Discussion:
[openstack-dev] Should we add a tempest-slow job?
Ghanshyam Mann
2018-05-13 04:24:23 UTC
Permalink
The tempest-full job used to run API and scenario tests concurrently, and if
you go back far enough I think it also ran slow tests.
Well it's a bit more subtle than that. Skipping slow tests was added right
https://github.com/openstack/tempest/commit/68a8060b24abd6b6bf99c4f9296bf418a8349a2d
Note those are in separate testr jobs which we migrated to the full job a bit
later in that cycle. The full job back then ran using nose and ran things
serially. But back then we didn't actually have any tests tagged as slow. It was
more of a future proofing thing because we were planning to add a bunch of
really slow heat tests we didn't want to run on every commit to each project.
The slow tags were first added for heat tests which came later in the havana
cycle.
Sometime in the last year or so, the full job was changed to run the
scenario tests in serial and exclude the slow tests altogether. So the API
tests run concurrently first, and then the scenario tests run in serial.
During that change, some other tests were identified as 'slow' and marked as
such, meaning they don't get run in the normal tempest-full job.
https://github.com/openstack/tempest/commit/49505df20f3dc578506e479c2afa4a4f02e464bf
There are some valuable scenario tests marked as slow, however, like the
only encrypted volume testing we have in tempest is marked slow so it
doesn't get run on every change for at least nova.
There is only one job that can be run against nova changes which runs the
slow tests but it's in the experimental queue so people forget to run it.
As a test, I've proposed a nova-slow job [1] which only runs the slow tests
and only the compute API and scenario tests. Since there currently no
compute API tests marked as slow, it's really just running slow scenario
tests. Results show it runs 37 tests in about 37 minutes [2]. The overall
job runtime was 1 hour and 9 minutes, which is on average less than the
tempest-full job. The nova-slow job is also running scenarios that nova
patches don't actually care about, like the neutron IPv6 scenario tests.
My question is, should we make this a generic tempest-slow job which can be
run either in the integrated-gate or at least in nova/neutron/cinder
consistently (I'm not sure if there are slow tests for just keystone or
glance)? I don't know if the other projects already have something like this
that they gate on. If so, a nova-specific job for nova changes is fine for
me.
So there used to be an experimental queue tempest-all job which ran everything
in tempest, including the slow tests. I can't find it in the .zuul.yaml in the
tempest repo, so my assumption is that got dropped during the v3 migration.
It is there with name "legacy-periodic-tempest-dsvm-all-master" [3].
This runs as experimental and periodic for Tempest. It is not yet
migrated, i will plan to migrate that in tempest repo.
I'm fine with adding a general purpose job for just running the slow tests to
the integrated gate if we think there is enough value from that. It's mostly
just a question of weighing the potential value from the increased coverage vs
the increased resource consumption for adding yet another job to the integrated
gate. Personally, I'm fine with that tradeoff.
-Matt Treinish
[1] https://review.openstack.org/#/c/567697/
[2] http://logs.openstack.org/97/567697/1/check/nova-slow/bedfafb/job-output.txt.gz#_2018-05-10_23_46_47_588138
..3 http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-jobs.yaml#n1579

-gmann
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/ma
Ghanshyam Mann
2018-05-13 04:20:55 UTC
Permalink
The tempest-full job used to run API and scenario tests concurrently, and if
you go back far enough I think it also ran slow tests.
Sometime in the last year or so, the full job was changed to run the
scenario tests in serial and exclude the slow tests altogether. So the API
tests run concurrently first, and then the scenario tests run in serial.
During that change, some other tests were identified as 'slow' and marked as
such, meaning they don't get run in the normal tempest-full job.
There are some valuable scenario tests marked as slow, however, like the
only encrypted volume testing we have in tempest is marked slow so it
doesn't get run on every change for at least nova.
Yes, basically slow tests were selected based on
https://ethercalc.openstack.org/nu56u2wrfb2b and there were frequent
gate failure for heavy tests mainly from ssh checks so we tried to
mark more tests as slow.
I agree that some of them are not really slow at least in today situation.
There is only one job that can be run against nova changes which runs the
slow tests but it's in the experimental queue so people forget to run it.
Tempest job "legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend"
run those slow tests including migration and LVM multibackend tests.
This job runs on tempest check pipeline and experimental (as you
mentioned) on nova and cinder [3]. We marked this as n-v to check its
stability and now it is good to go as voting on tempest.
As a test, I've proposed a nova-slow job [1] which only runs the slow tests
and only the compute API and scenario tests. Since there currently no
compute API tests marked as slow, it's really just running slow scenario
tests. Results show it runs 37 tests in about 37 minutes [2]. The overall
job runtime was 1 hour and 9 minutes, which is on average less than the
tempest-full job. The nova-slow job is also running scenarios that nova
patches don't actually care about, like the neutron IPv6 scenario tests.
My question is, should we make this a generic tempest-slow job which can be
run either in the integrated-gate or at least in nova/neutron/cinder
consistently (I'm not sure if there are slow tests for just keystone or
glance)? I don't know if the other projects already have something like this
that they gate on. If so, a nova-specific job for nova changes is fine for
me.
+1 on idea. As of now slow marked tests are from nova, cinder and
neutron scenario tests and 2 API swift tests only [4]. I agree that
making a generic job in tempest is better for maintainability. We can
use existing job for that with below modification-
- We can migrate
"legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend" job
zuulv3 in tempest repo
- We can see if we can move migration tests out of it and use
"nova-live-migration" job (in tempest check pipeline ) which is much
better in live migration env setup and controlled by nova.
- then it can be name something like
"tempest-scenario-multinode-lvm-multibackend".
- run this job in nova, cinder, neutron check pipeline instead of experimental.

Another update on slow tests is that we are trying the possibility of
taking back the slow tests in tempest-full with new job
"tempest-full-parallel" [5]. Currently this job is n-v and if
everything works fine in this new job then, we can make tempest-full
job to run the slow tests are it used to do previously.
[1] https://review.openstack.org/#/c/567697/
[2]
http://logs.openstack.org/97/567697/1/check/nova-slow/bedfafb/job-output.txt.gz#_2018-05-10_23_46_47_588138
..3 http://codesearch.openstack.org/?q=legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend&i=nope&files=&repos=
..4 https://github.com/openstack/tempest/search?utf8=%E2%9C%93&q=%22type%3D%27slow%27%22&type=
..5 https://github.com/openstack/tempest/blob/9c628189e798f46de8c4b9484237f4d6dc6ade7e/.zuul.yaml#L48


-gmann
--
Thanks,
Matt
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.o
Ghanshyam Mann
2018-05-14 02:06:28 UTC
Permalink
Post by Ghanshyam Mann
The tempest-full job used to run API and scenario tests concurrently, and if
you go back far enough I think it also ran slow tests.
Sometime in the last year or so, the full job was changed to run the
scenario tests in serial and exclude the slow tests altogether. So the API
tests run concurrently first, and then the scenario tests run in serial.
During that change, some other tests were identified as 'slow' and marked as
such, meaning they don't get run in the normal tempest-full job.
There are some valuable scenario tests marked as slow, however, like the
only encrypted volume testing we have in tempest is marked slow so it
doesn't get run on every change for at least nova.
Yes, basically slow tests were selected based on
https://ethercalc.openstack.org/nu56u2wrfb2b and there were frequent
gate failure for heavy tests mainly from ssh checks so we tried to
mark more tests as slow.
I agree that some of them are not really slow at least in today situation.
There is only one job that can be run against nova changes which runs the
slow tests but it's in the experimental queue so people forget to run it.
Tempest job "legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend"
run those slow tests including migration and LVM multibackend tests.
This job runs on tempest check pipeline and experimental (as you
mentioned) on nova and cinder [3]. We marked this as n-v to check its
stability and now it is good to go as voting on tempest.
As a test, I've proposed a nova-slow job [1] which only runs the slow tests
and only the compute API and scenario tests. Since there currently no
compute API tests marked as slow, it's really just running slow scenario
tests. Results show it runs 37 tests in about 37 minutes [2]. The overall
job runtime was 1 hour and 9 minutes, which is on average less than the
tempest-full job. The nova-slow job is also running scenarios that nova
patches don't actually care about, like the neutron IPv6 scenario tests.
My question is, should we make this a generic tempest-slow job which can be
run either in the integrated-gate or at least in nova/neutron/cinder
consistently (I'm not sure if there are slow tests for just keystone or
glance)? I don't know if the other projects already have something like this
that they gate on. If so, a nova-specific job for nova changes is fine for
me.
+1 on idea. As of now slow marked tests are from nova, cinder and
neutron scenario tests and 2 API swift tests only [4]. I agree that
making a generic job in tempest is better for maintainability. We can
use existing job for that with below modification-
- We can migrate
"legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend" job
zuulv3 in tempest repo
- We can see if we can move migration tests out of it and use
"nova-live-migration" job (in tempest check pipeline ) which is much
better in live migration env setup and controlled by nova.
- then it can be name something like
"tempest-scenario-multinode-lvm-multibackend".
- run this job in nova, cinder, neutron check pipeline instead of experimental.
Like this - https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:scenario-tests-job

That makes scenario job as generic with running all scenario tests
including slow tests with concurrency 2. I made few cleanup and moved
live migration tests out of it which is being run by
'nova-live-migration' job. Last patch making this job as voting on
tempest side.

If looks good, we can use this to run on project side pipeline as voting.

-gmann
Post by Ghanshyam Mann
Another update on slow tests is that we are trying the possibility of
taking back the slow tests in tempest-full with new job
"tempest-full-parallel" [5]. Currently this job is n-v and if
everything works fine in this new job then, we can make tempest-full
job to run the slow tests are it used to do previously.
[1] https://review.openstack.org/#/c/567697/
[2]
http://logs.openstack.org/97/567697/1/check/nova-slow/bedfafb/job-output.txt.gz#_2018-05-10_23_46_47_588138
..3 http://codesearch.openstack.org/?q=legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend&i=nope&files=&repos=
..4 https://github.com/openstack/tempest/search?utf8=%E2%9C%93&q=%22type%3D%27slow%27%22&type=
..5 https://github.com/openstack/tempest/blob/9c628189e798f46de8c4b9484237f4d6dc6ade7e/.zuul.yaml#L48
-gmann
--
Thanks,
Matt
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.ope

Loading...