Ghanshyam Mann
2018-05-13 04:24:23 UTC
The tempest-full job used to run API and scenario tests concurrently, and if
you go back far enough I think it also ran slow tests.
Well it's a bit more subtle than that. Skipping slow tests was added rightyou go back far enough I think it also ran slow tests.
https://github.com/openstack/tempest/commit/68a8060b24abd6b6bf99c4f9296bf418a8349a2d
Note those are in separate testr jobs which we migrated to the full job a bit
later in that cycle. The full job back then ran using nose and ran things
serially. But back then we didn't actually have any tests tagged as slow. It was
more of a future proofing thing because we were planning to add a bunch of
really slow heat tests we didn't want to run on every commit to each project.
The slow tags were first added for heat tests which came later in the havana
cycle.
Sometime in the last year or so, the full job was changed to run the
scenario tests in serial and exclude the slow tests altogether. So the API
tests run concurrently first, and then the scenario tests run in serial.
During that change, some other tests were identified as 'slow' and marked as
such, meaning they don't get run in the normal tempest-full job.
https://github.com/openstack/tempest/commit/49505df20f3dc578506e479c2afa4a4f02e464bfscenario tests in serial and exclude the slow tests altogether. So the API
tests run concurrently first, and then the scenario tests run in serial.
During that change, some other tests were identified as 'slow' and marked as
such, meaning they don't get run in the normal tempest-full job.
There are some valuable scenario tests marked as slow, however, like the
only encrypted volume testing we have in tempest is marked slow so it
doesn't get run on every change for at least nova.
There is only one job that can be run against nova changes which runs the
slow tests but it's in the experimental queue so people forget to run it.
As a test, I've proposed a nova-slow job [1] which only runs the slow tests
and only the compute API and scenario tests. Since there currently no
compute API tests marked as slow, it's really just running slow scenario
tests. Results show it runs 37 tests in about 37 minutes [2]. The overall
job runtime was 1 hour and 9 minutes, which is on average less than the
tempest-full job. The nova-slow job is also running scenarios that nova
patches don't actually care about, like the neutron IPv6 scenario tests.
My question is, should we make this a generic tempest-slow job which can be
run either in the integrated-gate or at least in nova/neutron/cinder
consistently (I'm not sure if there are slow tests for just keystone or
glance)? I don't know if the other projects already have something like this
that they gate on. If so, a nova-specific job for nova changes is fine for
me.
So there used to be an experimental queue tempest-all job which ran everythingonly encrypted volume testing we have in tempest is marked slow so it
doesn't get run on every change for at least nova.
There is only one job that can be run against nova changes which runs the
slow tests but it's in the experimental queue so people forget to run it.
As a test, I've proposed a nova-slow job [1] which only runs the slow tests
and only the compute API and scenario tests. Since there currently no
compute API tests marked as slow, it's really just running slow scenario
tests. Results show it runs 37 tests in about 37 minutes [2]. The overall
job runtime was 1 hour and 9 minutes, which is on average less than the
tempest-full job. The nova-slow job is also running scenarios that nova
patches don't actually care about, like the neutron IPv6 scenario tests.
My question is, should we make this a generic tempest-slow job which can be
run either in the integrated-gate or at least in nova/neutron/cinder
consistently (I'm not sure if there are slow tests for just keystone or
glance)? I don't know if the other projects already have something like this
that they gate on. If so, a nova-specific job for nova changes is fine for
me.
in tempest, including the slow tests. I can't find it in the .zuul.yaml in the
tempest repo, so my assumption is that got dropped during the v3 migration.
This runs as experimental and periodic for Tempest. It is not yet
migrated, i will plan to migrate that in tempest repo.
I'm fine with adding a general purpose job for just running the slow tests to
the integrated gate if we think there is enough value from that. It's mostly
just a question of weighing the potential value from the increased coverage vs
the increased resource consumption for adding yet another job to the integrated
gate. Personally, I'm fine with that tradeoff.
-Matt Treinish
..3 http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-jobs.yaml#n1579the integrated gate if we think there is enough value from that. It's mostly
just a question of weighing the potential value from the increased coverage vs
the increased resource consumption for adding yet another job to the integrated
gate. Personally, I'm fine with that tradeoff.
-Matt Treinish
[1] https://review.openstack.org/#/c/567697/
[2] http://logs.openstack.org/97/567697/1/check/nova-slow/bedfafb/job-output.txt.gz#_2018-05-10_23_46_47_588138
[2] http://logs.openstack.org/97/567697/1/check/nova-slow/bedfafb/job-output.txt.gz#_2018-05-10_23_46_47_588138
-gmann
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/ma