[openstack-dev] [all][requirements] uncapping eventlet

Post by Matthew Thode
eventlet-0.22.1 has been out for a while now, we should try and use it.
Going to be fun times.
I have a review projects can depend upon if they wish to test.
https://review.openstack.org/533021

It looks like we may have an issue with oslo.service -
https://review.openstack.org/#/c/559144/ is failing gates.

Also - what is the dance for this to get merged? It doesn't look like we
can merge this while oslo.service has the old requirement restrictions.

Post by Matthew Thode
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.

Matthew Thode

2018-04-05 19:26:19 UTC

It looks like we may have an issue with oslo.service -
https://review.openstack.org/#/c/559144/ is failing gates.
Also - what is the dance for this to get merged? It doesn't look like we
can merge this while oslo.service has the old requirement restrictions.

The dance is as follows.

0. provide review for projects to test new eventlet version
projects using eventlet should make backwards compat code changes at
this time.
1. uncap requirements for eventlet (do not raise upper constraint)
step 0 does not have to be done for this to occur, but it'd be nice.
2. make sure all projects in projects.txt uncap eventlet (this is harder
now that we have per-project requirements)
3. raise the constraint for eventlet, optionally also raise the global
requirement for it as well

--
Matthew Thode (prometheanfire)

Jens Harbott

2018-04-06 09:02:29 UTC

It looks like we may have an issue with oslo.service -
https://review.openstack.org/#/c/559144/ is failing gates.
Also - what is the dance for this to get merged? It doesn't look like we
can merge this while oslo.service has the old requirement restrictions.

The dance is as follows.
0. provide review for projects to test new eventlet version
projects using eventlet should make backwards compat code changes at
this time.

But this step is currently failing. Keystone doesn't even start when
eventlet-0.22.1 is installed, because loading oslo.service fails with
its pkg definition still requiring the capped eventlet:

http://logs.openstack.org/21/533021/4/check/legacy-requirements-integration-dsvm/7f7c3a8/logs/screen-keystone.txt.gz#_Apr_05_16_11_27_748482

So it looks like we need to have an uncapped release of oslo.service
before we can proceed here.

Matthew Thode

2018-04-06 16:34:33 UTC

It looks like we may have an issue with oslo.service -
https://review.openstack.org/#/c/559144/ is failing gates.
Also - what is the dance for this to get merged? It doesn't look like we
can merge this while oslo.service has the old requirement restrictions.

The dance is as follows.
0. provide review for projects to test new eventlet version
projects using eventlet should make backwards compat code changes at
this time.

But this step is currently failing. Keystone doesn't even start when
eventlet-0.22.1 is installed, because loading oslo.service fails with
http://logs.openstack.org/21/533021/4/check/legacy-requirements-integration-dsvm/7f7c3a8/logs/screen-keystone.txt.gz#_Apr_05_16_11_27_748482
So it looks like we need to have an uncapped release of oslo.service
before we can proceed here.

Ya, we may have to uncap and rely on upper-constraints to keep openstack
gate from falling over. The new steps would be the following:

1. uncap eventlet https://review.openstack.org/559367
2. push uncapped eventlet out via requirements updates to all consumers
3. make review in requirements changing upper-constraints.txt for
eventlet
4. projects depend on requirements change to do work on the new eventlet
the patch generated should merge into project without the
requirements change merged (this means the change should pass in the
dependant review (to test 0.22.1) AND in a separate non-dependant
review (test the current constraint). You would merge the
non-dependant once both reviews are passing.
5. Once some non-determined set of projects work with the new eventlet
we'd merge the updated upper-constraint into requirements.

steps 2 and 3 can happen in parallel, projects can move to step 4 after
step 3 is done (step 2 is only needed for their project and their
project's dependencies).

There is bound to be projects that will break as they didn't take the
opportunity to fix themselves, but this should help reduce breakage. I
suggest a 1 month deadline after step 2/3 is considered complete before
step 5 is performed.

--
Matthew Thode (prometheanfire)

Clark Boylan

2018-04-06 16:41:07 UTC

It looks like we may have an issue with oslo.service -
https://review.openstack.org/#/c/559144/ is failing gates.
Also - what is the dance for this to get merged? It doesn't look like we
can merge this while oslo.service has the old requirement restrictions.

The dance is as follows.
0. provide review for projects to test new eventlet version
projects using eventlet should make backwards compat code changes at
this time.

Ya, we may have to uncap and rely on upper-constraints to keep openstack

My understanding of our use of upper constraints was that this should (almost) always be the case for (almost) all dependencies. We should rely on constraints instead of requirements caps. Capping libs like pbr or eventlet and any other that is in use globally is incredibly difficult to work with when you want to uncap it because you have to coordinate globally. Instead if using constraints you just bump the constraint and are done.

It is probably worthwhile examining if we have any other deps in the situation and proactively addressing them rather than waiting for when we really need to fix them.

Clark

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailm

Matthew Thode

2018-04-06 16:45:57 UTC

Post by Clark Boylan

It looks like we may have an issue with oslo.service -
https://review.openstack.org/#/c/559144/ is failing gates.
Also - what is the dance for this to get merged? It doesn't look like we
can merge this while oslo.service has the old requirement restrictions.

The dance is as follows.
0. provide review for projects to test new eventlet version
projects using eventlet should make backwards compat code changes at
this time.

Ya, we may have to uncap and rely on upper-constraints to keep openstack

My understanding of our use of upper constraints was that this should (almost) always be the case for (almost) all dependencies. We should rely on constraints instead of requirements caps. Capping libs like pbr or eventlet and any other that is in use globally is incredibly difficult to work with when you want to uncap it because you have to coordinate globally. Instead if using constraints you just bump the constraint and are done.
It is probably worthwhile examining if we have any other deps in the situation and proactively addressing them rather than waiting for when we really need to fix them.

That's constantly on our list of things to do. In the past the only
time we've capped is when we know upstream is realeasing breaking
versions and we want to hold off for a cycle or until it's fixed. It
also has the benefit of telling consumers/packagers about something
'hard breaking'.

networkx is next on the list...

--
Matthew Thode (prometheanfire)

Tony Breeds

2018-04-09 03:39:30 UTC

Post by Clark Boylan
My understanding of our use of upper constraints was that this should
(almost) always be the case for (almost) all dependencies. We should
rely on constraints instead of requirements caps. Capping libs like
pbr or eventlet and any other that is in use globally is incredibly
difficult to work with when you want to uncap it because you have to
coordinate globally. Instead if using constraints you just bump the
constraint and are done.

Part of the reason that we have the caps it to prevent the tools that
auto-generate the constraints syncs from considering these versions and
then depending on the requirements team to strip that from the bot
change before committing (assuming it passes CI).

Once the work Doug's doing is complete we could consider tweaking the
tools to use a different mechanism, but that's only part of the reason
for the caps in g-r.

Yours Tony.

Doug Hellmann

2018-04-09 13:58:28 UTC

Part of the reason that we have the caps it to prevent the tools that
auto-generate the constraints syncs from considering these versions and
then depending on the requirements team to strip that from the bot
change before committing (assuming it passes CI).
Once the work Doug's doing is complete we could consider tweaking the
tools to use a different mechanism, but that's only part of the reason
for the caps in g-r.
Yours Tony.

Now that projects don't have to match the global requirements list
entries exactly we should be able to remove caps from within the
projects and keep caps in the global list for cases like this where we
know we frequently encounter breaking changes in new releases. The
changes to support that were part of
https://review.openstack.org/#/c/555402/

Doug

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cg

Tony Breeds

2018-04-10 05:50:26 UTC

Post by Doug Hellmann
Now that projects don't have to match the global requirements list
entries exactly we should be able to remove caps from within the
projects and keep caps in the global list for cases like this where we
know we frequently encounter breaking changes in new releases. The
changes to support that were part of
https://review.openstack.org/#/c/555402/

True. I was trying to add context to why we don't always rely on
upper-constraints.txt to save us. So yeah we can start working towards
removing the caps per project.

Yours Tony.

IWAMOTO Toshihiro

2018-04-11 09:19:02 UTC

On Mon, 09 Apr 2018 22:58:28 +0900,

Part of the reason that we have the caps it to prevent the tools that
auto-generate the constraints syncs from considering these versions and
then depending on the requirements team to strip that from the bot
change before committing (assuming it passes CI).
Once the work Doug's doing is complete we could consider tweaking the
tools to use a different mechanism, but that's only part of the reason
for the caps in g-r.
Yours Tony.

As eventlet has been uncapped in g-r, requirements-check is
complaining on unrelated project-local requirement changes.
I'm not quite sure but doesn't seem to be a intended behavior.

http://logs.openstack.org/57/451257/16/check/requirements-check/c32ee69/job-output.txt.gz

--
IWAMOTO Toshihiro

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.openstac

Doug Hellmann

2018-04-11 12:55:52 UTC

Post by IWAMOTO Toshihiro
On Mon, 09 Apr 2018 22:58:28 +0900,

Part of the reason that we have the caps it to prevent the tools that
auto-generate the constraints syncs from considering these versions and
then depending on the requirements team to strip that from the bot
change before committing (assuming it passes CI).
Once the work Doug's doing is complete we could consider tweaking the
tools to use a different mechanism, but that's only part of the reason
for the caps in g-r.
Yours Tony.

This error is related to the change in
https://review.openstack.org/#/c/560050/ which applies the matching
rules to all requirements settings any time any requirements-related
file is touched. The change was made because we are less in-sync than we
thought and because we're allowing "bad" settings to stay in place.

To correct the problem in the log you linked to, remove the cap from
eventlet in neutron.

Doug

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listin

Chen CH Ji

2018-04-11 13:09:29 UTC

sorry, I didn't see any solution for following error found in [1]
I just rechecked the patch and is this kind of issue already fixed?

ubuntu-xenial | Requirement for package eventlet : Requirement
(package=u'eventlet', location='',
specifiers='!=0.18.3,!=0.20.1,<0.21.0,>=0.18.2', markers=u'', comment=u'#
MIT', extras=frozenset([])) does not match openstack/requirements value :
set([Requirement(package='eventlet', location='',
specifiers='!=0.18.3,!=0.20.1,>=0.18.2', markers='', comment='# MIT',
extras=frozenset([]))])

[1]
logs.openstack.org/87/523387/32/check/requirements-check/408e28c/job-output.txt.gz

Best Regards!

Kevin (Chen) Ji ŒÍ ³¿

Engineer, zVM Development, CSTL
Notes: Chen CH Ji/China/***@IBMCN Internet: ***@cn.ibm.com
Phone: +86-10-82451493
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District,
Beijing 100193, PRC

From: Doug Hellmann <***@doughellmann.com>
To: openstack-dev <openstack-***@lists.openstack.org>
Date: 04/11/2018 08:56 PM
Subject: Re: [openstack-dev] [all][requirements] uncapping eventlet

Post by IWAMOTO Toshihiro
On Mon, 09 Apr 2018 22:58:28 +0900,

Part of the reason that we have the caps it to prevent the tools that
auto-generate the constraints syncs from considering these versions and
then depending on the requirements team to strip that from the bot
change before committing (assuming it passes CI).
Once the work Doug's doing is complete we could consider tweaking the
tools to use a different mechanism, but that's only part of the reason
for the caps in g-r.
Yours Tony.

https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_555402_&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=8sI5aZT88Uetyy_XsOddbPjIiLSGM-sFnua3lLy2Xr0&m=y9YWvP5nDDQCyw3QGNWGvQS-CVeHBeXA9rfHaLf3JpQ&s=P99Z7BlpiP8Sg9_5Ku4JMW_tJWXARpd2ldSvFFlFBpU&e=

Post by IWAMOTO Toshihiro
As eventlet has been uncapped in g-r, requirements-check is
complaining on unrelated project-local requirement changes.
I'm not quite sure but doesn't seem to be a intended behavior.

https://urldefense.proofpoint.com/v2/url?u=http-3A__logs.openstack.org_57_451257_16_check_requirements-2Dcheck_c32ee69_job-2Doutput.txt.gz&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=8sI5aZT88Uetyy_XsOddbPjIiLSGM-sFnua3lLy2Xr0&m=y9YWvP5nDDQCyw3QGNWGvQS-CVeHBeXA9rfHaLf3JpQ&s=6uHgERcFttsqFakjBTrjvKZhk5n-tZO-e0QMd7zj0nc&e=
This error is related to the change in
https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_560050_&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=8sI5aZT88Uetyy_XsOddbPjIiLSGM-sFnua3lLy2Xr0&m=y9YWvP5nDDQCyw3QGNWGvQS-CVeHBeXA9rfHaLf3JpQ&s=1hIA6J9OfM1mhcTDq89NkGmoAQi_fDfhel7q5dgcwIA&e=
which applies the matching
rules to all requirements settings any time any requirements-related
file is touched. The change was made because we are less in-sync than we
thought and because we're allowing "bad" settings to stay in place.

To correct the problem in the log you linked to, remove the cap from
eventlet in neutron.

Doug

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Ddev&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=8sI5aZT88Uetyy_XsOddbPjIiLSGM-sFnua3lLy2Xr0&m=y9YWvP5nDDQCyw3QGNWGvQS-CVeHBeXA9rfHaLf3JpQ&s=nEWykx9Cfm6deVwO9Sdge-_Q31mCbdfAmvp_KoPaenc&e=

Doug Hellmann

2018-04-11 13:23:45 UTC

Post by Chen CH Ji
sorry, I didn't see any solution for following error found in [1]
I just rechecked the patch and is this kind of issue already fixed?
ubuntu-xenial | Requirement for package eventlet : Requirement
(package=u'eventlet', location='',
specifiers='!=0.18.3,!=0.20.1,<0.21.0,>=0.18.2', markers=u'', comment=u'#
set([Requirement(package='eventlet', location='',
specifiers='!=0.18.3,!=0.20.1,>=0.18.2', markers='', comment='# MIT',
extras=frozenset([]))])

The error message is correct. The requirements specification does not
match and needs to be fixed by removing the cap from eventlet.

Doug

Post by Chen CH Ji
[1]
logs.openstack.org/87/523387/32/check/requirements-check/408e28c/job-output.txt.gz
Best Regards!
Kevin (Chen) Ji 纪晨
Engineer, zVM Development, CSTL
Phone: +86-10-82451493
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District,
Beijing 100193, PRC
Date: 04/11/2018 08:56 PM
Subject: Re: [openstack-dev] [all][requirements] uncapping eventlet

Post by IWAMOTO Toshihiro
On Mon, 09 Apr 2018 22:58:28 +0900,

Post by Clark Boylan
My understanding of our use of upper constraints was that this

should

Post by Clark Boylan
(almost) always be the case for (almost) all dependencies. We

should

Post by Clark Boylan
rely on constraints instead of requirements caps. Capping libs like
pbr or eventlet and any other that is in use globally is incredibly
difficult to work with when you want to uncap it because you have

Post by Clark Boylan
coordinate globally. Instead if using constraints you just bump the
constraint and are done.

Part of the reason that we have the caps it to prevent the tools that
auto-generate the constraints syncs from considering these versions

and

Post by Tony Breeds
then depending on the requirements team to strip that from the bot
change before committing (assuming it passes CI).
Once the work Doug's doing is complete we could consider tweaking the
tools to use a different mechanism, but that's only part of the

reason

Post by Tony Breeds
for the caps in g-r.
Yours Tony.

Ben Nemec

2018-04-09 18:12:30 UTC

It looks like we may have an issue with oslo.service -
https://review.openstack.org/#/c/559144/ is failing gates.
Also - what is the dance for this to get merged? It doesn't look like we
can merge this while oslo.service has the old requirement restrictions.

The dance is as follows.
0. provide review for projects to test new eventlet version
projects using eventlet should make backwards compat code changes at
this time.

I've proposed a patch[1] to uncap eventlet in oslo.service, but it's
failing the unit tests[2]. I'll look into it, but I thought I'd provide
an update in the meantime.

1: https://review.openstack.org/559800
2:
http://logs.openstack.org/00/559800/1/check/openstack-tox-py27/cef8fcb/job-output.txt.gz

Ben Nemec

2018-04-09 21:32:28 UTC

Post by Ben Nemec

It looks like we may have an issue with oslo.service -
https://review.openstack.org/#/c/559144/ is failing gates.
Also - what is the dance for this to get merged? It doesn't look like we
can merge this while oslo.service has the old requirement restrictions.

The dance is as follows.
0. provide review for projects to test new eventlet version
projects using eventlet should make backwards compat code changes at
this time.

I've proposed a patch[1] to uncap eventlet in oslo.service, but it's
failing the unit tests[2]. I'll look into it, but I thought I'd provide
an update in the meantime.

Oh, the unit test failures are unrelated. Apparently the unit tests
have been failing in oslo.service for a while. dims has a patch up at
https://review.openstack.org/#/c/559831/ that looks to be addressing the
problem, although it's also failing the unit tests. :-/

Post by Ben Nemec
1: https://review.openstack.org/559800
http://logs.openstack.org/00/559800/1/check/openstack-tox-py27/cef8fcb/job-output.txt.gz
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Ben Nemec

2018-04-10 21:47:48 UTC

Post by Ben Nemec

It looks like we may have an issue with oslo.service -
https://review.openstack.org/#/c/559144/ is failing gates.
Also - what is the dance for this to get merged? It doesn't look like we
can merge this while oslo.service has the old requirement
restrictions.

The dance is as follows.
0. provide review for projects to test new eventlet version
projects using eventlet should make backwards compat code changes at
this time.

I've proposed a patch[1] to uncap eventlet in oslo.service, but it's
failing the unit tests[2]. I'll look into it, but I thought I'd
provide an update in the meantime.

We finally got the uncap patch merged and a release request is up at
https://review.openstack.org/560163. Hopefully once that is in u-c
we'll be past this issue.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Doug Hellmann

2018-04-11 16:20:46 UTC

I have proposed a bunch of patches to projects to remove the cap
for eventlet [1]. If they don't pass tests, please take them over
and fix them up as needed (I anticipate some trouble with the new
check-requirements rules, for example).

Doug

[1] https://review.openstack.org/#/q/topic:uncap-eventlet+(status:open+OR+status:merged)

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/m

Doug Hellmann

2018-04-11 16:26:37 UTC

And please go ahead and abandon any that are duplicates for patches that
are already being worked on elsewhere. It was easier to just update
everything than to script something to figure out when updates were
needed.

Doug

Doug Hellmann

2018-04-20 20:06:17 UTC

We have made great progress on this but we do still have quite a
few of these patches that have not been approved. Many are failing
test jobs and will need a little attention ( the failing requirements
jobs are real problems and rechecking will not fix them). If you
have time to help, please jump in and take over a patch and get it
working.

https://review.openstack.org/#/q/status:open+topic:uncap-eventlet

Thanks,
Doug

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.openstack.o

Michel Peterson

2018-05-07 14:54:02 UTC