[openstack-dev] [nova][placement] Trying to summarize bp/glance-image-traits scheduling alternatives for rebuild

[Matt Riedemann]

We seem to be at a bit of an impasse in this spec amendment [1] so I
want to try and summarize the alternative solutions as I see them.

The overall goal of the blueprint is to allow defining traits via image
properties, like flavor extra specs. Those image-defined traits are used
to filter hosts during scheduling of the instance. During server create,
that filtering happens during the normal "GET /allocation_candidates"
call to placement.

The problem is during rebuild with a new image that specifies new
required traits. A rebuild is not a move operation, but we run through
the scheduler filters to make sure the new image (if one is specified),
is valid for the host on which the instance is currently running.

We don't currently call "GET /allocation_candidates" during rebuild
because that could inadvertently filter out the host we know we need
[2]. Also, since flavors don't change for rebuild, we haven't had a need
for getting allocation candidates during rebuild since we're not
allocating new resources (pretend bug 1763766 [3] does not exist for now).

Now that we know the problem, here are some of the solutions that have
been discussed in the spec amendment, again, only for rebuild with a new
image that has new traits:

1. Fail in the API saying you can't rebuild with a new image with new
required traits.

Pros:

- Simple way to keep the new image off a host that doesn't support it.
- Similar solution to volume-backed rebuild with a new image.

Cons:

- Confusing user experience since they might be able to rebuild with
some new images but not others with no clear explanation about the
difference.

2. Have the ImagePropertiesFilter call "GET
/resource_providers/{rp_uuid}/traits" and compare the compute node root
provider traits against the new image's required traits.

Pros:

- Avoids having to call "GET /allocation_candidates" during rebuild.
- Simple way to compare the required image traits against the compute
node provider traits.

Cons:

- Does not account for nested providers so the scheduler could reject
the image due to its required traits which actually apply to a nested
provider in the tree. This is somewhat related to bug 1763766.

3. Slight variation on #2 except build a set of all traits from all
providers in the same tree.

Pros:

- Handles the nested provider traits issue from #2.

Cons:

- Duplicates filtering in ImagePropertiesFilter that could otherwise
happen in "GET /allocation_candidates".

4. Add a microversion to change "GET /allocation_candidates" to make two
changes:

a) Add an "in_tree" filter like in "GET /resource_providers". This would
be needed to limit the scope of what gets returned since we know we only
want to check against one specific host (the current host for the instance).

b) Make "resources" optional since on a rebuild we don't want to
allocate new resources (again, notwithstanding bug 1763766).

Pros:

- We can call "GET /allocation_candidates?in_tree=<current node rp
UUID>&required=<new image required traits>" and if nothing is returned,
we know the new image's required traits don't work with the current node.
- The filtering is baked into "GET /allocation_candidates" and not
client-side in ImagePropertiesFilter.

Cons:

- Changes to the "GET /allocation_candidates" API which is going to be
more complicated and more up-front work, but I don't have a good idea of
how hard this would be to add since we already have the same "in_tree"
logic in "GET /resource_providers".
- Potentially slows down the completion of the overall blueprint.

===========

My personal thoughts are, I don't like option 1 since it adds technical
debt which we'll eventually just need to solve later (think about [4]).
Similar feelings for #2. #3 might be a short-term solution until #4 is
done, but I think the best long-term solution to this problem is #4.

[1] https://review.openstack.org/#/c/560718/
[2] https://review.openstack.org/#/c/546357/
[3] https://bugs.launchpad.net/nova/+bug/1763766
[4] https://review.openstack.org/#/c/532407/

[Eric Fried]

Semantically, GET /allocation_candidates where we don't actually want to
allocate anything (i.e. we don't want to use the returned candidates) is
goofy, and talking about what the result would look like when there's no
`resources` is going to spider into some weird questions.

Like what does the response payload look like? In the "good" scenario,
you would be expecting an allocation_request like:

"allocations": {
$rp_uuid: {
"resources": {
# Nada
}
},
}

...which is something we discussed recently [1] in relation to "anchor"
providers, and killed.

No, the question you're really asking in this case is, "Do the resource
providers in this tree contain (or not contain) these traits?" Which to
me, translates directly to:

GET /resource_providers?in_tree=$rp_uuid&required={$TRAIT|!$TRAIT, ...}

...which we already support. The answer is a list of providers. Compare
that to the providers from which resources are already allocated, and
Bob's your uncle.

(I do find it messy/weird that the required/forbidden traits in the
image meta are supposed to apply *anywhere* in the provider tree. But I
get that that's probably going to make the most sense.)

[1]
http://lists.openstack.org/pipermail/openstack-dev/2018-April/129408.html

Post by Matt Riedemann
We seem to be at a bit of an impasse in this spec amendment [1] so I
want to try and summarize the alternative solutions as I see them.
The overall goal of the blueprint is to allow defining traits via image
properties, like flavor extra specs. Those image-defined traits are used
to filter hosts during scheduling of the instance. During server create,
that filtering happens during the normal "GET /allocation_candidates"
call to placement.
The problem is during rebuild with a new image that specifies new
required traits. A rebuild is not a move operation, but we run through
the scheduler filters to make sure the new image (if one is specified),
is valid for the host on which the instance is currently running.
We don't currently call "GET /allocation_candidates" during rebuild
because that could inadvertently filter out the host we know we need
[2]. Also, since flavors don't change for rebuild, we haven't had a need
for getting allocation candidates during rebuild since we're not
allocating new resources (pretend bug 1763766 [3] does not exist for now).
Now that we know the problem, here are some of the solutions that have
been discussed in the spec amendment, again, only for rebuild with a new
1. Fail in the API saying you can't rebuild with a new image with new
required traits.
- Simple way to keep the new image off a host that doesn't support it.
- Similar solution to volume-backed rebuild with a new image.
- Confusing user experience since they might be able to rebuild with
some new images but not others with no clear explanation about the
difference.
2. Have the ImagePropertiesFilter call "GET
/resource_providers/{rp_uuid}/traits" and compare the compute node root
provider traits against the new image's required traits.
- Avoids having to call "GET /allocation_candidates" during rebuild.
- Simple way to compare the required image traits against the compute
node provider traits.
- Does not account for nested providers so the scheduler could reject
the image due to its required traits which actually apply to a nested
provider in the tree. This is somewhat related to bug 1763766.
3. Slight variation on #2 except build a set of all traits from all
providers in the same tree.
- Handles the nested provider traits issue from #2.
- Duplicates filtering in ImagePropertiesFilter that could otherwise
happen in "GET /allocation_candidates".
4. Add a microversion to change "GET /allocation_candidates" to make two
a) Add an "in_tree" filter like in "GET /resource_providers". This would
be needed to limit the scope of what gets returned since we know we only
want to check against one specific host (the current host for the instance).
b) Make "resources" optional since on a rebuild we don't want to
allocate new resources (again, notwithstanding bug 1763766).
- We can call "GET /allocation_candidates?in_tree=<current node rp
UUID>&required=<new image required traits>" and if nothing is returned,
we know the new image's required traits don't work with the current node.
- The filtering is baked into "GET /allocation_candidates" and not
client-side in ImagePropertiesFilter.
- Changes to the "GET /allocation_candidates" API which is going to be
more complicated and more up-front work, but I don't have a good idea of
how hard this would be to add since we already have the same "in_tree"
logic in "GET /resource_providers".
- Potentially slows down the completion of the overall blueprint.
===========
My personal thoughts are, I don't like option 1 since it adds technical
debt which we'll eventually just need to solve later (think about [4]).
Similar feelings for #2. #3 might be a short-term solution until #4 is
done, but I think the best long-term solution to this problem is #4.
[1] https://review.openstack.org/#/c/560718/
[2] https://review.openstack.org/#/c/546357/
[3] https://bugs.launchpad.net/nova/+bug/1763766
[4] https://review.openstack.org/#/c/532407/

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.o

[Matt Riedemann]

Post by Eric Fried
No, the question you're really asking in this case is, "Do the resource
providers in this tree contain (or not contain) these traits?" Which to
GET /resource_providers?in_tree=$rp_uuid&required={$TRAIT|!$TRAIT, ...}
...which we already support. The answer is a list of providers. Compare
that to the providers from which resources are already allocated, and
Bob's your uncle.

OK and that will include filtering the required traits on nested
providers in that tree rather than just against the root provider? If
so, then yeah that sounds like an improvement on option 2 or 3 in my
original email and resolves the issue without having to call (or change)
"GET /allocation_candidates". I still think it should happen from within
ImagePropertiesFilter, but that's an implementation detail.

[Eric Fried]

Following the discussion on IRC, here's what I think you need to do:

- Assuming the set of traits from your new image is called image_traits...
- Use GET /allocations/{instance_uuid} and pull out the set of all RP
UUIDs. Let's call this instance_rp_uuids.
- Use the SchedulerReportClient.get_provider_tree_and_ensure_root method
[1] to populate and return the ProviderTree for the host. (If we're
uncomfortable about the `ensure_root` bit, we can factor that away.)
Call this ptree.
- Collect all the traits in the RPs you've got allocated to your instance:

traits_in_instance_rps = set()
for rp_uuid in instance_rp_uuids:
traits_in_instance_rps.update(ptree.data(rp_uuid).traits)

- See if any of your image traits are *not* in those RPs.

missing_traits = image_traits - traits_in_instance_rps

- If there were any, it's a no go.

if missing_traits:
FAIL(_("The following traits were in the image but not in the
instance's RPs: %s") % ', '.join(missing_traits))

[1]
https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L986

Post by Matt Riedemann

Post by Eric Fried
No, the question you're really asking in this case is, "Do the resource
providers in this tree contain (or not contain) these traits?"  Which to
  GET /resource_providers?in_tree=$rp_uuid&required={$TRAIT|!$TRAIT, ...}
...which we already support.  The answer is a list of providers. Compare
that to the providers from which resources are already allocated, and
Bob's your uncle.

OK and that will include filtering the required traits on nested
providers in that tree rather than just against the root provider? If
so, then yeah that sounds like an improvement on option 2 or 3 in my
original email and resolves the issue without having to call (or change)
"GET /allocation_candidates". I still think it should happen from within
ImagePropertiesFilter, but that's an implementation detail.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/

[Jay Pipes]

Post by Matt Riedemann
We seem to be at a bit of an impasse in this spec amendment [1] so I
want to try and summarize the alternative solutions as I see them.
The overall goal of the blueprint is to allow defining traits via image
properties, like flavor extra specs. Those image-defined traits are used
to filter hosts during scheduling of the instance. During server create,
that filtering happens during the normal "GET /allocation_candidates"
call to placement.
The problem is during rebuild with a new image that specifies new
required traits. A rebuild is not a move operation, but we run through
the scheduler filters to make sure the new image (if one is specified),
is valid for the host on which the instance is currently running.

What you are discussing above is simple a validation that the compute
node performing the rebuild for an instance supports the capabilities
that were required by the original image.

How about just having the conductor call GET
/resource_providers?in_tree=<CN_UUID>&required=<IMAGE_TRAITS>, see if
there is a result, and if not, don't even call the scheduler at all
(because conductor would already know there would be a NoValidHost
returned)?

If there's no image traits, or if there is a result from GET
/resource_providers, continue to do the existing call-the-scheduler
behaviour in order to fulfill the ComputeCapabilitiesFilter and
ImageMetadataFilter requirements that exist today.

So, in short, just do a quick pre-flight check from the conductor if
image traits are found before ever calling the scheduler. Otherwise,
proceed as normal.

Best,
-jay

[Arvind N]

Thanks for the detailed options Matt/eric/jay.

Just few of my thoughts,

For #1, we can make the explanation very clear that we rejected the request
because the original traits specified in the original image and the new
traits specified in the new image do not match and hence rebuild is not
supported.

For #2,

Other Cons:

1. None of the filters currently make other API requests and my
understanding is we want to avoid reintroducing such a pattern. But
definitely workable solution.
2. If the user disables the image properties filter, then traits based
filtering will not be run in rebuild case

For #3,

Even though it handles the nested provider, there is a potential issue.

Lets say a host with two SRIOV nic. One is normal SRIOV nic(VF1), another
one with some kind of offload feature(VF2).(Described by alex)

Initial instance launch happens with VF:1 allocated, rebuild launches with
modified request with traits=HW_NIC_OFFLOAD_X, so basically we want the
instance to be allocated VF2.

But the original allocation happens against VF1 and since in rebuild the
original allocations are not changed, we have wrong allocations.

for #4, there is good amount of pushback against modifying the
allocation_candiadates api to not have resources.

Jay:
for the GET /resource_providers?in_tree=<CN_UUID>&required=<IMAGE_TRAITS>,
nested resource providers and allocation pose a problem see #3 above.

I will investigate erics option and update the spec.

[Eric Fried]

for the GET
/resource_providers?in_tree=<CN_UUID>&required=<IMAGE_TRAITS>, nested
resource providers and allocation pose a problem see #3 above.

This *would* work as a quick up-front check as Jay described (if you get
no results from this, you know that at least one of your image traits
doesn't exist anywhere in the tree) except that it doesn't take sharing
providers into account :(

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.openstac

[Arvind N]

Post by Matt Riedemann
1. Fail in the API saying you can't rebuild with a new image with new
required traits.

Pros: - Simple way to keep the new image off a host that doesn't support it.

Post by Matt Riedemann
- Similar solution to volume-backed rebuild with a new image.

Cons: - Confusing user experience since they might be able to rebuild with

Post by Matt Riedemann
some new images but not others with no clear explanation about the
difference.

Still want to get thoughts on Option 1 from the community, the only main
con can be addressed by a better error message.

My main concern is the amount of complexity being introduced now but also
what we are setting ourselfs up for the future.

When/If we decide to support forbidden traits, granular resource traits,
preferred traits etc based on image properties, we would have to handle all
those complexities for the rebuild case and possibly re-implement some of
the logic already within placement to handle these cases.

IMHO, i dont see a whole lot of benefit when weighing against the cost.
Feedback is appreciated. :)

Arvind

Post by Matt Riedemann

Post by Arvind N
for the GET
/resource_providers?in_tree=<CN_UUID>&required=<IMAGE_TRAITS>, nested
resource providers and allocation pose a problem see #3 above.

This *would* work as a quick up-front check as Jay described (if you get
no results from this, you know that at least one of your image traits
doesn't exist anywhere in the tree) except that it doesn't take sharing
providers into account :(
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Alex Xu]

Post by Arvind N
Thanks for the detailed options Matt/eric/jay.
Just few of my thoughts,
For #1, we can make the explanation very clear that we rejected the
request because the original traits specified in the original image and the
new traits specified in the new image do not match and hence rebuild is not
supported.
For #2,
1. None of the filters currently make other API requests and my
understanding is we want to avoid reintroducing such a pattern. But
definitely workable solution.
2. If the user disables the image properties filter, then traits based
filtering will not be run in rebuild case
For #3,
Even though it handles the nested provider, there is a potential issue.
Lets say a host with two SRIOV nic. One is normal SRIOV nic(VF1), another
one with some kind of offload feature(VF2).(Described by alex)
Initial instance launch happens with VF:1 allocated, rebuild launches with
modified request with traits=HW_NIC_OFFLOAD_X, so basically we want the
instance to be allocated VF2.
But the original allocation happens against VF1 and since in rebuild the
original allocations are not changed, we have wrong allocations.

Yes, that is the case what I said, and none of #1,2,3,4 and the proposal in
this threads works also.

The problem isn't just checking the traits in the nested resource provider.
We also need to ensure the trait in the exactly same child resource
provider. Or we need to adjust allocations for the child resource provider.

Post by Arvind N
for #4, there is good amount of pushback against modifying the
allocation_candiadates api to not have resources.
for the GET /resource_providers?in_tree=<CN_UUID>&required=<IMAGE_TRAITS>,
nested resource providers and allocation pose a problem see #3 above.
I will investigate erics option and update the spec.
--
Arvind N
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Balázs Gibizer]

Post by Alex Xu

Post by Arvind N
Thanks for the detailed options Matt/eric/jay.
Just few of my thoughts,
For #1, we can make the explanation very clear that we rejected the
request because the original traits specified in the original image
and the new traits specified in the new image do not match and hence
rebuild is not supported.
For #2,
None of the filters currently make other API requests and my
understanding is we want to avoid reintroducing such a pattern. But
definitely workable solution.
If the user disables the image properties filter, then traits based
filtering will not be run in rebuild case
For #3,
Even though it handles the nested provider, there is a potential issue.
Lets say a host with two SRIOV nic. One is normal SRIOV nic(VF1),
another one with some kind of offload feature(VF2).(Described by
alex)
Initial instance launch happens with VF:1 allocated, rebuild
launches with modified request with traits=HW_NIC_OFFLOAD_X, so
basically we want the instance to be allocated VF2.
But the original allocation happens against VF1 and since in rebuild
the original allocations are not changed, we have wrong allocations.

Yes, that is the case what I said, and none of #1,2,3,4 and the
proposal in this threads works also.
The problem isn't just checking the traits in the nested resource
provider. We also need to ensure the trait in the exactly same child
resource provider. Or we need to adjust allocations for the child
resource provider.

I agree that in_tree only ensure that the compute node tree has the
required traits but it does not take into account that only some of
those RPs from the tree provides resources for the current allocation.
The algorithm Eric provided in a previous mail do the filtering for the
RPs that are part of the instance allocation so that sounds good to me.

I think we should not try to adjust allocations during a rebuild.
Changing the allocation would mean it is not a rebuild any more but a
resize.

Cheers,
gibi

Post by Alex Xu

Post by Arvind N
for #4, there is good amount of pushback against modifying the
allocation_candiadates api to not have resources.
for the GET
/resource_providers?in_tree=<CN_UUID>&required=<IMAGE_TRAITS>,
nested resource providers and allocation pose a problem see #3 above.
I will investigate erics option and update the spec.
--
Arvind N
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Matt Riedemann]

Post by Balázs Gibizer
The algorithm Eric provided in a previous mail do the filtering for the
RPs that are part of the instance allocation so that sounds good to me.

Yeah I've been wondering if that solves this VF case.

Post by Balázs Gibizer
I think we should not try to adjust allocations during a rebuild.
Changing the allocation would mean it is not a rebuild any more but a
resize.

Agree.

[Eric Fried]

Post by Alex Xu
The problem isn't just checking the traits in the nested resource
provider. We also need to ensure the trait in the exactly same child
resource provider.

No, we can't get "granular" with image traits. We accepted this as a
limitation for the spawn aspect of this spec [1], for all the same
reasons [2]. And by the time we've spawned the instance, we've lost the
information about which granular request groups (from the flavor) were
satisfied by which resources - retrofitting that information from a new
image would be even harder. So we need to accept the same limitation
for rebuild.

[1] "Due to the difficulty of attempting to reconcile granular request
groups between an image and a flavor, only the (un-numbered) trait group
is supported. The traits listed there are merged with those of the
un-numbered request group from the flavor."
(http://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/glance-image-traits.html#proposed-change)
[2]
https://review.openstack.org/#/c/554305/2/specs/rocky/approved/glance-image-***@86

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailm

[Alex Xu]

Post by Eric Fried

Post by Alex Xu
The problem isn't just checking the traits in the nested resource
provider. We also need to ensure the trait in the exactly same child
resource provider.

No, we can't get "granular" with image traits. We accepted this as a
limitation for the spawn aspect of this spec [1], for all the same
reasons [2]. And by the time we've spawned the instance, we've lost the
information about which granular request groups (from the flavor) were
satisfied by which resources - retrofitting that information from a new
image would be even harder. So we need to accept the same limitation
for rebuild.
[1] "Due to the difficulty of attempting to reconcile granular request
groups between an image and a flavor, only the (un-numbered) trait group
is supported. The traits listed there are merged with those of the
un-numbered request group from the flavor."
(http://specs.openstack.org/openstack/nova-specs/specs/
rocky/approved/glance-image-traits.html#proposed-change)
[2]
https://review.openstack.org/#/c/554305/2/specs/rocky/

Why we can return a RP which has a specific trait but we won't consume any
resources on it?
If the case is that we request two VFs, and this two VFs have different
required traits, then that should be granular request.

Post by Eric Fried
__________________________________

________________________________________

Post by Eric Fried
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Eric Fried]

Alex-

Post by Alex Xu
The problem isn't just checking the traits in the nested resource
provider. We also need to ensure the trait in the exactly same child
resource provider.

No, we can't get "granular" with image traits.  We accepted this as a
limitation for the spawn aspect of this spec [1], for all the same
reasons [2].  And by the time we've spawned the instance, we've lost the
information about which granular request groups (from the flavor) were
satisfied by which resources - retrofitting that information from a new
image would be even harder.  So we need to accept the same limitation
for rebuild.
[1] "Due to the difficulty of attempting to reconcile granular request
groups between an image and a flavor, only the (un-numbered) trait group
is supported. The traits listed there are merged with those of the
un-numbered request group from the flavor."
(http://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/glance-image-traits.html#proposed-change
<http://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/glance-image-traits.html#proposed-change>)
[2]
Why we can return a RP which has a specific trait but we won't consume
any resources on it?
If the case is that we request two VFs, and this two VFs have different
required traits, then that should be granular request.

We don't care about RPs we're not consuming resources from. Forget
rebuild - if the image used for the original spawn request has traits
pertaining to VFs, we folded those traits into the un-numbered request
group, which means the VF resources would have needed to be in the
un-numbered request group in the flavor as well. That was the
limitation discussed at [2]: trying to correlate granular groups from an
image to granular groups in a trait would require nontrivial invention
beyond what we're willing to do at this point. So we're limited at
spawn time to VFs (or whatever) where we can't tell which trait belongs
to which. The best we can do is ensure that the end result of the
un-numbered request group will collectively satisfy all the traits from
the image. And this same limitation exists, for the same reasons, on
rebuild. It even goes a bit further, because if there are *other* VFs
(or whatever) that came from numbered groups in the original request, we
have no way to know that; so if *those* guys have traits required by the
new image, we'll still pass. Which is almost certainly okay.

-efried

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mai

[Jay Pipes]

Post by Arvind N
Thanks for the detailed options Matt/eric/jay.
Just few of my thoughts,
For #1, we can make the explanation very clear that we rejected the
request because the original traits specified in the original image and
the new traits specified in the new image do not match and hence rebuild
is not supported.

I believe I had suggested that on the spec amendment patch. Matt had
concerns about an error message being a poor user experience (I don't
necessarily disagree with that) and I had suggested a clearer error
message to try and make that user experience slightly less sucky.

Post by Arvind N
For #3,
Even though it handles the nested provider, there is a potential issue.
Lets say a host with two SRIOV nic. One is normal SRIOV nic(VF1),
another one with some kind of offload feature(VF2).(Described by alex)
Initial instance launch happens with VF:1 allocated, rebuild launches
with modified request with traits=HW_NIC_OFFLOAD_X, so basically we want
the instance to be allocated VF2.
But the original allocation happens against VF1 and since in rebuild the
original allocations are not changed, we have wrong allocations.

Yep, that is certainly an issue. The only solution to this that I can
see would be to have the conductor ask the compute node to do the
pre-flight check. The compute node already has the entire tree of
providers, their inventories and traits, along with information about
providers that share resources with the compute node. It has this
information in the ProviderTree object in the reportclient that is
contained in the compute node resource tracker.

The pre-flight check, if run on the compute node, would be able to grab
the allocation records for the instance and determine if the required
traits for the new image are present on the actual resource providers
allocated against for the instance (and not including any child
providers not allocated against).

Or... we chalk this up as a "too bad" situation and just either go with
option #1 or simply don't care about it.

Best,
-jay

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.o

[Sylvain Bauza]

Sorry folks for the late reply, I'll try to also weigh in the Gerrit change.

Post by Jay Pipes

Post by Arvind N
Thanks for the detailed options Matt/eric/jay.
Just few of my thoughts,
For #1, we can make the explanation very clear that we rejected the
request because the original traits specified in the original image and the
new traits specified in the new image do not match and hence rebuild is not
supported.

I believe I had suggested that on the spec amendment patch. Matt had
concerns about an error message being a poor user experience (I don't
necessarily disagree with that) and I had suggested a clearer error message
to try and make that user experience slightly less sucky.
For #3,

Post by Arvind N
Even though it handles the nested provider, there is a potential issue.
Lets say a host with two SRIOV nic. One is normal SRIOV nic(VF1), another
one with some kind of offload feature(VF2).(Described by alex)
Initial instance launch happens with VF:1 allocated, rebuild launches
with modified request with traits=HW_NIC_OFFLOAD_X, so basically we want
the instance to be allocated VF2.
But the original allocation happens against VF1 and since in rebuild the
original allocations are not changed, we have wrong allocations.

Yep, that is certainly an issue. The only solution to this that I can see
would be to have the conductor ask the compute node to do the pre-flight
check. The compute node already has the entire tree of providers, their
inventories and traits, along with information about providers that share
resources with the compute node. It has this information in the
ProviderTree object in the reportclient that is contained in the compute
node resource tracker.
The pre-flight check, if run on the compute node, would be able to grab
the allocation records for the instance and determine if the required
traits for the new image are present on the actual resource providers
allocated against for the instance (and not including any child providers
not allocated against).

Yup, that. We also have pre-flight checks for move operations like live and
cold migrations, and I'd really like to keep all the conditionals in the
conductor, because it knows better than the scheduler which operation is
asked.
I'm not really happy with adding more in the scheduler about "yeah, it's a
rebuild, so please do something exceptional", and I'm also not happy with
having a filter (that can be disabled) calling the Placement API.

Post by Jay Pipes
Or... we chalk this up as a "too bad" situation and just either go with
option #1 or simply don't care about it.

Also, that too. Maybe just provide an error should be enough, nope?
Operators, what do you think ? (cross-calling openstack-operators@)

-Sylvain

Post by Jay Pipes
Best,
-jay
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Arvind N]

Reminder for Operators, Please provide feedback either way.

In cases of rebuilding of an instance using a different image where the
image traits have changed between the original launch and the rebuild, is
it reasonable to ask to just re-launch a new instance with the new image?

The argument for this approach is that given that the requirements have
changed, we want the scheduler to pick and allocate the appropriate host
for the instance.

The approach above also gives you consistent results vs the other
approaches where the rebuild may or may not succeed depending on how the
original allocation of resources went.

For example(from Alex Xu) ,if you launched an instance on a host which has
two SRIOV nic. One is normal SRIOV nic(A), another one with some kind of
offload feature(B).

So, the original request is: resources=SRIOV_VF:1 The instance gets a VF
from the normal SRIOV nic(A).

But with a new image, the new request is: resources=SRIOV_VF:1
traits=HW_NIC_OFFLOAD_XX
With all the solutions discussed in the thread, a rebuild request like
above may or may not succeed depending on whether during the initial launch
whether nic A or nic B was allocated.

Remember that in rebuild new allocation don't happen, we have to reuse the
existing allocations.

Given the above background, there seems to be 2 competing options.

1. Fail in the API saying you can't rebuild with a new image with new
required traits.

2. Look at the current allocations for the instance and try to match the
new requirement from the image with the allocations.

With #1, we get consistent results in regards to how rebuilds are treated
when the image traits changed.

With #2, the rebuild may or may not succeed, depending on how well the
original allocations match up with the new requirements.

#2 will also need to need to account for handling preferred traits or
granular resource traits if we decide to implement them for images at some
point...


[1]
https://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/glance-image-traits.html
[2] https://review.openstack.org/#/c/560718/

Post by Sylvain Bauza
Sorry folks for the late reply, I'll try to also weigh in the Gerrit change.

Post by Jay Pipes

Post by Arvind N
Thanks for the detailed options Matt/eric/jay.
Just few of my thoughts,
For #1, we can make the explanation very clear that we rejected the
request because the original traits specified in the original image and the
new traits specified in the new image do not match and hence rebuild is not
supported.

I believe I had suggested that on the spec amendment patch. Matt had
concerns about an error message being a poor user experience (I don't
necessarily disagree with that) and I had suggested a clearer error message
to try and make that user experience slightly less sucky.
For #3,

Post by Arvind N
Even though it handles the nested provider, there is a potential issue.
Lets say a host with two SRIOV nic. One is normal SRIOV nic(VF1),
another one with some kind of offload feature(VF2).(Described by alex)
Initial instance launch happens with VF:1 allocated, rebuild launches
with modified request with traits=HW_NIC_OFFLOAD_X, so basically we want
the instance to be allocated VF2.
But the original allocation happens against VF1 and since in rebuild the
original allocations are not changed, we have wrong allocations.

Yep, that is certainly an issue. The only solution to this that I can see
would be to have the conductor ask the compute node to do the pre-flight
check. The compute node already has the entire tree of providers, their
inventories and traits, along with information about providers that share
resources with the compute node. It has this information in the
ProviderTree object in the reportclient that is contained in the compute
node resource tracker.
The pre-flight check, if run on the compute node, would be able to grab
the allocation records for the instance and determine if the required
traits for the new image are present on the actual resource providers
allocated against for the instance (and not including any child providers
not allocated against).

Yup, that. We also have pre-flight checks for move operations like live
and cold migrations, and I'd really like to keep all the conditionals in
the conductor, because it knows better than the scheduler which operation
is asked.
I'm not really happy with adding more in the scheduler about "yeah, it's a
rebuild, so please do something exceptional", and I'm also not happy with
having a filter (that can be disabled) calling the Placement API.

Post by Jay Pipes
Or... we chalk this up as a "too bad" situation and just either go with
option #1 or simply don't care about it.

Also, that too. Maybe just provide an error should be enough, nope?
-Sylvain

Post by Jay Pipes
Best,
-jay
____________________________________________________________
______________
OpenStack Development Mailing List (not for usage questions)
e
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Matt Riedemann]

Post by Arvind N
In cases of rebuilding of an instance using a different image where the
image traits have changed between the original launch and the rebuild,
is it reasonable to ask to just re-launch a new instance with the new image?
The argument for this approach is that given that the requirements have
changed, we want the scheduler to pick and allocate the appropriate host
for the instance.

We don't know if the requirements have changed with the new image until
we check them.

Here is another option:

What if the API compares the original image required traits against the
new image required traits, and if the new image has required traits
which weren't in the original image, then (punt) fail in the API? Then
you would at least have a chance to rebuild with a new image that has
required traits as long as those required traits are less than or equal
to the originally validated traits for the host on which the instance is
currently running.

Post by Arvind N
The approach above also gives you consistent results vs the other
approaches where the rebuild may or may not succeed depending on how the
original allocation of resources went.

Consistently frustrating, I agree. :) Because as a user, I can rebuild
with some images (that don't have required traits) and can't rebuild
with other images (that do have required traits).

I see no difference with this and being able to rebuild (with a new
image) some instances (image-backed) and not others (volume-backed).
Given that, I expect if we punt on this, someone will just come along
asking for the support later. Could be a couple of years from now when
everyone has moved on and it then becomes someone else's problem.

Post by Arvind N
For example(from Alex Xu) ,if you launched an instance on a host which
has two SRIOV nic. One is normal SRIOV nic(A), another one with some
kind of offload feature(B).
So, the original request is: resources=SRIOV_VF:1 The instance gets a VF
from the normal SRIOV nic(A).
But with a new image, the new request is: resources=SRIOV_VF:1
traits=HW_NIC_OFFLOAD_XX
With all the solutions discussed in the thread, a rebuild request like
above may or may not succeed depending on whether during the initial
launch whether nic A or nic B was allocated.
Remember that in rebuild new allocation don't happen, we have to reuse
the existing allocations.
Given the above background, there seems to be 2 competing options.
1. Fail in the API saying you can't rebuild with a new image with new
required traits.
2. Look at the current allocations for the instance and try to match the
new requirement from the image with the allocations.
With #1, we get consistent results in regards to how rebuilds are
treated when the image traits changed.
With #2, the rebuild may or may not succeed, depending on how well the
original allocations match up with the new requirements.
#2 will also need to need to account for handling preferred traits or
granular resource traits if we decide to implement them for images at
some point...

Option 10: Don't support image-defined traits at all. I know that won't
happen though.

At this point I'm exhausted with this entire issue and conversation and
will probably bow out and need someone else to step in with different
perspective, like melwitt or dansmith.

All of the solutions are bad in their own way, either because they add
technical debt and poor user experience, or because they make rebuild
more complicated and harder to maintain for the developers.

[Arvind N]

Post by Matt Riedemann
What if the API compares the original image required traits against the

new image required traits, and if the new image has required traits which
weren't in the original image, then (punt) fail in the API? Then you would
at least have a chance > to rebuild with a new image that has required
traits as long as those required traits are less than or equal to the
originally validated traits for the host on which the instance is currently
running.

This is what i was proposing with #1, sorry if it was unclear. Will make it
more explicit.

1. Reject the rebuild request indicating that rebuilding with a new image
with **different** required traits compared to the original request is not
supported.
If the new image has the same or reduced set of traits as the old image,
then the request will be passed through to the conductor etc

Pseudo code

Post by Matt Riedemann
if not set(new_image.traits_required).issubset(

set(original_image.traits_required))

Post by Matt Riedemann
raise exception

Post by Arvind N
In cases of rebuilding of an instance using a different image where the
image traits have changed between the original launch and the rebuild, is
it reasonable to ask to just re-launch a new instance with the new image?
The argument for this approach is that given that the requirements have
changed, we want the scheduler to pick and allocate the appropriate host
for the instance.

We don't know if the requirements have changed with the new image until we
check them.
What if the API compares the original image required traits against the
new image required traits, and if the new image has required traits which
weren't in the original image, then (punt) fail in the API? Then you would
at least have a chance to rebuild with a new image that has required traits
as long as those required traits are less than or equal to the originally
validated traits for the host on which the instance is currently running.

Post by Arvind N
The approach above also gives you consistent results vs the other
approaches where the rebuild may or may not succeed depending on how the
original allocation of resources went.

Consistently frustrating, I agree. :) Because as a user, I can rebuild
with some images (that don't have required traits) and can't rebuild with
other images (that do have required traits).
I see no difference with this and being able to rebuild (with a new image)
some instances (image-backed) and not others (volume-backed). Given that, I
expect if we punt on this, someone will just come along asking for the
support later. Could be a couple of years from now when everyone has moved
on and it then becomes someone else's problem.
For example(from Alex Xu) ,if you launched an instance on a host which has

Post by Arvind N
two SRIOV nic. One is normal SRIOV nic(A), another one with some kind of
offload feature(B).
So, the original request is: resources=SRIOV_VF:1 The instance gets a VF
from the normal SRIOV nic(A).
But with a new image, the new request is: resources=SRIOV_VF:1
traits=HW_NIC_OFFLOAD_XX
With all the solutions discussed in the thread, a rebuild request like
above may or may not succeed depending on whether during the initial launch
whether nic A or nic B was allocated.
Remember that in rebuild new allocation don't happen, we have to reuse
the existing allocations.
Given the above background, there seems to be 2 competing options.
1. Fail in the API saying you can't rebuild with a new image with new
required traits.
2. Look at the current allocations for the instance and try to match the
new requirement from the image with the allocations.
With #1, we get consistent results in regards to how rebuilds are treated
when the image traits changed.
With #2, the rebuild may or may not succeed, depending on how well the
original allocations match up with the new requirements.
#2 will also need to need to account for handling preferred traits or
granular resource traits if we decide to implement them for images at some
point...

Option 10: Don't support image-defined traits at all. I know that won't
happen though.
At this point I'm exhausted with this entire issue and conversation and
will probably bow out and need someone else to step in with different
perspective, like melwitt or dansmith.
All of the solutions are bad in their own way, either because they add
technical debt and poor user experience, or because they make rebuild more
complicated and harder to maintain for the developers.
--
Thanks,
Matt
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Jay Pipes]

Post by Matt Riedemann

Post by Arvind N
In cases of rebuilding of an instance using a different image where
the image traits have changed between the original launch and the
rebuild, is it reasonable to ask to just re-launch a new instance with
the new image?
The argument for this approach is that given that the requirements
have changed, we want the scheduler to pick and allocate the
appropriate host for the instance.

We don't know if the requirements have changed with the new image until
we check them.
What if the API compares the original image required traits against the
new image required traits, and if the new image has required traits
which weren't in the original image, then (punt) fail in the API? Then
you would at least have a chance to rebuild with a new image that has
required traits as long as those required traits are less than or equal
to the originally validated traits for the host on which the instance is
currently running.

That's pretty much what I had suggested earlier, yeah.

Post by Matt Riedemann
Option 10: Don't support image-defined traits at all. I know that won't
happen though.
At this point I'm exhausted with this entire issue and conversation and
will probably bow out and need someone else to step in with different
perspective, like melwitt or dansmith.
All of the solutions are bad in their own way, either because they add
technical debt and poor user experience, or because they make rebuild
more complicated and harder to maintain for the developers.

I hear your frustration. And I agree all of the solutions are bad in
their own way.

My personal preference is to add less technical debt and go with a
solution that checks if image traits have changed in nova-api and if so,
simply refuse to perform a rebuild.

Best,
-jay

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/

[Matt Riedemann]

Post by Jay Pipes
My personal preference is to add less technical debt and go with a
solution that checks if image traits have changed in nova-api and if so,
simply refuse to perform a rebuild.

So, what if when I created my server, the image I used, let's say
image1, had required trait A and that fit the host.

Then some external service removes (or somehow changes) trait A from the
compute node resource provider (because people can and will do this,
there are a few vmware specs up that rely on being able to manage traits
out of band from nova), and then I rebuild my server with image2 that
has required trait A. That would match the original trait A in image1
and we'd say, "yup, lgtm!" and do the rebuild even though the compute
node resource provider wouldn't have trait A anymore.

Having said that, it could technically happen before traits if the
operator changed something on the underlying compute host which
invalidated instances running on that host, but I'd think if that
happened the operator would be migrating everything off the host and
disabling it from scheduling before making whatever that kind of change
would be, let's say they change the hypervisor or something less drastic
but still image property invalidating.

[Arvind N]

Isnt this an existing issue with traits specified in flavor as well?

Server is created using flavor1 requiring trait A on RP1. Before the
rebuild is called, the underlying RP1 can be updated to remove trait A and
when a rebuild is requested(regardless of whether the image is updated or
not), we skip scheduling and allow the rebuild to go through.

Now, even though the flavor1 requests trait A, the underlying RP1 does not
have that trait the rebuild will succeed...

I think maybe there should be some kind of report or query which runs
periodically to ensure continued conformance with respect to instance
running and their traits. But since traits are intend to provide hints for
scheduling, this is different problem to solve IMO.

Post by Jay Pipes
My personal preference is to add less technical debt and go with a
solution that checks if image traits have changed in nova-api and if so,
simply refuse to perform a rebuild.

So, what if when I created my server, the image I used, let's say image1,
had required trait A and that fit the host.
Then some external service removes (or somehow changes) trait A from the
compute node resource provider (because people can and will do this, there
are a few vmware specs up that rely on being able to manage traits out of
band from nova), and then I rebuild my server with image2 that has required
trait A. That would match the original trait A in image1 and we'd say,
"yup, lgtm!" and do the rebuild even though the compute node resource
provider wouldn't have trait A anymore.
Having said that, it could technically happen before traits if the
operator changed something on the underlying compute host which invalidated
instances running on that host, but I'd think if that happened the operator
would be migrating everything off the host and disabling it from scheduling
before making whatever that kind of change would be, let's say they change
the hypervisor or something less drastic but still image property
invalidating.
--
Thanks,
Matt
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[melanie witt]

Post by Matt Riedemann

Post by Jay Pipes
My personal preference is to add less technical debt and go with a
solution that checks if image traits have changed in nova-api and if so,
simply refuse to perform a rebuild.

So, what if when I created my server, the image I used, let's say
image1, had required trait A and that fit the host.
Then some external service removes (or somehow changes) trait A from the
compute node resource provider (because people can and will do this,
there are a few vmware specs up that rely on being able to manage traits
out of band from nova), and then I rebuild my server with image2 that
has required trait A. That would match the original trait A in image1
and we'd say, "yup, lgtm!" and do the rebuild even though the compute
node resource provider wouldn't have trait A anymore.
Having said that, it could technically happen before traits if the
operator changed something on the underlying compute host which
invalidated instances running on that host, but I'd think if that
happened the operator would be migrating everything off the host and
disabling it from scheduling before making whatever that kind of change
would be, let's say they change the hypervisor or something less drastic
but still image property invalidating.

This is a scenario I was thinking about too. In the land of software
licenses, this would be analogous to removing a license from a compute
host, say. The instance is already there but should we let a rebuild
proceed that is going to violate the image traits currently supported by
that host? Do we potentially prolong the life of that instance by
letting it be re-imaged?

I'm late to this thread but I finally went through the replies and my
thought is, we should do a pre-flight check to verify with placement
whether the image traits requested are 1) supported by the compute host
the instance is residing on and 2) coincide with the already-existing
allocations. Instead of making an assumption based on "last image" vs
"new image" and artificially limiting a rebuild that should be valid to
go ahead. I can imagine scenarios where a user is trying to do a rebuild
that their cloud admin says should be perfectly valid on their
hypervisor, but it's getting rejected because old image traits != new
image traits. It seems like unnecessary user and admin pain.

It doesn't seem correct to reject the request if the current compute
host can fulfill it, and if I understood correctly, we have placement
APIs we can call from the conductor to verify the image traits requested
for the rebuild can be fulfilled. Is there a reason not to do that?

-melanie

[Dan Smith]

Post by melanie witt
I'm late to this thread but I finally went through the replies and my
thought is, we should do a pre-flight check to verify with placement
whether the image traits requested are 1) supported by the compute
host the instance is residing on and 2) coincide with the
already-existing allocations. Instead of making an assumption based on
"last image" vs "new image" and artificially limiting a rebuild that
should be valid to go ahead. I can imagine scenarios where a user is
trying to do a rebuild that their cloud admin says should be perfectly
valid on their hypervisor, but it's getting rejected because old image
traits != new image traits. It seems like unnecessary user and admin
pain.

Yeah, I think we have to do this.

Post by melanie witt
It doesn't seem correct to reject the request if the current compute
host can fulfill it, and if I understood correctly, we have placement
APIs we can call from the conductor to verify the image traits
requested for the rebuild can be fulfilled. Is there a reason not to
do that?

Well, it's a little itcky in that it makes a random part of conductor a
bit like the scheduler in its understanding of and iteraction with
placement. I don't love it, but I think it's what we have to do. Trying
to do the trait math with what was used before, or conservatively
rejecting the request and being potentially wrong about that is not
reasonable, IMHO.

--Dan

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-b

[Matt Riedemann]

Post by Dan Smith
Well, it's a little itcky in that it makes a random part of conductor a
bit like the scheduler in its understanding of and iteraction with
placement. I don't love it, but I think it's what we have to do. Trying
to do the trait math with what was used before, or conservatively
rejecting the request and being potentially wrong about that is not
reasonable, IMHO.

The upside to doing the check in conductor is we have a specific code
flow for rebuild in conductor and we should be able to just put a
private method off to the side for this validation scenario. That's
preferable to baking more rebuild logic into the scheduler. It also
means we are always going to do this validation regardless of whether or
not the ImagePropertiesFilter is enabled, but that (1) seems OK and (2)
no one probably ever disables the ImagePropertiesFilter anyway.

[Eric Fried]

Post by melanie witt
verify with placement
whether the image traits requested are 1) supported by the compute
host the instance is residing on and 2) coincide with the
already-existing allocations.

Note that #2 is a subset of #1. The only potential advantage of
including #1 is efficiency: We can do #1 in one API call and bail early
if it fails; but if it passes, we have to do #2 anyway, which is
multiple steps. So would we rather save one step in the "good path" or
potentially N-1 steps in the failure case? IMO the cost of the
additional dev/test to implement #1 is higher than that of the potential
extra API calls. (TL;DR: just implement #2.)

-efried

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/l

[Matt Riedemann]

We also have pre-flight checks for move operations like live and cold
migrations, and I'd really like to keep all the conditionals in the
conductor, because it knows better than the scheduler which operation is
asked.

I'm not sure what "pre-flight checks" we have for cold migration. The
conductor migrate task asks the scheduler for a host and then casts to
the destination compute to start the migration.

The conductor live migration task does do some checking on the source
and dest computes before proceeding, I agree with you there.

I'm not really happy with adding more in the scheduler about "yeah,

it's a rebuild, so please do something exceptional"

Agree that building more special rebuild logic into the scheduler isn't
ideal and hopefully we could resolve this in conductor if possible,
despite the fact that ImagePropertiesFilter is optional (although I'm
pretty sure everyone enables it).

[Matt Riedemann]

Post by Arvind N
For #1, we can make the explanation very clear that we rejected the
request because the original traits specified in the original image and
the new traits specified in the new image do not match and hence rebuild
is not supported.

We don't reject rebuild requests today where you rebuild with a new
image as long as that new image passes the scheduler filters for the
host on which the instance is already running. I don't see why we'd just
immediately fail in the API because the new image has required traits,
when we have no idea, from the nova-api service, whether or not those
image-defined required traits are going to match the current host or
not. That's just adding technical debt to rebuild, like we have for
rebuilding a volume-backed instance with a new image (you can't do it
today because it wasn't thought about early enough in the design process).

Post by Arvind N
For #2,
1. None of the filters currently make other API requests and my
understanding is we want to avoid reintroducing such a pattern. But
definitely workable solution.

For a rebuild-specific request (which we can determine already), I'm OK
with this - we're already not calling GET /allocation_candidates in this
case, so if people are worried about performance, it's just a trade of
one REST API call for another.

Post by Arvind N
2. If the user disables the image properties filter, then traits based
filtering will not be run in rebuild case

The user doesn't disable the filter, the operator does, and likely for
good reason. I don't see a problem with this.

Post by Arvind N
For #3,
Even though it handles the nested provider, there is a potential issue.
Lets say a host with two SRIOV nic. One is normal SRIOV nic(VF1),
another one with some kind of offload feature(VF2).(Described by alex)
Initial instance launch happens with VF:1 allocated, rebuild launches
with modified request with traits=HW_NIC_OFFLOAD_X, so basically we want
the instance to be allocated VF2.
But the original allocation happens against VF1 and since in rebuild the
original allocations are not changed, we have wrong allocations.

I don't know what to say about this. We shouldn't have any quantitative
resource allocation changes as a result of a rebuild. This actually
sounds like a case for option #4 with using GET /allocation_candidates
and then being able to filter out if rebuliding the instance with the
new image with new required traits but on the same host would result in
new allocation requests, and if so, we should fail - but we can (only?)
determine that via the response from GET /allocation_candidates.

[Matt Riedemann]

Post by Jay Pipes
How about just having the conductor call GET
/resource_providers?in_tree=<CN_UUID>&required=<IMAGE_TRAITS>, see if
there is a result, and if not, don't even call the scheduler at all
(because conductor would already know there would be a NoValidHost
returned)?

This makes filtering on image properties required, which is something I
was pushing back on because the ImagePropertiesFilter today, by design
of all scheduler filters, is configurable and optional, which is why I
wanted to add the filtering logic for image-defined required traits into
the ImagePropertiesFilter itself.