-
Notifications
You must be signed in to change notification settings - Fork 257
HIVE-2302, HIVE-2644: Pass metadata.json through opaquely #2729
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HIVE-2302, HIVE-2644: Pass metadata.json through opaquely #2729
Conversation
|
@2uasimojo: This pull request references HIVE-2302 which is a valid jira issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Skipping CI for Draft Pull Request. |
|
@2uasimojo: This pull request references HIVE-2302 which is a valid jira issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
TODO: Deprovisioner side |
8707a79 to
d09e43e
Compare
|
@2uasimojo: This pull request references HIVE-2302 which is a valid jira issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
1 similar comment
|
@2uasimojo: This pull request references HIVE-2302 which is a valid jira issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
d09e43e to
13ad458
Compare
13ad458 to
ace243e
Compare
|
@2uasimojo: This pull request references HIVE-2302 which is a valid jira issue. This pull request references HIVE-2644 which is a valid jira issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@2uasimojo: This pull request references HIVE-2302 which is a valid jira issue. This pull request references HIVE-2644 which is a valid jira issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
ace243e to
cf8610a
Compare
|
/test e2e e2e-azure e2e-gcp e2e-vsphere e2e-openstack 🤞 |
cf8610a to
268d7cc
Compare
|
/test e2e-azure e2e-vsphere |
268d7cc to
bce1629
Compare
/test e2e-vsphere |
bce1629 to
cb1b5d5
Compare
|
/hold for moar testings |
e2b93af to
553ba5b
Compare
Well, mostly. Previously any time installer added a field to metadata.json, we would need to evaluate and possibly add a bespoke field and code path for it to make sure it was supplied to the destroyer at deprovision time. With this change, we're offloading metadata.json verbatim (except in some cases we have to scrub/replace credentials fields -- see HIVE-2804 / openshift#2612) to a new Secret in the ClusterDeployment's namespace, referenced from a new field: ClusterDeployment.Spec.ClusterMetadata.MetadataJSONSecretRef. For legacy clusters -- those created before this change -- we attempt to retrofit the new Secret based on the legacy fields. This is best effort and may not always work. In the future (but not here!) instead of building the installer's ClusterMetadata structure for the destroyer with individual fields from the CD's ClusterMetadata, we'll unmarshal it directly from the contents of this Secret.
An earlier commit ensures that ClusterDeployments have an associated Secret containing the metadata.json emitted by the installer. This change adds a new generic destroyer via the (existing) `hiveutil deprovision` command that consumes this metadata.json to deprovision the cluster. This new behavior is the default, but we also include an escape hatch to run the platform-specific legacy destroyer by setting the following annotation on the ClusterDeployment: `hive.openshift.io/legacy-deprovision: "true"`
A couple of providers (nutanix, vsphere) need bespoke code to populate credentials in the metadata.json object for destroying a cluster. In a prior commit this was being done in the deprovisioner (the new one, that uses metadata.json directly, per HIVE-2302) after ConfigureCreds. Since ConfigureCreds is where we (stay with me) configure creds, and is already platform-specific, it makes more sense to do this work there. This commit refactors to do so. Legacy code paths pass in a `nil` metadata object, which is coded to result in no change from the previous functionality. (In particular, ConfigureCreds is also used when provisioning, where no metadata object is present/necessary.)
553ba5b to
42c37de
Compare
|
/test verify |
|
/lgtm @2uasimojo merge when you're ready |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: 2uasimojo, dlom The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/test verify e2e-azure |
|
The |
|
What started this was HIVE-2302 / openshift#2729 where I wanted to be able to use the ConfigureCreds map from installmanager as well as deprovision, but couldn't do it easily due to circular imports. I got that done, but also knocked down some other tech debt at the same time. I moved things closer to where they're used, in many cases eliminating the packages they came from, addressed some deprecations, deleted some unused stuff, shaved off a few lines of code in some places, got rid of some soft linting squiggles, DRYed some duplicated symbols, that kind of thing.
What started this was HIVE-2302 / openshift#2729 where I wanted to be able to use the ConfigureCreds map from installmanager as well as deprovision, but couldn't do it easily due to circular imports. I got that done, but also knocked down some other tech debt at the same time. I moved things closer to where they're used, in many cases eliminating the packages they came from, addressed some deprecations, deleted some unused stuff, shaved off a few lines of code in some places, got rid of some soft linting squiggles, DRYed some duplicated symbols, that kind of thing.
|
/retest-required |
What started this was HIVE-2302 / openshift#2729 where I wanted to be able to use the ConfigureCreds map from installmanager as well as deprovision, but couldn't do it easily due to circular imports. I got that done, but also knocked down some other tech debt at the same time. I moved things closer to where they're used, in many cases eliminating the packages they came from, addressed some deprecations, deleted some unused stuff, shaved off a few lines of code in some places, got rid of some soft linting squiggles, DRYed some duplicated symbols, that kind of thing.
|
/hold cancel QE looks good. There is one pending issue (start reading here) but it would need to be coordinated with the installer team, if we decide to address it at all. Let's go! |
|
/test e2e-gcp |
|
/test verify |
|
@2uasimojo: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
What started this was HIVE-2302 / openshift#2729 where I wanted to be able to use the ConfigureCreds map from installmanager as well as deprovision, but couldn't do it easily due to circular imports. I got that done, but also knocked down some other tech debt at the same time. I moved things closer to where they're used, in many cases eliminating the packages they came from, addressed some deprecations, deleted some unused stuff, shaved off a few lines of code in some places, got rid of some soft linting squiggles, DRYed some duplicated symbols, that kind of thing.
What started this was HIVE-2302 / openshift#2729 where I wanted to be able to use the ConfigureCreds map from installmanager as well as deprovision, but couldn't do it easily due to circular imports. I got that done, but also knocked down some other tech debt at the same time. I moved things closer to where they're used, in many cases eliminating the packages they came from, addressed some deprecations, deleted some unused stuff, shaved off a few lines of code in some places, got rid of some soft linting squiggles, DRYed some duplicated symbols, that kind of thing.
For purposes of this work, there are three kinds of clusters: - IPI, where hive manages the provisioning via installer. - UPI, where a ClusterInstall implementation provisions the cluster and hive just watches and copies over the status. - Fake, used for testing purposes. When we implemented HIVE-2302 / openshift#2729, we didn't account for UPI/ClusterInstall, which we don't really have a good way to test, and ended up injecting a bug: Via that effort, we started populating a new Secret containing metadata.json produced by the installer. For legacy clusters (those that existed before upgrading to a version with this feature) we need to retrofit that Secret based on the ClusterMetadata (among other things), which was previously how we saved off the metadata.json. For IPI, that ClusterMetadata always had a Platform section. The changes we had to make for fake clusters saw us spoofing a very sparse metadata.json and then populating it later. That process relied on the existence of the CD.Spec.ClusterMetadata.Platform section, so we were creating it for the sake of that fake cluster path. However, it turns out that ClusterInstall subclasses don't (and don't need to) populate the ClusterMetadata.Platform section. Since we blindly copy the ClusterMetadata into the ClusterDeployment, we could end up with that Platform section being absent when we come to retrofit the metadata.json. We would then hit the path designed for fake clusters where we would create that section (empty). No problem, right? Except that we have a validating admission webhook that forbids making changes to the ClusterMetadata section, and that new, empty Platform field was flagged as such a change, and bounced by the webhook. Phew. So with this change, we condition populating that Platform section explicitly on fake clusters only.
In HIVE-2302 / openshift#2729 we rearranged the way installmanager figures out which platform-specific ConfigureCreds() function to call, but forgot to include IBMCloud in the lookup table. (BareMetal is there, which we don't use, and which is probably why this was missed, as it made the number of entries correct.)
Previously any time installer added a field to metadata.json, we would need to evaluate and possibly add a bespoke field and code path for it to make sure it was supplied to the destroyer at deprovision time.
With this change, we're offloading metadata.json verbatim (except in some cases we have to scrub/replace credentials fields -- see HIVE-2804 / #2612) to a new Secret in the ClusterDeployment's namespace, referenced from a new field: ClusterDeployment.Spec.ClusterMetadata.MetadataJSONSecretRef.
For legacy clusters -- those created before this change -- we attempt to retrofit the new Secret based on the legacy fields. This is best effort and may not always work.
This change then adds a new generic destroyer via the (existing)
hiveutil deprovisioncommand that consumes this metadata.json to deprovision the cluster.This new behavior is the default, but we also include an escape hatch to run the platform-specific legacy destroyer by setting the following annotation on the ClusterDeployment:
hive.openshift.io/legacy-deprovision: "true"