MCO-1679: Add MCO periodics #64752
MCO-1679: Add MCO periodics #64752
Conversation
openshift-ci-robot
added
the
jira/valid-reference
|
@yuqi-zhang: This pull request references MCO-1679 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.20.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci
bot
added
the
approved
|
/pj-rehearse |
|
@djoshy: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
yuqi-zhang
force-pushed
the
add-mco-periodics
branch
from
e119da9 to
fea489f
Compare
|
@yuqi-zhang: This pull request references MCO-1679 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.20.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@yuqi-zhang: This pull request references MCO-1679 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.20.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
yuqi-zhang
force-pushed
the
add-mco-periodics
branch
from
fea489f to
878bccb
Compare
| @@ -578,6 +578,117 @@ tests: | |||
| CHECK_MOD_LIST: "false" | |||
| test: | |||
| - ref: go-verify-deps | |||
| - as: e2e-aws-mco-disruptive | |||
There was a problem hiding this comment.
Question: Do these tests need to be defined for 4.21 as well?
There was a problem hiding this comment.
Good question, I'm not really sure what the guidance is there. I was under the impression that when we come to 4.21 branching we would set up the config aligned with main so I opted not to do it yet. If that's not the case I'm happy to copy the configs over as well.
There was a problem hiding this comment.
I'm not sure what the guidance is either, but keeping this as you have it for now sounds fair to me!
There was a problem hiding this comment.
So actually you don't want periodics defined in main and for the release specific config you should have a separate periodic variant file similar to openshift-cluster-control-plane-machine-set-operator-release-4.20__periodics.yaml the 4.21 version will get created when 4.20 branches.
yuqi-zhang
force-pushed
the
add-mco-periodics
branch
from
878bccb to
820531a
Compare
|
Per suggestion, change the metal/vsphere jobs to run in early hours UTC/EST to reduce load on those platforms |
yuqi-zhang
force-pushed
the
add-mco-periodics
branch
from
820531a to
e232304
Compare
|
/pj-rehearse Now that the suite exists, see if these can run |
|
@yuqi-zhang: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
Hmm, some of the installs failed. Of the successful ones:
That's good to see I think, since the suite is indeed empty. @neisw would we be allowed to merge failing periodics for now? Or would we need to add tests via openshift/origin#29776 first? |
| @@ -598,6 +598,117 @@ tests: | |||
| test: | |||
| - ref: windows-e2e-operator-test | |||
| workflow: ipi-aws-ovn-hybrid | |||
| - as: e2e-aws-mco-disruptive | |||
| interval: 48h | |||
There was a problem hiding this comment.
Since you are looking to create these jobs before you have any tests, how about reducing the frequency this runs until you get the tests added. Unless you expect a quick turnaround there.
There was a problem hiding this comment.
Modified to run every 7 days for now, but we do intend to add tests within the next few sprints.
| ci-operator.openshift.io/cloud: aws | ||
| ci-operator.openshift.io/cloud-cluster-profile: aws-3 | ||
| ci.openshift.io/generator: prowgen | ||
| pj-rehearse.openshift.io/can-be-rehearsed: "true" |
There was a problem hiding this comment.
When you have the 4.20 release __periodics.yaml variant defined that will add the job-release label that is needed for Sippy / CR to pick up the job data
There was a problem hiding this comment.
Ack, I think I misunderstood the configuration initially. Updated it to instead be 4.19/4.20 __periodic variants instead and regenerated the job config.
yuqi-zhang
force-pushed
the
add-mco-periodics
branch
from
e232304 to
1058cb6
Compare
openshift-ci
bot
removed
the
approved
|
Updated based on suggestions, the tests will run every 7 days for now, and moved the tests to |
|
/pj-rehearse more |
|
@yuqi-zhang: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
Hmm, the which is expected, but the 4.19 and 4.20 both fail with 4.19 I would expect, but openshift/origin#29800 is in the release-4.20 branch since that's fast-forwarded. Anything we need to do specifically there? |
yuqi-zhang
force-pushed
the
add-mco-periodics
branch
from
1058cb6 to
3b5c8a2
Compare
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: yuqi-zhang The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
@yuqi-zhang: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@yuqi-zhang: requesting more than one rehearsal in one comment is not supported. If you would like to rehearse multiple specific jobs, please separate the job names by a space in a single command. |
3 similar comments
|
@yuqi-zhang: requesting more than one rehearsal in one comment is not supported. If you would like to rehearse multiple specific jobs, please separate the job names by a space in a single command. |
|
@yuqi-zhang: requesting more than one rehearsal in one comment is not supported. If you would like to rehearse multiple specific jobs, please separate the job names by a space in a single command. |
|
@yuqi-zhang: requesting more than one rehearsal in one comment is not supported. If you would like to rehearse multiple specific jobs, please separate the job names by a space in a single command. |
|
/pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-vsphere-mco-disruptive periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-metal-ipi-ovn-ipv6-mco-disruptive periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-azure-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-gcp-mco-disruptive-techpreview |
|
@yuqi-zhang: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
ah, I guess I need to wait for https://amd64.ocp.releases.ci.openshift.org/releasestream/4.20.0-0.nightly/release/4.20.0-0.nightly-2025-06-10-072846 to get into nightlies |
|
/pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-vsphere-mco-disruptive periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-metal-ipi-ovn-ipv6-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-azure-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-gcp-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-aws-mco-disruptive-techpreview |
|
@yuqi-zhang: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
We had our first test passes! Added a fix for vsphere, metal continues to fail due to resource quota. Trying again: /pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-vsphere-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-metal-ipi-ovn-ipv6-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-azure-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-gcp-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-aws-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-metal-ipi-ovn-ipv4-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-metal-ipi-ovn-dualstack-mco-disruptive-techpreview |
|
@yuqi-zhang: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
Ah, the metal jobs need to run off build05 I just realized. Updated and trying the failing jobs. VSphere, AWS and GCP have passed: /pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-metal-ipi-ovn-ipv6-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-azure-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-metal-ipi-ovn-ipv4-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-metal-ipi-ovn-dualstack-mco-disruptive-techpreview |
|
@yuqi-zhang: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@yuqi-zhang, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
@yuqi-zhang: job(s): either don't exist or were not found to be affected, and cannot be rehearsed |
|
The ipv4 metal job passed, and the tests were passing on the other metal jobs. Bumped the timeouts just in case but @neisw I think this is ready to go. The 4.19 jobs will need openshift/origin#29918 , I've backported those, PTAL. I think the final goal is to get these to 4.18, but let's just do 4.19/4.20 for now to get some signal |
|
[REHEARSALNOTIFIER]
A total of 30 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs. A full list of affected jobs can be found here Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
Running again to make sure /pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-vsphere-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-metal-ipi-ovn-ipv6-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-azure-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-gcp-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-aws-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-metal-ipi-ovn-ipv4-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-metal-ipi-ovn-dualstack-mco-disruptive-techpreview |
|
@yuqi-zhang: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@yuqi-zhang: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-metal-ipi-ovn-ipv6-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-metal-ipi-ovn-dualstack-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-gcp-mco-disruptive-techpreview |
|
@yuqi-zhang: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/hold To make sure the new test this exercises works. Also so this doesn't disturb release signal for MCN/PIS. |
openshift-ci
bot
added
the
do-not-merge/hold
|
/hold cancel openshift/api#2288 has merged so this shouldn't cause any issues, and all 7 platforms have passed |
openshift-ci
bot
removed
the
do-not-merge/hold
Add periodic MCO jobs to run the new suite defined in openshift/origin#29776
Added a total of 14 MCO periodic jobs aligned with feature graduation requirements in o/api: https://github.com/openshift/api/blob/9052dea866947c1b70a662aa0ff3d90fa662a93d/tools/codegen/cmd/featuregate-test-analyzer.go#L331-L370
Currently they are set to 48h intervals, but maybe we should run the metal jobs less frequently, and run techpreview more than graduated, to help us generate signal? Or should it be the other way around?
Side note: baremetalds-e2e-ovn-* workflows don't seem to have FEATURESET enabled, so copied
from other techpreview jobs.