Emit restart events when a job is enqueued #6816

d3flex · 2025-10-22T12:33:55Z

With this commit once the job goes to its end, it will check if it can restarted and if it fulfills the criteria, it will send the new job to the minion queue via enqueue_restart, where now it should populate the event.

issue: https://progress.opensuse.org/issues/190557

d3flex · 2025-10-22T12:37:25Z

I want to see where it will fail (some test failed on my side) and the codecov report
also I cant see where a test for this should go. there is t/10-jobs.t as an obvious candidate but it doesnt seem to cover the events

codecov · 2025-10-22T12:52:41Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.26%. Comparing base (ad78558) to head (c8fe5b4).
⚠️ Report is 16 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #6816   +/-   ##
=======================================
  Coverage   99.26%   99.26%           
=======================================
  Files         402      402           
  Lines       41373    41384   +11     
=======================================
+ Hits        41067    41078   +11     
  Misses        306      306

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

perlpunk · 2025-10-22T13:09:47Z

also I cant see where a test for this should go. there is t/10-jobs.t as an obvious candidate but it doesnt seem to cover the events

t/23-amqp.t looks right in this case. Take a job and call enqueue_restart on it

okurz

Needs tests

okurz

code looks good so far. I suggest you do a manual system test with a local openQA instance and ensure that no duplicate events are sent.

Also in your commit message I don't quite understand "With this commit once the job goes to its end, it will check if it can restarted and if it fulfills the criteria". What "end" and what "criteria"? Also "check if it can restarted" is grammatically incorrect

d3flex · 2025-10-23T11:43:31Z

code looks good so far. I suggest you do a manual system test with a local openQA instance and ensure that no duplicate events are sent.

Also in your commit message I don't quite understand "With this commit once the job goes to its end, it will check if it can restarted and if it fulfills the criteria". What "end" and what "criteria"? Also "check if it can restarted" is grammatically incorrect

d3flex · 2025-10-23T13:05:58Z

code looks good so far. I suggest you do a manual system test with a local openQA instance and ensure that no duplicate events are sent.

Also in your commit message I don't quite understand "With this commit once the job goes to its end, it will check if it can restarted and if it fulfills the criteria". What "end" and what "criteria"? Also "check if it can restarted" is grammatically incorrect

updated.

perlpunk · 2025-10-23T13:11:21Z

With this commit once the job goes to execute its final stage, it will
check if it can be restarted. If yes, it will send the new job to
the minion queue via enqueue_restart, where now it should populate
the event.

The only change in this commit is that, if the job is restarted via enqueue_restart, it now publishes a restart event.
The description sounds overly complicated ("goes to execute its final stage").
" it will check if it can be restarted" - that's already happening without your commit.

How about: "When a job is enqueued to be restarted via the RETRY feature, it now publishes a job.restart event."

d3flex · 2025-10-23T13:59:25Z

With this commit once the job goes to execute its final stage, it will
check if it can be restarted. If yes, it will send the new job to
the minion queue via enqueue_restart, where now it should populate
the event.

The only change in this commit is that, if the job is restarted via enqueue_restart, it now publishes a restart event. The description sounds overly complicated ("goes to execute its final stage"). " it will check if it can be restarted" - that's already happening without your commit.

How about: "When a job is enqueued to be restarted via the RETRY feature, it now publishes a job.restart event."

I avoid to reffer RETRY because is not the only path to enqueue_restart. are you good with When a job is enqueued to be restarted, it now publishes a job.restart event?

perlpunk · 2025-10-23T15:46:56Z

When a job is enqueued to be restarted as part of the $job->done handling, a job.restart event is now published

Martchus · 2025-10-24T07:27:34Z

t/23-amqp.t

+    qr/Enqueued restarting openQA job 99945 via Minion job \d+/, 'Debug log for enqueue_restart is present';
+    ok $minion_id, 'Got a Minion job ID back';
+    # each event contains keys(headers args body)
+    is scalar(%published), 3, 'Exactly one event was published';


The number and the description are inconsistent.

Martchus · 2025-10-24T07:30:12Z

t/23-amqp.t

+    combined_like { $minion_id = $test_job->enqueue_restart }
+    qr/Enqueued restarting openQA job 99945 via Minion job \d+/, 'Debug log for enqueue_restart is present';
+    ok $minion_id, 'Got a Minion job ID back';
+    # each event contains keys(headers args body)


I would remove this comment. If this is important it should be checked with code.

I will answer in your both comments here.
The amqp event contains three things, right? It has a headers, a body and the args. This is one amqp and the description is emphasize to that. the comment+ the description intended to make that clear.
the importance is that there is one event and not what is in this.

Maybe just remove the comment then. It doesn't help me understand the test code better. (And it lacks punctuation.)

Martchus · 2025-10-24T07:33:53Z

lib/OpenQA/Schema/Result/Jobs.pm

    my $openqa_job_id = $self->id;
    my $minion_job_id = OpenQA::App->singleton->gru->enqueue(restart_job => [$openqa_job_id], $options)->{minion_id};
    log_debug "Enqueued restarting openQA job $openqa_job_id via Minion job $minion_job_id";
+    OpenQA::App->singleton->emit_event(openqa_job_restart => {id => $openqa_job_id});


I'm not sure whether this is the right place. Here the job hasn't actually been restarted yet because the according Minion job has only been enqueued. I would prefer to emit the event only when the restart has actually successfully happened.

Use the `auto_duplicate` to emit an event as it is invoked by other functions which try to handle individual job restarts. That should make sure that the event is send to AMQP for jobs with `RETRY` once it is actually triggered. issue: https://progress.opensuse.org/issues/190557 Signed-off-by: Ioannis Bonatakis <[email protected]>

perlpunk · 2025-10-26T19:09:56Z

lib/OpenQA/Schema/Result/Jobs.pm

    $dup->{cluster_cloned} = {map { $_ => $clones->{$_}->{clone} } keys %$clones};
    $dup->{comments_created} = $args->{comments};
    log_debug("Job $job_id duplicated as $clone_id");
+    OpenQA::Events->singleton->emit_event('job_restart', data => {id => $clone_id});


auto_duplicate is also called when doing a restart via API or web UI, and in that case the code emits an event already, see
https://github.com/os-autoinst/openQA/blob/master/lib/OpenQA/WebAPI/Controller/API/V1/Job.pm#L821
So we would get two events.

Yes, so please take care we don't get two events.

Martchus · 2025-10-27T10:59:49Z

t/20-stale-job-detection.t

+    my $mock_events = Test::MockModule->new('OpenQA::Events');
+    $mock_events->redefine(
+        emit_event => sub {
+            my ($self, $type, %args) = @_;


Use signatures in new code.

d3flex added the not-ready label Oct 22, 2025

perlpunk mentioned this pull request Oct 22, 2025

Emit restart events from _hook_script #6813

Closed

okurz requested changes Oct 23, 2025

View reviewed changes

d3flex force-pushed the feat/amqp_restart branch from 8632b55 to e5e5fd2 Compare October 23, 2025 07:44

okurz requested changes Oct 23, 2025

View reviewed changes

d3flex force-pushed the feat/amqp_restart branch from e5e5fd2 to 44e622e Compare October 23, 2025 08:12

d3flex force-pushed the feat/amqp_restart branch from 44e622e to 0b69cf5 Compare October 23, 2025 13:05

d3flex marked this pull request as ready for review October 23, 2025 13:06

d3flex removed the not-ready label Oct 23, 2025

Martchus requested changes Oct 24, 2025

View reviewed changes

d3flex force-pushed the feat/amqp_restart branch from 0b69cf5 to c3478ec Compare October 24, 2025 19:55

d3flex force-pushed the feat/amqp_restart branch from c3478ec to c8fe5b4 Compare October 25, 2025 04:12

okurz approved these changes Oct 25, 2025

View reviewed changes

perlpunk reviewed Oct 26, 2025

View reviewed changes

Martchus reviewed Oct 27, 2025

View reviewed changes

Uh oh!

Emit restart events when a job is enqueued #6816

Are you sure you want to change the base?

Emit restart events when a job is enqueued #6816

Conversation

d3flex commented Oct 22, 2025

Uh oh!

d3flex commented Oct 22, 2025

Uh oh!

codecov bot commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

perlpunk commented Oct 22, 2025

Uh oh!

okurz left a comment

Choose a reason for hiding this comment

Uh oh!

okurz left a comment

Choose a reason for hiding this comment

Uh oh!

d3flex commented Oct 23, 2025

Uh oh!

d3flex commented Oct 23, 2025

Uh oh!

perlpunk commented Oct 23, 2025

Uh oh!

d3flex commented Oct 23, 2025

Uh oh!

perlpunk commented Oct 23, 2025

Uh oh!

Martchus Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

Martchus Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

d3flex Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

Martchus Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Martchus Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

perlpunk Oct 26, 2025

Choose a reason for hiding this comment

Uh oh!

Martchus Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Martchus Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Oct 22, 2025 •

edited

Loading

Martchus Oct 24, 2025 •

edited

Loading