Additional progress information for batch job monitoring #78
Conversation
…hes remaining and a remaining time estimate based on the time taken so far
# Conflicts: # VERSION.txt
There was a problem hiding this comment.
Pull Request Overview
This PR enhances batch job monitoring by adding progress tracking and time estimation capabilities. When processing batches of TDR jobs, users now see the current batch number, total batches remaining, and an estimated completion time based on historical batch durations.
Key Changes:
- Added time tracking for batch durations to calculate estimated remaining time
- Enhanced logging to display batch progress (current/total) and time estimates
- Added type hints for improved code clarity
Reviewed Changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| ops_utils/tdr_utils/tdr_job_utils.py | Added batch duration tracking, time estimation logic, and enhanced progress logging with batch counts and remaining time |
| VERSION.txt | Updated version to 12.2.1 with description of the new feature |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| failed_jobs = [] | ||
| failed_jobs: list[str] = [] | ||
| total_jobs = len(self.job_args_list) | ||
| batch_durations: list[Any] = [] |
There was a problem hiding this comment.
The type annotation list[Any] is too generic. Based on usage at line 162 where time.time() - start_time is appended, this should be list[float] to accurately represent batch duration values in seconds.
| batch_durations: list[Any] = [] | |
| batch_durations: list[float] = [] |
| batches_left = (total_jobs + self.batch_size - 1) // self.batch_size - batch_num | ||
| logging.info( | ||
| f"Submitting jobs for batch {i // self.batch_size + 1} with {len(current_batch)} jobs." | ||
| f"Submitting jobs for batch {batch_num} of {batches_left} with {len(current_batch)} jobs " |
There was a problem hiding this comment.
The batch progress display is incorrect. Line 131 shows 'batch {batch_num} of {batches_left}', but batches_left represents remaining batches, not total batches. It should be 'batch {batch_num} of {total_batches}' where total_batches = (total_jobs + self.batch_size - 1) // self.batch_size.
When waiting for a batch of jobs to complete, show the number of batches remaining and a remaining time estimate based on the time taken so far.