Sync upstream #54
Closed
Sync upstream #54
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…elery#9243) * Added docker cleanup auto-fixture to improve smoke tests stability * Use docker API instead of subprocess
…lery#9222) * print is not thread-safe, so should not be used in signal handler * Moved unit tests to class test_WorkerApp * only writes when fd has file descriptor value * use the original __stdout__ and __stderr__ * sys.__stderr__ is not mutable * no format change for better diff * retain function interface --------- Co-authored-by: Asif Saif Uddin <[email protected]> Co-authored-by: Tomer Nosrati <[email protected]>
* Bump version: 5.5.0b2 → 5.5.0b3 * Added Changelog for v5.5.0b3
…_time_limit (celery#9246) * Correct the error description in exception message when validate soft_time_limit * Update celery/app/task.py --------- Co-authored-by: Asif Saif Uddin <[email protected]>
internal helper so no public API impact, but we may want to add to changelog
Updates the requirements on [elasticsearch](https://github.com/elastic/elasticsearch-py) to permit the latest version. - [Release notes](https://github.com/elastic/elasticsearch-py/releases) - [Commits](elastic/elasticsearch-py@0.4.1...v8.15.1) --- updated-dependencies: - dependency-name: elasticsearch dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Migrate workflows to Blacksmith * Removed "if: startsWith(matrix.os, ubuntu-)" * Skip "apt-get install" on windows --------- Co-authored-by: blacksmith-sh[bot] <157653362+blacksmith-sh[bot]@users.noreply.github.com> Co-authored-by: Tomer Nosrati <[email protected]>
…elery#9247) * Fixes celery#9119: inject dispatch_uid for retry-wrapped receivers - edited _make_lookup_key instead of _make_id, doesn't seem to be much of a difference, but I wanted the change as far up the stack as possible so devs can see it sooner - we can potentially also use functools.wraps and `__wrapped__` (i.e., `functools.wraps(fun)(retry_over_time)`, but this is a bit too generic for this type of solution, which may cause other issues * linting --------- Co-authored-by: Asif Saif Uddin <[email protected]> Co-authored-by: Omer Katz <[email protected]>
Need to set ALL the values (not clear in documentation still) for it to take effect, and conflicts between apps cause issues Per celery#7651 and celery#5935
* Bump pytest-celery to 1.1.2 * Revert "Added docker cleanup auto-fixture to improve smoke tests stability (celery#9243)" This reverts commit 90feae3. * Marked xfail for test_prefetch_count_restored with Redis - flaky test * Marked xfail for test_max_prefetch_not_passed_on_broker_restart with Redis - flaky test
* [TMP] removed unit/int tests * sudo sysctl -w vm.overcommit_memory=1 * Use redis.conf for the redis containers in the smoke tests * Changed Smoke-stamping max reruns from 3 to 5 in the CI only * Revert "[TMP] removed unit/int tests" This reverts commit 3376b82.
…chord (celery#9774) * fix: (celery#9773) task_id must not be empty with chain as body of a chord * fix:(celery#9773) pytest parameter name * Update t/unit/tasks/test_canvas.py Co-authored-by: Copilot <[email protected]> * fix: (celery#9773) pass new task_id (if none) and group_id in chord.run.freeze * feat: (celery#9773) add check_logs_for_error utility for log message verification --------- Co-authored-by: Asif Saif Uddin <[email protected]> Co-authored-by: Copilot <[email protected]>
…c queue (celery#9766) * Adds regression test for multiple chords using quorum queues and custom routing * Removes changes from Dockerfile * Fixes linting issues * Fixes long line linting * Removes fixed sleep * Update t/integration/test_rabbitmq_chord_unlock_routing.py * Update t/integration/test_rabbitmq_chord_unlock_routing.py Co-authored-by: Copilot <[email protected]> * Update t/integration/test_rabbitmq_chord_unlock_routing.py Co-authored-by: Copilot <[email protected]> * Uses logging instead of prints --------- Co-authored-by: Asif Saif Uddin <[email protected]> Co-authored-by: Copilot <[email protected]>
…ult_* settings (celery#9765) * Add test for fallback to 'direct' exchange and 'classic' queue when no routing is used This test verifies that Celery currently ignores task_default_exchange_type and task_default_queue_type for the default 'celery' queue/exchange, falling back to 'direct' and 'classic' respectively. Marked as xfail with strict=True to track future fix. * Fixes linting issues * Update t/integration/test_rabbitmq_default_queue_type_fallback.py Co-authored-by: Copilot <[email protected]> * Uses pyamqp instead of amqp on integration test * Tries resolving rabbitmq connection on CI * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update t/integration/test_rabbitmq_default_queue_type_fallback.py Co-authored-by: Copilot <[email protected]> * Update t/integration/test_rabbitmq_default_queue_type_fallback.py Waits for both rabbit and redis ports Co-authored-by: Copilot <[email protected]> * Fixes linting * Increases timeouts for rabbitmq connection and test execution --------- Co-authored-by: Asif Saif Uddin <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…elery#9770) * Add xfail test for RabbitMQ quorum queue QoS race condition This test simulates a quorum queue cluster propagation race where the first worker fails quorum detection and others succeed. It starts multiple workers concurrently and expects at least one AMQP error (e.g., 540 NOT_IMPLEMENTED) caused by applying global QoS to a quorum queue. The test is marked xfail since this behavior is a known RabbitMQ limitation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update t/integration/test_quorum_queue_qos_cluster_simulation.py * Update t/integration/test_quorum_queue_qos_cluster_simulation.py * Final version with comments * Uses processes instead of threads for better isolation * Fixes for CI * Tries adding extra safeguard to stop test * Tries to make connection parameters to rabbit and redis compatible with CI * Tries to fix test on CI * Fixes linting * Tries to force execution with no cache * Replaces cache sanitization steps. Tries fixing actions indentation. * Tries improving termination * Fixes pytest cache clearing. Tries to terminate process when catches error. * Fixes linting * Removes cache cleaning. Total refactory to try to avoid ci hanging * Adds missing xfail * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixes linting * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixes linting * Yet another try to avoid hanging on CI * Improves comments (just to trigger tests again that seem to be hanging on test_canvas) * Tries to improve termination to fix it on CI * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Final tentative * Adds missing xfail * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixes linting * Forces xfail * Tries yet another time to fix it on CI * Tries to add extra safeguard for CI, cleaning up processes that might be stranded * Yet another try * Tries using multiprocessing manager * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixes linting * Tries to force spawning and daemonizing * Messy WORKING version (local) * Cleans up WORKING local version * Addresses PR suggestions * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Removes debug step from integration tests flow * Reverts changes on python-package.yml * Reverts comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update t/integration/conftest.py * Update t/integration/test_security.py --------- Co-authored-by: Asif Saif Uddin <[email protected]>
…elery#9788) * fix: (celery#8786) time out when chord header fails with group body * fix: (celery#8786) PyPy mock mapping compatibility
* ISSUE-9738: Add root_id and parent_id for .apply() * ISSUE-9738: Flake8 fix * Update celery/app/task.py * tests: (celery#9738) root_id and parent_id for .apply() --------- Co-authored-by: Asif Saif Uddin <[email protected]>
…lery#9793) * Replace DelayedDelivery connection creation to use context manger * Fixed failing test because of usage of Mock which doesn't support context manager, replaced those places with MagicMock instead * Modify test test_start_native_delayed_delivery_topic_exchange to check connection context was used
…ns. (celery#9795) * Fix celery#9794: Pydantic integration fails with __future__.annotations. When a project uses `from __future__ import annotations`, all annotations will be stored as strings. This is fairly common, and many projects dictate that any use of type annotations must be accompanied by this import. The Pydantic integration in Celery introspects the annotations to check if any parameters and/or the return type are subclasses of `pydantic.BaseModel`. This fails when the annotations are `str` instead of the actual class. This change fixes the issue by optimistically using `typing.get_type_hints()` instead of relying on the annotations included in the result of `inspect.signature()`. This works in most cases, although there can be cases where `get_type_hints()` fails due to circular import chains. In this case, we fall back to the old implementation. A new test has been added to t/integration/test_tasks.py to validate the issue. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Asif Saif Uddin <[email protected]>
* add go and rust implementation in docs * Update docs/getting-started/introduction.rst * Update docs/getting-started/introduction.rst * Update docs/getting-started/introduction.rst --------- Co-authored-by: Asif Saif Uddin <[email protected]>
* Fix memory leak in exception handling (Issue celery#8882) - Enhanced traceback cleanup in celery/app/trace.py to prevent memory leaks - Added proper cleanup of ExceptionInfo objects and traceback references - Optimized traceback_clear() function by removing redundant f_locals access - Added comprehensive memory leak test suite in t/integration/test_memory_leak_8882.py - Fixed code quality issues: removed unused imports, cleaned whitespace, added noqa comments Memory usage improvement: 92% reduction (from ~70MB to ~0.6MB for 500 failing tasks) Addresses reference cycles that prevent garbage collection of traceback frames. All pre-commit hooks passing. * Better file name * Update t/integration/test_memory_leak_8882.py * Update t/integration/test_memory_leak_8882.py * Review changes: Add unit test * Update celery/app/trace.py * Review comments: Clear exceptions * separate the unit and integration tests * Update t/integration/test_mem_leak_in_exception_handling.py * Update t/integration/test_mem_leak_in_exception_handling.py * Update t/integration/test_mem_leak_in_exception_handling.py * Update t/integration/test_mem_leak_in_exception_handling.py * Update celery/app/trace.py * Update t/integration/test_mem_leak_in_exception_handling.py * Update t/integration/test_mem_leak_in_exception_handling.py * Update celery/app/trace.py * Update t/integration/test_mem_leak_in_exception_handling.py * precommit fix --------- Co-authored-by: Asif Saif Uddin <[email protected]>
Co-authored-by: Asif Saif Uddin <[email protected]>
Celery's use of `importlib_metadata` was removed in celery#9612 (since Celery now requires Python 3.8 which contains `importlib.metadata` in the standard library), but a few conditional imports were left behind. Co-authored-by: Asif Saif Uddin <[email protected]>
This reverts commit 9bf0546.
Replace deprecated datetime.utcfromtimestamp() with datetime.fromtimestamp() using timezone.utc. The deprecated method was removed in Python 3.12+. Also fix test timezone handling to create proper UTC timestamps and update assertions to expect timezone-aware datetime format. Fixes failing tests: - test_on_event_task_received - test_on_event_non_task
* Dockerfile Build Optimizations * Update docker/Dockerfile * Review Fixes --------- Co-authored-by: Asif Saif Uddin <[email protected]>
…up-docker-builder@v1 in the CI (celery#9846)
pytest-celery is a plugin with a different API, this page is about celery.contrib.pytest plugin.
* Revert "Use Django DB max age connection setting" This reverts commit f0c9b40. This reverts PR celery#6134 and stops using the close_if_unusable_or_obsolete API since there are edge cases where it's unable to detect if a connection if actually unusable. This is most obvious when Celery interrupts a query in progress via a time limit handler. Django has marked this issue as wontfix (https://code.djangoproject.com/ticket/30646). Since this is effectively an optimization for Celery that can't be reliably used, Celery ought to close the connection after each task instead of trying to manage connections in a way similar to how the Django application does. * Ensure django fixup never calls close_if_unusable_or_obsolete This API can fail to close unusable connections in certain scenarios, namely database failovers and ungraceful terminations (e.g. signal handler for time limit exceeded tasks). This makes close_if_unusable_or_obsolete adequate for HTTP request lifecycle management but inappropriate for use within celery workers. See also: https://code.djangoproject.com/ticket/30646 https://forum.djangoproject.com/t/close-if-unusable-or-obsolete-fails-to-close-unusable-connections/41900 * Add test for close_cache
* Add call to remove_pending_result, to counter add_pending_result in then.
* Add unittest for checking if remove_pending_result is called after a call to forget.
---------
Co-authored-by: Asif Saif Uddin {"Auvi":"অভি"} <[email protected]>
Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 5. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4...v5) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Asif Saif Uddin {"Auvi":"অভি"} <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note: Before submitting this pull request, please review our contributing
guidelines.
Description