Cron-based scheduling

[RealOrangeOne]

It's currently possible to execute a task after a given time with run_after. However, it's not possible to schedule a task "At 3am every day", for example.

[jribbens]

I've mentioned this in issue #22. I think this is a vital feature for a 1.0 release. Even the database backend for django-tasks itself needs it, for expiring old tasks from the database.

[RealOrangeOne]

I wouldn't consider it a "requirement". It's important, and useful, but you can always use an external scheduler for the cleanup, which is probably what people are using to run manage.py clearsessions

[jribbens]

I'd be willing to bet a huge number of installations (perhaps most) are simply not running clearsessions...

[chrisgrande]

Any thought to if scheduling should be possibly database backed? Similar to how django-celery-beat works? Or just in code registration?

One thing I would like would be a syntax similar to Laravel, it just reads better compared to cron syntax:

ScheduleTask("tasks.send_email").daily().weekdays().at("01:30")

[RealOrangeOne]

When it's supported, yes it should work with the database backend.

As for the syntax, it might end up being something more static. Perhaps as arguments to the @task decorator. But I agree I think literal cron syntax can be confusing, and in many ways limiting.

[jribbens]

Database backend is useful of course, but I think it's essential it also works from code. e.g. perhaps something in settings.py and also AppConfig - you want to be able to distribute apps that need, say, a daily background task without having to say "after installing this app, manually add an entry to the ScheduledTasks database table saying <...>"

[RealOrangeOne]

It could be something more dynamic, similar to how permissions work, but I agree the point of this is that it should be low-friction, so it shouldn't require any manual steps. Which is why I suggested something more module-level as opposed to something you might be tempted to define as part of say a view.

Keeping the state the database might work, and is probably required for concurrency reasons.

[andreasnuesslein]

But I agree I think literal cron syntax can be confusing, and in many ways limiting.

especially the absence of sub-minute interval support (at least in the standard cron, yes yes i know some crons can do something like */60 but that sucks even harder)

[hmpf]

You could have a standard managment command "cron_task" with "queue name" as argument that an external process can run. Then you don't have to think about scheduling syntax at all and leave that to cron/systemd/k8s/whatever.

[andreasnuesslein]

while this would be nice to have too, i think, in too many scenarios, it shifts the scheduling away from Dev to Ops.
of course no problem for buzzword DevOps, but enough developers don't have quick/direct control over the systemd-timers or sometimes even the cronjobs. i think it's often "cleaner" to tell the Ops just "we need this daemon" (e.g. celery beat) and what it does is Dev responsibility.

(also i feel like this kind of management command is quite easily hacked together in 15 loc)

[RealOrangeOne]

The problem with cron tasks is more about how to define them. Executing them I agree is very simple.

However, it's also not simple. This method would rely on one (exactly one) management command running. If there's an infrastructure issue where multiple instances are running, tasks would run multiple times. If the command isn't running when a task should be run, then it would be missed.

I think the ideal solution for this is to have "cron" style tasks executed by the same workers as conventional tasks (perhaps with some opt-out ability for throughput reasons).

[adnathanail]

I've implemented a very hacky task scheduler, to avoid having to have a separate container just for scheduling, using a view middleware and the Django cache.

This does add overhead to my page loads, but I have a very small throughput of requests, and pay for my hosting by the instance, so this makes sense for me for now. Might be useful to someone else!

from datetime import UTC, datetime, timedelta
from typing import Any

from django.core.cache import cache
from django.utils import timezone
from django_tasks import Task

from .tasks import a_task_that_runs_every_hour, a_task_that_runs_every_day

TASK_SCHEDULE: tuple[tuple[Task[Any, Any], int]] = (
    (a_task_that_runs_every_hour, 1),
    (a_task_that_runs_every_day, 24),
)


class ScheduledTaskMiddleware:
    """
    Hacky task scheduler, to avoid having to have another server
    ONLY SAFE TO BE USED WITH EXACTLY 1 WEB SERVER, AND 1 TASK SERVER

    Configure by adding tasks, and how many hours between each run, to the TASK_SCHEDULE variable above

    Stores time since last run of each task in Django cache
      this is wiped on server restart, so all tasks will run on each restart
    """

    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        response = self.get_response(request)

        for task, hours_between_runs in TASK_SCHEDULE:
            cache_key = f"scheduled_task_middleware_{task.name}_last_started"
            # Run the task on first request, or after the allotted time has expired
            task_last_started_timestamp = cache.get(cache_key)
            if task_last_started_timestamp is not None:
                task_last_started = datetime.fromtimestamp(
                    task_last_started_timestamp,
                    tz=UTC,
                )
                if (timezone.now() - task_last_started) < timedelta(
                    hours=hours_between_runs,
                ):
                    continue

            task.enqueue()
            cache.set(cache_key, timezone.now().timestamp())

        return response

[tcely]

I very much like the way that huey handled periodic tasks with the crontab function included.

https://huey.readthedocs.io/en/latest/guide.html#periodic-tasks

You can build on this fairly easily by having a scheduled task run every minute that is responsible for scheduling any tasks that were missed or that need to run in the next minute.

This allows for running tasks as many times per minute as you need and adjustments using eta and expires.

It is also possible to use a custom function that decides to run much less often if crontab doesn't fit your scheduling requirements.

The one thing that I don't enjoy with that design is the separation between periodic and normal tasks. It makes sense why they are different and not precisely interchangeable though.

The coordination between consumer processes could be improved as well. Consumers should really communicate and select which is responsible for scheduling new tasks into the queue without manual intervention.

I want to run as many consumers as I prefer without multiple tasks being injected into the queue for the same timestamp.

[djpeacher]

I really wanted to run a few cron tasks for a personal project so I "forked" the db_worker command and added a few lines to enqueue tasks defined in a CRON_TASKS setting. Probably not the best approach, but it does the job for me!

CRON_TASKS = [
    ("apps.tasks.send_daily_report", "0 9 * * *"),      # daily at 9 AM
    ("apps.tasks.backup_database", "0 0 * * 0"),        # weekly on Sunday at midnight
    ("apps.tasks.cleanup_old_files", "*/30 * * * *"),   # every 30 minutes
    ("apps.tasks.sync_external_data", "0 */4 * * *"),   # every 4 hours
    ("apps.tasks.send_reminders", "0 12 * * 1-5"),      # weekdays at noon
    ("apps.tasks.update_cache", "*/15 * * * *"),        # every 15 minutes
    ("apps.tasks.generate_reports", "0 1 1 * *"),       # monthly on 1st at 1 AM
]

$ python manage.py db_worker
[2025-08-04 09:42:01] Enqueuing 'apps.tasks.example_task'. Next run at 2025-08-04 09:43:00 (*/1 * * * *).
Task id=84e96bd8-d0ef-4f22-a86e-f0d103101398 path=apps.tasks.example_task state=RUNNING
Example task
Task id=84e96bd8-d0ef-4f22-a86e-f0d103101398 path=apps.tasks.example_task state=SUCCEEDED

Appreciation

• This comment for the idea to use the cache to track run times.
• This comment for the idea to enqueue tasks using the main worker.