Command airflow db clean does not work anymore with BashOperator in Airflow 3+

[jlandercy]

I use Airflow 3.1.0 with Docker Compose (official file).
I would like to automatize Airflow cleaning using Airflow.

If I had to make it by hand, I would open the worker container:

docker exec -it airflow-airflow-worker-1 bash

And issue the following command:

airflow db clean --clean-before-timestamp 20250930 --yes

I can also confirm the container does know the connection string:

echo $AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
postgresql+psycopg2://airflow:secret@postgres/airflow

Anyway when I run this command using a BashOperator:

    clean_database = BashOperator(
        task_id="clean-database",
        bash_command=f'airflow db clean --clean-before-timestamp {clean_before.isoformat()} --yes',
    )

I get the following error:

sqlalchemy.exc.ArgumentError: Could not parse SQLAlchemy URL from string 'airflow-db-not-allowed:///'

Which seems to be an explicit forbidden with a volontary bad shaped connection string.
I have not found any relevant documentation about this error.

I have confirmed that the BashOperator does execute within the same container airflow-airflow-worker-1.

My question is why my BashOperator blocked when issuing this command and how can I configure it properly to make it work ?

Related question I opened on SO: https://stackoverflow.com/questions/79779432/how-can-i-run-airflow-db-clean-command-using-bashoperator-with-airlfow-3

[salimuddin07]

You have a few ways to handle this:

1. Use airflow db clean outside the DAG

The simplest approach is to keep DB cleaning as a manual or scheduled job outside Airflow:

Use a cron job or systemd timer that executes the command inside the container.

Example:

docker exec airflow-airflow-worker-1 airflow db clean --clean-before-timestamp 20250930 --yes

2. Use a separate script and run it as a container task

Create a small Docker container (or script) that runs the cleaning command.

Schedule it via Airflow using a KubernetesPodOperator or DockerOperator instead of BashOperator.

The container can safely have the real DB connection string, bypassing the “blocked” environment.

3. Explicitly pass the DB connection as an environment variable

If you really want to use BashOperator, you can override the environment:

clean_database = BashOperator(
    task_id="clean-database",
    bash_command=f'airflow db clean --clean-before-timestamp {clean_before.isoformat()} --yes',
    env={
        "AIRFLOW__DATABASE__SQL_ALCHEMY_CONN": "postgresql+psycopg2://airflow:secret@postgres/airflow"
    }
)

Caution: This works, but it bypasses the security protection, so use it only if you understand the risks.

4. Use Airflow API / maintenance commands instead

If your goal is to clean old DAG runs or logs, you can use Airflow’s Python API (airflow.models.DagRun deletion) in a PythonOperator, which is safer than calling destructive CLI commands.

Recommended Approach

For production, the safest and most maintainable way is:

Keep airflow db clean outside DAGs, running either via cron or as a one-off container job.

If you need to automate via DAG, use DockerOperator or KubernetesPodOperator with explicit DB connection, not BashOperator.

Avoid passing destructive commands directly inside DAGs, because Airflow 3 intentionally prevents it for safety.

[jlandercy]

Thank you for answering with such details.
It does make sense now.

As a side note 3. does not work in my configuration.

Have a good day.
Cheers