Task context environment variables are not set in callbacks running in dag processor

[deepak4babu]

Apache Airflow version

Other Airflow 2/3 version (please specify below)

If "Other Airflow 2/3 version" selected, which one?

2.10.3

What happened?

Task context environment variables are not set in callbacks running in dag processor.

We use the dag_id from AIRFLOW_CTX_DAG_ID environment variable to create some connections in our internal packages.

What you think should happen instead?

The task context environment variables should be set in callbacks run in dag processor especially on_failure_callbacks

How to reproduce

Create a dag with on failure callback set for a task. In the callback function, get the environment variables and save it to a file (as logging is not available for callbacks). The env list will not have AIRFLOW_CTX_* keys.

Operating System

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.8.3
apache-airflow-providers-cncf-kubernetes==10.1.0
apache-airflow-providers-common-compat==1.2.1
apache-airflow-providers-common-io==1.4.2
apache-airflow-providers-common-sql==1.19.0
apache-airflow-providers-fab==1.5.2
apache-airflow-providers-ftp==3.11.1
apache-airflow-providers-http==4.13.2
apache-airflow-providers-imap==3.7.0
apache-airflow-providers-smtp==1.8.0
apache-airflow-providers-sqlite==3.9.0

Deployment

Other

Deployment details

We use a slightly modified version of offical helm chart

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

The AIRFLOW_CTX_DAG_ID is only set during parsing, not during execution of callback. It is intended purely for optimisation of dynamic Dag parsing in worker. That's the design.

I am not sure why you would need it also in the callback - can't you get it from the context passed to the callback?

converting to a discussion if more discussion is needed

[deepak4babu]

@potiuk we have built some logic around the dag_id to provide database connection objects, abstracting from user.
Its not just this env, the envs injected as part of airflow policy is also not available in the callbacks

[potiuk]

Policy is about updating the Dags / tasks themselves not injecting extra information to callbacks. I think any meaningful discussion about it would need the examples of the code you use to help you to do things better possibly

[potiuk]

To be very clear - the fact that you do something with Python around and additionally to what airflow provides, does not mean that it is correct or supported. You might simply abuse some side effects that are not guaranteed or even desireable. The only thi g that you can rely on is the Public Interface described in Airflow docs.