Changed ownership of the logs/ folder causing outage

[gudata]

Apache Airflow version

Airflow 3 version (please specify below)

3.0.4

What happened?

I am running an airflow from the compose file found at airflow samples.
I have ~150 virtual machines accross 4 aws accounts running the airflow-airflow-worker-1 process only, The workers are scaled from 1 to N.
I have each aws account one main node with the api-server, scheduler, triggerer.

Today all 3 production, 2 dev environments was not working because the logs/ folder has root:root ownership instead of airflow:root ownership as previously.

Only the qa setup was working because it was on airflow 3.0.6 version.

What you think should happen instead?

The logs/ folder should not be changed.

How to reproduce

No idea.

Operating System

amazon linux 2023

Versions of Apache Airflow Providers

the ones from the compose -apache/airflow:3.0.4

Deployment

Docker-Compose

Deployment details

Using celery.
On each node, the logs/ folder is mounted in each container.

Anything else?

This setup was working 31 days without any changes.
Today at the same time, with the same error failed 5 airflow installs in 4 different aws accounts with the same problem.

I assume airflow has changed some script which is dynamically pulled on boot of the workers/airflow
This starts happening from 2025-09-26 00:00:00 EET

after manually restoring the permissions everythings works fine.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[gudata]

On the machines there are also running crowdstrike agent, and datadog.
From their logs we see that the logs/ folder is not modified by them.

[potiuk]

I think you likely had a side effect of bad settings in "productionising" of your docker-compose.

The https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/docker-compose/docker-compose.yaml is clearly explained bot in the docs and the compose file itself as not suitable for production environment:

In the file:

# Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL.
#
# WARNING: This configuration is for local development. Do not use it in a production deployment.
#
# This configuration supports basic configuration using environment variables or an .env file

In the docs:

This procedure can be useful for learning and exploration. However, adapting it for use in real-world situations can be complicated and the docker compose file does not provide any security guarantees required for production system. Making changes to this procedure will require specialized expertise in Docker & Docker Compose, and the Airflow community may not be able to help you.

For that reason, we recommend using Kubernetes with the Official Airflow Community Helm Chart when you are ready to run Airflow in production.

Also it is explained what is the "Deployment Manager" (i.e. you) responsibilities when using installation with just docker containers - and that includes making your docker compose adjusted for any production settings. Basically the docker-compose you get from us is for learning and exploration and it's up to you entirely to make it works in the way your deployment and production requirement expect to have. This also means that you are responsible for appropriate configuration and automation of all the environment your docker compose runs.

So diagnosing and solving the problem is on you - as you are solely responsible to make the docker-compose configuration suitable to run in your production environment.

However I can provide some hints:

There are few processes in the learning docker-compse that are run with root if improperly configured - and it's possible that when they are run with root they change permissions, or maybe you have some impersonation (sudo) - this is the way how ownership of a folder can be changed to root - when the folder is created by "root" user.

Airflow user (which is used by most processes) cannot physically do it - unless you geve it sudo capabilitites. This is basic UNIX and container knowledge, nothing specific to airflow.

Those "root" containers are only run when compose is restarted and they change owner of folders created to AIRFLOW_UID - which (as explained in the warning if you don't do it) shoudl have AIRFLOW_UID set so that it can change the permissions. This is also explained in the docs. Possibly your setup caused execution of that init container without AIRFLOW_UID set. But this is (again) part of your deployment and docker-compose expert knowledge that you have to have when deciding on creating your production setup to investigate and diagnose when and how it happened (if my hypothesis is right).

There might also be other reasons, but only you can tell what happened looking at your logs and knowing how you run things.

Also - if you find out, it would be great you report it back here, that might help other people who might have similar issues with their setup.

Here is the part that you should start looking at:

  airflow-init:
    <<: *airflow-common
    entrypoint: /bin/bash
    # yamllint disable rule:line-length
    command:
      - -c
      - |
        if [[ -z "${AIRFLOW_UID}" ]]; then
          echo
          echo -e "\033[1;33mWARNING!!!: AIRFLOW_UID not set!\e[0m"
          echo "If you are on Linux, you SHOULD follow the instructions below to set "
          echo "AIRFLOW_UID environment variable, otherwise files will be owned by root."
          echo "For other operating systems you can get rid of the warning with manually created .env file:"
          echo "    See: https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#setting-the-right-airflow-user"
          echo
          export AIRFLOW_UID=$$(id -u)
        fi
        one_meg=1048576
        mem_available=$$(($$(getconf _PHYS_PAGES) * $$(getconf PAGE_SIZE) / one_meg))
        cpus_available=$$(grep -cE 'cpu[0-9]+' /proc/stat)
        disk_available=$$(df / | tail -1 | awk '{print $$4}')
        warning_resources="false"
        if (( mem_available < 4000 )) ; then
          echo
          echo -e "\033[1;33mWARNING!!!: Not enough memory available for Docker.\e[0m"
          echo "At least 4GB of memory required. You have $$(numfmt --to iec $$((mem_available * one_meg)))"
          echo
          warning_resources="true"
        fi
        if (( cpus_available < 2 )); then
          echo
          echo -e "\033[1;33mWARNING!!!: Not enough CPUS available for Docker.\e[0m"
          echo "At least 2 CPUs recommended. You have $${cpus_available}"
          echo
          warning_resources="true"
        fi
        if (( disk_available < one_meg * 10 )); then
          echo
          echo -e "\033[1;33mWARNING!!!: Not enough Disk space available for Docker.\e[0m"
          echo "At least 10 GBs recommended. You have $$(numfmt --to iec $$((disk_available * 1024 )))"
          echo
          warning_resources="true"
        fi
        if [[ $${warning_resources} == "true" ]]; then
          echo
          echo -e "\033[1;33mWARNING!!!: You have not enough resources to run Airflow (see above)!\e[0m"
          echo "Please follow the instructions to increase amount of resources available:"
          echo "   https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#before-you-begin"
          echo
        fi
        echo
        echo "Creating missing opt dirs if missing:"
        echo
        mkdir -v -p /opt/airflow/{logs,dags,plugins,config}
        echo
        echo "Airflow version:"
        /entrypoint airflow version
        echo
        echo "Files in shared volumes:"
        echo
        ls -la /opt/airflow/{logs,dags,plugins,config}
        echo
        echo "Running airflow config list to create default config file if missing."
        echo
        /entrypoint airflow config list >/dev/null
        echo
        echo "Files in shared volumes:"
        echo
        ls -la /opt/airflow/{logs,dags,plugins,config}
        echo
        echo "Change ownership of files in /opt/airflow to ${AIRFLOW_UID}:0"
        echo
        chown -R "${AIRFLOW_UID}:0" /opt/airflow/
        echo
        echo "Change ownership of files in shared volumes to ${AIRFLOW_UID}:0"
        echo
        chown -v -R "${AIRFLOW_UID}:0" /opt/airflow/{logs,dags,plugins,config}
        echo
        echo "Files in shared volumes:"
        echo
        ls -la /opt/airflow/{logs,dags,plugins,config}

[potiuk]

Converted it into a discussion.