Intermittent "recorded hostname does not match instance hostname" errors with multiple active schedulers (LocalExecutor, Airflow 2.7.0)

[starlight1805]

We are running into an intermittent issue with Airflow 2.7.0 where many tasks fail at once with the following error:

[2025-09-23, 06:23:08 IST] {local_task_job_runner.py:253} ERROR -
The recorded hostname hostname1.com does not match this instance's hostname hostname2.com

Architecture

• Airflow 2.7.0
• LocalExecutor
• MySQL metadata DB
• All logs, DAGs, and configs stored on a NAS mount (shared between nodes)
• Active-active schedulers enabled (3 schedulers total)

Cluster setup:

• Node 1 → webserver + scheduler
• Node 2 → scheduler only
• Node 3 → scheduler only

Observed behavior

• Normally jobs run fine across all three nodes.
• About once every two weeks, ~50 DAGs fail simultaneously with the above hostname mismatch error.
• The mismatch is always the same pattern:
• This happens even if the job is actually running on Node 1, Node 2, or Node 3.
• After this window, jobs continue to run normally until the next occurrence.