How to override hydra.run.dir using Python structured config?

[funnym0nk3y]

Prolog: Please don't pay attention to the LLM-styled text. I used it to summarize everything I have done. I proofread for accuracy.

Hi all,

I’m working on a computer vision project using NanoDet, fully configured with Hydra’s structured configs in Python only — no YAML files involved. Our goal is to override the default hydra.run.dir so that all outputs — including Hydra’s internal .hydra/ directory, logs, and application artifacts like saved images and timings.log — consistently go into a single stable, timestamped output folder relative to our project root.

We want the output path to look like:

${NANODET_PROJECT_ROOT}/outputs/${now:%Y-%m-%d_%H-%M-%S}/

where NANODET_PROJECT_ROOT is an environment variable set to our absolute project root (e.g., /home/michael/masterarbeit-git/code-git/).

---

✅ Setup

• We use the @hydra.main() decorator in external/models/nanodet/demos/demo/demo.py.
• All config is defined via Python @dataclass structured configs; no YAML or CLI overrides for hydra.run.dir.
• Hydra version is >= 1.3.
• We want to set hydra.run.dir purely in Python code, not via CLI or YAML.
• We want the entire config, including core Hydra settings, to come from structured config only.

---

What we tried

1. Set environment variable:
   Exported

   export NANODET_PROJECT_ROOT="/home/michael/masterarbeit-git/code-git"

   before running the script.

2. Defined hydra.run.dir in a Python config run.py:

   # external/models/nanodet/nanodet/conf/hydra/run.py
   from dataclasses import dataclass
   
   @dataclass
   class RunConfig:
       # Override Hydra’s run directory to a stable location.
       # Uses env var NANODET_PROJECT_ROOT, which must be set.
       dir: str = "${oc.env:NANODET_PROJECT_ROOT}/outputs/${now:%Y-%m-%d_%H-%M-%S}"

3. Added hydra/run to base config defaults:

   # external/models/nanodet/nanodet/conf/base_config.py
   from dataclasses import dataclass, field
   from typing import Any, List
   
   @dataclass
   class Config:
       _defaults_: List[Any] = field(default_factory=lambda: [
           {"_self_": None},
           {"hydra/run": "_self_"}  # Added hydra.run config here
       ])
       project_root: str = "${nanodet_root:}"
       save_dir: str = (
           "${oc.env:PROJECT_OUTPUT_ROOT,${project_root}/outputs}/nanodet_outputs/"
           "${oc.env:JOB_NAME,nanodet_demo}/${oc.env:JOB_TIMESTAMP,${now:%Y-%m-%d_%H-%M-%S}}"
       )
       # other fields...

4. Our demo entry point:

   # external/models/nanodet/demos/demo/demo.py
   import hydra
   from nanodet.conf.base_config import Config
   
   @hydra.main(
       version_base=None,
       config_name="nanodet_plus_m_1_5x_416",
       config_path="../../nanodet/conf",
   )
   def main(cfg: Config) -> None:
       print(f"Hydra run dir: {cfg.hydra.run.dir}")  # Trying to access
       print(f"Application save dir: {cfg.save_dir}")
       # rest of demo logic...
   
   if __name__ == "__main__":
       main()

---

❌ What happens / fails

• Despite the above, hydra.run.dir still defaults to outputs/<date>/<time> relative to the current working directory from which demo.py is run.
• Hydra’s .hydra/ directory (with config.yaml, hydra.yaml) is generated inside these default folders, not our desired stable path.
• Our application artifacts (save_dir) correctly go to the intended stable path, but this causes a persistent split:
  Hydra internal outputs → default folder
  App outputs → stable folder
• Attempting to access cfg.hydra.run.dir sometimes results in Key 'hydra' is not in struct, indicating the Hydra config is not properly overridden or merged into the structured config.
• We get warnings about version_base if not set explicitly in the decorator.

---

❓ What I’m asking

• How do I reliably override hydra.run.dir via Python structured config only, respecting Hydra’s internal config system?
• Is there a recommended or supported pattern to inject core Hydra config like hydra.run.dir during config composition, when using @hydra.main() with structured configs?
• Do I need to register or define a special Hydra config class or use a particular syntax beyond adding it to _defaults_?
• How to avoid the "two output folders" problem cleanly in a pure Python config setup?

---

Thanks so much for any help! If needed, I can share minimal repro code or snippets — just ask!