[RFC] structured configs for verl

[eric-haibin-lin]

As initially mentioned in #1941, having structured configuration classes in verl makes argument passing easier for testing and validation.

This is an extended thread on the current implementation of configuration schema in verl. Related PRs:

• [cfg, perf] refactor: add omega_conf_to_dataclass API, rename WorkerProfiler to DistProfiler, add unit test based on ProfilerConfig #2117
• [cfg] refactor: add ActorConfig, EngineConfig, and ActorWorker unit test, refactor validation code #2621

Motivation

By moving from loose omegaconfig.DictConfig-based parameters to structured dataclasses, we gain:

• Type safety & IDE support when accessing fields (e.g. cfg.optim.lr).
• Validation hooks via post_init in each class.
• Immutable defaults with controlled mutability (e.g., an extra field).
• Seamless Hydra/OmegaConf integration and easy per-recipe extension.

Core: BaseConfig

All config dataclasses inherit from BaseConfig, which:

• Implements collections.abc.Mapping → dict-like iteration/access.
• Freezes attributes once set, unless listed in _mutable_fields.
• Provides an extra: dict[str, Any] for unchecked extensions.

@dataclass
class BaseConfig(collections.abc.Mapping):
    """Dict-like, frozen dataclass with opt-in mutability."""
    _mutable_fields: set[str] = {"extra"}
    extra: dict[str, Any] = field(default_factory=dict)

    def __setattr__(self, name: str, value):
        if name in self.__dict__ and name not in self._mutable_fields:
            raise FrozenInstanceError(f"Field '{name}' is frozen")
        super().__setattr__(name, value)

    # Mapping methods: get, __getitem__, __iter__, __len__ …

Example Config Classes (verl/trainer/config)

Each sub-component of the trainer has its own dataclass, inheriting BaseConfig.

critic:
  checkpoint:
    _target_: verl.trainer.config.CheckpointConfig
    save_contents: ["model","optimizer","extra"]
    load_contents: ["model","optimizer","extra"]
    async_save: false

Definition:

@dataclass
class CheckpointConfig(BaseConfig):
    """What to save/load and async behavior."""
    save_contents: list[str] = field(default_factory=lambda: ["model","optimizer","extra"])
    load_contents: list[str] = field(default_factory=lambda: ["model","optimizer","extra"])
    async_save: bool = False

    def __post_init__(self):
        # validation checks go here after initialization


ckpt_cfg = CheckpointConfig(async_save=True)
print(ckpt_cfg.save_contents)
print(ckpt_cfg.get("save_contents", default_value))
print(ckpt_cfg["save_contents"])

# converting hydra-generated omegaconf.DictConfig to the dataclass config:
from verl.utils.config import omegaconf_to_dataclass
ckpt_cfg_from_cli = omegaconf_to_dataclass(config.critic.checkpoint)

Extending existing config classes

Because now configs become structured, unexpected keys would raise exceptions. To add new keys, there are two ways:

Explicit class extensions:

from verl.workers.config import FSDPActorConfig

@dataclass
class SPPOActorConfig(FSDPActorConfig):
    """Add SPPO-specific temperature/penalty."""
    sppo_eta: float = 1.0

When using yaml or from command line, update the target config class:

hydra:
  searchpath:
    - file://verl/trainer/config
defaults:
  - ppo_trainer      # base trainer config
  - _self_               # then apply these overrides

actor_rollout_ref:
  actor:
    _target_:  recipe.sppo.config.SPPOActorConfig # **new target dataclass required for extension **
    sppo_eta: 1.0  

or directly from command line:

python main_sppo.py \
  actor_rollout_ref.actor._target_=recipe.sppo.config.SPPOActorConfig \
  actor_rollout_ref.actor.sppo_eta=1.0

Leverage the extra field

Adding more keys to the extra field of any dataclass that inherits from BaseConfig also works. This way there's no need to define your own dataclass in python:

hydra:
  searchpath:
    - file://verl/trainer/config
defaults:
  - ppo_trainer      # base trainer config
  - _self_               # then apply these overrides

actor_rollout_ref:
  actor:
    extra:
        sppo_eta: 1.0  

Declaring mutable fields

For historical reasons some fields in the configs are mutated inplace in the codebase such as batch size for data/sequence parallelism. We are in the process of deprecating this kind of behavior. However, if you want to intentionally mutate one field, specify it with the _mutable_fields attr:

@dataclass
class CheckpointConfig(BaseConfig):
    """What to save/load and async behavior."""
    _mutable_fields = BaseConfig._mutable_fields | {"save_contents"} # mark save_contents as mutable.

    save_contents: list[str] = field(default_factory=lambda: ["model","optimizer","extra"])
    load_contents: list[str] = field(default_factory=lambda: ["model","optimizer","extra"])
    async_save: bool = False

Other helpful resources

verl default trainer configs combines the following config files together, specified in the _defaults_ field: https://github.com/volcengine/verl/blob/main/verl/trainer/config/ppo_trainer.yaml#L1-L36

• verl/trainer/config/ppo_trainer.yaml # main config for entrypoint
• verl/trainer/config/actor/dp_actor.yaml
• verl/trainer/config/critic/dp_critic.yaml
• verl/trainer/config/reward_model/dp_reward_model.yaml
• verl/trainer/config/rollout/rollout.yaml

To quickly peek the default full config in a single file, you can check the auto-generated full config in https://github.com/volcengine/verl/blob/main/verl/trainer/config/_generated_ppo_trainer.yaml

[vadimkantorov]

I propose to upstream dataclass config classes from Trinity:

It builds up on Verl and validated the approach with dataclass configs and OmegaConf.

E.g. here is an example of partial Verl config dataclass:

https://github.com/nkkarpov/Trinity-RFT/blob/main/trinity/common/verl_config.py