Official Repository for Scenario Dreamer

Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environments
Luke Rowe^1,2,6, Roger Girgis^1,3,6, Anthony Gosselin^1,3, Liam Paull^1,2,5, Christopher Pal^1,2,3,5, Felix Heide^4,6
1 Mila, ² Université de Montréal, ³ Polytechnique Montréal, ⁴ Princeton University, ⁵ CIFAR AI Chair, ⁶ Torc Robotics

Computer Vision and Pattern Recognition (CVPR), 2025

We propose Scenario Dreamer, a fully data-driven closed-loop generative simulator for autonomous vehicle planning.

scenario_dreamer.mp4

Repository Timeline

• [06/11/2025] Environment setup
• [06/11/2025] Dataset Preprocessing
• [ETA: 07/06/2025] Train Scenario Dreamer autoencoder model on Waymo
• [ETA: 07/06/2025] Train Scenario Dreamer latent diffusion model on Waymo
• [ETA: 07/06/2025] Support visualization of Scenario Dreamer initial scenes
• [ETA: 07/13/2025] Support inpainting and lane-conditioned object generation modes on Waymo
• [ETA: 07/13/2025] Support evaluation of Scenario Dreamer model on Waymo
• [ETA: 07/13/2025] Compatibility with nuPlan dataset
• [ETA: 07/13/2025] Release of pre-trained Scenario Dreamer models on Waymo and nuPlan
• [ETA: 07/13/2025] Train CtRL-Sim behaviour model on Waymo
• [ETA: 07/31/2025] Train Scenario-Dreamer compatible agents in GPUDrive
• [ETA: 07/31/2025] Evaluate planners in Scenario Dreamer environments
• [ETA: 07/31/2025] SLEDGE and DriveSceneGen baseline reproduction and evaluation

Setup

Start by cloning the repository

git clone https://github.com/princeton-computational-imaging/scenario-dreamer.git
cd scenario-dreamer-waymo

This repository assumes you have a "scratch" directory for larger files (datasets, checkpoints, etc.). If disk space is not an issue, you can keep everything in the repository directory:

export SCRATCH_ROOT=$(pwd) # prefer a separate drive? Point SCRATCH_ROOT there instead.

Define environment variables to let the code know where things live:

source $(pwd)/scripts/define_env_variables.sh

Conda Setup

# create conda environment
conda env create -f environment.yml
conda activate scenario-dreamer

Waymo Dataset Preparation

Download the Waymo Open Motion Dataset (v1.1.0) into your scratch directory with the following directory structure:

$SCRATCH_ROOT/waymo_open_dataset_motion_v_1_1_0/
├── training/
│   ├── training.tfrecord-00000-of-01000
│   ├── …
│   └── training.tfrecord-00999-of-01000
├── validation/
│   ├── validation.tfrecord-00000-of-00150
│   ├── …
│   └── validation.tfrecord-00149-of-00150
└── testing/
    ├── testing.tfrecord-00000-of-00150
    ├── …
    └── testing.tfrecord-00149-of-00150

Then, we preprocess the waymo dataset to prepare for Scenario Dreamer model training. The first script takes ~12hrs and the second script takes ~12hrs (8 CPU cores, 64GB RAM):

bash scripts/extract_waymo_data.sh # extract relevant data from tfrecords and create train/val/test splits
bash scripts/preprocess_waymo_dataset.sh # preprocess data to facilitate efficient model training

Citation

@InProceedings{rowe2025scenariodreamer,
  title={Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environments},
  author={Rowe, Luke and Girgis, Roger and Gosselin, Anthony and Paull, Liam and Pal, Christopher and Heide, Felix},
  booktitle = {CVPR},
  year={2025}
}