Skip to content

Conversation

@EllingtonKirby
Copy link
Collaborator

Adding this as a sanity check, mainly for the nuplan data.

My assumption is that when replaying the log trajectories we would see 100% goal completion, and very low off-road and collision rates. This would indicate that "perfect" driving on these scenarios is possible (I had a slight fear that the nuplan agents had a higher off-road rate due to an issue with the data).

When I evaluate 1300 nuplan scenarios with the expert trajectories we see:

        goal_achieved_frac      collided_frac      off_road_frac      other_frac     
                      mean  std          mean  std          mean  std       mean  std
dataset                                                                              
test                 100.0  0.0           0.0  0.6           1.2  6.1        0.0  0.0
train                100.0  0.0           0.0  0.5           1.4  8.6        0.0  0.0

Agent-based metrics 

Total agents: 20918.0 in 1300 scenes
Collision rate: 0.028683431446552277
Offroad rate: 0.5975714884698391
Goal rate: 100.0
Other rate: 0.0

The off-road rate is still a bit higher than I might expect, but it is still significantly lower than the best we have recorded on nuplan scenarios so far (~8%).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants