-
Notifications
You must be signed in to change notification settings - Fork 23
Open
Description
I get the following error when running an implementation (mostly the same code) as eval/inspet_ai/scicode.py
through Inspect AI:
eval/inspect_ai/scicode.py
at line 172:
│ 455 │ │ │ │ │ │ ) │
│ 456 │ │ │ │ │ │ self.previous_llm_code[prev_step] = function_code │
│ 457 │ │ │ │ │ else: │
│ ❱ 458 │ │ │ │ │ │ raise Exception(f'Generating problem {prob_id} step {num_steps} │
│ 459 │ │ │
│ 460 │ │ prompt, previous_code = self.generate_prompt_with_steps( │
│ 461 │ │ │ prob_data, │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Exception: Generating problem 13 step 7 ahead of step 6.
this is run on the test split, and the error seems to go away on the validation split.
changing the number of --max-connections
has no effect on this error. This was run with openai/o3
as a model.
Metadata
Metadata
Assignees
Labels
No labels