Backtrack handler not supporting ChainOfThought (Or Predict)

[jnakhle-sabis]

What happened?

When I try executing the provided code, I keep getting this error:

ERROR dspy.utils.parallelizer: Error for Example({'question': "Question here", 'output': '1'}) (input_keys={'question'}): 'NoneType' object is not callable
Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/dspy/utils/parallelizer.py", line 52, in safe_func
return user_function(item)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/dspy/evaluate/evaluate.py", line 154, in process_item
score = metric(example, prediction)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not callable

Steps to reproduce

dff=pd.DataFrame(df).sample(10)

dspy.settings.configure(lm=lm,trace=[])

import random
import ast
import dspy

trainset=dff[:7]
testset=dff[7:]

trainset = [dspy.Example(question=trainset.loc[index, 'Student_answer'], output=trainset.loc[index, 'Grade']).with_inputs("question") for index in trainset.index]
testset = [dspy.Example(question=testset.loc[index, 'Student_answer'], output=testset.loc[index, 'Grade']).with_inputs("question") for index in testset.index]

class ReferenceAnswer(dspy.Signature):

question=dspy.InputField()
output=dspy.OutputField()

class PredictModel(dspy.Module):
def init(self):
self.predict=dspy.ChainOfThought(ReferenceAnswer)
def forward(self,question):
return self.predict( question=question)

suzan=PredictModel()

from dspy.evaluate import Evaluate
from dspy.evaluate.metrics import answer_exact_match

evaluate_program=Evaluate(devset=testset, metrics=answer_exact_match, provide_traceback=True)

evaluate_program(suzan)

DSPy version

dspy-2.6.11

[chenmoneygithub]

I cannot reproduce the error locally, closing it for now. If the issue still exists, I suggest directly run answer_exact_match on the output on the problematic data record so that you can get a clear stacktrace.