Skip to content

Move dsp/metrics into evaluate/metrics #8402

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

TomeHirata
Copy link
Collaborator

We had dsp/utils/metrics and evaluate/metrics separately and re-export some metrics under dsp/utils/metrics in evaluate/metrics. To have a clear separation of concerns, this PR moves functions in dsp/utils/metrics into evaluate/metrics.

Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Centralize metric implementations by moving functions from dspy/dsp/utils/metrics.py into dspy/evaluate/metrics.py and updating exports.

  • Copied EM, F1, HotPotF1, normalize_text, em_score, f1_score, hotpot_f1_score, and precision_score into evaluate/metrics.py
  • Updated dspy/evaluate/__init__.py to re-export relocated metrics and removed old imports
  • Deleted the old metrics module and cleaned up imports in dspy/dsp/utils/__init__.py

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
dspy/evaluate/metrics.py Added and integrated core metric functions into the evaluation package
dspy/evaluate/init.py Updated top-level exports for relocated metrics functions
dspy/dsp/utils/metrics.py Removed legacy metrics module
dspy/dsp/utils/init.py Removed metrics import from utils initialization
Comments suppressed due to low confidence (3)

dspy/evaluate/metrics.py:29

  • Public function normalize_text lacks a docstring. Add a brief description of its behavior and parameters.
def normalize_text(s):

dspy/evaluate/init.py:3

  • Functions F1 and HotPotF1 are defined in metrics.py but not re-exported here. If they’re part of the public API, add them to the imports and __all__.
from dspy.evaluate.metrics import EM, answer_exact_match, answer_passage_match, normalize_text

dspy/evaluate/metrics.py:99

  • New function precision_score has no associated unit tests. Consider adding tests for normal and edge cases (e.g., empty inputs) to validate behavior.
def precision_score(prediction, ground_truth):

@@ -1,48 +1,50 @@
import dspy
from dspy.predict.chain_of_thought import ChainOfThought
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change here is for avoiding circular imports

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant