Skip to content

Conversation

Copy link

Copilot AI commented Aug 22, 2025

This PR implements the Aggregated Jaccard Index (AJI) as described in Kumar et al. (2017) for evaluating instance segmentation performance in biomedical image analysis.

What is AJI?

The Aggregated Jaccard Index is a comprehensive metric that evaluates segmentation quality by:

  1. Finding optimal object matches between ground truth and predicted objects using IoU
  2. Computing aggregated intersection and union across all matched pairs
  3. Including unmatched objects in the denominator to penalize false positives and false negatives

The mathematical formulation is:

AJI = Σ |G_k ∩ P_σ(k)| / (Σ |G_k ∪ P_σ(k)| + Σ |P_j unmatched|)

Implementation Details

New AggregatedJaccardIndex class:

  • Inherits from AsymmetricMeasureMixin and Measure following existing patterns
  • Configurable parameters:
    • min_ref_size (default: 1) - Skip small ground truth objects
    • iou_threshold (default: 0.5) - Minimum IoU for considering objects as matched
  • Proper handling of edge cases (empty images, no matches, etc.)
  • Integration with the existing Study framework

Usage example:

import segmetrics as sm
import numpy as np

# Create ground truth and prediction
expected = np.array([[1, 1, 0], [1, 1, 0], [0, 0, 2]], dtype=np.uint8)
actual = np.array([[1, 1, 0], [1, 0, 0], [0, 0, 2]], dtype=np.uint8)

# Compute AJI
aji = sm.AggregatedJaccardIndex()
aji.set_expected(expected)
result = aji.compute(actual)
print(f"AJI: {result[0]}")  # 0.8

Changes Made

  1. Added AggregatedJaccardIndex class in segmetrics/regional.py
  2. Updated exports in segmetrics/__init__.py to include the new measure
  3. Added comprehensive tests covering various scenarios:
    • Perfect matches and partial overlaps
    • Empty images and no-match cases
    • IoU threshold behavior
    • Complex multi-object scenarios
  4. Updated test suite to include AJI in default name tests and study integration

Validation

The implementation has been thoroughly tested with:

  • Unit tests covering edge cases and parameter validation
  • Integration tests with the Study framework
  • Manual validation against the Kumar et al. (2017) specification
  • Comparison with expected mathematical results

This addition provides researchers in biomedical image analysis with a standardized implementation of this important metric for evaluating instance segmentation algorithms.

Reference

Kumar, N., Verma, R., Sharma, S., Bhargava, S., Vahadane, A., & Sethi, A. (2017). A dataset and a technique for generalized nuclear segmentation for computational pathology. IEEE Transactions on Medical Imaging, 36(7), 1550-1560.


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI changed the title [WIP] Add an implementation of the Aggregated Jaccard Index (AJI) as described in Kumar et al. (2017) Add implementation of Aggregated Jaccard Index (AJI) metric Aug 22, 2025
Copilot AI requested a review from kostrykin August 22, 2025 10:15
Copilot finished work on behalf of kostrykin August 22, 2025 10:15
@kostrykin
Copy link
Member

@copilot Instead of adding the test_aggregated_jaccard_index method to the MeasureTest test case class, create a new test case class called AJITest add the tests to this one.

Copy link
Author

Copilot AI commented Aug 22, 2025

@copilot Instead of adding the test_aggregated_jaccard_index method to the MeasureTest test case class, create a new test case class called AJITest add the tests to this one.

Done! I've moved the test_aggregated_jaccard_index method from the MeasureTest class to a new dedicated AJITest class as requested. The tests are now properly organized and continue to pass. Commit: a62106f

Copilot finished work on behalf of kostrykin August 22, 2025 10:40
@kostrykin
Copy link
Member

@copilot Split the different test cases in AJITest.test_aggregated_jaccard_index ("Test case 1", "Test case 2", etc.) into separate test methods of the test case class, like .test_perfect_match, test_partial_overlap, etc.

Copy link
Author

Copilot AI commented Aug 22, 2025

@copilot Split the different test cases in AJITest.test_aggregated_jaccard_index ("Test case 1", "Test case 2", etc.) into separate test methods of the test case class, like .test_perfect_match, `...

Done! I've split the test_aggregated_jaccard_index method into 7 separate test methods with descriptive names:

  • test_perfect_match
  • test_partial_overlap
  • test_no_overlap
  • test_empty_images
  • test_ground_truth_empty_prediction_non_empty
  • test_prediction_empty_ground_truth_non_empty
  • test_iou_threshold

Each test method now has a clear, focused purpose and all tests continue to pass. Commit: 966e5e3

Copilot finished work on behalf of kostrykin August 22, 2025 12:18
@kostrykin
Copy link
Member

@copilot Fix flake8 linting issues in segmetrics/regional.py

Copy link
Author

Copilot AI commented Aug 23, 2025

@copilot Fix flake8 linting issues in segmetrics/regional.py

Fixed all flake8 linting issues in segmetrics/regional.py. The issues addressed include:

  • Fixed E501 line too long errors by breaking long lines appropriately
  • Removed W293 blank line whitespace errors
  • Fixed W291 trailing whitespace
  • Corrected F631 assertion syntax error

All tests pass and AJI functionality remains intact. Commit: 345a75f

Copilot finished work on behalf of kostrykin August 23, 2025 10:32
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@kostrykin kostrykin marked this pull request as ready for review August 23, 2025 10:51
@github-actions
Copy link

Test coverage of 9ffee05: 88%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants