Skip to content

Commit 215fda4

Browse files
hrch3nhrch3n
committed
v1.2
1 parent c9ecd4e commit 215fda4

File tree

133 files changed

+1291
-80
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

133 files changed

+1291
-80
lines changed

‎LICENSE

Lines changed: 21 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,21 @@
1-
MIT License
2-
3-
Copyright (c) 2021 Haoran Chen
4-
5-
Permission is hereby granted, free of charge, to any person obtaining a copy
6-
of this software and associated documentation files (the "Software"), to deal
7-
in the Software without restriction, including without limitation the rights
8-
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9-
copies of the Software, and to permit persons to whom the Software is
10-
furnished to do so, subject to the following conditions:
11-
12-
The above copyright notice and this permission notice shall be included in all
13-
copies or substantial portions of the Software.
14-
15-
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16-
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17-
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18-
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19-
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20-
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21-
SOFTWARE.
1+
MIT License
2+
3+
Copyright (c) 2021 Haoran Chen, Robert F. Murphy and Ce Zhang
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md

Lines changed: 89 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -1,59 +1,89 @@
1-
# Cell Segmentation Evaluator: evaluation of cell segmentation methods without reference segmentations
2-
Haoran Chen and Robert F. Murphy\
3-
Carnegie Mellon University\
4-
V1.0.2 Aug 24, 2022
5-
6-
## Description of idea
7-
We present here an approach that seeks to evaluate cell segmentation methods without relying upon comparison to results from humans.
8-
For this, we defined a series of segmentation quality metrics that can be applied to multichannel fluorescence images.
9-
We calculated these metrics for 11 previously-described segmentation methods applied to datasets from 4 multiplexed microscope modalities covering 5 tissues.
10-
Using principal component analysis to combine the metrics we defined an overall cell segmentation quality score and ranked the segmentation methods.
11-
12-
## Use of package
13-
This package allows users to
14-
1. Reproduce all results that we generated for the paper on multichannel images from HuBMAP public portal.
15-
2. Seek the most suitable segmentation method for one or a batch of multichannel images.
16-
3. Obtain the segmentation quality score on given segmentation masks.
17-
18-
The package was tested under Ubuntu 18.04 LTS.
19-
20-
21-
## Execution
22-
### Step 0: Setup environment
23-
```bash
24-
pip install -r requirements.txt
25-
```
26-
27-
### Step 1: Setup configuration file
28-
```bash
29-
python generate_config.py
30-
```
31-
This step generates config.json which contains all configurational parameters and options in order to run the evaluation pipeline. The users will be asked a series of questions about how they would like to utilize this package. The answers will be stored in config.json for the pipeline to read. Meanwhile, all necessay data, softwares and dependencies will also be automatically downloaded or installed based on users' responses.
32-
33-
### Step 2: Run the pipeline
34-
```bash
35-
python run_pipeline.py configuration_file_path
36-
```
37-
This step runs the evaluation pipeline given the generated configuration file.
38-
39-
## Segmentation masks of HuBMAP images
40-
All segmentation masks and evaluation results we generated for the paper can be download from Google Drive folder below:
41-
42-
https://drive.google.com/drive/folders/14tw4qrXWTt2eg64zOpYjE9cZ7OFo1i6b?usp=sharing
43-
44-
The segmentation masks would be automatically downloaded if you choose to reproduce all results of the paper at step 1 above.
45-
46-
## Documentation
47-
48-
For more detailed introduction of segmentation quality metrics among other image quality metrics, please see
49-
[HuBMAP Image Quality Control Metrics](http://hubmap.scs.cmu.edu/wp-content/uploads/2021/09/HuBMAP-Image-Quality-Control-Metrics-v1.5.pdf)
50-
51-
## Citation
52-
If you find our package useful in your research, please cite our bioRxiv paper:
53-
> Chen, Haoran, and Robert F. Murphy. "Evaluation of cell segmentation methods without reference segmentations." bioRxiv (2022): 2021-09.
54-
55-
## Contact
56-
57-
Robert F. Murphy - [email protected]\
58-
Haoran Chen - [email protected]
59-
1+
# Cell Segmentation Evaluator: evaluation of cell segmentation methods without reference segmentations
2+
Haoran Chen and Robert F. Murphy and Ted Zhang\
3+
Carnegie Mellon University\
4+
V1.2 Jan 05, 2023
5+
6+
## Cell Segmentation Approach
7+
This package implements an approach for cell segmentation evaluation (CSE) without relying upon comparison to results from humans.
8+
For this, we defined a series of segmentation quality metrics that can be applied to multichannel fluorescence images.
9+
We calculated these metrics for 11 previously-described segmentation methods applied to 2D images from 4 multiplexed microscope modalities covering 5 tissues.
10+
Using principal component analysis to combine the metrics we defined an overall cell segmentation quality score.
11+
We also defined similar metrics and quality score for 3D images.
12+
13+
Reference: Haoran Chen and Robert F. Murphy (2022) Evaluation of cell segmentation methods without reference segmentations. Molecular Biology of the Cell. https://doi.org/10.1091/mbc.E22-08-0364
14+
15+
## Package contents
16+
This package contains two implementations of the cell segmentation metrics as well as example images for testing. The first implementation ("SimpleCSE") just calculates metrics and quality score for one or more images and corresponding cell segmentation masks.
17+
It was tested on python >=3.8 under Ubuntu 18.04 LTS.
18+
19+
## SimpleCSE
20+
21+
This folder contains a simplified version of the CSE that calculate the metrics and quality score given a multichannel image and a corresponding cell mask.
22+
23+
It is provided as a python main program and as an example Jupyter Notebook ("SegEvalExample.ipynb").
24+
25+
The multichannel image should be in a format readable by AICSimageio (e.g., OME TIFF). The masks should be in a similar format with an indexed image for cell masks in the first channel and an indexed image (with corresponding indices) for nuclear masks in the second channel.
26+
27+
The output is a JSON file with the metrics and the scores.
28+
29+
## Execution from command line
30+
### Step 1:
31+
Download the "SimpleCSE" folder and change your default directory ("cd SimpleCSE") to that folder
32+
### Step 2:
33+
Run
34+
```bash
35+
pip install -r requirements.txt
36+
```
37+
### Step 3
38+
Run
39+
```bash
40+
python seg_eval_main.py --img-dir [img_dir_path] --mask-dir [mask_dir_path]
41+
```
42+
Please download and extract example images and masks from [example_data.zip](https://drive.google.com/file/d/1qtAbPQBiU4SxXFDxl-UhOShTaQszB1Ca/view?usp=share_link). The "example_data" folder contains example 2D CODEX and 3D IMC images and their corresponding cell masks. To run them, download the example data use
43+
44+
python seg_eval_main.py --img-dir [example_data/imgs/2D_CODEX.ome.tiff] --mask-dir [example_data/masks/2D_CODEX.ome.tiff]
45+
46+
python seg_eval_main.py --img-dir [example_data/imgs/3D_IMC.ome.tiff] --mask-dir [example_data/masks/3D_IMC.ome.tiff]
47+
48+
49+
##full CSE pipeline
50+
51+
This implementation can be used to seek the most suitable segmentation method for one or a batch of multichannel images.
52+
53+
## Execution: find most suitable segmentation method for multichannel images
54+
### Step 1:
55+
Download "full_pipeline" folder and change your default directory to that folder
56+
### Step 2: Setup environment
57+
```bash
58+
pip install -r requirements.txt
59+
```
60+
61+
### Step 3: Setup configuration file
62+
```bash
63+
python generate_config.py
64+
```
65+
This step generates config.json which contains all configurational parameters and options in order to run the evaluation pipeline. The user will be asked a series of questions about how they would like to utilize this package. The answers will be stored in config.json for the pipeline to read. Meanwhile, all necessary data, software and dependencies will also be automatically downloaded or installed based on user responses.
66+
67+
### Step 4: Run the pipeline
68+
```bash
69+
python run_pipeline.py configuration_file_path
70+
```
71+
This step runs the evaluation pipeline given the generated configuration file.
72+
73+
## Segmentation masks of HuBMAP images
74+
All segmentation masks and evaluation results we generated for the paper can be download from the Google Drive folder below:
75+
76+
https://drive.google.com/drive/folders/14tw4qrXWTt2eg64zOpYjE9cZ7OFo1i6b?usp=sharing
77+
78+
The segmentation masks will be automatically downloaded if you choose to reproduce all results of the paper at step 1 above.
79+
80+
## Documentation
81+
82+
For more detailed introduction of segmentation quality metrics among other image quality metrics, please see
83+
[HuBMAP Image Quality Control Metrics](http://hubmap.scs.cmu.edu/wp-content/uploads/2021/09/HuBMAP-Image-Quality-Control-Metrics-v1.5.pdf)
84+
85+
86+
## Contact
87+
88+
Robert F. Murphy - [email protected]\
89+
Haoran Chen - [email protected]

‎SimpleCSE/.DS_Store

6 KB
Binary file not shown.

‎SimpleCSE/2D_PCA_model.pickle

1.83 KB
Binary file not shown.

‎SimpleCSE/3D_PCA_model.pickle

1.75 KB
Binary file not shown.

‎SimpleCSE/SegEvalExample.ipynb

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"## EXAMPLE MAIN PROGRAM TO CALCULATE SEGMENTATION EVALUATION STATISTICS FOR A SINGLE IMAGE AND MASK\n",
8+
"## Authors: Haoran Chen and Robert F. Murphy and Ted Zhang\n",
9+
"## this assumes that the required python packages have already been installed using\n",
10+
"### pip install -r requirements.txt"
11+
]
12+
},
13+
{
14+
"cell_type": "code",
15+
"execution_count": 1,
16+
"metadata": {},
17+
"outputs": [],
18+
"source": [
19+
"from pathlib import Path\n",
20+
"import pickle\n",
21+
"from read_and_eval_seg import read_and_eval_seg"
22+
]
23+
},
24+
{
25+
"cell_type": "code",
26+
"execution_count": 2,
27+
"metadata": {},
28+
"outputs": [
29+
{
30+
"name": "stdout",
31+
"output_type": "stream",
32+
"text": [
33+
"Calculating single-method metrics v1.5 for example_data/imgs\n",
34+
"{'Matched Cell': {'NumberOfCellsPer100SquareMicrons': 1.3776182999409543, 'FractionOfForegroundOccupiedByCells': 0.9139552591250374, '1-FractionOfBackgroundOccupiedByCells': 0.07701391043377503, 'FractionOfCellMaskInForeground': 0.7648416245431748, '1/(ln(StandardDeviationOfCellSize)+1)': 0.16336128718458506, 'FractionOfMatchedCellsAndNuclei': 1.0, '1/(AvgCVForegroundOutsideCells+1)': 0.09343860787391506, 'FractionOfFirstPCForegroundOutsideCells': 0.7157825825930578}, 'Nucleus (including nuclear membrane)': {'1/(AvgOfWeightedAvgCVMeanCellIntensitiesOver1~10NumberOfClusters+1)': 0.2045461213373561, 'AvgOfWeightedAvgFractionOfFirstPCMeanCellIntensitiesOver1~10NumberOfClusters': 0.4881621793693247, 'AvgSilhouetteOver2~10NumberOfClusters': 0.057437420796382}, 'Cell Not Including Nucleus (cell membrane plus cytoplasm)': {'1/(AvgOfWeightedAvgCVMeanCellIntensitiesOver1~10NumberOfClusters+1)': 0.20268239360924561, 'AvgOfWeightedAvgFractionOfFirstPCMeanCellIntensitiesOver1~10NumberOfClusters': 0.5332347317496605, 'AvgSilhouetteOver2~10NumberOfClusters': 0.2254152572310381}, 'QualityScore': 0.13032760796679677}\n"
35+
]
36+
}
37+
],
38+
"source": [
39+
"# illustration for a 2D image \n",
40+
"img_path = Path('example_data/imgs/2D_CODEX.ome.tiff')\n",
41+
"mask_path = Path('example_data/masks/2D_CODEX.ome.tiff')\n",
42+
"output_directory = Path('results')\n",
43+
"\n",
44+
"# two choices: give empty model in which case read_and_eval_seg will use the default 2D model\n",
45+
"PCA_model = []\n",
46+
"# or give a specific model\n",
47+
"#PCA_model = pickle.load( open( \"2D_PCA_model.pickle\", \"rb\" ))\n",
48+
"\n",
49+
"seg_metrics = read_and_eval_seg(img_path, mask_path, PCA_model, output_directory)\n",
50+
"\n",
51+
"print(seg_metrics)"
52+
]
53+
},
54+
{
55+
"cell_type": "code",
56+
"execution_count": 3,
57+
"metadata": {},
58+
"outputs": [
59+
{
60+
"name": "stdout",
61+
"output_type": "stream",
62+
"text": [
63+
"Calculating single-method metrics v1.5 for example_data/imgs\n",
64+
"{'Matched Cell': {'NumberOfCellsPer100SquareMicrons': 1.3776182999409543, 'FractionOfForegroundOccupiedByCells': 0.9139643615647609, '1-FractionOfBackgroundOccupiedByCells': 0.07703737026852353, 'FractionOfCellMaskInForeground': 0.7650158452081655, '1/(ln(StandardDeviationOfCellSize)+1)': 0.16336128718458506, 'FractionOfMatchedCellsAndNuclei': 1.0, '1/(AvgCVForegroundOutsideCells+1)': 0.0934432339297593, 'FractionOfFirstPCForegroundOutsideCells': 0.7163386307662011}, 'Nucleus (including nuclear membrane)': {'1/(AvgOfWeightedAvgCVMeanCellIntensitiesOver1~10NumberOfClusters+1)': 0.20430029228313443, 'AvgOfWeightedAvgFractionOfFirstPCMeanCellIntensitiesOver1~10NumberOfClusters': 0.4886003316890298, 'AvgSilhouetteOver2~10NumberOfClusters': 0.05786529575259167}, 'Cell Not Including Nucleus (cell membrane plus cytoplasm)': {'1/(AvgOfWeightedAvgCVMeanCellIntensitiesOver1~10NumberOfClusters+1)': 0.2028591160797523, 'AvgOfWeightedAvgFractionOfFirstPCMeanCellIntensitiesOver1~10NumberOfClusters': 0.5333378213018224, 'AvgSilhouetteOver2~10NumberOfClusters': 0.2258065871400864}, 'QualityScore': 0.13137588446137866}\n"
65+
]
66+
}
67+
],
68+
"source": [
69+
"img_path = Path('example_data/imgs/2D_CODEX.ome.tiff')\n",
70+
"mask_path = Path('example_data/masks/2D_CODEX.ome.tiff')\n",
71+
"output_directory = Path('results')\n",
72+
"\n",
73+
"# use a specific model (actually the same as the default)\n",
74+
"PCA_model = pickle.load( open( \"2D_PCA_model.pickle\", \"rb\" ))\n",
75+
"\n",
76+
"seg_metrics2 = read_and_eval_seg(img_path, mask_path, PCA_model, output_directory)\n",
77+
"\n",
78+
"print(seg_metrics2)"
79+
]
80+
},
81+
{
82+
"cell_type": "code",
83+
"execution_count": 4,
84+
"metadata": {},
85+
"outputs": [
86+
{
87+
"name": "stdout",
88+
"output_type": "stream",
89+
"text": [
90+
"Calculating single-method metrics v1.5 for example_data/imgs\n",
91+
"{'Matched Cell': {'NumberOfCellsPer100CubicMicrons': 0.7077738508763257, 'FractionOfForegroundOccupiedByCells': 0.14266366802153443, '1-FractionOfBackgroundOccupiedByCells': 1.0025906439112604, 'FractionOfCellMaskInForeground': 0.9822079836750122, '1/(ln(StandardDeviationOfCellSize)+1)': 0.14996549000573892, '1/(AvgCVForegroundOutsideCells+1)': 0.31104260524313904, 'FractionOfFirstPCForegroundOutsideCells': 0.1725867652852338}, 'Nucleus (including nuclear membrane)': {'1/(AvgOfWeightedAvgCVMeanCellIntensitiesOver1~10NumberOfClusters+1)': 0.5107179921178507, 'AvgOfWeightedAvgFractionOfFirstPCMeanCellIntensitiesOver1~10NumberOfClusters': 0.1667737531485708, 'AvgSilhouetteOver2~10NumberOfClusters': 0.07133055907968484}, 'Cell Not Including Nucleus (cell membrane plus cytoplasm)': {'1/(AvgOfWeightedAvgCVMeanCellIntensitiesOver1~10NumberOfClusters+1)': 0.5567450565441361, 'AvgOfWeightedAvgFractionOfFirstPCMeanCellIntensitiesOver1~10NumberOfClusters': 0.18322358897618432, 'AvgSilhouetteOver2~10NumberOfClusters': 0.0876643704226369}, 'QualityScore': 39.85449093420033}\n"
92+
]
93+
}
94+
],
95+
"source": [
96+
"img_path = Path('example_data/imgs/3D_IMC.ome.tiff')\n",
97+
"mask_path = Path('example_data/masks/3D_IMC.ome.tiff')\n",
98+
"output_directory = Path('results')\n",
99+
"\n",
100+
"# two choices: give empty model in which case read_and_eval_seg will use the default 3D model\n",
101+
"PCA_model = []\n",
102+
"# or give a specific model\n",
103+
"#PCA_model = pickle.load( open( \"3D_PCA_model.pickle\", \"rb\" ))\n",
104+
"\n",
105+
"seg_metrics3 = read_and_eval_seg(img_path, mask_path, PCA_model, output_directory)\n",
106+
"\n",
107+
"print(seg_metrics3)"
108+
]
109+
},
110+
{
111+
"cell_type": "code",
112+
"execution_count": null,
113+
"metadata": {},
114+
"outputs": [],
115+
"source": []
116+
}
117+
],
118+
"metadata": {
119+
"kernelspec": {
120+
"display_name": "Python 3 (ipykernel)",
121+
"language": "python",
122+
"name": "python3"
123+
},
124+
"language_info": {
125+
"codemirror_mode": {
126+
"name": "ipython",
127+
"version": 3
128+
},
129+
"file_extension": ".py",
130+
"mimetype": "text/x-python",
131+
"name": "python",
132+
"nbconvert_exporter": "python",
133+
"pygments_lexer": "ipython3",
134+
"version": "3.9.13"
135+
}
136+
},
137+
"nbformat": 4,
138+
"nbformat_minor": 4
139+
}

0 commit comments

Comments
 (0)