Single-cell perturbation benchmark (scPerturBench)

Introduction

Various computational methods have been developed to predict perturbation effects, but despite claims of promising performance, concerns about the true efficacy of these models continue to be raised, particularly when the models are evaluated across diverse unseen cellular contexts and unseen perturbations. To address this, a comprehensive benchmark was conducted for 27 single-cell perturbation response prediction methods, including methods concerning genetic and chemical perturbations; 29 datasets were used, and various evaluation metrics were applied to assess the generalizability of the methods to unseen cellular contexts and perturbations. Recommendations regarding the method limitations, method generalization and method selection were obtained. Finally, an solution that leverages prior knowledge through cellular context embedding to improve the generalizability of models to new cellular contexts is presented.

Workflow

Cellular context generalization scenario

In the cellular context generalization scenario, we evaluate the prediction of known perturbations in previously unobserved cellular contexts. Specifically, we assessed the accuracy of 10 published methods and the four baseline model across 12 datasets using six evaluation metrics including MSE, PCC-delta, E-distance, Wasserstein distance, KL-divergence, and Common-DEGs. The cellular context generalization scenario can be further divided into two distinct test settings based on the partitioning of the training and test datasets: i.i.d (independent and identically distributed or in-distribution) and o.o.d (out-of-distribution) setting. i.i.d contained the script used in the i.i.d setting. o.o.d contained the script used in the o.o.d. setting. calPerformance is the script for performance calculation and generic function。

Perturbation generalization scenario

In the perturbation generalization scenario, we assess the ability of models to predict the effects of previously unobserved perturbations within a specific cellular context. Depending on the type of perturbation, this scenario can be further divided into two categories: genetic perturbation effects prediction and chemical perturbation effects prediction. (1) Genetic perturbation effect prediction. (2) Chemical perturbation effect prediction. Genetic contained the script used in the genetic setting. Chemical contained the script used in the chemical setting. calPerformance_genetic is the script for genetic performance calculation and calPerformance_chemical is the script for chemical performance calculation and

bioLord-emCell

We posit that improving generalization in the cellular context generalization scenario requires models to effectively capture the inter-heterogeneity in perturbation responses across cellular contexts. This can be achieved in one of two main ways: (1) training on large-scale, diverse datasets to directly learn inter-heterogeneity or (2) leveraging existing prior knowledge. Given the scarcity of large-scale cellular perturbation datasets, the second approach is more feasible.Therefore, we propose a generalizable framework to improve model generalizability across different cellular contexts via cell line embedding and disentanglement representation. bioLord-emCell contained the script we used to implement our framework.

We use sciplex3 dataset as a demo case to run biolord-emCell. We recommend using Anaconda / Miniconda to create a conda environment for using biolord-emCell. You can create a python environment using the following command:

conda env create -f environment.yml

python  biolord-emCell.py

sciplex3_cell_embs.pkl was obtained by Get_embedding.py
For more details, please refer to our manuscript and scGPT tutorial.

scPerturBench reproducibility

To facilitate the community in three key ways—(1) reproducing our work more easily, (2) visualizing the benchmark results more intuitively, and (3) more importantly, comparing the performance of newly developed tools with those included in our benchmark—we have taken two major steps:

• Podman image for full reproducibility: We have packaged all the major benchmark scripts, conda environments, and dependencies used in our study into a Podman image, which is a modern and rootless alternative to Docker. This image allows users to effortlessly reproduce the results presented in our study without manually installing any dependencies. It also comes pre-installed with most of the commonly used tools in the single-cell perturbation analysis field, including pertpy, scanpy, and torch, making it a convenient platform for both end users and tool developers.

• Website for result visualization: We have developed a web interface that visualizes the key results of all tools included in our benchmark. The website allows users to explore and download the raw outputs of each method. Furthermore, if a developer evaluates their own tool using the same evaluation metrics defined in our paper, they can easily compare its performance with existing methods. The website is available at: [XXX].

Below, we provide detailed instructions on how to use the Podman image to reproduce our benchmark results.

• Download Podman image from Onedrive, load and run the image with the following commands,

gunzip scperturbench_v1.tar.gz
podman load -i  scperturbench_v1.tar
podman  run -it --device nvidia.com/gpu=all --security-opt=label=disable scperturbench:latest

• Due to the large size of the above Podman image (40GB), we have created a separate image containing only the cpa environment to facilitate testing (scperturbench_cpa.tar.gz, 12GB). You can download it from Zenedo or Figshare for testing first,

gunzip scperturbench_cpa.tar.gz
podman load -i  scperturbench_cpa.tar
podman  run -it --device nvidia.com/gpu=all --security-opt=label=disable scperturbench_cpa:latest

• We created nine separate conda environments to accommodate the different dependency requirements of various tools. In the container instantiated from this image, you can list the environments with the following command

conda env list

• In our image, we have preloaded three example datasets: KangCrossCell, Papalexi, and Sciplex3-A549. /home/project/Pertb_benchmark/manuscript1 directory contains the scripts related to cellular context generalization scenario, /home/project/Pertb_benchmark/manuscript2 contains the scripts related to perturbation generalization scenario, and /home/project/Pertb_benchmark/manuscript3 contains the scripts related to bioLord-emCell. For example, you can easily train the trVAE model on the KangCrossCell dataset with minimal setup,

conda activate cpa
cd /home/project/Pertb_benchmark/manuscript1/ood
python mytrVAE.py

• Then, you can calculate the performance with

conda activate pertpyV7
export OPENBLAS_NUM_THREADS=20
export JAX_PLATFORMS=cpu
python calPerformance.py

• If you want to run other models on other datasets, just download the datasets from our Figshare and Zenedo, place it in the same folder of KangCrossCell and just change the DataSet paremeter in the corresponding scripts. Below, we list the environment needed by the models.

Environment	Software
cpa	biolord, inVAE, scDisInFact, scPRAM, scPreGAN, SCREEN, trVAE, CPA, scouter, cycleCDR, PRnet
cellot	CellOT
scarches	scGen
scVIDR	scVIDR
gears	AttentionPert, GEARS, GenePert, scFoundation, scELMo, GeneCompass,
chemCPA	chemCPA
scGPT	scGPT
linearModel	linearModel
pertpyV7	calculate the performance

Benchmark datasets summary

All datasets analyzed in our study are listed in the Workflow. We have uploaded all benchmark datasets to Figshare and Zenodo, which can be obtained from Figshare-Cellular, Figshare-Perturbation, Zenodo-Cellular and Zenodo-perturbation.

Benchmark methods

All benchmark methods analyzed in our study are listed below. Details of the setting were available in our manuscript.

Method	Article	Time	Title	Version
biolord	Nature Biotechnology	2024	Disentanglement of single-cell data with biolord	0.0.3
CellOT	Nature Methods	2023	Learning single-cell perturbation responses using neural optimal transport	0.0.1
inVAE	Bioengineering	2023	Homogeneous Space Construction and Projection for Single-Cell Expression Prediction Based on Deep Learning	0.0.1
scDisInFact	Nature Communications	2024	scDisInFact: disentangled learning for integration and prediction of multi-batch multi-condition single-cell RNA-sequencing data	0.1.0
scGen	Nature Methods	2019	scGen predicts single-cell perturbation responses	2.1.0
scPRAM	Bioinformatics	2024	scPRAM accurately predicts single-cell gene expression perturbation response based on attention mechanism	0.0.1
scPreGAN	Bioinformatics	2022	scPreGAN, a deep generative model for predicting the response of single-cell expression to perturbation	0.0.1
SCREEN	Frontiers of Computer Science	2024	SCREEN: predicting single-cell gene expression perturbation responses via optimal transport	0.0.1
scVIDR	Patterns	2023	Generative modeling of single-cell gene expression for dose-dependent chemical perturbations	0.0.3
trVAE	Bioinformatics	2020	Conditional out-of-distribution generation for unpaired data using transfer VAE	1.1.2
AttentionPert	Bioinformatics	2021	AttentionPert: Accurately Modeling Multiplexed Genetic Perturbations with Multi-scale Effects	0.0.1
CPA	Molecular Systems Biology	2023	Predicting cellular responses to complex perturbations in high-throughput screens	0.8.5
GEARS	Nature Biotechnology	2022	Predicting transcriptional outcomes of novel multigene perturbations with GEARS	0.1.0
GenePert	bioRxiv	2024	GenePert: Leveraging GenePT Embeddings for Gene Perturbation Prediction	0.0.1
linearModel	bioRxiv	2024	Deep learning-based predictions of gene perturbation effects do not yet outperform simple linear methods	0.0.1
scGPT	Nature Methods	2024	scGPT: toward building a foundation model for single-cell multi-omics using generative AI	0.2.1
scFoundation	Nature Methods	2024	Large-scale foundation model on single-cell transcriptomics	0.0.1
chemCPA	arXiv	2022	Predicting Cellular Responses to Novel Drug Perturbations at a Single-Cell Resolution	2.0.0
scouter	bioRxiv	2024	Scouter: Predicting Transcriptional Responses to Genetic Perturbations with LLM embeddings	0.0.1
scELMo	bioRxiv	2024	scELMo: Embeddings from Language Models are Good Learners for Single-cell Data Analysis	0.0.1
GeneCompass	Cell Research	2024	GeneCompass: deciphering universal gene regulatory mechanisms with a knowledge-informed cross-species foundation model	0.0.1
cycleCDR	bioRxiv	2024	Predicting single-cell cellular responses to perturbations using cycle consistency learning	0.0.1
PRnet	Nature communication	2024	Predicting transcriptional responses to novel chemical perturbations using deep generative model for drug discovery	0.0.1

Citation

Zhiting Wei, Yiheng Wang, Yicheng Gao, Qi Liu et al. Benchmarking algorithms for generalizable single-cell perturbation response prediction, submitted, 2025.

Contacts

[email protected], [email protected]