Construct Validity for Single-Cell Integration Metrics
Construct Validity for Single-Cell Integration Benchmarks: Five Metrics at Chance, an Analytic Failure Bound, and a Validated Replacement
Five widely used single-cell integration metrics — silhouette, isolated label silhouette, cell-type ASW, graph connectivity, and kBET — rank foundation model embeddings at chance (mean inversion rate = 0.50 across 60 model pairs, 4 tissues). An analytic failure bound explains why: embedding dimension exceeds the number of batches, and metric discriminability saturates. This condition holds for every foundation model tested, including Geneformer (V1, V2-104M, V2-316M), scGPT, UCE, and scPRINT. The failure is not model-specific but structural — any embedding whose dimension exceeds the batch count will hit the same ceiling.
We develop a replacement metric: the Spearman correlation coefficient between pairwise batch distances and downstream task performance. The replacement achieves ρ = 0.67 across 25 tissues, transfers to held-out tasks, and discriminates between models that the standard metrics rank identically. Open-source library.