Genetics Chemistry
Methods that report strong benchmark numbers sometimes fail outside the benchmark. These papers test specific claims at the claim level — identifying which survive external scrutiny, which are benchmark-bound, and where the validity boundary lies.
Mechanism-Dependent Validity of Cis-pQTL Mendelian Randomization
Abundance or Activity? Mechanism-Dependent Validity of Cis-pQTL Mendelian Randomization
Zenodo 10.5281/zenodo.21207253
A cis-pQTL instrument measures lifelong variation in circulating protein abundance, yet evaluations of pQTL-based Mendelian randomization for drug-target validation report composite accuracy across all drug mechanisms. This conflation predicts a mechanism-dependent domain of validity: MR should be informative when drug and instrument act on the same causal axis (protein level) and uninformative when they do not (protein function). Pre-registered, outcome-blind evaluation of a zero-parameter classifier against 195 Phase III drug-target pairs across 24 diseases confirms the prediction. For activity-blocking targets (n = 82), MR is uninformative (balanced accuracy = 0.515, 90% CI [0.460, 0.579]). For abundance-modulating targets (n = 72), MR is informative (BA = 0.610, [0.530, 0.694]; AUC = 0.599, permutation p = 0.024). The pooled BA of 0.578 is a mixture of these strata, consistent with benchmarks reporting lower pQTL enrichment than GWAS-only approaches. The instrument class has a definable validity boundary set by drug mechanism.
Heterogeneous Validity in ML for Chemistry
Heterogeneous Validity in Machine Learning for Chemistry: A Claim-Level Audit of 52 Published Methods
119 ML-for-chemistry claims across 12 families — spectral foundation models, metabolomics biomarkers, molecular property prediction, reaction prediction, materials discovery, spectral matching, multi-omic integration, chemistry LLMs, docking/AlphaFold, protein design — scored against 16 typed failure modes organized into four categories: data integrity, generalization, interpretation, and reproducibility. Of the scored claims, 68% are Benchmark-Bound: real on the reported benchmark but with transportability unresolved. Three entries earn Validated — AlphaFold2 for single-chain static structure, ProteinMPNN for inverse folding, and RFdiffusion for backbone generation — all validated on real experimental endpoints (CASP, X-ray, cryo-EM). 17 are Disconfirmed. The dominant unmeasured failure is external-validation collapse driven by batch/instrument confound and split-design leakage.