Theme — eval

Items in corpus: 45

Papers

• Virtual Cell Challenge — a Turing test for the virtual cell — CZI / Roohani / Hsu et al., Cell, 2025
  What success is supposed to look like, operationalised.
• Earliest 'linear baselines are competitive' warning — Boiarsky et al., NeurIPS workshop, 2023
  First warning shot — read it to see how early the signal was.
• scPerturBench replication — Csendes et al., BM2 Lab preprint, 2024
  Showed the original scGPT split was leaky.
• Deep-learning predictions of gene expression don't generalize — Ahlmann-Eltze & Huber, Nature Methods, 2025
  THE canonical reckoning paper — start here.
• Limits of zero-shot foundation models in single-cell biology — Kedzierska et al., Genome Biology, 2025
  Extends the result to UCE and the zero-shot setting.
• latent-additive is the new baseline floor — Wenkel et al., Nature Methods, 2025
  Proposed the stronger baseline current FMs still don't beat.
• 27 methods × 29 datasets × 6 metrics — Wu et al., Nature Methods, 2026
  First axis-by-axis failure decomposition.
• No single scFM consistently outperforms — Wu et al., Genome Biology, 2025
  6 scFMs, cell-ontology-grounded metrics.
• scEval — challenges the necessity of sc-FMs — Liu et al., Advanced Science, 2026
  Strongest 'is the paradigm worth it' framing.
• Parameter-free baseline beats sc-FMs — bioRxiv, 2026
  Cleanest post-reckoning headline; direct Ahlmann successor.
• PertEval-scFM — standardised evaluation framework — ICML, 2025
  Formal venue stamp on the perturbation critique.
• CellBench-LS — stratified low-supervision benchmark — bioRxiv, 2026
  FMs lead cell-type ID; classical methods win gene-expression.
• Real-world RNA-seq integration — Han et al., bioRxiv, 2026
  Industry-authored — deployment-grade robustness gaps.
• Zero-shot scFMs fail to recover cellular dynamics — bioRxiv, 2026
  Extends the critique to RNA-velocity / dynamics.
• Foundation Models Improve Perturbation Response — bioRxiv, 2026
  Contrarian voice — FMs DO improve with enough data.
• Beyond Alignment — Synergistic Information Score — bioRxiv (Microsoft), 2026
  Microsoft's metric for whether a multimodal FM extracts genuinely cross-modal information.
• Deep deterministic UQ — van Amersfoort et al., 2024
  Uncertainty-quantification method for the clinical-grade-FM track.
• Conformal prediction — Angelopoulos & Bates, 2023
  Distribution-free uncertainty paired with the UQ track.
• Sukumaran et al. — clinical-decision-support evaluation — JCO Precision Oncology, 2025
  Framework for clinical AI evaluation.
• Evaluating single-cell perturbation response models is far from straightforward — Heidari, Karimpour, Srivatsa & Montazeri, bioRxiv, 2026
  Second front in the perturbation-evaluation reckoning — Wasserstein/Energy distances mislead in high-dim gene space.

Tools

• scPRINT-2 — Kalfon, Peyré & Cantini, bioRxiv, 2025
  Next-gen cell FM AND benchmark suite in one preprint.
• MedAgentGym
  Agentic environment / benchmark for training and evaluating medical AI agents.
• Pertpy — end-to-end framework for perturbation analysis — Heumos, Ji, May, Theis et al., Nature Methods, 2025
  scverse-blessed perturbation toolbox — distances, perturbation spaces, harmonisation. The canonical analysis layer.

Benchmarks

• Open Problems v2
  Community single-cell benchmark suite.
• PerturBench — Altos Labs
  Altos Labs perturbation-prediction benchmark.
• scPerturBench — BM2-Lab
  Benchmark with adversarial perturbation/cell-type splits.
• CASP15 / CAPRI
  Community structure- and complex-prediction assessments.
• ProteinGym — Notin et al., NeurIPS, 2023
  Protein variant-effect benchmark.
• Compositional-generalisation benchmark (Hetzel et al.) — NeurIPS LMRL, 2024
  Compositional generalisation benchmark referenced in supplementary §C.3.
• VCBench — arXiv
  Named benchmark for virtual cells — details still being chased down.
• Campanella et al. clinical-grade pathology panel — Campanella et al., Nature Communications, 2025
  Multi-task pathology benchmark for FM evaluation.
• Polaris Hub — drug-discovery ML benchmark community
  Public leaderboard for drug-discovery models.
• Therapeutics Data Commons (TDC-2) — Marinka Zitnik
  Therapeutic ML benchmark suite.
• CZI Virtual Cells Platform Benchmarks
  Official virtual-cell evaluation suite hosted by CZI.
• sc-Arena — single-cell benchmark suite
  Third-party-maintained sc-FM benchmark.

Organisations

• Ahlmann-Eltze / Huber (EMBL Heidelberg)
  Authored the 2025 reckoning paper; Ahlmann-Eltze now at Isomorphic Labs.

People

• Constantin Ahlmann-Eltze — Isomorphic Labs (ex-EMBL)
  Linear-baseline-2025 first author; recently moved to Isomorphic.
• Wolfgang Huber — EMBL
  Bioconductor + genomics-statistics elder; co-author of the reckoning paper.
• Kasia Z. Kedzierska — Oxford / MSR
  Zero-shot-critique first author.
• Qi Liu — Tongji University
  BM2 Lab — CausCell, STAMP, scPerturBench. Methods-critique + tooling powerhouse from China.

Competition

• CZ Biohub Virtual Cell Challenge 2025
  First community virtual-cell benchmark, $50k prize.
• Echoes of Silenced Genes — Myllia Biotechnology cell challenge — Myllia, 2026
  Biotech-run challenge focused on perturbation prediction.
• AI Proteomics Competition (AIPC) — Westlake University
  Westlake-run AI-proteomics challenge — bridges AIVC and mass-spec ML.

Event

• Virtual Cell Challenge 2025 — first round — CZI / Biohub, 2025
  The annual benchmark cycle whose 2025 launch crystallised the field.
• Arc Virtual Cell Challenge 2025
  Community benchmark with $50k prize; first formal VC evaluation.

---

Regenerated 2026-05-15 from a 277-item corpus by resourcelib-views.