Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmark

Open original

Researchers created a fully synthetic but realistic electronic health record dataset with 1,268 patients and 5,602 encounters, designed to let AI models be tested on clinical tasks without privacy concerns. Physicians couldn't reliably distinguish it from real charts, but current language models still fall short—the best achieved an F1 score of 0.73 on reconstructing patient problem lists, below the average physician's 0.89.

  • The dataset solves two major barriers: it's openly shareable (no protected health information) and has verifiable ground truth (every diagnosis and finding is grounded in standard medical ontologies and traced back to source materials)
  • The benchmark is genuinely difficult—top models miss roughly half of clinically relevant findings when summarizing charts and don't approach human physician performance
  • Physicians in blinded review distinguished synthetic records from real ones at near-chance rates (53%), validating the realism of the dataset

The researchers built Synthetic Hospital to solve a real problem in clinical AI: you can't openly share real patient data for training and testing AI models because of privacy laws and ethics rules, but synthetic data usually lacks the grounding in actual medical facts that makes benchmarks useful. Their solution was to construct 1,268 longitudinal patient records with 5,602 encounters entirely from public medical education materials, with zero protected health information. Every diagnosis, finding, and temporal relationship gets tied back to standard medical coding systems (ICD-10-CM, SNOMED CT, LOINC) and traces back to its original source material. The dataset lives in a simulated EHR system that mimics real hospital infrastructure—complete with standard APIs, role-based access controls, and function-calling interfaces that models can actually interact with. To validate that the synthetic records feel realistic, they had physicians review them blind against real patient charts. The doctors couldn't reliably tell them apart, hitting only 53% accuracy when trying to distinguish synthetic from real (basically chance). That matters because it means the benchmark isn't some sterile, obviously-artificial dataset. When they tested ten different large language models on reconstructing a patient's complete problem list with severity weighting, the best one achieved an F1 score of 0.73—matching the average performance of seven physicians on the same subset but trailing the best physician at 0.89. Most models also missed roughly half of clinically relevant findings when asked to summarize a chart. The results show the benchmark actually has teeth. No model is close to ceiling performance, and there's still a visible gap between the best AI and the best human doctors. That gap matters for clinical deployment—it's the difference between a tool that catches most things and one that might miss dangerous stuff. The researchers essentially created the first openly shareable, verifiable longitudinal EHR benchmark that doesn't require you to break privacy law or sacrifice ground truth.

Ask about this article