OpenAI's o1 Model Outperforms Physicians in ER Diagnosis Study, But Experts Warn AI Not Ready for Real-World Use

OpenAI's o1 AI model outperforms physicians in emergency room diagnoses with a 67% accuracy rate versus 55% and 50% for human doctors, according to a Harvard Medical School study, though experts warn the AI remains unready for real-world use and critics question the fairness of comparing it to internal medicine doctors rather than actual ER specialists.

• A new Harvard Medical School and Beth Israel Deaconess Medical Center study finds OpenAI's o1 model outperforms two internal medicine physicians in emergency room diagnoses, correctly identifying exact or near-exact diagnoses in 67% of triage cases compared to 55% and 50% for the human doctors.
• Researchers stress that AI is not yet ready for real-world life-or-death medical decisions, calling for formal prospective trials and noting there is currently no accountability framework for AI-generated diagnoses.
• Critics push back on the findings, arguing the comparison is flawed since AI was tested against internal medicine physicians rather than actual ER specialists, and that an ER doctor's primary goal is identifying life-threatening conditions, not guessing a final diagnosis.