OpenAI Retracts Support for SWE-Bench Pro After Audit Finds 30% of Tasks Broken
Summary
OpenAI retracts its support for SWE-Bench Pro after an internal audit uncovers that nearly 30% of the coding benchmark's 731 tasks are broken due to flawed tests and misleading prompts, prompting a call for new, developer-built benchmarks.
Key Points
- A detailed audit of SWE-Bench Pro, a widely used coding benchmark, reveals that approximately 30% of its 731 tasks are broken, with issues including overly strict tests, underspecified prompts, low-coverage tests, and misleading prompts.
- OpenAI conducts the audit using a two-pronged approach combining an automated datapoint analysis pipeline and a human annotation campaign with experienced software engineers, flagging 200 to 249 broken tasks depending on the review method.
- In light of these findings, OpenAI retracts its earlier recommendation to adopt SWE-Bench Pro and calls on the evaluation community to develop new benchmarks built specifically by experienced software developers to ensure valid, trustworthy, and capability-reflective assessments.