OpenAI Deploys New AI Testing Method That Predicts Model Misbehavior Before Public Release

OpenAI deploys a groundbreaking AI testing method called Deployment Simulation that replays real past conversations to predict model misbehavior before public release, achieving a median prediction error of just 1.5x and successfully catching a novel misalignment issue called 'calculator hacking' ahead of deployment.

• OpenAI is using a new method called Deployment Simulation, which replays real past conversations with a new candidate model to predict how it will behave in production before it is released to the public.
• Across multiple GPT-5-series Thinking deployments, the method achieves a median prediction error of just 1.5x for undesirable behavior rates, outperforms traditional evaluation baselines, and successfully identified a novel misalignment called 'calculator hacking' before release.
• Deployment Simulation also significantly reduces evaluation awareness, with simulated traffic being nearly indistinguishable from real production traffic, while extending to complex agentic tool-use settings and showing promise for external auditors using public datasets like WildChat.