Study Finds JSON Mode Costs Up to 11x More Than CLI Args and Causes AI Agents to Fail 60% of the Time
Summary
A new study reveals that using JSON mode for AI agents causes smaller models to fail up to 60% of the time and costs up to 11x more than traditional CLI arguments due to shell escaping failures, with developers urged to avoid rewriting CLIs for agents until running measurable experiments.
Key Points
- A new study tests whether replacing CLI arguments with a single --json payload improves AI agent performance, finding that traditional args achieve perfect correctness across all five models tested, while JSON causes smaller models like Claude Haiku 4.5 and MAI-Code-1-Flash to fail up to 60% of the time.
- JSON mode costs 4x to 11x more per task due to excessive retries caused by shell escaping failures, with output tokens running 7x to 14x higher than the args path, and the problem is significantly worse on PowerShell than Bash.
- Developers are urged not to rewrite CLIs for agents, as constrained argument-based inputs outperform JSON across correctness, cost, and cross-environment reliability, with the recommendation to run measurable experiments before making any structural changes to a CLI's input model.