New Open-Source Framework Evaluates 500+ Robot AI Models Across 17 Benchmarks at 47x Faster Speed

A powerful new open-source framework called vla-evaluation-harness launches with the ability to evaluate 500+ robot AI models across 17 benchmarks at up to 47x faster speeds, completing 2,000 simulation episodes in just 18 minutes on a single GPU.

• A new open-source framework called vla-evaluation-harness is now available, offering a unified system to evaluate any Vision-Language-Action (VLA) model across 17 robot simulation benchmarks, with a leaderboard tracking 500+ models aggregated from over 1,700 papers.
• The framework delivers up to 47x faster evaluation throughput through batch parallel processing, completing 2,000 LIBERO episodes in just 18 minutes on a single H100 GPU by combining episode sharding with batched GPU inference.
• Benchmarks run inside isolated Docker containers to eliminate dependency conflicts, while model servers are set up as single-file scripts requiring zero manual configuration, making cross-benchmark evaluation fully reproducible and accessible.