Moondream's Photon Engine Cuts GPU Idle Time, Boosts VLM Decode Throughput by Up to 35%

Moondream's new Photon inference engine eliminates GPU idle time through pipelined decoding, boosting vision-language model throughput by up to 35% on NVIDIA B200 hardware and achieving near-realtime inference at approximately 33ms.

• Moondream's inference engine, Photon, eliminates GPU idle time — known as GPU bubbles — by using pipelined decoding, overlapping CPU bookkeeping with GPU computation to achieve up to 35% higher decode throughput and near-realtime VLM inference at approximately 33ms on NVIDIA B200 hardware.
• Pipelined decoding relies on three core mechanisms: ping-pong slots that prevent buffer collisions between overlapping steps, a forward-now-sample-later approach that allows constrained decoding to run ahead without blocking the next GPU forward pass, and a zombie refcounting system that safely handles sequences that finish mid-pipeline without requiring complex cancellation logic.
• Performance gains scale directly with GPU speed — a 3090 sees roughly 6-12% improvement while a B200 sees up to 35% — because faster hardware shrinks the forward pass, making the fixed CPU bookkeeping cost a larger share of each step, and pipelining is what hides that growing overhead.