Perplexity's New 'Lily' Engine Outperforms MLX-LM on Apple Silicon with 35% Faster Decode Speeds
Summary
Perplexity Engineering unveils Lily, a custom inference engine built for Apple Silicon that outperforms MLX-LM by 23% on prefill and 35% on decode speeds, using fused kernels, GPU-resident routing, and smart memory optimizations on the Qwen3-35B model.
Key Points
- Perplexity Engineering introduces Lily, a custom local inference engine built specifically for Apple Silicon and the Qwen3.6-35B-A3B model, achieving 1.23x prefill and 1.35x decode throughput compared to MLX-LM on an M5 Max MacBook Pro.
- Lily improves prefill performance by keeping sparse expert routing and recurrent state on the GPU, fusing dequantization into matrix multiplication, and using bounded prompt chunking to cap peak memory usage.
- Decode performance is optimized by minimizing data movement per token through fused kernel chains, GPU-resident token handoffs, GQA packing to reuse cached KV rows, and a context-length-aware attention layout switch that delivers up to 40% improvement at 128K tokens.