Rust-Powered Tokenizer Gigatoken Hits 24.53 GB/s, Outpaces HuggingFace by Over 1,000x
A new Rust-based tokenizer called Gigatoken is shattering speed records, achieving 24.53 GB/s and running over 1,000x faster than HuggingFace's tokenizers by leveraging SIMD instructions and aggressive caching, with the ability to tokenize all of Common Crawl in just 6.5 hours.