Firehose

The Hugging Face team has released a new version of the tokenizers library (v1) with improved performance, including a hand-written splitter instead of a regex engine, a cache that answers repeated words without merging, and a merge loop that never touches the allocator. The library now scales across cores and processes multiple pre-token spans in one call. The new version is faster than the previous version (v0.23) by 3-30 times, depending on the model and input size. AI summary