Tokenizers Files

Fast State-of-the-Art Tokenizers optimized for Research and Production

This is an exact mirror of the Tokenizers project, hosted at https://github.com/huggingface/tokenizers. SourceForge is not affiliated with Tokenizers.

The interactive file manager requires Javascript. Please enable it or use sftp or scp.
You may still browse the files here.

Name	Modified	InfoDownloads / Week
v0.23.3	2026-10-09	2
v0.23.2	2026-09-03	0
v0.23.1	2026-04-27	0
v0.22.2	2025-12-02	2
v0.22.1	2025-09-19	0
Totals: 5 Items		4

Other Useful Business Software

Build Agents and Models on One Platform

Everything you need to build production-ready agents and models. Access 200+ Google and third-party AI models and tools.

Gemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.

Start Free

$300 Free Credits to Build on Google Cloud

New customers can spin up VMs, build with AI, and query data at no cost.

Put your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.

Start Free

Build Agents and Models on One Platform

Everything you need to build production-ready agents and models. Access 200+ Google and third-party AI models and tools.

Start Free

Recommended Projects

VibeVoice
Open-source multi-speaker long-form text-to-speech model
AI Engineering from Scratch
Learn it. Build it. Ship it for others
Fast Python Xmlrpc Library
An extremely fast implementation of the xmlrpc spec for Python (written in C). It supports both blocking and non-blocking clients and servers on Windows and POSIX platforms. Version 0.8.1 is 100% compliant with the www.xmlrpc.com validator.
PDF split by bookmarks
SplitPDF -SplitPDF.jar- is a ‘command-line driven’ Java-program, it splits a PDF-file by bookmarks into separated PDF’s. The bookmark is used as title for the newly created PDF. Extremely usefull and fast in a batch processing environment.
textacy
NLP, before and after spaCy