Tokenizers Files

Fast State-of-the-Art Tokenizers optimized for Research and Production

This is an exact mirror of the Tokenizers project, hosted at https://github.com/huggingface/tokenizers. SourceForge is not affiliated with Tokenizers.

The interactive file manager requires Javascript. Please enable it or use sftp or scp.
You may still browse the files here.

Name	Modified	InfoDownloads / Week
v0.23.2	2026-09-03	3
v0.23.1	2026-04-27	0
v0.22.2	2025-12-02	0
v0.22.1	2025-09-19	2
v0.22.0	2025-08-29	2
Totals: 5 Items		7

Other Useful Business Software

$300 Free Credits to Build on Google Cloud

New customers can spin up VMs, build with AI, and query data at no cost.

Put your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.

Start Free

Build Agents and Models on One Platform

Everything you need to build production-ready agents and models. Access 200+ Google and third-party AI models and tools.

Gemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.

Start Free

$300 Free Credits to Build on Google Cloud

New customers can spin up VMs, build with AI, and query data at no cost.

Put your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.

Start Free

Recommended Projects

BioNLP UIMA Component Repository
The BioNLP UIMA Component Repository provides UIMA wrappers for novel and well-known 3rd-party NLP tools used in biomedical text prosessing, such as tokenizers, parsers, named entity taggers, and tools for evaluation.
Tokenize.jl
Tokenization for Julia source code
twitter-text
This code is used at Twitter to tokenize and parse text
VibeVoice
Open-source multi-speaker long-form text-to-speech model
Automa.jl
A julia code generator for regular expressions

Tokenizers Files

Fast State-of-the-Art Tokenizers optimized for Research and Production

Get an email when there's a new version of Tokenizers