Tokenizers Files

Fast State-of-the-Art Tokenizers optimized for Research and Production

This is an exact mirror of the Tokenizers project, hosted at https://github.com/huggingface/tokenizers. SourceForge is not affiliated with Tokenizers. For more information, see the SourceForge Open Source Mirror Directory.

The interactive file manager requires Javascript. Please enable it or use sftp or scp.
You may still browse the files here.

Name	Modified	Size	InfoDownloads / Week
Parent folder
README.md	2025-07-04	259 Bytes	0
v0.21.3 source code.tar.gz	2025-07-04	1.5 MB	1
v0.21.3 source code.zip	2025-07-04	1.7 MB	4
Totals: 3 Items		3.2 MB	5

What's Changed

Clippy fixes. by @Narsil in https://github.com/huggingface/tokenizers/pull/1818
Fixed an introduced backward breaking change in our Rust APIs.

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.21.2...v0.21.3

Source: README.md, updated 2025-07-04

Other Useful Business Software

Our Free Plans just got better! | Auth0

With up to 25k MAUs and unlimited Okta connections, our Free Plan lets you focus on what you do best—building great apps.

You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.

Try free now

MongoDB Atlas runs apps anywhere

Deploy in 115+ regions with the modern database for every enterprise.

MongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.

Start Free

Recommended Projects

AdaNet
Fast and flexible AutoML with learning guarantees
DeSmuME: Nintendo DS emulator
DeSmuME is a Nintendo DS emulator
Apache OpenOffice
The free and Open Source productivity suite
7-Zip
A free file archiver for extremely high compression
KeePass
A lightweight and easy-to-use password manager

Tokenizers Files

Fast State-of-the-Art Tokenizers optimized for Research and Production

Get an email when there's a new version of Tokenizers

What's Changed