Tokenizers Files

Fast State-of-the-Art Tokenizers optimized for Research and Production

This is an exact mirror of the Tokenizers project, hosted at https://github.com/huggingface/tokenizers. SourceForge is not affiliated with Tokenizers.

The interactive file manager requires Javascript. Please enable it or use sftp or scp.
You may still browse the files here.

Name	Modified	Size	InfoDownloads / Week
Parent folder
README.md	2026-10-09	716 Bytes	0
Release v0.23.3 (python only) source code.tar.gz	2026-10-09	1.5 MB	0
Release v0.23.3 (python only) source code.zip	2026-10-09	1.6 MB	2
Totals: 3 Items		3.1 MB	2

Release v0.23.3

Python package release, other packages are unchanged

:rotating_light: Fixs a breaking change introduced in v0.23.1

What's changed

Patch for the python package with three fixes:

Loosen the huggingface_hub requirements to allow huggingface_hub v2 https://github.com/huggingface/tokenizers/pull/2486#issuecomment-6062880361 - Unblocks transformers installs
Fix type hints not resolving for pyright: https://github.com/huggingface/tokenizers/issues/2456
Fix a breaking change from 0.23.1 regarding truncation and overflowing: https://github.com/huggingface/tokenizers/issues/2091

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.23.2...v0.23.3

Source: README.md, updated 2026-10-09

Other Useful Business Software

$300 Free Credits to Build on Google Cloud Icon

$300 Free Credits to Build on Google Cloud

New customers can spin up VMs, build with AI, and query data at no cost.

Put your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.

Start Free

Veeam Data Platform v13.1 Icon

Veeam Data Platform v13.1

Move workloads across hypervisors and clouds with no vendor lock-in. Try VDP free today.

Try Veeam Data Platform today. Experience the unified platform that's secure by design, portable by default, and proven to recover clean, fast, and anywhere.

Try Now

$300 Free Credits to Build on Google Cloud

New customers can spin up VMs, build with AI, and query data at no cost.

Put your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.

Start Free

Recommended Projects

VibeVoice
Open-source multi-speaker long-form text-to-speech model
AI Engineering from Scratch
Learn it. Build it. Ship it for others
Fast Python Xmlrpc Library
An extremely fast implementation of the xmlrpc spec for Python (written in C). It supports both blocking and non-blocking clients and servers on Windows and POSIX platforms. Version 0.8.1 is 100% compliant with the www.xmlrpc.com validator.
PDF split by bookmarks
SplitPDF -SplitPDF.jar- is a ‘command-line driven’ Java-program, it splits a PDF-file by bookmarks into separated PDF’s. The bookmark is used as title for the newly created PDF. Extremely usefull and fast in a batch processing environment.
textacy
NLP, before and after spaCy