Tiktoken Files

tiktoken is a fast BPE tokeniser for use with OpenAI's models

This is an exact mirror of the Tiktoken project, hosted at https://github.com/openai/tiktoken. SourceForge is not affiliated with Tiktoken.

The interactive file manager requires Javascript. Please enable it or use sftp or scp.
You may still browse the files here.

Name	Modified	Size	InfoDownloads / Week
0.13.0	2026-05-15		0
0.12.0	2025-10-06		0
0.11.0	2025-08-08		0
0.9.0	2025-02-14		0
0.8.0	2024-10-03		0
Totals: 5 Items			0

Other Useful Business Software

MongoDB Atlas runs apps anywhere

Deploy in 115+ regions with the modern database for every enterprise.

MongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.

Start Free

Ship Agents Faster

Transform your applications and workflows into powerful agentic systems at global scale.

Gemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.

Get Started Free

MongoDB Atlas runs apps anywhere

Deploy in 115+ regions with the modern database for every enterprise.

Start Free

Recommended Projects

SentencePiece
Unsupervised text tokenizer for Neural Network-based text generation
MiMo Audio
Audio Language Models are Few-Shot Learners
Codepage Converter
Codepage Converter - Convert HTML/Text files to different encoding formats e.g. ANSI to UTF-8 or Unicode. Convert multiple files with 1 click. Works with all encodings. * The issue with UTF-8 has now been fixed.
Libb64
libb64 is a library of ANSI C routines for fast encoding/decoding data into and from a base64-encoded format. C++ wrappers are included, as well as the source code for standalone encoding and decoding executables.
GifComponents
A library written in C# for encoding and decoding animated Graphics Interchange Format (GIF) images. Based on NGif http://www.codeproject.com/KB/GDI-plus/NGif.aspx. Includes sample apps for encoding and decoding GIF files.

Tiktoken Files

tiktoken is a fast BPE tokeniser for use with OpenAI's models

Get an email when there's a new version of Tiktoken