Tokenizers Files

This is the last v0 release, we are moving to v1!!

More details coming soon 👀

What's Changed

• Update daachorse from 1.0.1 to 3.0.0 by @musicinmybrain in https://github.com/huggingface/tokenizers/pull/2050
• Ci test fixes by @sebpop in https://github.com/huggingface/tokenizers/pull/2079
• Avoid full vocab clone in get_vocab_size() by @eunseo9311 in https://github.com/huggingface/tokenizers/pull/2074
• ci: pin benchmark jobs to a larger dedicated runner by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/2084
• ci: target benchmark runners by group, not nonexistent label by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/2090
• Bindings: acquire model read-lock once per call instead of per pre-token by @sebpop in https://github.com/huggingface/tokenizers/pull/2072
• ci: split audit in a separate job by @SBrandeis in https://github.com/huggingface/tokenizers/pull/2103
• ci: install libcairo on benchmark runners by @SBrandeis in https://github.com/huggingface/tokenizers/pull/2100
• feat(ci): enable npm's trusted publisher for releases by @McPatate in https://github.com/huggingface/tokenizers/pull/2111
• chore: bump pyo3 to version 0.29 by @SBrandeis in https://github.com/huggingface/tokenizers/pull/2115
• perf: add benchmarks for the decode API by @SBrandeis in https://github.com/huggingface/tokenizers/pull/2099
• ci: cache HF test data and authenticate Hub downloads to avoid rate limits by @SBrandeis in https://github.com/huggingface/tokenizers/pull/2102
• Implementing Parity-aware BPE by @cimeister in https://github.com/huggingface/tokenizers/pull/1974
• Make doc builds faster by @mishig25 in https://github.com/huggingface/tokenizers/pull/2158
• chore(deps-dev): bump websocket-driver from 0.7.4 to 0.7.5 in /tokenizers/examples/unstable_wasm/www by @dependabot[bot] in https://github.com/huggingface/tokenizers/pull/2208
• chore(deps-dev): bump shell-quote from 1.8.3 to 1.10.0 in /tokenizers/examples/unstable_wasm/www by @dependabot[bot] in https://github.com/huggingface/tokenizers/pull/2244
• chore: remove stale examples (kills 50% of dependabot traffic) by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/2285
• chore(node): drop 8 unused devDependencies by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/2286
• ci: pin every action to one SHA, drop stale audit suppressions by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/2291
• chore(node): take node security alerts from 33 to 0 by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/2287
• fix(python): pin the ruff rule set so a ruff release can't redden main by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/2292
• Add a ParityBpeTrainer example and list it in the trainers API docs by @cimeister in https://github.com/huggingface/tokenizers/pull/2217

New Contributors

• @eunseo9311 made their first contribution in https://github.com/huggingface/tokenizers/pull/2074
• @cimeister made their first contribution in https://github.com/huggingface/tokenizers/pull/1974

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.23.1...v0.23.2