Tokenizers Files

What's Changed

• Bump on-headers and compression in /tokenizers/examples/unstable_wasm/www by @dependabot[bot] in https://github.com/huggingface/tokenizers/pull/1827
• Implement from_bytes and read_bytes Methods in WordPiece Tokenizer for WebAssembly Compatibility by @sondalex in https://github.com/huggingface/tokenizers/pull/1758
• fix: use AHashMap to fix compile error by @b00f in https://github.com/huggingface/tokenizers/pull/1840
• New stream by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1856
• [docs] Add more decoders by @pcuenca in https://github.com/huggingface/tokenizers/pull/1849
• Fix missing parenthesis in EncodingVisualizer.calculate_label_colors by @Liam-DeVoe in https://github.com/huggingface/tokenizers/pull/1853
• Update quicktour.mdx re: Issue [#1625] by @WilliamPLaCroix in https://github.com/huggingface/tokenizers/pull/1846
• remove stray comment by @sanderland in https://github.com/huggingface/tokenizers/pull/1831
• Fix typo in README by @aisk in https://github.com/huggingface/tokenizers/pull/1808
• RUSTSEC-2024-0436 - replace paste with pastey by @nystromjd in https://github.com/huggingface/tokenizers/pull/1834
• Tokenizer: Add native async bindings, via py03-async-runtimes. by @michaelfeil in https://github.com/huggingface/tokenizers/pull/1843

New Contributors

• @b00f made their first contribution in https://github.com/huggingface/tokenizers/pull/1840
• @pcuenca made their first contribution in https://github.com/huggingface/tokenizers/pull/1849
• @Liam-DeVoe made their first contribution in https://github.com/huggingface/tokenizers/pull/1853
• @WilliamPLaCroix made their first contribution in https://github.com/huggingface/tokenizers/pull/1846
• @sanderland made their first contribution in https://github.com/huggingface/tokenizers/pull/1831
• @aisk made their first contribution in https://github.com/huggingface/tokenizers/pull/1808
• @nystromjd made their first contribution in https://github.com/huggingface/tokenizers/pull/1834
• @michaelfeil made their first contribution in https://github.com/huggingface/tokenizers/pull/1843

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.21.3...v0.22.0rc0