YouTokenToMe

YouTokenToMe is a fast and efficient unsupervised text tokenization library designed for training subword embeddings, particularly useful for NLP models.

Features

Implements Byte Pair Encoding (BPE) and Unigram language models
Optimized for processing large text corpora
Provides a lightweight and fast tokenization pipeline
Supports vocabulary pruning and model compression
Works with Unicode and multilingual text inputs

Project Samples

Project Activity

See All Activity >

License

MIT License

Follow YouTokenToMe

YouTokenToMe Web Site

Other Useful Business Software

Earn up to 16% annual interest with Nexo.

Let your crypto work for you

Put idle assets to work with competitive interest rates, borrow without selling, and trade with precision. All in one platform. Geographic restrictions, eligibility, and terms apply.

Get started with Nexo.

Rate This Project

User Reviews

Be the first to post a review of YouTokenToMe!

Additional Project Details

Operating Systems

Linux, Mac, Windows

Programming Language

C++

Related Categories

C++ Natural Language Processing (NLP) Tool

Registered

2025-01-24

Similar Business Software

LM-Kit.NET

LM-Kit.NET is a cutting-edge, high-level inference SDK designed specifically to bring the advanced capabilities of Large Language Models (LLM) into the C# ecosystem. Tailored for developers working within .NET, LM-Kit.NET provides a comprehensive suite of powerful Generative AI tools, making...

See Software
Google AI Studio

Google AI Studio is a unified development platform that helps teams explore, build, and deploy applications using Google’s most advanced AI models, including Gemini 3. It brings text, image, audio, and video models together in one interactive playground. With vibe coding, developers can use...

See Software
kama.ai

A Responsible AI Agent platform providing accurate, accountable, and safe AI for your organization. As a Composite (hybrid) platform, it combines Knowledge Graph AI, governed Generative AI, and Intelligent Automation technologies. This combination gives you trusted answers that are accurate...

See Software
Enterprise Bot

Enterprise Bot, based in Switzerland, is a pioneer in Conversational AI, Process Automation, and Generative AI. With the trust of esteemed enterprise giants across industries like Generali, SIX, SBB, DHL, and SWICA, Enterprise Bot is revolutionizing both customer and employee experiences....

See Software
GPT-4

GPT-4 (Generative Pre-trained Transformer 4) is a large-scale unsupervised language model, yet to be released by OpenAI. GPT-4 is the successor to GPT-3 and part of the GPT-n series of natural language processing models, and was trained on a dataset of 45TB of text to produce human-like text...

See Software
Gensim

Gensim is a free, open source Python library designed for unsupervised topic modeling and natural language processing, focusing on large-scale semantic modeling. It enables the training of models like Word2Vec, FastText, Latent Semantic Analysis (LSA), and Latent Dirichlet Allocation (LDA),...

See Software

Report inappropriate content

YouTokenToMe

Unsupervised text tokenizer focused on computational efficiency

Get an email when there's a new version of YouTokenToMe

Features

Project Samples

Project Activity

Categories

License

Follow YouTokenToMe

User Reviews

Additional Project Details

Operating Systems

Programming Language

Related Categories

Registered