Search Results for "token"

LLM-TLDR is a Python-based tool designed to dramatically reduce the amount of code a large language model needs to read by extracting the essential structure and context from a codebase and presenting only the most relevant parts to the model. Traditional approaches often dump entire files into a model’s context, which quickly exceeds token limits; LLM-TLDR instead indexes project structure, traces dependencies, and summarizes code in a way that preserves semantic relevance while shrinking input size by up to 95 %. This makes queries and analysis much faster and cheaper, with dramatic token savings and latency improvements for LLM-driven development workflows. The project supports multiple programming languages and includes utilities for warming an index and then generating LLM-ready contexts or summaries of specific parts of a project.

Tokenize is a Julia package that serves a similar purpose and API as the tokenize module in Python but for Julia. This is to take a string or buffer containing Julia code, perform lexical analysis and return a stream of tokens.