Search Results for "ids"

tiktoken is a high-performance, tokenizer library (based on byte-pair encoding, BPE) designed for use with OpenAI’s models. It handles encoding and decoding text to token IDs efficiently, with minimal overhead. Because tokenization is a fundamental step in preparing text for models, tiktoken is optimized for speed, memory, and correctness in model contexts (e.g. matching OpenAI’s internal tokenization). The repo supports multiple encodings (e.g. “cl100k_base”) and lets users switch encoding names to match different model contexts. ...

PRM800K is a process supervision dataset accompanying the paper Let’s Verify Step by Step, providing 800,000 step-level correctness labels on model-generated solutions to problems from the MATH dataset. The repository releases the raw labels and the labeler instructions used in two project phases, enabling researchers to study how human raters graded intermediate reasoning. Data are stored as newline-delimited JSONL files tracked with Git LFS, where each line is a full solution sample that...

...The released model is a 14.7 MB ONNX artifact based on a fine-tuned MiniLM-L6-H384 encoder with approximately 18.5 million parameters and a 35-label BIO classification head covering 17 entity types. It works alongside a deterministic recognizer that handles structured identifiers such as phone numbers and IDs, forming a defense-in-depth client-side redaction system. Rampart supports English, Spanish, French, German, Italian, Portuguese, and Dutch, and is designed to run efficiently even on older devices, making it suitable for browsers, mobile and applications.