Search Results for "text encoding"
Sort By:
Showing 127 open source projects for "text encoding"
View related business solutions-
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
1
Text Encoding Initiative
TEI produces the TEI Guidelines and associated software
The TEI is an international and interdisciplinary standard used by libraries, museums, publishers, and academics to represent all kinds of literary and linguistic texts, using an encoding scheme that is maximally expressive and minimally obsolescent. -
2
Tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models
tiktoken is a high-performance, tokenizer library (based on byte-pair encoding, BPE) designed for use with OpenAI’s models. It handles encoding and decoding text to token IDs efficiently, with minimal overhead. Because tokenization is a fundamental step in preparing text for models, tiktoken is optimized for speed, memory, and correctness in model contexts (e.g. matching OpenAI’s internal tokenization). The repo supports multiple encodings (e.g. -
3
cryptii
Web app and framework offering modular conversion and encoding
Web app and framework offering modular conversion, encoding and encryption. Translations are done client-side without any server interaction. This framework and web app aims to support a wide variety of ciphers, formats, algorithms and methods (called 'Bricks') while keeping them easily combinable. There are currently two types of Bricks: Encoders and Viewers. Encoders manipulate content by encoding or decoding in a specific way and using specific settings while Viewers allow users to access and edit the content fed into or outputted by Encoders in a certain way and format. ... -
4
httpexpect
End-to-end HTTP and REST API testing for Go
...Basically, httpexpect is a set of chainable builders for HTTP requests and assertions for HTTP responses and payload, on top of net/http and several utility packages. URL path construction, with simple string interpolation provided by go-interpol package. URL query parameters (encoding using go-querystring package). Headers, cookies, payload: JSON, urlencoded or multipart forms (encoding using form package), plain text. Custom reusable request builders and request transformers. Type-specific assertions, supported types: object, array, string, number, boolean, null, datetime. Regular expressions. Simple JSON queries (using subset of JSONPath), provided by jsonpath package. ... -
Train ML Models With SQL You Already KnowBuild and deploy ML models using familiar SQL. Automate data prep with built-in Gemini. Query 1 TB and store 10 GB free monthly.
-
5
Endroid QR Code
QR Code Generator
Endroid QR Code is a PHP library that allows developers to generate QR codes with customizable parameters. It supports creating QR codes in various formats, including PNG and SVG, and offers options for encoding URLs, text, or other data. The library is flexible and easy to integrate into applications that require QR code generation, such as ticketing systems or payment gateways. -
6
EditorConfig
Sublime Text plugin for EditorConfig
EditorConfig Sublime is a Sublime Text plugin that applies EditorConfig coding-style rules to files opened in the editor. It helps teams maintain consistent formatting conventions even when developers use different editors or operating systems. Supported settings include indentation style and size, line endings, character encoding, trailing-whitespace removal, and final-newline insertion. -
7
SentencePiece
Unsupervised text tokenizer for Neural Network-based text generation
SentencePiece is an unsupervised text tokenizer and detokenizer mainly for Neural Network-based text generation systems where the vocabulary size is predetermined prior to the neural model training. SentencePiece implements subword units (e.g., byte-pair-encoding (BPE) [Sennrich et al.]) and unigram language model [Kudo.]) with the extension of direct training from raw sentences. -
8
Ksoup
Ksoup is a lightweight Kotlin Multiplatform library for parsing HTML
Ksoup is a lightweight Kotlin Multiplatform library for parsing HTML, extracting HTML tags, attributes, and text, and encoding and decoding HTML entities. -
9
PHP QR Code
A PHP QR Code generator and reader with a user-friendly API
chillerlan/php-qrcode is a modern, flexible PHP library for generating QR codes. It supports various customization options such as size, encoding, error correction, and logo embedding. The library is PSR-compliant and built for ease of use in modern PHP projects, making it suitable for generating QR codes in web apps, receipts, and authentication systems. -
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
10
zpdf
Zero-copy PDF text extraction library written in Zig
...It implements multiple PDF decompression filters and handles common font encoding pathways, which are essential for turning raw PDF content streams into readable text. It also understands both classic cross-reference tables and newer cross-reference streams, including PDF 1.5+ features, and it offers configurable strict vs permissive error handling depending on whether you prioritize correctness or robustness. -
11
Render
Go package for easily rendering JSON, XML, binary data, and HTML
...XML: Uses the encoding/xml package to marshal data into an XML-encoded response. Binary data: Passes the incoming data straight through to the HTTP.ResponseWriter. Text: Passes the incoming string straight through to the http.ResponseWriter. Render comes with a variety of configuration options. By default Render will attempt to load templates with a '.tmpl' extension from the "templates" directory. -
12
MIME Component
Allows manipulating MIME messages
...It is commonly used for handling email content and attachments in Symfony applications. The component supports building complex email structures, including multi-part messages, and correctly encoding text and binary data. Mime is an essential part of Symfony Mailer but can be utilized independently to handle MIME message generation. -
13
Nette Utility Classes
Lightweight utilities for string & array manipulation, image handling
In package nette/utils you will find a set of useful classes for everyday use. Lightweight utilities for string & array manipulation, image handling, safe JSON encoding/decoding, validation, slug or strong password generating etc. -
14
ripgrep
Regex pattern directory search tool that respects your .gitignore
...By default, ripgrep will ignore your .gitignore and skip hidden files or directories and binary files automatically. ripgrep has first class support on Windows, macOS and Linux, with binary downloads available for every release. ripgrep is similar to other popular search tools like The Silver Searcher, ack and grep. ripgrep supports arbitrary input preprocessing filters which could be PDF text extraction, less supported decompression, decrypting, automatic encoding detection and so on. In other words, use ripgrep if you like speed, filtering by default, fewer bugs and Unicode support. -
15
pdf-inspector
Fast Rust library for PDF inspection, classification
pdf-inspector is a fast Rust library for classifying PDFs and extracting structured text without OCR. It distinguishes text-based, scanned, image-based, and mixed documents while returning confidence scores and pages that may need OCR. Its position-aware parser preserves font data, coordinates, reading order, and multi-column layouts. The converter produces clean Markdown with headings, lists, code blocks, links, page breaks, and formatted tables. It supports CID fonts, several encodings,... -
16
Yjs
Shared data types for building collaborative software
Yjs is a high-performance, open-source CRDT (Conflict-free Replicated Data Type) implementation for building collaborative, real-time applications. It enables multiple users to edit shared data structures—such as text documents, arrays, maps, and XML trees—synchronously and offline. Yjs is network-agnostic and works with WebRTC, WebSocket, or any other transport layer, making it ideal for collaborative editors, whiteboards, and design tools. Its compact updates and powerful reconciliation... -
17
LLaMA-Mesh
Unifying 3D Mesh Generation with Language Models
LLaMA-Mesh is a research framework that extends large language models so they can understand and generate 3D mesh data alongside text. The system introduces a method for representing 3D meshes in a textual format by encoding vertex coordinates and face definitions as sequences that can be processed by a language model. By serializing 3D geometry into text tokens, the approach allows existing transformer architectures to generate and interpret 3D models without requiring specialized visual tokenizers. ... -
18
International Components for Unicode
The home of the ICU project source code
...ICU is released under a nonrestrictive open-source license that is suitable for use with both commercial software and with other open-source or free software. Convert text data to or from Unicode and nearly any other character set or encoding. ICU's conversion tables are based on charset data collected by IBM over the course of many decades and is the most complete available anywhere. Compare strings according to the conventions and standards of a particular language, region or country. ICU's collation is based on the Unicode Collation Algorithm plus locale-specific comparison rules from the Common Locale Data Repository, a comprehensive source for this type of data. -
19
Kafdrop
Kafka Web UI
Kafdrop is a web UI for viewing Kafka topics and browsing consumer groups. The tool displays information such as brokers, topics, partitions, and consumers, and lets you view messages. This project is a reboot of Kafdrop 2.x, dragged kicking and screaming into the world of Java 17+, Kafka 2.x, Helm and Kubernetes. It's a lightweight application that runs on Spring Boot and is dead-easy to configure, supporting SASL and TLS-secured brokers. -
20
eslint-plugin-unicorn
More than 100 powerful ESLint rules
More than 100 powerful ESLint rules. You might want to check out XO, which includes this plugin. Each rule has emojis denoting if it belongs to the recommended configuration if some problems reported by the rule are automatically fixable by the --fix command line option, or if some problems reported by the rule are manually fixable by editor suggestions. Use a preset config or configure each rules in package.json. -
21
Redpanda Console
Redpanda Console is a developer-friendly UI for managing your workload
...Explore your topics' messages in our message viewer through ad-hoc queries and dynamic filters. Find any message you want using JavaScript functions to filter messages. Supported encodings are JSON, Avro, Protobuf, XML, MessagePack, Text and Binary (hex view). The used encoding (except Protobuf) is recognized automatically. Redpanda is a Kafka®-compatible streaming data platform that is proven to be 10x faster and 6x lower in total costs. It is also JVM-free, ZooKeeper®-free, Jepsen-tested, and source available. A single binary with built-in everything. No ZooKeeper®. ... -
22
vim-airline
Very light status/tabline for vim
Lean & mean status/tabline for vim that's light as air. When the plugin is correctly loaded, there will be a nice statusline at the bottom of each vim window. That line consists of several sections, each one displaying some piece of information. For a better look, any sections can be colored differently, depending on various conditions (e.g. the mode or whether the current file is 'modified'). Tiny core written with extensibility in mind (open/closed principle). Integrates with a variety of... -
23
Loro
Make your JSON data collaborative and version-controlled with CRDTs
loro is a high-performance CRDT (Conflict-free Replicated Data Type) engine designed for building collaborative applications that sync in real time across multiple peers or devices. Written in Rust, loro is designed to be compact, fast, and embeddable in a wide range of environments, from desktop to mobile to web via WebAssembly. Its architecture supports multiple data types like text, maps, and lists, and it offers automatic conflict resolution with minimal data overhead. It is ideal for... -
24
Qwen-2.5-VL
Qwen2.5-VL is the multimodal large language model series
Qwen2.5 is a series of large language models developed by the Qwen team at Alibaba Cloud, designed to enhance natural language understanding and generation across multiple languages. The models are available in various sizes, including 0.5B, 1.5B, 3B, 7B, 14B, 32B, and 72B parameters, catering to diverse computational requirements. Trained on a comprehensive dataset of up to 18 trillion tokens, Qwen2.5 models exhibit significant improvements in instruction following, long-text generation... -
25
MiniMax-01
Large-language-model & vision-language-model based on Linear Attention
MiniMax-01 is the official repository for two flagship models: MiniMax-Text-01, a long-context language model, and MiniMax-VL-01, a vision-language model built on top of it. MiniMax-Text-01 uses a hybrid attention architecture that blends Lightning Attention, standard softmax attention, and Mixture-of-Experts (MoE) routing to achieve both high throughput and long-context reasoning. It has 456 billion total parameters with 45.9 billion activated per token and is trained with advanced parallel...
