Search Results for "unicode"

...Tesseract can recognize over 100 languages out-of-the-box, and can be trained to recognize other languages. It supports various output formats, including plain text, HTML, PDF and more. It also has unicode (UTF-8) support.

NeMo Curator is a Python library specifically designed for fast and scalable dataset preparation and curation for large language model (LLM) use-cases such as foundation model pretraining, domain-adaptive pretraining (DAPT), supervised fine-tuning (SFT) and paramter-efficient fine-tuning (PEFT). It greatly accelerates data curation by leveraging GPUs with Dask and RAPIDS, resulting in significant time savings. The library provides a customizable and modular interface, simplifying pipeline...

minbpe is a minimal, clean implementation of byte-level Byte Pair Encoding (BPE), the tokenization approach widely used in modern language models. It operates on UTF-8 encoded bytes rather than Unicode characters, which makes it robust to arbitrary text inputs and avoids needing a language-specific character vocabulary. The repository is structured as a teaching-oriented implementation that shows how to train a tokenizer by learning merge rules, then apply those merges to encode text into token IDs and decode tokens back into text. ...

**Burn To The Brim** is a highly efficient archiving utility designed to solve the classic subset-sum (bin packing) optimization challenge. It intelligently selects and groups files and directories (documents, high-fidelity media, or raw back-ups) to optimally fill recordable Blu-Rays, USB drives or custom-capacity storage drives. By recursively scanning your designated folders, BTTB matches item sizes to your media capacity, finding a near-perfect selection in milliseconds and an...

Unitag is a language-independent Unicode-based part-of-speech tagging system. Written entirely in ANSI-compatible C, it should (in theory) compile on any OS, but has been tested on 32-bit Windows.

YouTokenToMe is a fast and efficient unsupervised text tokenization library designed for training subword embeddings, particularly useful for NLP models.

CLISP is a portable ANSI Common Lisp implementation and development environment by Bruno Haible. Interpreter, compiler, debugger, CLOS, MOP, FFI, Unicode, sockets, CLX. UI in English, German, French, Spanish, Dutch, Russian, and Danish.

Program Q AIML is a C++ Qt-based library offering a simple API to parse AIML XML files and then interacting with a user input with Latin/Arabic support (Unicode). AIML is a technology permitting to have an AI chat bot. A sample application is provided.

...Romanization is normally done using ASCII and extended ASCII syllables, which is easy to process but difficult to work with. Romanize project converts the Indian languages in their unicode form to english alphabets. Compared to the existing schemes of romanizations, this project focuses on few main points - Readability, easy typability, English alphabet combinations only, incoporations with existing popular schemes, phonetically equivallent transliterations and most importantly non-ambiguity across the languages (9) using the same transliteration mapping set.

The first Free Text-to-Speech (TTS) synthesis system in Malayalam based on festival speech synthesis system. It gives Orca - a free domain screen reading software, the additional functionality to read Unicode Malayalam documents and websites as well.

Our Objective is to create a GPU-based system that can accept scanned inputs of printed Devanagari texts, and produce outputs of the same in Unicode with a very high accuracy (>99.9%).

Collaborative development and distribution of Windows Speech Recognition (WSR) application macros to 1) improve the accessibility of personal computing for impaired users, and 2) improve the efficiency of personal computing for all users.

...Include: contour recognition; vectorisation; matrix letter feature recognition; auto page segmentation and detect rotation; SS3 ASM core; XML base; web-based GUI; 99,6% printed Unicode text recognition; letter base up to 1200 letters.

NOTE: I couldn't keep up this project to align with latest Unicode spec. Not sure I may be continuing. You can try Myanmar3 from Myanmar NLP or WinUniInnwa or https://sourceforge.net/projects/prahita/ or something better compliant font. ~Victor --- [This is UniBurma - UniMM project workshop area. This project currently have two productions, UniBurma and UniMM. For more descriptive info about this project, please visit http://unimm.org/.

Auvai is a Java API and Java Swing based application for Text to Speech conversion of Unicode Tamil. Future direction of this API and application is to support Text to Speech conversion for all "Indic" languages.

Kathak is a Bangla text to speech synthesizer which can produce the speech from unicode bangla text input. We are developing the system based on Festvox framework.The Festival Speech Synthesis System was used as a base for developing Kathak.

K is a C++ library featuring Genetic Programming, cryptography, 64 bits arithmetic, cross-platform macros and unicode conversion routines. It compiles on NewtonOS, Mac < X, Unix and Windows and it makes little use of C++ templates.