Search Results for "tokenizer"

...It includes MusicGen for music generation conditioned on text (and optionally melody) and AudioGen for text-conditioned sound effects and environmental audio. Both models operate over discrete audio tokens produced by a neural codec (EnCodec), which acts like a tokenizer for waveforms and enables efficient sequence modeling. The repo provides inference scripts, checkpoints, and simple Python APIs so you can generate clips from prompts or incorporate the models into applications. It also contains training code and recipes, so researchers can fine-tune on custom data or explore new objectives without building infrastructure from scratch. ...

javalang is a pure Python library for working with Java source code. javalang provides a lexer and parser targeting Java 8. The implementation is based on the Java language spec.

flex is a tool for generating scanners. A scanner, sometimes called a tokenizer, is a program which recognizes lexical patterns in text. The flex program reads user-specified input files, or its standard input if no file names are given, for a description of a scanner to generate. The description is in the form of pairs of regular expressions and C code, called rules. Flex generates a C source file named, "lex.yy.c", which defines the function yylex().

BASIC interpreter for the 16bit PIC microcontroller 24FJ64GA002. The interpreter runs on the chip only, no compiler/tokenizer is needed. Communication with PC is done by USB-to-serial converter cable.

Similar to StringTokenizer already in the Java standard library, but capable of discerning different types of data as tokens (e.g. "a=1" will yield 3 tokens) as it doesn't rely on whitespaces or some single delimiter (as StringTokenizer does).

The tokenization and segmentation for the Czech language.

JavaDoc style comment parser for PHP code, written in PHP using the PHP tokenizer extension. Designed with an emphasis on speed and simplicity.

A JAVA suite for parsing arbitrary text data. Not just HTML or XML or Java, but all of them. Use it when the JDK tokenizers are too limited, JavaCC, JTB etc. are too complicated, or You need dynamic parser configuration

Pure Python implementation of GOLD Parser Engine. GOLD Parser Engine is a LALR(1) parser with DFA tokenizer. It uses compiled grammar table generated by GOLD Parser Builder (not included - available on http://www.devincook.com/goldparser)

A run-time configurable character stream tokenizer that allows the user to define token classes via regular expressions. The developer is not limited to predefined notions of whitespace, commenting, or word modalities.

NQXML is a pure Ruby implementation of a non-validating XML processor. It includes an XML tokenizer, a SAX-style streaming XML parser, a DOM-style tree parser, an XML writer, and a context-sensitive callback mechanism.