Search Results for "tokenizer"

A C++ high-performance regex library and Flex-compatible lexical analyzer generator with full Unicode support, new indentation anchors, lazy quantifiers, and many other modern features. Accepts Flex lexer specification syntax and is compatible with Bison/Yacc parsers. Generates reusable source code that is easy to understand. Supports fast scanning of UTF-8/16/32 files, strings, and streams. The reflex scanner generator generates clean C++ lexer class code that is thread-safe. Generates...

flex is a tool for generating scanners. A scanner, sometimes called a tokenizer, is a program which recognizes lexical patterns in text. The flex program reads user-specified input files, or its standard input if no file names are given, for a description of a scanner to generate. The description is in the form of pairs of regular expressions and C code, called rules. Flex generates a C source file named, "lex.yy.c", which defines the function yylex().

Tokenizer and translator for the IDL language, principally used with the IDL environment from ITTVis and GDL.

Pure Python implementation of GOLD Parser Engine. GOLD Parser Engine is a LALR(1) parser with DFA tokenizer. It uses compiled grammar table generated by GOLD Parser Builder (not included - available on http://www.devincook.com/goldparser)