Search Results for "token"

ULS is a class library for creating lexical analyzer from language specification file. It's provided as C/C++/C# and Java (class) libraries with a few of tools on Linux/Windows platform.

...The slg tool takes a .lexer file that contains the description of a lexical analyzer as input and produces C++ source code for a lexical analyzer as output. The produced lexical analyzer is a finite state machine that recognizes patterns described as regular expressions and return corresponding token identifiers to the parser. The spg tool takes .parser files that contain descriptions of parsers as input and produces C++ source code for parser classes as output. The parser generator produces a recursive descent top-down backtracking parser that use the lexical analyzer generated by slg to tokenize the input.

The goal of this project is to provide a generator for lexical analyzers of maximum computational efficiency and maximum range of applications. This includes the support for Unicode (UTF8, UTF16, ...) and a large variety of other encodings directly and via nested converters such as ICU(tm) and IConv. Sophisticated buffer handling allows to operate on plain file streams, on sockets, or manually fed buffer content. 'Ready-to-build' examples explain related concepts and facilitate practical...

Kayacc is an alternate name for the TOMBO project. I am in the process of updating the Tombo project page with version 2.1.0. Please download Tombo2.1.0.tar.gz from the Files tab. Tombo is a compiler generation toolkit for constructing compilers and interpreters. It includes a BNF compiler that generates parsers for context free grammars, and a class library to support compiler development. The class library supports symbol tables, Unicode text manipulation, reserved word mappings,...