Search Results for "token"

ULS is a class library for creating lexical analyzer from language specification file. It's provided as C/C++/C# and Java (class) libraries with a few of tools on Linux/Windows platform.

Kayacc is an alternate name for the TOMBO project. I am in the process of updating the Tombo project page with version 2.1.0. Please download Tombo2.1.0.tar.gz from the Files tab. Tombo is a compiler generation toolkit for constructing compilers and interpreters. It includes a BNF compiler that generates parsers for context free grammars, and a class library to support compiler development. The class library supports symbol tables, Unicode text manipulation, reserved word mappings,...

This project is derived from the dk.brics.automaton DFA library, version 1.11-8. The most significant change is the addition of a 'info' field to automaton states, allowing them to be associated with a token type. A few automaton operations were modified to preserve the value of this field. I have not given much thought to whether it would even make sense to preserve the 'info' field through some of the other operations. This project also includes an extension of RunAutomaton, called TokenAutomaton, that can be used as a lexer, as well as its support classes and examples of its use. ...

A run-time configurable character stream tokenizer that allows the user to define token classes via regular expressions. The developer is not limited to predefined notions of whitespace, commenting, or word modalities.