Search Results for "token"

ULS is a class library for creating lexical analyzer from language specification file. It's provided as C/C++/C# and Java (class) libraries with a few of tools on Linux/Windows platform.

The goal of this project is to provide a generator for lexical analyzers of maximum computational efficiency and maximum range of applications. This includes the support for Unicode (UTF8, UTF16, ...) and a large variety of other encodings directly and via nested converters such as ICU(tm) and IConv. Sophisticated buffer handling allows to operate on plain file streams, on sockets, or manually fed buffer content. 'Ready-to-build' examples explain related concepts and facilitate practical...

Kayacc is an alternate name for the TOMBO project. I am in the process of updating the Tombo project page with version 2.1.0. Please download Tombo2.1.0.tar.gz from the Files tab. Tombo is a compiler generation toolkit for constructing compilers and interpreters. It includes a BNF compiler that generates parsers for context free grammars, and a class library to support compiler development. The class library supports symbol tables, Unicode text manipulation, reserved word mappings,...

This project is derived from the dk.brics.automaton DFA library, version 1.11-8. The most significant change is the addition of a 'info' field to automaton states, allowing them to be associated with a token type. A few automaton operations were modified to preserve the value of this field. I have not given much thought to whether it would even make sense to preserve the 'info' field through some of the other operations. This project also includes an extension of RunAutomaton, called TokenAutomaton, that can be used as a lexer, as well as its support classes and examples of its use. ...

Modern scripting language and the first one that presents the concept of programming with direct fiction description using a clear subset of pure human language. Using Supernova you can write the source code using Arabic and/or English keywords and you can create GUI applications for Windows and Linux.

CTokenizer accepts C++ syntax as input and outputs a vector of CToken objects, which contain a token string and a(n enumerated) token type, which can then be used to parse the input.

A run-time configurable character stream tokenizer that allows the user to define token classes via regular expressions. The developer is not limited to predefined notions of whitespace, commenting, or word modalities.