Search Results for "token"

Logos is a lexer generator for Rust, designed to create fast and efficient lexers with minimal code. It leverages Rust's procedural macros to define token patterns, streamlining the process of lexical analysis.​

Parser for Rust source code is a major Rust crate for parsing Rust source code (token streams) into a syntax tree (AST) that procedural macros can inspect or transform. The primary target is macro authors: you can parse TokenStreams into syn::File, syn::Item, syn::Expr, syn::Type, etc. It offers rich data structures, fine-grained parsing, span tracking (for error reporting), traversal and mutation APIs (visit, fold, visit_mut), printing back to tokens, and strong feature-gating so you only compile what you need. ...

...The slg tool takes a .lexer file that contains the description of a lexical analyzer as input and produces C++ source code for a lexical analyzer as output. The produced lexical analyzer is a finite state machine that recognizes patterns described as regular expressions and return corresponding token identifiers to the parser. The spg tool takes .parser files that contain descriptions of parsers as input and produces C++ source code for parser classes as output. The parser generator produces a recursive descent top-down backtracking parser that use the lexical analyzer generated by slg to tokenize the input.

...Their state machines are precompiled and serialized, so run time initialization is fast no matter how large the rule set. JPlex is not a parser generator; it only generates an interface for its parsers, and it does not build a parse tree. When the lexer identifies a token that has a named event associated with it, it passes the token to the parser method associated with that event. A lexer can dispatch token events to any number of parsers. JPlex is steadily being improved, and the Subversion repo will almost always be newer and better than the binary download.