Search Results for "token"

Logos is a lexer generator for Rust, designed to create fast and efficient lexers with minimal code. It leverages Rust's procedural macros to define token patterns, streamlining the process of lexical analysis.​

Parser for Rust source code is a major Rust crate for parsing Rust source code (token streams) into a syntax tree (AST) that procedural macros can inspect or transform. The primary target is macro authors: you can parse TokenStreams into syn::File, syn::Item, syn::Expr, syn::Type, etc. It offers rich data structures, fine-grained parsing, span tracking (for error reporting), traversal and mutation APIs (visit, fold, visit_mut), printing back to tokens, and strong feature-gating so you only compile what you need. ...

...Their state machines are precompiled and serialized, so run time initialization is fast no matter how large the rule set. JPlex is not a parser generator; it only generates an interface for its parsers, and it does not build a parse tree. When the lexer identifies a token that has a named event associated with it, it passes the token to the parser method associated with that event. A lexer can dispatch token events to any number of parsers. JPlex is steadily being improved, and the Subversion repo will almost always be newer and better than the binary download.