LexGupthor / Blog: Recent posts

Astrologers proclaim week of the lexer. Lexer population doubles!

V0.1.0 (@revision 48) is the first LexGupthor release. It takes an input stream of Gupta source code and gives you back an STD (Gupta Syntax Tree Description). As an STD consumer you can traverse it and use the few specialized nodes (for function, parameters section, local/static variables section) or analyze yourself the lexed product contained in the non-specialized nodes.... read more

• Overhaul string search (use new simultaneous string search rather than sequence of substring-then-compare).
• Intern whitespaces and punctuation at primitive and token level.

Terribly surprised and a bit frustrated to benchmark after implementing these, that the speed and memory consumption does not change AT ALL. Interning the strings does change a bit the memory fill pattern, not the maximum value though and not the execution speed. Suffering SUCCOTASH!!!... read more

The lexer mechanics are pretty much complete, it offers the minimum required look-behind and take-ahead to be dumb but useful (most of the logic will be offloaded onto the STD). From now on, support for new markup+tokens should be a basic matter of adding them.

The Gupta STD (Gupta Syntax Tree Description ^_^) is fresh but already in a "probably done" state. Its build pass #1 arranges the linear lexed product in an content agnostic tree. Pass #2 traverses and attempts upgrading the nodes into node subclasses. Most of the following work will be on implementing these.... read more

Wohoo! Milestone: lexer can do "empty" Gupta files ("empty" in Gupta terms == many KB of content to start with).

Look ma we're online! So far we have:

• low level: tokenizer (works)
• mid level: lexer (very basic, works in primitive cases :P)
• high level: abstract syntax tree (just an empty class ^_^)
• thorough tests for what is already implemented