Tokenize is a Julia package that serves a similar purpose and API as the tokenize module in Python but for Julia. This is to take a string or buffer containing Julia code, perform lexical analysis and return a stream of tokens.

Features

  • Fast, it currently lexes all of Julia source files in ~0.25 seconds (580 files, 2 million Tokens)
  • Round trippable, that is, from a stream of tokens the original string should be recoverable exactly
  • Round trippable, that is, from a stream of tokens the original string should be recoverable exactly
  • The function tokenize is the main entrypoint for generating Tokens
  • Each Token is represented by where it starts and ends, what string it contains and what type it is
  • Documentation available

Project Samples

Tokenize.jl Screenshot 1

Project Activity

See All Activity >

Categories

3D Modeling, Data Visualization, Libraries

License

MIT License

Follow Tokenize.jl

Tokenize.jl Web Site

Other Useful Business Software
99.99% Uptime for MySQL and PostgreSQL Databases Icon
99.99% Uptime for MySQL and PostgreSQL Databases

Sub-second maintenance. 2x read/write performance. Built-in vector search for AI apps.

Cloud SQL Enterprise Plus delivers near-zero downtime with 35 days of point-in-time recovery. Supports MySQL, PostgreSQL, and SQL Server.
Try Free
Rate This Project
Login To Rate This Project

User Reviews

Be the first to post a review of Tokenize.jl!

Additional Project Details

Programming Language

Julia

Related Categories

Julia 3D Modeling Software, Julia Data Visualization Software, Julia Libraries

Registered

2023-12-07
Report inappropriate content