tokenizer free download

Showing 4 open source projects for "tokenizer"

View related business solutions

PHP Clear Filters & Widen Search

Stop vibe-debugging.
Plug Claude into your app's actual errors.

AppSignal's MCP server hands Claude, Cursor, or Zed your real errors, traces, and the deploy that shipped them. AI writes the fix; you review the diff.

Free 30 days.
$300 Free Credits to Build on Google Cloud
New to Google Cloud? Get $300 in credits to explore Compute Engine, BigQuery, Cloud Run, Gemini Enterprise Agent Platform, and more.

Start your next project with $300 in free Google Cloud credit. Spin up VMs, run containers, query petabytes in BigQuery, or build agents with Gemini Enterprise Agent Platform. Once your credits are used, keep building with 20+ always-free tier products including Compute Engine, Cloud Storage, GKE, and Cloud Run functions. No commitment required—just sign up and start building.

Claim $300 Free
1

Tokenizer

A small library for converting tokenized PHP source code into XML

A small library for converting tokenized PHP source code into XML. You can add this library as a local, per-project dependency to your project using Composer. If you only need this library during development, for instance to run your project's test suite, then you should add it as a development-time dependency.

Downloads: 1 This Week

Last Update: 2025-12-08
See Project
2

EWBTokenFactory for PHP

EWBTokenFactory for PHP is a whitespace inclusive tokenizer. Most tokenizers will grab each word as a token and drop the whitespace. This tokenizer will also tokenize the spaces in between into their own tokens. Also supports associated token data.

Downloads: 0 This Week

Last Update: 2013-04-03
See Project
3

Nstag

Namespaced Template Engine (PHP extension) - An extremely powerful Tokenizer driven Template engine with XSL-like syntax. Nstag works based on the idea to have special tags in a separate namespace to apply view related logic or just assignments.

Downloads: 1 This Week

Last Update: 2014-03-29
See Project
4

PHPDoctor: The PHP Documentation Creator

JavaDoc style comment parser for PHP code, written in PHP using the PHP tokenizer extension. Designed with an emphasis on speed and simplicity.

Downloads: 6 This Week

Last Update: 2013-04-16
See Project
Build Agents and Models on One Platform
Everything you need to build production-ready agents and models. Access 200+ Google and third-party AI models and tools.

Gemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.

Try It Free