Tokenizer Files

A small library for converting tokenized PHP source code into XML

This is an exact mirror of the Tokenizer project, hosted at https://github.com/theseer/tokenizer. SourceForge is not affiliated with Tokenizer.

The interactive file manager requires Javascript. Please enable it or use sftp or scp.
You may still browse the files here.

Name	Modified	Size	InfoDownloads / Week
Parent folder
README.md	2025-12-08	275 Bytes	0
Release 2.0.1 source code.tar.gz	2025-12-08	9.0 kB	0
Release 2.0.1 source code.zip	2025-12-08	13.3 kB	0
Totals: 3 Items		22.6 kB	0

Removed the custom token T_AMPERSAND as PHP 8.1+ provides tokens for it already and our 2.0.0 release overwrote it. See #44 for details.

Full Changelog: https://github.com/theseer/tokenizer/compare/2.0.0...2.0.1

Source: README.md, updated 2025-12-08

Other Useful Business Software

Gemini 3 and 200+ AI Models on One Platform

Access Google's best plus Claude, Llama, and Gemma. Fine-tune and deploy from one console.

Build generative AI apps with Vertex AI. Switch between models without switching platforms.

Start Free

Full-stack observability with actually useful AI | Grafana Cloud

Our generous forever free tier includes the full platform, including the AI Assistant, for 3 users with 10k metrics, 50GB logs, and 50GB traces.

Built on open standards like Prometheus and OpenTelemetry, Grafana Cloud includes Kubernetes Monitoring, Application Observability, Incident Response, plus the AI-powered Grafana Assistant. Get started with our generous free tier today.

Create free account

Gemini 3 and 200+ AI Models on One Platform

Access Google's best plus Claude, Llama, and Gemma. Fine-tune and deploy from one console.

Build generative AI apps with Vertex AI. Switch between models without switching platforms.

Start Free

Recommended Projects

docconv
Converts PDF, DOC, DOCX, XML, HTML, RTF, etc to plain text
Bedrock
WordPress boilerplate with modern development tools
Xml2Json Converter
Simple tool for converting large XML-files to JSON or JSON to XML
Android Network Tools
Set of useful android network tools
JWT
A simple library to work with JSON Web Token and JSON Web Signature

Tokenizer Files

A small library for converting tokenized PHP source code into XML

Get an email when there's a new version of Tokenizer