EWBTokenFactory for PHP

Status: Beta

Brought to you by: ericburns

Downloads: 0 This Week

Last Update: 2013-04-03

Add to Preferred Sources

EWBTokenFactory for PHP is a whitespace inclusive tokenizer. Most tokenizers will grab each word as a token and drop the whitespace. This tokenizer will also tokenize the spaces in between into their own tokens. Also supports associated token data.

Project Activity

See All Activity >

License

GNU Library or Lesser General Public License version 2.0 (LGPLv2)

Follow EWBTokenFactory for PHP

EWBTokenFactory for PHP Web Site

Other Useful Business Software

Veeam Data Platform v13.1 - Get Your Free Trial Icon

Veeam Data Platform v13.1 - Get Your Free Trial

Secure by design, portable by default. Recover clean, fast, anywhere. Start a free trial.

Try Veeam Data Platform today. Experience the unified platform that's secure by design, portable by default, and proven to recover clean, fast, and anywhere.

Try it Free

Rate This Project

Login To Rate This Project

User Reviews

Be the first to post a review of EWBTokenFactory for PHP!

Additional Project Details

Languages

English

Intended Audience

Developers

Programming Language

Registered

2009-12-12

Report inappropriate content

Build Agents and Models on One Platform

Everything you need to build production-ready agents and models. Access 200+ Google and third-party AI models and tools.

Gemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.

Start Free

Recommended Projects

PHPDoctor: The PHP Documentation Creator
JavaDoc style comment parser for PHP code, written in PHP using the PHP tokenizer extension. Designed with an emphasis on speed and simplicity.
Tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models
Whitespace development IDE
wspacegen is an interactive IDE for programing with whitespace (http://compsoc.dur.ac.uk/whitespace/). So it's possible to create .ws code relative easily and debug it as well. But attention, you'll addicted to it (-:
Tokenizers
Fast State-of-the-Art Tokenizers optimized for Research and Production
GPoSTTL: Enhanced Brill's Tagger
Enhanced version of Brill's Parts-of-Speech Tagger with built-in Tokenizer and Lemmatizer.