Search Results for "tokenizer"

...Starting from version 3.0, IK has developed into a common word segmentation component for Java, independent of the Lucene project, and at the same time provides a default optimized implementation of Lucene. In the 2012 version, IK implemented a simple word segmentation ambiguity elimination algorithm, marking the evolution of the IK tokenizer from pure dictionary word segmentation to analog semantic word segmentation.

We have implemented a core summarizer of scientific articles written in Spanish, with the following components: a tokenizer, a grammar checker, a clarity checker, a cohesion-coherence checker, a common-topic extractor and an output formatter.

...Here we are hosting our projects related to solr and lucene. log4jSolr - log4j appender (and more) to index all log events solr_core - solr analysis extensions such as filters oder tokenizer

Similar to StringTokenizer already in the Java standard library, but capable of discerning different types of data as tokens (e.g. "a=1" will yield 3 tokens) as it doesn't rely on whitespaces or some single delimiter (as StringTokenizer does).

NLPTools-ES is a Spanish plugin for GATE (General Architecture for Text Engineering). It includes a tokenizer, sentence splitter, gazetteer, pos tagger.

a tokenizer that is general purpose, and multilanguage. it will provide consistent usage across the languages, and be tied into that language's unique common usage types.

LACE means "Lucene Analyzer for CJK (Chinese/Japanese/Korean) & English". It's a simple tokenizer that can handle English-CJK mixed text. Chinese words are handled using a dictionary based method.

Framework for tokenizing information contained in legacy files.

A JAVA suite for parsing arbitrary text data. Not just HTML or XML or Java, but all of them. Use it when the JDK tokenizers are too limited, JavaCC, JTB etc. are too complicated, or You need dynamic parser configuration

BPWJ is Java package that allows users to quickly develop parsers for application specific mini-languages. New languages are created incrumentally from generic tokenizer and parser classes.

A run-time configurable character stream tokenizer that allows the user to define token classes via regular expressions. The developer is not limited to predefined notions of whitespace, commenting, or word modalities.

a Lucene/Solr Analyzer/Tokenizer for documents written in Japanese using Igo morphological analyzer.

Making Hebrew properly searchable by IR software. Right now, most work is being done in our mailing list (planning), and on our github repository (concept code, see below).