Java Suffix array library for phrase discovery. Inspired initially by the classic paper of Yamamoto & Church, with newer ideas from Abouelhoda et al and Kim et al. Adapted for large alphabet so that words can be tokenized as alphabet characters.

Features

  • Adapted to large alphabet for NLP
  • Includes tokenizers, normalizers and symbol table.
  • Calculates term and document frequency; full distribution across texts
  • Modular design, user can apply various statistics to phrases
  • Includes Aho Corasick automaton, also for large alphabet
  • Needs: Better sorting, though radix qsort usually works okay
  • Needs: improvement to the symbol table. This is the slowest part.
  • Needs: Tokenizers, normalizers for more languages.
  • Needs: Some links to foma (foma.sourceforge.net/)

Project Activity

See All Activity >

Categories

Linguistics, Natural Language Processing (NLP)

License

Apache Software License

Follow suffix arrays for phrase extraction

suffix arrays for phrase extraction Web Site

You Might Also Like
Our Free Plans just got better! | Auth0 by Okta Icon
Our Free Plans just got better! | Auth0 by Okta

With up to 25k MAUs and unlimited Okta connections, our Free Plan lets you focus on what you do best—building great apps.

You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your secuirty. Auth0 now, thank yourself later.
Try free now
Rate This Project
Login To Rate This Project

User Reviews

Be the first to post a review of suffix arrays for phrase extraction!

Additional Project Details

Intended Audience

Information Technology

Programming Language

Java

Related Categories

Java Linguistics Software, Java Natural Language Processing (NLP) Tool

Registered

2010-05-02
Report inappropriate content