Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo


Building NLP

  • Hello,

    I want to build a NLP and I know that the first block to build is the parser so I can get words from a text.

    But I am not sure on what kind of information that should be prepared in the database? Is it a list of words (stems)? Is it the word patterns (regular expression list that describe morpheme of each word).

    Thank you,


    • Thomas Morton
      Thomas Morton

        To really address your post I'd need to write an introduction to how text processing is done and specifically how its down in opennlp.  There are plenty of resource for the first question but here is a good start.


      The second question is addressed implicitly in the README where the processing stages are specified in the examples.  These are the types of processing that opennlp provides in roughly the order the need to be applied.

      Low-level Processing
      Sentece Detection
      Tokenization (word detection)

      Grammatical Analysis
      POS-Tagging (part-of-speech identification)
      Phrase Chunking
      Full-Parsing (construct a grammatical tree for a sentence)

      Semantic Analysis
      Named-Entity Detection (identify the semantic type of many proper nouns)

      Discourse Analysis
      Coreference Resolution (resolve pronouns and other referring expressions).

      Hope this helps...Tom

      • I have read several reference from aaai and have downloaded the opennlp tools and models. Can I build the tools with NetBeans?

        Thank you,


    • SFMania

      Sorry, but I am new to this NLP, and I have a similar question regarding NLP in data-mining. Where should I go from here?! Your above answer is excellent, but I am still confused.

      Please note that I was successfully building the openNLP, and running a few tests on it; however, I have no clue on what to do next, and where to find an easy example for it, e.g.: semantic, sql query building, etc. Can you kindly enlight us? Thank you very much for your help.

      • Hello sle98,

        I am new to NLP too. I think it is a good start to make study mate? :)
        What do you want to achieve by studying NLP?

        My dream is to make NLP for Indonesia language. The NLP will read text in Indonesia and user can post question to NLP and NLP can answer. Hopefully I can make the NLP learn from the text it read and become smarter. Well, it is just a dream.

        I have downloaded the opennlp tools, too but I still not sure how to use the tools and I don't know how to use the example. Can you help me?

        Thank you,


    • Thomas Morton
      Thomas Morton

         There are some examples in the README which should give you an sense of what the processing does. 
         As far as what you can do with text processing and specifically opennlp there isn't a direct application provided in these tools.  They are just a set of tools that may be useful in applications that contain a lot of other programming.

         Also that state of research in NLP/AI is not at the point where there is any ability to read in the human sense of the word.  Understanding at a sentence level is basically an unsolved problem. 

      Things that are tractable include:

      Putting text into categories.
      Grammatical processing.
      Some discourse processing.

      There are some applications which give a sense of understanding, but if one looks at the details they are basically complicated classification systems.  These include essay grading, and machine translation.

      Hope this helps...Tom