Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

Building NLP

2008-01-30
2013-04-16
  • Hello,

    I want to build a NLP and I know that the first block to build is the parser so I can get words from a text.

    But I am not sure on what kind of information that should be prepared in the database? Is it a list of words (stems)? Is it the word patterns (regular expression list that describe morpheme of each word).

    Thank you,

    fasisi

     
    • Thomas Morton
      Thomas Morton
      2008-01-30

      Hi,
        To really address your post I'd need to write an introduction to how text processing is done and specifically how its down in opennlp.  There are plenty of resource for the first question but here is a good start.

      http://www.aaai.org/AITopics/html/natlang.html

      The second question is addressed implicitly in the README where the processing stages are specified in the examples.  These are the types of processing that opennlp provides in roughly the order the need to be applied.

      Low-level Processing
      ---------------------
      Sentece Detection
      Tokenization (word detection)

      Grammatical Analysis
      --------------------
      POS-Tagging (part-of-speech identification)
      Phrase Chunking
      Full-Parsing (construct a grammatical tree for a sentence)

      Semantic Analysis
      -----------------
      Named-Entity Detection (identify the semantic type of many proper nouns)

      Discourse Analysis
      ------------------
      Coreference Resolution (resolve pronouns and other referring expressions).

      Hope this helps...Tom

       
      • I have read several reference from aaai and have downloaded the opennlp tools and models. Can I build the tools with NetBeans?

        Thank you,

        fasisi

         
    • SFMania
      SFMania
      2008-01-30

      Sorry, but I am new to this NLP, and I have a similar question regarding NLP in data-mining. Where should I go from here?! Your above answer is excellent, but I am still confused.

      Please note that I was successfully building the openNLP, and running a few tests on it; however, I have no clue on what to do next, and where to find an easy example for it, e.g.: semantic, sql query building, etc. Can you kindly enlight us? Thank you very much for your help.

       
      • Hello sle98,

        I am new to NLP too. I think it is a good start to make study mate? :)
        What do you want to achieve by studying NLP?

        My dream is to make NLP for Indonesia language. The NLP will read text in Indonesia and user can post question to NLP and NLP can answer. Hopefully I can make the NLP learn from the text it read and become smarter. Well, it is just a dream.

        I have downloaded the opennlp tools, too but I still not sure how to use the tools and I don't know how to use the example. Can you help me?

        Thank you,

        fasisi

         
    • Thomas Morton
      Thomas Morton
      2008-02-04

      Hi,
         There are some examples in the README which should give you an sense of what the processing does. 
         As far as what you can do with text processing and specifically opennlp there isn't a direct application provided in these tools.  They are just a set of tools that may be useful in applications that contain a lot of other programming.

         Also that state of research in NLP/AI is not at the point where there is any ability to read in the human sense of the word.  Understanding at a sentence level is basically an unsolved problem. 

      Things that are tractable include:

      Putting text into categories.
      Grammatical processing.
      Some discourse processing.

      There are some applications which give a sense of understanding, but if one looks at the details they are basically complicated classification systems.  These include essay grading, and machine translation.

      Hope this helps...Tom