JoBimText Wiki

Linking Language to Knowledge with Distributional Semantics

Status: Beta

Brought to you by: apanchenko, biem-tuda, coppolab, eugenso, and 4 others

jobimtext_pipeline

There is a newer version of this page. You can find it here.

Getting Started with the JoBim Text Project

This page describes, how to use the jobim text project to use distributional similarities or contextualized similarities in your project. In this example we consider the that you work on the same computer, the Hadoop server is running. The used compenents use some components from dkpro, uimafit and OpenNLP.

Getting Started with the JoBim Text Project

Holing System: Extract the Features

The files needed for this tutorial can be downloaded from the download section and are contained in the archive jobimtext_pipline_vXXX.tar.gz. For the feature extraction we need also text files. The format of the files should be plain text. A corpus of Web data and of sentences extracted from English Wikipedia are available here and start with the prefix dataset_. We advise to split them, so UIMA does not have to keep the complete file in the memory.
This can be done using the split command from linux:

split news10M splitted/news10M-part-

Then the holing system can be started to extract the features. This can be done using the shell script holing_operation.sh in the download section, or by executing the jobimtext.example.holing.HolingHadoop class in the maven svn project jobimtext/jobimtext.example. The script can be executed as following

sh holing_operation.sh path pattern output extractor_configuration holing_system_name

and has following parameters:

path: path where the files are, or zip in format: jar:file:/dir/file.zip!
pattern: pattern the files matches e.g. *.txt for all txt files
holing_system_name: Ngram[hole_position,ngram] or MaltParser (MaltParser is only working when using the source code directly and not the jar, as there are some problems with the MlatParser library)
extractor_configuration: file that contains all informations needed to configure the output format for Keys and Features

An example for the extractor_configuration file is shown below:

<jobimtext.holing.extractor.JobimExtractorConfiguration>
  <keyValuesDelimiter>  </keyValuesDelimiter>
  <extractorClassName>
      jobimtext.holing.extractor.TokenExtractors$LemmaPos
  </extractorClassName>
  <attributeDelimiter>#</attributeDelimiter>
  <valueDelimiter>_</valueDelimiter>
  <valueRelationPattern>$relation($values)</valueRelationPattern>
  <holeSymbol>@</holeSymbol>
</jobimtext.holing.extractor.JobimExtractorConfiguration>

The output of the holing system using this configuration file leads to a tab separated (specified by keyValuesDelimiter) key and context feature separated list. The element extractorClassName specifies how an entry is concatenated, in this case the lemma and the POS tag of a word are used and concatenated using a dash (#) as defined with attributeDelimiter. The name of the relation and the context features are concatenated following the valueRelationPattern pattern. An output of the holing system using the MaltParser with the introduced extractor file for the sentence "I gave the book to the girl" leads to the following result:

I#PRP   -nsubj(@_give#VB)
give#VB nsubj(@_I#PRP)
give#VB prep(@_to#TO)
give#VB dobj(@_book#NN)
give#VB punct(@_.#.)
the#DT  -det(@_book#NN)
book#NN -dobj(@_give#VB)
book#NN det(@_the#DT)
to#TO   pobj(@_girl#NN)
to#TO   -prep(@_give#VB)
the#DT  -det(@_girl#NN)
girl#NN det(@_the#DT)
girl#NN -pobj(@_to#TO)
.#.     -punct(@_give#VB)

One can observe that the tokens are lemmatized and the Pos tags are concatenated to the lemma of the token using the dash.

Calculate the Distributional Similarities

Afterwards the file should be splitted again and then transferred to the file system (hdfs) of the MapReduce server:

split -a 5 -d news10M_hadoop_input splitted/news10M_maltdependency_part-
hadoop dfs -copyFromLocal splitted news10M_maltdependency

the execution pipeline for the MapReducer can be generated using the script generateHadoopScript using the following parameters:

generateHadoopScript.py dataset s t p significance simsort_count [computer file_prefix]

with

dataset: e.g. news10M_ngram_1_3, news10M_maltparser
s: minimal threshold used for the significance between word and feature
t: minimal threshold for the word count
p: number of features used per word
significance: LMI, PMI, LL
simsort count: number of similar terms sorted in the last step
computer: computer to copy the files to
file_prefix: prefix, the files (distributional similarity, WordCount, FeatureCount) will be copied to

for example the command

python generateHadoopScript.py news10M_maltdependency 0 0 1000 LL 200 desktop_computer dt/

will lead to the output file named news10M_maltdependency_s0_t0_p1000_LL_simsort200 with the following content:

#hadoop dfs -rmr wordcount_out  news10M_maltdependency__WordCount
#hadoop dfs -rmr context_out    news10M_maltdependency__WordFeatureCount
#hadoop dfs -rmr featurecount_out       news10M_maltdependency__FeatureCount
#hadoop dfs -rmr freqsig_out    news10M_maltdependency__FreqSigLL_s_0_t_0
#hadoop dfs -rmr prunegraph_out news10M_maltdependency__FreqSigLL_s_0_t_0__PruneGraph_p_1000
#hadoop dfs -rmr aggregate_out  news10M_maltdependency__FreqSigLL_s_0_t_0__PruneGraph_p_1000__AggrPerFt
#hadoop dfs -rmr simcount_out   news10M_maltdependency__FreqSigLL_s_0_t_0__PruneGraph_p_1000__AggrPerFt__SimCounts1WithFeatures
#hadoop dfs -rmr simsort_out    news10M_maltdependency__FreqSigLL_s_0_t_0__PruneGraph_p_1000__AggrPerFt__SimCounts1WithFeatures__SimSortlimit_200
hadoop jar distributional_thesaurus/hadoop.jar jobimtext.thesaurus.distributional.hadoop.mapreduce.WordFeatureCount news10M_maltdependency news10M_maltdependency__WordFeatureCount True
hadoop jar distributional_thesaurus/hadoop.jar jobimtext.thesaurus.distributional.hadoop.mapreduce.FeatureCount news10M_maltdependency__WordFeatureCount news10M_maltdependency__FeatureCount True
hadoop jar distributional_thesaurus/hadoop.jar jobimtext.thesaurus.distributional.hadoop.mapreduce.WordCount news10M_maltdependency__WordFeatureCount news10M_maltdependency__WordCount True
pig  -param s=0 -param t=0 -param wordcountout=news10M_maltdependency__WordCount -param featurecountout=news10M_maltdependency__FeatureCount -param contextout=news10M_maltdependency__WordFeatureCount -param freqsigout=news10M_maltdependency__FreqSigLL_s_0_t_0 distributional_thesaurus/pig/FreqSigLL.pig
pig  -param p=1000 -param freqsigout=news10M_maltdependency__FreqSigLL_s_0_t_0 -param prunegraphout=news10M_maltdependency__FreqSigLL_s_0_t_0__PruneGraph_p_1000 distributional_thesaurus/pig/PruneGraph.pig
hadoop jar distributional_thesaurus/hadoop.jar jobimtext.thesaurus.distributional.hadoop.mapreduce.AggrPerFt news10M_maltdependency__FreqSigLL_s_0_t_0__PruneGraph_p_1000 news10M_maltdependency__FreqSigLL_s_0_t_0__PruneGraph_p_1000__AggrPerFt True
hadoop jar distributional_thesaurus/hadoop.jar jobimtext.thesaurus.distributional.hadoop.mapreduce.SimCounts1WithFeatures news10M_maltdependency__FreqSigLL_s_0_t_0__PruneGraph_p_1000__AggrPerFt news10M_maltdependency__FreqSigLL_s_0_t_0__PruneGraph_p_1000__AggrPerFt__SimCounts1WithFeatures True
pig  -param limit=200 -param IN=news10M_maltdependency__FreqSigLL_s_0_t_0__PruneGraph_p_1000__AggrPerFt__SimCounts1WithFeatures -param OUT=news10M_maltdependency__FreqSigLL_s_0_t_0__PruneGraph_p_1000__AggrPerFt__SimCounts1WithFeatures__SimSortlimit_200 distributional_thesaurus/pig/SimSort.pig
ssh desktop_computer   mkdir dt
hadoop dfs -text  news10M_maltdependency__WordCount/p* | ssh desktop_computer  ->  dt/news10M_maltdependency__WordCount
hadoop dfs -text  news10M_maltdependency__WordFeatureCount/p* | ssh desktop_computer  ->  dt/news10M_maltdependency__WordFeatureCount
hadoop dfs -text  news10M_maltdependency__FeatureCount/p* | ssh desktop_computer  ->  dt/news10M_maltdependency__FeatureCount
hadoop dfs -text  news10M_maltdependency__FreqSigLL_s_0_t_0__PruneGraph_p_1000__AggrPerFt__SimCounts1WithFeatures__SimSortlimit_200/p* | ssh desktop_computer  ->  dt/news10M_maltdependency__FreqSigLL_s_0_t_0__PruneGraph_p_1000__AggrPerFt__SimCounts1WithFeatures__SimSortlimit_200

The first lines of the script are commented and can be used to delete the files from the server. After executing the script we have to wait until the Hadoop server is finished. The files are copied to the specified computer into the folder specified by the prefix.

Add data to be used as database:

Yet we support two databases: MySQL and DCA, a memory-based data server provided within this project. Here we will only describe the DCA server. The configuration files for the DCA are generated using the create_db_dca.sh script:

sh create_db_dca.sh folder prefix database_server

folder: folder where all the files, from the Hadoop step are located
prefix: prefix for the files (e.g. news10M_maltparser)
database_server: name of server, where the database runs

This command create two files PREFIX_dcaserver and PREFIX_dcaserver_tables.xml with the following content:

<jobimtext.util.db.conf.DatabaseTableConfiguration>
  <tableOrder2>subset_wikipedia-maltparser__FreqSigLL_s_0_t_0__PruneGraph_p_1000__AggrPerFt__SimCounts1WithFeatures__SimSortlimit_200</tableOrder2>
  <tableOrder1>subset_wikipedia-maltparser__FreqSigLL_s_0_t_0</tableOrder1>
  <tableValues>subset_wikipedia-maltparser__FeatureCount</tableValues>
  <tableKey>subset_wikipedia-maltparser__WordCount</tableKey>
</jobimtext.util.db.conf.DatabaseTableConfiguration>

and

# TableID       ValType TCPP#   TableLines      CacheSize       MaxValues               DataAllocation          InputFileNames/Dir      FileFilter
subset_wikipedia-maltparser__FreqSigLL_s_0_t_0__PruneGraph_p_1000__AggrPerFt__SimCounts1WithFeatures__SimSortlimit_200  TABLE   8080    0       10000   10000   server[0-19228967]      /home/user/data/out/dt/subset_wikipedia-maltparser__FreqSigLL_s_0_t_0__PruneGraph_p_1000__AggrPerFt__SimCounts1WithFeatures__SimSortlimit_200   NONE
subset_wikipedia-maltparser__FreqSigLL_s_0_t_0  TABLE   8081    0       10000   10000   server[0-19228967]      /home/user/data/out/dt/subset_wikipedia-maltparser__FreqSigLL_s_0_t_0   NONE
subset_wikipedia-maltparser__FeatureCount       SCORE   8082    0       10000   10000   server[0-19228967]      /home/user/data/out/dt/subset_wikipedia-maltparser__FeatureCount        NONE
subset_wikipedia-maltparser__WordCount  SCORE   8083    0       10000   10000   server[0-19228967]      /home/user/data/out/dt/subset_wikipedia-maltparser__WordCount   NONE

Further details for the DCA server are specified within the README file withint the DCA project in the subversion. The server then be started using the PREFIX_dcaserver configuration file using the following command:

java -Xmx... -Xms.... -cp $(echo lib/*jar| tr ' ' ':') com.ibm.sai.dca.server.Server PREFIX_dcaserver

sh apply_dt_ct.sh $APPLY_FOLDER $APPLY_FILE $EXTRACTOR $HOLINGSYSTEM $FILE"_dcaserver" $FILE"_dcaserver_tables.xml"

When also specfying the computer and the folder using the script from the previous script, the data is availabel locally.

Get expansions for new text

When all the data is loaded into the database, we can use script apply_dt_ct.sh to get expansions of words for new documents.

The

--------------------------------------------------------------------------------
sh apply_dt_ct.sh path pattern holing_system_name extractor_configuration database_configuration database_tables
--------------------------------------------------------------------------------

path: path of the files (also zip files could be used e.g.: jar:file:/dir/file.zip!
pattern: pattern the files matches, that should be expanded (e.g. *.txt for all txt files)
extractor_configuration: file that contains all informations needed for the output format for Keys and Features
holing_system_name: Ngram[hole_position,ngram] or MaltParser (Standard)
database_configuration: configuration file needed for the dca server
database_tables: condfiguration file for the java software, specifying the table names
targetword: if true the target word has to be encapsulated using <target>word</target>. Otherwise every word will be expanded. (Default value : true)
--------------------------------------------------------------------------------

The input format of the files can be plain text, when exanding all words. Therefore, the parameter targetword should be set to false. When expanding solely selected words they should be encapsulated by <target>target_word</target>.

For the impatient ones

Here we show an example to execute all steps, where everything (hadoop server) is running on one system using the MaltParser. Probably the number of lines the files are splitted should be adjusted to the used dataset.

Calculate the distributional thesaurus

FILEDIR=/home/user/data
FILE=textfile
OUTPUT=/home/user/data/out


DB_SERVER=server
EXTRACTOR=extractor_standard.xml
HOLINGSYSTEM=MaltParser
HOLINGNAME=maltparser

#Holing Operation
mkdir -p $OUTPUT
mkdir -p $OUTPUT/splitted/
split $FILEDIR/$FILE $OUTPUT/splitted/$FILE
sh holing_operation.sh $OUTPUT/splitted $FILE* $OUTPUT/$FILE-$HOLINGNAME $EXTRACTOR $HOLINGSYSTEM
mkdir $OUTPUT/$FILE-$HOLINGNAME-splitted/

#Compute distributional similarity
split -a 5 -l 2500000 -d $OUTPUT/$FILE-$HOLINGNAME $OUTPUT/$FILE-$HOLINGNAME-splitted/part-
hadoop dfs -copyFromLocal $OUTPUT/$FILE-$HOLINGNAME-splitted $FILE-$HOLINGNAME
mkdir $OUTPUT/dt/
python generateHadoopScript.py $FILE-$HOLINGNAME 0 0 1000 LL 200 localhost $OUTPUT/dt/
sh $FILE-$HOLINGNAME"_s0_t0_p1000_LL_simsort200"

Start the databser server

#Load and start databaseserver
sh create_db_dca.sh $OUTPUT/dt/ $FILE $DB_SERVER
java -Xmx3g -cp $(echo lib/*jar| tr ' ' ':') com.ibm.sai.dca.server.Server $FILE"_dcaserver"

Expand the text in a given text file

APPLY_FOLDER=./
APPLY_FILE=test.txt

#start dt and ct on file
sh apply_dt_ct.sh $APPLY_FOLDER $APPLY_FILE $EXTRACTOR $HOLINGSYSTEM $FILE"_dcaserver" $FILE"_dcaserver_tables.xml"