https://sourceforge.net/p/jobimtext/wiki/FAQ/Recent changes to FAQenWed, 10 Jul 2013 12:17:33 -0000https://sourceforge.net/p/jobimtext/wiki/FAQ/<div class="markdown_content"><pre>--- v1 +++ v2 @@ -1,4 +1,3 @@ - [TOC] # Similarity Calculations using Hadoop @@ -15,3 +14,5 @@ ~~~~~~ python generateHadoopScript.py dataset 1000 0 0 1000 LMI 200 ~~~~~~ + +If the error still remains, you might use a version before 0.0.6 and should change to the latest version. If you don't want to change, follow the documentation found [here](https://sourceforge.net/p/jobimtext/wiki/jobimtext_pipeline/?version=33). </pre> </div>Martin RiedlWed, 10 Jul 2013 12:17:33 -0000https://sourceforge.net8f6cfda8d6ed329ab45095d9889bf0ba6f513414https://sourceforge.net/p/jobimtext/wiki/FAQ/<div class="markdown_content"><div class="toc"> <ul> <li><a href="#similarity-calculations-using-hadoop">Similarity Calculations using Hadoop</a><ul> <li><a href="#which-are-the-recommended-parameter-to-generate-the-hadoop-similarity-pipeline">Which are the recommended parameter to generate the Hadoop similarity pipeline?</a></li> <li><a href="#the-computations-of-the-similarities-stop-with-an-exception-eg-error-2997-encountered-ioexception-file-pigfreqsig1000pig-does-not-exist">The computations of the similarities stop with an exception e.g. ERROR 2997: Encountered IOException. File pig/FreqSig1000.pig does not exist.</a></li> </ul> </li> </ul> </div> <h1 id="similarity-calculations-using-hadoop">Similarity Calculations using Hadoop</h1> <h2 id="which-are-the-recommended-parameter-to-generate-the-hadoop-similarity-pipeline">Which are the recommended parameter to generate the Hadoop similarity pipeline?</h2> <p>We recommend to use only features that occur with less then 1000 words and specify to use only positive significance scores. We recommend, using the LMI significance measure and keep only the top 1000 features per term with the highest significance scores. Normally it should be sufficient to keep the top 200 most similar terms for each term. This is achieved with:</p> <div class="codehilite"><pre><span class="n">python</span> <span class="n">generateHadoopScript</span><span class="p">.</span><span class="n">py</span> <span class="n">dataset</span> <span class="mi">1000</span> <span class="mi">0</span> <span class="mi">0</span> <span class="mi">1000</span> <span class="n">LMI</span> <span class="mi">200</span> </pre></div> <h2 id="the-computations-of-the-similarities-stop-with-an-exception-eg-error-2997-encountered-ioexception-file-pigfreqsig1000pig-does-not-exist">The computations of the similarities stop with an exception e.g. <em>ERROR 2997: Encountered IOException. File pig/FreqSig1000.pig does not exist.</em></h2> <p>The reason for that error is a wrong parameter, when generating the script (generateHadoopScript.py) to run the Hadoop pipeline. Instead of a significance measure a number was given to the script. Following parameters could be used for generating the script:</p> <div class="codehilite"><pre><span class="n">python</span> <span class="n">generateHadoopScript</span><span class="p">.</span><span class="n">py</span> <span class="n">dataset</span> <span class="mi">1000</span> <span class="mi">0</span> <span class="mi">0</span> <span class="mi">1000</span> <span class="n">LMI</span> <span class="mi">200</span> </pre></div> </div>Martin RiedlWed, 10 Jul 2013 12:15:20 -0000https://sourceforge.net6fe36e4ec0f6d838b3f1c82f43780cd5293df755