https://sourceforge.net/p/text-analysis/wiki/StringMatching/Recent changes to StringMatchingenTue, 20 May 2014 09:36:46 -0000https://sourceforge.net/p/text-analysis/wiki/StringMatching/<div class="markdown_content"><h2 id="string-matching">String Matching</h2> <p>Performing a dictionary-based search of terms on long texts may be very time expensive, if you use a naive approach such as String#indexOf(.).<br /> The Aho-Corasick algorithm finds all matches against a dictionary in linear time, that means independently regards to the number of matches or the size of the dictionary. It particularly helpful for dictionary-based Named Entity recognition tasks. </p> <p>Below some results from the comparison between the Aho-Corasick algorithm and indexOf: </p> <p>Terms in dictionary: 10100 </p> <p>Text lenght<br /> Aho-Corasick (ms)<br /> indexof (ms) </p> <p>1000<br /> 164<br /> 128 </p> <p>2000<br /> 296<br /> 260 </p> <p>3000<br /> 292<br /> 401 </p> <p>4000<br /> 341<br /> 531 </p> <p>5000<br /> 137<br /> 639 </p> <p>6000<br /> 151<br /> 770 </p> <p>7000<br /> 284<br /> 895 </p> <p>8000<br /> 254<br /> 1074 </p> <p>9000<br /> 337<br /> 1161 </p> <p>10000<br /> 348<br /> 1321 </p> <p>Terms in dictionary: 25100 </p> <p>Text lenght<br /> Aho-Corasick (ms)<br /> indexof (ms) </p> <p>1000<br /> 771<br /> 401 </p> <p>2000<br /> 633<br /> 646 </p> <p>3000<br /> 853<br /> 961 </p> <p>4000<br /> 714<br /> 1308 </p> <p>5000<br /> 978<br /> 1616 </p> <p>6000<br /> 660<br /> 1899 </p> <p>7000<br /> 680<br /> 2319 </p> <p>8000<br /> 647<br /> 2546 </p> <p>9000<br /> 871<br /> 2916 </p> <p>10000<br /> 580<br /> 3186 </p> <p>Terms in dictionary: 40100 </p> <p>Text lenght<br /> Aho-Corasick (ms)<br /> indexof (ms) </p> <p>1000<br /> 1652<br /> 512 </p> <p>2000<br /> 1047<br /> 1017 </p> <p>3000<br /> 986<br /> 1531 </p> <p>4000<br /> 1767<br /> 2054 </p> <p>5000<br /> 890<br /> 2524 </p> <p>6000<br /> 1496<br /> 3124 </p> <p>7000<br /> 1107<br /> 3589 </p> <p>8000<br /> 883<br /> 4163 </p> <p>9000<br /> 3546<br /> 4618 </p> <p>10000<br /> 983<br /> 5135 </p> <p>Usually the Aho-Corasick exceed indexOf for texts longer than 2000 chars. </p> <p>As you can see in the following diagram, while the indexOf-performance increases linearly by longer texts, the Aho-Corarick- performance remains quite invariable. <img alt="" src="/userapps/trac/kostia76/raw-attachment/wiki/StringMatching/Performance%20Comparison%20Diagram.jpg" /></p></div>KostiaTue, 20 May 2014 09:36:46 -0000https://sourceforge.net7edd1d48561dbd9ae80f70e700ce73688941e74e