The tokeniser has been updated in version 1.6 - this is to remove false tokenisations when presented with strings with multiple adjoining delimiter characters (i.e. two space characters together, "previously bad tokenisation example")
This tokeniser is now fixed, this may cause a slight alteration in metric scores to token based metrics in the rare cases where multiple space characters are together in the given comparison strings.
Log in to post a comment.
Sign up for the SourceForge newsletter:
You seem to have CSS turned off.
Please don't fill out this field.