FYI - Whitespace Tokeniser - altered in v1.6

Developers
2006-11-27
2013-04-25
  • ReverendSam
    ReverendSam
    2006-11-27

    The tokeniser has been updated in version 1.6 - this is to remove false tokenisations when presented with strings with multiple adjoining delimiter characters (i.e. two space characters together, "previously  bad  tokenisation  example")

    This tokeniser is now fixed, this may cause a slight alteration in metric scores to token based metrics in the rare cases where multiple space characters are together in the given comparison strings.