Bug with character encoding

Brought to you by: reverendsam

#3 Bug with character encoding

Status: open-later

Owner: nobody

Labels: None

Priority: 5

Updated: 2011-05-18

Created: 2008-07-07

Creator: Anonymous

Private: No

Some of the metrics (for example BlockDistance) fail if one of the strings has a unicode 160 (non-blocking space) in.

ReverendSam - 2011-05-18

by fail how do you mean - not the expected results - currently the tokenisation is not very unicode aware, development is currently focused on lower bit unicode as a primary focus, this should be easy to address in the tokenisation code however

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

ReverendSam - 2011-05-18

status: open --> open-later
If you would like to refer to this comment somewhere else in this project, copy and paste the following link: