If supplying a multibyte string as either "haystack" or "needle", results are unpredictable:
>>> ngram.compare('dfsédfsdf', 'dfsédfsdf')
Note that the trigrams in the middle are not trigrams at all, but di-grams because the multibyte character is recognized as two characters. Essentially, in this case, as everything get's treated the same, the end-result is correct. However, when supplying the ctrings as unicode objects, it all works as expected:
>>> ngram.compare(u'dfsédfsdf', u'dfsédfsdf')
For this reason, I will add a type-check that will only allow unicode objects to be passed down into the module. This may also reveal possible encoding trouble, beacuse the unicode conversion will most likely fail in that case. So the module will not accept data that is obviously wrong.
Log in to post a comment.