#26 CJKTokenizer.Next() function bug

[Anonymous]

The bug is in the CJKTokenizer.Next() function.

Here's the code fragment:

---------
dataLen = input.Read(ioBuffer, 0, ioBuffer.Length);
...
...
if (dataLen == -1)

---------

The issue is that this Read method of the TextReader
returns "0" if called when no more characters are
left to read, while the code is checking for -1!

Below's the documentation from MSDN for the
TextReader.Read Method (Char[], Int32, Int32)

Return Value
The number of characters that have been read. The
number will be less than or equal to count, depending
on whether the data is available within the stream.
This method returns zero if called when no more
characters are left to read.