CScanner is a Chinese lexical scanner using the forward maximum matching algorithm to tokenize BIG5 Chinese text or text mixed with BIG5 and ASCII characters. (I did not implement other algorithms which are more complex.) It is an attempt to hide the complexity of Chinese word identification by providing an interface at a higher level, which is realized as a lexical scanner.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm been trying to generate a C++ interface of libtabe's y and these days.
I intended to implement hao's "lexical scanner" idea at 2 levels: the first pass at "chunk" level (in libtabe's terminology), and the second at "token" level. Thanks to hao's CScanner, it's a lot easier now. :)
As for a C++ wrapper, I had some difficulty in cleanning up libtabe's memory management codes about Chu, Chunk, and Tsi. So perhaps I'll re-structure these parts a lot.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I guess the use of the word "chunk" originated from my old MMSEG. Not sure if it is an appropriate word. By the way, I just released version 0.20 of CScanner, in which all algorithms implemented in the MMSEG system have been re-analyzed and re-implemented. The code should be more clean and much easier to read. As to the performance, even when using the set of complex algorithms, the performance was still very good. A lot better than the old implementation.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Finally I was able to get rid of the nasty for-loops and make mmseg recursive, so the new code is cleaner.
For strict sequential algorithms such as forward maximum matching, a "chunk" of text is not really necessary. However, for algorithms requireing a larger context, there is indeed a need for that "chunk."
As to romanization, I would suggest to use "ju" for y, "ci" for , "ciyin" for , "zhuyin" for `, etc, in the source code and documentation. My feeling was that using internationally recognized standard romanization may make libtabe more accessible and readable to people outside Taiwan who want to understand libtabe but are not able to read BIG5 characters. If you don't like hanyu pinyin, just ignore this suggestion.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
http://www.geocities.com/hao510/cscanner/
CScanner is a Chinese lexical scanner using the forward maximum matching algorithm to tokenize BIG5 Chinese text or text mixed with BIG5 and ASCII characters. (I did not implement other algorithms which are more complex.) It is an attempt to hide the complexity of Chinese word identification by providing an interface at a higher level, which is realized as a lexical scanner.
I'm been trying to generate a C++ interface of libtabe's y and these days.
I intended to implement hao's "lexical scanner" idea at 2 levels: the first pass at "chunk" level (in libtabe's terminology), and the second at "token" level. Thanks to hao's CScanner, it's a lot easier now. :)
As for a C++ wrapper, I had some difficulty in cleanning up libtabe's memory management codes about Chu, Chunk, and Tsi. So perhaps I'll re-structure these parts a lot.
I guess the use of the word "chunk" originated from my old MMSEG. Not sure if it is an appropriate word. By the way, I just released version 0.20 of CScanner, in which all algorithms implemented in the MMSEG system have been re-analyzed and re-implemented. The code should be more clean and much easier to read. As to the performance, even when using the set of complex algorithms, the performance was still very good. A lot better than the old implementation.
I confess that I was only serious about Tsi's memory management, not Chu or Chunk. 8Q
Let me know if I can fix something.
Finally I was able to get rid of the nasty for-loops and make mmseg recursive, so the new code is cleaner.
For strict sequential algorithms such as forward maximum matching, a "chunk" of text is not really necessary. However, for algorithms requireing a larger context, there is indeed a need for that "chunk."
As to romanization, I would suggest to use "ju" for y, "ci" for , "ciyin" for , "zhuyin" for `, etc, in the source code and documentation. My feeling was that using internationally recognized standard romanization may make libtabe more accessible and readable to people outside Taiwan who want to understand libtabe but are not able to read BIG5 characters. If you don't like hanyu pinyin, just ignore this suggestion.
Actually I have no preference.
The use of Chu, Tsi, Tsiyin and ZuYin is a straightforward decision to me. The change can be made, but it's time consuming.
Chunk is a vital component in my opinion. Without chunk, the segmentation algorithm will have to consider the difference between ASCCI and Big5.