#15 Support for Big5 encoding (larger character sets)

[Anonymous]

The following illustrates changes to the JLog parser (pTokenizerTable
and pTokenizeStream) to add support for large character sets (e.g., for
Chinese), and the big5 encoding (the ** lines below appear to indicate
lines to add to the methods they appear in):

ubc.cs.JLog.Parser.pTokenizerTable:

public pTokenizerTable() {
table = new int[65280];
//table = new int[256]; resetSyntax(); // comment out these lines
};

beacause every word in big5 start with 4 byte so I increase
the table to 65536-256 = 65280
another change: ubc.cs.JLog.Parser.pTokenizeStream

protected void initTable() {
table.setTokenType('%', TOKEN_LINECOMMENT);
table.setTokenType(256, 65279, TOKEN_WORDS); ....
}
protected void initSinglesTable() { singles_table.setTokenType(256, 65279, TOKEN_WORDS);
....
}
protected void initBaseNumberTable() {
basenumber_table.setTokenType(256, 65279, TOKEN_WORDS);
....
}
protected void initStringTable() { string_table.setTokenType(256, 65279, TOKEN_WORDS); .
...
}
protected void initArrayTable() {
array_table.setTokenType(256, 65279, TOKEN_WORDS);
....
}
protected void initLineCommentTable() { linecomment_table.setTokenType(256, 65279, TOKEN_WORDS); ....
}
protected void initCommentTable() {
** comment_table.setTokenType(256, 65279, TOKEN_WORDS); ....
}