Hi! I'm the developer for XOWA -- an offline Wikipedia application. About a month ago, I changed XOWA to use Luaj to process the lua statements in Wikipedia's offline dumps. I'd just like to say how impressed I was in Luaj's functionality and performance. So far, it handles all the lua code in 30+ wikis, including en.wikipedia.org, en.wiktionary.org, etc..
In the process of integrating luaj into XOWA, I came across a minor number of issues. I don't know if this board is still active, but I wanted to list them below. Admittedly, the format is unstructured. If there is a formal process for submitting patches to Luaj, let me know, and I will follow up accordingly.
old:if(anchor)break;new:if(anchor)break;if(soffset>=srclen)break;// assert soffset is in bounds, else will throw ArrayIndexOutOfBounds exception;
string.gsub fails if string is empty
file: /src/core/org/luaj/vm2/lib/StringLib.java
proc: gsub
staticVarargsgsub(Varargsargs){LuaStringsrc=args.checkstring(1);finalintsrclen=src.length();if(srclen==0)returnvarargsOf(src,LuaValue.ZERO);// exit early
old:case'G':fdsc.format(result,args.checkdouble(arg));new:case'G':Stringfmt_str=newString(fmt.m_bytes,fdsc_bgn-1,fdsc.length+1);// -1 to include %; +1 to account for included %; basically get everything between % and f; EX: a%.1fb -> %.1f fdsc.format(result,fmt_str,args.checkdouble(arg));
* proc: format
old:buf.append(v));new:// buf.append( String.valueOf( x ) );if(fmt.startsWith("%0."))fmt="%"+fmt.substring(2);// remove leading 0, else MissingFormatWidthExceptionintfmt_len=fmt.length();if(fmt_len>1&&fmt.charAt(fmt_len-2)=='.')// penultimmate char has "."fmt=fmt.substring(0,fmt_len-1)+"0"+fmt.charAt(fmt_len-1);// add trailing 0, else UnknownFormatConversionException; EX: "02.f" -> "02.0f" buf.append(String.format(fmt,v));// call String.format
// trim wsintidx=i;while(idx<j){switch(m_bytes[idx]){case9:case10:case13:case32:++idx;i=idx;break;default:idx=j;break;}}idx=j-1;while(idx>=i){switch(m_bytes[idx]){case9:case10:case13:case32:j=idx;--idx;break;default:idx=i-1;break;}}
multi-byte strings not fully supported
file: /src/core/org/luaj/vm2/LuaString.java
publicstaticLuaStringvalueOf(char[]chars,intoff,intlen){// COMMENTED: does not handle 2+ byte chars; assumes 1 char = 1 byte// byte[] b = new byte[len];// for ( int i=0; i<len; i++ )// b[i] = (byte) chars[i + off];// return valueOf(b, 0, len);intbry_len=0;for(inti=0;i<len;i++){// iterate over chars to sum all single / multi-byte charsintb_len=LuaString.Utf16_Len_by_char((int)(chars[i+off]));if(b_len==4)++i;// 4 bytes; surrogate pair; skip next char;bry_len+=b_len;}byte[]bry=newbyte[bry_len];intbry_idx=0;inti=0;while(i<len){charc=chars[i+off];intb_len=Utf16_Encode_char(c,chars,i,bry,bry_idx);bry_idx+=b_len;i+=b_len==4?2:1;// 4 bytes; surrogate pair; skip next char;}returnvalueOf(bry,0,bry_len);}publicstaticStringdecodeAsUtf8(byte[]bytes,intoffset,intlength){// COMMENTED: does not handle 3+ byte chars// int i,j,n,b;// for ( i=offset,j=offset+length,n=0; i<j; ++n ) {// switch ( 0xE0 & bytes[i++] ) {// case 0xE0: ++i;// case 0xC0: ++i;// }// }// char[] chars=new char[n];// for ( i=offset,j=offset+length,n=0; i<j; ) {// chars[n++] = (char) (// ((b=bytes[i++])>=0||i>=j)? b:// (b<-32||i+1>=j)? (((b&0x3f) << 6) | (bytes[i++]&0x3f)):// (((b&0xf) << 12) | ((bytes[i++]&0x3f)<<6) | (bytes[i++]&0x3f)));// }// return new String(chars);returnnewString(bytes,offset,length,java.nio.charset.Charset.forName("UTF-8"));}publicstaticintlengthAsUtf8(char[]chars){// COMMENTED: does not handle 3+ byte chars// int i,b;// char c;// for ( i=b=chars.length; --i>=0; )// if ( (c=chars[i]) >=0x80 )// b += (c>=0x800)? 2: 1;// return b;intlen=chars.length;intrv=0;for(inti=0;i<len;i++){intb_len=LuaString.Utf16_Len_by_char(chars[i]);if(b_len==4)++i;// 4 bytes; surrogate pair; skip next char; rv+=b_len;}returnrv;}publicstaticintencodeToUtf8(char[]chars,intnchars,byte[]bytes,intoff){// COMMENTED: does not handle 4+ byte chars; already using Encode_by_int, so might as well be consistent// char c;// int j = off;// for ( int i=0; i<nchars; i++ ) {// if ( (c = chars[i]) < 0x80 ) {// bytes[j++] = (byte) c;// } else if ( c < 0x800 ) {// bytes[j++] = (byte) (0xC0 | ((c>>6) & 0x1f));// bytes[j++] = (byte) (0x80 | ( c & 0x3f)); // } else {// bytes[j++] = (byte) (0xE0 | ((c>>12) & 0x0f));// bytes[j++] = (byte) (0x80 | ((c>>6) & 0x3f));// bytes[j++] = (byte) (0x80 | ( c & 0x3f)); // }// }// return j - off;intbry_idx=off;inti=0;while(i<nchars){charc=chars[i];intbytes_read=Utf16_Encode_char(c,chars,i,bytes,bry_idx);bry_idx+=bytes_read;i+=bytes_read==4?2:1;// 4 bytes; surrogate pair; skip next char;}returnnchars;// NOTE: code returned # of bytes which is wrong; Globals.UTF8Stream.read caches rv as j which is used as index to char[] not byte[]; will throw out of bounds exception if bytes returned}privatestaticintUtf16_Len_by_char(intc){if((c>-1)&&(c<128))return1;// 1 << 7elseif(c<2048)return2;// 1 << 11elseif((c>55295)// 0xD800&&(c<56320))return4;// 0xDFFFelseif(c<65536)return3;// 1 << 16elsethrownewRuntimeException("UTF-16 int must be between 0 and 2097152; char="+c);}publicstaticintUtf16_Len_by_int(intc){if((c>-1)&&(c<128))return1;// 1 << 7elseif(c<2048)return2;// 1 << 11elseif(c<65536)return3;// 1 << 16elseif(c<2097152)return4;elsethrownewRuntimeException("UTF-16 int must be between 0 and 2097152; char="+c);}publicstaticintUtf8_Len_of_char_by_1st_byte(byteb){// SEE:w:UTF-8inti=b&0xff;// PATCH.JAVA:need to convert to unsigned byteswitch(i){case0:case1:case2:case3:case4:case5:case6:case7:case8:case9:case10:case11:case12:case13:case14:case15:case16:case17:case18:case19:case20:case21:case22:case23:case24:case25:case26:case27:case28:case29:case30:case31:case32:case33:case34:case35:case36:case37:case38:case39:case40:case41:case42:case43:case44:case45:case46:case47:case48:case49:case50:case51:case52:case53:case54:case55:case56:case57:case58:case59:case60:case61:case62:case63:case64:case65:case66:case67:case68:case69:case70:case71:case72:case73:case74:case75:case76:case77:case78:case79:case80:case81:case82:case83:case84:case85:case86:case87:case88:case89:case90:case91:case92:case93:case94:case95:case96:case97:case98:case99:case100:case101:case102:case103:case104:case105:case106:case107:case108:case109:case110:case111:case112:case113:case114:case115:case116:case117:case118:case119:case120:case121:case122:case123:case124:case125:case126:case127:case128:case129:case130:case131:case132:case133:case134:case135:case136:case137:case138:case139:case140:case141:case142:case143:case144:case145:case146:case147:case148:case149:case150:case151:case152:case153:case154:case155:case156:case157:case158:case159:case160:case161:case162:case163:case164:case165:case166:case167:case168:case169:case170:case171:case172:case173:case174:case175:case176:case177:case178:case179:case180:case181:case182:case183:case184:case185:case186:case187:case188:case189:case190:case191:return1;case192:case193:case194:case195:case196:case197:case198:case199:case200:case201:case202:case203:case204:case205:case206:case207:case208:case209:case210:case211:case212:case213:case214:case215:case216:case217:case218:case219:case220:case221:case222:case223:return2;case224:case225:case226:case227:case228:case229:case230:case231:case232:case233:case234:case235:case236:case237:case238:case239:return3;case240:case241:case242:case243:case244:case245:case246:case247:return4;default:thrownewRuntimeException("invalid initial utf8 byte; byte="+b);}}publicstaticintUtf16_Decode_to_int(byte[]ary,intpos){byteb0=ary[pos];if((b0&0x80)==0){returnb0;}elseif((b0&0xE0)==0xC0){return(b0&0x1f)<<6|(ary[pos+1]&0x3f);}elseif((b0&0xF0)==0xE0){return(b0&0x0f)<<12|((ary[pos+1]&0x3f)<<6)|(ary[pos+2]&0x3f);}elseif((b0&0xF8)==0xF0){return(b0&0x07)<<18|((ary[pos+1]&0x3f)<<12)|((ary[pos+2]&0x3f)<<6)|(ary[pos+3]&0x3f);}elsethrownewRuntimeException("invalid utf8 byte: byte="+b0);}publicstaticintUtf16_Encode_int(intc,byte[]src,intpos){if((c>-1)&&(c<128)){src[pos]=(byte)c;return1;}elseif(c<2048){src[pos]=(byte)(0xC0|(c>>6));src[++pos]=(byte)(0x80|(c&0x3F));return2;}elseif(c<65536){src[pos]=(byte)(0xE0|(c>>12));src[++pos]=(byte)(0x80|(c>>6)&0x3F);src[++pos]=(byte)(0x80|(c&0x3F));return3;}elseif(c<2097152){src[pos]=(byte)(0xF0|(c>>18));src[++pos]=(byte)(0x80|(c>>12)&0x3F);src[++pos]=(byte)(0x80|(c>>6)&0x3F);src[++pos]=(byte)(0x80|(c&0x3F));return4;}elsethrownewRuntimeException("UTF-16 int must be between 0 and 2097152; char="+c);}publicstaticintUtf16_Encode_char(intc,char[]c_ary,intc_pos,byte[]b_ary,intb_pos){if((c>-1)&&(c<128)){b_ary[b_pos]=(byte)c;return1;}elseif(c<2048){b_ary[b_pos]=(byte)(0xC0|(c>>6));b_ary[++b_pos]=(byte)(0x80|(c&0x3F));return2;}elseif((c>55295)// 0xD800&&(c<56320)){// 0xDFFFif(c_pos>=c_ary.length)thrownewRuntimeException("incomplete surrogate pair at end of string; char="+c);intnxt_char=c_ary[c_pos+1];intv=Utf16_Surrogate_merge(c,nxt_char);b_ary[b_pos]=(byte)(0xF0|(v>>18));b_ary[++b_pos]=(byte)(0x80|(v>>12)&0x3F);b_ary[++b_pos]=(byte)(0x80|(v>>6)&0x3F);b_ary[++b_pos]=(byte)(0x80|(v&0x3F));return4;}else{b_ary[b_pos]=(byte)(0xE0|(c>>12));b_ary[++b_pos]=(byte)(0x80|(c>>6)&0x3F);b_ary[++b_pos]=(byte)(0x80|(c&0x3F));return3;}}privatestaticintUtf16_Surrogate_merge(inthi,intlo){// REF: http://perldoc.perl.org/Encode/Unicode.htmlreturn0x10000+(hi-0xD800)*0x400+(lo-0xDC00);}
if(c>UCHAR_MAX)lexerror("escape sequence too large",TK_STRING);save(c,false);// NOTE: specify that c is integer and does not need conversion; EX: \128 -> 128 -> (char)128, not Utf8_encode(128)
voidsave(intc){save(c,true);}voidsave(intc,booleanc_might_be_utf8){intbytes_len=c_might_be_utf8?LuaString.Utf8_Len_of_char_by_1st_byte((byte)c):1;if(bytes_len>1){// c is 1st byte of utf8 multi-byte sequence; read required number of bytes and convert to char; EX: left-arrow is serialized in z as 226,134,144; c is currently 226; read 134 and 144 and convert to left-arrowtemp_bry[0]=(byte)c;for(inti=1;i<bytes_len;i++){nextChar();temp_bry[i]=(byte)current;}c=LuaString.Utf16_Decode_to_int(temp_bry,0);}if(buff==null||nbuff+1>buff.length)buff=LuaC.realloc(buff,nbuff*2+1);buff[nbuff++]=(char)c;}privatestaticbyte[]temp_bry=newbyte[6];
Last edit: gnosygnu 2014-05-07
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi! I'm the developer for XOWA -- an offline Wikipedia application. About a month ago, I changed XOWA to use Luaj to process the lua statements in Wikipedia's offline dumps. I'd just like to say how impressed I was in Luaj's functionality and performance. So far, it handles all the lua code in 30+ wikis, including en.wikipedia.org, en.wiktionary.org, etc..
In the process of integrating luaj into XOWA, I came across a minor number of issues. I don't know if this board is still active, but I wanted to list them below. Admittedly, the format is unstructured. If there is a formal process for submitting patches to Luaj, let me know, and I will follow up accordingly.
You can also download XOWA's modified source of luaj here: https://sourceforge.net/projects/xowa/files/support/luaj/v1/
Thanks for developing such a great library!
Last edit: gnosygnu 2014-05-07