Re: [CLucene-dev] Question about CLucene's internal character representation.

[Rene R. <ren...@ya...>]

Wohoo rewrite the analyzers, yeah that would be alot of fun :)=0A=0AActuall=
y I played around with ICU conversion API's yesterday, and it comes out=0At=
hat on my platform, x86 Fedora Code 6, gcc doesn't use UCS-4 but instead=0A=
it uses UTF-32LE :) So right now everything happens to be working.=0ANow I =
just need to find out what Windows XP uses, I think it was UTF-16 with BOM'=
s (Byte=0AOrder Mark's).=0A=0AAnyway thanks for the help and happy new year=
 to everyone :)=0A=0A> ----- Original Message ----=0A> From: Ben van Klinke=
n <bva...@gm...>=0A> To: clu...@li...=
=0A> Sent: Thursday, December 28, 2006 10:36:37 PM=0A> Subject: Re: [CLucen=
e-dev] Question about CLucene's internal character representation.=0A> =0A>=
 you could always do everything in utf8 and re-write the analyzers to=0A> h=
andle utf8 ;-)=0A> =0A> ben=0A> =0A> On 28/12/06, Rene Rattur <renerattur@y=
ahoo.com> wrote:=0A>=0A> Looks like I'm gonna be having some fun with ICU.=
=0A> Thanks.=0A>=0A> >----- Original Message ----=0A> >From: Ben van Klinke=
n <bva...@gm...>=0A> >To: clu...@li...=
=0A> >Sent: Thursday, December 28, 2006 2:12:51 PM=0A> >Subject: Re: [CLuce=
ne-dev] Question about CLucene's internal character=0A> representation.=0A>=
 >=0A> >Yes, but are you sure all linux machines use UCS-4? I think some=0A=
> >distro versions/GCC versions used UCS-2.=0A> >ben=0A> >=0A> >On 28/12/06=
, Rene Rattur <ren...@ya...> wrote:=0A> >=0A> > Hi Ben,=0A> >=0A> >=
 On linux wchar_t is 4 bytes,=0A> > so when I pass const TChar* string to C=
Lucene, what encoding should it be=0A> in=0A> > ?=0A> > On windows UTF-16, =
on linux UCS-4 ???=0A> >=0A> >=0A> > ----- Original Message ----=0A> > From=
: Ben van Klinken <bva...@gm...>=0A> > To: clucene-developers@list=
s.sourceforge.net=0A> > Sent: Wednesday, December 27, 2006 5:59:07 PM=0A> >=
 Subject: Re: [CLucene-dev] Question about CLucene's internal character=0A>=
 > representation.=0A> >=0A> > Rene,=0A> >=0A> > It uses the wchar_t type. =
So whatever your platform defines that as,=0A> > is what we use. I know for=
 windows + msvc it uses ucs2, but i can't=0A> > speak for everything. We su=
pport unicode, so 2 bytes doesn't cover the=0A> > entire unicode character-=
set.=0A> >=0A> > ben=0A> >=0A> > On 27/12/06, Rene Rattur <renerattur@yahoo=
.com> wrote:=0A> > >=0A> > > Hi,=0A> > >=0A> > > I was wondering what's the=
 internal character format of CLucene?=0A> > > Is it UCS-2 or UCS-4, and if=
 it's UCS-4 isn't it kinda overkill, to=0A> store=0A> > a=0A> > > 2-byte co=
deunit=0A> > > in a 4-byte datatype ?=0A> > >=0A> > > _____________________=
_____________________________=0A> > > Do You Yahoo!?=0A> > > Tired of spam?=
 Yahoo! Mail has the best spam protection around=0A> > > http://mail.yahoo.=
com=0A> > >=0A> >=0A> -----------------------------------------------------=
--------------------=0A> > > Take Surveys. Earn Cash. Influence the Future =
of IT=0A> > > Join SourceForge.net's Techsay panel and you'll get the chanc=
e to share=0A> > your=0A> > > opinions on IT & business topics through brie=
f surveys - and earn cash=0A> > >=0A> >=0A> http://www.techsay.com/default.=
php?page=3Djoin.php&p=3Dsourceforge&CID=3DDEVDEV=0A> > >=0A> > > __________=
_____________________________________=0A> > > CLucene-developers mailing li=
st=0A> > > CLu...@li...=0A> > >=0A> >=0A> https=
://lists.sourceforge.net/lists/listinfo/clucene-developers=0A> > >=0A> > >=
=0A> > >=0A> >=0A> >=0A> --------------------------------------------------=
-----------------------=0A> > Take Surveys. Earn Cash. Influence the Future=
 of IT=0A> > Join SourceForge.net's Techsay panel and you'll get the chance=
 to share=0A> your=0A> > opinions on IT & business topics through brief sur=
veys - and earn cash=0A> >=0A> http://www.techsay.com/default.php?page=3Djo=
in.php&p=3Dsourceforge&CID=3DDEVDEV=0A> > _________________________________=
______________=0A> > CLucene-developers mailing list=0A> > CLucene-develope=
rs...@li...=0A> >=0A> https://lists.sourceforge.net/lists/list=
info/clucene-developers=0A> >=0A> >=0A> > _________________________________=
_________________=0A> > Do You Yahoo!?=0A> > Tired of spam? Yahoo! Mail has=
 the best spam protection around=0A> > http://mail.yahoo.com=0A> >=0A> ----=
---------------------------------------------------------------------=0A> >=
 Take Surveys. Earn Cash. Influence the Future of IT=0A> > Join SourceForge=
.net's Techsay panel and you'll get the chance to share=0A> your=0A> > opin=
ions on IT & business topics through brief surveys - and earn cash=0A> >=0A=
> http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D=
DEVDEV=0A> >=0A> > _______________________________________________=0A> > CL=
ucene-developers mailing list=0A> > CLu...@li...=
t=0A> >=0A> https://lists.sourceforge.net/lists/listinfo/clucene-developers=
=0A> >=0A> >=0A> >=0A>=0A> ------------------------------------------------=
-------------------------=0A> Take Surveys. Earn Cash. Influence the Future=
 of IT=0A> Join SourceForge.net's Techsay panel and you'll get the chance t=
o share your=0A> opinions on IT & business topics through brief surveys - a=
nd earn cash=0A> http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsou=
rceforge&CID=3DDEVDEV=0A> _______________________________________________=
=0A> CLucene-developers mailing list=0A> CLu...@li...urcefor=
ge.net=0A> https://lists.sourceforge.net/lists/listinfo/clucene-developers=
=0A>=0A>=0A> __________________________________________________=0A> Do You =
Yahoo!?=0A> Tired of spam? Yahoo! Mail has the best spam protection around=
=0A> http://mail.yahoo.com=0A> --------------------------------------------=
-----------------------------=0A> Take Surveys. Earn Cash. Influence the Fu=
ture of IT=0A> Join SourceForge.net's Techsay panel and you'll get the chan=
ce to share your=0A> opinions on IT & business topics through brief surveys=
 - and earn cash=0A> http://www.techsay.com/default.php?page=3Djoin.php&p=
=3Dsourceforge&CID=3DDEVDEV=0A>=0A> _______________________________________=
________=0A> CLucene-developers mailing list=0A> CLucene-developers@lists.s=
ourceforge.net=0A> https://lists.sourceforge.net/lists/listinfo/clucene-dev=
elopers=0A>=0A>=0A>=0A> =0A> ----------------------------------------------=
---------------------------=0A> Take Surveys. Earn Cash. Influence the Futu=
re of IT=0A> Join SourceForge.net's Techsay panel and you'll get the chance=
 to share your=0A> opinions on IT & business topics through brief surveys -=
 and earn cash=0A> http://www.techsay.com/default.php?page=3Djoin.php&p=3Ds=
ourceforge&CID=3DDEVDEV=0A> _______________________________________________=
=0A> CLucene-developers mailing list=0A> CLu...@li...urcefor=
ge.net=0A> https://lists.sourceforge.net/lists/listinfo/clucene-developers=
=0A=0A=0A> =0A=0A=0A__________________________________________________=0ADo=
 You Yahoo!?=0ATired of spam?  Yahoo! Mail has the best spam protection aro=
und =0Ahttp://mail.yahoo.com