Re: [Modeling-users] working in unicode?

[Yannick G. <ygi...@yg...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sunday 20 April 2003 08:58, Sebastien Bigaret wrote:
> latin-1?  No, I never assumed a particular encoding for strings. It
> can be that the framework does not behave correctly because of the
> unicode type, but if this happens this is definitely a bug, it was
> not intended.

We use unicode all the time.  The 1st time you create and object, if
you don't pass ASCII only data to an attibute of an object, you have
to explicitly encode it with the encoding of your choice.

We use MySQL which does not support unicode at all.  We have do our
encoding explicitly but you may have more luck with a unicode DB like
postgres.  Encoding explicitly has a few draw backs, we can't do a
case insensitive search and we can't be sure of the length of the
encoded string before we encode it.  You have to plan larger fields in
your model.  Three times larger than the intended length is safe in most
situations.  Latin-1 to UTF8 is about 1.5 time larger but cyrillic
clip near 2.5.  Do you own test with a reliable sample of data, UTF8
may take up to 8 bytes per characters:

   print len(u'\u58f7\u58d9\u58bc\u585e\u5859'.encode("utf-8"))

Typing the hex values is boring I made a PyQt app in which I can paste
characters I copied from kcharselect : 

#!/usr/bin/python
from autogen.UniToPyGui import UniToPyGui
from qt import *
import sys

class UniToPyDialog(UniToPyGui):
    def __init__(self, *args):
        UniToPyGui.__init__(self, *args)
        self.connect(self.okBtn, SIGNAL("clicked()"), self.convertString)

    def convertString(self):
        unistr = unicode(self.uniTxt.text())
        self.uniSizeTxt.setText(str(len(unistr)))
        
        pystr = repr(unistr)
        self.pyTxt.setText(pystr)
        self.pySizeTxt.setText(str(len(pystr)))
        
        utfstr = unistr.encode("utf-8")
        self.utfTxt.setText(utfstr)
        self.utfSizeTxt.setText(str(len(utfstr)))
    

if __name__ == "__main__":
    qapp = QApplication(sys.argv)
    dialog = UniToPyDialog()
    qapp.setMainWidget(dialog)
    dialog.show()
    qapp.exec_loop()

#           The END !

All you need is a UniToPyGui form with a OK button, an a few text edits.

- -- 
Yannick Gingras
Coder for OBB : Onymous Barrelled Baneberry
http://OpenBeatBox.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)

iD8DBQE+oqZCrhy5Fqn/MRARAqeBAJ0ZaDCwUEKpNSITQZI8gRhj3F38/QCcCO5l
N2l/G7zy4NUhhXk6CzFreyc=
=u0lD
-----END PGP SIGNATURE-----