Re: [cx-oracle-users] Unicode support

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

On Fri, Oct 17, 2008 at 2:59 AM, Sébastien Sablé
<sa...@us...> wrote:
> Hi Anthony,
>
> thank you for those improvements!

You're welcome.

> I don't expect to work again on Unicode in my application before a few
> weeks; but when I will, I will give you some feedback whether it works
> or not.

Thanks.

> For your information, I made a prototype of my application running in
> Unicode (with some Chinese text) for a demo which worked great. I used
> cx_Oracle revision 142 with a database encoded in AL32UTF8 (AL16UTF16
> was working fine also; UTF-8 was not).

Please try again when you have a moment with the latest code. I
believe UTF-8 encoded databases should work fine now as well but I'd
appreciate confirmation of that fact. Thanks.

> regards
>
> --
> Sébastien Sablé
>
>
>
> Anthony Tuininga a écrit :
>> Hi Sebastien,
>>
>> I just committed changes to use byte semantics instead of character
>> semantics and all appears to be well -- but I'm not sure completely so
>> I'd appreciate some testing. Note that I am also in the process of
>> adding "full" unicode support in preparation for a port to Python 3.0.
>> This implies that all strings are both expected in unicode and will be
>> returned in unicode -- but its not complete yet. The test cases
>> currently pass but there may still be some holes. Please advise if you
>> find one. :-)
>>
>> Anthony
>>
>> On Mon, Oct 6, 2008 at 10:21 AM, Sébastien Sablé
>> <sa...@us...> wrote:
>>> Hi Anthony,
>>>
>>> I did the test with a database using AL16UTF16 and it works fine.
>>> So I guess the problem only happens when using variable width character
>>> set. I will stick with AL16UTF16 for the moment.
>>>
>>> Thanks for your help
>>>
>>> --
>>> Sébastien Sablé
>>>
>>>
>>>
>>> Anthony Tuininga a écrit :
>>>> Hmm, do you have access to a database with a server side fixed width
>>>> character set? If not, I'll have to try creating a database with a
>>>> variable width character set like UTF8 and try it -- but my time is
>>>> stretched right now so it will have to wait if you want me to look
>>>> into it. :-)
>>>>
>>>> Anthony
>>>>
>>>> On Thu, Oct 2, 2008 at 10:50 AM, Sébastien Sablé
>>>> <sa...@us...> wrote:
>>>>> Hi Anthony,
>>>>>
>>>>> I use a 10.2.0.1.0 client on linux, connecting to a 10.2.0.3.0 server on
>>>>> sunos 10.
>>>>>
>>>>> Here are the parameters in my database:
>>>>>
>>>>> $ sqlplus SSA_UTF8/sungard@UTF102NP
>>>>>
>>>>> SQL*Plus: Release 10.2.0.1.0 - Production on Thu Oct 2 18:38:43 2008
>>>>>
>>>>> Copyright (c) 1982, 2005, Oracle.  All rights reserved.
>>>>>
>>>>>
>>>>> Connected to:
>>>>> Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
>>>>> With the Partitioning, OLAP and Data Mining options
>>>>>
>>>>> SQL> select * from nls_database_parameters where instr(parameter,
>>>>> 'CHARACTERSET') != 0;
>>>>>   2
>>>>> PARAMETER
>>>>> ------------------------------
>>>>> VALUE
>>>>> --------------------------------------------------------------------------------
>>>>> NLS_CHARACTERSET
>>>>> UTF8
>>>>>
>>>>> NLS_NCHAR_CHARACTERSET
>>>>> UTF8
>>>>>
>>>>>
>>>>> So this may be linked to the different charset being used.
>>>>> regards
>>>>>
>>>>> --
>>>>> Sébastien Sablé
>>>>>
>>>>>
>>>>> Anthony Tuininga a écrit :
>>>>>> Hi,
>>>>>>
>>>>>> I tried the code you provided as is and it worked for me. :-) I
>>>>>> checked the error message and the cause claims to be that you cannot
>>>>>> use OCI_ATTR_CHAR_COUNT when the server or client character set is
>>>>>> variable length. I was using a fixed length character set so I tried a
>>>>>> variable length character set but I still didn't get an error. I am
>>>>>> using the Oracle 10.2.0.3 instant client on Windows and the full
>>>>>> Oracle 11.1.0.6 client on Linux. Neither of these seem to have any
>>>>>> difficulty. Can anyone else tell me if they are getting the same
>>>>>> behavior? Please specify the Oracle client being used and the value
>>>>>> for connection.encoding and connection.encoding as well as the results
>>>>>> of
>>>>>>
>>>>>> select * from nls_database_parameters where instr(parameter,
>>>>>> 'CHARACTERSET') != 0;
>>>>>>
>>>>>> For example, with the Windows instant client:
>>>>>>
>>>>>> Oracle Instant client version 10.2.0.3
>>>>>> connection.encoding = "WINDOWS-1252"
>>>>>> connection.nencoding = "WINDOWS-1252"
>>>>>>
>>>>>> PARAMETER                      VALUE
>>>>>> ------------------------------ ----------------------------------------
>>>>>> NLS_NCHAR_CHARACTERSET         AL16UTF16
>>>>>> NLS_CHARACTERSET               WE8MSWIN1252
>>>>>>
>>>>>> Anthony
>>>>>>
>>>>>> On Thu, Oct 2, 2008 at 6:55 AM, Sébastien Sablé
>>>>>> <sa...@us...> wrote:
>>>>>>> Hi Anthony,
>>>>>>>
>>>>>>> thank you for this evolution of cx_Oracle!
>>>>>>>
>>>>>>> It tested it and it seems to work fine for output, however when doing
>>>>>>> some input I always get the same kind of error:
>>>>>>>
>>>>>>> $ python test_oracle.py
>>>>>>> Traceback (most recent call last):
>>>>>>>   File "test_oracle.py", line 18, in <module>
>>>>>>>     cursor.execute('insert into personne values(:0, :1)', [u'abc', u'def'])
>>>>>>> cx_Oracle.DatabaseError: ORA-24363: measurements in characters illegal here
>>>>>>>
>>>>>>> Here is the trivial test I ran:
>>>>>>>
>>>>>>> import cx_Oracle
>>>>>>>
>>>>>>> def OutputTypeHandler(cursor, name, defaultType, size, precision, scale):
>>>>>>>     if defaultType in (cx_Oracle.STRING, cx_Oracle.FIXED_CHAR):
>>>>>>>         return cursor.var(unicode, size, cursor.arraysize)
>>>>>>>
>>>>>>> connection = cx_Oracle.Connection('SSA_UTF8/sungard@UTF102NP')
>>>>>>> connection.outputtypehandler = OutputTypeHandler
>>>>>>> cursor = connection.cursor()
>>>>>>>
>>>>>>> try:
>>>>>>>     cursor.execute('drop table personne')
>>>>>>> except cx_Oracle.DatabaseError:
>>>>>>>     pass
>>>>>>>
>>>>>>> cursor.execute('create table personne (name nchar(4), first_name nchar(6))')
>>>>>>>
>>>>>>> cursor.execute('insert into personne values(:0, :1)', [u'abc', u'def'])
>>>>>>>
>>>>>>> cursor.execute("select * from personne")
>>>>>>> for row in cursor:
>>>>>>>     print "Row:", row
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Is there something to configure in order to allow unicode strings input?
>>>>>>> Thanks in advance
>>>>>>>
>>>>>>> --
>>>>>>> Sébastien Sablé
>>>>>>>
>>>>>>>
>>>>>>> Anthony Tuininga a écrit :
>>>>>>>> All,
>>>>>>>>
>>>>>>>> I just checked in to Subversion support for Unicode. Thanks go to
>>>>>>>> Amaury Forgeot d'Arc for the patch he provided earlier this year and
>>>>>>>> to everyone else for their patience. :-) Since I have little
>>>>>>>> experience with languages other than English and I only have access to
>>>>>>>> little endian machines I'd appreciate those of you with such
>>>>>>>> experience or machines to test the new code for me and let me know the
>>>>>>>> results.
>>>>>>>>
>>>>>>>> Note that now nchar, nvarchar2 data are now returned as unicode. It is
>>>>>>>> also possible to bind unicode strings in to cursors. SQL statements
>>>>>>>> and varchar2 and char data are still returned as encoded strings. Note
>>>>>>>> that with another new feature in cx_Oracle you can force all strings
>>>>>>>> to be returned as unicode if you so desire. Below is an example that
>>>>>>>> will run with the data in the test suite:
>>>>>>>>
>>>>>>>> import cx_Oracle
>>>>>>>>
>>>>>>>> def OutputTypeHandler(cursor, name, defaultType, size, precision, scale):
>>>>>>>>     if defaultType in (cx_Oracle.STRING, cx_Oracle.FIXED_CHAR):
>>>>>>>>         return cursor.var(unicode, size, cursor.arraysize)
>>>>>>>>
>>>>>>>> connection = cx_Oracle.Connection("cx_Oracle/dev@eadev")
>>>>>>>> connection.outputtypehandler = OutputTypeHandler
>>>>>>>> cursor = connection.cursor()
>>>>>>>> cursor.execute("select * from teststrings")
>>>>>>>> for row in cursor:
>>>>>>>>     print "Row:", row
>>>>>>>>
>>>>>>>> If you have questions, let me know. If you have feedback, please
>>>>>>>> provide it. Thanks.
>>>>>>>>
>>>>>>>> Anthony
>>>>>>>>
>>>>>>>> -------------------------------------------------------------------------
>>>>>>>> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
>>>>>>>> Build the coolest Linux based applications with Moblin SDK & win great prizes
>>>>>>>> Grand prize is a trip for two to an Open Source event anywhere in the world
>>>>>>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>>>>>>>> _______________________________________________
>>>>>>>> cx-oracle-users mailing list
>>>>>>>> cx-...@li...
>>>>>>>> https://lists.sourceforge.net/lists/listinfo/cx-oracle-users
>>>>>>> -------------------------------------------------------------------------
>>>>>>> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
>>>>>>> Build the coolest Linux based applications with Moblin SDK & win great prizes
>>>>>>> Grand prize is a trip for two to an Open Source event anywhere in the world
>>>>>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>>>>>>> _______________________________________________
>>>>>>> cx-oracle-users mailing list
>>>>>>> cx-...@li...
>>>>>>> https://lists.sourceforge.net/lists/listinfo/cx-oracle-users
>>>>>>>
>>>>>> -------------------------------------------------------------------------
>>>>>> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
>>>>>> Build the coolest Linux based applications with Moblin SDK & win great prizes
>>>>>> Grand prize is a trip for two to an Open Source event anywhere in the world
>>>>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>>>>>> _______________________________________________
>>>>>> cx-oracle-users mailing list
>>>>>> cx-...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/cx-oracle-users
>>>>> -------------------------------------------------------------------------
>>>>> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
>>>>> Build the coolest Linux based applications with Moblin SDK & win great prizes
>>>>> Grand prize is a trip for two to an Open Source event anywhere in the world
>>>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>>>>> _______________________________________________
>>>>> cx-oracle-users mailing list
>>>>> cx-...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/cx-oracle-users
>>>>>
>>>> -------------------------------------------------------------------------
>>>> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
>>>> Build the coolest Linux based applications with Moblin SDK & win great prizes
>>>> Grand prize is a trip for two to an Open Source event anywhere in the world
>>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>>>> _______________________________________________
>>>> cx-oracle-users mailing list
>>>> cx-...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/cx-oracle-users
>>>
>>> -------------------------------------------------------------------------
>>> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
>>> Build the coolest Linux based applications with Moblin SDK & win great prizes
>>> Grand prize is a trip for two to an Open Source event anywhere in the world
>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>>> _______________________________________________
>>> cx-oracle-users mailing list
>>> cx-...@li...
>>> https://lists.sourceforge.net/lists/listinfo/cx-oracle-users
>>>
>>
>> -------------------------------------------------------------------------
>> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
>> Build the coolest Linux based applications with Moblin SDK & win great prizes
>> Grand prize is a trip for two to an Open Source event anywhere in the world
>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>> _______________________________________________
>> cx-oracle-users mailing list
>> cx-...@li...
>> https://lists.sourceforge.net/lists/listinfo/cx-oracle-users
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> cx-oracle-users mailing list
> cx-...@li...
> https://lists.sourceforge.net/lists/listinfo/cx-oracle-users
>