From: James R E. <ea...@ba...> - 2005-11-17 15:59:13
|
Hi folks, Let me start off by thanking y'all for all of the great effort into Jython. It's a dream come true to be able to use python and java together! I'm trying to pickle some unicode strings, but the pickler (both cPickle and pickle) raise Errors. With cPickle, I get: >>> pickle.dumps(foo) Traceback (innermost last): File "<console>", line 1, in ? UnpickleableError: Cannot pickle Alors vous imaginez ma surprise, au lever du jour, quand une dr?le de petit voix m?a r?veill?. Elle disait: ? S?il vous pla?t? dessine-moi un mouton! ? objects (Note that there are two bugs going on: unicode should be a pickleable type, and the error raised should say "Cannot pickle unicode objects" instead of the string repr.) From looking at the jython-2.2a1 source, the exception is raised at cPickle.java line 925, but I'm not familiar enough with the internals of jython or __reduce__ to fix it myself. Can anyone suggest a workaround? As a temporary measure, I can get by with dropping the unicode data to ascii, but for some reason I'm not able to do so. Am I doing something wrong here, or is jython?: >>> foo.encode('ascii', 'replace') Traceback (innermost last): File "<console>", line 1, in ? UnicodeError: ascii encoding error: ordinal not in range(128) Thanks for your help! James |
From: Alan K. <jyt...@xh...> - 2005-11-17 16:19:15
|
[James R Eagan] > Let me start off by thanking y'all for all of the great effort into > Jython. It's a dream come true to be able to use python and java together! > > I'm trying to pickle some unicode strings, but the pickler (both cPickle > and pickle) raise Errors. With cPickle, I get: > > >>> pickle.dumps(foo) > Traceback (innermost last): > File "<console>", line 1, in ? > UnpickleableError: Cannot pickle Alors vous imaginez ma surprise, au > lever du jour, quand une dr?le de petit voix m?a r?veill?. Elle disait: > ? S?il vous pla?t? dessine-moi un mouton! ? > objects James, Can you show us what the definition of "foo" is? Here is a transcript of an interactive session with a jython 2.1 interpreter on Windows 2K3, with the terminal encoding as cp850. x:\>chcp Active code page: 850 x:\>jython Jython 2.1 on java1.4.2_09 (JIT: null) Type "copyright", "credits" or "license" for more information. >>> import pickle >>> pickle.dumps(u"áéíóú") 'V\xA0\\u201A\xA1\xA2\xA3\np0\n.' >>> import cPickle >>> cPickle.dumps(u"áéíóú") 'V\xA0\\u201A\xA1\xA2\xA3\np0\n.' >>> cPickle.loads(_) u'\xA0\u201A\xA1\xA2\xA3' >>> Can you create a complete script that illustrates the problem and post it here for us? Regards, Alan. |
From: James R E. <ea...@ba...> - 2005-11-17 16:44:29
|
Wow, thanks for the quick response! This is on Mac OS X Panther, for =20= what it's worth. I fetch a unicode string from a url, but get back a =20= plain string, which is pickleable. I then convert that string to it's =20= utf8 unicode equivalent, where I run into the problem. In this demo, =20= I'm converting from a str to unicode object before I pickle, but in my =20= real usage, I don't have the option of pickling the str and encoding it =20= as a unicode string upon unpickling. Here's the transcript: pasatiempo : ~ $ jython2.2 Jython 2.2a1 on java1.4.2_05 (JIT: null) Type "copyright", "credits" or "license" for more information. >>> import sys, urllib, cPickle as pickle >>> sys.getdefaultencoding() 'ascii' >>> foo =3D =20 urllib.urlopen('http://stuff.vandervossen.net/external/2003/=20 petit.utf8').read() >>> foo 'Alors vous imaginez ma surprise, au lever du jour, quand une =20 dr\xC3\xB4le de petit voix m\xE2\x80\x99a r\xC3\xA9veill\xC3\xA9. Elle =20= disait: \xC2\xAB S\xE2\x80\x99il vous pla\xC3\xAEt\xE2\x80\xA6 =20 dessine-moi un mouton! \xC2\xBB\n' >>> pickle.dumps(foo) "S'Alors vous imaginez ma surprise, au lever du jour, quand une =20 dr\\xC3\\xB4le de petit voix m\\xE2\\x80\\x99a =20 r\\xC3\\xA9veill\\xC3\\xA9. Elle disait: \\xC2\\xAB S\\xE2\\x80\\x99il =20= vous pla\\xC3\\xAEt\\xE2\\x80\\xA6 dessine-moi un mouton! =20 \\xC2\\xBB\\n'\np1\n." >>> foo =3D unicode(foo, 'utf8') >>> foo u'Alors vous imaginez ma surprise, au lever du jour, quand une dr\xF4le =20= de petit voix m\u2019a r\xE9veill\xE9. Elle disait: \xAB S\u2019il vous =20= pla\xEEt\u2026 dessine-moi un mouton! \xBB\n' >>> pickle.dumps(foo) Traceback (innermost last): File "<console>", line 1, in ? UnpickleableError: Cannot pickle Alors vous imaginez ma surprise, au =20 lever du jour, quand une dr?le de petit voix m?a r?veill?. Elle disait: =20= ? S?il vous pla?t? dessine-moi un mouton! ? objects On Nov 17, 2005, at 11:18 AM, Alan Kennedy wrote: > Can you show us what the definition of "foo" is? > > Here is a transcript of an interactive session with a jython 2.1 =20 > interpreter on Windows 2K3, with the terminal encoding as cp850. > > x:\>chcp > Active code page: 850 > > x:\>jython > Jython 2.1 on java1.4.2_09 (JIT: null) > Type "copyright", "credits" or "license" for more information. > >>> import pickle > >>> pickle.dumps(u"=E1=E9=ED=F3=FA") > 'V\xA0\\u201A\xA1\xA2\xA3\np0\n.' > >>> import cPickle > >>> cPickle.dumps(u"=E1=E9=ED=F3=FA") > 'V\xA0\\u201A\xA1\xA2\xA3\np0\n.' > >>> cPickle.loads(_) > u'\xA0\u201A\xA1\xA2\xA3' > >>> > > Can you create a complete script that illustrates the problem and post = =20 > it here for us? > > Regards, > > Alan. > > > > ------------------------------------------------------- > This SF.Net email is sponsored by the JBoss Inc. Get Certified Today > Register for a JBoss Training Course. Free Certification Exam > for All Training Attendees Through End of 2005. For more info visit: > http://ads.osdn.com/?ad_id=3D7628&alloc_id=3D16845&op=3Dclick > _______________________________________________ > Jython-users mailing list > Jyt...@li... > https://lists.sourceforge.net/lists/listinfo/jython-users > |
From: Alan K. <jyt...@xh...> - 2005-11-17 18:23:58
|
[James R Eagan] > $ jython2.2 > Jython 2.2a1 on java1.4.2_05 (JIT: null) As Frank pointed out in another mail, pickle on 2.2a1 has known problems. Do you have a specific need for jython 2.2a1? Is there any reason not to use jython 2.1? Regards, Alan. |
From: James R E. <ea...@ba...> - 2005-11-17 18:34:03
|
So it seems. Oh well. Sadly, I -do- need jython 2.2 for various libraries I'm using. My temporary solution has been to omit support for unicode strings for now. Whenever I get a unicode string, I do the following (it's horribly ugly, but it seems to work): def _encoder(ustr): ''' Convert a unicode string to a utf8 plain string. ''' try: return ustr.encode('utf8', 'replace') except UnicodeError: # This should not be necessary, but it seems to be :-( def _replace(ch): if 0 <= ord(ch) <= 127: return ch return '?' return reduce(lambda s, ch: s + _replace(ch), ustr, '') On Nov 17, 2005, at 1:23 PM, Alan Kennedy wrote: > [James R Eagan] >> $ jython2.2 >> Jython 2.2a1 on java1.4.2_05 (JIT: null) > > As Frank pointed out in another mail, pickle on 2.2a1 has known > problems. > > Do you have a specific need for jython 2.2a1? Is there any reason not > to use jython 2.1? > > Regards, > > Alan. > > > > ------------------------------------------------------- > This SF.Net email is sponsored by the JBoss Inc. Get Certified Today > Register for a JBoss Training Course. Free Certification Exam > for All Training Attendees Through End of 2005. For more info visit: > http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click > _______________________________________________ > Jython-users mailing list > Jyt...@li... > https://lists.sourceforge.net/lists/listinfo/jython-users > |
From: Frank W. <fwi...@gm...> - 2005-11-17 16:21:42
|
> From looking at the jython-2.2a1 source, the exception is raised at > cPickle.java line 925, but I'm not familiar enough with the internals > of jython or __reduce__ to fix it myself. Pickling in the alpha version of Jython is known to be broken. I'm not sure when this is going to get fixed since I'm not even sure how much work it will be to get it unbroken... I wish I could give you better news. -Frank |