From: Fernando M. <fer...@cm...> - 2013-08-28 21:53:22
|
sorry, forgot to reply to list, see below On 08/28/2013 10:43 PM, Jim Baker wrote: > ... > # -*- coding: utf-8 -*- > > from java.io <http://java.io> import File, FileOutputStream > from contextlib import closing > > x = u"首页" > with closing(FileOutputStream(File("output.txt"))) as f: > f.write("<b>%s</b>" % (x,)) # FileOutputStream takes bytes... > f.write("<b>%s</b>" % (x.encode("UTF-8"),)) > > This results in the following output: > > <b>??</b><b>首页</b> > > Interestingly, if you were to change the above to x = "首页" (or > equivalently in 2.7, x = b"首页"), you would setting x to a bytestring > with the underlying bytes in UTF-8 (because that's what the source > encoding was, we said so in the code) and you would get this output: > > <b>首页</b><b>首页</b> > > The moral here is that you need to carefully track your encodings. Thanks Jim, I'm using jython 2.5 and this works: x = "José" toClient.println("<li>%s</li>" % x) In my example, row[0] is of type PyUnicode and the db uses encoding utf8. So, some part of the pipelining is changing the encoding. After a search, I got to http://old.nabble.com/Re%3A-On-String,-PyString-and-PyUnicode-p18767753.html where you state that PyUnicode uses utf-16. I can't test now, but I guess .encode('utf-8') should do the trick, right? And I don't suppose there is a way to avoid this extra conversion? BTW, what is the state of jython 2.7 bugwise? Regards, Fernando |
From: Jim B. <jb...@zy...> - 2013-08-28 22:39:26
|
On Wed, Aug 28, 2013 at 3:53 PM, Fernando Martins <fer...@cm...>wrote: > .... > > > I'm using jython 2.5 and this works: > > x = "José" > toClient.println("<li>%s</li>" % x) > > In my example, row[0] is of type PyUnicode and the db uses encoding utf8. > So, some part of the pipelining is changing the encoding. After a search, I > got to > > > http://old.nabble.com/Re%3A-On-String,-PyString-and-PyUnicode-p18767753.html > > where you state that PyUnicode uses utf-16. I can't test now, but I guess > .encode('utf-8') should do the trick, right? And I don't suppose there is a > way to avoid this extra conversion? > Correct, this conversion is really required. Sometimes you may be working with input stream/output streams that are character-oriented, but not in this case. But it's just a simple matter of .encode("UTF-8") The fact that the internal representation of java.lang.String, as wrapped by PyUnicode, is UTF-16 is usually sort of irrelevant here. But occasionally we do see the effect of the representation in corner cases, such as in this test_bad_unicode script: x = u"Some illegal codepoints: \uD800 or \uD801 or ... or \uDBFF" print x $ python test_bad_unicode.py Some illegal codepoints: � or � or ... or � $ jython27 test_bad_unicode.py UnicodeDecodeError: 'unicodeescape' codec can't decode bytes in position 27-33: illegal Unicode character Now there are Python modules out there in pypi that use such illegal codepoints, although they tend to be rare because this requires a fairly sophisticated knowledge of how to abuse unicode. This includes such favorites as pip: pip/vendor/html5lib/inputstream.py 168: self.replaceCharactersRegexp = re.compile("([\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF])") We need to send them a pull request, although working on other things to support pip first; see below: > > BTW, what is the state of jython 2.7 bugwise? > Working hard on it! ;) There are a bunch of little things that need to be fixed. There are 293 FIXME tests out of ~5000 tests in the test suite right now, with skips there because regrtest can fail to complete otherwise. Some of these FIXMEs we still probably won't fix this time around, despite their value, such as codecs for encodings like big5 and shift_jis, especially given there are workarounds for these issues. This also doesn't count some new functionality that we will be delivering such as ssl support; for what it's worth, I recently pushed up a branch of an experiment on this that allows me to use easy_install ( https://bitbucket.org/jimbaker/jython-ssl); pip still fails because of some Unicode issue and the lack of os.O_NOFOLLOW, something we could readily fix if we restrict 2.7 to Java 7 (as I've started to advocate on #jython, given the recent end-of-life for Java 6 for users without Oracle support contracts). So - a lot of work - but also something that looks manageable. We will get there: we have a track record of eventually delivering :) Also, I really need Jython 2.7 for some projects at Rackspace, specifically support for running Python on Storm, so there's that. - Jim |
From: Fernando M. <fer...@cm...> - 2013-08-29 22:08:01
|
On 08/29/2013 12:38 AM, Jim Baker wrote: > > > > On Wed, Aug 28, 2013 at 3:53 PM, Fernando Martins > <fer...@cm... <mailto:fer...@cm...>> wrote: > >> .... > > I'm using jython 2.5 and this works: > > x = "José" > toClient.println("<li>%s</li>" % x) > > In my example, row[0] is of type PyUnicode and the db uses > encoding utf8. So, some part of the pipelining is changing the > encoding. After a search, I got to > > http://old.nabble.com/Re%3A-On-String,-PyString-and-PyUnicode-p18767753.html > > where you state that PyUnicode uses utf-16. I can't test now, but > I guess .encode('utf-8') should do the trick, right? And I don't > suppose there is a way to avoid this extra conversion? > > > Correct, this conversion is really required. Sometimes you may be > working with input stream/output streams that are character-oriented, > but not in this case. But it's just a simple matter of .encode("UTF-8") > For the record, I have it working without .encode("UTF-8"). I think the issue was that the encoding of the response has to be set before getting the writer. I think I got mislead by the browser initially reporting the page as UTF-8, incorrectly because the servlet docs says the default is ISO-8859-1. At this stage I had not explicitly set the encoding and latter on I set it after creating the writer (incorrectly). Note also that creating a string with x = "josé" will not work even if the .py file states encoding utf8!! it needs x = unicode("josé", "UTF-8") However, it works well (no call to encode) from a text file including UTF-8 characters!! For the record, complete test code: # -*- coding: utf-8 -*- from javax.servlet.http import HttpServlet from com.ziclix.python.sql import zxJDBC from java.lang import System import sys class JythonServlet1 (HttpServlet): def doGet(self,request,response): self.doPost (request,response) def doPost(self,request,response): response.setContentType("text/html; charset=UTF-8") # encoding must be set before getWriter toClient = response.getWriter() d = "jdbc:mysql://localhost/database" u, p, v = 'xxxx', 'xxxx', 'com.mysql.jdbc.Driver' db = zxJDBC.connect(d, u, p, v) cursor = db.cursor() cursor.execute('SELECT * FROM TABLE') html = unicode("<html><head><title>Servlet Test</title>" +\ "<body><h1>José</h1><ul>", "UTF-8") toClient.println(html) for row in cursor.fetchall(): toClient.println("<li>%s</li>" % row[1]) toClient.println("</ul></body></html>") > ... > This also doesn't count some new functionality that we will be > delivering such as ssl support; for what it's worth, I recently pushed > up a branch of an experiment on this that allows me to use > easy_install (https://bitbucket.org/jimbaker/jython-ssl); pip still > fails because of some Unicode issue and the lack of os.O_NOFOLLOW, > something we could readily fix if we restrict 2.7 to Java 7 (as I've > started to advocate on #jython, given the recent end-of-life for Java > 6 for users without Oracle support contracts). > I don't know yet if I'm going to use jython for sure but it is likely as I'll have to port a python web site to the official java platform. Jython might give me a cheap way to "port" from python to java ;) In that case, it will be java 6 and I have no idea if the systems will be upgraded to 7 any time soon. But I don't think supporting older platforms is a good criteria to delay progress of Jython unless it pays your bread and butter :) Personally I am very glad to have a shot at using python on java and very grateful for your work!! Cheers, Fernando |
From: Pekka K. <pe...@ik...> - 2013-08-29 06:39:59
|
2013/8/29 Jim Baker <jb...@zy...>: > Fernando Martins <fer...@cm...> wrote: >> BTW, what is the state of jython 2.7 bugwise? > > Working hard on it! ;) > > There are a bunch of little things that need to be fixed. There are 293 > FIXME tests out of ~5000 tests in the test suite right now, with skips there > because regrtest can fail to complete otherwise. Some of these FIXMEs we > still probably won't fix this time around, despite their value, such as > codecs for encodings like big5 and shift_jis, especially given there are > workarounds for these issues. > > This also doesn't count some new functionality that we will be delivering > such as ssl support; for what it's worth, I recently pushed up a branch of > an experiment on this that allows me to use easy_install > (https://bitbucket.org/jimbaker/jython-ssl); pip still fails because of some > Unicode issue and the lack of os.O_NOFOLLOW, something we could readily fix > if we restrict 2.7 to Java 7 (as I've started to advocate on #jython, given > the recent end-of-life for Java 6 for users without Oracle support > contracts). +1 for requiring Java 7. Anything that makes Jython 2.7 easier to finalize is great, and in this case the question is about supporting or not supporting an unsupported platform. Those who must keep on using older Java versions can keep on using Jython 2.5 too. > So - a lot of work - but also something that looks manageable. We will get > there: we have a track record of eventually delivering :) Also, I really > need Jython 2.7 for some projects at Rackspace, specifically support for > running Python on Storm, so there's that. This is great to hear. Do you have any educated guesses when the first release candidates could be available? At that point I think we could change Robot Framework <http://robotframework.org> to require Python/Jython/IronPython 2.7, and finally add the often requested Python 3(.3) support. Supporting Python 3 syntax along with 2.5 syntax would be too much work, and we cannot really drop Jython support either. Cheers, .peke -- Agile Tester/Developer/Consultant :: http://eliga.fi Lead Developer of Robot Framework :: http://robotframework.org |
From: Nick R. <ni...@ca...> - 2013-08-30 18:21:58
|
On 29/08/2013 07:39, Pekka Klärck wrote: > +1 for requiring Java 7. Sooner or later, yes, but that would take me out of the game; I need to load Jython via JNI into an audio app. under OS X which also needs to load 32-bit plugins. I'm using Jython 2.7b1 at the moment. (Of course, Java 6 under OS X is dying, so the problem is looming anyway.) -- N. |
From: Pekka K. <pe...@ik...> - 2013-08-31 11:24:23
|
2013/8/30 Nick Rothwell <ni...@ca...>: > On 29/08/2013 07:39, Pekka Klärck wrote: >> +1 for requiring Java 7. > Sooner or later, yes, but that would take me out of the game; I need to > load Jython via JNI into an audio app. under OS X which also needs to > load 32-bit plugins. I'm using Jython 2.7b1 at the moment. Is there a reason you couldn't upgrade to Java 7 or keep on using Jython version (including 2.7b1) that still work with Java 6? > (Of course, Java 6 under OS X is dying, so the problem is looming anyway.) This is pretty much the reason I think dropping Java 6 now makes sense. If I've understood it correctly, Jython 2.7 isn't exactly around the corner, so there's a very big change that Java 6 is effectively dead when it is finally released. If requiring Java 7 would speed up the development, or would allow optimizations not possibly otherwise, it would be a big win for the majority of the users. Cheers, .peke -- Agile Tester/Developer/Consultant :: http://eliga.fi Lead Developer of Robot Framework :: http://robotframework.org |
From: Nick R. <ni...@ca...> - 2013-08-31 14:01:10
|
On 31/08/2013 12:24, Pekka Klärck wrote: > Is there a reason you couldn't upgrade to Java 7 or keep on using > Jython version (including 2.7b1) that still work with Java 6? Java 7 is 64-bit only, so I wouldn't be able to load a Java 7 JVM into a 32-bit hosting app. > This is pretty much the reason I think dropping Java 6 now makes sense. from Jython's point of view, sure; my particular use case (needing 32-bit Java under OS X) is a little obscure. On the other hand, isn't it it a little counter-intuitive for Jython 2.7 betas to work in Java 6 but the final version to not? -- N. |
From: <Matthew.Webber@Diamond.ac.uk> - 2013-08-31 17:19:50
|
> Java 7 is 64-bit only, so I wouldn't be able to load a Java 7 JVM into a > 32-bit hosting app. That's not correct; Java 7 is available for 32 and 64 bit Windows and Linux. -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom |
From: Nick R. <ni...@ca...> - 2013-09-01 09:38:47
|
On 31/08/2013 18:18, Matthew.Webber@Diamond.ac.uk wrote: >> Java 7 is 64-bit only, so I wouldn't be able to load a Java 7 JVM into a >> 32-bit hosting app. > That's not correct; Java 7 is available for 32 and 64 bit Windows and Linux.. Sure; I'm on OS X, where Java 7 is 64-bit only. -- N. |
From: Jim B. <jb...@zy...> - 2013-09-05 01:18:19
|
So it's a bit more clear to me re Java 6 vs Java 7. Basically for OS X only, Java 6 support continues without requiring getting a support contract. In particular, on my OS X laptop, I have a current Java 6 release when I switch my Java home accordingly: $ java -version java version "1.6.0_51" (I usually run Java 7 of course.) Having said that, I think it makes sense for us to move to Java 7 where it will plug specific bugs and gaps; experience has shown that it can be a lot easier for us to make pip and other applications work than trying to push changes upstream. After then figuring this out on Java 7, we can then also selectively downgrade this capability, eg os.O_NOFOLLOW is available only on Java 7.This means pip doesn't work, but easy_install should. However, this requires a bit more workaround, so Java 6 support won't necessarily be immediately available in trunk as we make this transition. Thoughts? - Jim On Sun, Sep 1, 2013 at 3:38 AM, Nick Rothwell <ni...@ca...> wrote: > On 31/08/2013 18:18, Matthew.Webber@Diamond.ac.uk wrote: > >> Java 7 is 64-bit only, so I wouldn't be able to load a Java 7 JVM into a > >> 32-bit hosting app. > > That's not correct; Java 7 is available for 32 and 64 bit Windows and > Linux.. > Sure; I'm on OS X, where Java 7 is 64-bit only. > > -- N. > > > ------------------------------------------------------------------------------ > Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! > Discover the easy way to master current and previous Microsoft technologies > and advance your career. Get an incredible 1,500+ hours of step-by-step > tutorial videos with LearnDevNow. Subscribe today and save! > http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk > _______________________________________________ > Jython-users mailing list > Jyt...@li... > https://lists.sourceforge.net/lists/listinfo/jython-users > |
From: Nick R. <ni...@ca...> - 2013-09-05 10:52:10
|
On 05/09/2013 02:17, Jim Baker wrote: > (I usually run Java 7 of course.) Me too, for anything other than MaxMSP, which (I presume) asks for a 32-bit JVM and gets Java 6. > Having said that, I think it makes sense for us to move to Java 7 > where it will plug specific bugs and gaps; experience has shown that > it can be a lot easier for us to make pip and other applications work > than trying to push changes upstream. Sure. > After then figuring this out on Java 7, we can then also selectively > downgrade this capability, eg os.O_NOFOLLOW is available only on Java > 7.This means pip doesn't work, but easy_install should. However, this > requires a bit more workaround, so Java 6 support won't necessarily be > immediately available in trunk as we make this transition. I guess the Right Answer would be for Oracle to roll out Java 7 in 32- and 64-bit versions, but it looks as if they abandoned the 32-bit version late in 2011. -- N. |
From: Alan K. <jyt...@xh...> - 2013-09-07 14:37:20
|
+1 for requiring java 7. Java 6 is end of life already, so java 7 is the only supported version available. Also, there are features of java 7 that will make supporting, e.g., modern networking, easier: cleaner ipv6 support, non-blocking multicast, etc. Conversely, not having up-to-date fixes and features available in java 6 either delays or makes it more difficult to provide robust functionality. On Thu, Sep 5, 2013 at 2:17 AM, Jim Baker <jb...@zy...> wrote: > So it's a bit more clear to me re Java 6 vs Java 7. Basically for OS X > only, Java 6 support continues without requiring getting a support > contract. In particular, on my OS X laptop, I have a current Java 6 release > when I switch my Java home accordingly: > > $ java -version > java version "1.6.0_51" > > (I usually run Java 7 of course.) > > Having said that, I think it makes sense for us to move to Java 7 where it > will plug specific bugs and gaps; experience has shown that it can be a lot > easier for us to make pip and other applications work than trying to push > changes upstream. > > After then figuring this out on Java 7, we can then also selectively > downgrade this capability, eg os.O_NOFOLLOW is available only on Java > 7.This means pip doesn't work, but easy_install should. However, this > requires a bit more workaround, so Java 6 support won't necessarily be > immediately available in trunk as we make this transition. > > Thoughts? > > - Jim > > > > > On Sun, Sep 1, 2013 at 3:38 AM, Nick Rothwell <ni...@ca...> wrote: > >> On 31/08/2013 18:18, Matthew.Webber@Diamond.ac.uk wrote: >> >> Java 7 is 64-bit only, so I wouldn't be able to load a Java 7 JVM into >> a >> >> 32-bit hosting app. >> > That's not correct; Java 7 is available for 32 and 64 bit Windows and >> Linux.. >> Sure; I'm on OS X, where Java 7 is 64-bit only. >> >> -- N. >> >> >> ------------------------------------------------------------------------------ >> Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! >> Discover the easy way to master current and previous Microsoft >> technologies >> and advance your career. Get an incredible 1,500+ hours of step-by-step >> tutorial videos with LearnDevNow. Subscribe today and save! >> >> http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk >> _______________________________________________ >> Jython-users mailing list >> Jyt...@li... >> https://lists.sourceforge.net/lists/listinfo/jython-users >> > > > > ------------------------------------------------------------------------------ > Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! > Discover the easy way to master current and previous Microsoft technologies > and advance your career. Get an incredible 1,500+ hours of step-by-step > tutorial videos with LearnDevNow. Subscribe today and save! > http://pubads.g.doubleclick.net/gampad/clk?id=58041391&iu=/4140/ostg.clktrk > _______________________________________________ > Jython-users mailing list > Jyt...@li... > https://lists.sourceforge.net/lists/listinfo/jython-users > > |