From: Jeff A. <ja...@fa...> - 2012-09-18 08:08:30
|
I've been looking at the io module with a view to getting test.test_io to pass, and bringing the io module as seen from Python as close as can be to to the CPython picture. test_io has a high level of failures (25), errors (52) and skips (82) in at the moment. A copy of the Python implementation of PEP-3116 (_pyio.py) masquerades as _io by occupying _io.py, whereas the CPython io.py thinks it is delegating to a C implementation when it imports _io. I know a number of you have contributed to a Java implementation close to PEP-3116 that resides in org.python.core.io . I intend to build on that. You understand it better than I do, so I'd like to check mine is a viable plan. 1. The package org.python.modules._fileio is my starting point, but for strict correspondence with CPython, ought in version 2.7 to be _io not _fileio. 2. The new Java package org.python.modules._io will contain exposed classes corresponding to Python's _io.* classes. These classes will have the signatures Python expects, but delegate the work to corresponding org.python.core.io classes. The existing PyFileIO can be my exemplar in how to do this. 3. There should be a static open() function in org.python.modules._io._io.java . 4. fileno() should return something the Python user treats as an opaque handle, and that open() and the constructors of streams will have to accept, where currently their CPython implementations expect an int. I read the discussion around the proper return type fileno() (http://comments.gmane.org/gmane.comp.lang.jython.devel/3994 and refs therein). 5. I can make these changes progressively by ditching _io.py (clone of _pyio.py) and replacing the current CPython io.py with one that delegates to _pyio.py initially. Then class by class, I change its delegation from _pyio to _io (Java implementation). In the end, we go back to the CPython io.py. Thoughts welcome. Jeff |
From: Jeff A. <ja...@fa...> - 2012-09-19 07:31:02
|
Ok, amended idea: 5. Rename _io.py (adapted from prototype v2.6 io.py) to _jyio.py. and replace the current CPython io.py with one that delegates to _jyio.py initially. Then class by class, I change its delegation back from _jyio to _io (Java implementation). In the end, we go back to the CPython io.py. |
From: Philip J. <pj...@un...> - 2012-09-20 20:32:21
|
On Sep 18, 2012, at 1:07 AM, Jeff Allen wrote: > I've been looking at the io module with a view to getting test.test_io > to pass, and bringing the io module as seen from Python as close as can > be to to the CPython picture. test_io has a high level of failures (25), > errors (52) and skips (82) in at the moment. A copy of the Python > implementation of PEP-3116 (_pyio.py) masquerades as _io by occupying > _io.py, whereas the CPython io.py thinks it is delegating to a C > implementation when it imports _io. > > I know a number of you have contributed to a Java implementation close > to PEP-3116 that resides in org.python.core.io . I intend to build on > that. You understand it better than I do, so I'd like to check mine is a > viable plan. > > 1. The package org.python.modules._fileio is my starting point, but for > strict correspondence with CPython, ought in version 2.7 to be _io not > _fileio. > > 2. The new Java package org.python.modules._io will contain exposed > classes corresponding to Python's _io.* classes. These classes will have > the signatures Python expects, but delegate the work to corresponding > org.python.core.io classes. The existing PyFileIO can be my exemplar in > how to do this. Hey Jeff, I haven't thought about all this in too much detail but I should point out (as the author of core.io): o It's "loosely" based on PEP 3116 because there are some differences between py2 and py3 file: - universal newlines mode is more configurable (you can choose the 'newline' to use) and it now supports writing - since the buffer/raw layers aren't exposed in Py 2 file, I didn't bother making them threadsafe (PyFile is responsible for the locking) - no 'encoding' arg to open() functionality was needed for Py 2 - other small things (like some of the exceptions IOBase raises should be different in Py 3) and probably other things I've forgotten o It was written before CPython's _io and obviously before your new buffer stuff. IIRC the lower layers of CPython's _io heavily use Py_Buffer whereas core.io works on java.nio.Buffer. I'm not sure how much of it might benefit from the new Buffer stuff (I haven't followed the buffer work very much unfortunately). o I was hoping PyString might eventually be based on bytes instead of char. I also thought that *possibly* future Py_Buffer support in Jython might be based/or somehow integrate with java.nio.Buffer (I'm not sure that's even a great goal though, you might have some insight. Integrating with a ByteBuffer is simple if you have an underlying Java byte[] array somewhere). - So note that core.io is well optimized right now, though it could actually gain a slight speedup in Py2 if we got rid of the extra bytes->String (for PyString) conversion Basically, adapting core.io to _io will take some doing and I'm not sure how to handle the 2 vs 3 differences. We should also keep in mind that the work shouldn't affect Py2 file performance negatively as that's ultimately more important to Py 2 code. Adding locks to all of the layers could hurt (though maybe Java 6 escape analysis/lock coarsening helps here) In fact, I'm not sure the io module is very heavily used in Py 2 code at all (probably just in some cases of Py3 compat)? You might want to consider doing the bare minimum to get it working for now, and leave optimizing it until later (maybe even until Jython 3). Then you can basically defer on all the points I'm worrying about =] > 3. There should be a static open() function in > org.python.modules._io._io.java . > > 4. fileno() should return something the Python user treats as an opaque > handle, and that open() and the constructors of streams will have to > accept, where currently their CPython implementations expect an int. I > read the discussion around the proper return type fileno() > (http://comments.gmane.org/gmane.comp.lang.jython.devel/3994 and refs > therein). We should have this already unless I'm missing something > > 5. I can make these changes progressively by ditching _io.py (clone of > _pyio.py) and replacing the current CPython io.py with one that > delegates to _pyio.py initially. Then class by class, I change its > delegation from _pyio to _io (Java implementation). In the end, we go > back to the CPython io.py. It was a little simpler in 2.6 in that the bare minimum you needed to implement for the pure Python version of io to work was _fileio.FileIO. The 2.7 _pyio is a little strange in that it refers to io.IO/RawIO/Buffered/TextIOBase to register as ABCs (which requires _io). You can probably get away with implementing just io.FileIO and SEEK_SET/CUR/END (as a builtin that'd replace io.py). Then comment out the ABC registration calls in _pyio. -- Philip Jenvey |
From: Jeff A. <ja...@fa...> - 2012-09-21 20:32:39
|
Thanks for that Philip. I'll start with _io.FileIO, as you suggest, and its hierarchy up to _IOBase made to look like the corresponding CPython classes. I hope to populate this basically by abstraction from your PyFileIO. I've just drawn out the inheritance hierarchy of _io in CPython, and there are a lot of classes there. On 20/09/2012 21:32, Philip Jenvey wrote: > On Sep 18, 2012, at 1:07 AM, Jeff Allen wrote: >> ... > Hey Jeff, I haven't thought about all this in too much detail but I should point out (as the author of core.io): > > o It's "loosely" based on PEP 3116 because there are some differences between py2 and py3 file: > - universal newlines mode is more configurable (you can choose the 'newline' to use) and it now supports writing > - since the buffer/raw layers aren't exposed in Py 2 file, I didn't bother making them threadsafe (PyFile is responsible for the locking) > - no 'encoding' arg to open() functionality was needed for Py 2 > - other small things (like some of the exceptions IOBase raises should be different in Py 3) and probably other things I've forgotten In mentioning Py 2 and Py 3 here, do you mean that your implementation is Py 3 in character and needs some down-shifting to match he Py 2 capability? Obviously I'm targeting the Py 2.7 tests. > o I was hoping PyString might eventually be based on bytes instead of char. I think so too, for Jython 3. BaseBytes.java was written with this in mind, although I think refactoring that hierarchy is overdue. > I also thought that *possibly* future Py_Buffer support in Jython might be based/or somehow integrate with java.nio.Buffer (I'm not sure that's even a great goal though, you might have some insight. Integrating with a ByteBuffer is simple if you have an underlying Java byte[] array somewhere). I agree and I know where it would fit: it can't *be* a PyBuffer, nor be extended, but it could (probably should) replace the thing I called BufferPointer that encapsulates a byte[] and an offset into it. I think revisiting readinto() etc. will make me do this. > - So note that core.io is well optimized right now, though it could actually gain a slight speedup in Py2 if we got rid of the extra bytes->String (for PyString) conversion I think I noticed some of this unwelcome conversion. > Basically, adapting core.io to _io will take some doing and I'm not sure how to handle the 2 vs 3 differences. We should also keep in mind that the work shouldn't affect Py2 file performance negatively as that's ultimately more important to Py 2 code. Adding locks to all of the layers could hurt (though maybe Java 6 escape analysis/lock coarsening helps here) > > In fact, I'm not sure the io module is very heavily used in Py 2 code at all (probably just in some cases of Py3 compat)? You might want to consider doing the bare minimum to get it working for now, and leave optimizing it until later (maybe even until Jython 3). Then you can basically defer on all the points I'm worrying about =] I was going for correct first. >> 3. There should be a static open() function in >> org.python.modules._io._io.java . >> >> 4. fileno() should return something the Python user treats as an opaque >> handle, and that open() and the constructors of streams will have to >> accept, where currently their CPython implementations expect an int. I >> read the discussion around the proper return type fileno() >> (http://comments.gmane.org/gmane.comp.lang.jython.devel/3994 and refs >> therein). > We should have this already unless I'm missing something fileno() returns it, but open() doesn't accept it, only a string, nor if I comment out the type test in open, does the FileIO constructor. It shouldn't be too difficult. >> 5. I can make these changes progressively by ditching _io.py (clone of >> _pyio.py) and replacing the current CPython io.py with one that >> delegates to _pyio.py initially. Then class by class, I change its >> delegation from _pyio to _io (Java implementation). In the end, we go >> back to the CPython io.py. > It was a little simpler in 2.6 in that the bare minimum you needed to implement for the pure Python version of io to work was _fileio.FileIO. The 2.7 _pyio is a little strange in that it refers to io.IO/RawIO/Buffered/TextIOBase to register as ABCs (which requires _io). > > You can probably get away with implementing just io.FileIO and SEEK_SET/CUR/END (as a builtin that'd replace io.py). Then comment out the ABC registration calls in _pyio. I plan to keep io.py and mirror exactly the hierarchy in CPython. That way I expect to be able to use existing Python implementations of classes I haven't implement in Java. My understanding of how classes and modules cross that boundary should improve a lot! By which I mean I'll be asking whether I've done it right. Jeff |
From: Philip J. <pj...@un...> - 2012-09-21 21:25:24
|
On Sep 21, 2012, at 1:32 PM, Jeff Allen wrote: > Thanks for that Philip. I'll start with _io.FileIO, as you suggest, and > its hierarchy up to _IOBase made to look like the corresponding CPython > classes. I hope to populate this basically by abstraction from your > PyFileIO. I've just drawn out the inheritance hierarchy of _io in > CPython, and there are a lot of classes there. > > On 20/09/2012 21:32, Philip Jenvey wrote: >> On Sep 18, 2012, at 1:07 AM, Jeff Allen wrote: >>> ... >> Hey Jeff, I haven't thought about all this in too much detail but I should point out (as the author of core.io): >> >> o It's "loosely" based on PEP 3116 because there are some differences between py2 and py3 file: >> - universal newlines mode is more configurable (you can choose the 'newline' to use) and it now supports writing >> - since the buffer/raw layers aren't exposed in Py 2 file, I didn't bother making them threadsafe (PyFile is responsible for the locking) >> - no 'encoding' arg to open() functionality was needed for Py 2 >> - other small things (like some of the exceptions IOBase raises should be different in Py 3) and probably other things I've forgotten > In mentioning Py 2 and Py 3 here, do you mean that your implementation > is Py 3 in character and needs some down-shifting to match he Py 2 > capability? Obviously I'm targeting the Py 2.7 tests. Yep, but I mention the differences just as a warning if we're to start using these classes for the io module, particularly the text layers where most of the differences are. Then core.io would somehow have to straddle Py 2 (when used w/ file) functionality vs Py 3 (when used via io). Right now you're going to start with only FileIO, so you probably won't run into any big 2 vs 3 differences except for the differing exception types raised. >> o I was hoping PyString might eventually be based on bytes instead of char. > I think so too, for Jython 3. BaseBytes.java was written with this in > mind, although I think refactoring that hierarchy is overdue. This will definitely have to happen for 3 but it could also benefit 2. We'd save the bytes -> String conversion in core.io. Plus, right now if you have 1mb of bytes as a Jython 2 str it takes 2mb of memory =[ >> I also thought that *possibly* future Py_Buffer support in Jython might be based/or somehow integrate with java.nio.Buffer (I'm not sure that's even a great goal though, you might have some insight. Integrating with a ByteBuffer is simple if you have an underlying Java byte[] array somewhere). > I agree and I know where it would fit: it can't *be* a PyBuffer, nor be > extended, but it could (probably should) replace the thing I called > BufferPointer that encapsulates a byte[] and an offset into it. I think > revisiting readinto() etc. will make me do this. >> - So note that core.io is well optimized right now, though it could actually gain a slight speedup in Py2 if we got rid of the extra bytes->String (for PyString) conversion > I think I noticed some of this unwelcome conversion. >> Basically, adapting core.io to _io will take some doing and I'm not sure how to handle the 2 vs 3 differences. We should also keep in mind that the work shouldn't affect Py2 file performance negatively as that's ultimately more important to Py 2 code. Adding locks to all of the layers could hurt (though maybe Java 6 escape analysis/lock coarsening helps here) >> >> In fact, I'm not sure the io module is very heavily used in Py 2 code at all (probably just in some cases of Py3 compat)? You might want to consider doing the bare minimum to get it working for now, and leave optimizing it until later (maybe even until Jython 3). Then you can basically defer on all the points I'm worrying about =] > I was going for correct first. >>> 3. There should be a static open() function in >>> org.python.modules._io._io.java . >>> >>> 4. fileno() should return something the Python user treats as an opaque >>> handle, and that open() and the constructors of streams will have to >>> accept, where currently their CPython implementations expect an int. I >>> read the discussion around the proper return type fileno() >>> (http://comments.gmane.org/gmane.comp.lang.jython.devel/3994 and refs >>> therein). >> We should have this already unless I'm missing something > fileno() returns it, but open() doesn't accept it, only a string, nor if > I comment out the type test in open, does the FileIO constructor. It > shouldn't be too difficult. builtin open shouldn't accept it, but os.open should (and already does) and io.open allows it too. >>> 5. I can make these changes progressively by ditching _io.py (clone of >>> _pyio.py) and replacing the current CPython io.py with one that >>> delegates to _pyio.py initially. Then class by class, I change its >>> delegation from _pyio to _io (Java implementation). In the end, we go >>> back to the CPython io.py. >> It was a little simpler in 2.6 in that the bare minimum you needed to implement for the pure Python version of io to work was _fileio.FileIO. The 2.7 _pyio is a little strange in that it refers to io.IO/RawIO/Buffered/TextIOBase to register as ABCs (which requires _io). >> >> You can probably get away with implementing just io.FileIO and SEEK_SET/CUR/END (as a builtin that'd replace io.py). Then comment out the ABC registration calls in _pyio. > I plan to keep io.py and mirror exactly the hierarchy in CPython. That > way I expect to be able to use existing Python implementations of > classes I haven't implement in Java. My understanding of how classes and > modules cross that boundary should improve a lot! By which I mean I'll > be asking whether I've done it right. Cool, sounds like we're on the same page. If you have any quick questions you can also find me and others on IRC (irc.freenode.net #jython) -- Philip Jenvey |
From: Jeff A. <ja...@fa...> - 2012-09-30 14:41:48
|
I've spread the goodness of _io.PyFileIO.java up a hierearchy of PyFileIO.java, PyRawIOBase.java and _IOBase.java in a way I think is roughly correct, but nothing works yet: the test program (test_io) cannot construct the classes that extend mine. This is really taxing my understanding of how Jython is implemented. It is more accurate to say I have almost no idea how Jython is implemented, and don't know where to look. At least, I've no idea what should happen when Jython tries to instantiate CMockIOWithoutRead. I've spent some time single-stepping through attempts at reflective invocation of my constructors. I can see plausible stuff happening but it is very difficult to know whether it is meant to happen, or is some kind of fall-back. The mro is [<class '__main__.CMockRawIOWithoutRead'>, <class '__main__.MockRawIOWithoutRead'>, <class 'io.RawIOBase'>, <type 'org.python.modules._io.PyRawIOBase'>, <class 'io.IOBase'>, <type '_io._IOBase'>, <type 'object'>]. Along the way, Jython must have to create a PyRawIOBase, or maybe a PyRawIOBaseDerived, that in Python is one of the multiply-inherited base classes of CMockRawIOWithoutRead. Eventually it should find and call MockRawIOWithoutRead.__init__(), and later MockRawIOWithoutRead.readinto() (we're in test_RawIOBase_read). The argument MockRawIOWithoutRead.__init__() should get, which is a tuple of strings to be treated as lines of a virtual file is not something any PyRawIOBase constructor is likely to expect, but Jython is looking for one and giving up when it is not found. Should there be one accepting the generic args[], kwargs[] arguments to be called and do nothing when _io._RawIOBase is the "wrong" base class? I could really do with a picture of the data structures that Jython creates in these circumstances, and a narrative on how it would traverse them to find methods and constructors. Is there really *nothing* written down for the next generation of developers? Jeff Allen |
From: <fwi...@gm...> - 2012-09-30 21:05:31
|
On Sun, Sep 30, 2012 at 7:40 AM, Jeff Allen <ja...py...@fa...> wrote: > I've spread the goodness of _io.PyFileIO.java up a hierearchy of > PyFileIO.java, PyRawIOBase.java and _IOBase.java in a way I think is > roughly correct, but nothing works yet: the test program (test_io) > cannot construct the classes that extend mine. This is really taxing my > understanding of how Jython is implemented. > > It is more accurate to say I have almost no idea how Jython is > implemented, and don't know where to look. At least, I've no idea what > should happen when Jython tries to instantiate CMockIOWithoutRead. I've > spent some time single-stepping through attempts at reflective > invocation of my constructors. I can see plausible stuff happening but > it is very difficult to know whether it is meant to happen, or is some > kind of fall-back. > > The mro is [<class '__main__.CMockRawIOWithoutRead'>, <class > '__main__.MockRawIOWithoutRead'>, <class 'io.RawIOBase'>, <type > 'org.python.modules._io.PyRawIOBase'>, <class 'io.IOBase'>, <type > '_io._IOBase'>, <type 'object'>]. Along the way, Jython must have to > create a PyRawIOBase, or maybe a PyRawIOBaseDerived, that in Python is > one of the multiply-inherited base classes of CMockRawIOWithoutRead. > Eventually it should find and call MockRawIOWithoutRead.__init__(), and > later MockRawIOWithoutRead.readinto() (we're in test_RawIOBase_read). > The argument MockRawIOWithoutRead.__init__() should get, which is a > tuple of strings to be treated as lines of a virtual file is not > something any PyRawIOBase constructor is likely to expect, but Jython is > looking for one and giving up when it is not found. Should there be one > accepting the generic args[], kwargs[] arguments to be called and do > nothing when _io._RawIOBase is the "wrong" base class? > > I could really do with a picture of the data structures that Jython > creates in these circumstances, and a narrative on how it would traverse > them to find methods and constructors. Is there really *nothing* written > down for the next generation of developers? > The method dispatch code is indeed entirely undocumented and fairly complicated - would you mind posting a patch somewhere (maybe here)? I think looking at what you have and getting my own head together on the subject as I try to explain could help me start a doc. -Frank |
From: Jeff A. <ja...@fa...> - 2012-10-01 07:05:16
|
Thanks Frank. I've got a change set to push (just the renaming of modules) then when I've tidied up a bit -- no point confusing things further -- I'll post a patch of my broken bits so far. Jeff On 30/09/2012 22:05, fwi...@gm... wrote: > > > On Sun, Sep 30, 2012 at 7:40 AM, Jeff Allen <ja...py...@fa... > <mailto:ja...py...@fa...>> wrote: > > I've spread the goodness of _io.PyFileIO.java up a hierearchy of > PyFileIO.java, PyRawIOBase.java and _IOBase.java in a way I think is > roughly correct, but nothing works yet: the test program (test_io) > cannot construct the classes that extend mine. This is really > taxing my > understanding of how Jython is implemented. > > It is more accurate to say I have almost no idea how Jython is > implemented, and don't know where to look. At least, I've no idea what > should happen when Jython tries to instantiate CMockIOWithoutRead. > I've > spent some time single-stepping through attempts at reflective > invocation of my constructors. I can see plausible stuff happening but > it is very difficult to know whether it is meant to happen, or is some > kind of fall-back. > > The mro is [<class '__main__.CMockRawIOWithoutRead'>, <class > '__main__.MockRawIOWithoutRead'>, <class 'io.RawIOBase'>, <type > 'org.python.modules._io.PyRawIOBase'>, <class 'io.IOBase'>, <type > '_io._IOBase'>, <type 'object'>]. Along the way, Jython must have to > create a PyRawIOBase, or maybe a PyRawIOBaseDerived, that in Python is > one of the multiply-inherited base classes of CMockRawIOWithoutRead. > Eventually it should find and call > MockRawIOWithoutRead.__init__(), and > later MockRawIOWithoutRead.readinto() (we're in test_RawIOBase_read). > The argument MockRawIOWithoutRead.__init__() should get, which is a > tuple of strings to be treated as lines of a virtual file is not > something any PyRawIOBase constructor is likely to expect, but > Jython is > looking for one and giving up when it is not found. Should there > be one > accepting the generic args[], kwargs[] arguments to be called and do > nothing when _io._RawIOBase is the "wrong" base class? > > I could really do with a picture of the data structures that Jython > creates in these circumstances, and a narrative on how it would > traverse > them to find methods and constructors. Is there really *nothing* > written > down for the next generation of developers? > > The method dispatch code is indeed entirely undocumented and fairly > complicated - would you mind posting a patch somewhere (maybe here)? I > think looking at what you have and getting my own head together on the > subject as I try to explain could help me start a doc. > > -Frank |
From: Jeff A. <ja...@fa...> - 2012-10-03 20:00:51
|
Some of this is beginning to make sense. I've discovered: * The "plausible stuff happening ... difficult to know whether it is meant to happen", is not supposed to happen. I made a typo in CoreExposed.includes, and that's why constructors in my PyRawIOBase were being searched reflectively. * If PyRawIOBase is properly exposed, __new__ is called and as we return, __init__ would be called in some circumstances. Now that __new__ is being called, I can see better how this is supposed to work. But I still don't really have the picture in my head of the data structure being built and navigated. Can anyone say where (in Java) MockRawIOWithoutRead.__init__() should get called from, and when? Is that done in the generated code defined in Python or in support code written in Java? Jeff On 01/10/2012 08:03, Jeff Allen wrote: > Thanks Frank. I've got a change set to push (just the renaming of > modules) then when I've tidied up a bit -- no point confusing things > further -- I'll post a patch of my broken bits so far. > > Jeff > > On 30/09/2012 22:05, fwi...@gm... wrote: >> >> >> On Sun, Sep 30, 2012 at 7:40 AM, Jeff Allen <ja...py...@fa... >> <mailto:ja...py...@fa...>> wrote: >> >> I've spread the goodness of _io.PyFileIO.java up a hierearchy of >> PyFileIO.java, PyRawIOBase.java and _IOBase.java in a way I think is >> roughly correct, but nothing works yet: the test program (test_io) >> cannot construct the classes that extend mine. This is really >> taxing my >> understanding of how Jython is implemented. >> >> It is more accurate to say I have almost no idea how Jython is >> implemented, and don't know where to look. At least, I've no idea >> what >> should happen when Jython tries to instantiate >> CMockIOWithoutRead. I've >> spent some time single-stepping through attempts at reflective >> invocation of my constructors. I can see plausible stuff >> happening but >> it is very difficult to know whether it is meant to happen, or is >> some >> kind of fall-back. >> >> The mro is [<class '__main__.CMockRawIOWithoutRead'>, <class >> '__main__.MockRawIOWithoutRead'>, <class 'io.RawIOBase'>, <type >> 'org.python.modules._io.PyRawIOBase'>, <class 'io.IOBase'>, <type >> '_io._IOBase'>, <type 'object'>]. Along the way, Jython must have to >> create a PyRawIOBase, or maybe a PyRawIOBaseDerived, that in >> Python is >> one of the multiply-inherited base classes of CMockRawIOWithoutRead. >> Eventually it should find and call >> MockRawIOWithoutRead.__init__(), and >> later MockRawIOWithoutRead.readinto() (we're in test_RawIOBase_read). >> The argument MockRawIOWithoutRead.__init__() should get, which is a >> tuple of strings to be treated as lines of a virtual file is not >> something any PyRawIOBase constructor is likely to expect, but >> Jython is >> looking for one and giving up when it is not found. Should there >> be one >> accepting the generic args[], kwargs[] arguments to be called and do >> nothing when _io._RawIOBase is the "wrong" base class? >> >> I could really do with a picture of the data structures that Jython >> creates in these circumstances, and a narrative on how it would >> traverse >> them to find methods and constructors. Is there really *nothing* >> written >> down for the next generation of developers? >> >> The method dispatch code is indeed entirely undocumented and fairly >> complicated - would you mind posting a patch somewhere (maybe here)? >> I think looking at what you have and getting my own head together on >> the subject as I try to explain could help me start a doc. >> >> -Frank > > > ------------------------------------------------------------------------------ > Got visibility? > Most devs has no idea what their production app looks like. > Find out how fast your code is with AppDynamics Lite. > http://ad.doubleclick.net/clk;262219671;13503038;y? > http://info.appdynamics.com/FreeJavaPerformanceDownload.html > > > _______________________________________________ > Jython-dev mailing list > Jyt...@li... > https://lists.sourceforge.net/lists/listinfo/jython-dev |
From: Jeff A. <ja...@fa...> - 2012-10-13 23:10:11
|
Time I checked into this thread with a status update. My idea has been to copy _pyio.py to _jyio.py (as a second Python implementation of _io) and then progressively replace parts of it with Java in _io. This half works, but the _jyio classes expect features (like a _checkWritable method) that the exposed classes do not provide (yet). I've got a better idea now about how methods are invoked, particularly constructors. The hierarchy around io is complex and it's pleasing to see Jython copes with multiple inheritance. I couldn't explain how it works yet, but it's beginning to look familiar in the debugger. I'm not sure what will happen when we get to the multiple inheritance of exposed classes. I've already encountered a multiple inheritance problem with the exceptions (which do not to work like exposed classes, I know). io.BlockingError and io.UnsupportedOperation both inherit IOError and another exception type. Can anyone explain how to do this in Java? And when I get there, what will the Java for class BufferedRandom(BufferedWriter, BufferedReader): look like? The main progress is that I now have a pretty faithful imitation in Java of _io.open() (from CPython _iomodule.c). It gets called from Python where it should and can construct an (only slightly defective) exposed _io.FileIO that delegates to Philip's org.core.io.PyFileIO. My Java _io.open() method is able to construct instances of the classes in _jyio and wrap the raw FileIO in them. It looks like this: // Get constructor io.BufferedWriter PyObject globals = __builtin__.globals(); PyObject io = globals.__finditem__("io"); PyObject bufferType = io.__getattr__("BufferedWriter"); PyInteger pyBuffering = new PyInteger(buffering); PyIOBase buffer = (PyIOBase)bufferType.__call__(raw, pyBuffering); This is exciting as I'd have had no idea how to do that a couple of weeks ago. But I think it would fail if module io were not already loaded. This will hardly ever be the case, but what could I do in my code to make sure it gets loaded? Jeff On 03/10/2012 21:00, Jeff Allen wrote: > Some of this is beginning to make sense. I've discovered: > > * The "plausible stuff happening ... difficult to know whether it is > meant to happen", is not supposed to happen. I made a typo in > CoreExposed.includes, and that's why constructors in my > PyRawIOBase were being searched reflectively. > * If PyRawIOBase is properly exposed, __new__ is called and as we > return, __init__ would be called in some circumstances. > > > Now that __new__ is being called, I can see better how this is > supposed to work. But I still don't really have the picture in my head > of the data structure being built and navigated. > > Can anyone say where (in Java) MockRawIOWithoutRead.__init__() should > get called from, and when? Is that done in the generated code defined > in Python or in support code written in Java? > > Jeff > > |
From: Jeff A. <ja...@fa...> - 2012-10-24 18:54:19
|
I continue to make progress. You're not seeing anything from me in the code base because I resolved not to push the io work to the repository until my (Java) version scored fewer test_io errors and failures than the existing (Python) version. If it really doesn't matter how broken I've made it while I work, I'll happily put it where others can see. I'm certainly learning a lot about the Jython plumbing through this. Writing bytearray required only a little understanding by comparison: the clever bits are in the Java. Here Philip Jenvey has mostly done the operations and the challenges stem from the interplay of Java and Python. Even the simplest operations have to take into account the possibility that a subclass defined in Python code has overridden them. This theme accounts for many things in the CPython code I didn't understand at first and is probably behind many still-failing tests. I have basic open, write, read and close going on in Java for unbuffered binary files. Operations all involve delegation to Philip's FileIO class, although I found it necessary to implement the logic of read and readall myself because they invoke the possibly overridden "readinto". I think there's more of this Python-overrides-Java stuff to find. I got this overriding to happen, by exposing __dict__ from (just) the right place. Thanks to others who helped me to the right solution. The Python buffered and text wrappers still work, even though they get their base class from the Java code. (At least, I think they mostly do: quite a lot of test failure in this area.) I have adapted the Closer concept from PyFile so that it invokes "close" by name: hence it invokes the overridden close, rather than just closing the underlying (delegate) file. Unlike a PyFile, a PyFileIO remains alive until that can happen. At the moment I'm working on getting close(), closed(), _checkClosed() and their delegate versions to play together nicely, reproducing the behaviour of CPython. Cold towel time. Jeff On 14/10/2012 00:09, Jeff Allen wrote: > Time I checked into this thread with a status update. > > My idea has been to copy _pyio.py to _jyio.py (as a second Python > implementation of _io) and then progressively replace parts of it with > Java in _io. This half works, but the _jyio classes expect features > (like a _checkWritable method) that the exposed classes do not provide > (yet). > > I've got a better idea now about how methods are invoked, particularly > constructors. The hierarchy around io is complex and it's pleasing to > see Jython copes with multiple inheritance. I couldn't explain how it > works yet, but it's beginning to look familiar in the debugger. I'm > not sure what will happen when we get to the multiple inheritance of > exposed classes. > > I've already encountered a multiple inheritance problem with the > exceptions (which do not to work like exposed classes, I know). > io.BlockingError and io.UnsupportedOperation both inherit IOError and > another exception type. Can anyone explain how to do this in Java? > > And when I get there, what will the Java for > class BufferedRandom(BufferedWriter, BufferedReader): > look like? > > The main progress is that I now have a pretty faithful imitation in > Java of _io.open() (from CPython _iomodule.c). It gets called from > Python where it should and can construct an (only slightly defective) > exposed _io.FileIO that delegates to Philip's org.core.io.PyFileIO. My > Java _io.open() method is able to construct instances of the classes > in _jyio and wrap the raw FileIO in them. It looks like this: > > // Get constructor io.BufferedWriter > PyObject globals = __builtin__.globals(); > PyObject io = globals.__finditem__("io"); > PyObject bufferType = io.__getattr__("BufferedWriter"); > > PyInteger pyBuffering = new PyInteger(buffering); > PyIOBase buffer = (PyIOBase)bufferType.__call__(raw, pyBuffering); > > This is exciting as I'd have had no idea how to do that a couple of > weeks ago. But I think it would fail if module io were not already > loaded. This will hardly ever be the case, but what could I do in my > code to make sure it gets loaded? > > Jeff > > On 03/10/2012 21:00, Jeff Allen wrote: >> Some of this is beginning to make sense. I've discovered: >> >> * The "plausible stuff happening ... difficult to know whether it >> is meant to happen", is not supposed to happen. I made a typo in >> CoreExposed.includes, and that's why constructors in my >> PyRawIOBase were being searched reflectively. >> * If PyRawIOBase is properly exposed, __new__ is called and as we >> return, __init__ would be called in some circumstances. >> >> >> Now that __new__ is being called, I can see better how this is >> supposed to work. But I still don't really have the picture in my >> head of the data structure being built and navigated. >> >> Can anyone say where (in Java) MockRawIOWithoutRead.__init__() should >> get called from, and when? Is that done in the generated code defined >> in Python or in support code written in Java? >> >> Jeff >> >> |
From: Jeff A. <ja...@fa...> - 2012-11-02 09:19:38
|
Test failures in the revised approach to PEP-3116 IO are now slightly lower than when I started work. I have (I think) passed through a phase of being thoroughly confused by the interplay of inheritance (in Java, Python, CPython and Jython), PEP-3116 layering, and delegation to classes from org.python.core.io. In the end, I had to make a big spreadsheet of all the parts I'd worked on and how they implement their methods. I think I have learned that the only safe thing is to mimick the Python reference implementation _pyio as closely as possible. I have been following a design, outlined at the start of this thread, that involves delegation to the implementation Philip Jenvey has given us in org.python.core.io. I'm now convinced this won't work as planned. Philip's implementation does indeed mimick the _pyio one well (or differences that may have appeared in _pyio after his work could easily be added), but the overriding of abstract or stub methods in base classes is only available to Java, whereas it must be possible to override them in Python. As an isolated example, a call to flush() occurring in FileIO.close() must invoke a flush() defined by a Python subclass of _io.FileIO. The only solution I see is for my class to provide its own implementation of _io.FileIO.close(), rather than delegate to Philip's, and the same logic applied across all classes an methods means my implementation in org.python.modules._io has to stand alone. Philip's code is certainly a good model for it, but it doesn't seem possible to delegate to it. (There may be something about *Derived classes that I'm missing.) Given the test scores, I'm at the point where pushing to the repo would do slightly more good than harm, technically. However, I can see major change on the way. What would be most useful? Jeff Allen |
From: Philip J. <pj...@un...> - 2012-11-02 20:01:03
|
On Nov 2, 2012, at 2:18 AM, Jeff Allen wrote: > Test failures in the revised approach to PEP-3116 IO are now slightly lower than when I started work. > > I have (I think) passed through a phase of being thoroughly confused by the interplay of inheritance (in Java, Python, CPython and Jython), PEP-3116 layering, and delegation to classes from org.python.core.io. In the end, I had to make a big spreadsheet of all the parts I'd worked on and how they implement their methods. I think I have learned that the only safe thing is to mimick the Python reference implementation _pyio as closely as possible. > > I have been following a design, outlined at the start of this thread, that involves delegation to the implementation Philip Jenvey has given us in org.python.core.io. I'm now convinced this won't work as planned. Philip's implementation does indeed mimick the _pyio one well (or differences that may have appeared in _pyio after his work could easily be added), but the overriding of abstract or stub methods in base classes is only available to Java, whereas it must be possible to override them in Python. As an isolated example, a call to flush() occurring in FileIO.close() must invoke a flush() defined by a Python subclass of _io.FileIO. The only solution I see is for my class to provide its own implementation of _io.FileIO.close(), rather than delegate to Philip's, and the same logic applied across all classes an methods means my implementation in org.python.modules._io has to stand alone. Philip's code is certainly a good model for it, but it doesn't seem possible to delegate to it. (There may be something about *Derived classes that I'm missing.) > > Given the test scores, I'm at the point where pushing to the repo would do slightly more good than harm, technically. However, I can see major change on the way. What would be most useful? Jeff, it'd be great if you could put the code up, say on your bitbucket account, then we could probably have a better look -- Philip Jenvey |
From: Jeff A. <ja...@fa...> - 2012-11-03 19:42:14
|
On 02/11/2012 19:39, Philip Jenvey wrote: > On Nov 2, 2012, at 2:18 AM, Jeff Allen wrote: > >> I have been following a design, outlined at the start of this thread, that involves delegation to the implementation Philip Jenvey has given us in org.python.core.io. I'm now convinced this won't work as planned. ... > Jeff, it'd be great if you could put the code up, say on your bitbucket account, then we could probably have a better look > I now have a Bitbucket account. (What great facility once over the learning curve!) I have pushed the code I am working on to this repo: https://bitbucket.org/tournesol/jython-ja-review I'd be grateful for thoughts about whether the delegation pattern that seemed promising at the start can deliver the inheritance behaviour we need. Jeff |
From: Jeff A. <ja...@fa...> - 2012-11-15 20:01:32
|
On 02/11/2012 19:39, Philip Jenvey wrote: >> Jeff, it'd be great if you could put the code up, say on your bitbucket account, then we could probably have a better look >> > I now have a Bitbucket account. (What great facility once over the > learning curve!) I have pushed the code I am working on to this repo: > https://bitbucket.org/tournesol/jython-ja-review In case anyone was still thinking of it, don't bother reviewing what is up there. In a few days it will change again, probably with new questions from me. I have taken a new direction on this: delegation works ok with the most derived java class (PyFileIO, the implementation of _io.FileIO) delegating some actions to ...core.io.FileIO, but I can't satisfactorily use this trick in PyIOBase and PyRawIOBase because everything interesting there has to be able to "invoke" methods overridden in Python, while ...core.io.FileIO calls methods that can only ever be overridden in Java. I think I finally understand how *Derived classes fit in, and that they don't quite solve this problem. (It's been a long time dawning.) I defined methods in the *Derived classes to mirror most of the Python-overridable methods. It works, but feels overly manual, and has few parallels elsewhere in the core so perhaps I haven't found the smartest way. I do understand the incl: tag, so I'm only defining things once. These false starts make the stack of change sets I have into a bit of muddle. When I prepare this stored work to be pushed to the core, I'll make it look like I got it right first time. ;-) Jeff |
From: Jeff A. <ja...@fa...> - 2012-11-18 00:03:51
|
On 15/11/2012 19:58, Jeff Allen wrote: > On 02/11/2012 19:39, Philip Jenvey wrote: >>> Jeff, it'd be great if you could put the code up, say on your bitbucket account, then we could probably have a better look >>> >> I now have a Bitbucket account. (What great facility once over the >> learning curve!) I have pushed the code I am working on to this repo: >> https://bitbucket.org/tournesol/jython-ja-review For those interested in commenting, it would now be worth taking a look. I've pulled through the revised delegation idea and it seems to work. I'm getting substantially fewer errors from test_io now (13 fail, 18 error) although this is after adding a skip for lack of POSIX signals on Windows. I hand-crafted a lot of wrappers in _io._IOBase.derived and _io._RawIOBase.derived. I'd like to know if that was really necessary, or if I'm still missing something about *Derived classes. Have I created the _io.UnsupportedOperation exception type correctly (in _io.java)? When thrown, I get an ArrayIndexOutOfBoundsException during its raising into Python. I suspect a bug in the core. Jeff |
From: Jeff A. <ja...@fa...> - 2012-12-09 21:33:33
|
I've concluded that my direction of travel must be ok, and have pushed a bunch of material today. It has been quite helpful to re-work that material in smaller increments. I've noticed that a few of the failures in test_io are because the wrong type of exception is being raised. Also, test_fileio makes extensive accommodations to the fact that Jython throws a ValueError when CPython would throw an IOError. This mostly occurs in tests on an IO object where the underlying file has been closed by someone "behind its back". I've scratched my head most of the day about why this became necessary and can say with some confidence that it comes down to calling checkClosed() (or _checkClosed()) at the start of operations where CPython does not. checkClosed() raises ValueError, and checkReadable(), checkWritable() and checkSeekable() raise IOError. (These are correct behaviours in themselves.) I've ventured a little way down the road of a possible solution. I think I could make Jython behave the same as CPython, and remove the concessions and skips that allow us to get away with the divergence, by removing or re-thinking these tests. Each operation has its nuances to be dealt with case by case, but that's the essence of it. Quite a few of these checks are in io.FileIO and ancestors, rather than _io.PyFileIO. I'm trying to change the former as little as I can, but this would change nearly every method (in a small way). Does anyone (Philip particularly) foresee impacts to other code that I might not notice in testing? Jeff Allen On 03/11/2012 19:40, Jeff Allen wrote: > On 02/11/2012 19:39, Philip Jenvey wrote: >> On Nov 2, 2012, at 2:18 AM, Jeff Allen wrote: >> >>> I have been following a design, outlined at the start of this thread, that involves delegation to the implementation Philip Jenvey has given us in org.python.core.io. I'm now convinced this won't work as planned. ... >> Jeff, it'd be great if you could put the code up, say on your bitbucket account, then we could probably have a better look >> > I now have a Bitbucket account. (What great facility once over the > learning curve!) I have pushed the code I am working on to this repo: > https://bitbucket.org/tournesol/jython-ja-review > > I'd be grateful for thoughts about whether the delegation pattern that > seemed promising at the start can deliver the inheritance behaviour we need. > > Jeff > > ------------------------------------------------------------------------------ > LogMeIn Central: Instant, anywhere, Remote PC access and management. > Stay in control, update software, and manage PCs from one command center > Diagnose problems and improve visibility into emerging IT issues > Automate, monitor and manage. Do more in less time with Central > http://p.sf.net/sfu/logmein12331_d2d > _______________________________________________ > Jython-dev mailing list > Jyt...@li... > https://lists.sourceforge.net/lists/listinfo/jython-dev > |
From: <fwi...@gm...> - 2012-12-12 00:09:46
|
On Sun, Dec 9, 2012 at 1:32 PM, Jeff Allen <ja...py...@fa...> wrote: > I've ventured a little way down the road of a possible solution. I think I > could make Jython behave the same as CPython, and remove the concessions and > skips that allow us to get away with the divergence, by removing or > re-thinking these tests. Each operation has its nuances to be dealt with > case by case, but that's the essence of it. > > Quite a few of these checks are in io.FileIO and ancestors, rather than > _io.PyFileIO. I'm trying to change the former as little as I can, but this > would change nearly every method (in a small way). Does anyone (Philip > particularly) foresee impacts to other code that I might not notice in > testing? Philip would be best at answering here, but I for one am very glad you are having a look at this. Every little bit of compatibility helps. BTW I'm not at all sure that this is relevant to the exact issue you are looking at, but just in case you hadn't seen it, the OS and IO exception hierarchy got an overhaul in 3.3: http://docs.python.org/3/whatsnew/3.3.html#pep-3151-reworking-the-os-and-io-exception-hierarchy Thanks again for looking at these issues! -Frank |
From: Jeff A. <ja...@fa...> - 2012-12-12 23:59:05
|
On 12/12/2012 00:09, fwi...@gm... wrote: > On Sun, Dec 9, 2012 at 1:32 PM, Jeff Allen<ja...py...@fa...> wrote: >> I've ventured a little way down the road of a possible solution. I think I >> could make Jython behave the same as CPython, and remove the concessions and >> skips that allow us to get away with the divergence, by removing or >> re-thinking these tests. Each operation has its nuances to be dealt with >> case by case, but that's the essence of it. >> >> Quite a few of these checks are in io.FileIO and ancestors, rather than >> _io.PyFileIO. I'm trying to change the former as little as I can, but this >> would change nearly every method (in a small way). Does anyone (Philip >> particularly) foresee impacts to other code that I might not notice in >> testing? > Philip would be best at answering here, but I for one am very glad you > are having a look at this. Every little bit of compatibility helps. > BTW I'm not at all sure that this is relevant to the exact issue you > are looking at, but just in case you hadn't seen it, the OS and IO > exception hierarchy got an overhaul in 3.3: > http://docs.python.org/3/whatsnew/3.3.html#pep-3151-reworking-the-os-and-io-exception-hierarchy > > Thanks again for looking at these issues! > > -Frank I find the only way to get past the tests is to stick very closely to the way CPython has implemented it. I'm allowing myself some small divergence in implementation where it seems too good to miss, but usually it bites me. The IO and OS hierarchy rework in 3.3 will not help with the ValueError vs. IOError problem. I think I've got this straight in _io.FileIO itself now. It's confusing because CPython _io.FileIO does not actually use the _check* methods from _io._IOBase that call readable() and chums: it uses state it collected from the open mode string. I think the remaining ValueError vs. IOError problems stem from core.io.FileIO: I'm treating it as the file descriptor that _io.FileIO wraps, and as such, if it is closed, I should get an IOError. But it wants to behave like _io.FileIO and give me a ValueError. There's a dissonance between CPython and Jython with the core.io classes wanting to be at the same time a file descriptor and mimic the behaviour of _io. This is very difficult, but a good educational puzzle. Soon I'll have to stop whining about how the core is so complex and undocumented and have to start feeling guilty myself for not documenting it. Jeff |
From: Philip J. <pj...@un...> - 2012-12-16 21:07:26
|
On Dec 12, 2012, at 3:57 PM, Jeff Allen wrote: > On 12/12/2012 00:09, fwi...@gm... wrote: >> On Sun, Dec 9, 2012 at 1:32 PM, Jeff Allen<ja...py...@fa...> wrote: >>> I've ventured a little way down the road of a possible solution. I think I >>> could make Jython behave the same as CPython, and remove the concessions and >>> skips that allow us to get away with the divergence, by removing or >>> re-thinking these tests. Each operation has its nuances to be dealt with >>> case by case, but that's the essence of it. >>> >>> Quite a few of these checks are in io.FileIO and ancestors, rather than >>> _io.PyFileIO. I'm trying to change the former as little as I can, but this >>> would change nearly every method (in a small way). Does anyone (Philip >>> particularly) foresee impacts to other code that I might not notice in >>> testing? >> Philip would be best at answering here, but I for one am very glad you >> are having a look at this. Every little bit of compatibility helps. >> BTW I'm not at all sure that this is relevant to the exact issue you >> are looking at, but just in case you hadn't seen it, the OS and IO >> exception hierarchy got an overhaul in 3.3: >> http://docs.python.org/3/whatsnew/3.3.html#pep-3151-reworking-the-os-and-io-exception-hierarchy >> >> Thanks again for looking at these issues! >> >> -Frank > I find the only way to get past the tests is to stick very closely to > the way CPython has implemented it. I'm allowing myself some small > divergence in implementation where it seems too good to miss, but > usually it bites me. The IO and OS hierarchy rework in 3.3 will not help > with the ValueError vs. IOError problem. I think I've got this straight > in _io.FileIO itself now. It's confusing because CPython _io.FileIO does > not actually use the _check* methods from _io._IOBase that call > readable() and chums: it uses state it collected from the open mode string. Sounds like you've mostly got it figured out now. I think the the Python 2.6 pure Python FileIO would call into IOBase's check methods (I based this impl off of that, this was before it was ported to C), then I think this was changed in 2.7. So basically I'd recommend trying to change it to match CPython as closely as possible -- Philip Jenvey |