From: Samuele P. <pe...@in...> - 2001-07-26 15:05:57
|
Hi, > > We're working on a project involving ExtensionClasses and have written > some code to that end. (The project is open source, but we haven't > "announced" it yet, because we still need to build up some support > infrastructure.) > > In order to pickle an ExtensionClass PyMetaClass, I had a couple of > changes... > > $ diff PyMetaClass.java ~/jython-20010719/org/python/core/PyMetaClass.java > 5d4 > < public PyObject __basicnew__(); > > > $ diff cPickle.java ~/jython-20010719/org/python/modules/cPickle.java > 996c996 > < else if (object instanceof PyClass) > --- > > else if (cls == ClassType) > 1821,1824c1821 > < if (klass instanceof PyMetaClass) > < value = ((PyMetaClass) klass).__basicnew__(); > < else > < value = new PyInstance((PyClass)klass); > --- > > value = new PyInstance((PyClass)klass); > > Basically, this creates a requirement for a __basicnew__ method in a > PyMetaClass. I'm not sure if there would be a cleaner way to designate what > kind of instance object should be created. This seems like a pretty > straightforward interface. From empty to non-empty is universe creation <wink> Seriously, up now I have had no time to make up my mind on this a bit of patience ... regards, Samuele Pedroni. |
From: Samuele P. <pe...@in...> - 2001-07-30 13:56:53
|
Hi, I have studied a bit the situation comparing pickle, CPython cPickle and Jython cPickle the only bug/difference that I found (related to you issue) is that if (args.length == 0 && klass instanceof PyClass && klass.__findattr__("__getinitargs__") == null) { value = new PyInstance((PyClass)klass); } else { value = klass.__call__(args); } should be: if (args.length == 0 && klass.getClass() == PyClass && klass.__findattr__("__getinitargs__") == null) { value = new PyInstance((PyClass)klass); } else { value = klass.__call__(args); } I will fix that. OTOH I have briefly looked at ExtensionClasses and it seems that this drives pickling through __reduce__ (is that correct?) so I suggest to use the same technique for the moment also in Jython, if you encounter any problem/bug trying to do this let me know. I agree that something similar to your proposal for pickling make sense wrt to Java idioms and PyMetaClass but the latter has a somehow transitory status (for the time being) so I prefer not to add further related/dependent code to Jython. regards, Samuele Pedroni. |
From: Kevin D. <kda...@we...> - 2001-07-30 17:36:24
|
Hi, (comments below) ----- Original Message ----- From: "Samuele Pedroni" <pe...@in...> To: <jyt...@li...>; <kda...@we...> Sent: Monday, July 30, 2001 9:56 AM Subject: Re: [Jython-dev] Pickling for PyMetaClass > Hi, > > I have studied a bit the situation comparing > pickle, CPython cPickle and Jython cPickle > > the only bug/difference that I found (related to you issue) is that > > if (args.length == 0 && klass instanceof PyClass && > klass.__findattr__("__getinitargs__") == null) { > value = new PyInstance((PyClass)klass); > } else { > value = klass.__call__(args); > } > > should be: > > if (args.length == 0 && klass.getClass() == PyClass && > klass.__findattr__("__getinitargs__") == null) { > value = new PyInstance((PyClass)klass); > } else { > value = klass.__call__(args); > } > I will fix that. Hmm. Not sure I understand why the change is needed. (That should be PyClass.class, right?) > OTOH I have briefly looked at ExtensionClasses and it seems that this drives pickling through > __reduce__ (is that correct?) > > so I suggest to use the same technique for the moment also in Jython, > if you encounter any problem/bug trying to do this let me know. Yes, I think you are correct. It looks like C ExtensionClass uses __reduce__. I tried adding __reduce__ to my package, but it never gets called. The EClasses and EInstances look (to the cPickle module) just like PyClasses and PyInstances, so it never gets past the call to save_type. With a small patch to __builtin__.type(), we can get around this: public static PyClass type(PyObject o) { if (o instanceof PyInstance) { return PyJavaClass.lookup(o.getClass()); } else { return o.__class__; } } (replacing PyJavaClass.lookup(PyInstance.class) with PyJavaClass.lookup(o.getClass()) While there is a reduce in the CExtensionClass package, the resulting pickles don't really look like "reduced" pickles. In CPython I get something like this: cectest Point (small amount of binary data in which x, y, and foo show up.) Basically, it's an ectest.Point object which has values for x, y and foo. My __reduce__ returns: (<jextension class ectest.Point at 1491648>, (), {'y': 3, 'x': 2, 'foo': 42}) type(Point) returns org.python.PyClass The pickle I get is: cectest Point q cectest Point (binary data in which all of the variables show up 4 times) My original patch produced output very similar to what you get from CPython. I'll keep looking at cPickle and ExtensionClass to see if there's some bit of behavior I'm missing... > > I agree that something similar to your proposal for pickling make sense wrt to Java idioms > and PyMetaClass but the latter has a somehow transitory status (for the time being) > so I prefer not to add further related/dependent code to Jython. I understand. I haven't been following the Python development timeline... any idea how long it will be until 2.2's semantics are finalized? Kevin |
From: Samuele P. <pe...@in...> - 2001-07-30 18:39:26
|
> > Hi, > > > > I have studied a bit the situation comparing > > pickle, CPython cPickle and Jython cPickle > > > > the only bug/difference that I found (related to you issue) is that > > > > if (args.length == 0 && klass instanceof PyClass && > > klass.__findattr__("__getinitargs__") == null) { > > value = new PyInstance((PyClass)klass); > > } else { > > value = klass.__call__(args); > > } > > > > should be: > > > > if (args.length == 0 && klass.getClass() == PyClass && > > klass.__findattr__("__getinitargs__") == null) { > > value = new PyInstance((PyClass)klass); > > } else { > > value = klass.__call__(args); > > } > > I will fix that. > > Hmm. Not sure I understand why the change is needed. (That should be > PyClass.class, right?) Right, OTOH this fixes a bug but don't solve your problem, I know. > > OTOH I have briefly looked at ExtensionClasses and it seems that this > drives pickling through > > __reduce__ (is that correct?) > > > > so I suggest to use the same technique for the moment also in Jython, > > if you encounter any problem/bug trying to do this let me know. > > Yes, I think you are correct. It looks like C ExtensionClass uses > __reduce__. I tried adding __reduce__ to my package, but it never gets > called. The EClasses and EInstances look (to the cPickle module) just like > PyClasses and PyInstances, so it never gets past the call to save_type. > > With a small patch to __builtin__.type(), we can get around this: > > public static PyClass type(PyObject o) { > if (o instanceof PyInstance) { > return PyJavaClass.lookup(o.getClass()); > } else { > return o.__class__; > } > } > > (replacing PyJavaClass.lookup(PyInstance.class) with > PyJavaClass.lookup(o.getClass()) Yes, I see the point. I should admit that I feel a bit uncomfortable with this design driven by the matter of events and without a whole picture. But the patch can go in with the same transitional status of the PyMetaClass hook. See below ... > While there is a reduce in the CExtensionClass > package, the resulting pickles don't really look like "reduced" pickles. > > In CPython I get something like this: > > cectest > Point > (small amount of binary data in which x, y, and foo show up.) > > Basically, it's an ectest.Point object which has values for x, y and foo. > > My __reduce__ returns: > (<jextension class ectest.Point at 1491648>, (), {'y': 3, 'x': 2, 'foo': > 42}) > > type(Point) returns org.python.PyClass This is strange, any idea why you get this instead of your derived class? > > The pickle I get is: > cectest > Point > q cectest > Point > (binary data in which all of the variables show up 4 times) > > My original patch produced output very similar to what you get from CPython. > I'll keep looking at cPickle and ExtensionClass to see if there's some bit > of behavior I'm missing... Possible, because up to the bug and type() behavior CPython and Jython cPickle code seem to do the same thing... > > > > I agree that something similar to your proposal for pickling make sense > wrt to Java idioms > > and PyMetaClass but the latter has a somehow transitory status (for the > time being) > > so I prefer not to add further related/dependent code to Jython. > > I understand. I haven't been following the Python development timeline... > any idea how long it will be until 2.2's semantics are finalized? It seems that to make things work I should add code :(. About 2.2 semantics, I think their are mostly finalized on CPython side. OTOH I promised GvR to do a revision wrt Jython implications. In any case it will not be implemented in Jython very soon - we should finish 2.1 - and it's a lot of work because adding that without refactorising the old Jython stuff will produce just a mess. regards, Samuele Pedroni. |
From: Kevin D. <kda...@we...> - 2001-07-30 19:21:50
|
----- Original Message ----- From: "Samuele Pedroni" <pe...@in...> To: <jyt...@li...>; <kda...@we...> Sent: Monday, July 30, 2001 2:39 PM Subject: Re: [Jython-dev] Pickling for PyMetaClass > > > if (args.length == 0 && klass.getClass() == PyClass && > > > klass.__findattr__("__getinitargs__") == null) { > > > value = new PyInstance((PyClass)klass); > > > } else { > > > value = klass.__call__(args); > > > } > > > I will fix that. > > > > Hmm. Not sure I understand why the change is needed. (That should be > > PyClass.class, right?) > Right, OTOH this fixes a bug but don't solve your problem, I know. Ahh... now I see why your patch would be useful for the metaclasses. I'll think some more about if there's an approach that doesn't involve specific references to PyMetaClass. > > > OTOH I have briefly looked at ExtensionClasses and it seems that this > > drives pickling through > > > __reduce__ (is that correct?) > > > > > > so I suggest to use the same technique for the moment also in Jython, > > > if you encounter any problem/bug trying to do this let me know. > > > > Yes, I think you are correct. It looks like C ExtensionClass uses > > __reduce__. I tried adding __reduce__ to my package, but it never gets > > called. The EClasses and EInstances look (to the cPickle module) just like > > PyClasses and PyInstances, so it never gets past the call to save_type. > > > > With a small patch to __builtin__.type(), we can get around this: > > > > public static PyClass type(PyObject o) { > > if (o instanceof PyInstance) { > > return PyJavaClass.lookup(o.getClass()); > > } else { > > return o.__class__; > > } > > } > > > > (replacing PyJavaClass.lookup(PyInstance.class) with > > PyJavaClass.lookup(o.getClass()) > Yes, I see the point. I should admit that I feel a bit uncomfortable > with this design driven by the matter of events and without a whole > picture. But the patch can go in with the same transitional status > of the PyMetaClass hook. See below ... It is true that PyMetaClass is transitional... It seems like there would always be some value to being able to subclass PyClass and PyInstance to extend the behavior of those objects (for things like the __of__ protocol). In order for that to generally work, it would seem that it would be useful to use code that doesn't rely on PyClass and PyInstance specifically, but rather instanceof PyClass and PyInstance. > > While there is a reduce in the CExtensionClass > > package, the resulting pickles don't really look like "reduced" pickles. > > > > In CPython I get something like this: > > > > cectest > > Point > > (small amount of binary data in which x, y, and foo show up.) > > > > Basically, it's an ectest.Point object which has values for x, y and foo. > > > > My __reduce__ returns: > > (<jextension class ectest.Point at 1491648>, (), {'y': 3, 'x': 2, 'foo': > > 42}) > > > > type(Point) returns org.python.PyClass > This is strange, any idea why you get this instead of your derived > class? I did that on purpose. I don't think I need to do anything to make the class itself pickle, so I just left __class__ as org.python.core.PyClass. It's the instances that I'm really focusing on. (If I *did* change the class' __class__, I would have had to make a reduce method and would have gotten an even larger difference from the CPython cPickle.) [snip] > > > I agree that something similar to your proposal for pickling make sense > > wrt to Java idioms > > > and PyMetaClass but the latter has a somehow transitory status (for the > > time being) > > > so I prefer not to add further related/dependent code to Jython. > > > > I understand. I haven't been following the Python development timeline... > > any idea how long it will be until 2.2's semantics are finalized? > > It seems that to make things work I should add code :(. I'll keep looking to see if I can figure out the difference in behavior. If cPickle.c doesn't make special allowances for ExtensionClasses, it seems like it should be possible to make cPickle.java do the same. > About 2.2 semantics, I think their are mostly finalized on CPython side. > OTOH I promised GvR to do a revision wrt Jython implications. > In any case it will not be implemented in Jython very soon - we should > finish 2.1 - and it's a lot of work because adding that without refactorising > the old Jython stuff will produce just a mess. Agreed. There are some significant changes proposed in the PEPs. Thanks for the feedback. Kevin |
From: Kevin D. <kda...@we...> - 2001-07-31 20:12:13
|
Good news. A bit of reading through cPickle.c and cPickle.java showed me the way. ExtensionClass.c does, in fact, use reduce, and I discovered that load_reduce looks for __basicnew__ if the args tuple is None. So, that all works fine. (see below...) > > With a small patch to __builtin__.type(), we can get around this: > > > > public static PyClass type(PyObject o) { > > if (o instanceof PyInstance) { > > return PyJavaClass.lookup(o.getClass()); > > } else { > > return o.__class__; > > } > > } > > > > (replacing PyJavaClass.lookup(PyInstance.class) with > > PyJavaClass.lookup(o.getClass()) > Yes, I see the point. I should admit that I feel a bit uncomfortable > with this design driven by the matter of events and without a whole > picture. But the patch can go in with the same transitional status > of the PyMetaClass hook. See below ... This patch is necessary to support user-defined Instance classes and pickling... It looks like no other change is needed for this to work, though. Kevin |
From: Samuele P. <pe...@in...> - 2001-08-08 18:34:43
|
Hi. > > Good news. A bit of reading through cPickle.c and cPickle.java showed me the > way. ExtensionClass.c does, in fact, use reduce, and I discovered that > load_reduce looks for __basicnew__ if the args tuple is None. So, that all > works fine. (see below...) > > > > With a small patch to __builtin__.type(), we can get around this: > > > > > > public static PyClass type(PyObject o) { > > > if (o instanceof PyInstance) { > > > return PyJavaClass.lookup(o.getClass()); > > > } else { > > > return o.__class__; > > > } > > > } > > > > > > (replacing PyJavaClass.lookup(PyInstance.class) with > > > PyJavaClass.lookup(o.getClass()) > > Yes, I see the point. I should admit that I feel a bit uncomfortable > > with this design driven by the matter of events and without a whole > > picture. But the patch can go in with the same transitional status > > of the PyMetaClass hook. See below ... > > This patch is necessary to support user-defined Instance classes and > pickling... It looks like no other change is needed for this to work, > though. > > Kevin > I have commited that. BTW with the experimental status of the other PyMetaClass related stuff. regards, Samuele Pedroni. |