jython-dev

[Jython-dev] Jython and native java merged with simplicity

[john c. <joh...@ya...>]

After examining the Jython JAPI, it is quite clear
that the internal operation of the Interpreter is
quite different.  The jAPI of Jython leverages java's
object oriented nature to implement a Python
equivalent in  java.  The jython API has some
interesting features but has some flaws as well.  I
would like to see these things changed in future
releases of Jython:

It is very easy to create a true Python class in Java
using the Jython Api.  But, there is, as usual a
better way to go about things which would make things
run much faster and with greater flexibility.

Now, one could create a true Python class in java, but
the performance boost is minimal and not worth the
time.

Native java code runs much faster.  Proof of this is
the cPickle module implementation in java.  The speed
of "jPickle" is the same speed as cPickle.

Here is a code comparison for a simple program:
from time import time
def test():
    start = time()
    total = 0.0
    for i in range(0,1000000):
        total = total + i + i*2.0
    print time()- start

This test on my computer:
  python2.0:                     2.77 seconds
  jython2.1a1:                   7.40 seconds
  native java:                   0.014 seconds

197 times faster than Python
528 times faster than Jython

It would be nice if a python class could leverage the
speed of java, while having the flexibility of Python.
We should in theory be able to do this using a Mix-In
approach.

public class jSpam implements PyObject {
    public pyOb = new PyObject();

    public void setOb(PyObject ob) {
        pyOb = ob;
    }
}


import jSpam # true java class 
class Dummy: pass

Class Foo(Dummy,jSpam):
    def __init__(self,name):
        self.name=name

Now, in theory this code should work:

foo=Foo("foo")

This code bombs because inhereting from jSpam has
interfered with our inheritence from Dummy.

The reason it bombs is because we cannot set
attributes on the java class we inherited from.  This
makes sense since java is statically typed. We should
be able to set attributes on the Foo portion of the
object.  Jython does not understand this.


We really need to include a feature in Jython that can
leverage Java's speed with a simple api layer to do
so.
For lack of better terminology I'll call it a ghosting
and skeletons.

Ghosting is the layer in that will provide hooks into
python types mapped into java skeleton classes. So
everytime we change a python attribute, the attribute
is updated in the java skeleton automatically.  This
will in effect pin the Python class to a java class
removing some of the dynamic flexibility we have with
the python classes, but the speed gain is tremendous,
so worth our while.

so in java maybe we have:

   public class mySkeleton implements ghosting

the Ghosting interface are the hooks needed by Jython
to map a Python object to it's skeleton in java

perhaps syntax in Jython would look like:
   from spam import Spam with jSpam skeleton

   This would import Spam from the spam module and
   associate the Spam class with the jSpam skeleton.

Alternate syntax may look like:
   from spam import Spam with jSpam skeleton as Spam1
   from spam import Spam with jSpam2 skeleton as Spam2

Where Spam1 and Spam2 have been linked with different
java skeletons.

A system such as this would provide an easy way to
utilize java's speed and enable Jython to run several
times faster than cPython where computationally
intensive algorithms could be done in the native java
skeleton class.

John Coppola


__________________________________________________
Do You Yahoo!?
Make international calls for as low as $.04/minute with Yahoo! Messenger
http://phonecard.yahoo.com/

Re: [Jython-dev] Jython and native java merged with simplicity

[Titus B. <ti...@ca...>]

-> so in java maybe we have:
-> 
->    public class mySkeleton implements ghosting

[ munch ]

Hi, John,

there is a problem with adding keywords into Jython that aren't
in CPython: then Jython is no longer "Python", but is rather
"Python-like".

In my project, I'm writing code that works in both.  It's not clear
in the end how necessary this approach will be, or how useful, but
it's something to keep in mind.

As far as speed: why not simply "harden" Jython classes into
Java when you have it all worked out?  My plan is to use
Jython partly as a rapid prototyping tool, and then (as necessary)
rewrite classes into Java.

--t

Re: [Jython-dev] Jython and native java merged with simplicity

[Samuele P. <pe...@in...>]

Hi, I found your mail a bit confusing, maybe there was just too
much meta comments <wink>.

I see three different points:
- jython is slow vs. java
- subclassing a PyObject subclass from jython have problem with mutability of
the instances of the resulting class
  This should be solved, probably in 2.2, given that to have a jython 2.2 we
should somehow implement the descr changes
  happened to CPython
- a way to solve (partially) both the above problems, although in my opinion
they are unrelated.

In any case adding new keywords and strange semantics is a no-go.

Although I have maybe understood what was the idea behind your words, I don't
want to try to re-explain it confusing
things further, so please please restate it more clearly...

in any case this is an open source projects, prosal and patches are always
welcome.

Samuele Pedroni.



----- Original Message -----
From: john coppola <joh...@ya...>
To: <jyt...@li...>
Sent: Tuesday, August 07, 2001 2:03 AM
Subject: [Jython-dev] Jython and native java merged with simplicity


> After examining the Jython JAPI, it is quite clear
> that the internal operation of the Interpreter is
> quite different.  The jAPI of Jython leverages java's
> object oriented nature to implement a Python
> equivalent in  java.  The jython API has some
> interesting features but has some flaws as well.  I
> would like to see these things changed in future
> releases of Jython:
>
> It is very easy to create a true Python class in Java
> using the Jython Api.  But, there is, as usual a
> better way to go about things which would make things
> run much faster and with greater flexibility.
>
> Now, one could create a true Python class in java, but
> the performance boost is minimal and not worth the
> time.
>
> Native java code runs much faster.  Proof of this is
> the cPickle module implementation in java.  The speed
> of "jPickle" is the same speed as cPickle.
>
> Here is a code comparison for a simple program:
> from time import time
> def test():
>     start = time()
>     total = 0.0
>     for i in range(0,1000000):
>         total = total + i + i*2.0
>     print time()- start
>
> This test on my computer:
>   python2.0:                     2.77 seconds
>   jython2.1a1:                   7.40 seconds
>   native java:                   0.014 seconds
>
> 197 times faster than Python
> 528 times faster than Jython
>
> It would be nice if a python class could leverage the
> speed of java, while having the flexibility of Python.
> We should in theory be able to do this using a Mix-In
> approach.
>
> public class jSpam implements PyObject {
>     public pyOb = new PyObject();
>
>     public void setOb(PyObject ob) {
>         pyOb = ob;
>     }
> }
>
>
> import jSpam # true java class
> class Dummy: pass
>
> Class Foo(Dummy,jSpam):
>     def __init__(self,name):
>         self.name=name
>
> Now, in theory this code should work:
>
> foo=Foo("foo")
>
> This code bombs because inhereting from jSpam has
> interfered with our inheritence from Dummy.
>
> The reason it bombs is because we cannot set
> attributes on the java class we inherited from.  This
> makes sense since java is statically typed. We should
> be able to set attributes on the Foo portion of the
> object.  Jython does not understand this.
>
>
> We really need to include a feature in Jython that can
> leverage Java's speed with a simple api layer to do
> so.
> For lack of better terminology I'll call it a ghosting
> and skeletons.
>
> Ghosting is the layer in that will provide hooks into
> python types mapped into java skeleton classes. So
> everytime we change a python attribute, the attribute
> is updated in the java skeleton automatically.  This
> will in effect pin the Python class to a java class
> removing some of the dynamic flexibility we have with
> the python classes, but the speed gain is tremendous,
> so worth our while.
>
> so in java maybe we have:
>
>    public class mySkeleton implements ghosting
>
> the Ghosting interface are the hooks needed by Jython
> to map a Python object to it's skeleton in java
>
> perhaps syntax in Jython would look like:
>    from spam import Spam with jSpam skeleton
>
>    This would import Spam from the spam module and
>    associate the Spam class with the jSpam skeleton.
>
> Alternate syntax may look like:
>    from spam import Spam with jSpam skeleton as Spam1
>    from spam import Spam with jSpam2 skeleton as Spam2
>
> Where Spam1 and Spam2 have been linked with different
> java skeletons.
>
> A system such as this would provide an easy way to
> utilize java's speed and enable Jython to run several
> times faster than cPython where computationally
> intensive algorithms could be done in the native java
> skeleton class.
>
> John Coppola
>
>
> __________________________________________________
> Do You Yahoo!?
> Make international calls for as low as $.04/minute with Yahoo! Messenger
> http://phonecard.yahoo.com/
>
> _______________________________________________
> Jython-dev mailing list
> Jyt...@li...
> http://lists.sourceforge.net/lists/listinfo/jython-dev
>

Re: [Jython-dev] Jython and native java merged with simplicity

[john c. <joh...@ya...>]

I think I will need to create my own customized
branch. Introducing new keywords is probably not a
good idea. However the feature doesn't rely on that, a
customized builtin function could also do the trick
and be catchable by cPython.

The inconsistency in lets say "hardening" code using
java is that the code is no longer compatable with
cPython, even though syntatically it is.  So in an
abstract way, the skeletons and ghosting concept
promotes compatability between Python and Jython by
allowing certain methods to be overridden in the
skeleton java class (ghost class).  The original
Python class could be used in both Jython and Python. 
Skeleton gives Jython a means to map Python data to
native java data so that java's calculation efficiency
could be exploited directly.  So I am in favor of the
skeletons approach.

I believe that there are few things I will have to
include to the Jython api.  When an attribute is set,
first check the class an see if it has a ghost.  Then,
via introspection or reflection, check to see if the
attribute is defined in the ghost class, if so set the
attribute in the ghost class as well as the python
class.  I will need to check the object being set to
see if it supports ghosting  recursively because
ultimately we want to use the java native types (or
ghosts) for speed.

When getting a method, check to see if the attribute
has a ghost.  If so check to see if the method is
defined in the ghost class, if so, invoke the ghost
method.

Fairly simple.  Within my coding ability to pull off.


--- Samuele Pedroni <pe...@in...> wrote:
> Hi, I found your mail a bit confusing, maybe there
> was just too
> much meta comments <wink>.
> 
> I see three different points:
> - jython is slow vs. java
> - subclassing a PyObject subclass from jython have
> problem with mutability of
> the instances of the resulting class
>   This should be solved, probably in 2.2, given that
> to have a jython 2.2 we
> should somehow implement the descr changes
>   happened to CPython
> - a way to solve (partially) both the above
> problems, although in my opinion
> they are unrelated.
> 
> In any case adding new keywords and strange
> semantics is a no-go.
> 
> Although I have maybe understood what was the idea
> behind your words, I don't
> want to try to re-explain it confusing
> things further, so please please restate it more
> clearly...
> 
> in any case this is an open source projects, prosal
> and patches are always
> welcome.
> 
> Samuele Pedroni.
> 
> 
> 
> ----- Original Message -----
> From: john coppola <joh...@ya...>
> To: <jyt...@li...>
> Sent: Tuesday, August 07, 2001 2:03 AM
> Subject: [Jython-dev] Jython and native java merged
> with simplicity
> 
> 
> > After examining the Jython JAPI, it is quite clear
> > that the internal operation of the Interpreter is
> > quite different.  The jAPI of Jython leverages
> java's
> > object oriented nature to implement a Python
> > equivalent in  java.  The jython API has some
> > interesting features but has some flaws as well. 
> I
> > would like to see these things changed in future
> > releases of Jython:
> >
> > It is very easy to create a true Python class in
> Java
> > using the Jython Api.  But, there is, as usual a
> > better way to go about things which would make
> things
> > run much faster and with greater flexibility.
> >
> > Now, one could create a true Python class in java,
> but
> > the performance boost is minimal and not worth the
> > time.
> >
> > Native java code runs much faster.  Proof of this
> is
> > the cPickle module implementation in java.  The
> speed
> > of "jPickle" is the same speed as cPickle.
> >
> > Here is a code comparison for a simple program:
> > from time import time
> > def test():
> >     start = time()
> >     total = 0.0
> >     for i in range(0,1000000):
> >         total = total + i + i*2.0
> >     print time()- start
> >
> > This test on my computer:
> >   python2.0:                     2.77 seconds
> >   jython2.1a1:                   7.40 seconds
> >   native java:                   0.014 seconds
> >
> > 197 times faster than Python
> > 528 times faster than Jython
> >
> > It would be nice if a python class could leverage
> the
> > speed of java, while having the flexibility of
> Python.
> > We should in theory be able to do this using a
> Mix-In
> > approach.
> >
> > public class jSpam implements PyObject {
> >     public pyOb = new PyObject();
> >
> >     public void setOb(PyObject ob) {
> >         pyOb = ob;
> >     }
> > }
> >
> >
> > import jSpam # true java class
> > class Dummy: pass
> >
> > Class Foo(Dummy,jSpam):
> >     def __init__(self,name):
> >         self.name=name
> >
> > Now, in theory this code should work:
> >
> > foo=Foo("foo")
> >
> > This code bombs because inhereting from jSpam has
> > interfered with our inheritence from Dummy.
> >
> > The reason it bombs is because we cannot set
> > attributes on the java class we inherited from. 
> This
> > makes sense since java is statically typed. We
> should
> > be able to set attributes on the Foo portion of
> the
> > object.  Jython does not understand this.
> >
> >
> > We really need to include a feature in Jython that
> can
> > leverage Java's speed with a simple api layer to
> do
> > so.
> > For lack of better terminology I'll call it a
> ghosting
> > and skeletons.
> >
> > Ghosting is the layer in that will provide hooks
> into
> > python types mapped into java skeleton classes. So
> > everytime we change a python attribute, the
> attribute
> > is updated in the java skeleton automatically. 
> This
> > will in effect pin the Python class to a java
> class
> > removing some of the dynamic flexibility we have
> with
> > the python classes, but the speed gain is
> tremendous,
> > so worth our while.
> >
> > so in java maybe we have:
> >
> >    public class mySkeleton implements ghosting
> >
> > the Ghosting interface are the hooks needed by
> Jython
> > to map a Python object to it's skeleton in java
> >
> > perhaps syntax in Jython would look like:
> >    from spam import Spam with jSpam skeleton
> >
> >    This would import Spam from the spam module and
> >    associate the Spam class with the jSpam
> skeleton.
> >
> > Alternate syntax may look like:
> >    from spam import Spam with jSpam skeleton as
> Spam1
> >    from spam import Spam with jSpam2 skeleton as
> Spam2
> >
> > Where Spam1 and Spam2 have been linked with
> different
> > java skeletons.
> >
> > A system such as this would provide an easy way to
> > utilize java's speed and enable Jython to run
> several
> > times faster than cPython where computationally
> > intensive algorithms could be done in the native
> java
> > skeleton class.
> >
> > John Coppola
> >
> >
> > __________________________________________________
> > Do You Yahoo!?
> > Make international calls for as low as $.04/minute
> with Yahoo! Messenger
> > http://phonecard.yahoo.com/
> >
> > _______________________________________________
> > Jython-dev mailing list
> > Jyt...@li...
> >
>
http://lists.sourceforge.net/lists/listinfo/jython-dev
> >
> 
=== message truncated ===




__________________________________________________
Do You Yahoo!?
Make international calls for as low as $.04/minute with Yahoo! Messenger
http://phonecard.yahoo.com/

Multiple inheritance, was Re: [Jython-dev] Jython and native java merged with simplicity

[Kevin B. <kb...@ca...>]

john coppola wrote:

> It would be nice if a python class could leverage the
> speed of java, while having the flexibility of Python.
> We should in theory be able to do this using a Mix-In
> approach.

I think the "ghosting" approach is the wrong approach for this.

The example you cited is trivially solved by multiple inheritance, as you pointed out:

...
> import jSpam # true java class
> class Dummy: pass
> 
> Class Foo(Dummy,jSpam):
>     def __init__(self,name):
>         self.name=name
> 
> Now, in theory this code should work:
> 
> foo=Foo("foo")
> 
> This code bombs because inhereting from jSpam has
> interfered with our inheritence from Dummy.

...

So multiple inheritance seems the right way to approach this (it gives you the ghosting feature, all the multiple inheritance/mixins features, and does it in established Python syntax).

Early versions of jpython allowed multiple inheritance of Java classes. Later versions disabled it explicitly, mostly because it wasn't supported in jpythonc. I believe this was a mistake, but I have yet to put my code where my mouth is, so I should probably shut up about it.  But here goes anyway.  :-)

Ideally, Python code should not care whether a class instance is implemented in Python or in Java (or C...), including when you try to inherit from it.

I haven't investigated the code, but I expect it wouldn't be hard to re-enable.

Here's the FAQ entry that talks about it the history:
http://www.jython.org/cgi-bin/faqw.py?req=show&file=faq03.001.htp

This links to: http://mail.python.org/pipermail/jpython-interest/1998-April/000213.html
and a bad link, that should probably be replaced with:
http://mail.python.org/pipermail/jpython-interest/1999-June/001874.html
http://mail.python.org/pipermail/jpython-interest/1999-January/001162.html

I'd propose that we allow the multiple inheritance behavior, and make jythonc encode that behavior in some rational way based on delegation (multiple inheritance in Python doesn't _have_ to make to inheritance in Java, otherwise we couldn't implement Python on the JVM).

This makes jythonc's job a bit harder - but only for classes that use multiple inheritance.  

The basic approach would be to have the derived class implementation inherit from the _first_ base class.  Any other "base classes" become a contained member as shown below. This lets single inheritance generate exactly the code it does today, and supports multiple inheritance in a reasonable way.

Issues (and suggestions):

- access to protected methods/data of base classes (not needed, because Jython doesn't support it)
- derived methods delegating to base classes (provide 'super_method' that calls super.method)
- base class methods that call derived methods - template method pattern (derived method delegates to method on combined class)
- methods with the same name in two base classes (delegate to method on combined class)
- methods of X that access both A and B (delegate to method on combined class)
- data members with the same name in two base classes (really painful - maybe leave that as a restriction in jythonc?)

class X( A, B, C ):
	def MyCMethod( self ):
		# do stuff, then
		return C.MyCMethod( self )

becomes:

public class X extends A
{
	public X( args )
	{
		super( a_args );
		b = new B_X( this, b_args );
		c = new C_X( this, c_args );
	}

	// overridden methods of A, B, and C
	
	public RETURNVALUE MyCMethod()
	{
		// do stuff, then
		return c.super_MyCMethod();
	}

	// non-overridden methods of just B and C delegate to b and c

	// clients need to access fields of B & C and
	// pass to pass B & C references to other code
	// these could be protected w/ accessor methods
	public B b;  
	public C c;

	// may want to add a 'convenience member'
	public A a = this;
}


class B_X extends B
{
	protected X x;
	B_X( X x, b_args )
	{
		super( b_args );
		this.x = x;
	}

	// overridden methods of B delegate to x
}

class C_X extends C
{
	protected X x;
	C_X( X x, c_args )
	{
		super( c_args );
		this.x = x;
	}

	// overridden methods of C delegate to x
	public RETURNTYPE MyCMethod()
	{
		return x.MyCMethod();
	}

	// for any methods that are called from X code, generate the following:
	public RETURNTYPE super_MyCMethod()
	{
		return super.MyCMethod( ARGS );
	}
}

But again, no code to support the proposal at this point, but I'd appreciate feedback on the idea.

:-)  /   :-(

kb

Re: Multiple inheritance, was Re: [Jython-dev] Jython and native java merged with simplicity

[<bc...@wo...>]

[Kevin Butler]

>[...]
>
>So multiple inheritance seems the right way to approach this (it gives 
>you the ghosting feature, all the multiple inheritance/mixins features,
>and does it in established Python syntax).
>
>Early versions of jpython allowed multiple inheritance of Java classes. 
>Later versions disabled it explicitly, mostly because it wasn't supported
>in jpythonc.

I'm not at all sure that was the main motivation.

>I believe this was a mistake, but I have yet to put my code
>where my mouth is, so I should probably shut up about it.  But here goes
>anyway.  :-)
>
>Ideally, Python code should not care whether a class instance is implemented
>in Python or in Java (or C...), including when you try to inherit from it.

Wrong ideal IMO. As long as java classes and instance can be used and
created from jython, I don't mind this kind to semantic differences.

>I haven't investigated the code, but I expect it wouldn't be hard to re-enable.
>
>Here's the FAQ entry that talks about it the history:
>http://www.jython.org/cgi-bin/faqw.py?req=show&file=faq03.001.htp
>
>This links to: http://mail.python.org/pipermail/jpython-interest/1998-April/000213.html
>and a bad link, that should probably be replaced with:

Fixed. Thanks.

>I'd propose that we allow the multiple inheritance behavior, and make jythonc
>encode that behavior in some rational way based on delegation (multiple
>inheritance in Python doesn't _have_ to make to inheritance in Java,
>otherwise we couldn't implement Python on the JVM).

Well, I still can't see a big need for MI.

>This makes jythonc's job a bit harder - but only for classes that use multiple inheritance.  
>
>The basic approach would be to have the derived class implementation inherit from the _first_ base class.  Any other "base classes" become a contained member as shown below. This lets single inheritance generate exactly the code it does today, and supports multiple inheritance in a reasonable way.
>
>Issues (and suggestions):
>
>- access to protected methods/data of base classes (not needed, because Jython doesn't support it)
>- derived methods delegating to base classes (provide 'super_method' that calls super.method)
>- base class methods that call derived methods - template method pattern (derived method delegates to method on combined class)
>- methods with the same name in two base classes (delegate to method on combined class)
>- methods of X that access both A and B (delegate to method on combined class)
>- data members with the same name in two base classes (really painful - maybe leave that as a restriction in jythonc?)
>
>class X( A, B, C ):
>	def MyCMethod( self ):
>		# do stuff, then
>		return C.MyCMethod( self )
>
>becomes:
>
>public class X extends A

I assume that that X, B_X a C_X all implements PyProxy.

>{
>	public X( args )
>	{
>		super( a_args );
>		b = new B_X( this, b_args );
>		c = new C_X( this, c_args );

Keep in mind that the X.__init__ method may call the A, B and C __init__
method at different time with different arguments. The B_X and C_X
instances can't be created at the same time as the A instance. Instead a
reference to the B_X and C_X instances can be stored in the PyInstance
where a reference to the X instance is stored already.

>[...]

regards,
finn

Re: [Jython-dev] Jython and native java merged with simplicity

[<bc...@wo...>]

[john coppola on ghosting and skeletons]

Do you have any code available that I can play with? Based only on your
description, I find it very difficult to understand the philosophy and
goals behind your suggestions.

Assuming that Samuele understod you correctly in the jmerge example, it
seems like the java skeleton should override (or at least hide) the
similar python method. Sort of the same goes for attributes where the
set_/get_ methods in the skeleton is used to intercept attribute access.

Right so far?

I guess that you get a speed increase because the the test for skeleton
is located early in the __findattr__ method. The result of such an early
test is all non-skeleton instances will run a little bit slower, rigth?.
If that is the case, you will have a tough job selling it.

>    public int get_b () {
>        return b;
>    }

I note that references to "b" from python will create a new PyInteger to
hold the returned result. So this particular definition of "get_b"
favors a usage where the skeleton methods makes a lot of changes to "b"
and the python world only retrives the "b" value a few times.

How would a method implemented in the skeleton call a method on the
python instance? Your SpamSkeleton example does not seem to have a
reference to the python instance.

>from time import time
>def test():
>    start = time()
>    total = 0.0
>    for i in range(0,1000000):
>        total = total + i + i*2.0
>    print time()- start
>
>This test on my computer:
>  python2.0:                     2.77 seconds
>  jython2.1a1:                   7.40 seconds
>  native java:                   0.014 seconds

Now, that is the worst test you can possible use and it certainly will
not convince me of anything important. A far better test is the micro
benchmark by Marc-André Lemburg. 

 http://www.lemburg.com/files/python/index.html

(I haven't run this for a while, but older version could run with
jython)

The result of the benchmark can be confusing because of the way hotspot
works, but it clearly show changes (both good and bad) when making
performance related changes to the core. Adding new tests that exercise
a skeleton class should be possible.

Most importantly, I urge you to show some code. You could for instance
make a patch and put in jython's patch manager. 

regards,
finn