jython-users

[Jython-users] Interpreters within Interpreters

[Edward P. <epo...@te...>]

Hello All,

My Jython app creates and uses a few org.python.util.PythonInterpreter's. It
looks something like this:


|-----Shared namespace ------------|
Main app
Interpreter Interpreter Interpreter

So the interpreters and the main app share a namespace.

Can anyone answer these questions:

o If a script executed within one interpreter alters the state of the main
application (ie, a scripted class is added to some application list), and
the interpreter is deleted, what happens.
Is the interpreter really deleted, or is some reference going to be dangling
about.
If it is deleted, is the scripted class 'ok', or will weird and unpleasant
things happen.

o Each interpreter redefines sys.stderr. It seems that stderr is unique
amongst all interpreters within an app. Is there another way to deal with
this? I redefined both sys.stderr via interpreter.exec() calls, and
interpreter.setOut() ... same result. I can think of hacky ways to deal with
this, but there might be an easy solution out there.

o Anything else I should be aware of. From some stuff in the archive, I see
only one person saying that they had problems when interpreters were in diff
threads. But no solutions.

Sorry if these are basic questions, I have been only using Jython for about
2 months, but am having such a blast that it is my main dev language now.

Thanks,
-Ed

Re: [Jython-users] Interpreters within Interpreters

[Ype K. <yk...@xs...>]

Edward,

>Hello All,
>
>My Jython app creates and uses a few org.python.util.PythonInterpreter's. It
>looks something like this:
>
>
>|-----Shared namespace ------------|
>Main app
>Interpreter Interpreter Interpreter
>
>So the interpreters and the main app share a namespace.
>
>Can anyone answer these questions:
>
>o If a script executed within one interpreter alters the state of the main
>application (ie, a scripted class is added to some application list), and
>the interpreter is deleted, what happens.

When the state change is a done in the shared namespace it will stay
there when the interpreter is deleted.

It is a bit tricky to share namespaces this way. An alternative is to
use the namespace of a module shared between the main app and the interpreters.
You can set things in a module from outside the module.
You could for example import a module in the main app and set a variable
in this module from the main app to a shared object. This shared object
can then be also be used from the interpreters after they import the module.

>Is the interpreter really deleted, or is some reference going to be dangling
>about.

That depends on how you implement the shared namespace.
The system uses a the __dict__ member of a module for the namespace
of a module (see module new), and it keeps the modules in sys.modules.

>If it is deleted, is the scripted class 'ok', or will weird and unpleasant
>things happen.

Unpleasant things might happen when you explicitly clear a namespace too early.
Only removing references is safer.

>o Each interpreter redefines sys.stderr. It seems that stderr is unique
>amongst all interpreters within an app. Is there another way to deal with
>this? I redefined both sys.stderr via interpreter.exec() calls, and
>interpreter.setOut() ... same result. I can think of hacky ways to deal with
>this, but there might be an easy solution out there.

With a single shared namespace and  a single sys.stderr object this is not easy.
You'll need some criterion to distinguish the source of the data being
sent to sys.stderr.

>o Anything else I should be aware of. From some stuff in the archive, I see
>only one person saying that they had problems when interpreters were in diff
>threads. But no solutions.

You could use the thread id as the distinguishing criterion. Ie. replace
sys.stderr with an object that uses the thread id to send it's output
to different places. See demos/swing/Console.py on changing sys.stdout
and sys.stderr.

>Sorry if these are basic questions, I have been only using Jython for about
>2 months, but am having such a blast that it is my main dev language now.

They are not a basic questions, but they pop up often enough that a FAQ entry
might be considered.

You might consider using the namespace of a new.module() for each
interpreter and register this module at sys.modules. You'll have to find
an appropriate value for the module name (ie. __name__) to be able
to find each module in sys.modules. Only one at a time can be called
'__main__', unless you override the lookup in sys.modules to include
eg. the thread id.
With a namespace per interpreter you can also put a different sys
object in each interpreter, but this seems a bit drastic too me.

Some modules use their __name__ to find their module in sys.modules
(notably PyUnit) and supporting this is nice.
When each interpreter thread finishes you can remove all the references
from sys.modules to make sure nothing is dangling.
I have some bad experience with clearing such namespaces, so i'd recommend
to only remove the references. The garbage collector will find non
referenced objects in due time.

Finally: when you use exec of execfile() you don't need a PyInterpreter.
To set things in a module name space before it is started
just use module.__dict__['var'] = value.

Good luck,
Ype

-- 

Re: [Jython-users] Interpreters within Interpreters

[Edward P. <epo...@te...>]

Hi Ype,

> It is a bit tricky to share namespaces this way. An alternative is to
> use the namespace of a module shared between the main app and the
interpreters.
> You can set things in a module from outside the module.
I am not sure I see the difference. This is what I am doing:
MainApp:
sharedSpace={'this':this, 'that':that, 'whatever':whatever}
globals().update(sharedSpace)

Then I have 'plugins':
plugin.init(sharedSpace)

In the plugin:
def init(namespace):
    globals().update(namespace)

Now each plugin may create additional interpreters, and the main app
normally has at least one interpreter.

So far, I think this has the same effect as a module. But I see what you
mean:
import sharedSpace:
thatRef = sharedSpace.that
etc.

> Unpleasant things might happen when you explicitly clear a namespace too
early.
> Only removing references is safer.
I am sorry, my python newbieness is showing. Explicity clear a namespace?
If in my plugin I:
del that
then I remove it from the modules namespace. It is still alive elsewhere (I
assume) as the GC won't clean up until I del that from all the namespaces
that contain its reference. Right?

> You could use the thread id as the distinguishing criterion. Ie. replace
> sys.stderr with an object that uses the thread id to send it's output
> to different places. See demos/swing/Console.py on changing sys.stdout
> and sys.stderr.
I considered the thread ID, but not all interpreters are in different
threads.
Perhaps I could 'register' a module/threadID pair as a key, and then look
back at the call stack to see which module using stdout/err. And if no key
exists, then the original stdout is used. Heh, heh, I am not sure I can do
this, just assuming I can peek back into the call stack.

> You might consider using the namespace of a new.module()
is the new module available in Jython? 'import new' doesn't work here.


> Some modules use their __name__ to find their module in sys.modules
> (notably PyUnit) and supporting this is nice.
> When each interpreter thread finishes you can remove all the references
> from sys.modules to make sure nothing is dangling.
Thanks, didn't know about this, will have to look in the docs.

> I have some bad experience with clearing such namespaces, so i'd recommend
> to only remove the references. The garbage collector will find non
> referenced objects in due time.
Again, I am a little confused ... clearing vs removing references. I am
guessing, in an interpreters __del__
I set my variables to None? And leave stuff in sys.modules alone?

> Finally: when you use exec of execfile() you don't need a PyInterpreter.
> To set things in a module name space before it is started
> just use module.__dict__['var'] = value.
At the moment I am just adding a console to my app, so I can interactively
do things - an intermediate step befiore commiting to a user interface.

Thanks for all your tips, seems there is little one can't change in Python
... still trying to get used to this coming from c++/java :)

-Ed

Re: [Jython-users] Interpreters within Interpreters

[Ype K. <yk...@xs...>]

Edward,
>
>> It is a bit tricky to share namespaces this way. An alternative is to
>> use the namespace of a module shared between the main app and the
>interpreters.
>> You can set things in a module from outside the module.
>I am not sure I see the difference. This is what I am doing:
>MainApp:
>sharedSpace={'this':this, 'that':that, 'whatever':whatever}
>globals().update(sharedSpace)

I had the impression you where using globals() as
the shared namespace. Fortunately you are not.

>Then I have 'plugins':
>plugin.init(sharedSpace)
>
>In the plugin:
>def init(namespace):
>    globals().update(namespace)
>
>Now each plugin may create additional interpreters, and the main app
>normally has at least one interpreter.
>
>So far, I think this has the same effect as a module. But I see what you
>mean:
>import sharedSpace:
>thatRef = sharedSpace.that
>etc.

What I meant is:

import sharedSpace
sharedSpace.that = thatRef

From then on 'that' is available in module sharedSpace.

> > Unpleasant things might happen when you explicitly clear a namespace too
>early.
>> Only removing references is safer.
>I am sorry, my python newbieness is showing. Explicity clear a namespace?

Consider what happens when you do a sharedSpace.clear() too early.
The next time a name from this namespace is needed
you will get a NameError from the interpreter. This is what happened to me.
To make things worse I cleared the namespace from one thread
without proper synchronisation with another thread using the namespace.

>If in my plugin I:
>del that
>then I remove it from the modules namespace. It is still alive elsewhere (I
>assume) as the GC won't clean up until I del that from all the namespaces
>that contain its reference. Right?

That's right. But when you share a namespace there is only one shared
namespace. In that case a 'del that' will remove 'that' from each user of the
shared namespace. The first symptom of this might be NameError: that.

> > You could use the thread id as the distinguishing criterion. Ie. replace
>> sys.stderr with an object that uses the thread id to send it's output
>> to different places. See demos/swing/Console.py on changing sys.stdout
>> and sys.stderr.
>I considered the thread ID, but not all interpreters are in different
>threads.
>Perhaps I could 'register' a module/threadID pair as a key, and then look
>back at the call stack to see which module using stdout/err. And if no key
>exists, then the original stdout is used. Heh, heh, I am not sure I can do
>this, just assuming I can peek back into the call stack.

You can peek back by looking at the backtrace of a caught exception.

> > You might consider using the namespace of a new.module()
>is the new module available in Jython? 'import new' doesn't work here.

It works for me, but I don't have the code handy to show
you precisely how. Checks the docs and if this problem persists ask here.

>
>> Some modules use their __name__ to find their module in sys.modules
>> (notably PyUnit) and supporting this is nice.
> > When each interpreter thread finishes you can remove all the references
> > from sys.modules to make sure nothing is dangling.
>Thanks, didn't know about this, will have to look in the docs.

The docs are rather terse, ie. everything is mentioned once or perhaps
twice. The relevant parts are the execution model and the standard types,
quite dry material.

> > I have some bad experience with clearing such namespaces, so i'd recommend
>> to only remove the references. The garbage collector will find non
>> referenced objects in due time.
>Again, I am a little confused ... clearing vs removing references. I am
>guessing, in an interpreters __del__
>I set my variables to None? And leave stuff in sys.modules alone?

To avoid memory leaks after normal use:
- Remove things from sys.modules that you put in there yourself.
- Global vars (in modules or in a shared namespace as you showed above)
  might need to be set to None.
- Local variables disappear after the function returns, so you only
  need to set these to None before you wait a long time in the function.

> > Finally: when you use exec of execfile() you don't need a PyInterpreter.
>> To set things in a module name space before it is started
>> just use module.__dict__['var'] = value.
>At the moment I am just adding a console to my app, so I can interactively
>do things - an intermediate step befiore commiting to a user interface.
>
>Thanks for all your tips, seems there is little one can't change in Python
>... still trying to get used to this coming from c++/java :)

Have fun,
Ype

-- 

Re: [Jython-users] Interpreters within Interpreters

[Edward P. <epo...@te...>]

Thanks Ype, now it all makes sense :)
Now to get it all working!

-Ed