[Jython-bugs] [ jython-Bugs-1735864 ] Multithreading errors

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Bugs item #1735864, was opened at 2007-06-12 18:18
Message generated for change (Comment added) made by otmarhumbel
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=112867&aid=1735864&group_id=12867

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Core
Group: targeted for 2.2rc1
Status: Open
Resolution: None
Priority: 6
Private: No
Submitted By: Otmar Humbel (otmarhumbel)
Assigned to: Otmar Humbel (otmarhumbel)
Summary: Multithreading errors

Initial Comment:
Running PythonInterpreters in multiple java threads leads to non-deterministic ImportErrors and NameErrors in scripts which 'otherwise' are just fine.

An initial discussion can be found here:
http://www.nabble.com/Using-Jython-in-separate-threads-tf3263616.html

----------------------------------------------------------------------

>Comment By: Otmar Humbel (otmarhumbel)
Date: 2007-06-18 00:29

Message:
Logged In: YES 
user_id=105844
Originator: YES

The no_string_node_cache.patch solves the problem on my laptop.
I ran the same test as before, without any failures any more.

----------------------------------------------------------------------

Comment By: Charles Groves (cgroves)
Date: 2007-06-17 08:59

Message:
Logged In: YES 
user_id=1174327
Originator: NO

I think the patch I'm attaching now may fix this.  TreeBuilder.java kept a
single instance of each node type in a static nodes array, and just passed
out the correct instance for each id as it was requested.  For JJTNAME,
JJTSTRING, and JJTNUM the actual value of the node is stored in the image
field of the returned node, so sharing those between threads would cause
values to get swapped around.  The patch just returns a new node when any
of those types come in rather than caching it.

The test failed reliably for me with 50 threads, but doesn't fail at all
anymore.
File Added: no_string_node_cache.patch

----------------------------------------------------------------------

Comment By: Charles Groves (cgroves)
Date: 2007-06-15 18:14

Message:
Logged In: YES 
user_id=1174327
Originator: NO

I can't see how LiteralMakerForParser would cause any threading errors. 
It doesn't store state between method invocations, so it should be safe to
call from multiple threads.  Of course, I haven't found anything else
that's shared between compilation instances.

----------------------------------------------------------------------

Comment By: Otmar Humbel (otmarhumbel)
Date: 2007-06-15 16:30

Message:
Logged In: YES 
user_id=105844
Originator: YES

The field:
  private static IParserHost literalMkrForParser = new
LiteralMakerForParser()
in org.python.parser.java is highly under suspicion:
If I make it ThreadLocal, the situation dramatically improves.
I am investigating further.

----------------------------------------------------------------------

Comment By: Otmar Humbel (otmarhumbel)
Date: 2007-06-15 09:31

Message:
Logged In: YES 
user_id=105844
Originator: YES

I also had ThreadLocal under suspicion
(http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6550283), but switching
back to the pre 1.2 ThreadStateMapping made no difference.

----------------------------------------------------------------------

Comment By: Otmar Humbel (otmarhumbel)
Date: 2007-06-15 09:16

Message:
Logged In: YES 
user_id=105844
Originator: YES

The ImportErrors are compile time errors: There are wrong import
statements generated in CodeCompiler.java, as documented in
codecompiler.txt

----------------------------------------------------------------------

Comment By: Otmar Humbel (otmarhumbel)
Date: 2007-06-14 23:40

Message:
Logged In: YES 
user_id=105844
Originator: YES

>From debugging, I found that completely messed up import statements are
issued from PyFrame (see stacktraces.txt):
  "from urllib.swing import JButton"
  "from List.util import List"
It looks like on part originates from one line, and the other part from
another line of test.py

File Added: stacktraces.txt

----------------------------------------------------------------------

Comment By: Otmar Humbel (otmarhumbel)
Date: 2007-06-14 23:21

Message:
Logged In: YES 
user_id=105844
Originator: YES

File Added: test.py

----------------------------------------------------------------------

Comment By: Otmar Humbel (otmarhumbel)
Date: 2007-06-14 23:20

Message:
Logged In: YES 
user_id=105844
Originator: YES

I was able to reproduce the effects with the attached test program.
Platforms: Ubuntu 6.10, Windows XP
Processors: single core, single core HT (hyper threading), dual core
JDK's: JDK1.4.2_14, JDK1.5.0_11

The test program takes two options:
 - number of threads to start
 - number of seconds to pause before starting the threads (useful for
remote debugging)

ImportErrors or NameErrors occur between 5 and 50 threads on almost every
platform / JDK combination.
(I was not able to reproduce them with JDK1.6.0_01 on my single core
laptop).

Luckily problems do not go away if a debugger is attached and paused at
ImportErrors.
File Added: Test.java

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=112867&aid=1735864&group_id=12867