[Swig-devel] [ swig-Bugs-1863647 ] Multiple modules leads to unhandled exception

[SourceForge.net <no...@so...>]

Bugs item #1863647, was opened at 2008-01-03 23:25
Message generated for change (Comment added) made by benwebb
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=101645&aid=1863647&group_id=1645

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: code generation (general)
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Ben Webb (benwebb)
Assigned to: Nobody/Anonymous (nobody)
Summary: Multiple modules leads to unhandled exception

Initial Comment:
When using methods from multiple methods, Python crashes with "terminate called after throwing an 
instance of 'swig::stop_iteration'". Here is a simple example:

module1.i:
%module module1
%include "std_vector.i"
%template(vectori) std::vector<int>;

module2.i:
%module module2
%include "std_vector.i"
%template(vectorf) std::vector<float>;

Build with something like (this is with swig-1.3.33, Fedora 8):
swig -python -c++ module1.i
g++ -shared -I/usr/include/python2.5 -o _module1.so module1_wrap.cxx
swig -python -c++ module2.i
g++ -shared -I/usr/include/python2.5 -o _module2.so module2_wrap.cxx

Then run the following Python script:
import module1
import module2
a = module1.vectori()
for b in a: print b
a = module2.vectorf()
for b in a: print b

This crashes every time for me with:
terminate called after throwing an instance of 'swig::stop_iteration'

Incidentally, the following script always crashes for me too:
import module1
import module2
a = module1.vectori()
for b in a: print b

but the following always works:
import module2
import module1
a = module1.vectori()
for b in a: print b

So it looks to me as if one module is overriding the symbols of the other, and indeed, both modules are exporting thesame symbols:
$ nm -C _module1.so |grep stop
000242c8 V typeinfo for swig::stop_iteration
000242d0 V typeinfo name for swig::stop_iteration
$ nm -C _module2.so |grep stop
00024648 V typeinfo for swig::stop_iteration
00024650 V typeinfo name for swig::stop_iteration

On an Intel Mac (10.4) all of the examples work, but nm again shows that both dynamic libraries have the same symbols.

----------------------------------------------------------------------

>Comment By: Ben Webb (benwebb)
Date: 2008-01-12 22:35

Message:
Logged In: YES 
user_id=69439
Originator: YES

Unfortunately the #define workaround seems to break some part of the SWIG
type system. I'll poke around a little, but maybe others can shed some
light on it. So now the situation is as follows:

module1.i:
%module module1
#define PySwigIterator mod1_PySwigIterator
%include "std_vector.i"
%template(vectori) std::vector<int>;

Build with, e.g.
swig -python -c++ module1.i
g++ -g -shared -fPIC -I/usr/include/python2.5 -o _module1.so
module1_wrap.cxx

Then run test.py:
import module1
module1.vectori().begin()

This gives the output:
swig/python detected a memory leak of type '(null)', no destructor found.

This is irritating on my Linux box, but a showstopper on my Solaris box,
since its printf() tries to dereference the null pointer and segfaults. My
guess is the mod1_PySwigIterator destructor isn't being called where it
should, but I don't know why. Without the #define all proceeds normally
(but of course then the original two-module problem occurs).

----------------------------------------------------------------------

Comment By: Ben Webb (benwebb)
Date: 2008-01-04 17:49

Message:
Logged In: YES 
user_id=69439
Originator: YES

Thanks for the informative response - you are of course absolutely right.
I wasn't aware that Python did dlopen with RTLD_LOCAL, so that indeed
explains the behavior.

The #define workaround works for us. It seems to me that another possible
"more elegant" fix would be to split the SWIG type table into two, one of
which is shared in the usual way with all other modules with the same
runtime version and SWIG_TYPE_TABLE define, and the other which is not
shared at all. Obviously PySwigIterator would go in the second table. (It
seems odd to propose a module-specific name for PySwigIterator and then
share it globally.) But I don't know if this is feasible with the SWIG
design and the supported scripting languages.

----------------------------------------------------------------------

Comment By: Josh Cherry (jlcherry)
Date: 2008-01-04 17:04

Message:
Logged In: YES 
user_id=957678
Originator: NO

The symbols in one module are *not* overriding those of the other, which
is why things don't work.  These are weak symbols, as indicated by the 'V'.
 The intent is that just one typeinfo is used.  That way if, for example, a
stop_iteration from one shared library is thrown, and code in another
library tries to catch it, the catch will succeed since the type
information will compare equal.  By default, Python loads shared libraries
in such a way that they don't see each other's symbols.  If you change that
behavior, e.g., by doing

import sys, dl
sys.setdlopenflags(sys.getdlopenflags() | dl.RTLD_GLOBAL)

before importing your modules, you won't get the crash (I'm not suggesting
this as a practical solution, but if you do use it you probably want to
restore the original settings afterwards).

The issue arises because both modules wrap swig::PySwigIterator.  The
shadow object returned for the iterator is a module2.PySwigIterator, even
though it comes from a module1 object (you can see this by printing
iter(a)).  Calling its methods thus leads to calls into _modules2.so. 
Making the iterator class names different will fix the problem.  For
example, adding

#define PySwigIterator module1_PySwigIterator

as the second line of module1.i will work around the problem.  A more
elegant fix might involve file-specific namespaces.



----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=101645&aid=1863647&group_id=1645