#895 SWIG 1.3.33 does not support multiple Python interpreters

open
nobody
python (259)
5
2012-12-21
2008-02-22
No

As discussed on the mailing list: SWIG does not generate modules compatible with multiple python interpreters.

Bug #1: Infinite loop in the initialization function

Bug #2: There was some issue with the delete function

Bug #3: Each swig module loaded per interpreter leaks memory (horribly if your module is big)

Bug #4: If you start two interpreters and import swig modules in each (simultaneously), after ending the first interpreter, the second interpreter sometimes has cleaned up all of its swig objects, with tragic results

(Start 1
import swiga
(Start 2
import swiga
PyEndInterpreter 1,
interpreter2's swiga may be corrupted.

Bug #4: The static objects created in the object_swigregister functions are shared between interpreters, which allows them to get deleted by one interpreter and have that cross over into the other interpreter.

etc.

Discussion

  • rather_not_disclose

    Mostly working example (hacked SWIG interface) that demonstrates some issues

     
  • John Lenz

    John Lenz - 2008-02-22

    Logged In: YES
    user_id=153408
    Originator: NO

    How many of these are still issues in the latest svn code. We did commit a fix for initialization.

    As far as bug 3, I don't understand. The runtime system never, ever calls malloc anywhere at all. Never once! So I don't really understand how it can leak memory... it never allocates any memory anywhere. The type system was written specifically to remove any need for memory allocation.

    As far as bug 4, the type system has never really worked when modules are unloaded. Not even in ancient versions of swig. In older versions of swig, it is mostly luck that you can unload modules... there was never any code to properly unhook and remove types. I had some proposals a long time ago about how to remove modules, but it is a very tricky problem, and I was never convinced they would work properly.

     
  • rather_not_disclose

    Logged In: YES
    user_id=1598900
    Originator: YES

    For number 3: I should have recast this as 'creates references that are leaked'

    Martin v. Lowis and I did an experiment -- here were the results.
    I'll try and see if the experiemnt

    > Other than that, I'm at a loss as to where the memory is going.

    I put the following block into testfile.py:

    import gc
    gc.collect()
    counts = {}
    for o in gc.get_objects():
    o = type(o)
    counts[o] = counts.get(o,0)+1
    counts = [(b,a) for a,b in counts.items()]
    counts.sort()
    for a,b in counts[-10:]:
    print b.__name__,a

    From two subsequent iterations, I get

    Starting 92
    LEAK/INUSECHECK
    member_descriptor 85
    type 109
    method_descriptor 163
    weakref 186
    getset_descriptor 230
    wrapper_descriptor 339
    builtin_function_or_method 395
    dict 662
    function 1143
    tuple 3813
    Starting 93
    HELLOW
    Starting 93
    LEAK/INUSECHECK
    member_descriptor 85
    type 110
    method_descriptor 163
    weakref 187
    getset_descriptor 232
    wrapper_descriptor 339
    builtin_function_or_method 396
    dict 667
    function 1151
    tuple 3838

    So you can see that you gain per round
    - 1 type
    - 1 method_descriptor
    - 1 weakref
    - 2 getset_descriptor
    - 1 builtin_function_or_method
    - 5 dict
    - 8 function
    - 25 tuple

    I haven't analysed all of them, but for the types, it is easy to print all type.__name__s, and this shows that the type that you gain is MemoryHog; further investigation shows that it is memoryhoginterface.MemoryHog (I renamed it, and then the renamed type got leaked). I haven't looked into the SWIG code too much, yet, but chances are that MemoryHog_swigregister loses references.

    I'm particularly puzzled that PySwigClientData_Del doesn't Py_DECREF(data->klass). OTOH, PySwigClientData_Del isn't ever called.

     
  • rather_not_disclose

    Logged In: YES
    user_id=1598900
    Originator: YES

    I downloaded current sources as of today:

    #1 is fixed in CVS

    #2 is NOT fixed in CVS. The only way that I could get my example to work (it creates an interpreter, initializes it, runs a function in the swig module, and then ends the interpreter 100x) was to comment out the line: // Py_DECREF(SWIG_This());
    Otherwise it will eventually core dump on (usually) the third interpreter.

    #3 not sure -- I don't see any way to reclaim the swigregister functions and a variety of other ones.

     
  • rather_not_disclose

    Logged In: YES
    user_id=1598900
    Originator: YES

    Thanks for your bug report. I can reproduce this and I'm
    investigating.

     

Log in to post a comment.