#1301 python director bug with -builtin

None
closed
nobody
None
5
2013-07-01
2013-01-17
No

We found a bug in the director code for python with the -builtin option.
It looks like the heap gets corrupted (double free?)
we're using swig 2.0.8 and python 2.7.1

You can recreate the problem with:

$ cat sample.i
%module(directors="1") "sample";
%feature("director") Foo;

%inline %{
class Foo {
  public:
    Foo() {}
    virtual ~Foo() {}
};
%}

$ swig -builtin -O -python -c++ -o sample_wrap.cc sample.i
$ g++- -ggdb -fPIC -I/usr/share/python/2.7.1/linux64/include/python2.7 -c -o sample_wrap.o sample_wrap.cc
$ g++ -shared -o _sample.so sample_wrap.o

$ cat test.py
import sample

class MyFoo(sample.Foo):
    pass

for i in range(1000):
    print i
    MyFoo()

$ python test.py
1
2
....
49
50
Segmentation fault

Depending on what we do, sometimes the crash happens in the python garage collector, during python termination, or when creating/deleting new objects

It appears to me that we might have a corrupted heap due to a double free but I haven't been able to track it down to root cause yet.

The code works correctly without the -builtin option.

Discussion

  • SchoenleAndi

    SchoenleAndi - 2013-06-20

    I think we figured out partly what is going on:
    When the object is constructed, PyType_GenericAlloc is called

    python27_d.dll!PyType_GenericAlloc(_typeobject * type=0x01ed4060, int nitems=0)  Line 754   C
    python27_d.dll!PyType_GenericNew(_typeobject * type=0x01ed4060, _object * args=0x005b1038, _object * kwds=0x00000000)  Line 778 + 0x11 bytes    
    python27_d.dll!type_call(_typeobject * type=0x01ed4060, _object * args=0x005b1038, _object * kwds=0x00000000)  Line 721 + 0x17 bytes    C
    python27_d.dll!PyObject_Call(_object * func=0x01ed4060, _object * arg=0x005b1038, _object * kw=0x00000000)  Line 2529 + 0xf bytes   C
    python27_d.dll!do_call(_object * func=0x01ed4060, _object * * * pp_stack=0x0027fca0, int na=0, int nk=0)  Line 4231 + 0x8 bytes C
    python27_d.dll!call_function(_object * * * pp_stack=0x0027fca0, int oparg=0)  Line 4036 + 0x7 bytes C
    

    Inside PyType_GenericAlloc() the function

    PyObject *
    PyType_GenericAlloc(PyTypeObject *type, Py_ssize_t nitems)
    {
        PyObject *obj;
        const size_t size = _PyObject_VAR_SIZE(type, nitems+1);
        /* note that we need to add one, for the sentinel */
    
        if (PyType_IS_GC(type))
            obj = _PyObject_GC_Malloc(size);
        else
            obj = (PyObject *)PyObject_MALLOC(size);
    

    it drops into _PyObject_GC_Malloc(size) if a director class has been extended.
    But then, when going into the delete wrapper:

    #define SWIGPY_DESTRUCTOR_CLOSURE(wrapper)  \
    SWIGINTERN void                 \
    wrapper##_closure(PyObject *a) {        \
        SwigPyObject *sobj;             \
        sobj = (SwigPyObject *)a;           \
        if (sobj->own) {                \
        PyObject *o = wrapper(a, NULL);     \
        Py_XDECREF(o);              \
        }                       \
        PyObject_Del(a);                \
    }
    

    PyObject_Del(a) is called failing as MyFoo() has not been allocated with PyObject_MALLOC.
    Replacing the last line of the macro with this:

    if (! PyType_IS_GC(a ->ob_type)) \
            PyObject_Del(a);                
    

    makes the crash go away but I am to unfamiliar with the GC process and much else in python to be
    sure this is the right thing to do.
    Maybe, however this will help someone to figure out a patch - which we await anxiously as we had to
    switch back from builtin exactly due to this problem.

     
  • Christian Delbaere

    I tried this out on the testcase above and I found that while it gets rid of the crash, unfortunately it creates a memory leak.

    while 1:
        MyFoo()
    

    You can watch it grow in memory usage in top.

     
    Last edit: Christian Delbaere 2013-06-21
  • Christian Delbaere

    Aha! The clues you provided really helped solve this problem.

    I was reading the Python documentation for the tp_dealloc function, and it explains that if the object was created using PyObject_New the corresponding delete is PyObject_Del, but if the object was created using PyObject_GC_New, the corresponding delete is PyObject_GC_Del.

    So what I think is happening is that for instances of subclasses of the directors, since they're been created within in a python script:

    MyFoo()
    

    Python seems to allocate them using PyObject_GC_New. Whereas for the SWIG'd types, it uses PyObject_New to create them.

    Here's a slight modification to your suggestion that resolves the crash, but doesn't create a memory leak:

    if (PyType_IS_GC(a->ob_type)) \
            PyObject__GC_Del(a);  \
    } else {                      \
            PyObject_Del(a);      \
    }                             \
    
     
  • William Fulton

    William Fulton - 2013-07-01
    • status: open --> closed
    • Group: -->
     
  • William Fulton

    William Fulton - 2013-07-01

    Patch 342 applied for swig-2.0.11.

     

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks