pyode-user

[Pyode-user] bug: read from free'd memory when joint in jointgroup

[Chris B. <chr...@gm...>]

In one of my pyode progs I get an invalid memory read reported when
running under valgrind. It seems that dJointGroupDestroy is called by
pyode and destroys all of the joints in the joint group. However, if
there is another reference to the joint, then it will also call
dJointDestroy, causing a read of the already-free'd memory. From ode
src:

void dJointDestroy (dxJoint *j)
{
  dAASSERT (j);
  if (j->flags & dJOINT_INGROUP) return;
  removeJointReferencesFromAttachedBodies (j);
  removeObjectFromList (j);
  j->world->nj--;
  dFree (j,j->vtable->size);
}

j->flags & dJOINT_INGROUP returns true because the joint was in a
(already destroyed) joint group, but the read of j->flags is invalid
since the memory was already free'd.

For some reason I've only seen this with ContactJoints - I wonder if
there are extra references for these attached to an object somewhere.

Re: [Pyode-user] bug: read from free'd memory when joint in jointgroup

[Matthias B. <ba...@ir...>]

Chris Bainbridge wrote:
> In one of my pyode progs I get an invalid memory read reported when
> running under valgrind. It seems that dJointGroupDestroy is called by
> pyode and destroys all of the joints in the joint group. However, if
> there is another reference to the joint, then it will also call
> dJointDestroy, causing a read of the already-free'd memory. From ode
> src:
> 
> void dJointDestroy (dxJoint *j)
> {
>   dAASSERT (j);
>   if (j->flags & dJOINT_INGROUP) return;
>   removeJointReferencesFromAttachedBodies (j);
>   removeObjectFromList (j);
>   j->world->nj--;
>   dFree (j,j->vtable->size);
> }
> 
> j->flags & dJOINT_INGROUP returns true because the joint was in a
> (already destroyed) joint group, but the read of j->flags is invalid
> since the memory was already free'd.

Could it be that you had a JointGroup object that you didn't empty 
explicitly (by calling the empty() method)?
In this case, the destructor of the JointGroup object only calls 
dJointGroupDestroy() without notifying the contained Python joints about 
the destruction of the ODE joints.
I have just committed a fix for this. Let me know if this has also fixed 
your problem.

- Matthias -

Re: [Pyode-user] bug: read from free'd memory when joint in jointgroup

[Chris B. <chr...@gm...>]

On 15/08/06, Matthias Baas <ba...@ir...> wrote:
> Could it be that you had a JointGroup object that you didn't empty
> explicitly (by calling the empty() method)?
> In this case, the destructor of the JointGroup object only calls
> dJointGroupDestroy() without notifying the contained Python joints about
> the destruction of the ODE joints.
> I have just committed a fix for this. Let me know if this has also fixed
> your problem.

Thanks, that seems to have fixed it. Have you noticed any memory
corruption problems with pyrex-0.9.4.1 and pyode? With 0.9.3.1
everything runs fine, with 0.9.4.1 I get:

*** glibc detected *** double free or corruption (!prev): 0x0823fb40 ***

And if I run valgrind there are hundreds of errors (again, 0.9.3.1 is fine).

Re: [Pyode-user] bug: read from free'd memory when joint in jointgroup

[Chris B. <chr...@gm...>]

On 15/08/06, Chris Bainbridge <chr...@gm...> wrote:
> On 15/08/06, Matthias Baas <ba...@ir...> wrote:
> > Could it be that you had a JointGroup object that you didn't empty
> > explicitly (by calling the empty() method)?
> > In this case, the destructor of the JointGroup object only calls
> > dJointGroupDestroy() without notifying the contained Python joints about
> > the destruction of the ODE joints.
> > I have just committed a fix for this. Let me know if this has also fixed
> > your problem.

Argh, I spoke too soon. It fixed the problem, mostly, but I am still
getting double frees. I think the problem now is that dWorldDestroy is
destroying all of the joints that aren't part of a joint group, and
then the destructors again get called by pyode when the joint object
is freed. From your fix it looks like every python Joint object that
isn't in a group will have to be notified about the destruction of the
underlying object in World.__dealloc__

Log:

==3141== Invalid free() / delete / delete[]
==3141==    at 0x401C12D: free (in
/usr/lib/valgrind/x86-linux/vgpreload_memcheck.so)
==3141==    by 0x463A2AB: dJointDestroy (ode.cpp:989)
==3141==    by 0x45EA930: __pyx_tp_dealloc_3ode_Joint (ode_trimesh.c:5613)
==3141==    by 0x45D8169: __pyx_tp_dealloc_3ode_AMotor (ode_trimesh.c:16542)
==3141==    by 0x409259C: (within /usr/lib/libpython2.4.so.1.0)
==3141==    by 0x407F117: PyDict_Clear (in /usr/lib/libpython2.4.so.1.0)
==3141==    by 0x4081032: (within /usr/lib/libpython2.4.so.1.0)
==3141==    by 0x40DB82F: (within /usr/lib/libpython2.4.so.1.0)
==3141==    by 0x40DC0C5: PyGC_Collect (in /usr/lib/libpython2.4.so.1.0)
==3141==    by 0x40D4EC5: Py_Finalize (in /usr/lib/libpython2.4.so.1.0)
==3141==    by 0x40D3EF9: Py_Exit (in /usr/lib/libpython2.4.so.1.0)
==3141==    by 0x40D3FC1: (within /usr/lib/libpython2.4.so.1.0)
==3141==  Address 0x4BA88B0 is 0 bytes inside a block of size 560 free'd
==3141==    at 0x401C12D: free (in
/usr/lib/valgrind/x86-linux/vgpreload_memcheck.so)
==3141==    by 0x463B907: dWorldDestroy (ode.cpp:1266)
==3141==    by 0x45E122A: __pyx_tp_dealloc_3ode_World (ode_trimesh.c:2366)
==3141==    by 0x45E83FA: __pyx_tp_dealloc_3ode_Body (ode_trimesh.c:14884)
==3141==    by 0x45EA8A7: __pyx_tp_dealloc_3ode_Joint (ode_trimesh.c:15277)
==3141==    by 0x45D7E4B: __pyx_tp_dealloc_3ode_HingeJoint (ode_trimesh.c:15619)
==3141==    by 0x407F2E9: (within /usr/lib/libpython2.4.so.1.0)
==3141==    by 0x45F679E: __pyx_tp_dealloc_3ode_GeomObject (ode_trimesh.c:16716)
==3141==    by 0x45D869F: __pyx_tp_dealloc_3ode_GeomCCylinder
(ode_trimesh.c:17976)
==3141==    by 0x407F117: PyDict_Clear (in /usr/lib/libpython2.4.so.1.0)
==3141==    by 0x4081032: (within /usr/lib/libpython2.4.so.1.0)
==3141==    by 0x40DB82F: (within /usr/lib/libpython2.4.so.1.0)

Re: [Pyode-user] bug: read from free'd memory when joint in jointgroup

[Matthias B. <ba...@ir...>]

Chris Bainbridge wrote:
> Argh, I spoke too soon. It fixed the problem, mostly, but I am still
> getting double frees. I think the problem now is that dWorldDestroy is
> destroying all of the joints that aren't part of a joint group, and
> then the destructors again get called by pyode when the joint object
> is freed. From your fix it looks like every python Joint object that
> isn't in a group will have to be notified about the destruction of the
> underlying object in World.__dealloc__

Well, unfortunately, the ownership rules in ODE (if there are any) are 
rather vague. Who is the owner of a joint object who is responsible for 
the destruction of the object? In PyODE it's basically the Python Joint 
object that takes care of destroying the ODE joint (i.e. dJointDestroy() 
is always called inside __dealloc__()). If other objects take over 
ownership (such as the joint group or the world) and destroy the ODE 
joint they have to notify the corresponding Python Joint object about 
this by calling the _destroyed() method. Then the Python Joint object 
will not destroy the ODE joint again.

Now the situation you describe above (i.e. dWorldDestroy() is called 
before the Python joints are deallocated) shouldn't occur because the 
Python joints keep a reference to the World object they are in. I did 
this so that the World object is deallocated after the joints. But I'm 
not sure if Python really cares about the reference counts when the 
interpreter is actually shut down.

Does the crash in your program occur at the time the program is being 
terminated or does it also happen while the program is running?

- Matthias -

Re: [Pyode-user] bug: read from free'd memory when joint in jointgroup

[Chris B. <chr...@gm...>]

On 16/08/06, Matthias Baas <ba...@ir...> wrote:
> Now the situation you describe above (i.e. dWorldDestroy() is called
> before the Python joints are deallocated) shouldn't occur because the
> Python joints keep a reference to the World object they are in. I did
> this so that the World object is deallocated after the joints. But I'm
> not sure if Python really cares about the reference counts when the
> interpreter is actually shut down.

I think the key thing here is that I'm using garbage collection module
gc which doesn't enforce any ordering on the destruction of objects. I
just disabled it in one of my test cases and it no longer crashes.

Re: [Pyode-user] bug: read from free'd memory when joint in jointgroup

[Chris B. <chr...@gm...>]

The attached patch to current cvs fixes my memory problems. The
ode.World needs to keep track of all Joint, Body, Geom objects, since
dWorldDestroy removes the underlying objects (in the case of Geom the
body it links to is removed rather than the Geom itself, but Geom then
tries to unlink itself from the Body).

Re: [Pyode-user] bug: read from free'd memory when joint in jointgroup

[Chris B. <chr...@gm...>]

On 17/08/06, Chris Bainbridge <chr...@gm...> wrote:
> The attached patch to current cvs fixes my memory problems. The
> ode.World needs to keep track of all Joint, Body, Geom objects, since
> dWorldDestroy removes the underlying objects (in the case of Geom the
> body it links to is removed rather than the Geom itself, but Geom then
> tries to unlink itself from the Body).

Ok, it doesn't fix everything... to deal with non-deterministic
destructor order pyode would have to know about every pointer between
ode objects, and some are hidden and can't be easily changed through
the API without knowing if both underlying objects are still valid
(body->geom for example). So we would need to ensure that the
underlying ODE objects are deconstructed in the right order, and
deconstruction is delayed when we know that another python object that
is still valid has an ode internal pointer to the to-be-destroyed
object. Argh. It's probably easier to have an explicit cleanup
function to be called at the end of a simulation that detaches all
geoms, joints, and bodies, and then actually destroy the underlying
objects in the usual pyrex destructor.