From: Mateusz K. <mat...@an...> - 2011-06-30 10:21:28
|
Hi ! We recently tried to upgrade from ancient PyCXX version we used (5.4.1) to newest 6.2.3 but we faced big memory leak after that change and forced to switch back to 5.4.1. We have complex multithread C++/STL application running for hours, exposing it's calculation results via interface in Python. What would be best way to find where is source of memory leak is? Thanks in advance, regards, -- Mateusz Korniak |
From: Mateusz K. <mat...@an...> - 2011-06-30 10:32:35
|
On Thursday 30 of June 2011, Mateusz Korniak wrote: > Hi ! > We recently tried to upgrade from ancient PyCXX version we used (5.4.1) to > newest 6.2.3 but we faced big memory leak after that change and forced to > switch back to 5.4.1. One more note: We checked python object counts, and they stay constans while memory usage raises rapidly leading to OOM kills under Linux. -- Mateusz Korniak |
From: Barry S. <ba...@ba...> - 2011-07-25 23:45:46
|
On 30 Jun 2011, at 11:32, Mateusz Korniak wrote: > On Thursday 30 of June 2011, Mateusz Korniak wrote: >> Hi ! >> We recently tried to upgrade from ancient PyCXX version we used (5.4.1) to >> newest 6.2.3 but we faced big memory leak after that change and forced to >> switch back to 5.4.1. > > One more note: > We checked python object counts, and they stay constans while memory usage > raises rapidly leading to OOM kills under Linux. The following patch I was sent may fix the problem you are seeing. I think the patch is good. But I did see it cause pysvn to crash that I have not had time to investigate yet. Could you try this and let me know if it works for you? Barry Index: /Users/barry/wc/svn/PyCXX-Clean/CXX/Python2/ExtensionOldType.hxx =================================================================== --- /Users/barry/wc/svn/PyCXX-Clean/CXX/Python2/ExtensionOldType.hxx (revision 266) +++ /Users/barry/wc/svn/PyCXX-Clean/CXX/Python2/ExtensionOldType.hxx (working copy) @@ -178,7 +178,7 @@ Tuple self( 2 ); self[0] = Object( this ); - self[1] = Object( PyCObject_FromVoidPtr( method_def, do_not_dealloc ) ); + self[1] = Object( PyCObject_FromVoidPtr( method_def, do_not_dealloc ), true ); PyObject *func = PyCFunction_New( &method_def->ext_meth_def, self.ptr() ); Index: /Users/barry/wc/svn/PyCXX-Clean/CXX/Python2/ExtensionModule.hxx =================================================================== --- /Users/barry/wc/svn/PyCXX-Clean/CXX/Python2/ExtensionModule.hxx (revision 266) +++ /Users/barry/wc/svn/PyCXX-Clean/CXX/Python2/ExtensionModule.hxx (working copy) @@ -139,8 +139,8 @@ static PyObject *self = PyCObject_FromVoidPtr( this, do_not_dealloc ); Tuple args( 2 ); - args[0] = Object( self ); - args[1] = Object( PyCObject_FromVoidPtr( method_def, do_not_dealloc ) ); + args[0] = Object( self, true ); + args[1] = Object( PyCObject_FromVoidPtr( method_def, do_not_dealloc ), true ); PyObject *func = PyCFunction_New ( Index: /Users/barry/wc/svn/PyCXX-Clean/CXX/Python3/ExtensionOldType.hxx =================================================================== --- /Users/barry/wc/svn/PyCXX-Clean/CXX/Python3/ExtensionOldType.hxx (revision 266) +++ /Users/barry/wc/svn/PyCXX-Clean/CXX/Python3/ExtensionOldType.hxx (working copy) @@ -178,7 +178,7 @@ Tuple self( 2 ); self[0] = Object( this ); - self[1] = Object( PyCapsule_New( method_def, NULL, NULL ) ); + self[1] = Object( PyCapsule_New( method_def, NULL, NULL ), true ); PyObject *func = PyCFunction_New( &method_def->ext_meth_def, self.ptr() ); Index: /Users/barry/wc/svn/PyCXX-Clean/CXX/Python3/ExtensionModule.hxx =================================================================== --- /Users/barry/wc/svn/PyCXX-Clean/CXX/Python3/ExtensionModule.hxx (revision 266) +++ /Users/barry/wc/svn/PyCXX-Clean/CXX/Python3/ExtensionModule.hxx (working copy) @@ -135,8 +135,8 @@ static PyObject *self = PyCapsule_New( this, NULL, NULL ); Tuple args( 2 ); - args[0] = Object( self ); - args[1] = Object( PyCapsule_New( method_def, NULL, NULL ) ); + args[0] = Object( self, true ); + args[1] = Object( PyCapsule_New( method_def, NULL, NULL ), true ); PyObject *func = PyCFunction_New ( |
From: Barry S. <ba...@ba...> - 2011-07-30 21:01:52
|
I have checked in the fix for the memory leak as r267 on trunk. This will make it in to 6.2.4. Barry On 25 Jul 2011, at 23:54, Barry Scott wrote: > > On 30 Jun 2011, at 11:32, Mateusz Korniak wrote: > >> On Thursday 30 of June 2011, Mateusz Korniak wrote: >>> Hi ! >>> We recently tried to upgrade from ancient PyCXX version we used (5.4.1) to >>> newest 6.2.3 but we faced big memory leak after that change and forced to >>> switch back to 5.4.1. >> >> One more note: >> We checked python object counts, and they stay constans while memory usage >> raises rapidly leading to OOM kills under Linux. > > The following patch I was sent may fix the problem you are seeing. > I think the patch is good. But I did see it cause pysvn to crash > that I have not had time to investigate yet. > > Could you try this and let me know if it works for you? > > Barry > > Index: /Users/barry/wc/svn/PyCXX-Clean/CXX/Python2/ExtensionOldType.hxx > =================================================================== > --- /Users/barry/wc/svn/PyCXX-Clean/CXX/Python2/ExtensionOldType.hxx (revision 266) > +++ /Users/barry/wc/svn/PyCXX-Clean/CXX/Python2/ExtensionOldType.hxx (working copy) > @@ -178,7 +178,7 @@ > Tuple self( 2 ); > > self[0] = Object( this ); > - self[1] = Object( PyCObject_FromVoidPtr( method_def, do_not_dealloc ) ); > + self[1] = Object( PyCObject_FromVoidPtr( method_def, do_not_dealloc ), true ); > > PyObject *func = PyCFunction_New( &method_def->ext_meth_def, self.ptr() ); > > Index: /Users/barry/wc/svn/PyCXX-Clean/CXX/Python2/ExtensionModule.hxx > =================================================================== > --- /Users/barry/wc/svn/PyCXX-Clean/CXX/Python2/ExtensionModule.hxx (revision 266) > +++ /Users/barry/wc/svn/PyCXX-Clean/CXX/Python2/ExtensionModule.hxx (working copy) > @@ -139,8 +139,8 @@ > static PyObject *self = PyCObject_FromVoidPtr( this, do_not_dealloc ); > > Tuple args( 2 ); > - args[0] = Object( self ); > - args[1] = Object( PyCObject_FromVoidPtr( method_def, do_not_dealloc ) ); > + args[0] = Object( self, true ); > + args[1] = Object( PyCObject_FromVoidPtr( method_def, do_not_dealloc ), true ); > > PyObject *func = PyCFunction_New > ( > Index: /Users/barry/wc/svn/PyCXX-Clean/CXX/Python3/ExtensionOldType.hxx > =================================================================== > --- /Users/barry/wc/svn/PyCXX-Clean/CXX/Python3/ExtensionOldType.hxx (revision 266) > +++ /Users/barry/wc/svn/PyCXX-Clean/CXX/Python3/ExtensionOldType.hxx (working copy) > @@ -178,7 +178,7 @@ > Tuple self( 2 ); > > self[0] = Object( this ); > - self[1] = Object( PyCapsule_New( method_def, NULL, NULL ) ); > + self[1] = Object( PyCapsule_New( method_def, NULL, NULL ), true ); > > PyObject *func = PyCFunction_New( &method_def->ext_meth_def, self.ptr() ); > > Index: /Users/barry/wc/svn/PyCXX-Clean/CXX/Python3/ExtensionModule.hxx > =================================================================== > --- /Users/barry/wc/svn/PyCXX-Clean/CXX/Python3/ExtensionModule.hxx (revision 266) > +++ /Users/barry/wc/svn/PyCXX-Clean/CXX/Python3/ExtensionModule.hxx (working copy) > @@ -135,8 +135,8 @@ > static PyObject *self = PyCapsule_New( this, NULL, NULL ); > > Tuple args( 2 ); > - args[0] = Object( self ); > - args[1] = Object( PyCapsule_New( method_def, NULL, NULL ) ); > + args[0] = Object( self, true ); > + args[1] = Object( PyCapsule_New( method_def, NULL, NULL ), true ); > > PyObject *func = PyCFunction_New > ( > > > ------------------------------------------------------------------------------ > Storage Efficiency Calculator > This modeling tool is based on patent-pending intellectual property that > has been used successfully in hundreds of IBM storage optimization engage- > ments, worldwide. Store less, Store more with what you own, Move data to > the right place. Try It Now! http://www.accelacomm.com/jaw/sfnl/114/51427378/ > _______________________________________________ > CXX-Users mailing list > CXX...@li... > https://lists.sourceforge.net/lists/listinfo/cxx-users > |
From: Mateusz K. <mat...@an...> - 2011-07-31 10:08:26
|
On Saturday 30 of July 2011, Barry Scott wrote: > On 25 Jul 2011, at 23:54, Barry Scott wrote: > > On 30 Jun 2011, at 11:32, Mateusz Korniak wrote: > >> On Thursday 30 of June 2011, Mateusz Korniak wrote: > >>> We recently tried to upgrade from ancient PyCXX version we used (5.4.1) > >>> to newest 6.2.3 but we faced big memory leak after that change and > >>> forced to switch back to 5.4.1. > > > > The following patch I was sent may fix the problem you are seeing. > > I think the patch is good. But I did see it cause pysvn to crash > > that I have not had time to investigate yet. > > Could you try this and let me know if it works for you? Yes, after few days of testing it works great. Thanks ! > I have checked in the fix for the memory leak as r267 > on trunk. This will make it in to 6.2.4. Great ! -- Mateusz Korniak |