From:
<hir...@be...> - 2006-09-13 14:30:05
|
Hi, I got a memory leak-like problem. Here's a sample code: obj_list = [] for i in range( 100000 ): obj = SomeObject( parameters ... ) obj_list.append( obj ) for obj in obj_list: obj.expire() del obj The memory usage grows up, even if I expire(clear) the cache, and delete the object. My wish is as follows: 1. When an object is created, it occupies some memory and the object data is stored in the RDBMS( This is the default function of SQLObject). 2. The object should be cached in normal usage(This is the default). 3. When I delete the object, the occupied memory is freed. The object data still remains in the RDBMS. I tried a patch by Dan, which is posted at 2006-05-30 in this mailing-list, but no change is obtained in memory usage. I searched this mailing list and found similar problems: "Confused about caching" by Peter, 2006-01-24 "Problem with circular references" by Dan, 2006-05-30 "High memory consumption" by Brian, 2006-08-05 I found that my problem may be the problem of garbage collection on circular references, but I cannot find the solution. Any good idea? (sorry my broken english.) -- 株式会社ビービット 玉越 大輝 ユーザビリティ コンサルタント beBit,Inc. Tamakoshi Hiroki hir...@be... -------------------------------------------------------- 〒105-0001 東京都港区虎ノ門1-18-1 虎ノ門10森ビル7F TEL: 03-3509-7602 / FAX: 03-3509-7605 URL: http://www.bebit.co.jp/ -------------------------------------------------------- |
From: Oleg B. <ph...@ph...> - 2006-09-13 15:45:21
|
On Wed, Sep 13, 2006 at 11:29:39PM +0900, ?$B6L1[ ?$BBg51 wrote: > 3. When I delete the object, the occupied memory is freed. The memory is freed for Python, by Python doesn't return the memory to OS. So even after connection.cache.clear() the memory usage by the process remains the same. Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: <hir...@be...> - 2006-09-13 16:40:29
|
Thanks Oleg, hmm, it seems that an object inherites SQLObject is not freed, but a plain object is freed. The following code explains. class DBObject(SQLObject): name = StringCol() class NormalObject: name = "" obj_list = [] for i in xrange( 100000 ): obj = DBObject( name = "foo" ) obj.expire() obj_list.append( obj ) del obj_list[:] # memory usage is kept for i in xrange( 100000 ): obj = NormalObject obj.name = "foo" obj_list.append( obj ) del obj_list[:] # memory usage is EXACTLY freed from OS, by looking top(1) screen. The difference of the above two code is the class of the object in the obj_list. (DBObject is freed from cache by calling .expire()) And, memory is freed in the following code. for i in xrange( 1000 ): obj = DBObject( name = "foo" ) obj.expire() Each time obj is created, its cache is cleared and memory is freed from OS as expected. Thus, the difference is that the object is stored in obj_list or not, and the object is an SQLObject or not. But I cannot understand why these differences cause the different memory usage. 2006/9/14, Oleg Broytmann <ph...@ph...>: > On Wed, Sep 13, 2006 at 11:29:39PM +0900, ?$B6L1[ ?$BBg51 wrote: > > 3. When I delete the object, the occupied memory is freed. > > The memory is freed for Python, by Python doesn't return the memory to > OS. So even after connection.cache.clear() the memory usage by the process > remains the same. > > Oleg. > -- > Oleg Broytmann http://phd.pp.ru/ ph...@ph... > Programmers don't die, they just GOSUB without RETURN. > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > sqlobject-discuss mailing list > sql...@li... > https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss > -- 株式会社ビービット 玉越 大輝 ユーザビリティ コンサルタント beBit,Inc. Tamakoshi Hiroki hir...@be... -------------------------------------------------------- 〒105-0001 東京都港区虎ノ門1-18-1 虎ノ門10森ビル7F TEL: 03-3509-7602 / FAX: 03-3509-7605 URL: http://www.bebit.co.jp/ -------------------------------------------------------- |
From:
<hir...@be...> - 2006-09-14 01:16:20
|
Thanks Oleg, You are very kind because you reply for my question, even if the question is discussed in this mailing-list again and again. I had already tried the patch by Dan, but nothing is changed. You posted that: 1. how about connection.cache.clear() ? 2. memory is freed for python, but python doesn't return memory to OS. I want to clear the cache only for the object I want to delete, thus I want to avoid 1.(and 1. doesn't solve this problem). 2. is doubt, because python returns memory to OS in the following code(as posted in previous mail). class NormalObject: name = "dummy" obj_list = [] for i in xrange( 10000 ): obj_list.append( NormalObject ) # memory grows del obj_list[:] # memory freed This problem is a frequently asked question, and it is natural that "I want to free the object from memory, because it is stored in DBMS." I treat the objects with RDBMS which has > 100,000,000 rows, thus this is very serious problem... On Wed, 13 Sep 2006 21:21:39 +0400 Oleg Broytmann <ph...@ph...> wrote: > On Thu, Sep 14, 2006 at 01:40:20AM +0900, ?$B6L1[Bg51 wrote: > > hmm, it seems that an object inherites SQLObject is not freed, but a > > plain object is freed. > > There is a patch created by Dan Pascu <da...@ag...> that may fix > that. Can you test it (attached)? Does it help? > > Oleg. > -- > Oleg Broytmann http://phd.pp.ru/ ph...@ph... > Programmers don't die, they just GOSUB without RETURN. -- 株式会社ビービット 玉越 大輝 ユーザビリティ コンサルタント beBit,Inc. Tamakoshi Hiroki hir...@be... -------------------------------------------------------- 〒105-0001 東京都港区虎ノ門1-18-1 虎ノ門10森ビル7F TEL: 03-3509-7602 / FAX: 03-3509-7605 URL: http://www.bebit.co.jp/ -------------------------------------------------------- |
From: Oleg B. <ph...@ph...> - 2006-09-14 10:53:47
|
On Thu, Sep 14, 2006 at 09:55:54AM +0900, ?$B6L1[ ?$BBg51 wrote: > You are very kind Thank you! > I had already tried the patch by Dan, but nothing is changed. Ouch! Then I don't what to do. Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From:
<hir...@be...> - 2006-09-14 14:43:38
|
Thanks Oleg, I try to contact Dan directly. On Thu, 14 Sep 2006 14:53:35 +0400 Oleg Broytmann <ph...@ph...> wrote: > On Thu, Sep 14, 2006 at 09:55:54AM +0900, ?$B6L1[ ?$BBg51 wrote: > > You are very kind > > Thank you! > > > I had already tried the patch by Dan, but nothing is changed. > > Ouch! Then I don't what to do. > > Oleg. > -- > Oleg Broytmann http://phd.pp.ru/ ph...@ph... > Programmers don't die, they just GOSUB without RETURN. > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > sqlobject-discuss mailing list > sql...@li... > https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss > -- 株式会社ビービット 玉越 大輝 ユーザビリティ コンサルタント beBit,Inc. Tamakoshi Hiroki hir...@be... -------------------------------------------------------- 〒105-0001 東京都港区虎ノ門1-18-1 虎ノ門10森ビル7F TEL: 03-3509-7602 / FAX: 03-3509-7605 URL: http://www.bebit.co.jp/ -------------------------------------------------------- |
From: Rick F. <rf...@im...> - 2006-09-14 14:49:23
|
Maybe you already stated this, but I can't find it. What version of SQLObject are you using for this test case? Where can I find more information on Dan's patch? Thanks. -- Rick On Thu, 14 Sep 2006, =B6=CC=B1=DB =C2=E7=B5=B1 wrote: > Thanks Oleg, I try to contact Dan directly. > > > On Thu, 14 Sep 2006 14:53:35 +0400 > Oleg Broytmann <ph...@ph...> wrote: > >> On Thu, Sep 14, 2006 at 09:55:54AM +0900, ?$B6L1[ ?$BBg51 wrote: >>> You are very kind >> >> Thank you! >> >>> I had already tried the patch by Dan, but nothing is changed. >> >> Ouch! Then I don't what to do. >> >> Oleg. >> -- >> Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.r= u >> Programmers don't die, they just GOSUB without RETURN. >> >> ------------------------------------------------------------------------= - >> Using Tomcat but need to do more? Need to support web services, security= ? >> Get stuff done quickly with pre-integrated technology to make your job e= asier >> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geroni= mo >> http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat= =3D121642 >> _______________________________________________ >> sqlobject-discuss mailing list >> sql...@li... >> https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss >> > > -- > =B3=F4=BC=B0=B2=F1=BC=D2=A5=D3=A1=BC=A5=D3=A5=C3=A5=C8=A1=A1=B6=CC=B1=DB = =C2=E7=B5=B1 > =A5=E6=A1=BC=A5=B6=A5=D3=A5=EA=A5=C6=A5=A3 =A5=B3=A5=F3=A5=B5=A5=EB=A5=BF= =A5=F3=A5=C8 > beBit,Inc. Tamakoshi Hiroki hir...@be... > -------------------------------------------------------- > =A2=A9105-0001 =C5=EC=B5=FE=C5=D4=B9=C1=B6=E8=B8=D7=A5=CE=CC=E71-18-1 =B8= =D7=A5=CE=CC=E710=BF=B9=A5=D3=A5=EB7F > TEL: 03-3509-7602 / FAX: 03-3509-7605 > URL: http://www.bebit.co.jp/ > -------------------------------------------------------- > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job ea= sier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronim= o > http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat= =3D121642 > _______________________________________________ > sqlobject-discuss mailing list > sql...@li... > https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss > |
From: Oleg B. <ph...@ph...> - 2006-09-13 17:21:45
Attachments:
sqlobject-circular-reference-fix.patch
|
On Thu, Sep 14, 2006 at 01:40:20AM +0900, ?$B6L1[Bg51 wrote: > hmm, it seems that an object inherites SQLObject is not freed, but a > plain object is freed. There is a patch created by Dan Pascu <da...@ag...> that may fix that. Can you test it (attached)? Does it help? Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Dan P. <da...@ag...> - 2006-09-14 18:39:15
|
On Wednesday 13 September 2006 20:21, Oleg Broytmann wrote: > There is a patch created by Dan Pascu <da...@ag...> that may > fix that. Can you test it (attached)? Does it help? BTW, what is the status on this? Do you intend to include it? If so, what is holding it back for so long? -- Dan |
From: Oleg B. <ph...@ph...> - 2006-09-14 18:44:37
|
On Thu, Sep 14, 2006 at 09:38:59PM +0300, Dan Pascu wrote: > BTW, what is the status on this? Do you intend to include it? If so, what > is holding it back for so long? I didn't want to include in 0.7.1. I wanted to apply it after releasing 0.7.1. But now the question is - does it really help? Does the original poster have a different problem not covered by the patch? Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Dan P. <da...@ag...> - 2006-09-14 19:04:26
|
On Thursday 14 September 2006 21:44, Oleg Broytmann wrote: > On Thu, Sep 14, 2006 at 09:38:59PM +0300, Dan Pascu wrote: > > BTW, what is the status on this? Do you intend to include it? If so, > > what is holding it back for so long? > > I didn't want to include in 0.7.1. I wanted to apply it after > releasing 0.7.1. > But now the question is - does it really help? Does the original > poster have a different problem not covered by the patch? It helps to take off the burden from the garbage collector by removing a known circular reference, which is a good thing to do in my opinion. The garbage collector knows how to deal with circular references, but that is just to protect programs against overlooked references, not to encourage programmers to grow careless and ignore them if they know about them. I do not know what problem the original poster has, but I'm guessing that he experiences something else, because the circular reference problem is not evident unless you also enable memory tracing, in which case the garbage collector no longer frees circular references. Else the garbage collector does its job and eliminates the referenced objects, but as I said with a performance penalty which can be avoided if the patch is applied. So even if the specific problem of the original poster is not solved by this patch, it doesn't mean that the patch is not useful. Removing the circular references that we know about is always a good thing to do and is also harmless. -- Dan |
From: Oleg B. <ph...@ph...> - 2006-09-14 19:06:22
|
On Thu, Sep 14, 2006 at 10:04:19PM +0300, Dan Pascu wrote: > So even if the specific problem of the original poster is not solved by > this patch, it doesn't mean that the patch is not useful. Removing the > circular references that we know about is always a good thing to do and > is also harmless. Ok then. Will be in the trunk after 0.7.1 final release. Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Hiroki T. <hir...@be...> - 2006-09-15 08:43:00
|
Thanks Dan, Can I consult with you even if the problem I have is not the problem of circular references? The expected work is clear, and natural I think: 1. A sqlobject is persistent in RDBMS, 2. and when I want to delete the object, it is freed from memory(and cache), but its data is still persistent in RDBMS. Here is a sample code: obj_list = [] for i in xrange( 1000 ): obj = SomeObject( parameters ... ) obj_list.append( obj ) # memory grows for obj in obj_list: obj.expire() # clear cache del obj_list[:] # expected work: clear memory, but data remain in RDBMS. # actual work: memory doesn't seem to be cleared, data remain in RDBMS. On Thu, 14 Sep 2006 22:04:19 +0300 Dan Pascu <da...@ag...> wrote: > On Thursday 14 September 2006 21:44, Oleg Broytmann wrote: > > On Thu, Sep 14, 2006 at 09:38:59PM +0300, Dan Pascu wrote: > > > BTW, what is the status on this? Do you intend to include it? If so, > > > what is holding it back for so long? > > > > I didn't want to include in 0.7.1. I wanted to apply it after > > releasing 0.7.1. > > But now the question is - does it really help? Does the original > > poster have a different problem not covered by the patch? > > It helps to take off the burden from the garbage collector by removing a > known circular reference, which is a good thing to do in my opinion. The > garbage collector knows how to deal with circular references, but that is > just to protect programs against overlooked references, not to encourage > programmers to grow careless and ignore them if they know about them. > > I do not know what problem the original poster has, but I'm guessing that > he experiences something else, because the circular reference problem is > not evident unless you also enable memory tracing, in which case the > garbage collector no longer frees circular references. Else the garbage > collector does its job and eliminates the referenced objects, but as I > said with a performance penalty which can be avoided if the patch is > applied. > > So even if the specific problem of the original poster is not solved by > this patch, it doesn't mean that the patch is not useful. Removing the > circular references that we know about is always a good thing to do and > is also harmless. |
From: Markus G. <m.g...@gm...> - 2006-09-15 08:57:34
|
On 9/15/06, Hiroki Tamakoshi <hir...@be...> wrote: > Thanks Dan, > > Can I consult with you even if the problem I have is not the problem of > circular references? > > The expected work is clear, and natural I think: > 1. A sqlobject is persistent in RDBMS, > 2. and when I want to delete the object, it is freed from memory(and > cache), but its data is still persistent in RDBMS. > > Here is a sample code: > > obj_list = [] > for i in xrange( 1000 ): > obj = SomeObject( parameters ... ) > obj_list.append( obj ) # memory grows > > for obj in obj_list: > obj.expire() # clear cache > > del obj_list[:] > # expected work: clear memory, but data remain in RDBMS. > # actual work: memory doesn't seem to be cleared, data remain in RDBMS. del obj_list[:] in your example code is definitely not what you want. By [:] you create a copy of your list, and the statement you wrote deletes just this copy. In the following example obj_list is still there. obj_list = [] del obj_list[:] print obj_list You definitely want to del obj_list |
From: Hiroki T. <hir...@be...> - 2006-09-15 09:22:46
|
On Fri, 15 Sep 2006 10:57:31 +0200 "Markus Gritsch" <m.g...@gm...> wrote: > del obj_list[:] in your example code is definitely not what you want. > By [:] you create a copy of your list, and the statement you wrote > deletes just this copy. > > In the following example obj_list is still there. > obj_list = [] > del obj_list[:] > print obj_list > > You definitely want to > del obj_list "del obj_list" deletes obj_list itself and the containing objects. But, the memory usage doesn't seem to be free. Is it realy that del [:] doesn't delete the original objects? Even if I rewrite the code as follows, the memory is not freed. for obj in obj_list: del obj del obj_list best regards, |
From: Markus G. <m.g...@gm...> - 2006-09-15 09:30:16
|
On 9/15/06, Hiroki Tamakoshi <hir...@be...> wrote: > On Fri, 15 Sep 2006 10:57:31 +0200 > "Markus Gritsch" <m.g...@gm...> wrote: > > > del obj_list[:] in your example code is definitely not what you want. > > By [:] you create a copy of your list, and the statement you wrote > > deletes just this copy. > > > > In the following example obj_list is still there. > > obj_list = [] > > del obj_list[:] > > print obj_list > > > > You definitely want to > > del obj_list > > "del obj_list" deletes obj_list itself and the containing objects. > But, the memory usage doesn't seem to be free. Because you cannot explicitely delete an object. By calling del on a variable which holds a reference to it, you just remove this particular reference to the object. If it was the last reference pointing to the object, the garbage collector can decide to free the actual memory *at some time*. You can tell the garbage collector to collect all objects which are no longer referenced by calling gc.collect() > Is it realy that del [:] doesn't delete the original objects? > Even if I rewrite the code as follows, the memory is not freed. > > for obj in obj_list: > del obj > del obj_list see above |
From: Hiroki T. <hir...@be...> - 2006-09-15 10:03:08
|
On Fri, 15 Sep 2006 11:30:11 +0200 "Markus Gritsch" <m.g...@gm...> wrote: > Because you cannot explicitely delete an object. By calling del on a > variable which holds a reference to it, you just remove this > particular reference to the object. If it was the last reference > pointing to the object, the garbage collector can decide to free the > actual memory *at some time*. You can tell the garbage collector to > collect all objects which are no longer referenced by calling > gc.collect() calling gc.collect() doesn't change the memory usage. (Dan's patch is applied) obj_list = [] for i in xrange( 1000 ): obj = SomeObject( parameters ... ) obj_list.append( obj ) # memory grows for obj in obj_list: obj.expire() # clear cache del obj_list gc.collect() # <- added here, memory usage doesn't change. |
From: Dan P. <da...@ag...> - 2006-09-15 10:29:58
|
On Friday 15 September 2006 12:41, Hiroki Tamakoshi wrote: > On Fri, 15 Sep 2006 11:30:11 +0200 > > "Markus Gritsch" <m.g...@gm...> wrote: > > Because you cannot explicitely delete an object. By calling del on a > > variable which holds a reference to it, you just remove this > > particular reference to the object. If it was the last reference > > pointing to the object, the garbage collector can decide to free the > > actual memory *at some time*. You can tell the garbage collector to > > collect all objects which are no longer referenced by calling > > gc.collect() > > calling gc.collect() doesn't change the memory usage. > (Dan's patch is applied) > > obj_list = [] > for i in xrange( 1000 ): > obj = SomeObject( parameters ... ) > obj_list.append( obj ) # memory grows > > for obj in obj_list: > obj.expire() # clear cache > > del obj_list > gc.collect() # <- added here, memory usage doesn't change. If I recall correctly python allocates memory of the OS in chunks and then has an internal memory allocator which allocates from these chunks. Which means that it may not return it to the OS but keep it for later reuse. Here a very simple test: dan@host:~$ python >>> # ps aux here shows 4060 2552 (VSZ and RSS in kb) >>> l = range(1000000) >>> # ps aux here shows 19848 18272 (VSZ and RSS in kb) >>> del l >>> # ps aux here shows 15940 14364 (VSZ and RSS in kb) >>> # calling gc.collect() below will have no effect since we have >>> # no garbage actually, but is included just for reference >>> import gc; gc.collect() >>> # ps aux here shows 15940 14364 (VSZ and RSS in kb) >>> Which shows what I said (the above was a clean python interpreter with only a list created and destroyed). So this is not a SQLObject problem, but rather your misunderstanding of how memory allocation should behave. As I already said, with or without my patch a circular reference problem will not be apparent unless you enable memory tracing, because else the gc will do it's job, but with a performance cost without the patch. -- Dan |
From: Dan P. <da...@ag...> - 2006-09-15 10:35:36
|
On Friday 15 September 2006 12:41, Hiroki Tamakoshi wrote: > On Fri, 15 Sep 2006 11:30:11 +0200 > calling gc.collect() doesn't change the memory usage. > (Dan's patch is applied) > > obj_list = [] > for i in xrange( 1000 ): > obj = SomeObject( parameters ... ) > obj_list.append( obj ) # memory grows > > for obj in obj_list: > obj.expire() # clear cache > > del obj_list > gc.collect() # <- added here, memory usage doesn't change. If you want to see if you have memory leaks try to run your code in an infinite loop: while 1: obj_list = [] for i in xrange( 1000 ): obj = SomeObject( parameters ... ) obj_list.append( obj ) # memory grows for obj in obj_list: obj.expire() # clear cache del obj_list and monitor memory usage. If it only increases once while the first pool of 1000 objects is created, but then stays constant you are fine. I means that the memory is not returned to the OS but reused for the next bunch of 1000 objects and so on. If it constantly grows you have a mem leak. -- Dan |
From: Hiroki T. <hir...@be...> - 2006-09-15 11:46:23
|
Thanks Oleg, Dan and Markus!!!, and I could finaly understand the issue. The code by Dan helps me. On Fri, 15 Sep 2006 13:35:25 +0300 Dan Pascu <da...@ag...> wrote: > If you want to see if you have memory leaks try to run your code in an > infinite loop: > > while 1: > obj_list = [] > for i in xrange( 1000 ): > obj = SomeObject( parameters ... ) > obj_list.append( obj ) # memory grows > > for obj in obj_list: > obj.expire() # clear cache > > del obj_list > > and monitor memory usage. If it only increases once while the first pool > of 1000 objects is created, but then stays constant you are fine. I means > that the memory is not returned to the OS but reused for the next bunch > of 1000 objects and so on. If it constantly grows you have a mem leak. The memory usage doesn't grow up forever! I can understand Oleg and Dan's claim that "the memory is freed, but python doesn't return to OS." Thank you very much! |
From: Markus G. <m.g...@gm...> - 2006-09-15 10:15:07
|
On 9/15/06, Hiroki Tamakoshi <hir...@be...> wrote: > > On Fri, 15 Sep 2006 11:30:11 +0200 > "Markus Gritsch" <m.g...@gm...> wrote: > > > Because you cannot explicitely delete an object. By calling del on a > > variable which holds a reference to it, you just remove this > > particular reference to the object. If it was the last reference > > pointing to the object, the garbage collector can decide to free the > > actual memory *at some time*. You can tell the garbage collector to > > collect all objects which are no longer referenced by calling > > gc.collect() > > calling gc.collect() doesn't change the memory usage. > (Dan's patch is applied) > > obj_list = [] > for i in xrange( 1000 ): > obj = SomeObject( parameters ... ) > obj_list.append( obj ) # memory grows > > for obj in obj_list: > obj.expire() # clear cache > > del obj_list > gc.collect() # <- added here, memory usage doesn't change. What if you omit expiring every single object and do a sqlhub.processConnection.cache.clear() instead? |
From: Markus G. <m.g...@gm...> - 2006-09-15 10:17:33
|
On 9/15/06, Markus Gritsch <m.g...@gm...> wrote: > On 9/15/06, Hiroki Tamakoshi <hir...@be...> wrote: > > > > On Fri, 15 Sep 2006 11:30:11 +0200 > > "Markus Gritsch" <m.g...@gm...> wrote: > > > > > Because you cannot explicitely delete an object. By calling del on a > > > variable which holds a reference to it, you just remove this > > > particular reference to the object. If it was the last reference > > > pointing to the object, the garbage collector can decide to free the > > > actual memory *at some time*. You can tell the garbage collector to > > > collect all objects which are no longer referenced by calling > > > gc.collect() > > > > calling gc.collect() doesn't change the memory usage. > > (Dan's patch is applied) > > > > obj_list = [] > > for i in xrange( 1000 ): > > obj = SomeObject( parameters ... ) > > obj_list.append( obj ) # memory grows > > > > for obj in obj_list: > > obj.expire() # clear cache > > > > del obj_list > > gc.collect() # <- added here, memory usage doesn't change. > > What if you omit expiring every single object and do a > > sqlhub.processConnection.cache.clear() > > instead? Something like obj_list = [] for i in range( 1000 ): obj = SomeObject( parameters ... ) obj_list.append( obj ) # memory grows del obj_list sqlhub.processConnection.cache.clear() gc.collect() # maybe skip this -- I never explicitly force the gc to collect. |