[SQLObject] "Memory Leak" in sqlobject/cache.py

[Jason C. <ja...@me...>]

Hi,
I'm using an older copy of SQLObject but sqlobject/cache.py hasn't  
really changed in 2 years, so this applies to current and past versions.

I operate a website that runs around 4million requests a day through  
an application using SQLObject.
At about mid day I end up with large amount of weakref objects and  
slowing response times.

Top 6 objects in memory:
2887448 <type 'weakref'>
  209963 <type 'dict'>
   68189 <type 'tuple'>
   45584 <type 'weakproxy'>
   45584 <class 'sqlobject.main.SQLObjectState'>
   45584 <class 'sqlobject.declarative.sqlmeta'>

These objects are from the CacheFactory.expiredCache dictionary that  
holds weak references to SQLObject instances by id.  Some of these  
dictionaries reach ~1.5 million keys, each SQLObject.get() request has  
to look into this dictionary which results in an access slowdown and  
memory growth. When the object behind the weak reference is collected,  
a weakref with a dead object is left in the cache, In my case this is  
a memory leak.

Any comments or improvements on this method are appreciated.

There are lots of ways to solve this problem but I think this is the  
least disruptive.   Weakref callbacks can't be used because the  
"release" callback happens in the cull method.
I do 2 things in the cull method
1 - cull dead weak references out of the expiredCache.
2 - avoid placing dead weak references into the expiredCache

     def cull(self):
         """
         Runs through the cache and expires objects.  E.g., if
         ``cullFraction`` is 3, then every third object is moved to
         the 'expired' (aka weakref) cache.
         """
         self.lock.acquire()
         try:
             #remove dead references from the expired cache
             keys = self.expiredCache.keys()
             for key in keys:
                 if self.expiredCache[key]() is None:
                     self.expiredCache.pop(key, None)

             keys = self.cache.keys()
             for i in xrange(self.cullOffset, len(keys),  
self.cullFraction):
                 id = keys[i]
                 # create a weakref, then remove from the cache
                 obj = ref(self.cache[id])
                 del self.cache[id]

                 #the object may have been gc'd when removed from the  
cache
                 #above, no need to place in expiredCache
                 if obj() is not None:
                     self.expiredCache[id] = obj
             # This offset tries to balance out which objects we
             # expire, so no object will just hang out in the cache
             # forever.
             self.cullOffset = (self.cullOffset + 1) % self.cullFraction
         finally:
             self.lock.release()

Thanks,
Jason