Hi,
I'm using an older copy of SQLObject but sqlobject/cache.py hasn't
really changed in 2 years, so this applies to current and past versions.
I operate a website that runs around 4million requests a day through
an application using SQLObject.
At about mid day I end up with large amount of weakref objects and
slowing response times.
Top 6 objects in memory:
2887448 <type 'weakref'>
209963 <type 'dict'>
68189 <type 'tuple'>
45584 <type 'weakproxy'>
45584 <class 'sqlobject.main.SQLObjectState'>
45584 <class 'sqlobject.declarative.sqlmeta'>
These objects are from the CacheFactory.expiredCache dictionary that
holds weak references to SQLObject instances by id. Some of these
dictionaries reach ~1.5 million keys, each SQLObject.get() request has
to look into this dictionary which results in an access slowdown and
memory growth. When the object behind the weak reference is collected,
a weakref with a dead object is left in the cache, In my case this is
a memory leak.
Any comments or improvements on this method are appreciated.
There are lots of ways to solve this problem but I think this is the
least disruptive. Weakref callbacks can't be used because the
"release" callback happens in the cull method.
I do 2 things in the cull method
1 - cull dead weak references out of the expiredCache.
2 - avoid placing dead weak references into the expiredCache
def cull(self):
"""
Runs through the cache and expires objects. E.g., if
``cullFraction`` is 3, then every third object is moved to
the 'expired' (aka weakref) cache.
"""
self.lock.acquire()
try:
#remove dead references from the expired cache
keys = self.expiredCache.keys()
for key in keys:
if self.expiredCache[key]() is None:
self.expiredCache.pop(key, None)
keys = self.cache.keys()
for i in xrange(self.cullOffset, len(keys),
self.cullFraction):
id = keys[i]
# create a weakref, then remove from the cache
obj = ref(self.cache[id])
del self.cache[id]
#the object may have been gc'd when removed from the
cache
#above, no need to place in expiredCache
if obj() is not None:
self.expiredCache[id] = obj
# This offset tries to balance out which objects we
# expire, so no object will just hang out in the cache
# forever.
self.cullOffset = (self.cullOffset + 1) % self.cullFraction
finally:
self.lock.release()
Thanks,
Jason
|
