Re: [SQLObject] Slow/Large cache

[Oleg B. <ph...@ph...>]

Hello!

On Mon, Aug 27, 2012 at 10:32:50AM +0200, Gert Burger <ger...@gm...> wrote:
> We have a RPC server which CPU usage increases over time, after a week
> it will max out one core.
> 
> >From my testing(Using pyrasite's dump stack) it seems that all the time
> is spend in:
> 
>   File
> "/usr/local/lib/python2.6/dist-packages/SQLObject-1.3.1-py2.6.egg/sqlobject/cache.py",
> line 263, in allIDs
>     for id, value in self.expiredCache.items():
> 
> Currently the overhead for each commit and rollback is around 2-3 seconds.
> 
> Almost all of our tables have caching disabled via 'sqlmeta: cacheValues
> = False'.

   sqlmeta.cacheValues disables caching of attributes -- with
cacheValues=False every time a program reads and attribute
(value = row.column) SQLObject issues a SELECT query.
   There is also row caching -- SQLObject caches rows in the connection
cache. The code above is related to that cache. To disable the cache
pass cache=False to connection constructor or in the DB URI:
schema://host:port/db?cache=0.

> No state is kept in memory between requests and we typically write
> significantly more rows to the DB than what is read, ie. A lot of writes
> which are not read often and a few hundred rows which are read often and
> that can be changed by other processes.

   So you need mass insertion. See the related FAQ entry:
http://sqlobject.org/FAQ.html#how-to-do-mass-insertion

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.