sqlobject-discuss

[SQLObject] caching an expensive property

[Fetchinson . <fet...@go...>]

What's the best strategy for some kind of caching of an expensive property?

Let's say I have the following:

class obj( SQLObject ):
    somefield = StringCol( )
    someotherfield = StringCol( )

    def _get_expensivestuff( self ):
        # takes a long time
        return result

And if my obj instance is alive for a long time and expensivestuff
gets accessed many times I'd like to just compute result once, store
it somewhere (where?) and return it from there.

I'm aware of memoizing the result of a function using a suitable
decorator but I'm concerned about thread safety and I'm not terribly
knowledgeable about the internals of sqlobject.

Any ideas would help a lot. Thanks,
Daniel


-- 
Psss, psss, put it down! - http://www.cafepress.com/putitdown

Re: [SQLObject] caching an expensive property

[Oleg B. <ph...@ph...>]

Hi!

On Wed, Dec 10, 2014 at 12:47:17AM +0100, "Fetchinson ." <fet...@go...> wrote:
> What's the best strategy for some kind of caching of an expensive property?
> 
> Let's say I have the following:
> 
> class obj( SQLObject ):
>     somefield = StringCol( )
>     someotherfield = StringCol( )
> 
>     def _get_expensivestuff( self ):
>         # takes a long time
>         return result
> 
> And if my obj instance is alive for a long time and expensivestuff
> gets accessed many times I'd like to just compute result once, store
> it somewhere (where?) and return it from there.
> 
> I'm aware of memoizing the result of a function using a suitable
> decorator but I'm concerned about thread safety and I'm not terribly
> knowledgeable about the internals of sqlobject.
> 
> Any ideas would help a lot. Thanks,
> Daniel

   I hope you know that SQLObject converts _get_* and _set_* methods to
properties so you shouldn't decorate the methods. Also most caching
decorators are not thread-safe so you'd better implement your own
caching with proper locking.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

Re: [SQLObject] caching an expensive property

[Fetchinson . <fet...@go...>]

On 12/10/14, Oleg Broytman <ph...@ph...> wrote:
> Hi!
>
> On Wed, Dec 10, 2014 at 12:47:17AM +0100, "Fetchinson ."
> <fet...@go...> wrote:
>> What's the best strategy for some kind of caching of an expensive
>> property?
>>
>> Let's say I have the following:
>>
>> class obj( SQLObject ):
>>     somefield = StringCol( )
>>     someotherfield = StringCol( )
>>
>>     def _get_expensivestuff( self ):
>>         # takes a long time
>>         return result
>>
>> And if my obj instance is alive for a long time and expensivestuff
>> gets accessed many times I'd like to just compute result once, store
>> it somewhere (where?) and return it from there.
>>
>> I'm aware of memoizing the result of a function using a suitable
>> decorator but I'm concerned about thread safety and I'm not terribly
>> knowledgeable about the internals of sqlobject.
>>
>> Any ideas would help a lot. Thanks,
>> Daniel
>
>    I hope you know that SQLObject converts _get_* and _set_* methods to
> properties so you shouldn't decorate the methods.

Sure! I really like this feature of SQLObject.

> Also most caching
> decorators are not thread-safe so you'd better implement your own
> caching with proper locking.

Well, that was exactly the question, how would I do that? :)
Precisely for the reason you mention above I was wary of using a more
or less standard caching decorator. Can't I attach the computed value
to the instance itself somehow by setting a new property?

Cheers,
Daniel


> Oleg.
> --
>      Oleg Broytman            http://phdru.name/            ph...@ph...
>            Programmers don't die, they just GOSUB without RETURN.
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> _______________________________________________
> sqlobject-discuss mailing list
> sql...@li...
> https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
>


-- 
Psss, psss, put it down! - http://www.cafepress.com/putitdown

Re: [SQLObject] caching an expensive property

[Oleg B. <ph...@ph...>]

On Wed, Dec 10, 2014 at 01:09:50AM +0100, "Fetchinson ." <fet...@go...> wrote:
> On 12/10/14, Oleg Broytman <ph...@ph...> wrote:
> > you'd better implement your own
> > caching with proper locking.
> 
> Well, that was exactly the question, how would I do that? :)
> Precisely for the reason you mention above I was wary of using a more
> or less standard caching decorator. Can't I attach the computed value
> to the instance itself somehow by setting a new property?

   The classical approach is:

    def __init__(self):
        self.__cache = None
        self.__cache_lock = Lock()

    def _get_value(self):
        if self.__cache is not None:
            return self.__cache
        self.__cache_lock.acquire()
        try:
            if self.__cache is not None: # Calculated in another thread
                return self.__cache
            self.__cache = ...do expensive calculation once...
            return self.__cache
        finally: # finally works both for exceptions and returns
            self.__cache_lock.release()

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

Re: [SQLObject] caching an expensive property

[Fetchinson . <fet...@go...>]

On 12/10/14, Oleg Broytman <ph...@ph...> wrote:
> On Wed, Dec 10, 2014 at 01:09:50AM +0100, "Fetchinson ."
> <fet...@go...> wrote:
>> On 12/10/14, Oleg Broytman <ph...@ph...> wrote:
>> > you'd better implement your own
>> > caching with proper locking.
>>
>> Well, that was exactly the question, how would I do that? :)
>> Precisely for the reason you mention above I was wary of using a more
>> or less standard caching decorator. Can't I attach the computed value
>> to the instance itself somehow by setting a new property?
>
>    The classical approach is:
>
>     def __init__(self):
>         self.__cache = None
>         self.__cache_lock = Lock()
>
>     def _get_value(self):
>         if self.__cache is not None:
>             return self.__cache
>         self.__cache_lock.acquire()
>         try:
>             if self.__cache is not None: # Calculated in another thread
>                 return self.__cache
>             self.__cache = ...do expensive calculation once...
>             return self.__cache
>         finally: # finally works both for exceptions and returns
>             self.__cache_lock.release()

Great, thanks a lot, this will get me started.

But I was thinking, maybe I'm doing things not in an optimal way: the
reason I thought about caching is that my expensive property really
takes a long time. I have 'collections' which can contain 'articles'
and 'articles' can be 'tagged'. So there is a many-to-many
relationship between 'articles' and 'tags' done by RelatedJoin in both
directions. What I'd like to have is for each 'collection' have a list
of [ ( 'tag1', 'number-of-articles' ), ( 'tag2', 'number-of-articles'
), ..... ] so for each 'collection' a list of all tags and the number
of articles associated with each tag. And for a collection of even
moderate size, let's say about 1000 articles and about 100 tags, this
take on the order of 2-3 seconds. This is my current approach:

class collection( SQLObject ):
    articles = MultipleJoin( 'article' )

class article( SQLObject ):
    tags = RelatedJoin( 'tag' )

class tag( SQLObject ):
    name = StringCol( )
    articles = RelatedJoin( 'article' )
    def _get_articlecount( self ):
        """Count number of articles with given tag."""
        # maybe this could be done better?
        return tag.select( tag.q.name == self.name ).throughTo.items.count( )

coll = collection.get( 1 )
# get list of all articles, then list of all distinct tags of all articles
# I guess this could be done also better?
tags = { }
for art in coll.articles:
    for tag in art.tags:
        tags[tag] = 1
tags = tags.keys( )
# the above is quite slow, but the following is slower still
# how could I speed this up?
result = [ ( tag, tag.articlecount ) for tag in tags ]


Cheers,
Daniel


> Oleg.
> --
>      Oleg Broytman            http://phdru.name/            ph...@ph...
>            Programmers don't die, they just GOSUB without RETURN.
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> _______________________________________________
> sqlobject-discuss mailing list
> sql...@li...
> https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
>


-- 
Psss, psss, put it down! - http://www.cafepress.com/putitdown

Re: [SQLObject] caching an expensive property

[Fetchinson . <fet...@go...>]

On 12/10/14, Oleg Broytman <ph...@ph...> wrote:
> On Wed, Dec 10, 2014 at 01:09:50AM +0100, "Fetchinson ."
> <fet...@go...> wrote:
>> On 12/10/14, Oleg Broytman <ph...@ph...> wrote:
>> > you'd better implement your own
>> > caching with proper locking.
>>
>> Well, that was exactly the question, how would I do that? :)
>> Precisely for the reason you mention above I was wary of using a more
>> or less standard caching decorator. Can't I attach the computed value
>> to the instance itself somehow by setting a new property?
>
>    The classical approach is:
>
>     def __init__(self):
>         self.__cache = None
>         self.__cache_lock = Lock()
>
>     def _get_value(self):
>         if self.__cache is not None:
>             return self.__cache
>         self.__cache_lock.acquire()
>         try:
>             if self.__cache is not None: # Calculated in another thread
>                 return self.__cache
>             self.__cache = ...do expensive calculation once...
>             return self.__cache
>         finally: # finally works both for exceptions and returns
>             self.__cache_lock.release()

Hi Oleg, thanks again, this pushed me in the right direction and found

http://code.activestate.com/recipes/302997/

which is a pretty cool caching object including an implementation for
functions, just what I need. Now my application is a lot faster, in
some cases by a factor of 10 :)

Right now I'm happy with how things are, but I suspect that the
initial slowness was simply because I'm not very good with SQL or
SQLObject in particular and I'm not able to write the most optimal
queries. I guess I'll ask you about these things in a separate thread
:)

Thanks again,
Daniel


> Oleg.
> --
>      Oleg Broytman            http://phdru.name/            ph...@ph...
>            Programmers don't die, they just GOSUB without RETURN.
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> _______________________________________________
> sqlobject-discuss mailing list
> sql...@li...
> https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
>


-- 
Psss, psss, put it down! - http://www.cafepress.com/putitdown

Re: [SQLObject] caching an expensive property

[Oleg B. <ph...@ph...>]

Hi!

On Thu, Dec 11, 2014 at 12:34:35AM +0100, "Fetchinson ." <fet...@go...> wrote:
> Hi Oleg, thanks again, this pushed me in the right direction and found
> 
> http://code.activestate.com/recipes/302997/

   Very interesting but overly complex for my taste.

> I'm not very good with SQL or
> SQLObject in particular

   Don't worry, I'm not good with them either! ;-)

> Thanks again,
> Daniel

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.