Re: [pygccxml-development] Current status on additional caching support

[Allen B. <al...@vr...>]

Roman Yakovenko wrote:
> On 1/5/07, Allen Bierbaum <al...@vr...> wrote:
>   
>> I have been out of the loop for a couple of months and I was wondering
>> what the current status on supporting additional caching with py++ is.
>>     
>
> There were no changes in this area.
>
>   
>> I know that Matthias had proposed some ideas in October that would
>> minimize the number of symbols pulled in from gccxml.  What ever
>> happened with that effort?  Are you still using it Matthias?
>>     
>
> module_builder_t.__init__ has "start_with_declarations" argument.
> Take a look on GCC-XML help to find out more about it
>
>   
>> I had tried to increase performance by caching the decl parsing and
>> query optimization run in module_builder.init().  This worked very well
>> for me and I have code in goodie_perf_overrides.py that worked last time
>> I tried, but alas this code seems to have broken since the last time I
>> used it.  My guess is that module builder is now changed.
>>     
>
> May be. I don't support\upgrade goodies.
>   
Yep.  And I would not expect you too.  When I did the update to the 
latest py++ I expected that this would break.
>   
>> Roman: Is this idea of caching additional parts of the module builder
>> initialization something you are interested in integrating into Py++?
>>     
>
> Yes, but you will have to define exactly what you want. The previous one,
> implementation and definition, was problematic.
>   

If you look at goodi_perf_overrides.mb_override__init__ you can get a 
good idea of what I was caching.

The idea is this:

When the module builder is initialized in 
builder.module_builder_t.__init__, it goes through the following steps:

1 - parse the files using gccxml
2 - combine/merge/process all the parsed declarations
3 - run the query optimizer

All of this ends up in the __global_ns member of module_builder.  With 
the current caching in Py++ (non-goodies), the only thing that gets 
cached is the running of the files through gccxml (correct me if I am 
wrong here).  If the source files don't change then the processing from 
steps 2 and 3 is still done on the exact same input even though it will 
have the exact same output.  What my code did was cache the data in 
__global_ns after all the initial processing in module_builder.__init__ 
was complete.

Is that clear for a definition?  As far as implementation, you can see 
how I implemented it but I am fine with any implementation you want to use.

-Allen


>   
>> Are there any other optimizations floating around?
>>     
>
> No. Py++ is pretty usable and fast for me.
>
>   
>> If you can't tell
>> already I am pretty focused on performance on Py++.  In our workflow we
>> do a lot of iterative development and any tool that runs slow can grind
>> development progress to a halt.
>>     
>
> Py++ is now 20 - 30 % slower than it was. New features "dependency manager"
> and warnings introduced additional slowness. Patches are always welcome.
>
>   
>> Anyway, I just wanted to check in and see what is new that I should be
>> looking into.
>>     
>
> http://language-binding.net/pyplusplus/history/history.html
>
>