pygccxml-development

[pygccxml-development] Current status on additional caching support

[Allen B. <al...@vr...>]

I have been out of the loop for a couple of months and I was wondering 
what the current status on supporting additional caching with py++ is.

I know that Matthias had proposed some ideas in October that would 
minimize the number of symbols pulled in from gccxml.  What ever 
happened with that effort?  Are you still using it Matthias?

I had tried to increase performance by caching the decl parsing and 
query optimization run in module_builder.init().  This worked very well 
for me and I have code in goodie_perf_overrides.py that worked last time 
I tried, but alas this code seems to have broken since the last time I 
used it.  My guess is that module builder is now changed.

Roman: Is this idea of caching additional parts of the module builder 
initialization something you are interested in integrating into Py++?  
Atleast for me it had a significant impact on performance and made Py++ 
much more usable.

Are there any other optimizations floating around?  If you can't tell 
already I am pretty focused on performance on Py++.  In our workflow we 
do a lot of iterative development and any tool that runs slow can grind 
development progress to a halt.

Anyway, I just wanted to check in and see what is new that I should be 
looking into.

Thanks,
Allen

Re: [pygccxml-development] Current status on additional caching support

[Roman Y. <rom...@gm...>]

On 1/5/07, Allen Bierbaum <al...@vr...> wrote:
> I have been out of the loop for a couple of months and I was wondering
> what the current status on supporting additional caching with py++ is.

There were no changes in this area.

> I know that Matthias had proposed some ideas in October that would
> minimize the number of symbols pulled in from gccxml.  What ever
> happened with that effort?  Are you still using it Matthias?

module_builder_t.__init__ has "start_with_declarations" argument.
Take a look on GCC-XML help to find out more about it

> I had tried to increase performance by caching the decl parsing and
> query optimization run in module_builder.init().  This worked very well
> for me and I have code in goodie_perf_overrides.py that worked last time
> I tried, but alas this code seems to have broken since the last time I
> used it.  My guess is that module builder is now changed.

May be. I don't support\upgrade goodies.

> Roman: Is this idea of caching additional parts of the module builder
> initialization something you are interested in integrating into Py++?

Yes, but you will have to define exactly what you want. The previous one,
implementation and definition, was problematic.

> Are there any other optimizations floating around?

No. Py++ is pretty usable and fast for me.

> If you can't tell
> already I am pretty focused on performance on Py++.  In our workflow we
> do a lot of iterative development and any tool that runs slow can grind
> development progress to a halt.

Py++ is now 20 - 30 % slower than it was. New features "dependency manager"
and warnings introduced additional slowness. Patches are always welcome.

> Anyway, I just wanted to check in and see what is new that I should be
> looking into.

http://language-binding.net/pyplusplus/history/history.html

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Current status on additional caching support

[Allen B. <al...@vr...>]

Roman Yakovenko wrote:
> On 1/5/07, Allen Bierbaum <al...@vr...> wrote:
>   
>> I have been out of the loop for a couple of months and I was wondering
>> what the current status on supporting additional caching with py++ is.
>>     
>
> There were no changes in this area.
>
>   
>> I know that Matthias had proposed some ideas in October that would
>> minimize the number of symbols pulled in from gccxml.  What ever
>> happened with that effort?  Are you still using it Matthias?
>>     
>
> module_builder_t.__init__ has "start_with_declarations" argument.
> Take a look on GCC-XML help to find out more about it
>
>   
>> I had tried to increase performance by caching the decl parsing and
>> query optimization run in module_builder.init().  This worked very well
>> for me and I have code in goodie_perf_overrides.py that worked last time
>> I tried, but alas this code seems to have broken since the last time I
>> used it.  My guess is that module builder is now changed.
>>     
>
> May be. I don't support\upgrade goodies.
>   
Yep.  And I would not expect you too.  When I did the update to the 
latest py++ I expected that this would break.
>   
>> Roman: Is this idea of caching additional parts of the module builder
>> initialization something you are interested in integrating into Py++?
>>     
>
> Yes, but you will have to define exactly what you want. The previous one,
> implementation and definition, was problematic.
>   

If you look at goodi_perf_overrides.mb_override__init__ you can get a 
good idea of what I was caching.

The idea is this:

When the module builder is initialized in 
builder.module_builder_t.__init__, it goes through the following steps:

1 - parse the files using gccxml
2 - combine/merge/process all the parsed declarations
3 - run the query optimizer

All of this ends up in the __global_ns member of module_builder.  With 
the current caching in Py++ (non-goodies), the only thing that gets 
cached is the running of the files through gccxml (correct me if I am 
wrong here).  If the source files don't change then the processing from 
steps 2 and 3 is still done on the exact same input even though it will 
have the exact same output.  What my code did was cache the data in 
__global_ns after all the initial processing in module_builder.__init__ 
was complete.

Is that clear for a definition?  As far as implementation, you can see 
how I implemented it but I am fine with any implementation you want to use.

-Allen


>   
>> Are there any other optimizations floating around?
>>     
>
> No. Py++ is pretty usable and fast for me.
>
>   
>> If you can't tell
>> already I am pretty focused on performance on Py++.  In our workflow we
>> do a lot of iterative development and any tool that runs slow can grind
>> development progress to a halt.
>>     
>
> Py++ is now 20 - 30 % slower than it was. New features "dependency manager"
> and warnings introduced additional slowness. Patches are always welcome.
>
>   
>> Anyway, I just wanted to check in and see what is new that I should be
>> looking into.
>>     
>
> http://language-binding.net/pyplusplus/history/history.html
>
>   

Re: [pygccxml-development] Current status on additional caching support

[Roman Y. <rom...@gm...>]

On 1/6/07, Allen Bierbaum <al...@vr...> wrote:
> If you look at goodi_perf_overrides.mb_override__init__ you can get a
> good idea of what I was caching.
> ...
> Is that clear for a definition?  As far as implementation, you can see
> how I implemented it but I am fine with any implementation you want to use.

Now I remember what I didn't like - the implementation :-(.
I think Py++ should define new class that derives from pygccxml cache class and
implements all the work. Thus you will eliminate the unnecessary dumping of the
whole declaration tree. In your current implementation it will happen twice.

Also before introducing such functionality I would like to know the numbers.
How much time it will save to you?

Allen performance is not a top priority right now. The top priorities are:
* to improve alias handling in the huge projects
* single click setup for the whole environment
* better documentation
* new transformers

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Current status on additional caching support

[Matthias B. <ba...@ir...>]

Allen Bierbaum wrote:
> I know that Matthias had proposed some ideas in October that would 
> minimize the number of symbols pulled in from gccxml.  What ever 
> happened with that effort?  Are you still using it Matthias?

Yes, so far I'm still using it. Maybe with some additional work I could 
have also used the -fxml-start option from gccxml but at least an 
initial quick test had revealed some slight problems with it (I don't 
remember the details though).
My other optimization stuff in pypp_api (i.e. the query cache) is still 
not committed because I still have to catch up with the current version 
of Py++.

I'm currently trying to get everything working on OSX as I got myself a 
new Apple notebook. Basically it all works already, but I have to clean 
up some stuff. Maybe after that I can look into getting the current 
version of Py++ (but I'm afraid this will mean that I have to update a 
lot of things in my code to get it working with the latest version).

- Matthias -

Re: [pygccxml-development] Current status on additional caching support

[Roman Y. <rom...@gm...>]

On 1/10/07, Matthias Baas <ba...@ir...> wrote:
> Allen Bierbaum wrote:
> > I know that Matthias had proposed some ideas in October that would
> > minimize the number of symbols pulled in from gccxml.  What ever
> > happened with that effort?  Are you still using it Matthias?
>
> Yes, so far I'm still using it. Maybe with some additional work I could
> have also used the -fxml-start option from gccxml but at least an
> initial quick test had revealed some slight problems with it (I don't
> remember the details though).
> My other optimization stuff in pypp_api (i.e. the query cache) is still
> not committed because I still have to catch up with the current version
> of Py++.
>
> I'm currently trying to get everything working on OSX as I got myself a
> new Apple notebook. Basically it all works already, but I have to clean
> up some stuff. Maybe after that I can look into getting the current
> version of Py++ (but I'm afraid this will mean that I have to update a
> lot of things in my code to get it working with the latest version).

Except function transformation, everything should work.


-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/