pygccxml-development

[pygccxml-development] performance: make_flatten vs. _make_flatten_generator

[Allen B. <al...@vr...>]

Roman:

It looks like about 1/3 of the time is now being spent inside 
proceed_single in make_flatten. 

Is there any reason I shouldn't be able to use _make_flatten_generator 
instead of make_flatten so I can remove the performance hit of 
allocating the memory for such a large list every time make_flatten is 
called?

-Allen

Re: [pygccxml-development] Another performance tweak

[Allen B. <al...@vr...>]

Roman Yakovenko wrote:

> On 8/27/06, Allen Bierbaum <al...@vr...> wrote:
>
>> I have included comments below.  Before that though...
>>
>> I have found a few more optimizations:
>>
>> - Making create_identifier return identity adds about 15% to performance
>> - Adding a cache to module_builder that caches the global namespace tree
>> after parsing, joining, and running the query optimizer.  This boosts
>> performance by another 40%.  (both by saving more startup and making the
>> query optimization initialization less costly so it can be used)
>
>
> Can you  explain relationship between it and declarations cache in 
> pygccxml?

The module builder cache supercedes it. 

The module builder cache caches the decl tree after all loading, parsing 
in pygccxml and the additional processing in pyplusplus (optimization 
initialization).  The decl cache can/is still used if you don't use the 
module builder cache, but when using the module builder cache the decl 
cache is not really needed.

>> So now I have my run time down to aroung 59 seconds.  Although I would
>> really like to make it go even faster I think taking the running time
>> down from the original 12 minutes to 1 minute now is a good enough
>> improvement to make it so I can use py++ much easier in normal 
>> development.
>
>
> :-)
>
>> Roman:  Do you want a patch for module_builder caching or do you just
>> want me to commit it?  (all the changes are in the __init__ method and
>> are very clear so you can see what is in there and change it if you 
>> prefer)
>
>
> Please commit them, but still explain the relationship :-)

Okay.  I should have time to commit it later today.

>
>> Now on to the comments....
>>
>> Roman Yakovenko wrote:
>>
>> > On 8/26/06, Allen Bierbaum <al...@vr...> wrote:
>> >
>> >> >> - Caching the results of type_traits.remove_alias improved
>> >> performance
>> >> >> by 20%
>> >> >
>> >> >
>> >> > Yes, this algorithms is used every where. Please don't commit this
>> >> > change, I'd like
>> >> > to think about it. I see next problem here: there are a lot of 
>> types,
>> >> > so the cache
>> >> > memory size could go out of control. Can you find out how many 
>> memory
>> >> > Py++ takes with\without the cache?
>> >>
>> >> I can try to take a look at this.  I don't have an easy way to do it
>> >> right now but I will try later.
>> >
>> >
>> > I am think that I don't want to introduce cache of results of
>> > type_traits.
>> > I don't feel comfortable with them. One of the reasons is that it is
>> > not that easy to
>> > disable them.
>>
>> Maybe I am missing something, but why can't we control them by
>> introducing a flag in type_traits like "caching_enabled" or something
>> and then just testing that whenever the method is called.
>>
>> It would be a shame not to optimize this method when it gives a 20%
>> performance boost.
>
>
> You are right, I don't have good reason. I will commit the patch. By
> the way it was
> a tricky one. One of my test failed. It was very difficult ( few hours
> with debugger )
> to find the problem. That is exactly the reason why I hate cache, but
> what can I do
> if you like it :-) ?
>
>> Are you suggesting I add this as a new method and change the code that
>> calls it?
>
>
> No. I just committed the results.
>
>> In my own local copy I have replaced "created_identifier" with an
>> identify function and am getting very good results.
>
>
> Why do you have goodies? For a time being please put it in a goodies.

I had trouble putting it into goodies as an override because it looks 
like the method comes in through multiple module aliases.  In other 
words I haven't found a way to override all the of the uses of the method. 

I have run into a problem like this on another project and it is really 
tough to track down.  Basically it probably means there are multiple 
places where the module with create_identifier are being imported and 
the imports are using slightly different names to import the module.  So 
sys.modules[] ends up with multiple entries that are both pointing to 
different imports of the same code.  I think future versions of python 
are going to fix this problem by making you have to use the full 
package.module.name import no matter what file you are in, but for now 
this is a hard issue to track down.

>> Right now I am not quite ready to update to your changes.  I have some
>> concerns about the complexity and performance of the way you implemented
>> this.
>
>
> Do you have the numbers? I did not read the benchmark results. I will
> do this later.

I don't have numbers.  I didn't want to update and deal with the 
conflicts only to find I didn't want the update.  With the changes you 
have made now I think it will probably be safe for me to update.  I will 
let you know if something goes bad.

> You are right, I introduced one more "get attribute". I have 1 good 
> reason
> for this maintainability. I'd like to keep all "cached" values in 
> single place.
> Thus, I don't have to scan sources for "what I am caching" and more over
> it is very easy to create documentation to it. I am sure you will not 
> see this
> "get attribute" in the benchmark
>
>
>> - Why is there an "algorithms_cache_t" object as the base class to
>> "declaration_algs_cache_t"?  The split with a base class doesn't seem to
>> serve much of a purpose.
>
>
> Enable\disable functionality. But I think it is useless now. The
> better idea is to
> introduce type_algs_cache_t class with class variable , that will 
> control all
> instances of type_algs_cache_t.
>
>> - What is the performance implication of using "properties"?  I know you
>> like to use them in your code but how do they affect performance?
>
>
> I tried with and without them and I did not see the difference. May be 
> you will
> see. If you do see the difference, we can leave set_* methods and remove
> property
>
>> - Handling of enabled flag.  I think the handling of the enabled flag
>> should be done in the "set" method instead of the get method.  As it
>> stands with your change, our optimized path will require two if tests
>> (one in the local method to test for None, and one in the get method to
>> test enabled).  If we moved the enabled test to the set method we would
>> only pay the cost of that if when we have optimizations diabled.
>
>
> You are right. Fixed.
>
>>
>> > I left some interesting problem to you: it is possible to optimize
>> > declaration_path
>> > algorithm even more. In most cases its complexity could be O(1). This
>> > could be done
>> > by saving intermediate results. Another way to say it: to re-use
>> > parent declaration path
>> > cache.
>
>
> What about this?

I haven't looked at this at all.  I am not sure I understand what is 
going on here well enough to do it right and it isn't showing up high in 
any of my traces anymore.  So it really isn't a high priority for me.

-Allen

Re: [pygccxml-development] [pygccxml-commit] SF.net SVN: pygccxml: [451] pygccxml_dev/pygccxml/declarations/mdecl_wrapper.py

[Roman Y. <rom...@gm...>]

On 8/27/06, Allen Bierbaum <al...@vr...> wrote:
> Matthias:
>
> A word of warning, the most common problem I find with the settings in
> this file is that I often get conflicts when I update.  I haven't tried
> to track it down and make it more conflict free, but it is something to
> watch for.

I re-factored those files in a such way, that will allow you and
Matthias to add and
commit your settings.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Another performance tweak

[Roman Y. <rom...@gm...>]

On 8/27/06, Allen Bierbaum <al...@vr...> wrote:
> I have included comments below.  Before that though...
>
> I have found a few more optimizations:
>
> - Making create_identifier return identity adds about 15% to performance
> - Adding a cache to module_builder that caches the global namespace tree
> after parsing, joining, and running the query optimizer.  This boosts
> performance by another 40%.  (both by saving more startup and making the
> query optimization initialization less costly so it can be used)

Can you  explain relationship between it and declarations cache in pygccxml?

> So now I have my run time down to aroung 59 seconds.  Although I would
> really like to make it go even faster I think taking the running time
> down from the original 12 minutes to 1 minute now is a good enough
> improvement to make it so I can use py++ much easier in normal development.

:-)

> Roman:  Do you want a patch for module_builder caching or do you just
> want me to commit it?  (all the changes are in the __init__ method and
> are very clear so you can see what is in there and change it if you prefer)

Please commit them, but still explain the relationship :-)

> Now on to the comments....
>
> Roman Yakovenko wrote:
>
> > On 8/26/06, Allen Bierbaum <al...@vr...> wrote:
> >
> >> >> - Caching the results of type_traits.remove_alias improved
> >> performance
> >> >> by 20%
> >> >
> >> >
> >> > Yes, this algorithms is used every where. Please don't commit this
> >> > change, I'd like
> >> > to think about it. I see next problem here: there are a lot of types,
> >> > so the cache
> >> > memory size could go out of control. Can you find out how many memory
> >> > Py++ takes with\without the cache?
> >>
> >> I can try to take a look at this.  I don't have an easy way to do it
> >> right now but I will try later.
> >
> >
> > I am think that I don't want to introduce cache of results of
> > type_traits.
> > I don't feel comfortable with them. One of the reasons is that it is
> > not that easy to
> > disable them.
>
> Maybe I am missing something, but why can't we control them by
> introducing a flag in type_traits like "caching_enabled" or something
> and then just testing that whenever the method is called.
>
> It would be a shame not to optimize this method when it gives a 20%
> performance boost.

You are right, I don't have good reason. I will commit the patch. By
the way it was
a tricky one. One of my test failed. It was very difficult ( few hours
with debugger )
to find the problem. That is exactly the reason why I hate cache, but
what can I do
if you like it :-) ?

> Are you suggesting I add this as a new method and change the code that
> calls it?

No. I just committed the results.

> In my own local copy I have replaced "created_identifier" with an
> identify function and am getting very good results.

Why do you have goodies? For a time being please put it in a goodies.

> Right now I am not quite ready to update to your changes.  I have some
> concerns about the complexity and performance of the way you implemented
> this.

Do you have the numbers? I did not read the benchmark results. I will
do this later.
You are right, I introduced one more "get attribute". I have 1 good reason
for this maintainability. I'd like to keep all "cached" values in single place.
Thus, I don't have to scan sources for "what I am caching" and more over
it is very easy to create documentation to it. I am sure you will not see this
 "get attribute" in the benchmark


> - Why is there an "algorithms_cache_t" object as the base class to
> "declaration_algs_cache_t"?  The split with a base class doesn't seem to
> serve much of a purpose.

Enable\disable functionality. But I think it is useless now. The
better idea is to
introduce type_algs_cache_t class with class variable , that will control all
instances of type_algs_cache_t.

> - What is the performance implication of using "properties"?  I know you
> like to use them in your code but how do they affect performance?

I tried with and without them and I did not see the difference. May be you will
see. If you do see the difference, we can leave set_* methods and remove
property

> - Handling of enabled flag.  I think the handling of the enabled flag
> should be done in the "set" method instead of the get method.  As it
> stands with your change, our optimized path will require two if tests
> (one in the local method to test for None, and one in the get method to
> test enabled).  If we moved the enabled test to the set method we would
> only pay the cost of that if when we have optimizations diabled.

You are right. Fixed.

>
> > I left some interesting problem to you: it is possible to optimize
> > declaration_path
> > algorithm even more. In most cases its complexity could be O(1). This
> > could be done
> > by saving intermediate results. Another way to say it: to re-use
> > parent declaration path
> > cache.

What about this?

> > Can you publish top N lines of your benchmark result?
> >
> Sure.  Give me a bit to rerun it.  I can probably just send you a
> compress hotspot file so you can see the entire details.

Thanks, I got it and will take a look later, okay?


-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] [pygccxml-commit] SF.net SVN: pygccxml: [451] pygccxml_dev/pygccxml/declarations/mdecl_wrapper.py

[Allen B. <al...@vr...>]

Matthias:

A word of warning, the most common problem I find with the settings in 
this file is that I often get conflicts when I update.  I haven't tried 
to track it down and make it more conflict free, but it is something to 
watch for.

-Allen

Roman Yakovenko wrote:

>On 8/27/06, Matthias Baas <ba...@ir...> wrote:
>  
>
>>Roman Yakovenko wrote:
>>    
>>
>>>It could be nice if you and Matthias could run unit tests on regular basis.
>>>      
>>>
>>What additional configuration is required to run the unit tests? I
>>always get an error 'include path "" should exists or to be a valid
>>directory name.' whenever I run a test.
>>    
>>
>
>There are  3 files you should add you settings in:
>pygccxml:
>    unittests/autoconfig.py
>Py++:
>    pyplusplus_dev/ environment.py
>    pyplusplus_dev/unittests/autoconfig.py
>
>    You will have to in scons.
>
>Also both tests should be run, when pwd is relevant unit tests directory.
>Let me know if you need more help with this.
>
>  
>

Re: [pygccxml-development] [pygccxml-commit] SF.net SVN: pygccxml: [451] pygccxml_dev/pygccxml/declarations/mdecl_wrapper.py

[Roman Y. <rom...@gm...>]

On 8/27/06, Matthias Baas <ba...@ir...> wrote:
> Roman Yakovenko wrote:
> > It could be nice if you and Matthias could run unit tests on regular basis.
>
> What additional configuration is required to run the unit tests? I
> always get an error 'include path "" should exists or to be a valid
> directory name.' whenever I run a test.

There are  3 files you should add you settings in:
pygccxml:
    unittests/autoconfig.py
Py++:
    pyplusplus_dev/ environment.py
    pyplusplus_dev/unittests/autoconfig.py

    You will have to in scons.

Also both tests should be run, when pwd is relevant unit tests directory.
Let me know if you need more help with this.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Another performance tweak

[Allen B. <al...@vr...>]

I have included comments below.  Before that though...

I have found a few more optimizations:

- Making create_identifier return identity adds about 15% to performance
- Adding a cache to module_builder that caches the global namespace tree 
after parsing, joining, and running the query optimizer.  This boosts 
performance by another 40%.  (both by saving more startup and making the 
query optimization initialization less costly so it can be used)

So now I have my run time down to aroung 59 seconds.  Although I would 
really like to make it go even faster I think taking the running time 
down from the original 12 minutes to 1 minute now is a good enough 
improvement to make it so I can use py++ much easier in normal development.

Roman:  Do you want a patch for module_builder caching or do you just 
want me to commit it?  (all the changes are in the __init__ method and 
are very clear so you can see what is in there and change it if you prefer)

Now on to the comments....

Roman Yakovenko wrote:

> On 8/26/06, Allen Bierbaum <al...@vr...> wrote:
>
>> >> - Caching the results of type_traits.remove_alias improved 
>> performance
>> >> by 20%
>> >
>> >
>> > Yes, this algorithms is used every where. Please don't commit this
>> > change, I'd like
>> > to think about it. I see next problem here: there are a lot of types,
>> > so the cache
>> > memory size could go out of control. Can you find out how many memory
>> > Py++ takes with\without the cache?
>>
>> I can try to take a look at this.  I don't have an easy way to do it
>> right now but I will try later.
>
>
> I am think that I don't want to introduce cache of results of 
> type_traits.
> I don't feel comfortable with them. One of the reasons is that it is
> not that easy to
> disable them.

Maybe I am missing something, but why can't we control them by 
introducing a flag in type_traits like "caching_enabled" or something 
and then just testing that whenever the method is called.

It would be a shame not to optimize this method when it gives a 20% 
performance boost.

>> If you never call add_namespace_usage or add_namespace_alias, then will
>> create_identifier ever need to do anything?  Maybe we could make an
>> optimization that keeps a global flag around and just skips the work in
>> this method if you never set any namespace information.  What do you 
>> think?
>
>
> I think that you can replace create_identifier with "identity"
> function and this solution
> will always work :-)
>
> def create_identifier_fast( creator, full_name ):
>    return full_name
>
Are you suggesting I add this as a new method and change the code that 
calls it?

In my own local copy I have replaced "created_identifier" with an 
identify function and am getting very good results.

> And than replace create_identifier with the fast one from goodies.
>
> I committed small changes to "optimization" feature. New feature: it
> is possible to
> control cache in all project:
>
> for d in mb.decls():
>    d.cache.disable()
>
> It is not easy to achieve same goal with types cache.
>
> Would you mind to add documentation strings to the module?

Right now I am not quite ready to update to your changes.  I have some 
concerns about the complexity and performance of the way you implemented 
this.

- Why is there an "algorithms_cache_t" object as the base class to 
"declaration_algs_cache_t"?  The split with a base class doesn't seem to 
serve much of a purpose.
- What is the performance implication of using "properties"?  I know you 
like to use them in your code but how do they affect performance?
- Handling of enabled flag.  I think the handling of the enabled flag 
should be done in the "set" method instead of the get method.  As it 
stands with your change, our optimized path will require two if tests 
(one in the local method to test for None, and one in the get method to 
test enabled).  If we moved the enabled test to the set method we would 
only pay the cost of that if when we have optimizations diabled.


> I left some interesting problem to you: it is possible to optimize
> declaration_path
> algorithm even more. In most cases its complexity could be O(1). This
> could be done
> by saving intermediate results. Another way to say it: to re-use
> parent declaration path
> cache.
>
> Can you publish top N lines of your benchmark result?
>
Sure.  Give me a bit to rerun it.  I can probably just send you a 
compress hotspot file so you can see the entire details.

-Allen

Re: [pygccxml-development] [pygccxml-commit] SF.net SVN: pygccxml: [451] pygccxml_dev/pygccxml/declarations/mdecl_wrapper.py

[Matthias B. <ba...@ir...>]

Roman Yakovenko wrote:
> It could be nice if you and Matthias could run unit tests on regular basis.

What additional configuration is required to run the unit tests? I 
always get an error 'include path "" should exists or to be a valid 
directory name.' whenever I run a test.

- Matthias -

Re: [pygccxml-development] Another performance tweak

[Roman Y. <rom...@gm...>]

On 8/26/06, Allen Bierbaum <al...@vr...> wrote:
> >> - Caching the results of type_traits.remove_alias improved performance
> >> by 20%
> >
> >
> > Yes, this algorithms is used every where. Please don't commit this
> > change, I'd like
> > to think about it. I see next problem here: there are a lot of types,
> > so the cache
> > memory size could go out of control. Can you find out how many memory
> > Py++ takes with\without the cache?
>
> I can try to take a look at this.  I don't have an easy way to do it
> right now but I will try later.

I am think that I don't want to introduce cache of results of type_traits.
I don't feel comfortable with them. One of the reasons is that it is
not that easy to
disable them.

> If you never call add_namespace_usage or add_namespace_alias, then will
> create_identifier ever need to do anything?  Maybe we could make an
> optimization that keeps a global flag around and just skips the work in
> this method if you never set any namespace information.  What do you think?

I think that you can replace create_identifier with "identity"
function and this solution
will always work :-)

def create_identifier_fast( creator, full_name ):
    return full_name

And than replace create_identifier with the fast one from goodies.

I committed small changes to "optimization" feature. New feature: it
is possible to
control cache in all project:

for d in mb.decls():
    d.cache.disable()

It is not easy to achieve same goal with types cache.

Would you mind to add documentation strings to the module?
I left some interesting problem to you: it is possible to optimize
declaration_path
algorithm even more. In most cases its complexity could be O(1). This
could be done
by saving intermediate results. Another way to say it: to re-use
parent declaration path
cache.

Can you publish top N lines of your benchmark result?

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Another performance tweak

[Allen B. <al...@vr...>]

Roman Yakovenko wrote:

> On 8/26/06, Allen Bierbaum <al...@vr...> wrote:
>
>> I just found some more tweaks.
>
>
> Cool. I just committed a set of changes that will allow us to keep 
> under control
> cache results of different algorithms. I still have to write unit 
> tests. These
> changes allow you to continue grate work. Please take a look on them, 
> what
> do you think?

Not quite how I would have done it, but it looks like it should work.  
It may eliminate an extra attribute lookup I had to do so that could 
increase the performance even more.

>> - Caching the results of type_traits.remove_alias improved performance
>> by 20%
>
>
> Yes, this algorithms is used every where. Please don't commit this
> change, I'd like
> to think about it. I see next problem here: there are a lot of types,
> so the cache
> memory size could go out of control. Can you find out how many memory
> Py++ takes with\without the cache?

I can try to take a look at this.  I don't have an easy way to do it 
right now but I will try later.

>
>> - (danger) Having create_identifier just return the exact same string
>> passed as full_name improved performance another 20%.
>>
>> This last one was interesting because what I found was that (at least in
>> my project) create_identifier was called about 16,000 times and it
>> always returned the same value passed in.  This could be just because of
>> how my project works (no namespace aliases that I know of) and is
>> probably not portable.
>>
>> But, one very strange thing is why is this method being called for names
>> like "boost::python::arg" and "boost::python::default_call_policies"?
>> As far as I understand how py++ is working, these symbols are always
>> going to be valid.  So why waste the time calling an expensive method if
>> we already know the outcome?
>
>
> :-) It seems that you missed the point. Py++ has few cool features.
>
> mb = module_builder_t( ... )
> mb.build_code_creator(...)
> mb.code_creator.add_namespace_alias( "bpl", "::boost::python" )
> or
> mb.code_creator.add_namespace_usage( "::boost::python" )
>
> See what happens.
>
> The short version is that you can add to the code creators tree the 
> namespace
> usage and aliase and Py++ will generate code that takes into account 
> them.
>
> create_identfier function is responsible for creating such identifier.
>
> It is safe to add such optimization, but similar work should be done.
>
If you never call add_namespace_usage or add_namespace_alias, then will 
create_identifier ever need to do anything?  Maybe we could make an 
optimization that keeps a global flag around and just skips the work in 
this method if you never set any namespace information.  What do you think?

-Allen

Re: [pygccxml-development] Another performance tweak

[Roman Y. <rom...@gm...>]

On 8/26/06, Allen Bierbaum <al...@vr...> wrote:
> I just found some more tweaks.

Cool. I just committed a set of changes that will allow us to keep under control
cache results of different algorithms. I still have to write unit tests. These
changes allow you to continue grate work. Please take a look on them, what
do you think?

> - Caching the results of type_traits.remove_alias improved performance
> by 20%

Yes, this algorithms is used every where. Please don't commit this
change, I'd like
to think about it. I see next problem here: there are a lot of types,
so the cache
memory size could go out of control. Can you find out how many memory
Py++ takes with\without the cache?

> - (danger) Having create_identifier just return the exact same string
> passed as full_name improved performance another 20%.
>
> This last one was interesting because what I found was that (at least in
> my project) create_identifier was called about 16,000 times and it
> always returned the same value passed in.  This could be just because of
> how my project works (no namespace aliases that I know of) and is
> probably not portable.
>
> But, one very strange thing is why is this method being called for names
> like "boost::python::arg" and "boost::python::default_call_policies"?
> As far as I understand how py++ is working, these symbols are always
> going to be valid.  So why waste the time calling an expensive method if
> we already know the outcome?

:-) It seems that you missed the point. Py++ has few cool features.

mb = module_builder_t( ... )
mb.build_code_creator(...)
mb.code_creator.add_namespace_alias( "bpl", "::boost::python" )
or
mb.code_creator.add_namespace_usage( "::boost::python" )

See what happens.

The short version is that you can add to the code creators tree the namespace
usage and aliase and Py++ will generate code that takes into account them.

create_identfier function is responsible for creating such identifier.

It is safe to add such optimization, but similar work should be done.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Another performance tweak

[Allen B. <al...@vr...>]

I just found some more tweaks.

- Caching the results of type_traits.remove_alias improved performance 
by 20%

- (danger) Having create_identifier just return the exact same string 
passed as full_name improved performance another 20%.

This last one was interesting because what I found was that (at least in 
my project) create_identifier was called about 16,000 times and it 
always returned the same value passed in.  This could be just because of 
how my project works (no namespace aliases that I know of) and is 
probably not portable. 

But, one very strange thing is why is this method being called for names 
like "boost::python::arg" and "boost::python::default_call_policies"?    
As far as I understand how py++ is working, these symbols are always 
going to be valid.  So why waste the time calling an expensive method if 
we already know the outcome?

-Allen


Allen Bierbaum wrote:

>I just tracked down and fixed another major performance sink.
>
>I saw in the profiler output that the majority of the time from my run 
>was spent in __eq__ in declaration.py (line 121) and __eq_calldef.py 
>line 121.  This is the code that compares two calldefs to see if they 
>are equal.  (note this was also where most of the calls to 
>algorithm.declration_path were coming from).
>
>I was interested in tracking down where all these calls (over 6 million 
>of them) were coming from so I added some code to the __eq__ method of 
>calldef to keep track of all the ways it was called and store how many 
>hits it gets (an example of the output is at the end of the e-mail). 
>
>As a side note, I also counted the return value of true and false 
>separately just for fun.  I found that out of the over 6million times it 
>was called, it returned True only 33 times and those only came from a 
>call path starting with _join_declarations.   Every other test was false 
>every time, so there may be another optimization hiding in here to just 
>not call this test.
>
>As it ended up I found that the vast majority of these calls cam from 
>the member_functions method in scopedef.py.  I traced through there and 
>found that all the the __eq__ calls were coming from some nested calls 
>to _find_out_member_access_type that were coming from 
>access_type_matcher_t.  I never did find out where access_type_matcher_t 
>was coming from since I was just asking for all the members.  
>
>Anyway, the way pygccxml works the decls don't actually know their 
>access type.  Only their parents do.  So if you want to know a decl 
>access type you have to ask the parent and then it loops over all of 
>it's internal members for each access type until it find the one you are 
>asking about.  This meant that the member_functions method was at least 
>O(N^2) and possibly O(N^3).
>
>So back to what I did to fix it.  It seemed to me that for pygccxml the 
>access type of a member should remain static through a single 
>execution.  So I added a caching mechanism to the 
>find_out_member_access_type that just stores the access type with the 
>member decl.  Then the next time it is check we return it directly and 
>skip looping over all the lists and calling __eq__ so many millions of 
>times.
>
>In the end the number of __eq__ calls from  6,010,000  to 271,500.  This 
>took my build type from 344 seconds down to 116 seconds.
>
>So when you combine this change with the one from yesterday the 
>generation process is now 7 times faster.  Not bad for just modifying 
>two methods. :)
>
>-Allen
>
>PS. You can see the PerformanceTuning page on the wiki for pointers to 
>the tools I have been using.
>
>
>---------- Example call chaining for __eq__:  Eq: Called 238772 times 
>and *always* returned false  ------
>  [0, 238772]:  [('gen_bindings.py', 722, '?'), ('gen_bindings.py', 673, 
>'main'), 
>('/home/allenb/python/lib/python/pyplusplus/module_builder/builder.py', 
>236, 'build_code_creator'), 
>('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py', 
>541, 'create'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/algorithm.py', 
>268, 'apply_visitor'), 
>('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py', 
>704, 'visit_class'), 
>('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py', 
>348, '_is_wrapper_needed'), 
>('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py', 
>287, 'redefined_funcs'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py', 
>473, 'member_functions'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py', 
>326, '_find_multiple'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/matcher.py', 49, 
>'find'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py', 
>258, '<lambda>'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py', 83, 
>'__call__'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py', 61, 
>'__call__'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py', 
>478, '__call__'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py', 
>321, 'find_out_member_access_type'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/calldef.py', 310, 
>'__eq__'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/calldef.py', 139, 
>'__eq__')]
>
>
>-------------------------------------------------------------------------
>Using Tomcat but need to do more? Need to support web services, security?
>Get stuff done quickly with pre-integrated technology to make your job easier
>Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>_______________________________________________
>pygccxml-development mailing list
>pyg...@li...
>https://lists.sourceforge.net/lists/listinfo/pygccxml-development
>
>  
>

Re: [pygccxml-development] Another performance tweak

[Roman Y. <rom...@gm...>]

On 8/26/06, Allen Bierbaum <al...@vr...> wrote:
> Do you think there are cases that are not already covered by the current
> unit tests?

Yes.
> > What do you mean? module_builder_t.__init__ optimize_queries is turned on
> > by default. May be I am missing something?
>
> I mean that a month or so ago I found that if I turned them off, my
> script would run faster.  This is still true with the new optimizations.

Strange. We will take a look on this later, okay?

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Another performance tweak

[Allen B. <al...@vr...>]

Roman Yakovenko wrote:

> On 8/26/06, Allen Bierbaum <al...@vr...> wrote:
>
>> Sorry about this one then.  I can back out the revision if you like.
>
>
> No, no, no! You are doing grate job. I just want to create good unit 
> tests
> for every case. It can be very difficult to find a bug if there is a
> cache functionality
> somewhere.
>
Do you think there are cases that are not already covered by the current 
unit tests?

>> See my previous e-mail, I don't know how to finish up the previous
>> change because I don't understand the issue involved.  I made sure the
>> code passed the test suites, but that is about all I can do.  I think
>> you are going to have to look at it or come up with a test case that
>> shows the problem to me.
>
>
> I did not plan to work on this, but I think I will have some free time
> this evening,
> so I will work on both changes you introduced.
>
>> I will just keep any further performance increases in my own tree for 
>> now.
>
>
> Yes please. The better idea is to post them to the list.  It will take
> me day or two
> to complete the task.
>
>> If you are interested in looking into some of this I would point you at
>> check_name.  This method is now one of the most time consuming.  I can't
>> really understand what the implementation is doing but there may be room
>> for improvement.
>
>
> Yes of course. Let me finish with previous 2 and we will fix most( all
> ) of them.
>
>> It is also worth noting that when I turn on optimize_queries, the code
>> runs slower.
>
>
> What do you mean? module_builder_t.__init__ optimize_queries is turned on
> by default. May be I am missing something?

I mean that a month or so ago I found that if I turned them off, my 
script would run faster.  This is still true with the new optimizations.

-Allen

>
>

Re: [pygccxml-development] Another performance tweak

[Roman Y. <rom...@gm...>]

On 8/26/06, Allen Bierbaum <al...@vr...> wrote:
> Sorry about this one then.  I can back out the revision if you like.

No, no, no! You are doing grate job. I just want to create good unit tests
for every case. It can be very difficult to find a bug if there is a
cache functionality
somewhere.

> See my previous e-mail, I don't know how to finish up the previous
> change because I don't understand the issue involved.  I made sure the
> code passed the test suites, but that is about all I can do.  I think
> you are going to have to look at it or come up with a test case that
> shows the problem to me.

I did not plan to work on this, but I think I will have some free time
this evening,
so I will work on both changes you introduced.

> I will just keep any further performance increases in my own tree for now.

Yes please. The better idea is to post them to the list.  It will take
me day or two
to complete the task.

> If you are interested in looking into some of this I would point you at
> check_name.  This method is now one of the most time consuming.  I can't
> really understand what the implementation is doing but there may be room
> for improvement.

Yes of course. Let me finish with previous 2 and we will fix most( all
) of them.

> It is also worth noting that when I turn on optimize_queries, the code
> runs slower.

What do you mean? module_builder_t.__init__ optimize_queries is turned on
by default. May be I am missing something?


-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Another performance tweak

[Allen B. <al...@vr...>]

Roman Yakovenko wrote:

>On 8/26/06, Allen Bierbaum <al...@vr...> wrote:
>  
>
>>I just tracked down and fixed another major performance sink.
>>    
>>
>
>Allen, these are good news, but I explicitly asked you not to introduce new
>optimizations, untill we finish with previous one.
>  
>
Sorry about this one then.  I can back out the revision if you like.

See my previous e-mail, I don't know how to finish up the previous 
change because I don't understand the issue involved.  I made sure the 
code passed the test suites, but that is about all I can do.  I think 
you are going to have to look at it or come up with a test case that 
shows the problem to me.

I will just keep any further performance increases in my own tree for now.

If you are interested in looking into some of this I would point you at 
check_name.  This method is now one of the most time consuming.  I can't 
really understand what the implementation is doing but there may be room 
for improvement.

It is also worth noting that when I turn on optimize_queries, the code 
runs slower.

-Allen

>  
>
>>I saw in the profiler output that the majority of the time from my run
>>was spent in __eq__ in declaration.py (line 121) and __eq_calldef.py
>>line 121.  This is the code that compares two calldefs to see if they
>>are equal.  (note this was also where most of the calls to
>>algorithm.declration_path were coming from).
>>
>>I was interested in tracking down where all these calls (over 6 million
>>of them) were coming from so I added some code to the __eq__ method of
>>calldef to keep track of all the ways it was called and store how many
>>hits it gets (an example of the output is at the end of the e-mail).
>>
>>As a side note, I also counted the return value of true and false
>>separately just for fun.  I found that out of the over 6million times it
>>was called, it returned True only 33 times and those only came from a
>>call path starting with _join_declarations.   Every other test was false
>>every time, so there may be another optimization hiding in here to just
>>not call this test.
>>
>>As it ended up I found that the vast majority of these calls cam from
>>the member_functions method in scopedef.py.  I traced through there and
>>found that all the the __eq__ calls were coming from some nested calls
>>to _find_out_member_access_type that were coming from
>>access_type_matcher_t.  I never did find out where access_type_matcher_t
>>was coming from since I was just asking for all the members.
>>
>>Anyway, the way pygccxml works the decls don't actually know their
>>access type.  Only their parents do.  So if you want to know a decl
>>access type you have to ask the parent and then it loops over all of
>>it's internal members for each access type until it find the one you are
>>asking about.  This meant that the member_functions method was at least
>>O(N^2) and possibly O(N^3).
>>
>>So back to what I did to fix it.  It seemed to me that for pygccxml the
>>access type of a member should remain static through a single
>>execution.  So I added a caching mechanism to the
>>find_out_member_access_type that just stores the access type with the
>>member decl.  Then the next time it is check we return it directly and
>>skip looping over all the lists and calling __eq__ so many millions of
>>times.
>>
>>In the end the number of __eq__ calls from  6,010,000  to 271,500.  This
>>took my build type from 344 seconds down to 116 seconds.
>>
>>So when you combine this change with the one from yesterday the
>>generation process is now 7 times faster.  Not bad for just modifying
>>two methods. :)
>>
>>-Allen
>>
>>PS. You can see the PerformanceTuning page on the wiki for pointers to
>>the tools I have been using.
>>
>>
>>---------- Example call chaining for __eq__:  Eq: Called 238772 times
>>and *always* returned false  ------
>>  [0, 238772]:  [('gen_bindings.py', 722, '?'), ('gen_bindings.py', 673,
>>'main'),
>>('/home/allenb/python/lib/python/pyplusplus/module_builder/builder.py',
>>236, 'build_code_creator'),
>>('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py',
>>541, 'create'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/algorithm.py',
>>268, 'apply_visitor'),
>>('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py',
>>704, 'visit_class'),
>>('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py',
>>348, '_is_wrapper_needed'),
>>('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py',
>>287, 'redefined_funcs'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py',
>>473, 'member_functions'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py',
>>326, '_find_multiple'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/matcher.py', 49,
>>'find'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py',
>>258, '<lambda>'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py', 83,
>>'__call__'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py', 61,
>>'__call__'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py',
>>478, '__call__'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py',
>>321, 'find_out_member_access_type'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/calldef.py', 310,
>>'__eq__'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/calldef.py', 139,
>>'__eq__')]
>>
>>
>>-------------------------------------------------------------------------
>>Using Tomcat but need to do more? Need to support web services, security?
>>Get stuff done quickly with pre-integrated technology to make your job easier
>>Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>>_______________________________________________
>>pygccxml-development mailing list
>>pyg...@li...
>>https://lists.sourceforge.net/lists/listinfo/pygccxml-development
>>
>>    
>>
>
>
>  
>

[pygccxml-development] [new feature]exception translation

[Roman Y. <rom...@gm...>]

Hi. I just committed new feature - exception translation.

In general the only code that a user have to write is "translate"
code. Py++ will
do the rest. It will create translate and register functions.

Usage example:

struct my_exception
{
   std::string& what() throw();
   ...
};

mb = module_builder_t( ... )
exc_class = mb.class_( "my_exception" )
exc_class.exception_translation_code \
    = "PyErr_SetString(PyExc_RuntimeError, exc.what().c_str() );"

Py++ also provides a convenience method, that converts exception to string:

exc_class.translate_exception_to_string(
    'PyExc_RuntimeError', 'exc.what().c_str()'  )

Here you can find the full working example ( unit tests ):
http://svn.sourceforge.net/viewvc/pygccxml/pyplusplus_dev/unittests/cppexceptions_tester.py?view=markup

Enjoy

P.S. Matthias can you update pypp_api?

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] special functions/operators

[Matthias B. <ba...@ir...>]

Neal Becker wrote:
>> root.Class("X").cdef("__iadd__", "addAssign", return_self())
>>
>> (see https://realityforge.vrsource.org/view/PyppApi/Pypp_decorating)
>>
> Doesn't seem to work though, return_self isn't passed to generated code:

Oops, right, policies were ignored, I forgot that this was still on my 
todo list. I've just committed the fix (r470).

- Matthias -

Re: [pygccxml-development] Enums inside a class

[Roman Y. <rom...@gm...>]

On 8/26/06, Balasubramanyam, Shivakumar <sba...@qu...> wrote:
>
>
>
> Roman,
>
> Py++ is not generating code for enums defined inside a class. Are you aware
> of this problem?

No, more over I have such use cases and they works just fine. Please
send me the test case.


-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Another performance tweak

[Roman Y. <rom...@gm...>]

On 8/26/06, Allen Bierbaum <al...@vr...> wrote:
> I just tracked down and fixed another major performance sink.

Allen, these are good news, but I explicitly asked you not to introduce new
optimizations, untill we finish with previous one.

> I saw in the profiler output that the majority of the time from my run
> was spent in __eq__ in declaration.py (line 121) and __eq_calldef.py
> line 121.  This is the code that compares two calldefs to see if they
> are equal.  (note this was also where most of the calls to
> algorithm.declration_path were coming from).
>
> I was interested in tracking down where all these calls (over 6 million
> of them) were coming from so I added some code to the __eq__ method of
> calldef to keep track of all the ways it was called and store how many
> hits it gets (an example of the output is at the end of the e-mail).
>
> As a side note, I also counted the return value of true and false
> separately just for fun.  I found that out of the over 6million times it
> was called, it returned True only 33 times and those only came from a
> call path starting with _join_declarations.   Every other test was false
> every time, so there may be another optimization hiding in here to just
> not call this test.
>
> As it ended up I found that the vast majority of these calls cam from
> the member_functions method in scopedef.py.  I traced through there and
> found that all the the __eq__ calls were coming from some nested calls
> to _find_out_member_access_type that were coming from
> access_type_matcher_t.  I never did find out where access_type_matcher_t
> was coming from since I was just asking for all the members.
>
> Anyway, the way pygccxml works the decls don't actually know their
> access type.  Only their parents do.  So if you want to know a decl
> access type you have to ask the parent and then it loops over all of
> it's internal members for each access type until it find the one you are
> asking about.  This meant that the member_functions method was at least
> O(N^2) and possibly O(N^3).
>
> So back to what I did to fix it.  It seemed to me that for pygccxml the
> access type of a member should remain static through a single
> execution.  So I added a caching mechanism to the
> find_out_member_access_type that just stores the access type with the
> member decl.  Then the next time it is check we return it directly and
> skip looping over all the lists and calling __eq__ so many millions of
> times.
>
> In the end the number of __eq__ calls from  6,010,000  to 271,500.  This
> took my build type from 344 seconds down to 116 seconds.
>
> So when you combine this change with the one from yesterday the
> generation process is now 7 times faster.  Not bad for just modifying
> two methods. :)
>
> -Allen
>
> PS. You can see the PerformanceTuning page on the wiki for pointers to
> the tools I have been using.
>
>
> ---------- Example call chaining for __eq__:  Eq: Called 238772 times
> and *always* returned false  ------
>   [0, 238772]:  [('gen_bindings.py', 722, '?'), ('gen_bindings.py', 673,
> 'main'),
> ('/home/allenb/python/lib/python/pyplusplus/module_builder/builder.py',
> 236, 'build_code_creator'),
> ('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py',
> 541, 'create'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/algorithm.py',
> 268, 'apply_visitor'),
> ('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py',
> 704, 'visit_class'),
> ('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py',
> 348, '_is_wrapper_needed'),
> ('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py',
> 287, 'redefined_funcs'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py',
> 473, 'member_functions'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py',
> 326, '_find_multiple'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/matcher.py', 49,
> 'find'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py',
> 258, '<lambda>'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py', 83,
> '__call__'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py', 61,
> '__call__'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py',
> 478, '__call__'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py',
> 321, 'find_out_member_access_type'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/calldef.py', 310,
> '__eq__'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/calldef.py', 139,
> '__eq__')]
>
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> pygccxml-development mailing list
> pyg...@li...
> https://lists.sourceforge.net/lists/listinfo/pygccxml-development
>


-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

[pygccxml-development] Enums inside a class

[Balasubramanyam, S. <sba...@qu...>]

Roman,

Py++ is not generating code for enums defined inside a class. Are you
aware of this problem?

The class definition is however generated correctly. But py++ warns for
the class member functions referring to this enum.

I can send you the sample code if you would like.

Py++ code is probably 2 weeks old from sourceforge.

Thanks,
Shiva

[pygccxml-development] Another performance tweak

[Allen B. <al...@vr...>]

I just tracked down and fixed another major performance sink.

I saw in the profiler output that the majority of the time from my run 
was spent in __eq__ in declaration.py (line 121) and __eq_calldef.py 
line 121.  This is the code that compares two calldefs to see if they 
are equal.  (note this was also where most of the calls to 
algorithm.declration_path were coming from).

I was interested in tracking down where all these calls (over 6 million 
of them) were coming from so I added some code to the __eq__ method of 
calldef to keep track of all the ways it was called and store how many 
hits it gets (an example of the output is at the end of the e-mail). 

As a side note, I also counted the return value of true and false 
separately just for fun.  I found that out of the over 6million times it 
was called, it returned True only 33 times and those only came from a 
call path starting with _join_declarations.   Every other test was false 
every time, so there may be another optimization hiding in here to just 
not call this test.

As it ended up I found that the vast majority of these calls cam from 
the member_functions method in scopedef.py.  I traced through there and 
found that all the the __eq__ calls were coming from some nested calls 
to _find_out_member_access_type that were coming from 
access_type_matcher_t.  I never did find out where access_type_matcher_t 
was coming from since I was just asking for all the members.  

Anyway, the way pygccxml works the decls don't actually know their 
access type.  Only their parents do.  So if you want to know a decl 
access type you have to ask the parent and then it loops over all of 
it's internal members for each access type until it find the one you are 
asking about.  This meant that the member_functions method was at least 
O(N^2) and possibly O(N^3).

So back to what I did to fix it.  It seemed to me that for pygccxml the 
access type of a member should remain static through a single 
execution.  So I added a caching mechanism to the 
find_out_member_access_type that just stores the access type with the 
member decl.  Then the next time it is check we return it directly and 
skip looping over all the lists and calling __eq__ so many millions of 
times.

In the end the number of __eq__ calls from  6,010,000  to 271,500.  This 
took my build type from 344 seconds down to 116 seconds.

So when you combine this change with the one from yesterday the 
generation process is now 7 times faster.  Not bad for just modifying 
two methods. :)

-Allen

PS. You can see the PerformanceTuning page on the wiki for pointers to 
the tools I have been using.


---------- Example call chaining for __eq__:  Eq: Called 238772 times 
and *always* returned false  ------
  [0, 238772]:  [('gen_bindings.py', 722, '?'), ('gen_bindings.py', 673, 
'main'), 
('/home/allenb/python/lib/python/pyplusplus/module_builder/builder.py', 
236, 'build_code_creator'), 
('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py', 
541, 'create'), 
('/home/allenb/python/lib/python/pygccxml/declarations/algorithm.py', 
268, 'apply_visitor'), 
('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py', 
704, 'visit_class'), 
('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py', 
348, '_is_wrapper_needed'), 
('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py', 
287, 'redefined_funcs'), 
('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py', 
473, 'member_functions'), 
('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py', 
326, '_find_multiple'), 
('/home/allenb/python/lib/python/pygccxml/declarations/matcher.py', 49, 
'find'), 
('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py', 
258, '<lambda>'), 
('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py', 83, 
'__call__'), 
('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py', 61, 
'__call__'), 
('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py', 
478, '__call__'), 
('/home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py', 
321, 'find_out_member_access_type'), 
('/home/allenb/python/lib/python/pygccxml/declarations/calldef.py', 310, 
'__eq__'), 
('/home/allenb/python/lib/python/pygccxml/declarations/calldef.py', 139, 
'__eq__')]

Re: [pygccxml-development] Performance problems and fixes

[Allen B. <al...@vr...>]

Roman Yakovenko wrote:

> On 8/25/06, Allen Bierbaum <al...@vr...> wrote:
>
>> (all of these are in pygccxml)
>> - Getting names from declarations
>
>
> Please remove comments you added to the declaration_t class. We don't 
> really
> need them.

I was hoping to get some feedback on that commented out code in 
declaration.py.

See the diff:  
http://svn.sourceforge.net/viewvc/pygccxml/pygccxml_dev/pygccxml/declarations/declaration.py?r1=466&r2=465&pathrev=466

Can you explain what makes the commented out code fail, and is there 
anything that can be done to fix it up?

Also, why is there a _get_name_impl method at all.  Why isn't it just 
named _get_name?  It looked to me last night like a polymorphic 
_get_name method should work fine.

-Allen

Re: [pygccxml-development] Performance problems and fixes

[Allen B. <al...@vr...>]

Roman Yakovenko wrote:

> On 8/25/06, Allen Bierbaum <al...@vr...> wrote:
>
>> Today I ran my generation script through the python hotspot profiler and
>> found some interesting results.
>>
>> The majority of the time was being spent in algorithm.declaration_path.
>> I fixed this up and double the speed of my script. :)
>
>
> These are good news!
>
>> After that I decided it may be worth investigating some other spots.
>
>
> Before you do this, there is some work left on the previous patch :-(/
>
>> What I have found is that the following areas are consuming a lot of
>> time.  (not sure if it can be fixed or not, but this is where the time
>> is spent)
>>
>> (all of these are in pygccxml)
>> - Getting names from declarations
>> - Testing declarations and calldefs for equality
>> - Getting the decl_string (and creating it) for cpptypes.
>>
>> declarations.py  _get_name  line 151
>> class_declarations.py _get_name_impl    line 105
>> declaration.py  _eq_   line 119
>> calldef.py  _eq_  line 121 and 53
>> cpptypes.py  _get_decl_string   line 37
>> algorithm.py   full_name   line 30
>>
>> Once I get into the office tomorrow I will upload my hotspot trace so
>> other people can see what I ran into and hopefully find some other
>> places to improve the performance of py++. :)
>
>
> The issue that left.
> Consider next use case ( a real one ).
> I am not able to run a gccxml on boost.date_time library on windows. So
> I run the gccxml on Linux and than use created xml file on Windows. 
> The only
> issue I has is next: there is a template class that is instantiated
> with different values. So, on Windows I give class new name. Another
> issue is free template
> function on return type:
>
> template< class X>
> X do_smth(int, double);
>
> int do_smth( int );
>
> By default the generated code will not compile. User will have to rename
> instantiated function to "do_smth<xyz>".
>
> You can take a look on pyboost_dev/dev/date_time/generate_code.py script.
>
> The point is that if user rename a declaration, the cache value you 
> introduced
> should be reset to None. In case of namespace and class the internal 
> declaration
> cache values should be reset too. If you don't do this, Py++ will 
> generate bad
> code and it will take a lot of time to find out the error.
>
> The work, that still should be done:
> 1. To add new variable to declaration_t class.
> 2. To reset it and internal values on name change.
> 3. To write unit test ( pygccxml ).
>
> Please, do this before you introduce another changes. Thank you.

I sounds like you understand these issues much more then I do.  I don't 
even think I can replicate the problem you describe since I don't 
understand it.  If you can check in a testcase that causes the problem, 
I could look at it but otherwise I think I am going to need you to make 
these changes.

-Allen

>
>

Re: [pygccxml-development] Performance problems and fixes

[Roman Y. <rom...@gm...>]

On 8/25/06, Allen Bierbaum <al...@vr...> wrote:
> (all of these are in pygccxml)
> - Getting names from declarations

Please remove comments you added to the declaration_t class. We don't really
need them.

Thank you.
-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/