pygccxml-development

[pygccxml-development] Any recent commits that may affect performance

[Allen B. <al...@vr...>]

Have there been any recent commits (past 1-2 months) that may have had a 
significant affect on performance in py++?

I just update to the latest svn version and my code generation process 
is back up around 15 minutes.  Is there anything I should look for as a 
cause?

-Allen

Re: [pygccxml-development] Any recent commits that may affect performance

[Roman Y. <rom...@gm...>]

On 2/15/07, Allen Bierbaum <al...@vr...> wrote:
> Have there been any recent commits (past 1-2 months) that may have had a
> significant affect on performance in py++?
>
> I just update to the latest svn version and my code generation process
> is back up around 15 minutes.  Is there anything I should look for as a
> cause?

Yes, dependency manager. It checks that you expose all declarations
that are used in your interface.


-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Any recent commits that may affect performance

[Allen B. <al...@vr...>]

Roman Yakovenko wrote:
> On 2/15/07, Allen Bierbaum <al...@vr... 
> <mailto:al...@vr...>> wrote:
> > Have there been any recent commits (past 1-2 months) that may have had a
> > significant affect on performance in py++?
> >
> > I just update to the latest svn version and my code generation process
> > is back up around 15 minutes.  Is there anything I should look for as a
> > cause?
>
> Yes, dependency manager. It checks that you expose all declarations
> that are used in your interface.
>
Is there any user API to disable the dependency manager (atleast during 
debugging and development)?

What is the basic structure of the dependency management search 
algorithm?  Is it something like:

- For every part of an interface exposed
    - For each item used in the interface
        - Look up the item and make sure that is exposed

If it is something like this, then that could definitely be biting me.  
I have a *huge* number of symbols and something O(N^2) like this could 
really consume a lot of time.

In the meantime, I am trying to collect some profiler information to 
find out exactly where the time is being spent.  Once I have those 
numbers collected I will pass them along in case there is anything that 
can be done to increase performance.

-Allen

Re: [pygccxml-development] Any recent commits that may affect performance

[Allen B. <al...@vr...>]

Roman Yakovenko wrote:
> On 2/16/07, Allen Bierbaum <al...@vr...> wrote:
>> Is there any user API to disable the dependency manager (atleast during
>> debugging and development)?
>
> No, it does not exists.
>
>> What is the basic structure of the dependency management search
>> algorithm?  Is it something like:
>>
>> - For every part of an interface exposed
>>     - For each item used in the interface
>>         - Look up the item and make sure that is exposed
>>
>> If it is something like this, then that could definitely be biting me.
>> I have a *huge* number of symbols and something O(N^2) like this could
>> really consume a lot of time.
>
> Yes.
>
>> In the meantime, I am trying to collect some profiler information to
>> find out exactly where the time is being spent.  Once I have those
>> numbers collected I will pass them along in case there is anything that
>> can be done to increase performance.
>
> Lets wait for results. I want to be sure before I introduce
> functionality, which
> disables dependency manager.
You are right, removing the dependency manager did not help performance.

I have attached the detailed performance profile results to this e-mail 
with the results sorted by total time in the given methods.

The summary is:

   ncalls        tottime      percall  cumtime  percall 
filename:lineno(function)
49849182    452.827    0.000    755.786    0.000 
.../pygccxml/declarations/algorithm.py:37(full_name)
 50225914   338.388    0.000  1228.453    0.000 
.../pygccxml/declarations/matchers.py:224(check_name)
 50490184   244.846    0.000  1538.742    0.000 
.../pygccxml/declarations/matchers.py:205(__call__)
151030576  198.249    0.000    198.249    0.000 
.../pygccxml/declarations/matchers.py:160(_get_name)
102343991  169.654    0.000    169.654    0.000 
.../pygccxml/declarations/declaration.py:241(cache)
 49819921   153.485    0.000  1682.796    0.000 
.../pygccxml/declarations/scopedef.py:273(<lambda>)
 99681702   136.791    0.000    136.791    0.000 
.../pygccxml/declarations/algorithms_cache.py:30(_get_full_name)
    50591      84.602      0.002  1783.335    0.035 
.../pygccxml/declarations/matcher.py:33(find)
        1          46.046    46.046      46.047   46.047 
.../pygccxml/parser/declarations_cache.py:166(flush)
        1          32.485    32.485      32.535   32.535 
.../pygccxml/parser/source_reader.py:151(create_xml_file)
   6483/1      26.067      0.004      49.694   49.694 
.../pygccxml/declarations/scopedef.py:144(init_optimizer)
  6219137    12.618      0.000      12.618    0.000 
.../pyplusplus/decl_wrappers/algorithm.py:88(<lambda>)
    12960        9.767      0.001      27.079    0.002 
.../pyplusplus/decl_wrappers/algorithm.py:80(create_identifier)
3353123        9.533     0.000       21.040    0.000 
.../pygccxml/declarations/declaration.py:153(_get_name)
 
I need to look into it closer, but it looks like there is something in 
the code that is checking and/or getting the name of decls a *huge* 
number of times as part of some algorithm.  This may be a good 
opportunity for caching if we can be sure the value is set.

I know from our previous discussions of caching that you had two big 
worries (that I share with you by the way). 
  1. Caching doesn't work well in some cases where user code may want to 
change the values being cached  (runtime modification)
  2. It makes the code ugly putting all these caches everywhere.

That got me thinking about the python memoization pattern that can be 
handled by decorators.  See: 
http://www.phyast.pitt.edu/~micheles/python/documentation.html#memoize and
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/325205

What I was thinking is that in the context of py++ it would be nice if 
the user could make whatever changes they want to the system and then 
once the state is "set", call a method that turns on caching from that 
point forward.

Something like:

------------------------------
mb = ModuleBuilder(....)

<do user changes>

pyplusplus.memoizer.enable()   # Everything we cache is set from here on
------------------------------

We could also introduce an enabled state that asserted that the cache 
value equals the value that would have been calculated.  This would 
give  a nice sanity check to use during debugging to make sure the cache 
is not breaking things.

Anyway..... with a memoization pattern then the py++ internal code could 
look something like this:

class declaration_t:
   ....
   @memoize
    def getName(...):
        return value as normal.

So you see I think we could address both issues aboved.  The code would 
allow for runtime modification by users until they say they are done.  
It would also allow for clean code because you could just put a 
decorator (@memoize) on methods that needed it and then they would 
automatically work.

What do you think?  Is this a way we could increase the performance but 
still keep the code clean and flexible?

-Allen

Re: [pygccxml-development] Any recent commits that may affect performance

[Roman Y. <rom...@gm...>]

I need to study the report. The funny thing is that we do have cache for
"full_name" algorithm.


May be solution could come from gccxml. It seems that gccxml reports
the class names right. I need to check this. Also I have to add some support to
gccxml version, so I can safety select needed functionality.

On 2/16/07, Allen Bierbaum <al...@vr...> wrote:
> Roman Yakovenko wrote:
> > On 2/16/07, Allen Bierbaum <al...@vr...> wrote:
> >> Is there any user API to disable the dependency manager (atleast during
> >> debugging and development)?
> >
> > No, it does not exists.
> >
> >> What is the basic structure of the dependency management search
> >> algorithm?  Is it something like:
> >>
> >> - For every part of an interface exposed
> >>     - For each item used in the interface
> >>         - Look up the item and make sure that is exposed
> >>
> >> If it is something like this, then that could definitely be biting me.
> >> I have a *huge* number of symbols and something O(N^2) like this could
> >> really consume a lot of time.
> >
> > Yes.
> >
> >> In the meantime, I am trying to collect some profiler information to
> >> find out exactly where the time is being spent.  Once I have those
> >> numbers collected I will pass them along in case there is anything that
> >> can be done to increase performance.
> >
> > Lets wait for results. I want to be sure before I introduce
> > functionality, which
> > disables dependency manager.
> You are right, removing the dependency manager did not help performance.
>
> I have attached the detailed performance profile results to this e-mail
> with the results sorted by total time in the given methods.
>
> The summary is:
>
>    ncalls        tottime      percall  cumtime  percall
> filename:lineno(function)
> 49849182    452.827    0.000    755.786    0.000
> .../pygccxml/declarations/algorithm.py:37(full_name)
>  50225914   338.388    0.000  1228.453    0.000
> .../pygccxml/declarations/matchers.py:224(check_name)
>  50490184   244.846    0.000  1538.742    0.000
> .../pygccxml/declarations/matchers.py:205(__call__)
> 151030576  198.249    0.000    198.249    0.000
> .../pygccxml/declarations/matchers.py:160(_get_name)
> 102343991  169.654    0.000    169.654    0.000
> .../pygccxml/declarations/declaration.py:241(cache)
>  49819921   153.485    0.000  1682.796    0.000
> .../pygccxml/declarations/scopedef.py:273(<lambda>)
>  99681702   136.791    0.000    136.791    0.000
> .../pygccxml/declarations/algorithms_cache.py:30(_get_full_name)
>     50591      84.602      0.002  1783.335    0.035
> .../pygccxml/declarations/matcher.py:33(find)
>         1          46.046    46.046      46.047   46.047
> .../pygccxml/parser/declarations_cache.py:166(flush)
>         1          32.485    32.485      32.535   32.535
> .../pygccxml/parser/source_reader.py:151(create_xml_file)
>    6483/1      26.067      0.004      49.694   49.694
> .../pygccxml/declarations/scopedef.py:144(init_optimizer)
>   6219137    12.618      0.000      12.618    0.000
> .../pyplusplus/decl_wrappers/algorithm.py:88(<lambda>)
>     12960        9.767      0.001      27.079    0.002
> .../pyplusplus/decl_wrappers/algorithm.py:80(create_identifier)
> 3353123        9.533     0.000       21.040    0.000
> .../pygccxml/declarations/declaration.py:153(_get_name)
>
> I need to look into it closer, but it looks like there is something in
> the code that is checking and/or getting the name of decls a *huge*
> number of times as part of some algorithm.  This may be a good
> opportunity for caching if we can be sure the value is set.
>
> I know from our previous discussions of caching that you had two big
> worries (that I share with you by the way).
>   1. Caching doesn't work well in some cases where user code may want to
> change the values being cached  (runtime modification)
>   2. It makes the code ugly putting all these caches everywhere.
>
> That got me thinking about the python memoization pattern that can be
> handled by decorators.  See:
> http://www.phyast.pitt.edu/~micheles/python/documentation.html#memoize and
> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/325205
>
> What I was thinking is that in the context of py++ it would be nice if
> the user could make whatever changes they want to the system and then
> once the state is "set", call a method that turns on caching from that
> point forward.
>
> Something like:
>
> ------------------------------
> mb = ModuleBuilder(....)
>
> <do user changes>
>
> pyplusplus.memoizer.enable()   # Everything we cache is set from here on
> ------------------------------
>
> We could also introduce an enabled state that asserted that the cache
> value equals the value that would have been calculated.  This would
> give  a nice sanity check to use during debugging to make sure the cache
> is not breaking things.
>
> Anyway..... with a memoization pattern then the py++ internal code could
> look something like this:
>
> class declaration_t:
>    ....
>    @memoize
>     def getName(...):
>         return value as normal.
>
> So you see I think we could address both issues aboved.  The code would
> allow for runtime modification by users until they say they are done.
> It would also allow for clean code because you could just put a
> decorator (@memoize) on methods that needed it and then they would
> automatically work.
>
> What do you think?  Is this a way we could increase the performance but
> still keep the code clean and flexible?
>
> -Allen
>
>
>
>          617979712 function calls (616759372 primitive calls) in 2162.516 CPU seconds
>
>    Ordered by: internal time
>    List reduced from 1554 to 100 due to restriction <100>
>
>    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
> 49849182/49848887  452.827    0.000  755.786    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:37(full_name)
>  50225914  338.388    0.000 1228.453    0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:224(check_name)
>  50490184  244.846    0.000 1538.742    0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:205(__call__)
> 151030576  198.249    0.000  198.249    0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:160(_get_name)
> 102343991  169.654    0.000  169.654    0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:241(cache)
>  49819921  153.485    0.000 1682.796    0.000 /home/allenb/python/lib/python/pygccxml/declarations/scopedef.py:273(<lambda>)
>  99681702  136.791    0.000  136.791    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:30(_get_full_name)
>     50591   84.602    0.002 1783.335    0.035 /home/allenb/python/lib/python/pygccxml/declarations/matcher.py:33(find)
>         1   46.046   46.046   46.047   46.047 /home/allenb/python/lib/python/pygccxml/parser/declarations_cache.py:166(flush)
>         1   32.485   32.485   32.535   32.535 /home/allenb/python/lib/python/pygccxml/parser/source_reader.py:151(create_xml_file)
>    6483/1   26.067    0.004   49.694   49.694 /home/allenb/python/lib/python/pygccxml/declarations/scopedef.py:144(init_optimizer)
>   6219137   12.618    0.000   12.618    0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/algorithm.py:88(<lambda>)
>     12960    9.767    0.001   27.079    0.002 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/algorithm.py:80(create_identifier)
> 3353123/3353028    9.533    0.000   21.040    0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:153(_get_name)
>   1784028    9.249    0.000   22.834    0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:42(remove_alias)
>    277051    7.772    0.000    7.772    0.000 /home/allenb/python/lib/python/pygccxml/declarations/scopedef.py:112(__decl_types)
> 755882/755552    6.722    0.000   11.733    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:10(declaration_path)
>    487343    6.428    0.000   10.145    0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:14(__init__)
>   2154824    5.373    0.000   10.755    0.000 /home/allenb/python/lib/python/pyplusplus/code_creators/algorithm.py:40(make_flatten_generator)
> 2356010/2337404    5.234    0.000    5.382    0.000 /home/allenb/python/lib/python/pyplusplus/code_creators/algorithm.py:42(proceed_single)
>     50296    5.103    0.000 1795.412    0.036 /home/allenb/python/lib/python/pygccxml/declarations/scopedef.py:331(_find_multiple)
>       637    4.441    0.007   43.550    0.068 /usr/lib64/python2.4/site-packages/_xmlplus/sax/expatreader.py:205(feed)
>       291    4.374    0.015   16.067    0.055 /home/allenb/python/lib/python/pyplusplus/code_creators/class_declaration.py:113(_exported_base_classes)
>   2979382    4.353    0.000    4.353    0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:150(_get_name_impl)
> 373741/373740    4.337    0.000    7.154    0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:117(_get_name_impl)
>    487343    3.717    0.000    3.717    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:95(__init__)
>   2711357    3.436    0.000    3.436    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:99(_get_remove_alias)
>    174708    3.218    0.000   33.638    0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:166(startElement)
>     46858    3.118    0.000    3.389    0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/algorithm.py:30(_get_left_siblings)
> 169606/102367    2.694    0.000    4.253    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:71(proceed_single)
>     96123    2.473    0.000    3.149    0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:207(__read_location)
>    286719    2.422    0.000   11.120    0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:45(__eq__)
> 879364/455629    2.339    0.000    8.790    0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:40(_get_decl_string)
>   1145583    2.229    0.000    2.229    0.000 /home/allenb/python/lib/python/pygccxml/parser/patcher.py:113(<lambda>)
>       356    2.213    0.006    4.759    0.013 /home/allenb/python/lib/python/pygccxml/parser/patcher.py:109(__find_enum)
>   1389745    2.164    0.000    2.164    0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:171(_get_parent)
>   1026249    2.124    0.000    2.124    0.000 /usr/lib64/python2.4/site-packages/_xmlplus/sax/xmlreader.py:324(get)
>    100654    2.004    0.000    2.004    0.000 /usr/lib64/python2.4/posixpath.py:373(normpath)
> 295853/86792    1.991    0.000    2.331    0.000 /home/allenb/python/lib/python/pygccxml/parser/project_reader.py:453(get_from_type)
>   1493074    1.872    0.000    1.872    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:60(_get_declaration_path)
>     96175    1.739    0.000    1.739    0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/decl_wrapper.py:25(__init__)
>    290492    1.715    0.000   15.668    0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:265(remove_cv)
>         1    1.671    1.671   72.154   72.154 /home/allenb/python/lib/python/pygccxml/parser/source_reader.py:310(__parse_gccxml_created_file)
>     34097    1.613    0.000    5.760    0.000 /home/allenb/python/lib/python/pyplusplus/module_creator/call_policies_resolver.py:53(__call__)
>     57738    1.597    0.000    4.142    0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:319(__read_calldef)
>         1    1.536    1.536    6.534    6.534 /home/allenb/python/lib/python/pyplusplus/module_builder/builder.py:138(__filter_by_location)
>    146338    1.524    0.000    2.233    0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:256(get_members)
>     96175    1.513    0.000    2.848    0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:71(__init__)
>    985059    1.491    0.000    1.491    0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:196(_get_location)
> 229891/116390    1.484    0.000    1.894    0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:66(decompose_type)
>     52710    1.458    0.000    2.649    0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:307(__read_argument)
>    114946    1.351    0.000    2.088    0.000 /home/allenb/python/lib/python/pygccxml/parser/linker.py:24(_set_inst)
>     96175    1.335    0.000    1.335    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:12(__init__)
>         1    1.330    1.330    1.346    1.346 /home/allenb/python/lib/python/pygccxml/parser/declarations_cache.py:141(__load)
>    947095    1.287    0.000    1.287    0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:326(_get_base)
>    122541    1.261    0.000    4.489    0.000 /home/allenb/python/lib/python/pygccxml/declarations/function_traits.py:13(is_same_function)
>    217304    1.223    0.000    9.077    0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:24(__eq__)
>    375145    1.209    0.000    7.670    0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:216(is_const)
>    429350    1.183    0.000    1.655    0.000 extensions/pyexpat.c:465(CharacterData)
> 163346/70667    1.174    0.000    1.485    0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:31(__remove_alias)
>     79084    1.153    0.000    3.959    0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:282(adopt_declaration)
>     57738    1.151    0.000    1.999    0.000 /home/allenb/python/lib/python/pygccxml/parser/linker.py:61(__link_calldef)
>       581    1.118    0.002    1.408    0.002 /home/allenb/python/lib/python/pyplusplus/file_writers/writer.py:54(write_file)
>    624197    1.102    0.000    1.102    0.000 /usr/lib64/python2.4/site-packages/_xmlplus/sax/xmlreader.py:315(__getitem__)
>    254332    1.069    0.000    1.069    0.000 /usr/lib64/python2.4/logging/__init__.py:1136(getEffectiveLevel)
>    322753    1.062    0.000    1.062    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:148(does_match_exist)
>    174708    1.055    0.000    1.055    0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:203(endElement)
>    174708    1.050    0.000   34.924    0.000 /usr/lib64/python2.4/site-packages/_xmlplus/sax/expatreader.py:310(start_element)
>    303849    1.031    0.000    6.214    0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:322(__init__)
>     57752    1.003    0.000    2.576    0.000 /home/allenb/python/lib/python/pygccxml/declarations/calldef.py:371(function_type)
>     50590    0.998    0.000    4.635    0.000 /home/allenb/python/lib/python/pygccxml/declarations/scopedef.py:262(__create_matcher)
>      6431    0.994    0.000    1.169    0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:103(__init__)
>     53310    0.954    0.000    5.511    0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:336(__read_member_function)
>    290492    0.953    0.000    2.812    0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:249(is_volatile)
>    322753    0.949    0.000    2.011    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:171(__call__)
>    699219    0.945    0.000    0.945    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:26(enabled)
> 189878/189770    0.924    0.000    3.226    0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:350(_create_decl_string)
>    724484    0.903    0.000    0.903    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:50(_get_demangled_name)
>     12960    0.891    0.000    4.366    0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/algorithm.py:37(_get_definition_set)
>    253539    0.889    0.000    1.954    0.000 /usr/lib64/python2.4/logging/__init__.py:943(debug)
>    241132    0.873    0.000    4.267    0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:233(remove_declarated)
>         1    0.864    0.864    3.924    3.924 /home/allenb/python/lib/python/pygccxml/parser/project_reader.py:424(_relink_declarated_types)
>     81593    0.857    0.000    1.468    0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:320(find_out_member_access_type)
>         1    0.852    0.852   12.578   12.578 /home/allenb/python/lib/python/pygccxml/parser/project_reader.py:359(_join_class_hierarchy)
>    269445    0.847    0.000    1.269    0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/decl_wrapper.py:113(exclude)
>    159205    0.835    0.000    0.835    0.000 /home/allenb/python/lib/python/pygccxml/declarations/pattern_parser.py:24(has_pattern)
>    627455    0.825    0.000    0.825    0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:60(_get_related_class)
>     10535    0.811    0.000    2.844    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:175(find_all_declarations)
>     57350    0.808    0.000    2.596    0.000 /home/allenb/python/lib/python/pygccxml/declarations/calldef.py:108(__init__)
>     96156    0.791    0.000    2.486    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:70(reset)
>     72203    0.753    0.000    4.109    0.000 /home/allenb/python/lib/python/pygccxml/declarations/container_traits.py:40(get_container_or_none)
>    435851    0.750    0.000    0.750    0.000 /home/allenb/python/lib/python/pygccxml/declarations/calldef.py:145(_get_arguments)
>     30997    0.729    0.000    3.546    0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/__init__.py:118(create_member_function)
>    116184    0.723    0.000    1.176    0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:233(__read_access)
>    388905    0.721    0.000    0.721    0.000 /home/allenb/python/lib/python/pyplusplus/code_creators/algorithm.py:101(<lambda>)
>    159844    0.715    0.000    2.141    0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:341(__call__)
>    457956    0.704    0.000    0.704    0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:42(_get_file_name)
>     96125    0.703    0.000    1.609    0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:297(__call__)
>     73767    0.694    0.000    2.680    0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:118(__init__)
>     96175    0.667    0.000    1.048    0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:210(__update_membership)
>
>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys-and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> pygccxml-development mailing list
> pyg...@li...
> https://lists.sourceforge.net/lists/listinfo/pygccxml-development
>
>


-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Any recent commits that may affect performance

[Allen B. <al...@vr...>]

Roman Yakovenko wrote:
> I need to study the report. The funny thing is that we do have cache for
> "full_name" algorithm.
>   
I agree.  When I saw that it surprised me.  Is there anything you can 
think of that may have changed in the past several months that would 
increase the number of times that the full_name code is being called?  
In other words, is there a new algorithm in the code that needs to be 
made smarter to increase performance?

-Allen

>
> May be solution could come from gccxml. It seems that gccxml reports
> the class names right. I need to check this. Also I have to add some support to
> gccxml version, so I can safety select needed functionality.
>
> On 2/16/07, Allen Bierbaum <al...@vr...> wrote:
>   
>> Roman Yakovenko wrote:
>>     
>>> On 2/16/07, Allen Bierbaum <al...@vr...> wrote:
>>>       
>>>> Is there any user API to disable the dependency manager (atleast during
>>>> debugging and development)?
>>>>         
>>> No, it does not exists.
>>>
>>>       
>>>> What is the basic structure of the dependency management search
>>>> algorithm?  Is it something like:
>>>>
>>>> - For every part of an interface exposed
>>>>     - For each item used in the interface
>>>>         - Look up the item and make sure that is exposed
>>>>
>>>> If it is something like this, then that could definitely be biting me.
>>>> I have a *huge* number of symbols and something O(N^2) like this could
>>>> really consume a lot of time.
>>>>         
>>> Yes.
>>>
>>>       
>>>> In the meantime, I am trying to collect some profiler information to
>>>> find out exactly where the time is being spent.  Once I have those
>>>> numbers collected I will pass them along in case there is anything that
>>>> can be done to increase performance.
>>>>         
>>> Lets wait for results. I want to be sure before I introduce
>>> functionality, which
>>> disables dependency manager.
>>>       
>> You are right, removing the dependency manager did not help performance.
>>
>> I have attached the detailed performance profile results to this e-mail
>> with the results sorted by total time in the given methods.
>>
>> The summary is:
>>
>>    ncalls        tottime      percall  cumtime  percall
>> filename:lineno(function)
>> 49849182    452.827    0.000    755.786    0.000
>> .../pygccxml/declarations/algorithm.py:37(full_name)
>>  50225914   338.388    0.000  1228.453    0.000
>> .../pygccxml/declarations/matchers.py:224(check_name)
>>  50490184   244.846    0.000  1538.742    0.000
>> .../pygccxml/declarations/matchers.py:205(__call__)
>> 151030576  198.249    0.000    198.249    0.000
>> .../pygccxml/declarations/matchers.py:160(_get_name)
>> 102343991  169.654    0.000    169.654    0.000
>> .../pygccxml/declarations/declaration.py:241(cache)
>>  49819921   153.485    0.000  1682.796    0.000
>> .../pygccxml/declarations/scopedef.py:273(<lambda>)
>>  99681702   136.791    0.000    136.791    0.000
>> .../pygccxml/declarations/algorithms_cache.py:30(_get_full_name)
>>     50591      84.602      0.002  1783.335    0.035
>> .../pygccxml/declarations/matcher.py:33(find)
>>         1          46.046    46.046      46.047   46.047
>> .../pygccxml/parser/declarations_cache.py:166(flush)
>>         1          32.485    32.485      32.535   32.535
>> .../pygccxml/parser/source_reader.py:151(create_xml_file)
>>    6483/1      26.067      0.004      49.694   49.694
>> .../pygccxml/declarations/scopedef.py:144(init_optimizer)
>>   6219137    12.618      0.000      12.618    0.000
>> .../pyplusplus/decl_wrappers/algorithm.py:88(<lambda>)
>>     12960        9.767      0.001      27.079    0.002
>> .../pyplusplus/decl_wrappers/algorithm.py:80(create_identifier)
>> 3353123        9.533     0.000       21.040    0.000
>> .../pygccxml/declarations/declaration.py:153(_get_name)
>>
>> I need to look into it closer, but it looks like there is something in
>> the code that is checking and/or getting the name of decls a *huge*
>> number of times as part of some algorithm.  This may be a good
>> opportunity for caching if we can be sure the value is set.
>>
>> I know from our previous discussions of caching that you had two big
>> worries (that I share with you by the way).
>>   1. Caching doesn't work well in some cases where user code may want to
>> change the values being cached  (runtime modification)
>>   2. It makes the code ugly putting all these caches everywhere.
>>
>> That got me thinking about the python memoization pattern that can be
>> handled by decorators.  See:
>> http://www.phyast.pitt.edu/~micheles/python/documentation.html#memoize and
>> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/325205
>>
>> What I was thinking is that in the context of py++ it would be nice if
>> the user could make whatever changes they want to the system and then
>> once the state is "set", call a method that turns on caching from that
>> point forward.
>>
>> Something like:
>>
>> ------------------------------
>> mb = ModuleBuilder(....)
>>
>> <do user changes>
>>
>> pyplusplus.memoizer.enable()   # Everything we cache is set from here on
>> ------------------------------
>>
>> We could also introduce an enabled state that asserted that the cache
>> value equals the value that would have been calculated.  This would
>> give  a nice sanity check to use during debugging to make sure the cache
>> is not breaking things.
>>
>> Anyway..... with a memoization pattern then the py++ internal code could
>> look something like this:
>>
>> class declaration_t:
>>    ....
>>    @memoize
>>     def getName(...):
>>         return value as normal.
>>
>> So you see I think we could address both issues aboved.  The code would
>> allow for runtime modification by users until they say they are done.
>> It would also allow for clean code because you could just put a
>> decorator (@memoize) on methods that needed it and then they would
>> automatically work.
>>
>> What do you think?  Is this a way we could increase the performance but
>> still keep the code clean and flexible?
>>
>> -Allen
>>
>>
>>
>>          617979712 function calls (616759372 primitive calls) in 2162.516 CPU seconds
>>
>>    Ordered by: internal time
>>    List reduced from 1554 to 100 due to restriction <100>
>>
>>    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>> 49849182/49848887  452.827    0.000  755.786    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:37(full_name)
>>  50225914  338.388    0.000 1228.453    0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:224(check_name)
>>  50490184  244.846    0.000 1538.742    0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:205(__call__)
>> 151030576  198.249    0.000  198.249    0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:160(_get_name)
>> 102343991  169.654    0.000  169.654    0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:241(cache)
>>  49819921  153.485    0.000 1682.796    0.000 /home/allenb/python/lib/python/pygccxml/declarations/scopedef.py:273(<lambda>)
>>  99681702  136.791    0.000  136.791    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:30(_get_full_name)
>>     50591   84.602    0.002 1783.335    0.035 /home/allenb/python/lib/python/pygccxml/declarations/matcher.py:33(find)
>>         1   46.046   46.046   46.047   46.047 /home/allenb/python/lib/python/pygccxml/parser/declarations_cache.py:166(flush)
>>         1   32.485   32.485   32.535   32.535 /home/allenb/python/lib/python/pygccxml/parser/source_reader.py:151(create_xml_file)
>>    6483/1   26.067    0.004   49.694   49.694 /home/allenb/python/lib/python/pygccxml/declarations/scopedef.py:144(init_optimizer)
>>   6219137   12.618    0.000   12.618    0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/algorithm.py:88(<lambda>)
>>     12960    9.767    0.001   27.079    0.002 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/algorithm.py:80(create_identifier)
>> 3353123/3353028    9.533    0.000   21.040    0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:153(_get_name)
>>   1784028    9.249    0.000   22.834    0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:42(remove_alias)
>>    277051    7.772    0.000    7.772    0.000 /home/allenb/python/lib/python/pygccxml/declarations/scopedef.py:112(__decl_types)
>> 755882/755552    6.722    0.000   11.733    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:10(declaration_path)
>>    487343    6.428    0.000   10.145    0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:14(__init__)
>>   2154824    5.373    0.000   10.755    0.000 /home/allenb/python/lib/python/pyplusplus/code_creators/algorithm.py:40(make_flatten_generator)
>> 2356010/2337404    5.234    0.000    5.382    0.000 /home/allenb/python/lib/python/pyplusplus/code_creators/algorithm.py:42(proceed_single)
>>     50296    5.103    0.000 1795.412    0.036 /home/allenb/python/lib/python/pygccxml/declarations/scopedef.py:331(_find_multiple)
>>       637    4.441    0.007   43.550    0.068 /usr/lib64/python2.4/site-packages/_xmlplus/sax/expatreader.py:205(feed)
>>       291    4.374    0.015   16.067    0.055 /home/allenb/python/lib/python/pyplusplus/code_creators/class_declaration.py:113(_exported_base_classes)
>>   2979382    4.353    0.000    4.353    0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:150(_get_name_impl)
>> 373741/373740    4.337    0.000    7.154    0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:117(_get_name_impl)
>>    487343    3.717    0.000    3.717    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:95(__init__)
>>   2711357    3.436    0.000    3.436    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:99(_get_remove_alias)
>>    174708    3.218    0.000   33.638    0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:166(startElement)
>>     46858    3.118    0.000    3.389    0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/algorithm.py:30(_get_left_siblings)
>> 169606/102367    2.694    0.000    4.253    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:71(proceed_single)
>>     96123    2.473    0.000    3.149    0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:207(__read_location)
>>    286719    2.422    0.000   11.120    0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:45(__eq__)
>> 879364/455629    2.339    0.000    8.790    0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:40(_get_decl_string)
>>   1145583    2.229    0.000    2.229    0.000 /home/allenb/python/lib/python/pygccxml/parser/patcher.py:113(<lambda>)
>>       356    2.213    0.006    4.759    0.013 /home/allenb/python/lib/python/pygccxml/parser/patcher.py:109(__find_enum)
>>   1389745    2.164    0.000    2.164    0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:171(_get_parent)
>>   1026249    2.124    0.000    2.124    0.000 /usr/lib64/python2.4/site-packages/_xmlplus/sax/xmlreader.py:324(get)
>>    100654    2.004    0.000    2.004    0.000 /usr/lib64/python2.4/posixpath.py:373(normpath)
>> 295853/86792    1.991    0.000    2.331    0.000 /home/allenb/python/lib/python/pygccxml/parser/project_reader.py:453(get_from_type)
>>   1493074    1.872    0.000    1.872    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:60(_get_declaration_path)
>>     96175    1.739    0.000    1.739    0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/decl_wrapper.py:25(__init__)
>>    290492    1.715    0.000   15.668    0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:265(remove_cv)
>>         1    1.671    1.671   72.154   72.154 /home/allenb/python/lib/python/pygccxml/parser/source_reader.py:310(__parse_gccxml_created_file)
>>     34097    1.613    0.000    5.760    0.000 /home/allenb/python/lib/python/pyplusplus/module_creator/call_policies_resolver.py:53(__call__)
>>     57738    1.597    0.000    4.142    0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:319(__read_calldef)
>>         1    1.536    1.536    6.534    6.534 /home/allenb/python/lib/python/pyplusplus/module_builder/builder.py:138(__filter_by_location)
>>    146338    1.524    0.000    2.233    0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:256(get_members)
>>     96175    1.513    0.000    2.848    0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:71(__init__)
>>    985059    1.491    0.000    1.491    0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:196(_get_location)
>> 229891/116390    1.484    0.000    1.894    0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:66(decompose_type)
>>     52710    1.458    0.000    2.649    0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:307(__read_argument)
>>    114946    1.351    0.000    2.088    0.000 /home/allenb/python/lib/python/pygccxml/parser/linker.py:24(_set_inst)
>>     96175    1.335    0.000    1.335    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:12(__init__)
>>         1    1.330    1.330    1.346    1.346 /home/allenb/python/lib/python/pygccxml/parser/declarations_cache.py:141(__load)
>>    947095    1.287    0.000    1.287    0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:326(_get_base)
>>    122541    1.261    0.000    4.489    0.000 /home/allenb/python/lib/python/pygccxml/declarations/function_traits.py:13(is_same_function)
>>    217304    1.223    0.000    9.077    0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:24(__eq__)
>>    375145    1.209    0.000    7.670    0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:216(is_const)
>>    429350    1.183    0.000    1.655    0.000 extensions/pyexpat.c:465(CharacterData)
>> 163346/70667    1.174    0.000    1.485    0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:31(__remove_alias)
>>     79084    1.153    0.000    3.959    0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:282(adopt_declaration)
>>     57738    1.151    0.000    1.999    0.000 /home/allenb/python/lib/python/pygccxml/parser/linker.py:61(__link_calldef)
>>       581    1.118    0.002    1.408    0.002 /home/allenb/python/lib/python/pyplusplus/file_writers/writer.py:54(write_file)
>>    624197    1.102    0.000    1.102    0.000 /usr/lib64/python2.4/site-packages/_xmlplus/sax/xmlreader.py:315(__getitem__)
>>    254332    1.069    0.000    1.069    0.000 /usr/lib64/python2.4/logging/__init__.py:1136(getEffectiveLevel)
>>    322753    1.062    0.000    1.062    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:148(does_match_exist)
>>    174708    1.055    0.000    1.055    0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:203(endElement)
>>    174708    1.050    0.000   34.924    0.000 /usr/lib64/python2.4/site-packages/_xmlplus/sax/expatreader.py:310(start_element)
>>    303849    1.031    0.000    6.214    0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:322(__init__)
>>     57752    1.003    0.000    2.576    0.000 /home/allenb/python/lib/python/pygccxml/declarations/calldef.py:371(function_type)
>>     50590    0.998    0.000    4.635    0.000 /home/allenb/python/lib/python/pygccxml/declarations/scopedef.py:262(__create_matcher)
>>      6431    0.994    0.000    1.169    0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:103(__init__)
>>     53310    0.954    0.000    5.511    0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:336(__read_member_function)
>>    290492    0.953    0.000    2.812    0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:249(is_volatile)
>>    322753    0.949    0.000    2.011    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:171(__call__)
>>    699219    0.945    0.000    0.945    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:26(enabled)
>> 189878/189770    0.924    0.000    3.226    0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:350(_create_decl_string)
>>    724484    0.903    0.000    0.903    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:50(_get_demangled_name)
>>     12960    0.891    0.000    4.366    0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/algorithm.py:37(_get_definition_set)
>>    253539    0.889    0.000    1.954    0.000 /usr/lib64/python2.4/logging/__init__.py:943(debug)
>>    241132    0.873    0.000    4.267    0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:233(remove_declarated)
>>         1    0.864    0.864    3.924    3.924 /home/allenb/python/lib/python/pygccxml/parser/project_reader.py:424(_relink_declarated_types)
>>     81593    0.857    0.000    1.468    0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:320(find_out_member_access_type)
>>         1    0.852    0.852   12.578   12.578 /home/allenb/python/lib/python/pygccxml/parser/project_reader.py:359(_join_class_hierarchy)
>>    269445    0.847    0.000    1.269    0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/decl_wrapper.py:113(exclude)
>>    159205    0.835    0.000    0.835    0.000 /home/allenb/python/lib/python/pygccxml/declarations/pattern_parser.py:24(has_pattern)
>>    627455    0.825    0.000    0.825    0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:60(_get_related_class)
>>     10535    0.811    0.000    2.844    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:175(find_all_declarations)
>>     57350    0.808    0.000    2.596    0.000 /home/allenb/python/lib/python/pygccxml/declarations/calldef.py:108(__init__)
>>     96156    0.791    0.000    2.486    0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:70(reset)
>>     72203    0.753    0.000    4.109    0.000 /home/allenb/python/lib/python/pygccxml/declarations/container_traits.py:40(get_container_or_none)
>>    435851    0.750    0.000    0.750    0.000 /home/allenb/python/lib/python/pygccxml/declarations/calldef.py:145(_get_arguments)
>>     30997    0.729    0.000    3.546    0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/__init__.py:118(create_member_function)
>>    116184    0.723    0.000    1.176    0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:233(__read_access)
>>    388905    0.721    0.000    0.721    0.000 /home/allenb/python/lib/python/pyplusplus/code_creators/algorithm.py:101(<lambda>)
>>    159844    0.715    0.000    2.141    0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:341(__call__)
>>    457956    0.704    0.000    0.704    0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:42(_get_file_name)
>>     96125    0.703    0.000    1.609    0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:297(__call__)
>>     73767    0.694    0.000    2.680    0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:118(__init__)
>>     96175    0.667    0.000    1.048    0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:210(__update_membership)
>>
>>
>>
>> -------------------------------------------------------------------------
>> Take Surveys. Earn Cash. Influence the Future of IT
>> Join SourceForge.net's Techsay panel and you'll get the chance to share your
>> opinions on IT & business topics through brief surveys-and earn cash
>> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
>> _______________________________________________
>> pygccxml-development mailing list
>> pyg...@li...
>> https://lists.sourceforge.net/lists/listinfo/pygccxml-development
>>
>>
>>     
>
>
>   

Re: [pygccxml-development] Any recent commits that may affect performance

[Roman Y. <rom...@gm...>]

On 2/16/07, Allen Bierbaum <al...@vr...> wrote:
> Roman Yakovenko wrote:
> > I need to study the report. The funny thing is that we do have cache for
> > "full_name" algorithm.
> >
> I agree.  When I saw that it surprised me.  Is there anything you can
> think of that may have changed in the past several months that would
> increase the number of times that the full_name code is being called?
> In other words, is there a new algorithm in the code that needs to be
> made smarter to increase performance?

Can you do me a favor:
pygccxml.declarations.declaration_t.
_get_name function implemented as


def _get_name( self ):

    return self._get_name_impl()


can you change the implementation to "return self._name" and to run this under
profiler? I expect this change to increase performance.

Thanks

P.S. Please don't over quote.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Any recent commits that may affect performance

[Allen B. <al...@vr...>]

>>> I agree.  When I saw that it surprised me.  Is there anything you can
>>> think of that may have changed in the past several months that would
>>> increase the number of times that the full_name code is being called?
>>> In other words, is there a new algorithm in the code that needs to be
>>> made smarter to increase performance?
>>>       
>
> Can you do me a favor:
> pygccxml.declarations.declaration_t.
> _get_name function implemented as
>
>
> def _get_name( self ):
>
>     return self._get_name_impl()
>
>
> can you change the implementation to "return self._name" and to run this under
> profiler? I expect this change to increase performance.
>   
That increased performance by about 5%.  So a small speedup, but not a 
significant speedup.  Any other ideas?

-Allen

Re: [pygccxml-development] Any recent commits that may affect performance

[Roman Y. <rom...@gm...>]

On 2/16/07, Allen Bierbaum <al...@vr...> wrote:
>
> >>> I agree.  When I saw that it surprised me.  Is there anything you can
> >>> think of that may have changed in the past several months that would
> >>> increase the number of times that the full_name code is being called?
> >>> In other words, is there a new algorithm in the code that needs to be
> >>> made smarter to increase performance?
> >>>
> >
> > Can you do me a favor:
> > pygccxml.declarations.declaration_t.
> > _get_name function implemented as
> >
> >
> > def _get_name( self ):
> >
> >     return self._get_name_impl()
> >
> >
> > can you change the implementation to "return self._name" and to run this under
> > profiler? I expect this change to increase performance.
> >
> That increased performance by about 5%.  So a small speedup, but not a
> significant speedup.  Any other ideas?

I thought about it and I don't know. May be you are doing something wrong.
Python-Ogre project is not a smallest one, but it takes only 5 minutes( without
cache and files ) to generate code. During code generation they also parse
Ogre source tree to extract documentation. Can you show your script?

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Any recent commits that may affect performance

[Allen B. <al...@vr...>]

> I thought about it and I don't know. May be you are doing something wrong.

That is always possible although nothing has really changed in the 
script in the time since it used to run fast.

> Python-Ogre project is not a smallest one, but it takes only 5 minutes( without
> cache and files ) to generate code. During code generation they also parse
> Ogre source tree to extract documentation. Can you show your script?

The generation script is here: 
https://realityforge.vrsource.org/trac/pyopensg/browser/trunk/src/gen_bindings.py

I have added some additional timing analysis that shows the vast 
majority (>90%)of the time being spent in two places.

1) line 587, the call to the module builder initialization where all the 
code is parsed
2) line 1040, the call to build_code_creator where all the code creators 
are built up and everything is setup to run the code generation.

Neither of these is a surprise since this is where py++ does all of it's 
work.

I am still collecting more information that should be able to help track 
down why the full_name and get_name methods are being called so many 
millions of times.  (calling full_name 50 million times for a single run 
just seems excessive.)  I am using stack monitoring technique I used 
when I helped out with the performance last time.  I have added code to 
a local version of py++ that is not only counting the number of times 
full_name is called but also keeps a dictionary that contains a list of 
every call path (stack) into the code and how many times full_name has 
been called along each path.  This should be able to show us if there 
are some code paths that end up making the majority of the calls and 
could be optimized.

Collecting this information can take a long time though.  I have already 
had the instrumented code running for about 20 hours and I expect it 
will run for another 24 hours before it is complete.  Once the run is 
done I will post the resulting information so everyone can take a look 
and present any ideas they have for helping out performance.

-Allen

Re: [pygccxml-development] Any recent commits that may affect performance

[Roman Y. <rom...@gm...>]

On 2/19/07, Allen Bierbaum <al...@vr...> wrote:
>
> > I thought about it and I don't know. May be you are doing something
> wrong.
>
> That is always possible although nothing has really changed in the
> script in the time since it used to run fast.
>
> > Python-Ogre project is not a smallest one, but it takes only 5 minutes(
> without
> > cache and files ) to generate code. During code generation they also
> parse
> > Ogre source tree to extract documentation. Can you show your script?
>
> The generation script is here:
>
> https://realityforge.vrsource.org/trac/pyopensg/browser/trunk/src/gen_bindings.py
>
> I have added some additional timing analysis that shows the vast
> majority (>90%)of the time being spent in two places.
>
> 1) line 587, the call to the module builder initialization where all the
> code is parsed
> 2) line 1040, the call to build_code_creator where all the code creators
> are built up and everything is setup to run the code generation.
>
> Neither of these is a surprise since this is where py++ does all of it's
> work.
>
> I am still collecting more information that should be able to help track
> down why the full_name and get_name methods are being called so many
> millions of times.  (calling full_name 50 million times for a single run
> just seems excessive.)  I am using stack monitoring technique I used
> when I helped out with the performance last time.  I have added code to
> a local version of py++ that is not only counting the number of times
> full_name is called but also keeps a dictionary that contains a list of
> every call path (stack) into the code and how many times full_name has
> been called along each path.  This should be able to show us if there
> are some code paths that end up making the majority of the calls and
> could be optimized.
>
> Collecting this information can take a long time though.  I have already
> had the instrumented code running for about 20 hours and I expect it
> will run for another 24 hours before it is complete.  Once the run is
> done I will post the resulting information so everyone can take a look
> and present any ideas they have for helping out performance.
>
> -Allen
>
>
Grate. We make Py++ faster. Thank you for all your work.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/