Re: [pygccxml-development] Another performance tweak

[Allen B. <al...@vr...>]

I just found some more tweaks.

- Caching the results of type_traits.remove_alias improved performance 
by 20%

- (danger) Having create_identifier just return the exact same string 
passed as full_name improved performance another 20%.

This last one was interesting because what I found was that (at least in 
my project) create_identifier was called about 16,000 times and it 
always returned the same value passed in.  This could be just because of 
how my project works (no namespace aliases that I know of) and is 
probably not portable. 

But, one very strange thing is why is this method being called for names 
like "boost::python::arg" and "boost::python::default_call_policies"?    
As far as I understand how py++ is working, these symbols are always 
going to be valid.  So why waste the time calling an expensive method if 
we already know the outcome?

-Allen


Allen Bierbaum wrote:

>I just tracked down and fixed another major performance sink.
>
>I saw in the profiler output that the majority of the time from my run 
>was spent in __eq__ in declaration.py (line 121) and __eq_calldef.py 
>line 121.  This is the code that compares two calldefs to see if they 
>are equal.  (note this was also where most of the calls to 
>algorithm.declration_path were coming from).
>
>I was interested in tracking down where all these calls (over 6 million 
>of them) were coming from so I added some code to the __eq__ method of 
>calldef to keep track of all the ways it was called and store how many 
>hits it gets (an example of the output is at the end of the e-mail). 
>
>As a side note, I also counted the return value of true and false 
>separately just for fun.  I found that out of the over 6million times it 
>was called, it returned True only 33 times and those only came from a 
>call path starting with _join_declarations.   Every other test was false 
>every time, so there may be another optimization hiding in here to just 
>not call this test.
>
>As it ended up I found that the vast majority of these calls cam from 
>the member_functions method in scopedef.py.  I traced through there and 
>found that all the the __eq__ calls were coming from some nested calls 
>to _find_out_member_access_type that were coming from 
>access_type_matcher_t.  I never did find out where access_type_matcher_t 
>was coming from since I was just asking for all the members.  
>
>Anyway, the way pygccxml works the decls don't actually know their 
>access type.  Only their parents do.  So if you want to know a decl 
>access type you have to ask the parent and then it loops over all of 
>it's internal members for each access type until it find the one you are 
>asking about.  This meant that the member_functions method was at least 
>O(N^2) and possibly O(N^3).
>
>So back to what I did to fix it.  It seemed to me that for pygccxml the 
>access type of a member should remain static through a single 
>execution.  So I added a caching mechanism to the 
>find_out_member_access_type that just stores the access type with the 
>member decl.  Then the next time it is check we return it directly and 
>skip looping over all the lists and calling __eq__ so many millions of 
>times.
>
>In the end the number of __eq__ calls from  6,010,000  to 271,500.  This 
>took my build type from 344 seconds down to 116 seconds.
>
>So when you combine this change with the one from yesterday the 
>generation process is now 7 times faster.  Not bad for just modifying 
>two methods. :)
>
>-Allen
>
>PS. You can see the PerformanceTuning page on the wiki for pointers to 
>the tools I have been using.
>
>
>---------- Example call chaining for __eq__:  Eq: Called 238772 times 
>and *always* returned false  ------
>  [0, 238772]:  [('gen_bindings.py', 722, '?'), ('gen_bindings.py', 673, 
>'main'), 
>('/home/allenb/python/lib/python/pyplusplus/module_builder/builder.py', 
>236, 'build_code_creator'), 
>('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py', 
>541, 'create'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/algorithm.py', 
>268, 'apply_visitor'), 
>('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py', 
>704, 'visit_class'), 
>('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py', 
>348, '_is_wrapper_needed'), 
>('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py', 
>287, 'redefined_funcs'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py', 
>473, 'member_functions'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py', 
>326, '_find_multiple'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/matcher.py', 49, 
>'find'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py', 
>258, '<lambda>'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py', 83, 
>'__call__'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py', 61, 
>'__call__'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py', 
>478, '__call__'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py', 
>321, 'find_out_member_access_type'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/calldef.py', 310, 
>'__eq__'), 
>('/home/allenb/python/lib/python/pygccxml/declarations/calldef.py', 139, 
>'__eq__')]
>
>
>-------------------------------------------------------------------------
>Using Tomcat but need to do more? Need to support web services, security?
>Get stuff done quickly with pre-integrated technology to make your job easier
>Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>_______________________________________________
>pygccxml-development mailing list
>pyg...@li...
>https://lists.sourceforge.net/lists/listinfo/pygccxml-development
>
>  
>