[pygccxml-development] Another performance tweak

[Allen B. <al...@vr...>]

I just tracked down and fixed another major performance sink.

I saw in the profiler output that the majority of the time from my run 
was spent in __eq__ in declaration.py (line 121) and __eq_calldef.py 
line 121.  This is the code that compares two calldefs to see if they 
are equal.  (note this was also where most of the calls to 
algorithm.declration_path were coming from).

I was interested in tracking down where all these calls (over 6 million 
of them) were coming from so I added some code to the __eq__ method of 
calldef to keep track of all the ways it was called and store how many 
hits it gets (an example of the output is at the end of the e-mail). 

As a side note, I also counted the return value of true and false 
separately just for fun.  I found that out of the over 6million times it 
was called, it returned True only 33 times and those only came from a 
call path starting with _join_declarations.   Every other test was false 
every time, so there may be another optimization hiding in here to just 
not call this test.

As it ended up I found that the vast majority of these calls cam from 
the member_functions method in scopedef.py.  I traced through there and 
found that all the the __eq__ calls were coming from some nested calls 
to _find_out_member_access_type that were coming from 
access_type_matcher_t.  I never did find out where access_type_matcher_t 
was coming from since I was just asking for all the members.  

Anyway, the way pygccxml works the decls don't actually know their 
access type.  Only their parents do.  So if you want to know a decl 
access type you have to ask the parent and then it loops over all of 
it's internal members for each access type until it find the one you are 
asking about.  This meant that the member_functions method was at least 
O(N^2) and possibly O(N^3).

So back to what I did to fix it.  It seemed to me that for pygccxml the 
access type of a member should remain static through a single 
execution.  So I added a caching mechanism to the 
find_out_member_access_type that just stores the access type with the 
member decl.  Then the next time it is check we return it directly and 
skip looping over all the lists and calling __eq__ so many millions of 
times.

In the end the number of __eq__ calls from  6,010,000  to 271,500.  This 
took my build type from 344 seconds down to 116 seconds.

So when you combine this change with the one from yesterday the 
generation process is now 7 times faster.  Not bad for just modifying 
two methods. :)

-Allen

PS. You can see the PerformanceTuning page on the wiki for pointers to 
the tools I have been using.


---------- Example call chaining for __eq__:  Eq: Called 238772 times 
and *always* returned false  ------
  [0, 238772]:  [('gen_bindings.py', 722, '?'), ('gen_bindings.py', 673, 
'main'), 
('/home/allenb/python/lib/python/pyplusplus/module_builder/builder.py', 
236, 'build_code_creator'), 
('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py', 
541, 'create'), 
('/home/allenb/python/lib/python/pygccxml/declarations/algorithm.py', 
268, 'apply_visitor'), 
('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py', 
704, 'visit_class'), 
('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py', 
348, '_is_wrapper_needed'), 
('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py', 
287, 'redefined_funcs'), 
('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py', 
473, 'member_functions'), 
('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py', 
326, '_find_multiple'), 
('/home/allenb/python/lib/python/pygccxml/declarations/matcher.py', 49, 
'find'), 
('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py', 
258, '<lambda>'), 
('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py', 83, 
'__call__'), 
('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py', 61, 
'__call__'), 
('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py', 
478, '__call__'), 
('/home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py', 
321, 'find_out_member_access_type'), 
('/home/allenb/python/lib/python/pygccxml/declarations/calldef.py', 310, 
'__eq__'), 
('/home/allenb/python/lib/python/pygccxml/declarations/calldef.py', 139, 
'__eq__')]