Re: [pygccxml-development] Another performance tweak

[Allen B. <al...@vr...>]

Roman Yakovenko wrote:

>On 8/26/06, Allen Bierbaum <al...@vr...> wrote:
>  
>
>>I just tracked down and fixed another major performance sink.
>>    
>>
>
>Allen, these are good news, but I explicitly asked you not to introduce new
>optimizations, untill we finish with previous one.
>  
>
Sorry about this one then.  I can back out the revision if you like.

See my previous e-mail, I don't know how to finish up the previous 
change because I don't understand the issue involved.  I made sure the 
code passed the test suites, but that is about all I can do.  I think 
you are going to have to look at it or come up with a test case that 
shows the problem to me.

I will just keep any further performance increases in my own tree for now.

If you are interested in looking into some of this I would point you at 
check_name.  This method is now one of the most time consuming.  I can't 
really understand what the implementation is doing but there may be room 
for improvement.

It is also worth noting that when I turn on optimize_queries, the code 
runs slower.

-Allen

>  
>
>>I saw in the profiler output that the majority of the time from my run
>>was spent in __eq__ in declaration.py (line 121) and __eq_calldef.py
>>line 121.  This is the code that compares two calldefs to see if they
>>are equal.  (note this was also where most of the calls to
>>algorithm.declration_path were coming from).
>>
>>I was interested in tracking down where all these calls (over 6 million
>>of them) were coming from so I added some code to the __eq__ method of
>>calldef to keep track of all the ways it was called and store how many
>>hits it gets (an example of the output is at the end of the e-mail).
>>
>>As a side note, I also counted the return value of true and false
>>separately just for fun.  I found that out of the over 6million times it
>>was called, it returned True only 33 times and those only came from a
>>call path starting with _join_declarations.   Every other test was false
>>every time, so there may be another optimization hiding in here to just
>>not call this test.
>>
>>As it ended up I found that the vast majority of these calls cam from
>>the member_functions method in scopedef.py.  I traced through there and
>>found that all the the __eq__ calls were coming from some nested calls
>>to _find_out_member_access_type that were coming from
>>access_type_matcher_t.  I never did find out where access_type_matcher_t
>>was coming from since I was just asking for all the members.
>>
>>Anyway, the way pygccxml works the decls don't actually know their
>>access type.  Only their parents do.  So if you want to know a decl
>>access type you have to ask the parent and then it loops over all of
>>it's internal members for each access type until it find the one you are
>>asking about.  This meant that the member_functions method was at least
>>O(N^2) and possibly O(N^3).
>>
>>So back to what I did to fix it.  It seemed to me that for pygccxml the
>>access type of a member should remain static through a single
>>execution.  So I added a caching mechanism to the
>>find_out_member_access_type that just stores the access type with the
>>member decl.  Then the next time it is check we return it directly and
>>skip looping over all the lists and calling __eq__ so many millions of
>>times.
>>
>>In the end the number of __eq__ calls from  6,010,000  to 271,500.  This
>>took my build type from 344 seconds down to 116 seconds.
>>
>>So when you combine this change with the one from yesterday the
>>generation process is now 7 times faster.  Not bad for just modifying
>>two methods. :)
>>
>>-Allen
>>
>>PS. You can see the PerformanceTuning page on the wiki for pointers to
>>the tools I have been using.
>>
>>
>>---------- Example call chaining for __eq__:  Eq: Called 238772 times
>>and *always* returned false  ------
>>  [0, 238772]:  [('gen_bindings.py', 722, '?'), ('gen_bindings.py', 673,
>>'main'),
>>('/home/allenb/python/lib/python/pyplusplus/module_builder/builder.py',
>>236, 'build_code_creator'),
>>('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py',
>>541, 'create'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/algorithm.py',
>>268, 'apply_visitor'),
>>('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py',
>>704, 'visit_class'),
>>('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py',
>>348, '_is_wrapper_needed'),
>>('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py',
>>287, 'redefined_funcs'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py',
>>473, 'member_functions'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py',
>>326, '_find_multiple'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/matcher.py', 49,
>>'find'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py',
>>258, '<lambda>'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py', 83,
>>'__call__'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py', 61,
>>'__call__'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py',
>>478, '__call__'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py',
>>321, 'find_out_member_access_type'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/calldef.py', 310,
>>'__eq__'),
>>('/home/allenb/python/lib/python/pygccxml/declarations/calldef.py', 139,
>>'__eq__')]
>>
>>
>>-------------------------------------------------------------------------
>>Using Tomcat but need to do more? Need to support web services, security?
>>Get stuff done quickly with pre-integrated technology to make your job easier
>>Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>>_______________________________________________
>>pygccxml-development mailing list
>>pyg...@li...
>>https://lists.sourceforge.net/lists/listinfo/pygccxml-development
>>
>>    
>>
>
>
>  
>