Re: [pygccxml-development] Another performance tweak

[Roman Y. <rom...@gm...>]

On 8/26/06, Allen Bierbaum <al...@vr...> wrote:
> I just tracked down and fixed another major performance sink.

Allen, these are good news, but I explicitly asked you not to introduce new
optimizations, untill we finish with previous one.

> I saw in the profiler output that the majority of the time from my run
> was spent in __eq__ in declaration.py (line 121) and __eq_calldef.py
> line 121.  This is the code that compares two calldefs to see if they
> are equal.  (note this was also where most of the calls to
> algorithm.declration_path were coming from).
>
> I was interested in tracking down where all these calls (over 6 million
> of them) were coming from so I added some code to the __eq__ method of
> calldef to keep track of all the ways it was called and store how many
> hits it gets (an example of the output is at the end of the e-mail).
>
> As a side note, I also counted the return value of true and false
> separately just for fun.  I found that out of the over 6million times it
> was called, it returned True only 33 times and those only came from a
> call path starting with _join_declarations.   Every other test was false
> every time, so there may be another optimization hiding in here to just
> not call this test.
>
> As it ended up I found that the vast majority of these calls cam from
> the member_functions method in scopedef.py.  I traced through there and
> found that all the the __eq__ calls were coming from some nested calls
> to _find_out_member_access_type that were coming from
> access_type_matcher_t.  I never did find out where access_type_matcher_t
> was coming from since I was just asking for all the members.
>
> Anyway, the way pygccxml works the decls don't actually know their
> access type.  Only their parents do.  So if you want to know a decl
> access type you have to ask the parent and then it loops over all of
> it's internal members for each access type until it find the one you are
> asking about.  This meant that the member_functions method was at least
> O(N^2) and possibly O(N^3).
>
> So back to what I did to fix it.  It seemed to me that for pygccxml the
> access type of a member should remain static through a single
> execution.  So I added a caching mechanism to the
> find_out_member_access_type that just stores the access type with the
> member decl.  Then the next time it is check we return it directly and
> skip looping over all the lists and calling __eq__ so many millions of
> times.
>
> In the end the number of __eq__ calls from  6,010,000  to 271,500.  This
> took my build type from 344 seconds down to 116 seconds.
>
> So when you combine this change with the one from yesterday the
> generation process is now 7 times faster.  Not bad for just modifying
> two methods. :)
>
> -Allen
>
> PS. You can see the PerformanceTuning page on the wiki for pointers to
> the tools I have been using.
>
>
> ---------- Example call chaining for __eq__:  Eq: Called 238772 times
> and *always* returned false  ------
>   [0, 238772]:  [('gen_bindings.py', 722, '?'), ('gen_bindings.py', 673,
> 'main'),
> ('/home/allenb/python/lib/python/pyplusplus/module_builder/builder.py',
> 236, 'build_code_creator'),
> ('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py',
> 541, 'create'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/algorithm.py',
> 268, 'apply_visitor'),
> ('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py',
> 704, 'visit_class'),
> ('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py',
> 348, '_is_wrapper_needed'),
> ('/home/allenb/python/lib/python/pyplusplus/module_creator/creator.py',
> 287, 'redefined_funcs'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py',
> 473, 'member_functions'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py',
> 326, '_find_multiple'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/matcher.py', 49,
> 'find'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/scopedef.py',
> 258, '<lambda>'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py', 83,
> '__call__'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py', 61,
> '__call__'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/matchers.py',
> 478, '__call__'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py',
> 321, 'find_out_member_access_type'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/calldef.py', 310,
> '__eq__'),
> ('/home/allenb/python/lib/python/pygccxml/declarations/calldef.py', 139,
> '__eq__')]
>
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> pygccxml-development mailing list
> pyg...@li...
> https://lists.sourceforge.net/lists/listinfo/pygccxml-development
>


-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/