Re: [pygccxml-development] Any recent commits that may affect performance
Brought to you by:
mbaas,
roman_yakovenko
Menu
▾
▴
Re: [pygccxml-development] Any recent commits that may affect performance
|
From: Roman Y. <rom...@gm...> - 2007-02-16 20:10:00
|
I need to study the report. The funny thing is that we do have cache for "full_name" algorithm. May be solution could come from gccxml. It seems that gccxml reports the class names right. I need to check this. Also I have to add some support to gccxml version, so I can safety select needed functionality. On 2/16/07, Allen Bierbaum <al...@vr...> wrote: > Roman Yakovenko wrote: > > On 2/16/07, Allen Bierbaum <al...@vr...> wrote: > >> Is there any user API to disable the dependency manager (atleast during > >> debugging and development)? > > > > No, it does not exists. > > > >> What is the basic structure of the dependency management search > >> algorithm? Is it something like: > >> > >> - For every part of an interface exposed > >> - For each item used in the interface > >> - Look up the item and make sure that is exposed > >> > >> If it is something like this, then that could definitely be biting me. > >> I have a *huge* number of symbols and something O(N^2) like this could > >> really consume a lot of time. > > > > Yes. > > > >> In the meantime, I am trying to collect some profiler information to > >> find out exactly where the time is being spent. Once I have those > >> numbers collected I will pass them along in case there is anything that > >> can be done to increase performance. > > > > Lets wait for results. I want to be sure before I introduce > > functionality, which > > disables dependency manager. > You are right, removing the dependency manager did not help performance. > > I have attached the detailed performance profile results to this e-mail > with the results sorted by total time in the given methods. > > The summary is: > > ncalls tottime percall cumtime percall > filename:lineno(function) > 49849182 452.827 0.000 755.786 0.000 > .../pygccxml/declarations/algorithm.py:37(full_name) > 50225914 338.388 0.000 1228.453 0.000 > .../pygccxml/declarations/matchers.py:224(check_name) > 50490184 244.846 0.000 1538.742 0.000 > .../pygccxml/declarations/matchers.py:205(__call__) > 151030576 198.249 0.000 198.249 0.000 > .../pygccxml/declarations/matchers.py:160(_get_name) > 102343991 169.654 0.000 169.654 0.000 > .../pygccxml/declarations/declaration.py:241(cache) > 49819921 153.485 0.000 1682.796 0.000 > .../pygccxml/declarations/scopedef.py:273(<lambda>) > 99681702 136.791 0.000 136.791 0.000 > .../pygccxml/declarations/algorithms_cache.py:30(_get_full_name) > 50591 84.602 0.002 1783.335 0.035 > .../pygccxml/declarations/matcher.py:33(find) > 1 46.046 46.046 46.047 46.047 > .../pygccxml/parser/declarations_cache.py:166(flush) > 1 32.485 32.485 32.535 32.535 > .../pygccxml/parser/source_reader.py:151(create_xml_file) > 6483/1 26.067 0.004 49.694 49.694 > .../pygccxml/declarations/scopedef.py:144(init_optimizer) > 6219137 12.618 0.000 12.618 0.000 > .../pyplusplus/decl_wrappers/algorithm.py:88(<lambda>) > 12960 9.767 0.001 27.079 0.002 > .../pyplusplus/decl_wrappers/algorithm.py:80(create_identifier) > 3353123 9.533 0.000 21.040 0.000 > .../pygccxml/declarations/declaration.py:153(_get_name) > > I need to look into it closer, but it looks like there is something in > the code that is checking and/or getting the name of decls a *huge* > number of times as part of some algorithm. This may be a good > opportunity for caching if we can be sure the value is set. > > I know from our previous discussions of caching that you had two big > worries (that I share with you by the way). > 1. Caching doesn't work well in some cases where user code may want to > change the values being cached (runtime modification) > 2. It makes the code ugly putting all these caches everywhere. > > That got me thinking about the python memoization pattern that can be > handled by decorators. See: > http://www.phyast.pitt.edu/~micheles/python/documentation.html#memoize and > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/325205 > > What I was thinking is that in the context of py++ it would be nice if > the user could make whatever changes they want to the system and then > once the state is "set", call a method that turns on caching from that > point forward. > > Something like: > > ------------------------------ > mb = ModuleBuilder(....) > > <do user changes> > > pyplusplus.memoizer.enable() # Everything we cache is set from here on > ------------------------------ > > We could also introduce an enabled state that asserted that the cache > value equals the value that would have been calculated. This would > give a nice sanity check to use during debugging to make sure the cache > is not breaking things. > > Anyway..... with a memoization pattern then the py++ internal code could > look something like this: > > class declaration_t: > .... > @memoize > def getName(...): > return value as normal. > > So you see I think we could address both issues aboved. The code would > allow for runtime modification by users until they say they are done. > It would also allow for clean code because you could just put a > decorator (@memoize) on methods that needed it and then they would > automatically work. > > What do you think? Is this a way we could increase the performance but > still keep the code clean and flexible? > > -Allen > > > > 617979712 function calls (616759372 primitive calls) in 2162.516 CPU seconds > > Ordered by: internal time > List reduced from 1554 to 100 due to restriction <100> > > ncalls tottime percall cumtime percall filename:lineno(function) > 49849182/49848887 452.827 0.000 755.786 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:37(full_name) > 50225914 338.388 0.000 1228.453 0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:224(check_name) > 50490184 244.846 0.000 1538.742 0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:205(__call__) > 151030576 198.249 0.000 198.249 0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:160(_get_name) > 102343991 169.654 0.000 169.654 0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:241(cache) > 49819921 153.485 0.000 1682.796 0.000 /home/allenb/python/lib/python/pygccxml/declarations/scopedef.py:273(<lambda>) > 99681702 136.791 0.000 136.791 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:30(_get_full_name) > 50591 84.602 0.002 1783.335 0.035 /home/allenb/python/lib/python/pygccxml/declarations/matcher.py:33(find) > 1 46.046 46.046 46.047 46.047 /home/allenb/python/lib/python/pygccxml/parser/declarations_cache.py:166(flush) > 1 32.485 32.485 32.535 32.535 /home/allenb/python/lib/python/pygccxml/parser/source_reader.py:151(create_xml_file) > 6483/1 26.067 0.004 49.694 49.694 /home/allenb/python/lib/python/pygccxml/declarations/scopedef.py:144(init_optimizer) > 6219137 12.618 0.000 12.618 0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/algorithm.py:88(<lambda>) > 12960 9.767 0.001 27.079 0.002 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/algorithm.py:80(create_identifier) > 3353123/3353028 9.533 0.000 21.040 0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:153(_get_name) > 1784028 9.249 0.000 22.834 0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:42(remove_alias) > 277051 7.772 0.000 7.772 0.000 /home/allenb/python/lib/python/pygccxml/declarations/scopedef.py:112(__decl_types) > 755882/755552 6.722 0.000 11.733 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:10(declaration_path) > 487343 6.428 0.000 10.145 0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:14(__init__) > 2154824 5.373 0.000 10.755 0.000 /home/allenb/python/lib/python/pyplusplus/code_creators/algorithm.py:40(make_flatten_generator) > 2356010/2337404 5.234 0.000 5.382 0.000 /home/allenb/python/lib/python/pyplusplus/code_creators/algorithm.py:42(proceed_single) > 50296 5.103 0.000 1795.412 0.036 /home/allenb/python/lib/python/pygccxml/declarations/scopedef.py:331(_find_multiple) > 637 4.441 0.007 43.550 0.068 /usr/lib64/python2.4/site-packages/_xmlplus/sax/expatreader.py:205(feed) > 291 4.374 0.015 16.067 0.055 /home/allenb/python/lib/python/pyplusplus/code_creators/class_declaration.py:113(_exported_base_classes) > 2979382 4.353 0.000 4.353 0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:150(_get_name_impl) > 373741/373740 4.337 0.000 7.154 0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:117(_get_name_impl) > 487343 3.717 0.000 3.717 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:95(__init__) > 2711357 3.436 0.000 3.436 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:99(_get_remove_alias) > 174708 3.218 0.000 33.638 0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:166(startElement) > 46858 3.118 0.000 3.389 0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/algorithm.py:30(_get_left_siblings) > 169606/102367 2.694 0.000 4.253 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:71(proceed_single) > 96123 2.473 0.000 3.149 0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:207(__read_location) > 286719 2.422 0.000 11.120 0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:45(__eq__) > 879364/455629 2.339 0.000 8.790 0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:40(_get_decl_string) > 1145583 2.229 0.000 2.229 0.000 /home/allenb/python/lib/python/pygccxml/parser/patcher.py:113(<lambda>) > 356 2.213 0.006 4.759 0.013 /home/allenb/python/lib/python/pygccxml/parser/patcher.py:109(__find_enum) > 1389745 2.164 0.000 2.164 0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:171(_get_parent) > 1026249 2.124 0.000 2.124 0.000 /usr/lib64/python2.4/site-packages/_xmlplus/sax/xmlreader.py:324(get) > 100654 2.004 0.000 2.004 0.000 /usr/lib64/python2.4/posixpath.py:373(normpath) > 295853/86792 1.991 0.000 2.331 0.000 /home/allenb/python/lib/python/pygccxml/parser/project_reader.py:453(get_from_type) > 1493074 1.872 0.000 1.872 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:60(_get_declaration_path) > 96175 1.739 0.000 1.739 0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/decl_wrapper.py:25(__init__) > 290492 1.715 0.000 15.668 0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:265(remove_cv) > 1 1.671 1.671 72.154 72.154 /home/allenb/python/lib/python/pygccxml/parser/source_reader.py:310(__parse_gccxml_created_file) > 34097 1.613 0.000 5.760 0.000 /home/allenb/python/lib/python/pyplusplus/module_creator/call_policies_resolver.py:53(__call__) > 57738 1.597 0.000 4.142 0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:319(__read_calldef) > 1 1.536 1.536 6.534 6.534 /home/allenb/python/lib/python/pyplusplus/module_builder/builder.py:138(__filter_by_location) > 146338 1.524 0.000 2.233 0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:256(get_members) > 96175 1.513 0.000 2.848 0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:71(__init__) > 985059 1.491 0.000 1.491 0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:196(_get_location) > 229891/116390 1.484 0.000 1.894 0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:66(decompose_type) > 52710 1.458 0.000 2.649 0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:307(__read_argument) > 114946 1.351 0.000 2.088 0.000 /home/allenb/python/lib/python/pygccxml/parser/linker.py:24(_set_inst) > 96175 1.335 0.000 1.335 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:12(__init__) > 1 1.330 1.330 1.346 1.346 /home/allenb/python/lib/python/pygccxml/parser/declarations_cache.py:141(__load) > 947095 1.287 0.000 1.287 0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:326(_get_base) > 122541 1.261 0.000 4.489 0.000 /home/allenb/python/lib/python/pygccxml/declarations/function_traits.py:13(is_same_function) > 217304 1.223 0.000 9.077 0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:24(__eq__) > 375145 1.209 0.000 7.670 0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:216(is_const) > 429350 1.183 0.000 1.655 0.000 extensions/pyexpat.c:465(CharacterData) > 163346/70667 1.174 0.000 1.485 0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:31(__remove_alias) > 79084 1.153 0.000 3.959 0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:282(adopt_declaration) > 57738 1.151 0.000 1.999 0.000 /home/allenb/python/lib/python/pygccxml/parser/linker.py:61(__link_calldef) > 581 1.118 0.002 1.408 0.002 /home/allenb/python/lib/python/pyplusplus/file_writers/writer.py:54(write_file) > 624197 1.102 0.000 1.102 0.000 /usr/lib64/python2.4/site-packages/_xmlplus/sax/xmlreader.py:315(__getitem__) > 254332 1.069 0.000 1.069 0.000 /usr/lib64/python2.4/logging/__init__.py:1136(getEffectiveLevel) > 322753 1.062 0.000 1.062 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:148(does_match_exist) > 174708 1.055 0.000 1.055 0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:203(endElement) > 174708 1.050 0.000 34.924 0.000 /usr/lib64/python2.4/site-packages/_xmlplus/sax/expatreader.py:310(start_element) > 303849 1.031 0.000 6.214 0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:322(__init__) > 57752 1.003 0.000 2.576 0.000 /home/allenb/python/lib/python/pygccxml/declarations/calldef.py:371(function_type) > 50590 0.998 0.000 4.635 0.000 /home/allenb/python/lib/python/pygccxml/declarations/scopedef.py:262(__create_matcher) > 6431 0.994 0.000 1.169 0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:103(__init__) > 53310 0.954 0.000 5.511 0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:336(__read_member_function) > 290492 0.953 0.000 2.812 0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:249(is_volatile) > 322753 0.949 0.000 2.011 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:171(__call__) > 699219 0.945 0.000 0.945 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:26(enabled) > 189878/189770 0.924 0.000 3.226 0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:350(_create_decl_string) > 724484 0.903 0.000 0.903 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:50(_get_demangled_name) > 12960 0.891 0.000 4.366 0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/algorithm.py:37(_get_definition_set) > 253539 0.889 0.000 1.954 0.000 /usr/lib64/python2.4/logging/__init__.py:943(debug) > 241132 0.873 0.000 4.267 0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:233(remove_declarated) > 1 0.864 0.864 3.924 3.924 /home/allenb/python/lib/python/pygccxml/parser/project_reader.py:424(_relink_declarated_types) > 81593 0.857 0.000 1.468 0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:320(find_out_member_access_type) > 1 0.852 0.852 12.578 12.578 /home/allenb/python/lib/python/pygccxml/parser/project_reader.py:359(_join_class_hierarchy) > 269445 0.847 0.000 1.269 0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/decl_wrapper.py:113(exclude) > 159205 0.835 0.000 0.835 0.000 /home/allenb/python/lib/python/pygccxml/declarations/pattern_parser.py:24(has_pattern) > 627455 0.825 0.000 0.825 0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:60(_get_related_class) > 10535 0.811 0.000 2.844 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:175(find_all_declarations) > 57350 0.808 0.000 2.596 0.000 /home/allenb/python/lib/python/pygccxml/declarations/calldef.py:108(__init__) > 96156 0.791 0.000 2.486 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:70(reset) > 72203 0.753 0.000 4.109 0.000 /home/allenb/python/lib/python/pygccxml/declarations/container_traits.py:40(get_container_or_none) > 435851 0.750 0.000 0.750 0.000 /home/allenb/python/lib/python/pygccxml/declarations/calldef.py:145(_get_arguments) > 30997 0.729 0.000 3.546 0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/__init__.py:118(create_member_function) > 116184 0.723 0.000 1.176 0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:233(__read_access) > 388905 0.721 0.000 0.721 0.000 /home/allenb/python/lib/python/pyplusplus/code_creators/algorithm.py:101(<lambda>) > 159844 0.715 0.000 2.141 0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:341(__call__) > 457956 0.704 0.000 0.704 0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:42(_get_file_name) > 96125 0.703 0.000 1.609 0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:297(__call__) > 73767 0.694 0.000 2.680 0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:118(__init__) > 96175 0.667 0.000 1.048 0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:210(__update_membership) > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys-and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > pygccxml-development mailing list > pyg...@li... > https://lists.sourceforge.net/lists/listinfo/pygccxml-development > > -- Roman Yakovenko C++ Python language binding http://www.language-binding.net/ |