Re: [pygccxml-development] Any recent commits that may affect performance
Brought to you by:
mbaas,
roman_yakovenko
Menu
▾
▴
Re: [pygccxml-development] Any recent commits that may affect performance
|
From: Allen B. <al...@vr...> - 2007-02-16 20:40:29
|
Roman Yakovenko wrote: > I need to study the report. The funny thing is that we do have cache for > "full_name" algorithm. > I agree. When I saw that it surprised me. Is there anything you can think of that may have changed in the past several months that would increase the number of times that the full_name code is being called? In other words, is there a new algorithm in the code that needs to be made smarter to increase performance? -Allen > > May be solution could come from gccxml. It seems that gccxml reports > the class names right. I need to check this. Also I have to add some support to > gccxml version, so I can safety select needed functionality. > > On 2/16/07, Allen Bierbaum <al...@vr...> wrote: > >> Roman Yakovenko wrote: >> >>> On 2/16/07, Allen Bierbaum <al...@vr...> wrote: >>> >>>> Is there any user API to disable the dependency manager (atleast during >>>> debugging and development)? >>>> >>> No, it does not exists. >>> >>> >>>> What is the basic structure of the dependency management search >>>> algorithm? Is it something like: >>>> >>>> - For every part of an interface exposed >>>> - For each item used in the interface >>>> - Look up the item and make sure that is exposed >>>> >>>> If it is something like this, then that could definitely be biting me. >>>> I have a *huge* number of symbols and something O(N^2) like this could >>>> really consume a lot of time. >>>> >>> Yes. >>> >>> >>>> In the meantime, I am trying to collect some profiler information to >>>> find out exactly where the time is being spent. Once I have those >>>> numbers collected I will pass them along in case there is anything that >>>> can be done to increase performance. >>>> >>> Lets wait for results. I want to be sure before I introduce >>> functionality, which >>> disables dependency manager. >>> >> You are right, removing the dependency manager did not help performance. >> >> I have attached the detailed performance profile results to this e-mail >> with the results sorted by total time in the given methods. >> >> The summary is: >> >> ncalls tottime percall cumtime percall >> filename:lineno(function) >> 49849182 452.827 0.000 755.786 0.000 >> .../pygccxml/declarations/algorithm.py:37(full_name) >> 50225914 338.388 0.000 1228.453 0.000 >> .../pygccxml/declarations/matchers.py:224(check_name) >> 50490184 244.846 0.000 1538.742 0.000 >> .../pygccxml/declarations/matchers.py:205(__call__) >> 151030576 198.249 0.000 198.249 0.000 >> .../pygccxml/declarations/matchers.py:160(_get_name) >> 102343991 169.654 0.000 169.654 0.000 >> .../pygccxml/declarations/declaration.py:241(cache) >> 49819921 153.485 0.000 1682.796 0.000 >> .../pygccxml/declarations/scopedef.py:273(<lambda>) >> 99681702 136.791 0.000 136.791 0.000 >> .../pygccxml/declarations/algorithms_cache.py:30(_get_full_name) >> 50591 84.602 0.002 1783.335 0.035 >> .../pygccxml/declarations/matcher.py:33(find) >> 1 46.046 46.046 46.047 46.047 >> .../pygccxml/parser/declarations_cache.py:166(flush) >> 1 32.485 32.485 32.535 32.535 >> .../pygccxml/parser/source_reader.py:151(create_xml_file) >> 6483/1 26.067 0.004 49.694 49.694 >> .../pygccxml/declarations/scopedef.py:144(init_optimizer) >> 6219137 12.618 0.000 12.618 0.000 >> .../pyplusplus/decl_wrappers/algorithm.py:88(<lambda>) >> 12960 9.767 0.001 27.079 0.002 >> .../pyplusplus/decl_wrappers/algorithm.py:80(create_identifier) >> 3353123 9.533 0.000 21.040 0.000 >> .../pygccxml/declarations/declaration.py:153(_get_name) >> >> I need to look into it closer, but it looks like there is something in >> the code that is checking and/or getting the name of decls a *huge* >> number of times as part of some algorithm. This may be a good >> opportunity for caching if we can be sure the value is set. >> >> I know from our previous discussions of caching that you had two big >> worries (that I share with you by the way). >> 1. Caching doesn't work well in some cases where user code may want to >> change the values being cached (runtime modification) >> 2. It makes the code ugly putting all these caches everywhere. >> >> That got me thinking about the python memoization pattern that can be >> handled by decorators. See: >> http://www.phyast.pitt.edu/~micheles/python/documentation.html#memoize and >> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/325205 >> >> What I was thinking is that in the context of py++ it would be nice if >> the user could make whatever changes they want to the system and then >> once the state is "set", call a method that turns on caching from that >> point forward. >> >> Something like: >> >> ------------------------------ >> mb = ModuleBuilder(....) >> >> <do user changes> >> >> pyplusplus.memoizer.enable() # Everything we cache is set from here on >> ------------------------------ >> >> We could also introduce an enabled state that asserted that the cache >> value equals the value that would have been calculated. This would >> give a nice sanity check to use during debugging to make sure the cache >> is not breaking things. >> >> Anyway..... with a memoization pattern then the py++ internal code could >> look something like this: >> >> class declaration_t: >> .... >> @memoize >> def getName(...): >> return value as normal. >> >> So you see I think we could address both issues aboved. The code would >> allow for runtime modification by users until they say they are done. >> It would also allow for clean code because you could just put a >> decorator (@memoize) on methods that needed it and then they would >> automatically work. >> >> What do you think? Is this a way we could increase the performance but >> still keep the code clean and flexible? >> >> -Allen >> >> >> >> 617979712 function calls (616759372 primitive calls) in 2162.516 CPU seconds >> >> Ordered by: internal time >> List reduced from 1554 to 100 due to restriction <100> >> >> ncalls tottime percall cumtime percall filename:lineno(function) >> 49849182/49848887 452.827 0.000 755.786 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:37(full_name) >> 50225914 338.388 0.000 1228.453 0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:224(check_name) >> 50490184 244.846 0.000 1538.742 0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:205(__call__) >> 151030576 198.249 0.000 198.249 0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:160(_get_name) >> 102343991 169.654 0.000 169.654 0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:241(cache) >> 49819921 153.485 0.000 1682.796 0.000 /home/allenb/python/lib/python/pygccxml/declarations/scopedef.py:273(<lambda>) >> 99681702 136.791 0.000 136.791 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:30(_get_full_name) >> 50591 84.602 0.002 1783.335 0.035 /home/allenb/python/lib/python/pygccxml/declarations/matcher.py:33(find) >> 1 46.046 46.046 46.047 46.047 /home/allenb/python/lib/python/pygccxml/parser/declarations_cache.py:166(flush) >> 1 32.485 32.485 32.535 32.535 /home/allenb/python/lib/python/pygccxml/parser/source_reader.py:151(create_xml_file) >> 6483/1 26.067 0.004 49.694 49.694 /home/allenb/python/lib/python/pygccxml/declarations/scopedef.py:144(init_optimizer) >> 6219137 12.618 0.000 12.618 0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/algorithm.py:88(<lambda>) >> 12960 9.767 0.001 27.079 0.002 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/algorithm.py:80(create_identifier) >> 3353123/3353028 9.533 0.000 21.040 0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:153(_get_name) >> 1784028 9.249 0.000 22.834 0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:42(remove_alias) >> 277051 7.772 0.000 7.772 0.000 /home/allenb/python/lib/python/pygccxml/declarations/scopedef.py:112(__decl_types) >> 755882/755552 6.722 0.000 11.733 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:10(declaration_path) >> 487343 6.428 0.000 10.145 0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:14(__init__) >> 2154824 5.373 0.000 10.755 0.000 /home/allenb/python/lib/python/pyplusplus/code_creators/algorithm.py:40(make_flatten_generator) >> 2356010/2337404 5.234 0.000 5.382 0.000 /home/allenb/python/lib/python/pyplusplus/code_creators/algorithm.py:42(proceed_single) >> 50296 5.103 0.000 1795.412 0.036 /home/allenb/python/lib/python/pygccxml/declarations/scopedef.py:331(_find_multiple) >> 637 4.441 0.007 43.550 0.068 /usr/lib64/python2.4/site-packages/_xmlplus/sax/expatreader.py:205(feed) >> 291 4.374 0.015 16.067 0.055 /home/allenb/python/lib/python/pyplusplus/code_creators/class_declaration.py:113(_exported_base_classes) >> 2979382 4.353 0.000 4.353 0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:150(_get_name_impl) >> 373741/373740 4.337 0.000 7.154 0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:117(_get_name_impl) >> 487343 3.717 0.000 3.717 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:95(__init__) >> 2711357 3.436 0.000 3.436 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:99(_get_remove_alias) >> 174708 3.218 0.000 33.638 0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:166(startElement) >> 46858 3.118 0.000 3.389 0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/algorithm.py:30(_get_left_siblings) >> 169606/102367 2.694 0.000 4.253 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:71(proceed_single) >> 96123 2.473 0.000 3.149 0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:207(__read_location) >> 286719 2.422 0.000 11.120 0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:45(__eq__) >> 879364/455629 2.339 0.000 8.790 0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:40(_get_decl_string) >> 1145583 2.229 0.000 2.229 0.000 /home/allenb/python/lib/python/pygccxml/parser/patcher.py:113(<lambda>) >> 356 2.213 0.006 4.759 0.013 /home/allenb/python/lib/python/pygccxml/parser/patcher.py:109(__find_enum) >> 1389745 2.164 0.000 2.164 0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:171(_get_parent) >> 1026249 2.124 0.000 2.124 0.000 /usr/lib64/python2.4/site-packages/_xmlplus/sax/xmlreader.py:324(get) >> 100654 2.004 0.000 2.004 0.000 /usr/lib64/python2.4/posixpath.py:373(normpath) >> 295853/86792 1.991 0.000 2.331 0.000 /home/allenb/python/lib/python/pygccxml/parser/project_reader.py:453(get_from_type) >> 1493074 1.872 0.000 1.872 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:60(_get_declaration_path) >> 96175 1.739 0.000 1.739 0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/decl_wrapper.py:25(__init__) >> 290492 1.715 0.000 15.668 0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:265(remove_cv) >> 1 1.671 1.671 72.154 72.154 /home/allenb/python/lib/python/pygccxml/parser/source_reader.py:310(__parse_gccxml_created_file) >> 34097 1.613 0.000 5.760 0.000 /home/allenb/python/lib/python/pyplusplus/module_creator/call_policies_resolver.py:53(__call__) >> 57738 1.597 0.000 4.142 0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:319(__read_calldef) >> 1 1.536 1.536 6.534 6.534 /home/allenb/python/lib/python/pyplusplus/module_builder/builder.py:138(__filter_by_location) >> 146338 1.524 0.000 2.233 0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:256(get_members) >> 96175 1.513 0.000 2.848 0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:71(__init__) >> 985059 1.491 0.000 1.491 0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:196(_get_location) >> 229891/116390 1.484 0.000 1.894 0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:66(decompose_type) >> 52710 1.458 0.000 2.649 0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:307(__read_argument) >> 114946 1.351 0.000 2.088 0.000 /home/allenb/python/lib/python/pygccxml/parser/linker.py:24(_set_inst) >> 96175 1.335 0.000 1.335 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:12(__init__) >> 1 1.330 1.330 1.346 1.346 /home/allenb/python/lib/python/pygccxml/parser/declarations_cache.py:141(__load) >> 947095 1.287 0.000 1.287 0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:326(_get_base) >> 122541 1.261 0.000 4.489 0.000 /home/allenb/python/lib/python/pygccxml/declarations/function_traits.py:13(is_same_function) >> 217304 1.223 0.000 9.077 0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:24(__eq__) >> 375145 1.209 0.000 7.670 0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:216(is_const) >> 429350 1.183 0.000 1.655 0.000 extensions/pyexpat.c:465(CharacterData) >> 163346/70667 1.174 0.000 1.485 0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:31(__remove_alias) >> 79084 1.153 0.000 3.959 0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:282(adopt_declaration) >> 57738 1.151 0.000 1.999 0.000 /home/allenb/python/lib/python/pygccxml/parser/linker.py:61(__link_calldef) >> 581 1.118 0.002 1.408 0.002 /home/allenb/python/lib/python/pyplusplus/file_writers/writer.py:54(write_file) >> 624197 1.102 0.000 1.102 0.000 /usr/lib64/python2.4/site-packages/_xmlplus/sax/xmlreader.py:315(__getitem__) >> 254332 1.069 0.000 1.069 0.000 /usr/lib64/python2.4/logging/__init__.py:1136(getEffectiveLevel) >> 322753 1.062 0.000 1.062 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:148(does_match_exist) >> 174708 1.055 0.000 1.055 0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:203(endElement) >> 174708 1.050 0.000 34.924 0.000 /usr/lib64/python2.4/site-packages/_xmlplus/sax/expatreader.py:310(start_element) >> 303849 1.031 0.000 6.214 0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:322(__init__) >> 57752 1.003 0.000 2.576 0.000 /home/allenb/python/lib/python/pygccxml/declarations/calldef.py:371(function_type) >> 50590 0.998 0.000 4.635 0.000 /home/allenb/python/lib/python/pygccxml/declarations/scopedef.py:262(__create_matcher) >> 6431 0.994 0.000 1.169 0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:103(__init__) >> 53310 0.954 0.000 5.511 0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:336(__read_member_function) >> 290492 0.953 0.000 2.812 0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:249(is_volatile) >> 322753 0.949 0.000 2.011 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:171(__call__) >> 699219 0.945 0.000 0.945 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:26(enabled) >> 189878/189770 0.924 0.000 3.226 0.000 /home/allenb/python/lib/python/pygccxml/declarations/cpptypes.py:350(_create_decl_string) >> 724484 0.903 0.000 0.903 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:50(_get_demangled_name) >> 12960 0.891 0.000 4.366 0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/algorithm.py:37(_get_definition_set) >> 253539 0.889 0.000 1.954 0.000 /usr/lib64/python2.4/logging/__init__.py:943(debug) >> 241132 0.873 0.000 4.267 0.000 /home/allenb/python/lib/python/pygccxml/declarations/type_traits.py:233(remove_declarated) >> 1 0.864 0.864 3.924 3.924 /home/allenb/python/lib/python/pygccxml/parser/project_reader.py:424(_relink_declarated_types) >> 81593 0.857 0.000 1.468 0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:320(find_out_member_access_type) >> 1 0.852 0.852 12.578 12.578 /home/allenb/python/lib/python/pygccxml/parser/project_reader.py:359(_join_class_hierarchy) >> 269445 0.847 0.000 1.269 0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/decl_wrapper.py:113(exclude) >> 159205 0.835 0.000 0.835 0.000 /home/allenb/python/lib/python/pygccxml/declarations/pattern_parser.py:24(has_pattern) >> 627455 0.825 0.000 0.825 0.000 /home/allenb/python/lib/python/pygccxml/declarations/class_declaration.py:60(_get_related_class) >> 10535 0.811 0.000 2.844 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithm.py:175(find_all_declarations) >> 57350 0.808 0.000 2.596 0.000 /home/allenb/python/lib/python/pygccxml/declarations/calldef.py:108(__init__) >> 96156 0.791 0.000 2.486 0.000 /home/allenb/python/lib/python/pygccxml/declarations/algorithms_cache.py:70(reset) >> 72203 0.753 0.000 4.109 0.000 /home/allenb/python/lib/python/pygccxml/declarations/container_traits.py:40(get_container_or_none) >> 435851 0.750 0.000 0.750 0.000 /home/allenb/python/lib/python/pygccxml/declarations/calldef.py:145(_get_arguments) >> 30997 0.729 0.000 3.546 0.000 /home/allenb/python/lib/python/pyplusplus/decl_wrappers/__init__.py:118(create_member_function) >> 116184 0.723 0.000 1.176 0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:233(__read_access) >> 388905 0.721 0.000 0.721 0.000 /home/allenb/python/lib/python/pyplusplus/code_creators/algorithm.py:101(<lambda>) >> 159844 0.715 0.000 2.141 0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:341(__call__) >> 457956 0.704 0.000 0.704 0.000 /home/allenb/python/lib/python/pygccxml/declarations/declaration.py:42(_get_file_name) >> 96125 0.703 0.000 1.609 0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:297(__call__) >> 73767 0.694 0.000 2.680 0.000 /home/allenb/python/lib/python/pygccxml/declarations/matchers.py:118(__init__) >> 96175 0.667 0.000 1.048 0.000 /home/allenb/python/lib/python/pygccxml/parser/scanner.py:210(__update_membership) >> >> >> >> ------------------------------------------------------------------------- >> Take Surveys. Earn Cash. Influence the Future of IT >> Join SourceForge.net's Techsay panel and you'll get the chance to share your >> opinions on IT & business topics through brief surveys-and earn cash >> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >> _______________________________________________ >> pygccxml-development mailing list >> pyg...@li... >> https://lists.sourceforge.net/lists/listinfo/pygccxml-development >> >> >> > > > |
