#6388 Deep trace of single pageload

Dave Brondsema

Let's do a trace (e.g. start with TimerMiddleware verbose mode to get every pymongo call; maybe try line_profiler later) of common functionality to every page. Focus on just the core parts of allura, not looking into the specifics of whatever page you're on. Might be helpful to test hitting an external link URL, so there's not much tool-specific going on, just core route handling, project/tool lookup, permission checks, etc. Identify opportunities for improvement.


Chat: 52c596f60594ca5f48ca91d2
Chat: 52c596f70594ca5f48ca91d5
Chat: 52c596f70594ca5f48ca91d8
Chat: 52c596f70594ca5f48ca91db
Chat: 52c596f70594ca5f48ca91de
Chat: 52c596f70594ca5f48ca91e1
Chat: 52c596f70594ca5f48ca91e4
Chat: 52c596f70594ca5f48ca91e7
Chat: 52c5a39b0594ca5f48ca91fc
Chat: 52c5a39b0594ca5f48ca91ff
Chat: 52c5cae70594ca5f48ca920e
Chat: 52c5cae70594ca5f48ca9211


1 2 > >> (Page 1 of 2)
  • Dave Brondsema
    Dave Brondsema

    Based on New Relic "xray sessions", potential candidates to investigate more are:

    • controllers/root.py Neighborhood.query.get: 11% of avg _nav.json request. Time spent is all in mongo. Cache it for a while?
    • model/project.py install_anchored_tools: 15% of avg _nav.json request. Most of time spent is in mongo. Seems to be the self.app_configs property load?
    • lib/security.py obj.neighborhood_project: 7.3% of avg _nav.json request. Cache it for a while?
    • post_thread/post_widget.html has repeated sequential calls to Artifact.refs taking 13% of avg ticket view, backrefs (5.2%), user icon_url which is already memozied (8.4%)
    Last edit: Dave Brondsema 2013-12-09
    • Dave Brondsema
      Dave Brondsema

      Neighborhood.query.get is all over the place. For example in a discussion thread, each post calls User.icon_url which calls private_project which uses self.neighborhood but even though that is a LazyProperty it is instance-level so doesn't get re-used much. There are many other places too.

      We could have a general purpose Neighborhood cache system and use it in lots of places. Some tests might not want it to be cached though. Another even generalized possibility is to extend Ming's identity map. It currently indexes objects by _id only, but we could make it do lookups by any unique key(s) for a given model (e.g. url_prefix for Neighborhood).

  • Dave Brondsema
    Dave Brondsema


    There are a few performance & configuration improvements in this branch, but none of the particular areas mentioned above are addressed. Most of the work in this branch is a script to have a standard benchmark of call counts that can be run for any git revision (can't go earlier than the creation of the script though).

    Run the main script to get current call counts, and to see specific calls allura/scripts/perf$ ./call_count.py -v

    Use the shell wrapper to create a CSV file with multiple commits recorded (suitable for graphing in a spreadsheet app). E.g.: allura/scripts/perf$ ./call_count_walk_git.sh bdde98d..HEAD

    For integration with Jenkins, we should be able to run call_count.py --data-file /some/permanent/location.csv and hook that up to https://wiki.jenkins-ci.org/display/JENKINS/Plot+Plugin

1 2 > >> (Page 1 of 2)


Cancel   Add attachments