[pyprof-devel] RFC: Threading-Aware Profiler for Python

[Christian S. <ma...@ap...>]

Hi,

Python ships with a profiler module which, unfortunately, is almost useless in a multi-threaded environment. *

I've created an alternative profiler module which queries per-thread CPU usage via netlink/taskstats, which limits the applicability to Linux (which shouldn't be much of an issue, profiling is usually not done by end users). It implements two modes: a "sampling" (does CPU time accounting based on stack fraames 100 times per second, by default) and a "deterministic" profiler (does CPU time accounting on each function call/return, based on sys.profiler interface). The deterministic profiler is currently implemented in pure python (except for taskstats interface) and much slower than the sampling profiler.

Usage (don't forget make to build the c module):
python
>> from Profiler import *
>> def f(): do_something()
>> sampling_profiler(f)
or
>> deterministic_profiler(f)


Output is currently in the form of annotated source code (xyz.py.html, in the same directory where xyz.py resides). Before the *_profiler function returns, it iterates over all code objects it encountered and annotates the source files with 2 columns in front:
- 1st column: real time
- 2nd column: CPU time

numbers are log2(time_in_ns), colors are green-to-yellow for below-average and yellow-to-red for above-average metrics (relative to the average metric for all lines of the code object with a metric > 0).

Is there common need for such a module? 

Is it possible to have this included in the standard cPython distribution?

Which functional changes (besides a modification of the annotation output which shouldn't spread its result all over the FS) would be required to get this included?

Which non-functional changes would be required to get this included?


Please direct traffic regarding this subject to pyp...@li... (no I'm not subscribed to python-dev).


SF project page:

https://sourceforge.net/projects/pyprof/


git repository:

git://pyprof.git.sourceforge.net/gitroot/pyprof


Regards,
Christian


*) to be more exact there are at least three profiler modules: profile, cProfile, and hotshot, while I did only try (and failed) to use profile in a multi-threaded environment (by manually setting threading.profile to the profiling function), glancing at the source, I'm pretty sure that cProfile behaves similarly; I didn't test the hotshot module, but it does some other trade-offs (space-for-time), so I think that "pyprof" still adds some value