pyprof-devel

[pyprof-devel] RFC: Threading-Aware Profiler for Python

[Christian S. <ma...@ap...>]

Hi,

Python ships with a profiler module which, unfortunately, is almost useless in a multi-threaded environment. *

I've created an alternative profiler module which queries per-thread CPU usage via netlink/taskstats, which limits the applicability to Linux (which shouldn't be much of an issue, profiling is usually not done by end users). It implements two modes: a "sampling" (does CPU time accounting based on stack fraames 100 times per second, by default) and a "deterministic" profiler (does CPU time accounting on each function call/return, based on sys.profiler interface). The deterministic profiler is currently implemented in pure python (except for taskstats interface) and much slower than the sampling profiler.

Usage (don't forget make to build the c module):
python
>> from Profiler import *
>> def f(): do_something()
>> sampling_profiler(f)
or
>> deterministic_profiler(f)


Output is currently in the form of annotated source code (xyz.py.html, in the same directory where xyz.py resides). Before the *_profiler function returns, it iterates over all code objects it encountered and annotates the source files with 2 columns in front:
- 1st column: real time
- 2nd column: CPU time

numbers are log2(time_in_ns), colors are green-to-yellow for below-average and yellow-to-red for above-average metrics (relative to the average metric for all lines of the code object with a metric > 0).

Is there common need for such a module? 

Is it possible to have this included in the standard cPython distribution?

Which functional changes (besides a modification of the annotation output which shouldn't spread its result all over the FS) would be required to get this included?

Which non-functional changes would be required to get this included?


Please direct traffic regarding this subject to pyp...@li... (no I'm not subscribed to python-dev).


SF project page:

https://sourceforge.net/projects/pyprof/


git repository:

git://pyprof.git.sourceforge.net/gitroot/pyprof


Regards,
Christian


*) to be more exact there are at least three profiler modules: profile, cProfile, and hotshot, while I did only try (and failed) to use profile in a multi-threaded environment (by manually setting threading.profile to the profiling function), glancing at the source, I'm pretty sure that cProfile behaves similarly; I didn't test the hotshot module, but it does some other trade-offs (space-for-time), so I think that "pyprof" still adds some value

Re: [pyprof-devel] [Python-Dev] RFC: Threading-Aware Profiler for Python

[Aahz <aa...@py...>]

On Mon, May 04, 2009, Christian Schubert wrote:
>
> Python ships with a profiler module which, unfortunately, is almost
> useless in a multi-threaded environment. *
>
> I've created an alternative profiler module which queries per-thread
> CPU usage via netlink/taskstats, which limits the applicability to
> Linux (which shouldn't be much of an issue, profiling is usually
> not done by end users). It implements two modes: a "sampling" (does
> CPU time accounting based on stack fraames 100 times per second, by
> default) and a "deterministic" profiler (does CPU time accounting
> on each function call/return, based on sys.profiler interface). The
> deterministic profiler is currently implemented in pure python (except
> for taskstats interface) and much slower than the sampling profiler.

If you want to discuss this, please subscribe to python-ideas and repost
your message.  Generally speaking, in order to include modules like this,
they need to prove themselves over time and may require PEP approval.  If
you choose to move the discussion to python-ideas, it would help if you
mention known uses of your module.
-- 
Aahz (aa...@py...)           <*>         http://www.pythoncraft.com/

"It is easier to optimize correct code than to correct optimized code."
--Bill Harlan

Re: [pyprof-devel] [Python-Dev] RFC: Threading-Aware Profiler for Python

[Christian S. <ma...@ap...>]

Hi,

On 2009-05-04 17:56:04 AAhz wrote:
> If you want to discuss this, please subscribe to python-ideas and repost
> your message.  Generally speaking, in order to include modules like this,
> they need to prove themselves over time and may require PEP approval.  If
> you choose to move the discussion to python-ideas, it would help if you
> mention known uses of your module.

At the moment there are no known users except myself. So basically I'm trying to figure out whether this is something which is needed by others (thus worth putting further work in), or whether it's a complete waste of time. I might reconsider reposting this at python-* if there is enough interest - time will show.

Regards,
Christian

Re: [pyprof-devel] [Python-Dev] RFC: Threading-Aware Profiler for Python

[Bill J. <ja...@pa...>]

Hi, Christian.

Christian Schubert <ma...@ap...> wrote:

> I've created an alternative profiler module which queries per-thread
> CPU usage via netlink/taskstats, which limits the applicability to
> Linux (which shouldn't be much of an issue, profiling is usually not
> done by end users).

A surprisingly large # of developers are running on OS X these days,
though.  I suggest make it work there, too.

Bill

Re: [pyprof-devel] [Python-Dev] RFC: Threading-Aware Profiler for Python

[Paul M. <p.f...@gm...>]

2009/5/4 Bill Janssen <ja...@pa...>:
> Hi, Christian.
>
> Christian Schubert <ma...@ap...> wrote:
>
>> I've created an alternative profiler module which queries per-thread
>> CPU usage via netlink/taskstats, which limits the applicability to
>> Linux (which shouldn't be much of an issue, profiling is usually not
>> done by end users).
>
> A surprisingly large # of developers are running on OS X these days,
> though.  I suggest make it work there, too.

And Windows. I doubt that the various Windows-specific modules
available were developed on Linux. And I wouldn't assume that all of
the platform-neutral modules are developed on Linux, or even that the
developers have access to Linux. (I know I don't, short of building a
brand new virtual machine...)

Paul.

Re: [pyprof-devel] [Python-Dev] RFC: Threading-Aware Profiler for Python

[Christian S. <ma...@ap...>]

Hi Paul,

On 2009-05-04 21:52:02 you wrote:
> 2009/5/4 Bill Janssen <ja...@pa...>:
> > Hi, Christian.
> >
> > Christian Schubert <ma...@ap...> wrote:
> >
> >> I've created an alternative profiler module which queries per-thread
> >> CPU usage via netlink/taskstats, which limits the applicability to
> >> Linux (which shouldn't be much of an issue, profiling is usually not
> >> done by end users).
> >
> > A surprisingly large # of developers are running on OS X these days,
> > though.  I suggest make it work there, too.
> 
> And Windows. I doubt that the various Windows-specific modules
> available were developed on Linux. And I wouldn't assume that all of
> the platform-neutral modules are developed on Linux, or even that the
> developers have access to Linux. (I know I don't, short of building a
> brand new virtual machine...)

I don't see a problem there. As long as OS X and Windows provide functions to query the appropriate metrics, this should be not much of an issue. Except, that I don't have access to an OS X machine on which to test on (and I didn't to development on Windows in ages), so patches from interested parties might shorten the way to a semi-interoperable solution here.

Regards,
Christian