|
From: Sébastien S. <sa...@us...> - 2008-02-08 18:00:11
Attachments:
callgrind_helper.tar.bz2
patch-callgrind.diff
|
Hi, I am trying to profile an application which mixes C and Python. When running callgrind on such an application, I can clearly see the C calls of the application but all the python part is represented by a complex tree of Py* functions representing Python C internals. Using callgrind --fn-skip='Py*' option makes it possible to ignore most of Python internals but then a lot of information is missing. So I would like to extend callgrind so that it can handle Python calls. Python gives the possibility to attach a C function to be called each time a python function is called with PyEval_SetTrace. So I made of very simple callgrind_helper module to enable this callback. Then by modifying callgrind/callstack.c:push_call_stack I can get the Python context (file, function and lineno) for which callgrind_helper_callback is called (see provided patch). Now I need to push these calls into callgrind call stack. I have started to analyze callgrind code to see how this can be done, but I would appreciate some hints from someone familiar with callgrind internals on how this can best be done. Thanks in advance -- Sébastien Sablé |
|
From: Josef W. <Jos...@gm...> - 2008-02-08 19:22:17
|
On Friday 08 February 2008, Sébastien Sablé wrote:
> When running callgrind on such an application, I can clearly see the C
> calls of the application but all the python part is represented by a
> complex tree of Py* functions representing Python C internals.
>
> Using callgrind --fn-skip='Py*' option makes it possible to ignore most
> of Python internals but then a lot of information is missing.
>
> So I would like to extend callgrind so that it can handle Python calls.
Interesting.
Why do you not use the profiler built into python itself, for example
hotshot? There is a converter provided with the KCachegrind package.
On the other hand, if you are interested in cache misses and number of
instructions executed, your approach is reasonable. However, I am not
really sure this is the right approach, as the results will depend on the
implementation of the python interpreter/byte code compiler, and only
indirectly on your code (hmm.. but so will the real runtime).
> Python gives the possibility to attach a C function to be called each
> time a python function is called with PyEval_SetTrace. So I made of very
> simple callgrind_helper module to enable this callback.
>
> Then by modifying callgrind/callstack.c:push_call_stack I can get the
> Python context (file, function and lineno) for which
> callgrind_helper_callback is called (see provided patch).
Hmm... not really the right place ;-)
First:
To get a function name into the output, you have to create a fn_node
struct for it. A factory function for these structs is
"get_fn_node_inseg( si, filename, fnname)" in callgrind/fn.c
The first parameter specifies the image (shared lib/binary), a value
of 0 should be OK here.
Second:
You have to be aware that there exist actually
two shadow stacks in callgrind. The one which is modified in
"callgrind/callstack.c:push_call_stack" is "CLG_(current_call_stack)".
This one exactly mirrors the real stack and is synchronised via the
real stack pointer, e.g. to get longjmp's and C++ exception handling right.
The other one ("CLG_(current_fn_stack)"), which differs from the real
shadow stack, is responsible for separating counter values, and for
the profile output later on. E.g. by your use of "--fn-skip", you trigger
changes between the two shadow stacks: some real functions never make it
onto the second stack, and thus will not appear in the output.
The second stack is changed e.g. with push_cxt(). The important line
to change probably is callgrind/bbcc.c:756:
...
CLG_(push_cxt)(CLG_(get_fn_node)(bb));
....
Instead of pushing the fn_node struct for the current basic block (bb),
you could push another fn_node struct onto the second call stack here,
created dynamically (but please only once for every python function).
Of course, this should only happen when "bb" maps to your helper callback.
You should be able to access your python FrameObject via the "sp" pointer.
Hope this helps,
Josef
>
> Now I need to push these calls into callgrind call stack. I have started
> to analyze callgrind code to see how this can be done, but I would
> appreciate some hints from someone familiar with callgrind internals on
> how this can best be done.
>
> Thanks in advance
>
> --
> Sébastien Sablé
>
>
>
>
>
>
>
>
>
|
|
From: Sébastien S. <sa...@us...> - 2008-02-11 16:28:42
Attachments:
python_callgrind.diff
|
Hi Josef,
Thanks a lot for those explanations! It was exactly what I needed.
I have been able to push Python calls on the stack according to what you
explained and I can now correctly see them with Kcachegrind, which makes
profiling my application much easier.
I added a --python-mode option to callgrind, which will activate the
detection of PyEval_EvalFrameEx and use this function to get the python
context.
The reason why I prefer to use callgrind instead of
hotshot/cProfile/profile is that my application mixes C and Python
(about 30% python, 15% C generated by Pyrex and 55% legacy C); so
hotshot gives me some very superficial results.
I am interested by the time spent in the python runtime, but I need to
know to which python code it corresponds in order to be able to reduce
this time.
I also found that since hotshot is using timestamps rather than
instruction counts like callgrind, the result is much less reliable
(especially when profiling very fast loops as the time spent collecting
profiling data can generate a shift).
I attach a first patch. It needs a few changes as I would like to pass
get_fn_node_inseg a valid obj_node, so that I can more easily
distinguish python modules in Kcachegrind. But it works well as is.
Just compile valgrind with --enable-python then
valgrind --tool=callgrind --python-mode python foo.py
regards
--
Sébastien Sablé
Josef Weidendorfer a écrit :
> On Friday 08 February 2008, Sébastien Sablé wrote:
>> When running callgrind on such an application, I can clearly see the C
>> calls of the application but all the python part is represented by a
>> complex tree of Py* functions representing Python C internals.
>>
>> Using callgrind --fn-skip='Py*' option makes it possible to ignore most
>> of Python internals but then a lot of information is missing.
>>
>> So I would like to extend callgrind so that it can handle Python calls.
>
> Interesting.
> Why do you not use the profiler built into python itself, for example
> hotshot? There is a converter provided with the KCachegrind package.
>
> On the other hand, if you are interested in cache misses and number of
> instructions executed, your approach is reasonable. However, I am not
> really sure this is the right approach, as the results will depend on the
> implementation of the python interpreter/byte code compiler, and only
> indirectly on your code (hmm.. but so will the real runtime).
>
>> Python gives the possibility to attach a C function to be called each
>> time a python function is called with PyEval_SetTrace. So I made of very
>> simple callgrind_helper module to enable this callback.
>>
>> Then by modifying callgrind/callstack.c:push_call_stack I can get the
>> Python context (file, function and lineno) for which
>> callgrind_helper_callback is called (see provided patch).
>
> Hmm... not really the right place ;-)
>
> First:
> To get a function name into the output, you have to create a fn_node
> struct for it. A factory function for these structs is
> "get_fn_node_inseg( si, filename, fnname)" in callgrind/fn.c
> The first parameter specifies the image (shared lib/binary), a value
> of 0 should be OK here.
>
> Second:
> You have to be aware that there exist actually
> two shadow stacks in callgrind. The one which is modified in
> "callgrind/callstack.c:push_call_stack" is "CLG_(current_call_stack)".
> This one exactly mirrors the real stack and is synchronised via the
> real stack pointer, e.g. to get longjmp's and C++ exception handling right.
>
> The other one ("CLG_(current_fn_stack)"), which differs from the real
> shadow stack, is responsible for separating counter values, and for
> the profile output later on. E.g. by your use of "--fn-skip", you trigger
> changes between the two shadow stacks: some real functions never make it
> onto the second stack, and thus will not appear in the output.
> The second stack is changed e.g. with push_cxt(). The important line
> to change probably is callgrind/bbcc.c:756:
> ...
> CLG_(push_cxt)(CLG_(get_fn_node)(bb));
> ....
>
> Instead of pushing the fn_node struct for the current basic block (bb),
> you could push another fn_node struct onto the second call stack here,
> created dynamically (but please only once for every python function).
> Of course, this should only happen when "bb" maps to your helper callback.
> You should be able to access your python FrameObject via the "sp" pointer.
>
> Hope this helps,
> Josef
>
>
>> Now I need to push these calls into callgrind call stack. I have started
>> to analyze callgrind code to see how this can be done, but I would
>> appreciate some hints from someone familiar with callgrind internals on
>> how this can best be done.
>>
>> Thanks in advance
>>
>> --
>> Sébastien Sablé
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
|
|
From: Josef W. <Jos...@gm...> - 2008-02-11 18:01:40
|
Hi Sébastien, On Monday 11 February 2008, Sébastien Sablé wrote: > I have been able to push Python calls on the stack according to what you > explained and I can now correctly see them with Kcachegrind, which makes > profiling my application much easier. Cool. > I added a --python-mode option to callgrind, which will activate the > detection of PyEval_EvalFrameEx and use this function to get the python > context. > > The reason why I prefer to use callgrind instead of > hotshot/cProfile/profile is that my application mixes C and Python > (about 30% python, 15% C generated by Pyrex and 55% legacy C); so > hotshot gives me some very superficial results. Ah, I did not think about this use case of mixing C/Python. Good that this matches your requirements! The question now is if there is a possibility to integrate this with callgrind somehow. I understand that quite some people could find this useful. But to be true, the current solution looks quite "hacky" to me. To use it, one has to modify things in the python interpreter, and there is this special case for python in callgrind. Up to now, callgrind is independent from the language used, as long as there exists debug information for the binary. So, one way would be to add the patch as user contribution into a contrib/ directory. And if there is enough pressure for a distribution, it could provide the patched callgrind. Or I try to refactor most parts of callgrind to make it available as modules for other tools. And callgrind itself would shrink dramatically in code size. Not really realistic. Another way would be to try to achieve the same/similar result with a more general solution not specific for python, but for any type of interpreter. E.g. using client requests, which allow to specify function calls/returns happening in a high-level call-graph. I suppose that your PyEval_EvalFrameEx function would be the ideal place to notify callgrind about these events. Hmm... this would add another dimension to the code position such as (high-level function, low-level function), and thus to some "explosion" of positions. In fact, it is quite similar to another dimension I wanted to add since quite some time: accessed data. I just wonder if there is an easy extension of the format and visualization, or if it is not really worth it... Josef |
|
From: Nicholas N. <nj...@cs...> - 2008-02-11 21:12:28
|
On Mon, 11 Feb 2008, Josef Weidendorfer wrote: > But to be true, the current solution looks quite "hacky" to me. To use it, > one has to modify things in the python interpreter, and there is this > special case for python in callgrind. Up to now, callgrind is independent > from the language used, as long as there exists debug information for the > binary. I agree. We also try to avoid conditional compilation, and dependencies on external software, including headers. But it is useful to mixed Python/C programmers, so it would be nice if there is a cleaner way to do it. N |
|
From: Josef W. <Jos...@gm...> - 2008-02-11 18:06:00
|
On Monday 11 February 2008, Salvatore Iovene wrote: > On Mon, Feb 11, 2008 at 05:27:45PM +0100, =?ISO-8859-1?Q?S=E9bastien_Sabl=E9_ wrote: > > Just compile valgrind with --enable-python then > > valgrind --tool=callgrind --python-mode python foo.py > > Hi all, > I'm jumping in this thread because I have question which is probably > related. I work on some C/C++ libraries, that are tested with a C > tester program. This tester program, then, is run several times in a > python script (I need python to control several instances of this > program, that interact with each other). > > Nowadays, I'm profiling all this by having python call the tester > program within valgrind (memcheck), and then manually inspecting the > logs. > > Since I run this python test suite from the cmake/ctest suite, and > would like the profiling results to end up in the Dart dashboard, I > would need to actually valgrind python and catch the memory leaks of > the processes that python spawns. But unfortunately valgrind doesn't > seem to catch those, I end up with no leaks (even if there are some). > > What is this python-mode? Can it be useful in my use case? This thread is about profiling C code which is linked into python, and not about profiling processes spawn by python. So I do not think this is relevant to your use case. Josef > Thanks a lot in advance! > |
|
From: Sébastien S. <sa...@us...> - 2008-02-13 17:35:34
Attachments:
python_callgrind.diff
|
Hi Josef, I improved my patch so that python calls can now be sorted by "ELF object" in Kcachegrind, each "ELF object" representing a different python script. For some reason it only works with the --compress-strings=no option. I also found that I could get much more information about the Python context when calling Python C/API. This is useful for example to profile Python extension written in C or Pyrex. (this is done by instrumenting additional functions like PyObject_Call, PyObject_GetAttr...). It has the extra advantage of breaking many cycles, so that the trace is much easier to read in Kcachegrind. There is probably a lot more that can be done, and I will complete this patch in coming days. I intend to post an announce about it on python-users and pyrex mailing list once a more complete patch is available in order to get feedback from Python users. Josef Weidendorfer a écrit : > But to be true, the current solution looks quite "hacky" to me. To use it, > one has to modify things in the python interpreter, and there is > this special case for python in callgrind. Up to now, callgrind is independent > from the language used, as long as there exists debug information for the binary. This patch does not need any modification on the Python side. You can run callgrind with this patch on the standard Python interpreter or any C application linked with the Python libraries and it should work. However you are right that it is "hacky" in that it brings some code specific to Python in callgrind and some dependencies on the Python headers. > So, one way would be to add the patch as user contribution into a contrib/ directory. > And if there is enough pressure for a distribution, it could provide the patched callgrind. > > Or I try to refactor most parts of callgrind to make it available as modules for other > tools. And callgrind itself would shrink dramatically in code size. Not really realistic. > > Another way would be to try to achieve the same/similar result with a more general > solution not specific for python, but for any type of interpreter. > E.g. using client requests, which allow to specify function calls/returns > happening in a high-level call-graph. I suppose that your PyEval_EvalFrameEx > function would be the ideal place to notify callgrind about these events. > Hmm... this would add another dimension to the code position such as > (high-level function, low-level function), and thus to some "explosion" of positions. > In fact, it is quite similar to another dimension I wanted to add since quite some time: > accessed data. I just wonder if there is an easy extension of the format and > visualization, or if it is not really worth it... A patch in contrib/ would be fine for the moment. But I am sure that a generic way to push higher level functions would interest a lot of people (so that there could be some callgrind plug-ins to profile python2.5, python2.3, php, perl, whatever...). regards -- Sébastien Sablé |
|
From: Josef W. <Jos...@gm...> - 2008-02-13 19:37:23
|
Hi Sébastien, On Wednesday 13 February 2008, Sébastien Sablé wrote: > I improved my patch so that python calls can now be sorted by "ELF > object" in Kcachegrind, each "ELF object" representing a different > python script. Sorry, I do not understand this. Why is the "filename" field not enough? Or is this a workaround for a usability problem in KCachegrind? > For some reason it only works with the > --compress-strings=no option. What it the problem here? The parser in KCachegrind? > I also found that I could get much more information about the Python > context when calling Python C/API. This is useful for example to profile > Python extension written in C or Pyrex. (this is done by instrumenting > additional functions like PyObject_Call, PyObject_GetAttr...). > It has the extra advantage of breaking many cycles, so that the trace is > much easier to read in Kcachegrind. I have no idea about phython internals, so I can not comment on the usefulness of catching these additional functions. So you are "just" adding further artificial functions to the callgraph? > There is probably a lot more that can be done, and I will complete this > patch in coming days. I intend to post an announce about it on > python-users and pyrex mailing list once a more complete patch is > available in order to get feedback from Python users. > > Josef Weidendorfer a écrit : > > But to be true, the current solution looks quite "hacky" to me. To use it, > > one has to modify things in the python interpreter, and there is > > this special case for python in callgrind. Up to now, callgrind is independent > > from the language used, as long as there exists debug information for the binary. > > This patch does not need any modification on the Python side. Oh, then I misunderstood. When is "PyEval_EvalFrameEx" actually called? I thought this is a special run mode of Python, which needs modifications on the Python side... > You can > run callgrind with this patch on the standard Python interpreter or any > C application linked with the Python libraries and it should work. > However you are right that it is "hacky" in that it brings some code > specific to Python in callgrind and some dependencies on the Python headers. So if I want to try to abstract your needs for python from callgrind by adding client requests to define artificial function names for the call graph, this would mean an actual change of python, which currently is not needed? > A patch in contrib/ would be fine for the moment. But I am sure that a > generic way to push higher level functions would interest a lot of > people (so that there could be some callgrind plug-ins to profile > python2.5, python2.3, php, perl, whatever...). Yes. However, there currently does not exist any further extension of VG tools to allow for "callgrind plug-ins". But it should be possible to make a kind of library version of callgrind, to base other VG tools on it. Or even better: Support it via just an configuration file (or command line options). Now that Julian is adding support for reading debug info for data, it should be possible to look up arbitrary variables. Josef > > regards > > -- > Sébastien Sablé > |
|
From: Nicholas N. <nj...@cs...> - 2008-02-13 22:56:28
|
On Wed, 13 Feb 2008, Josef Weidendorfer wrote: >> A patch in contrib/ would be fine for the moment. But I am sure that a >> generic way to push higher level functions would interest a lot of >> people (so that there could be some callgrind plug-ins to profile >> python2.5, python2.3, php, perl, whatever...). > > Yes. > > However, there currently does not exist any further extension of VG > tools to allow for "callgrind plug-ins". But it should be possible to > make a kind of library version of callgrind, to base other VG tools > on it. > > Or even better: Support it via just an configuration file (or command line > options). Now that Julian is adding support for reading debug info for data, > it should be possible to look up arbitrary variables. I'd strongly encourage you to find a way, if possible, to make this feature generic if you plan to include it. Code that is not normally run (eg. due to being in contrib/, or only included if certain configure options are used) will be tested less and be much less robust. This is one of those things that is really important in practice. If the feature can be extended to other languages than Python that obviously makes it much better too. Nick |
|
From: Josef W. <Jos...@gm...> - 2008-02-13 23:32:01
|
On Wednesday 13 February 2008, Nicholas Nethercote wrote: > On Wed, 13 Feb 2008, Josef Weidendorfer wrote: > > >> A patch in contrib/ would be fine for the moment. But I am sure that a > >> generic way to push higher level functions would interest a lot of > >> people (so that there could be some callgrind plug-ins to profile > >> python2.5, python2.3, php, perl, whatever...). > > > > Yes. > > > > However, there currently does not exist any further extension of VG > > tools to allow for "callgrind plug-ins". But it should be possible to > > make a kind of library version of callgrind, to base other VG tools > > on it. > > > > Or even better: Support it via just an configuration file (or command line > > options). Now that Julian is adding support for reading debug info for data, > > it should be possible to look up arbitrary variables. > > I'd strongly encourage you to find a way, if possible, to make this feature > generic if you plan to include it. Yes, of course. I have not yet decided how to include it if at all. If I add client requests which looks like a general solution (as far as callgrind goes), the needed macros need to be included into python. While this is the better solution, it is not really helpful if such a modified python never finds its way into distributions. > Code that is not normally run (eg. due > to being in contrib/, or only included if certain configure options are > used) will be tested less and be much less robust. This is one of those > things that is really important in practice. Yes. There is probably no big difference if it would be in contrib/ or if Sebastian maintains it itself, or if the patch finds its way into contrib/ of python source: It won't be run normally, as you said. > If the feature can be extended to other languages than Python that obviously > makes it much better too. IMHO there is a lot of potential for it: a JITter could notify about the source relation of generated code. Probably even useful for self-hosting Valgrind runs (making it even slower ;-) Josef |
|
From: Sébastien S. <sa...@us...> - 2008-02-14 10:30:21
|
Hi Josef,
Josef Weidendorfer a écrit :
> Sorry, I do not understand this. Why is the "filename" field not enough?
> Or is this a workaround for a usability problem in KCachegrind?
There is a drop box widget in Kcachegrind where you can select how you
want to sort the functions. You can choose between (No Grouping), ELF
object, Source File, ...
The "filename" arg passed to get_fn_node_inseg in callgrind corresponds
to "Source File".
However the ELF object is determined by the obj_node in the SegInfo, and
since I was passing a NULL si, all the python functions were associated
to the "(unknown)" ELF File.
With my last patch, the python functions defined in the foo.py script
will be associated to the foo.py source file and foo.py ELF object.
It provides a different way of sorting which I find useful.
>> For some reason it only works with the
>> --compress-strings=no option.
>
> What it the problem here? The parser in KCachegrind?
I haven't looked in details yet. However when using
compress-strings=yes, some functions would be reported in the wrong
source file/ELF object.
> I have no idea about phython internals, so I can not comment on the usefulness
> of catching these additional functions.
> So you are "just" adding further artificial functions to the callgraph?
> Oh, then I misunderstood. When is "PyEval_EvalFrameEx" actually called? I thought
> this is a special run mode of Python, which needs modifications on the Python side...
I am going to try to explain what I did:
PyEval_EvalFrameEx, PyObject_CallMethod, PyObject_Call... are some C
functions in the Python runtime which are called at a low level to
execute some functions or methods. The Python functions which is
executed by them can be determined by the args passed to those functions.
These functions are called very often so when you run an unmodified
callgrind on a python application, you get cycles made of those
functions and it is difficult to analyze the trace.
What I am doing is detect those C functions thanks to callgrind, then
instead of pushing the name of those functions in the call stack, I get
the name of the python functions which are executed by looking into the
args of the function and I push this information instead in the call stack.
For example PyObject_CallMethod is used in the python runtime to call
the method of a class.
class Foo:
def bar(self):
pass
The following python code
foo.bar()
will be converted at the C Level in:
PyObject_CallMethod(object, "bar")
where object is of type Foo, so that in the C struct object->type->name
is "Foo".
So when callgrind encounters the C function PyObject_CallMethod, I get
the first and second args of this functions, and instead of pushing
PyObject_CallMethod in the call stack, I push the "virtual" function
"Foo.bar" instead.
As a result the trace has much less cycles, and I can actually see which
python functions get called instead of the internal of the python runtime.
So I am not exactly adding some events, but rather renaming them with a
more meaningful name.
I also try to skip some C functions in the python runtime which do not
bring information.
I hope this explanation makes things clearer.
> So if I want to try to abstract your needs for python from callgrind by adding
> client requests to define artificial function names for the call graph, this
> would mean an actual change of python, which currently is not needed?
I don't think any modification should be needed on the application being
analyzed. What would be great, would be if in callgrind each time a C
function call is detected (around bbcc.c:753), this function name (and
its args) could be passed to an extension for analysis. This extension
would dynamically decide if the function should be added to the call
stack as is, or if it should be skipped or renamed.
regards
--
Sébastien Sablé
|
|
From: Josef W. <Jos...@gm...> - 2008-02-14 17:48:15
|
[Julian: below there is the question what would be the best solution to make VG tools extendable, where the extension would be provided in a separate package. Any ideas?] Hi Sebastien, On Thursday 14 February 2008, Sébastien Sablé wrote: > With my last patch, the python functions defined in the foo.py script > will be associated to the foo.py source file and foo.py ELF object. Ok. From an more abstract view point, "ELF object" and "Source File" are just different groupings of functions. For the python functions, you chose to make these groupings equal. And the benefit would be that independent of whether you want to see the "ELF object" or "Source File" grouping for your C functions, you simultaneous get the "Python Source" grouping. Yes? > I haven't looked in details yet. However when using > compress-strings=yes, some functions would be reported in the wrong > source file/ELF object. Hmm. It could be that this is not about parsing the file, but because of some optimizations in KCachegrind where I took some semantics as given. > > I have no idea about phython internals, so I can not comment on the usefulness > > of catching these additional functions. > > So you are "just" adding further artificial functions to the callgraph? > > > Oh, then I misunderstood. When is "PyEval_EvalFrameEx" actually called? I thought > > this is a special run mode of Python, which needs modifications on the Python side... > > I am going to try to explain what I did: > > PyEval_EvalFrameEx, PyObject_CallMethod, PyObject_Call > ... OK, now I hope I understand. Am I right to say that your additions are specific to a given python implementation (the function names to catch etc.). Or is there some official specification which says that every implementation of a python interpreter has to call these functions on the given conditions with exactly this name? Further, you take a stack value and cast it to some python structure to get the function/file names. This also seems to be very specific to the given python implementation, as well as specific to the ABI of the current architecture (e.g. the patch probably would be different on x86-64). > > So if I want to try to abstract your needs for python from callgrind by adding > > client requests to define artificial function names for the call graph, this > > would mean an actual change of python, which currently is not needed? > > I don't think any modification should be needed on the application being > analyzed. What would be great, would be if in callgrind each time a C > function call is detected (around bbcc.c:753), this function name (and > its args) could be passed to an extension for analysis. This extension > would dynamically decide if the function should be added to the call > stack as is, or if it should be skipped or renamed. Yes, that's reasonable. The first thing would be to decide about the technical issues allowing for extensions of VG tools. You want to have access to python structures. So the extension probably has to be a binary blob callable in a callgrind run. As VG tools a static binaries, there is no way to dynamically link code to a tool... Hmm.. I have to think about it (Julian?). The other question is who should provide the python specific callgrind extension. As it dependson the given python implementation, I would say that it should be distributed with some python-devel package, using the extension support in callgrind. Josef |
|
From: Sébastien S. <sa...@us...> - 2008-02-19 14:00:47
|
Hi Josef, Josef Weidendorfer a écrit : > Ok. From an more abstract view point, "ELF object" and "Source File" are > just different groupings of functions. For the python functions, you chose > to make these groupings equal. And the benefit would be that independent > of whether you want to see the "ELF object" or "Source File" grouping for > your C functions, you simultaneous get the "Python Source" grouping. Yes? Yes, exactly; it is just for practical reasons. It provides a different way of sorting functions. >> I am going to try to explain what I did: >> >> PyEval_EvalFrameEx, PyObject_CallMethod, PyObject_Call >> ... > > OK, now I hope I understand. Great. > Am I right to say that your additions are specific to a given python implementation > (the function names to catch etc.). Or is there some official specification which > says that every implementation of a python interpreter has to call these functions > on the given conditions with exactly this name? > Further, you take a stack value and cast it to some python structure to get the > function/file names. This also seems to be very specific to the given python > implementation, as well as specific to the ABI of the current architecture (e.g. > the patch probably would be different on x86-64). You are right; The additions I did are specific to CPython 2.5.1 and need to be recompiled to match the ABI of the architecture. That being said, the Python C API is rather stable so provided that you recompile it with the right version of headers, the patch should work as is with Python 2.4 or only require some minimal changes (possibily also with 2.3). I didn't have time to improve the patch yet, but I have plenty of ideas to so. regards -- Sébastien Sablé |
|
From: Nicholas N. <nj...@cs...> - 2008-02-14 21:14:47
|
On Thu, 14 Feb 2008, Josef Weidendorfer wrote: > Yes, that's reasonable. The first thing would be to decide about the > technical issues allowing for extensions of VG tools. > You want to have access to python structures. So the extension probably > has to be a binary blob callable in a callgrind run. > As VG tools a static binaries, there is no way to dynamically link code > to a tool... Hmm.. I have to think about it (Julian?). Is there any way to encode this information in a file, and have Callgrind read that file? Then it wouldn't require any official notion of tool extension. Nick |
|
From: Josef W. <Jos...@gm...> - 2008-02-15 01:35:48
|
On Thursday 14 February 2008, Nicholas Nethercote wrote: > Is there any way to encode this information in a file, and have Callgrind > read that file? For this python extension, Sebastien needs to access the content of some variables on the stack when entering a given function. Potentially without data debug info, how would you do it? We already have a shared lib which is dynamically linked to client code when running a given VG tool. Could this somehow be used for an extension mechanism? E.g. by registering some functions on initialisation? The big problem here of course is that callgrind would like to run such a function on the real processor, and not on the simulated one... Hmm. In this case, it should even work to define function wrappers which execute client requests. Josef > Then it wouldn't require any official notion of tool > extension. > > Nick > |