Re: [PerfSuite-users] psprocess weirdness...

[Rick K. <rk...@nc...>]

Greetings all,

This note is a followup to the problem reported by Tirath Ramdas last week 
about psprocess profiling reports differing under certain circumstances.

Our experiments indicate that the problem source is indeed step 2 as 
described below, the use of BFD from within the PerfSuite Tcl BFD 
extension.  It doesn't seem to be consistent, but the likely cause is a 
failure related to BFD initialization that persists through the course of 
processing the input XML document.  The result is that no PC 
(program counter) values are mapped successfully to source code lines.
The problem seems to be more likely to occur if the binary being processed
was compiled without debugging/symbol information (-g).

I've not been able to reproduce this error on the machines that I have 
access to, if anyone has a test case (source) on which it occurs and would 
like to forward to me, please do.

In the meantime, a workaround has been implemented that bypasses the use 
of the PS Tcl extension in favor of using the "addr2line" utility as 
external process.  Those using psprocess on an IA-64 platform or who 
configured PerfSuite with --disable-binutils are already using this 
method, but for the rest the BFD Tcl extension is used by default.  The 
workaround gives the user of psprocess the option to force use of 
addr2line by setting an environment variable, PSPROCESS_MAPPER.

Tirath reports that the workaround produces more consistent results for 
the situations that previously failed.  This option has been implemented 
in the development version of PerfSuite (0.6.2a3) and will be included 
with the next release.  We'll also take a closer look at the Tcl extension 
itself to see if we can isolate the core problem location and address 
within there as well (using the extension is faster than using an external 
process, especially for profiles with a large number of addresses to 
which samples were attributed).

Many thanks to Tirath for the report and assist with tracking down,

Rick

> 
> psprocess' job is 
> conceptually pretty simple: it has to:
> 
> 1. Parse an XML document and extract the samples within (it uses Tcl/tDOM)
> 2. Map each sample to a source code location (using libbfd or addr2line)
> 3. Do some management/bookkeeping/summarization and display results

[ ... ]

> 
> To rule out (or implicate) step 2, I'll work with you off-list to verify 
> the PC->source mappings that are generated.  To do this, I'll modify the 
> portion of psprocess where this is going on to provide debugging output 
> for the individual PC values and where they are mapped.  This will take a 
> few days (it'll be based on the 0.6.1 version of PerfSuite which your 
> psinv output shows you are using).  I'll supply you a patch which 
> should simply be a drop-in file replacement (since psprocess is a 
> script, not a compiled object/binary). There is currently no such 
> tracing supported in psprocess but I think it will be a useful thing to 
> incorporate in general to track things like this.
> 
> If the mappings are all identical (as they should be, but that's what 
> we're checking for), then it'll point to data management within psprocess
> as the problem and we can go from there. 
> 
> In any case, we'll summarize the results back to this mailing list and 
> with luck will have pinpointed it for the upcoming 0.6.2a3 release.