Re: [PerfSuite-users] How to get combined profile results

[<rk...@il...>]

Sherry,

Your email to the PerfSuite user mailing list was bounced by SourceForge (I think because you are not a list subscriber), but I found a copy in my filtered email box.

Regarding displaying combined profile results with psprocess: I am not sure I would say that is a "problem", but Henry is correct in that there is no support for displaying profiling results from parallel programs (i.e., multiple XML profiling documents) from psprocess.  Primarily, this is due to the practical problem of how to summarize that type of information in the text-based output that psprocess concentrates on.  There is only so much "screen space" to work with when writing to the "console".

There isn't a PerfSuite-only solution to this situation, either in 0.x or 1.0.  However, recent versions of the TAU performance system from the University of Oregon does support graphical display of profiling results through TAU's visualizer "ParaProf".  ParaProf understands the PerfSuite file formats (you still have to translate the raw samples to source code locations through "psprocess -x").

Your user's questions are entirely reasonable.  There was/is support for examining parallel profiles through the VProf package from Sandia, but that package seems to no longer be active, so it is deprecated in PerfSuite (also, you would need to have VProf available in the first place for display).

Recent versions of PerfSuite (1.0) provide support for generating files that can be displayed by the Cube visualizer from the Scalasca project in Europe (http://www.scalasca.org), and I find that very useful and compact.  However, before you consider upgrading, I would like to know what the target platform is, primarily because I am aware that Altix platforms with ProPack 6+ can have difficulties with the psrun command.

If we can help you get things moving in a better way, please let us know, at present those are my comments on the current capabilities and options.

Rick


----------------
Sent By"Sherry Chang" <she...@na...>
On:	November 2, 2009 12:42 PM
To:	per...@li...
Cc:	She...@na...
Hi,

Our site (NASA Ames) is currently using PerfSuite version 0.6.2b1.

One of our users would like to get combined profile results from the individual
profile *.xml file and but was not able to do so.

Henry Jin mentioned that this is a known problem with version 0. Is this changed
in version 1?

Thank you,
Sherry Chang
User Services
NASA Advanced Supercomputing Division

>Sherry,

>The problem of not reporting combined profile results is known
>in the 0.x versions of PerfSuite.  I'm not sure if anything has been
>changed in the latest version (1.x).  I don't really know a solution.
>It's probably better to post an inquiry to
><per...@li...>
>Rick Kufrin is very responsive in answering questions.

>-Henry


On 10/31/09 4:49 PM, Sherry Chang wrote:

    Hi Henry,


    Using the counting mode, one can combine results from individual

    *.xml files and get an overall report of the whole code instead of

    each individual process. For example,


    mpirun -np 4 psrun -f ./new_pi_g > kkk

    psprocess -c new_pi_g.120*.cfe1.xml > all_counting.xml

    psprocess all_counting.xml > all_counting.psprocess.out


    In the all_counting.psprocess.out, one sees:


    Minimum and Maximum

    Min                 Max

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    ========================================================================

    % CPU utilization...................................        98.06

    [cfe1]        99.54 [cfe1]

    % cycles stalled on any resource....................        56.66

    [cfe1]        56.96 [cfe1]

    ...

    Aggregate Statistics                           Median

    Mean       StdDev          Sum

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    =

    ========================================================================

    % CPU utilization.......................        99.46

    99.13         0.71       396.51

    % cycles stalled on any resource........        56.73

    56.77         0.13       227.08

    Bandwidth used to level 1 cache (MB/s)..       814.28

    821.81        17.04      3287.26



    Instead of the counting mode, a user tried to use PerfSuite to profile

    his MPI code

    and get an overall statistics for his whole code instead of each

    process. He would like

    to know where in the code is taking longer time to run. But he was not

    able to do so.


    Here are part of his email:


        psrun generating a bunch of XML files, but I am unclear how to combine

        them to get overall statistics. First I tried,

        psprocess *.xml > out.txt


        That produced a report in plain text, but it looks like it just

        processed the first XML file.


        Then I tried,

        psprocess -c *.xml > all.out


        That produced an XML file, I think. It is much longer, but I am

    unclear

        what to do with it.


        I ran,

        psprocess all.out


        to process combined "out" file, but I got this error message:

        document contains profiling data (only vmon output is currently

    supported)


        I added --vmon and ran,

        psprocess --vmon all.out


        and got this error message:

        [stack]: cannot open `[stack]' (No such file or directory)


    I did similar experiments like he did and got the same behavior.

    The experiments I tried (papi_profile_cycles.xml and itimer.xml)

    both showed this behavior.


    Do you know any way that will aggregate the info from the profiling

    results

    of each process?


    Sherry