perfsuite-users

Re: [PerfSuite-users] Problems running examples

[Rick K. <rk...@nc...>]

Abdullah,

This is somewhat of a "catch-all"/generic error that can be a little 
hard to diagnose without additional information, such as what type of 
hardware you are running on, which version of PAPI you are using, which 
kernel support you have (perfctr or perfmon).

There are occasional posts in this mailing list from others that may be 
useful to have a look at, see for example:

http://sourceforge.net/mailarchive/forum.php?thread_name=20061117154330.GA25518%40ichec.ie&forum_name=perfsuite-users

For me, it is typically best do reconfigure/recompile PerfSuite with 
debugging enabled ("--enable-debug"), try to run an example with the 
environment variable PS_DEBUG set to the value 3 or higher, and examine 
the debugging output to learn where things are failing (or pass that on 
to me, as it's easier for me to make sense of it).

Rick

p.s. it has been a very long time since an update to PerfSuite has been 
released, but one currently being tested should hopefully be available 
within the next couple of weeks: 0.6.2a7

>
>
> Subject:
> Problems running examples
> From:
> "Abdullah Kayi" <ap...@gm...>
> Date:
> Mon, 23 Jun 2008 18:41:35 -0800
> To:
> per...@li...
>
> To:
> per...@li...
>
>
> Hi all,
>
> I am using PerfSuite 0.6.2a6. I compiled and installed succesfully but 
> I cannot run any of the examples. I am always getting the following error:
> libpsrun fatal error: error reported by performance software layer
>
> Any ideas on this?
>
> Regards,
>
> Abdullah

Re: [PerfSuite-users] Problems running examples

[Abdullah K. <ap...@gm...>]

And on top of these, for example if I try to run "hl" with "psrun -c
papi3_mflops.xml ./hl"

I am getting the following error;


hwpc.c:306 : Initialized global data
hwpc.c:315 : Signal handling off.
hwpc.c:2288 : Configuration file set to papi3_mflops.xml
hwpc.c:2333 : Attempting to parse "papi3_mflops.xml"
xmlparse.c:301 : XML parser: parsing expected XML document element
ps_hwpc_eventlist (class = "PAPI")
xmlparse.c:237 : XML parser: found preset: "PAPI_FP_OPS"
xmlparse.c:237 : XML parser: found preset: "PAPI_TOT_CYC"
xmlparse.c:490 : XML parser: found 2 events:
        PAPI_FP_OPS
        PAPI_TOT_CYC
hwpc.c:323 : Configuration file read
hwpc.c:325 : Read 2 events
hwpc.c:2158 : Initializing PAPI class...
hwpc.c:354 : Performance software supports 32 events max.
hwpc.c:447 : Calling package init routine
hwpc-papi.c:319 : PAPI/PerfSuite version check failed (require at least 3,
got -3)
libpsrun fatal error: error reported by performance software layer

Thanks in advance,

Regards,

Abdullah

On Tue, Jun 24, 2008 at 4:33 PM, Abdullah Kayi <ap...@gm...> wrote:

> Hi Rick,
>
> You can ignore number #1 as I found the problem with my configuration. Any
> help on the remaining issues are appreciated.
>
> Regards,
>
> Abdullah
>
>
> On Tue, Jun 24, 2008 at 3:07 PM, Abdullah Kayi <ap...@gm...> wrote:
>
>> Hi Rick,
>>
>> I made a clean install. Here is some info from the config.log and I solved
>> some of the problems that I had with the examples.
>>
>> ./configure --prefix=/wrkdir/kayi/PERFSUITE_DIR
>> --with-papi=/u2/wes/PET_HOME/pkgs/papi-3.5.0
>> --with-tdom=/wrkdir/kayi/TDOM_DIR PTHREAD_LIBS=-lpthread --enable-mpi
>> MPICPPFLAGS=-I/usr/local/pkg/voltairempi/voltairempi-S-1/mpi.pathcc.rsh/include
>> --enable-debug
>>
>> ## --------- ##
>> ## Platform. ##
>> ## --------- ##
>>
>> uname -m = x86_64
>> uname -r = 2.6.5-7.276-smp
>> uname -s = Linux
>>
>> However, I still have some questions and issues.
>>
>> 1- First of all, the "psprocess" wasn't installed although I have the tcl
>> and tdom in my system. So am I missing any requirements for that to
>> install?  how can i parse the xml output otherwise?
>>
>> 2- If I run the "cpi" with "psrun -c papi_profile_l2tcm.xml ./cpi" I am
>> getting the following error with PS_DEBUG=10
>>
>> hwpc-papi.c:750 : Initializing profiling in PAPI layer
>> hwpc-papi.c:152 : Entering setup_counters_internal
>> hwpc-papi.c:193 : Eventset address: 0x504070
>> hwpc-papi.c:209 : Created an eventset (0).
>> hwpc-papi.c:228 : Domain of eventset: unknown (0)
>> hwpc-papi.c:270 : Added 1 events to eventset 0
>> hwpc-papi.c:777 : Profiling using PAPI event code 80000007 (PAPI_L2_TCM)
>> hwpc-papi.c:789 : PAPI_sprofil_t structure allocated at 0x504bc0
>> hwpc-papi.c:810 : Calling PAPI_sprofil with arguments
>> sprofils  = 0x504bc0
>> nmaps     = 11
>> *eventset = 0
>> eventcode = 80000007
>> threshold = 100000
>> flags     = 0
>> hwpc-papi.c:818 : PAPI_sprofil error: PAPI_EINVAL
>> hwpc.c:602 : Error return from init_profiling
>> libpsrun fatal error: error reported by performance software layer
>>
>> If I run the mpirun version I am also getting the same error. So how can I
>> enable the profiling runs?
>>
>> 3- if I use psrun with mpirun (or charmrun) on a Charm++ code ChaNGa I am
>> getting the following error:
>>
>>
>> PAPI/PerfSuite version check failed (require at least 3, got -3)
>> libpsrun fatal error: error reported by performance software layer
>>  [psrun: Parent process forked, PID=1329]
>> [psrun: Child process forked, PID=2976]
>> hwpc-papi.c:718 : PAPI_shutdown called
>> hwpc.c:1046 : ps_hwpc_shutdown: returned from destroy counters
>> hwpc.c:1733 : No PID file scheduled for deletion.
>> fork(): _psrun_hwpc_shutdown returned 0
>> [psrun: Parent process forked, PID=2976]
>> [psrun: Parent process forked, PID=2976]
>> [psrun: Child process forked, PID=2977]
>> [psrun: Child process forked, PID=2978]
>>
>>
>> Do you have any idea what might be wrong in these different issues? Thanks
>> so much for your help and consideration.
>>
>> Regards,
>>
>> Abdullah
>>
>
>

Re: [PerfSuite-users] Problems running examples

[Rick K. <rk...@nc...>]

Abdullah -

It would appear that all of the errors you are encountering are related 
to use of PAPI calls within PerfSuite, so I would like to step back and 
make sure that the underlying software is functioning properly.  To help 
do that, I have some basic questions:

Can you clarify what processor you are using?  The config.log output you 
sent indicates it is an x86_64 machine, but something more specific 
would be helpful.   "cat /proc/cpuinfo" or the output of the psinv 
command may help.

Is this a perfctr or perfmon-patched system?

I'm also guessing you are using PAPI version 3.5.0, based on the 
pathname you provided via --with-papi on the configure command line.  
Unfortunately, I'm not sure if that is the official PAPI 3.5.0 release 
or possibly a version updated later via CVS.

Did you run the PAPI test suite when you built the library ("make 
fulltest" within the PAPI src subdirectory)?  If so, did all tests pass 
as expected?

Did you run the PerfSuite test suite as well ("make check")?  How about 
the outcome of those tests?  These are helpful tests to run as they help 
ensure that PAPI is functioning correctly outside of PerfSuite, 
particularly the test cases in the "libpshwpc" group.

If any of the above failed, then it would be helpful to see the relevant 
output before considering the examples as well as anything more 
substantial (like Charm++ runs).

Rick

Abdullah Kayi wrote:
> And on top of these, for example if I try to run "hl" with "psrun -c 
> papi3_mflops.xml ./hl"
>
> I am getting the following error;
>
>
> hwpc.c:306 : Initialized global data
> hwpc.c:315 : Signal handling off.
> hwpc.c:2288 : Configuration file set to papi3_mflops.xml
> hwpc.c:2333 : Attempting to parse "papi3_mflops.xml"
> xmlparse.c:301 : XML parser: parsing expected XML document element 
> ps_hwpc_eventlist (class = "PAPI")
> xmlparse.c:237 : XML parser: found preset: "PAPI_FP_OPS"
> xmlparse.c:237 : XML parser: found preset: "PAPI_TOT_CYC"
> xmlparse.c:490 : XML parser: found 2 events:
>         PAPI_FP_OPS
>         PAPI_TOT_CYC
> hwpc.c:323 : Configuration file read
> hwpc.c:325 : Read 2 events
> hwpc.c:2158 : Initializing PAPI class...
> hwpc.c:354 : Performance software supports 32 events max.
> hwpc.c:447 : Calling package init routine
> hwpc-papi.c:319 : PAPI/PerfSuite version check failed (require at 
> least 3, got -3)
> libpsrun fatal error: error reported by performance software layer
>
> Thanks in advance,
>
> Regards,
>
> Abdullah
>
> On Tue, Jun 24, 2008 at 4:33 PM, Abdullah Kayi <ap...@gm... 
> <mailto:ap...@gm...>> wrote:
>
>     Hi Rick,
>
>     You can ignore number #1 as I found the problem with my
>     configuration. Any help on the remaining issues are appreciated.
>
>     Regards,
>
>     Abdullah
>
>
>     On Tue, Jun 24, 2008 at 3:07 PM, Abdullah Kayi <ap...@gm...
>     <mailto:ap...@gm...>> wrote:
>
>         Hi Rick,
>
>         I made a clean install. Here is some info from the config.log
>         and I solved some of the problems that I had with the examples.
>
>         ./configure --prefix=/wrkdir/kayi/PERFSUITE_DIR
>         --with-papi=/u2/wes/PET_HOME/pkgs/papi-3.5.0
>         --with-tdom=/wrkdir/kayi/TDOM_DIR PTHREAD_LIBS=-lpthread
>         --enable-mpi
>         MPICPPFLAGS=-I/usr/local/pkg/voltairempi/voltairempi-S-1/mpi.pathcc.rsh/include
>         --enable-debug
>
>         ## --------- ##
>         ## Platform. ##
>         ## --------- ##
>
>         uname -m = x86_64
>         uname -r = 2.6.5-7.276-smp
>         uname -s = Linux
>
>         However, I still have some questions and issues.
>
>         1- First of all, the "psprocess" wasn't installed although I
>         have the tcl and tdom in my system. So am I missing any
>         requirements for that to install?  how can i parse the xml
>         output otherwise?
>
>         2- If I run the "cpi" with "psrun -c papi_profile_l2tcm.xml
>         ./cpi" I am getting the following error with PS_DEBUG=10
>
>         hwpc-papi.c:750 : Initializing profiling in PAPI layer
>         hwpc-papi.c:152 : Entering setup_counters_internal
>         hwpc-papi.c:193 : Eventset address: 0x504070
>         hwpc-papi.c:209 : Created an eventset (0).
>         hwpc-papi.c:228 : Domain of eventset: unknown (0)
>         hwpc-papi.c:270 : Added 1 events to eventset 0
>         hwpc-papi.c:777 : Profiling using PAPI event code 80000007
>         (PAPI_L2_TCM)
>         hwpc-papi.c:789 : PAPI_sprofil_t structure allocated at 0x504bc0
>         hwpc-papi.c:810 : Calling PAPI_sprofil with arguments
>         sprofils  = 0x504bc0
>         nmaps     = 11
>         *eventset = 0
>         eventcode = 80000007
>         threshold = 100000
>         flags     = 0
>         hwpc-papi.c:818 : PAPI_sprofil error: PAPI_EINVAL
>         hwpc.c:602 : Error return from init_profiling
>
>         libpsrun fatal error: error reported by performance software layer
>
>         If I run the mpirun version I am also getting the same error.
>         So how can I enable the profiling runs?
>
>         3- if I use psrun with mpirun (or charmrun) on a Charm++ code
>         ChaNGa I am getting the following error:
>
>          
>         PAPI/PerfSuite version check failed (require at least 3, got -3)
>
>         libpsrun fatal error: error reported by performance software layer
>         [psrun: Parent process forked, PID=1329]
>         [psrun: Child process forked, PID=2976]
>         hwpc-papi.c:718 : PAPI_shutdown called
>         hwpc.c:1046 : ps_hwpc_shutdown: returned from destroy counters
>         hwpc.c:1733 : No PID file scheduled for deletion.
>         fork(): _psrun_hwpc_shutdown returned 0
>         [psrun: Parent process forked, PID=2976]
>         [psrun: Parent process forked, PID=2976]
>         [psrun: Child process forked, PID=2977]
>         [psrun: Child process forked, PID=2978]
>
>
>         Do you have any idea what might be wrong in these different
>         issues? Thanks so much for your help and consideration.
>
>         Regards,
>
>         Abdullah
>
>
>
> ------------------------------------------------------------------------
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> ------------------------------------------------------------------------
>
> _______________________________________________
> PerfSuite-users mailing list
> Per...@li...
> https://lists.sourceforge.net/lists/listinfo/perfsuite-users
>   

Re: [PerfSuite-users] Problems running examples

[Abdullah K. <ap...@gm...>]

Hi Rick,

I made a clean install. Here is some info from the config.log and I solved
some of the problems that I had with the examples.

./configure --prefix=/wrkdir/kayi/PERFSUITE_DIR
--with-papi=/u2/wes/PET_HOME/pkgs/papi-3.5.0
--with-tdom=/wrkdir/kayi/TDOM_DIR PTHREAD_LIBS=-lpthread --enable-mpi
MPICPPFLAGS=-I/usr/local/pkg/voltairempi/voltairempi-S-1/mpi.pathcc.rsh/include
--enable-debug

## --------- ##
## Platform. ##
## --------- ##

uname -m = x86_64
uname -r = 2.6.5-7.276-smp
uname -s = Linux

However, I still have some questions and issues.

1- First of all, the "psprocess" wasn't installed although I have the tcl
and tdom in my system. So am I missing any requirements for that to
install?  how can i parse the xml output otherwise?

2- If I run the "cpi" with "psrun -c papi_profile_l2tcm.xml ./cpi" I am
getting the following error with PS_DEBUG=10

hwpc-papi.c:750 : Initializing profiling in PAPI layer
hwpc-papi.c:152 : Entering setup_counters_internal
hwpc-papi.c:193 : Eventset address: 0x504070
hwpc-papi.c:209 : Created an eventset (0).
hwpc-papi.c:228 : Domain of eventset: unknown (0)
hwpc-papi.c:270 : Added 1 events to eventset 0
hwpc-papi.c:777 : Profiling using PAPI event code 80000007 (PAPI_L2_TCM)
hwpc-papi.c:789 : PAPI_sprofil_t structure allocated at 0x504bc0
hwpc-papi.c:810 : Calling PAPI_sprofil with arguments
sprofils  = 0x504bc0
nmaps     = 11
*eventset = 0
eventcode = 80000007
threshold = 100000
flags     = 0
hwpc-papi.c:818 : PAPI_sprofil error: PAPI_EINVAL
hwpc.c:602 : Error return from init_profiling
libpsrun fatal error: error reported by performance software layer

If I run the mpirun version I am also getting the same error. So how can I
enable the profiling runs?

3- if I use psrun with mpirun (or charmrun) on a Charm++ code ChaNGa I am
getting the following error:


PAPI/PerfSuite version check failed (require at least 3, got -3)
libpsrun fatal error: error reported by performance software layer
[psrun: Parent process forked, PID=1329]
[psrun: Child process forked, PID=2976]
hwpc-papi.c:718 : PAPI_shutdown called
hwpc.c:1046 : ps_hwpc_shutdown: returned from destroy counters
hwpc.c:1733 : No PID file scheduled for deletion.
fork(): _psrun_hwpc_shutdown returned 0
[psrun: Parent process forked, PID=2976]
[psrun: Parent process forked, PID=2976]
[psrun: Child process forked, PID=2977]
[psrun: Child process forked, PID=2978]


Do you have any idea what might be wrong in these different issues? Thanks
so much for your help and consideration.

Regards,

Abdullah

Re: [PerfSuite-users] Problems running examples

[Abdullah K. <ap...@gm...>]

Hi Rick,

You can ignore number #1 as I found the problem with my configuration. Any
help on the remaining issues are appreciated.

Regards,

Abdullah

On Tue, Jun 24, 2008 at 3:07 PM, Abdullah Kayi <ap...@gm...> wrote:

> Hi Rick,
>
> I made a clean install. Here is some info from the config.log and I solved
> some of the problems that I had with the examples.
>
> ./configure --prefix=/wrkdir/kayi/PERFSUITE_DIR
> --with-papi=/u2/wes/PET_HOME/pkgs/papi-3.5.0
> --with-tdom=/wrkdir/kayi/TDOM_DIR PTHREAD_LIBS=-lpthread --enable-mpi
> MPICPPFLAGS=-I/usr/local/pkg/voltairempi/voltairempi-S-1/mpi.pathcc.rsh/include
> --enable-debug
>
> ## --------- ##
> ## Platform. ##
> ## --------- ##
>
> uname -m = x86_64
> uname -r = 2.6.5-7.276-smp
> uname -s = Linux
>
> However, I still have some questions and issues.
>
> 1- First of all, the "psprocess" wasn't installed although I have the tcl
> and tdom in my system. So am I missing any requirements for that to
> install?  how can i parse the xml output otherwise?
>
> 2- If I run the "cpi" with "psrun -c papi_profile_l2tcm.xml ./cpi" I am
> getting the following error with PS_DEBUG=10
>
> hwpc-papi.c:750 : Initializing profiling in PAPI layer
> hwpc-papi.c:152 : Entering setup_counters_internal
> hwpc-papi.c:193 : Eventset address: 0x504070
> hwpc-papi.c:209 : Created an eventset (0).
> hwpc-papi.c:228 : Domain of eventset: unknown (0)
> hwpc-papi.c:270 : Added 1 events to eventset 0
> hwpc-papi.c:777 : Profiling using PAPI event code 80000007 (PAPI_L2_TCM)
> hwpc-papi.c:789 : PAPI_sprofil_t structure allocated at 0x504bc0
> hwpc-papi.c:810 : Calling PAPI_sprofil with arguments
> sprofils  = 0x504bc0
> nmaps     = 11
> *eventset = 0
> eventcode = 80000007
> threshold = 100000
> flags     = 0
> hwpc-papi.c:818 : PAPI_sprofil error: PAPI_EINVAL
> hwpc.c:602 : Error return from init_profiling
> libpsrun fatal error: error reported by performance software layer
>
> If I run the mpirun version I am also getting the same error. So how can I
> enable the profiling runs?
>
> 3- if I use psrun with mpirun (or charmrun) on a Charm++ code ChaNGa I am
> getting the following error:
>
>
> PAPI/PerfSuite version check failed (require at least 3, got -3)
> libpsrun fatal error: error reported by performance software layer
> [psrun: Parent process forked, PID=1329]
> [psrun: Child process forked, PID=2976]
> hwpc-papi.c:718 : PAPI_shutdown called
> hwpc.c:1046 : ps_hwpc_shutdown: returned from destroy counters
> hwpc.c:1733 : No PID file scheduled for deletion.
> fork(): _psrun_hwpc_shutdown returned 0
> [psrun: Parent process forked, PID=2976]
> [psrun: Parent process forked, PID=2976]
> [psrun: Child process forked, PID=2977]
> [psrun: Child process forked, PID=2978]
>
>
> Do you have any idea what might be wrong in these different issues? Thanks
> so much for your help and consideration.
>
> Regards,
>
> Abdullah
>

Re: [PerfSuite-users] Problems running examples

[Rick K. <rk...@nc...>]

Abdullah -

Please try to use the SourceForge mailing list for questions/comments 
about PerfSuite.  That allows traffic to be archived that may benefit 
other people who have similar questions by browsing earlier postings.  
It also helps me keep my NCSA (work) email distinct, which in turn keeps 
my employer happy :)

As far as your question goes: much is dependent on the particular MPI 
implementation and how it manages processes/tasks.  For example, MPICH 
using the p4 device won't work with psrun at all (this was discussed in 
a perfsuite-users posting a long time ago).   You may have to experiment 
with your own favorite MPI.   Charm++ is another case that can work in 
some cases, others not.

The driver for the processor's performance monitoring unit (perfmon or 
perfctr for Linux) takes care of virtualizing the hardware counter data 
on a per-thread basis, so if your application spawns separate processes 
that use multiple cores, they should each generate an output XML 
document.  For multithreaded applications that use POSIX threads (either 
directly or through a Pthread-based interface like many OpenMP 
implementations), the option "-p" that psrun accepts will trap thread 
creation and generate per-thread performance data at thread exit.

If you are instead looking for a system-wide monitoring tool that would 
track all cores over a given period of time, that is not what PerfSuite 
was intended to do, but there are other tools available that fill that 
space.

Rick

Abdullah Kayi wrote:
> Hi Rick,
>
> I am pretty much getting the reasonable outputs from the tests that I 
> tried. So now, I am trying to use "psrun" with mpirun I couldn't find 
> an example for that other then cpi-mpi which works with libpshwpc. So 
> I  am not sure what is needed for psrun to work for mpi or charm++ 
> directly. Also, how can I get theoutput from all the cores? Do you 
> have any ideas on these? Thanks so much for your help.
>
> Regards,
>
> Abdullah
>
> On Wed, Jun 25, 2008 at 5:39 PM, Abdullah Kayi <ap...@gm... 
> <mailto:ap...@gm...>> wrote:
>
>     Ooops my bad, I shouldn't have used "psrun", now I am getting the
>     correct results. I will let you know about the ChaNGa results.
>
>     On Wed, Jun 25, 2008 at 4:14 PM, Abdullah Kayi <ap...@gm...
>     <mailto:ap...@gm...>> wrote:
>
>         Hey Rick,
>
>         I am running the "hl" example like this;
>
>
>         psrun -c papi3_mflops.xml ./hl
>
>         I am finally getting the xml output  fine but when I look at
>         the results.
>
>         Index
>         Description                                                             
>         Counter Value
>         ============================================================================================
>             1 Floating point
>         operations........................................                   
>         3
>             2 Total
>         cycles.....................................................              
>         629519
>
>         Event Index
>         ============================================================================================
>             1: PAPI_FP_OPS         2: PAPI_TOT_CYC
>
>         There should be something wrong how come there were only 3 FP
>         operations in that output, I am using the "hl.c" as it is. Do
>         you have any ideas on this?
>
>         Regards,
>
>         Abdullah
>
>
>