Re: [PerfSuite-users] How to Profile a daemon process

[Rick K. <rk...@nc...>]

Dawei,

You may be interested in a paper that was given at the 2005 Linux Clusters=
=20
Institute conference in which the step-by-step operation of the psrun=20
command is described.  An electronic copy of the paper can be found here:

http://perfsuite.ncsa.uiuc.edu/publications/LCI-2005.pdf

One of the most important points is that the psrun command itself does not=
=20
participate in the monitoring nor does it write out any of the XML=20
documents that result.  psrun's only purpose is to interpret command line=
=20
arguments and set up the runtime environment for the command that is being=
=20
monitored.  Strictly speaking, the psrun command does not need to be used=
=20
at all to do performance monitoring if one sets up the environment "by=20
hand" properly.

Regarding the XML documents attached in your message, I can see that they=
=20
are monitoring several different executables: timestensubd, ttcserver, and=
=20
ttDaemonAdmin, and it would appear that they correspond to the parent=20
process as well as spawned processes/threads, as you describe.

However, they appear to have consumed very little CPU time, which could=20
certainly be the case for daemon-type threads that may primarily be=20
waiting on I/O.  Additionally, they are not profiles but are documents=20
gathered in aggregate event count mode which by default uses counter=20
multiplexing from PAPI.  If a process does not use much CPU time, there=20
will not be enough time for counts to accumulate across all the events=20
selected and the resulting numbers will be zero.  You can also see this=20
behavior if you use psrun with a simple, short-running command like "ls":

   psrun ls

One way to test this is to run psrun with a configuration file that does=20
not require multiplexing.  You can do this by making a copy of the default=
=20
configuration file ("psrun -h" will show you its location) and removing=20
all but one of the events listed (e.g. PAPI_TOT_CYC), then rerun psrun
and see if the event counts are non-zero:

   setenv PS_HWPC_CONFIG your_new_configuration.xml
   psrun ls

On a separate matter: if you are indeed interested in gathering a profile=
=20
and not aggregate events, you may instead want to use a profiling=20
configuration, such as "papi_profile_cycles.xml" (contained in the=20
PerfSuite distribution).  Again, however, if a process does not use much=20
CPU time during its lifetime, you may not get meaningful results from=20
profiling since there would not be enough time to gather samples from the=
=20
statistical profiling method that PerfSuite uses.  Keep in mind also that=
=20
CPU time is different than wallclock time in that a process may be in=20
existence for a significant amount of wallclock time but still use very=20
little CPU time.

Rick

On Tue, 9 Jan 2007, liudawei wrote:

> Thank you for your reply.I have tried according your suggestion,but probl=
em does not resolved.I think I don not clearly described the problem, I don=
not think it is the programe donot consume enough cpu tim. the reason why t=
he profiled results is zero may be explain via the example here:
>
> consider the following command
>
> psrun -f p1.exe
>
> In fact, psrun will create a thread (say, t1) to execute p1.exe .
> but if the p1 is a type of daemon process. what will happen?
> t1 will create a new thread ( say,t2), when the t2 is created  successful=
ly, t2 will kill  t1,this t2 just is the daemon. (This is how the daemon ty=
pe process created.)
> but when t1 is killed, psrun will finish to collect CPU parameters right =
now.
> is this right?
>
> -------------------------------------------------------------------------=
--------------------------------------
> real application scenario:
>
> ttDaemonAdmin is a daemon type process,which is to start a database serve=
r,I just want to profile it via psrun.so I use the following command:
>
> psrun -f ttDaemonAdmin -start
>
> here,ttDaemonAdmin is real type of daemon process, when it is started, I =
will connect  it  from a remote client and send some query to server. This =
ttDaemonAdmin will received my request from the client and produce a subdem=
on process (is real type of daemon process too)  to do a few works in the s=
erver. when i finished my connection to the server. I will stop the ttDaemo=
nAdmin. so the psrun will finished too. Then ,I will  got a few of xml file=
s. but after processed by psprocess. the all results is zero.
>
> Attached the profile results,there are six files.
> the ttDaemonAdmin.15261.Node1.xml is the results correspond to the ttDaem=
onAdmin process.The other five xml files is the child process produced by d=
dDaemonAdmin, they are all the real type of daemon process too.
>
> anxious for how to profile these type of process?
>
> Best regards
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Rick Kufrin <rk...@nc...> =D0=B4=B5=C0=A3=BA Dawei,
>
> One thing you might consider trying is to replace the executable that
> corresponds to the daemon program with a wrapper script that invokes psru=
n
> on the true daemon program.  For example, rename Child.exe to
> Child.exe.orig and create a new file named Child.exe that is a shell
> script that runs the command "psrun Child.exe.orig", passing along any
> arguments necessary for Child.exe.  This would be the quickest way to see
> if things might work as a first cut.
>
> Also, psrun does not actually do any performance monitoring itself nor
> does it write out the XML result documents; this is done through librarie=
s
> that are "inserted" into the program that is being monitored.
>
> The usual reason for zeroes being reported are that the program does not
> consume enough CPU time to collect any data, and it sounds like this is
> what you may be seeing here.
>
> Rick
>
> On Sun, 7 Jan 2007, liudawei wrote:
>
>> Hi:
>>
>> I have encountered a problem when I use psrun to profile a Demon process=
 and its child process is also Demon process. The example is here.
>>
>> Psrun -f Daemon.exe
>>
>> Iet me describe the how this  Daemon work, I want to profile the Daemon =
and its child process. here, all the child process is daemon process too. T=
he problem I encountered is that all the results(xml file produced by psrun=
) is zero. I try to read the source code of the psrun. I guess that psrun w=
ill fork a process to execute the program which is profiled (here is the Da=
emon.exe). But as we know that if the program to be profiled is daemon proc=
ess, I think when the daemon process is created successfully, it will kill =
his father process, however, psrun is just waiting for this signal to finis=
hed the profile. so, if the Daemon is start,the psrun will stop right now. =
so, all the results is zero.
>>
>> Can any one give some suggestion? or give a patch to psrun, so let psrun=
 can monitor daemon type program.
>>
>> You can download the Oracle TimesTen , then use psrun to verify the prob=
lem I encountred.
>>
>>
>> Waiting for a reply.
>>
>> __________________________________________________
>> =B8=CF=BF=EC=D7=A2=B2=E1=D1=C5=BB=A2=B3=AC=B4=F3=C8=DD=C1=BF=C3=E2=B7=D1=
=D3=CA=CF=E4?
>> http://cn.mail.yahoo.com
>
> __________________________________________________
> =B8=CF=BF=EC=D7=A2=B2=E1=D1=C5=BB=A2=B3=AC=B4=F3=C8=DD=C1=BF=C3=E2=B7=D1=
=D3=CA=CF=E4?
> http://cn.mail.yahoo.com