Since this is distributed programming we are talking about, I have never found any profiler or even debugger to be useful.
Instead, I have always had to drop back down to the simplest forms of trouble shooting.
Which is writing off log messages to files.
It is easy to track the sequence of events and the time duration of each module by writing out time of day before and after ever major operation within the provider.

If the SAN is large, you may want to check whether you are using blocking IO or using asynchronous methods instead.


----- Original Message ----
From: sumeet gandhare <sumeetgandhare@gmail.com>
To: oprofile-list@lists.sourceforge.net
Cc: david_dillard@symantec.com; kirk_augustin@yahoo.com; maynardj@us.ibm.com; pegasus-l@openpegasus.org
Sent: Thursday, July 24, 2008 4:28:04 AM
Subject: Re: how to profile Pegasus CimServer Provider shared library??

Hi All,

I am working on a CIMPLE based Provider for Emulex HBA card for OpenPegaus CIMOM.

Please could you let me know if there is any tool to profile Pegasus cimserver with its shared libraries for provider. I tried oprofile, but the results I got are far from satisfactory.

I need to do this profiling so as to figure out where my provider is taking time. I suspect the underlying stack which obtains the SAN data from the SAN network like LUNs/Targets etc to be causing the delay but I want to confirm that.

In order to use gprof , i enabled "-pg " and "-lc_p" options with the environment variables, PEGASUS_EXTRA_C_FLAGS and PEGASUS_EXTRA_LINK_FLAGS.

On doing the build of cimserver with above options, the cimserver did not start up correctly (got hung) and some times it even gave segmentation fault. For your information I am using Pegasus-2.6.0 version on RHEL 4 U 5

My current Provider Library Profiling results got from Oprofile. Below BuildCimpleInfo is the function which invokes BuildFcInfo but from the results it seems the time for BuildCimpleInfo does not include time for BuildCimpleInfo.

Please kindly let me know what could be wrong.


Appreciate your help!!

Thanks and Regards,
Sumeet

[root@bglab-HSP3 src]# opreport --merge=all --symbols /root/sumeet/pegasus/lib/libELXHBA_CMPI_Provider.so
CPU: Core 2, speed 2333.51 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 6000
samples  %        symbol name
50       14.7059  cimple::BuildFcInfo(cimple::Array<cimple::Smi_Hba>*)
31        9.1176  cimple::Array<cimple::Instance*>::size() const
23        6.7647  cimple::CLogger::write(char const*, unsigned int, char const*, char const*)
17        5.0000  cimple::convertWWNToStr(char*, HBA_wwn const&)
15        4.4118  cimple::Array<cimple::Instance*>::operator[](unsigned int)
14        4.1176  cimple::CHbaApi::HBA_SendReadCapacity_t(unsigned int, HBA_wwn, long long, void*, unsigned int, void*, unsigned int)
14        4.1176  cimple::CHbaApi::HBA_SendScsiInquiry_t(unsigned int, HBA_wwn, long long, unsigned char, unsigned int, void*, unsigned int, void*, unsigned int)
13        3.8235  anonymous symbol from section .plt
13        3.8235  cimple::Array<cimple::Instance*>::data()
12        3.5294  cimple::String::String()
11        3.2353  cimple::ProviderAgent::BuildCimpleInfo()
9         2.6471  cimple::String::c_str() const
7         2.0588  cimple::Array<cimple::SMI_FcpSCSITarget>::append(cimple::SMI_FcpSCSITarget const&)
7         2.0588  cimple::Auto_Mutex::Auto_Mutex(cimple::Mutex&)
7         2.0588  cimple::CLogger::indentIncrease()
6         1.7647  cimple::Array<cimple::SMI_PortInfo>::operator[](unsigned int)
6         1.7647  cimple::__Array_Traits_Factory<cimple::SMI_FcpSCSITarget>::_construct(void*, void const*)
5         1.4706  cimple::CLogger::indentDecrease()
5         1.4706  cimple::__Array_Traits_Factory_CIM<cimple::String, 12>::traits()
4         1.1765  cimple::Array<unsigned short>::size() const
4         1.1765  cimple::Enum_Instances_Handler<cimple::ELXHBA_StorageVolume>::handle(cimple::ELXHBA_StorageVolume*)
3         0.8824  cimple::Array<cimple::String>::Array()
3         0.8824  cimple::Array<cimple::String>::size() const
3         0.8824  cimple::Array<cimple::String>::~Array()
3         0.8824  cimple::Array<unsigned short>::data() const
3         0.8824  cimple::Auto_Mutex::~Auto_Mutex()
3         0.8824  cimple::ELXHBA_StorageVolume::clone() const
3         0.8824  cimple::ELXHBA_StorageVolume_Provider::enum_instances(cimple::ELXHBA_StorageVolume const*, cimple::Enum_Instances_Handler<cimple::ELXHBA_StorageVolume>*)
3         0.8824  cimple::Instance_Map<cimple::ELXHBA_StorageVolume>::operator[](unsigned int)
3         0.8824  cimple::String::operator=(char const*)
3         0.8824  cimple::String::operator=(cimple::String const&)
3         0.8824  cimple::convertByteWWNToStr(char*, unsigned char*)
2         0.5882  cimple::Array<cimple::SMI_FcpSCSITarget>::data()
2         0.5882  cimple::Array<cimple::SMI_FcpSCSITarget>::operator[](unsigned int)
2         0.5882  cimple::Array<cimple::SMI_PortInfo>::data()
2         0.5882  cimple::Array<cimple::String>::operator=(cimple::Array<cimple::String> const&)
2         0.5882  cimple::Array<unsigned short>::Array(cimple::Array<unsigned short> const&)
2         0.5882  cimple::Array<unsigned short>::~Array()
2         0.5882  cimple::ELXHBA_SCSIProtocolEndpoint::clone() const
2         0.5882  cimple::Property<unsigned int>::set(unsigned int const&)
2         0.5882  cimple::assign(hba_AdapterAttributes&, hba_AdapterAttributes const&)
1         0.2941  cimple::Array<cimple::SMI_FcpSCSITarget>::Array(cimple::Array<cimple::SMI_FcpSCSITarget> const&)
1         0.2941  cimple::Array<cimple::SMI_FcpSCSITarget>::size() const
1         0.2941  cimple::Array<cimple::Smi_Hba>::data()
1         0.2941  cimple::Array<cimple::Smi_Hba>::operator[](unsigned int)
1         0.2941  cimple::Array<cimple::String>::operator[](unsigned int) const
1         0.2941  cimple::CHbaApi::HBA_CloseAdapter_t(unsigned int)
1         0.2941  cimple::CHbaApi::HBA_GetAdapterName_t(unsigned int, char*)
1         0.2941  cimple::CHbaApi::HBA_GetFcpTargetMapping_t(unsigned int, HBA_wwn, HBA_FCPTargetMapping**)
1         0.2941  cimple::ELXHBA_MethodsWrapper::HBACMD_GetDriverParam(HBA_NAME, bool)
1         0.2941  cimple::ELXHBA_MethodsWrapper::str2HBAName(HBA_NAME*, char const*) const
1         0.2941  cimple::ELXHBA_SCSIInitiatorTargetStorageVolumePath::create(bool)
1         0.2941  cimple::ELXHBA_StorageVolumeSystemDevice::create(bool)
1         0.2941  cimple::Instance_Map<cimple::ELXHBA_SCSIInitiatorTargetStorageVolumePath>::insert(cimple::ELXHBA_SCSIInitiatorTargetStorageVolumePath*)
1         0.2941  cimple::SwapInteger(unsigned char*, unsigned char*, unsigned char)
1         0.2941  cimple::__Array_Traits_Factory<cimple::Smi_Hba>::_construct(void*, void const*)
1         0.2941  cimple::assign(HBA_PortAttributes&, HBA_PortAttributes const&)
[root@bglab-HSP3 src]#


On Thu, Jul 24, 2008 at 4:20 AM, David Dillard <david_dillard@symantec.com> wrote:
Most profilers do not do emulation.  They either perform sampling or add instrumentation to the program.
 
For a good explanation of how profilers work see http://www.freescale.com/files/soft_dev_tools/doc/white_paper/CWCODEWP.pdf
 


From: Kirk Augustin [mailto:kirk_augustin@yahoo.com]
Sent: Wednesday, July 23, 2008 6:26 PM
To: sumeet gandhare; pegasus-l@openpegasus.org
Subject: Re: how to profile Pegasus CimServer Provider shared library??

Generally when you run a profiler, that is the actual program that is running.
You are not really running CimServer, but a variant of oprofile that is grabbing the CimServer executable code and emulating it line by line.
Just adding passive debug symbols to the build itself would not get anything to really happen.
So if there is a problem getting exactly what you want out of oprofile, what you need is advice on how to use oprofile, not Pegasus specific.
But the problem is probably that you need to tell oprofile what shared libraries you want to profile, and where the source for them is, so that it can emulate them line by line, as well.

----- Original Message ----
From: sumeet gandhare <sumeetgandhare@gmail.com>
To: pegasus-l@openpegasus.org
Sent: Wednesday, July 23, 2008 10:23:29 AM
Subject: how to profile Pegasus CimServer Provider shared library??

Hi,
 
I am trying to use oprofile to profile CimServer which dynamically loads shared libraries. From the opreport/opannotate output I am not able to see any profile information regarding the shared libraries function which I actually wish to profile.
 
Please could you let me know what special things I need to do get the profile done. Please note that I have built cimserver process (which is being profiled) and the dlls with debug information.
 
 
opcontrol -i /root/sumeet/pegasus/bin/cimserver
opcontrol -separate=lib
opcontrol -no-vmlinux
opcontrol --event=CPU_CLK_UNHALTED:50000
 
opcontrol --start
 
/root/sumeet/pegasus/bin/cimserver
 
(invoke cimcli to get the storage volume related information from the cimserver .. This would in turn invoke the code from the shared library)
 
opcontrol --dump
 
opcontrol --stop
opcontrol --shutdown
 
opreport -l /root/sumeet/pegasus/bin/cimserver
 
 
Thanks and Regards,
Sumeet R. Gandhare