From: Santos, J. R. G (J. R. Santos) <jos...@hp...> - 2005-04-11 22:41:11
|
>> -----Original Message----- >> From: xen...@li...=20 >> [mailto:xen...@li...] On Behalf Of=20 >> William Cohen >> Sent: Monday, April 11, 2005 1:27 PM >> To: xen...@li... >> Cc: oprofile-list >> Subject: [Xen-devel] [PATCH] Xenoprof: Enabling performance=20 >> profiling in Xen >>=20 >>=20 >> I took a quick look at the patches for supporting profiling=20 >> in Xen and I=20 >> have some questions about the patches. >>=20 >> -Do all domains have to use the same setup for the performance=20 >> monitoring counters? Or is there some virtualization of the=20 >> performance=20 >> counters? >>=20 Right now, there is no virtualization of the hardware performance counters. Our goal was to have a system wide profiling capability that could enable us understand better the behavior of Xen. Currently, only one profiling session can be run at a time and the=20 initiator domain for that session (typicaly domain 0) is the=20 domain that specify the performance events to be monitored. All other profiled domains just collect and decode samples of the same events. It would be nice to have hardware performance counter virtualization but it is not on our priority list. >> -How is this going to interact with other performance monitoring=20 >> infrastructure such as perfctr and perfmon? Or is the design=20 >> going to=20 >> need to be significantly revised for other performance monitoring=20 >> interfaces? I like OProfile as much as anyone else, but I=20 >> would like to=20 >> see the Xen support allow the other interfaces to=20 >> performance monitoring=20 >> hardware to work. >>=20 Our focus was mostly on enabling Oprofile to work on Xen.=20 Significant changes will be required for other performance=20 monitoring infraestructure, although some of the code could=20 be re-used. We are not planning to work on these in the near future.=20 >> -It appears that the multiple samples can be queued up in the=20 >> hypervisor. In the OProfile kernel support when a process=20 >> exits there is=20 >> a flush of the per cpu buffers to make sure that the VMAs=20 >> are mapped to=20 >> files and offsets before the the mappings are lost. Should=20 >> there be a=20 >> flush of the samples from the hypervisor to make sure that=20 >> they are read=20 >> out before the process exits and the memory maps are lost? >>=20 Samples are only queued for passive domains (on the initiator queue) (this is done to avoid waking up the initiator to process=20 samples that happen when running passive domains), or when executing=20 a critical part of the Xen code. PC samples for passive domains are not decoded into a file/offset and thus there is no issue. When the sample happens at a critical session of Xen, there is no issue either, since the PC refers to code in the hypervisor. When a counter overflows in an active domain, we guarantee that the samples in the hypervisor are transferred to the domain CPU buffer, before the domain continue execution (i.e. virtually no queue). Thus there is no process context switch or exiting processes in between a counter overflow and the receipt of a sample in the domain CPU buffer (for active domains). The normal flush of CPU buffer in the domain when a process exit still applies, as in standard linux. =20 >> -OProfile analysis tools expect the exectuable to be around when=20 >> analyzing the sample files. Is the oprofile user space making the=20 >> assumption that the domains have the same executables for=20 >> the active and=20 >> passive domains? Things could be messy if the domains are different,=20 >> e.g. Fedora Core 3 domain and a Rawhide domain. >>=20 Right now, passive domains samples do not get assigned to any=20 executable file. They are assigned to a general category domain_n (coarse granularity profiling). (i.e. no executable file is needed by analysis tools for passive domains). Active domains=20 keep their own sample files and thus there is no issue either. =20 Ian suggested that we add support to enable the user to specify (at the initiator) a file with symbol mappings for passive domains. This may be added in a future version to enable symbol mapping for the kernel, but user level symbols would still not be decoded for passive domains. >> -Should get rid of=20 >> xen-2.0.5/xen/arch/x86/oprofile/#Makefile# in the xen=20 >> hypervisor patch. >>=20 >>=20 Sure! Sorry, I missed that. Renato=20 PS: For context, those who did not see the original post, please check: =20 http://lists.xensource.com/archives/html/xen-devel/2005-04/msg00256.html >> -Will >>=20 >> _______________________________________________ >> Xen-devel mailing list >> Xen...@li... http://lists.xensource.com/xen-devel >>=20 |
From: Foong, A. <ann...@in...> - 2005-04-12 02:17:50
|
>> Right now, there is no virtualization of the hardware performance >> counters. Our goal was to have a system wide profiling capability >> that could enable us understand better the behavior of Xen. I have a tangential question, and thought you might be able to shed some light. I understand that Xen does not virtualize performance counters, but does the original Xen depend on hardware performance counters to work ? I am trying to understand how these counters are exposed by Xenoprof, and what HW dependency Xenoprof adds to Xen. =20 BTW, looking forward to using this wonderful tool for Xen. Thanks. -A =20 |
From: Santos, J. R. G (J. R. Santos) <jos...@hp...> - 2005-04-12 06:32:07
|
John, I am sorry you received this out of context. I was planing to send you a note describing our work=20 on enabling oprofile on the Xen virtual machine, but=20 the discussion on the Xen mailing list was forwarded to the OProfile list, before I had a chance to send you some information. Xen is an open source virtual machine monitor. While Vmware provides full virtualization where the OS believes is running on a physical machine, Xen provides paravirtualization. With paravirtualization, the OS is modified and made aware of the virtualization layer. This reduces the overhead of hardware emulation and improve system performance Before our work, it was not possile to use Oprofile in systems running Xen. Our group at HP Labs modified Oprofile and Xen to enable statistical profiling in Xen. Most of the code was written by Aravind Menon while working in our group.=20 =20 At a high level, we moved the X86 low level (hardware dependent) Oprofile component into the Xen hypervisor and exposed access to this functionality through a new hypercall (equivalent of system call at the Virtual machine layer). We then created a Xen specific low level driver for Oprofile which instead of programming the performance counters directly uses this new hypercall. I have attached the three patches that are needed, one for the Xen hypervisor, one for the linux kernel and one for the user level Oprofile code. The patch for the the linux code was=20 created for the Xen variation of linux 2.6.10 (commonly called xenolinux). This is linux with the modifications necessary for making the OS aware of the virtualization layer. Therefore, the patches would not apply cleanly to standard linux 2.6. We would be very interested in knowing your thoughts on this and if you would be interestd in incorporating this on the main oprofile distribution. Thanks Renato > -----Original Message----- > From: John Levon [mailto:mo...@co...] On Behalf Of=20 > John Levon > Sent: Monday, April 11, 2005 4:12 PM > To: Santos, Jose Renato G (Jose Renato Santos) > Cc: William Cohen; xen...@li...;=20 > oprofile-list; Aravind Menon; G John Janakiraman; Turner, Yoshio > Subject: Re: [Xen-devel] [PATCH] Xenoprof: Enabling=20 > performance profiling in Xen >=20 >=20 > On Mon, Apr 11, 2005 at 03:37:57PM -0700, Santos, Jose Renato=20 > G (Jose Renato Santos) wrote: >=20 > >=20 > http://lists.xensource.com/archives/html/xen-devel/2005-04/msg00256.ht > > ml >=20 > Where can I get the diff against oprofile userspace source,=20 > plus a diff of the changes to the kernel driver? >=20 > (It'd be really handy if somebody wrote up a little bit about=20 > Xen (no idea what a domain is) for us OProfile people too) >=20 > regards > john >=20 |
From: John L. <le...@mo...> - 2005-04-12 13:10:13
|
On Mon, Apr 11, 2005 at 11:32:05PM -0700, Santos, Jose Renato G (Jose Renato Santos) wrote: > I was planing to send you a note describing our work > on enabling oprofile on the Xen virtual machine, but Thanks. > level Oprofile code. The patch for the the linux code was > created for the Xen variation of linux 2.6.10 (commonly > called xenolinux). This is linux with the modifications > necessary for making the OS aware of the virtualization layer. > Therefore, the patches would not apply cleanly to standard > linux 2.6. I understand that. However, it would still be useful to see a diff so I can see exactly what has changed in drivers/oprofile/ (I see you added two new codes for the buffer etc.) > We would be very interested in knowing your thoughts on this > and if you would be interestd in incorporating this > on the main oprofile distribution. The userspace patch looks broadly OK, and I'd certainly be happy to look at this stuff. regards john |
From: Santos, J. R. G (J. R. Santos) <jos...@hp...> - 2005-04-12 06:35:59
|
Oh, I forgot to answer your question on Xen Xen uses the term "domain" to refer to a "virtual machine" Renato > -----Original Message----- > From: Santos, Jose Renato G (Jose Renato Santos)=20 > Sent: Monday, April 11, 2005 11:32 PM > To: 'John Levon' > Cc: oprofile-list; Aravind Menon; G John Janakiraman; Turner, Yoshio > Subject: RE: [Xen-devel] [PATCH] Xenoprof: Enabling=20 > performance profiling in Xen >=20 >=20 >=20 > John, >=20 > I am sorry you received this out of context. > I was planing to send you a note describing our work=20 > on enabling oprofile on the Xen virtual machine, but=20 > the discussion on the Xen mailing list was forwarded > to the OProfile list, before I had a chance to > send you some information. >=20 > Xen is an open source virtual machine monitor. > While Vmware provides full virtualization where the OS > believes is running on a physical machine, Xen provides > paravirtualization. With paravirtualization, the OS > is modified and made aware of the virtualization layer. > This reduces the overhead of hardware emulation and improve > system performance >=20 > Before our work, it was not possile to use Oprofile in > systems running Xen. > Our group at HP Labs modified Oprofile and Xen to enable > statistical profiling in Xen. Most of the code was written > by Aravind Menon while working in our group.=20 > =20 > At a high level, we moved the X86 low level (hardware dependent) > Oprofile component into the Xen hypervisor and exposed > access to this functionality through a new hypercall > (equivalent of system call at the Virtual machine layer). > We then created a Xen specific low level driver for Oprofile > which instead of programming the performance counters directly > uses this new hypercall. >=20 > I have attached the three patches that are needed, one for the > Xen hypervisor, one for the linux kernel and one for the user > level Oprofile code. The patch for the the linux code was=20 > created for the Xen variation of linux 2.6.10 (commonly > called xenolinux). This is linux with the modifications > necessary for making the OS aware of the virtualization layer. > Therefore, the patches would not apply cleanly to standard > linux 2.6. >=20 > We would be very interested in knowing your thoughts on this > and if you would be interestd in incorporating this > on the main oprofile distribution. >=20 > Thanks >=20 > Renato >=20 >=20 >=20 > > -----Original Message----- > > From: John Levon [mailto:mo...@co...] On Behalf Of > > John Levon > > Sent: Monday, April 11, 2005 4:12 PM > > To: Santos, Jose Renato G (Jose Renato Santos) > > Cc: William Cohen; xen...@li...;=20 > > oprofile-list; Aravind Menon; G John Janakiraman; Turner, Yoshio > > Subject: Re: [Xen-devel] [PATCH] Xenoprof: Enabling=20 > > performance profiling in Xen > >=20 > >=20 > > On Mon, Apr 11, 2005 at 03:37:57PM -0700, Santos, Jose Renato > > G (Jose Renato Santos) wrote: > >=20 > > >=20 > >=20 > http://lists.xensource.com/archives/html/xen-devel/2005-04/msg00256.ht > > > ml > >=20 > > Where can I get the diff against oprofile userspace source, > > plus a diff of the changes to the kernel driver? > >=20 > > (It'd be really handy if somebody wrote up a little bit about > > Xen (no idea what a domain is) for us OProfile people too) > >=20 > > regards > > john > >=20 >=20 |
From: Santos, J. R. G (J. R. Santos) <jos...@hp...> - 2005-04-12 06:56:07
|
> -----Original Message----- > From: Foong, Annie [mailto:ann...@in...]=20 > Sent: Monday, April 11, 2005 7:17 PM > To: Santos, Jose Renato G (Jose Renato Santos) > Cc: oprofile-list > Subject: RE: [Xen-devel] [PATCH] Xenoprof: Enabling=20 > performance profiling in Xen >=20 >=20 > >> Right now, there is no virtualization of the hardware=20 > performance > >> counters. Our goal was to have a system wide profiling=20 > capability > >> that could enable us understand better the behavior of Xen. >=20 > I have a tangential question, and thought you might be able=20 > to shed some light. I understand that Xen does not=20 > virtualize performance counters, but does the original Xen=20 > depend on hardware performance counters to > work ? I am trying to understand how these counters are exposed by > Xenoprof, and what HW dependency Xenoprof adds to Xen. =20 >=20 I am not sure if I understood your concerns ... Original Xen does not depend on any hardware performance counters. Xenoprof use the hardware performance counters, but only if oprofile is started.=20 Xenoprof does not prevent Xen from running in any hardware where it used to run before xenoprof Currently xenoprof only supports pentium 4 and pentium iii=20 CPU models. Xenoprof will not prevent Xen from running on other CPU models, except that running OProfile will not be possible on other models. I hope this answers your questions =20 Renato =20 > BTW, looking forward to using this wonderful tool for Xen. >=20 > Thanks. >=20 > -A =20 >=20 |
From: John L. <le...@mo...> - 2005-04-12 13:11:24
|
On Mon, Apr 11, 2005 at 11:56:04PM -0700, Santos, Jose Renato G (Jose Renato Santos) wrote: > Xenoprof does not prevent Xen from running in any hardware > where it used to run before xenoprof > Currently xenoprof only supports pentium 4 and pentium iii > CPU models. Xenoprof will not prevent Xen > from running on other CPU models, except that running OProfile > will not be possible on other models. Is it not possible to support running in timer interrupt mode in such cases? regards john |
From: Santos, J. R. G (J. R. Santos) <jos...@hp...> - 2005-04-12 15:27:51
|
>> -----Original Message----- >> From: John Levon [mailto:mo...@co...] On Behalf Of=20 >> John Levon >> Sent: Tuesday, April 12, 2005 6:11 AM >> To: Santos, Jose Renato G (Jose Renato Santos) >> Cc: Foong, Annie; oprofile-list; Aravind Menon; G John=20 >> Janakiraman; Turner, Yoshio >> Subject: Re: [Xen-devel] [PATCH] Xenoprof: Enabling=20 >> performance profiling in Xen >>=20 >>=20 >> On Mon, Apr 11, 2005 at 11:56:04PM -0700, Santos, Jose=20 >> Renato G (Jose Renato Santos) wrote: >>=20 >> > Xenoprof does not prevent Xen from running in any hardware >> > where it used to run before xenoprof >> > Currently xenoprof only supports pentium 4 and pentium iii=20 >> > CPU models. Xenoprof will not prevent Xen >> > from running on other CPU models, except that running OProfile >> > will not be possible on other models. >>=20 >> Is it not possible to support running in timer interrupt=20 >> mode in such cases? >>=20 Yes, sorry. That is what I should have said. Each domain (VM) would be able to run Oprofile in timer interrupt mode, but this will only profile the domain (virtual machine) and not the hypervisor. Renato >> regards >> john >>=20 |
From: Santos, J. R. G (J. R. Santos) <jos...@hp...> - 2005-04-12 17:59:05
|
>> >>=20 >> >> Is it not possible to support running in timer interrupt >> >> mode in such cases? >> >>=20 >> Yes, sorry. That is what I should have said. >> Each domain (VM) would be able to run Oprofile in timer interrupt >> mode, but this will only profile the domain (virtual machine) and >> not the hypervisor. >>=20 >> Renato I am sorry, but Aravind pointed out that my comment was incorrect. Unfortunately, in the current version there is no support for timer interrupt mode. I think it should not be difficult to add timer interrupt mode in the future. I will include this on our todo list. =20 Renato |
From: Santos, J. R. G (J. R. Santos) <jos...@hp...> - 2005-04-13 02:18:23
Attachments:
pmc.c
oprofile-xen.diff
|
John, Thanks your reply and interest >>=20 >> I understand that. However, it would still be useful to see=20 >> a diff so I can see exactly what has changed in=20 >> drivers/oprofile/ (I see you added two new codes for the buffer etc.) >>=20 I have attached the diffs for drivers/oprofile. I have also attached the Oprofile low level driver for Xen (arch/xen/oprofile/pmc.c). Let me know if there is anything else you need =20 >> > We would be very interested in knowing your thoughts on this >> > and if you would be interestd in incorporating this >> > on the main oprofile distribution. >>=20 >> The userspace patch looks broadly OK, and I'd certainly be=20 >> happy to look at this stuff. >>=20 Great! Thanks Please, send us a message if you have any question. =20 regards Renato >> regards >> john >>=20 |
From: John L. <le...@mo...> - 2005-04-13 15:48:20
|
On Tue, Apr 12, 2005 at 07:18:02PM -0700, Santos, Jose Renato G (Jose Renato Santos) wrote: > I have attached the diffs for drivers/oprofile. Thanks. Some initial comments: -static void add_kernel_ctx_switch(unsigned int in_kernel) +static void add_mode_switch(unsigned int cpu_mode) { add_event_entry(ESCAPE_CODE); - if (in_kernel) + if (cpu_mode == 0) + add_event_entry(USER_ENTER_SWITCH_CODE); + else if (cpu_mode == 1) add_event_entry(KERNEL_ENTER_SWITCH_CODE); else - add_event_entry(KERNEL_EXIT_SWITCH_CODE); + add_event_entry(XEN_ENTER_SWITCH_CODE); Please use defines for the cpu_mode, also make this function use switch() +static inline int is_dom_switch(unsigned long val) +{ + return val == ~1UL; Use a define for such a special EIP value. ESCAPE_CODE_DOMAIN or whatever. -void oprofile_add_sample(unsigned long eip, unsigned int is_kernel, +void oprofile_add_sample(unsigned long eip, unsigned int cpu_mode, You need to merge with recent Linux kernel, this has changed significantly... Can't Xen get its own entry point instead of modifying this one? + // aravind: In old code, why do we do the following? + // is_kernel = !!is_kernel; To ensure a binary value (i.e. exactly 0 or 1). + * aravind: also called on echo 1 > /dev/oprofile/dump Fair comment, but drop the name. +#include <linux/pagemap.h> +#include <linux/ctype.h> What are these needed for? +static ssize_t adomain_write(struct file *file, char const __user *buf, size_t count, loff_t * offset) Spaces after '*' here, and elsewhere. - dentry = d_alloc_name(root, name); + struct qstr qname; + qname.name = name; + qname.len = strlen(name); + qname.hash = full_name_hash(qname.name, qname.len); + dentry = d_alloc(root, &qname); More merging needed? regards john |
From: John L. <le...@mo...> - 2005-04-13 16:02:24
|
On Tue, Apr 12, 2005 at 07:18:02PM -0700, Santos, Jose Renato G (Jose Renato Santos) wrote: > >> The userspace patch looks broadly OK, and I'd certainly be > >> happy to look at this stuff. +opd_create_domain(char const *name) space after '*' +extern long long other_domain; Make this a member of struct transient. + if (image->name[strlen(image->name)] == other_domain + '0') + return image; Er, I don't think you mean this. Anyway, please just add a new field to struct kernel_image and compare against that in the domain list. + sprintf (name, "__domain_%d", (int)other_domain); No space before opening bracket. -static void code_kernel_exit(struct transient * trans) +static void code_user_enter(struct transient * trans) { verbprintf(vmisc, "KERNEL_EXIT_SWITCH to user-space\n"); Fix the verbprintf too. + /* subtlety: we must keep trans->cookie cached, + * even though it's meaningless for Xen - + * we won't necessarily get a cookie switch on + * Xen exit. See comments in opd_sfile.c. + * It seems that we can get away with in_kernel = 1 as long as we + * supply the correct Xen image, and its address range in startup + * find_kernel_image is modified to look in the Xen image also + */ Weird line wrapping? +static void code_domain_switch(struct transient *trans) Space after '*' if (!is_escape_code(code)) { + /* dirty dirty */ + if (other_domain != -1) + code = ~0ULL; opd_put_sample(&trans, code); Why do you need this? If we're in another domain, isn't the EIP from the kernel going to be 0? That's how I read your kernel patch anyway. Anyway, we need a more expansive comment that "dirty dirty" here. + /* canonicalise xen image filename. fix #637805 */ Drop the 'fix..' ? +# get start and end points of xen +get_xen_range() Please make a combined function for both kernel and xen that you pass arguments into. +# aravind menon: xen image option Drop comment + --xen) + --active-domains) + --passive-domains) You need to save the results of these into $SETUP_FILE regards, john |
From: John L. <le...@mo...> - 2005-04-11 23:11:56
|
On Mon, Apr 11, 2005 at 03:37:57PM -0700, Santos, Jose Renato G (Jose Renato Santos) wrote: > http://lists.xensource.com/archives/html/xen-devel/2005-04/msg00256.html Where can I get the diff against oprofile userspace source, plus a diff of the changes to the kernel driver? (It'd be really handy if somebody wrote up a little bit about Xen (no idea what a domain is) for us OProfile people too) regards john |
From: William C. <wc...@nc...> - 2005-04-12 03:11:04
|
John Levon wrote: > On Mon, Apr 11, 2005 at 03:37:57PM -0700, Santos, Jose Renato G (Jose Renato Santos) wrote: > > >>http://lists.xensource.com/archives/html/xen-devel/2005-04/msg00256.html > > > Where can I get the diff against oprofile userspace source, plus a diff > of the changes to the kernel driver? > > (It'd be really handy if somebody wrote up a little bit about Xen (no > idea what a domain is) for us OProfile people too) Xen is a virtualization system. It allows multiple domains (instances of the OS) to run on a computer using a hypervisor. Think of a domain as a Virtual Machine in the old 360 mainframes. Thus, a physical machine could appear to be multiple logical machines, each with a different disk image and installation running on it. Xen uses paravirtualization, so the kernel needs to be modified to make calls to the hypervisor rather than directly touch hardware like the memory management unit. This is bit different than VMware which emulates the machine and the OS doesn't need to be modified. However, Xen has better performance a a result of this. Note that user applications do not need to be modified to run on Xen. There is more information about Xen at: http://xen.sf.net/ Here are the start of the threads in the xen-devel mailing list discussing the patches to provide oprofile support: http://lists.xensource.com/archives/html/xen-devel/2005-04/msg00256.html http://lists.xensource.com/archives/html/xen-devel/2005-04/msg00257.html http://lists.xensource.com/archives/html/xen-devel/2005-04/msg00259.html -Will |