Thread: [perfmon2] Self monitoring threads
Status: Beta
Brought to you by:
seranian
From: john <sy...@vi...> - 2009-04-30 22:51:39
|
Hello Stephane et al., My goal is to have an arbitrary number of threads monitor themselves, each being interrupted after approximately 1ms worth of clock cycles have gone by. Libpfm's example program self_smpl_multi seemed to do something similar, so I rewrote it to take an arbitrary number of threads and an arbitrary number of desired interrupts per second. I followed self_smpl_multi's technique of arranging to receive a SIGIO in the thread associated with the Perfmon context, landing in the registered signal handler. The threads continue running until they've reached the targeted number of interrupts. For example, 10 interrupts/second times five seconds of runtime would be 50 total interrupts. This approach works fine up to about 25 interrupts/thread/second on my Intel Core2 Quad (4 core) system. After that, some of the threads never stop. Peering at the program in GDB leads me to believe that one or more threads doesn't get SIGIO anymore. My current guess is that if any one thread is handling SIGIO, any new notifications on other threads just get dropped on the floor; the descriptor doesn't get read, so the signal handler doesn't get invoked anymore. Would real time signals help me here (to queue up the signals), or is there some other approach to thread interruption I should explore? Thanks in advance for your time and help. -john |
From: stephane e. <er...@go...> - 2009-05-04 19:32:03
|
Hello John, On Fri, May 1, 2009 at 12:51 AM, john <sy...@vi...> wrote: > Hello Stephane et al., > > My goal is to have an arbitrary number of threads monitor themselves, > each being interrupted after approximately 1ms worth of clock cycles > have gone by. Libpfm's example program self_smpl_multi seemed to do > something similar, so I rewrote it to take an arbitrary number of > threads and an arbitrary number of desired interrupts per second. > > I followed self_smpl_multi's technique of arranging to receive a SIGIO > in the thread associated with the Perfmon context, landing in the > registered signal handler. The threads continue running until they've > reached the targeted number of interrupts. For example, 10 > interrupts/second times five seconds of runtime would be 50 total > interrupts. > > This approach works fine up to about 25 interrupts/thread/second on my > Intel Core2 Quad (4 core) system. After that, some of the threads never > stop. Peering at the program in GDB leads me to believe that one or more > threads doesn't get SIGIO anymore. My current guess is that if any one > thread is handling SIGIO, any new notifications on other threads just > get dropped on the floor; the descriptor doesn't get read, so the signal > handler doesn't get invoked anymore. > Yes, you are running into the issue with the pthread signal model. There is no guarantee asynchronous signals such as SIGIO get delivered to the right thread. Last time I looked a the kernel code, it works as a job queue for the process, which ever thread is exiting the kernel may pickup the next signal job on the queue. Therefore it is not possible to guarantee which thread is going to get called. But this is also why, the handler you be prepared to handle notifications from another thread. The file descriptor should be sufficient to identify which thread the overflow originated with. > Would real time signals help me here (to queue up the signals), or is > there some other approach to thread interruption I should explore? > It seems that using realtime signals does help, use a signal >33 and you should get a better behavior. |
From: john <sy...@vi...> - 2009-06-05 17:39:47
|
Hi Stephane, stephane eranian wrote: > Hello John, > > > On Fri, May 1, 2009 at 12:51 AM, john <sy...@vi...> wrote: > >> Hello Stephane et al., >> >> My goal is to have an arbitrary number of threads monitor themselves, >> each being interrupted after approximately 1ms worth of clock cycles >> have gone by. Libpfm's example program self_smpl_multi seemed to do >> something similar, so I rewrote it to take an arbitrary number of >> threads and an arbitrary number of desired interrupts per second. >> >> I followed self_smpl_multi's technique of arranging to receive a SIGIO >> in the thread associated with the Perfmon context, landing in the >> registered signal handler. The threads continue running until they've >> reached the targeted number of interrupts. For example, 10 >> interrupts/second times five seconds of runtime would be 50 total >> interrupts. >> >> This approach works fine up to about 25 interrupts/thread/second on my >> Intel Core2 Quad (4 core) system. After that, some of the threads never >> stop. Peering at the program in GDB leads me to believe that one or more >> threads doesn't get SIGIO anymore. My current guess is that if any one >> thread is handling SIGIO, any new notifications on other threads just >> get dropped on the floor; the descriptor doesn't get read, so the signal >> handler doesn't get invoked anymore. >> >> > Yes, you are running into the issue with the pthread signal model. There is > no guarantee asynchronous signals such as SIGIO get delivered to the right > thread. Last time I looked a the kernel code, it works as a job queue for the > process, which ever thread is exiting the kernel may pickup the next signal > job on the queue. Therefore it is not possible to guarantee which thread is > going to get called. But this is also why, the handler you be > prepared to handle > notifications from another thread. The file descriptor should be sufficient to > identify which thread the overflow originated with. > > > I do wish for the target threads themselves to be interrupted; this is so I can do things like periodic stack traces. Therefore, handling SIGIO in a different thread won't be sufficient. Is there any interest by others in a mechanism like this? Since Perfmon makes the kernel information available via a file descriptor, there aren't any reliable ways to interrupt the target thread when the PMU overflows. In theory, Perfmon could, for example, run a user-specified callback when the counter overflows. Since the thread is interrupted anyway, it could do this by simply delivering a signal directly to the target thread, and let the signal handling system do the rest. I don't really know Linux kernel internals, but it looks like send_sig() might do this. > > >> Would real time signals help me here (to queue up the signals), or is >> there some other approach to thread interruption I should explore? >> >> > It seems that using realtime signals does help, use a signal >33 and > you should get a better behavior. > I've tried this, and while signals do seem to get reliably delivered, they sometimes end up getting delivered to the wrong thread. Thanks again, john |
From: stephane e. <er...@go...> - 2009-06-05 20:19:56
|
John, Another possibility which I have not explored is to try pthread_kill() to direct a signal to the correct thread when received by the wrong thread. I don't know the internals of this call but from the definition, it may be the right thing to use. On Fri, Jun 5, 2009 at 7:39 PM, john <sy...@vi...> wrote: > Hi Stephane, > > > stephane eranian wrote: > >> Hello John, >> >> >> On Fri, May 1, 2009 at 12:51 AM, john <sy...@vi...> wrote: >> >> >>> Hello Stephane et al., >>> >>> My goal is to have an arbitrary number of threads monitor themselves, >>> each being interrupted after approximately 1ms worth of clock cycles >>> have gone by. Libpfm's example program self_smpl_multi seemed to do >>> something similar, so I rewrote it to take an arbitrary number of >>> threads and an arbitrary number of desired interrupts per second. >>> >>> I followed self_smpl_multi's technique of arranging to receive a SIGIO >>> in the thread associated with the Perfmon context, landing in the >>> registered signal handler. The threads continue running until they've >>> reached the targeted number of interrupts. For example, 10 >>> interrupts/second times five seconds of runtime would be 50 total >>> interrupts. >>> >>> This approach works fine up to about 25 interrupts/thread/second on my >>> Intel Core2 Quad (4 core) system. After that, some of the threads never >>> stop. Peering at the program in GDB leads me to believe that one or more >>> threads doesn't get SIGIO anymore. My current guess is that if any one >>> thread is handling SIGIO, any new notifications on other threads just >>> get dropped on the floor; the descriptor doesn't get read, so the signal >>> handler doesn't get invoked anymore. >>> >>> >>> >> Yes, you are running into the issue with the pthread signal model. There >> is >> no guarantee asynchronous signals such as SIGIO get delivered to the right >> thread. Last time I looked a the kernel code, it works as a job queue for >> the >> process, which ever thread is exiting the kernel may pickup the next >> signal >> job on the queue. Therefore it is not possible to guarantee which thread >> is >> going to get called. But this is also why, the handler you be >> prepared to handle >> notifications from another thread. The file descriptor should be >> sufficient to >> identify which thread the overflow originated with. >> >> >> >> > I do wish for the target threads themselves to be interrupted; this is so I > can do things like periodic stack traces. Therefore, handling SIGIO in a > different thread won't be sufficient. Is there any interest by others in a > mechanism like this? Since Perfmon makes the kernel information available > via a file descriptor, there aren't any reliable ways to interrupt the > target thread when the PMU overflows. In theory, Perfmon could, for example, > run a user-specified callback when the counter overflows. Since the thread > is interrupted anyway, it could do this by simply delivering a signal > directly to the target thread, and let the signal handling system do the > rest. I don't really know Linux kernel internals, but it looks like > send_sig() might do this. > > >> >> >>> Would real time signals help me here (to queue up the signals), or is >>> there some other approach to thread interruption I should explore? >>> >>> >>> >> It seems that using realtime signals does help, use a signal >33 and >> you should get a better behavior. >> >> > I've tried this, and while signals do seem to get reliably delivered, they > sometimes end up getting delivered to the wrong thread. > > Thanks again, > john > > |