From: Denys V. <dvl...@re...> - 2012-05-15 15:09:38
|
#include <stdio.h> #include <pthread.h> #include <unistd.h> #define NTHD 10 static int thd_no; static void *sub_thd(void *c) { printf("thread created: %d\n", ++thd_no); for (;;) getuid(); return NULL; } int main(int argc, char **argv) { int i; pthread_t thd; for (i = 0; i < NTHD; i++) pthread_create(&thd, NULL, sub_thd, NULL); sleep(1); return 0; } The above program runs like this: $ gcc -lpthread -Wall test.c $ ./a.out thread created: 3 thread created: 1 thread created: 2 thread created: 4 thread created: 6 thread created: 7 thread created: 8 thread created: 5 thread created: 9 thread created: 10 But under strace it most of the time never finishes: $ strace -f -o /dev/null ./a.out thread created: 1 thread created: 2 thread created: 3 thread created: 4 <...big pause...> thread created: 5 <...waits forever...> Testcase essentially starts threads, each if which does this: for (;;) getuid(); Since getuid is a very cheap syscall, it finishes very quickly. When there are 2-3 such threads, strace's waitpid() picks up one, restarts it, picks another, restarts it, ...and meanwhile first thread stops again at syscall entry/exit and is ready to be picked up again. It may happen so that the very first thread, which starts all others, even though it is also ready to be handled by strace, will never be picked up because there is always at least one other getuid'ing thread, and that thread gets picked up instead. One solution is to run waitpid() in a loop until it returns 0 "no more tracees to wait", then handle them all. It was NAKed a few years ago. Do we want to fix it, of do we chalk it to "thread scheduling fairness is busted up under strace, won't fix"? -- vda |
From: Dmitry V. L. <ld...@al...> - 2012-05-16 09:48:50
|
On Tue, May 15, 2012 at 05:09:21PM +0200, Denys Vlasenko wrote: > #include <stdio.h> > #include <pthread.h> > #include <unistd.h> > #define NTHD 10 > static int thd_no; > static void *sub_thd(void *c) { > printf("thread created: %d\n", ++thd_no); > for (;;) getuid(); > return NULL; > } > int main(int argc, char **argv) { > int i; > pthread_t thd; > for (i = 0; i < NTHD; i++) > pthread_create(&thd, NULL, sub_thd, NULL); > sleep(1); > return 0; > } > > The above program runs like this: > > $ gcc -lpthread -Wall test.c > $ ./a.out > thread created: 3 > thread created: 1 > thread created: 2 > thread created: 4 > thread created: 6 > thread created: 7 > thread created: 8 > thread created: 5 > thread created: 9 > thread created: 10 > > But under strace it most of the time never finishes: > > $ strace -f -o /dev/null ./a.out > thread created: 1 > thread created: 2 > thread created: 3 > thread created: 4 > <...big pause...> > thread created: 5 > <...waits forever...> > > Testcase essentially starts threads, each if which does this: > > for (;;) getuid(); > > Since getuid is a very cheap syscall, it finishes very quickly. > > When there are 2-3 such threads, strace's waitpid() picks up one, restarts it, > picks another, restarts it, ...and meanwhile first thread stops again at > syscall entry/exit and is ready to be picked up again. > > It may happen so that the very first thread, which starts all others, even > though it is also ready to be handled by strace, will never be picked up > because there is always at least one other getuid'ing thread, and that thread > gets picked up instead. > > One solution is to run waitpid() in a loop until it returns 0 "no more tracees > to wait", then handle them all. It was NAKed a few years ago. It would be nice to have a look at that discussion. Do you have a reference? What was the rationale that time? > Do we want to fix it, of do we chalk it to > "thread scheduling fairness is busted up under strace, won't fix"? If the fix has no drawbacks, I'd rather have it fixed. -- ldv |
From: Denys V. <dvl...@re...> - 2012-05-16 10:47:27
|
On 05/16/2012 11:48 AM, Dmitry V. Levin wrote: >> Testcase essentially starts threads, each of which does this: >> >> for (;;) getuid(); >> >> Since getuid is a very cheap syscall, it finishes very quickly. >> >> When there are 2-3 such threads, strace's waitpid() picks up one, restarts it, >> picks another, restarts it, ...and meanwhile first thread stops again at >> syscall entry/exit and is ready to be picked up again. >> >> It may happen so that the very first thread, which starts all others, even >> though it is also ready to be handled by strace, will never be picked up >> because there is always at least one other getuid'ing thread, and that thread >> gets picked up instead. >> >> One solution is to run waitpid() in a loop until it returns 0 "no more tracees >> to wait", then handle them all. It was NAKed a few years ago. > > It would be nice to have a look at that discussion. Do you have a > reference? What was the rationale that time? I guess Roland saw it as "ugly" and "trying to work around ptrace API which is a hopelessly bad API anyway". On the technical note, it adds additional waitpid call per every syscall entry/exit, and serializes strace even more: instead of servicing and restarting a thread as fast as we can, we collect other threads - keeping ready threads stuck a bit longer. I looked into in and _maybe_ signalfd may help us noticeably. I expect difficulties in waitpid area: it's likely we can't just block SIGCHLD and happily read their siginfo's in batch reads from signalfd: for one, some SIGCHLDs can be lost. Second, not doing waitpid on exiting tracees will turn them into zombies - thus, we _still_ will need to call waitpid... -- vda |
From: Oleg N. <ol...@re...> - 2012-05-17 17:54:22
|
On 05/16, Denys Vlasenko wrote: > > On 05/16/2012 11:48 AM, Dmitry V. Levin wrote: >>> >>> One solution is to run waitpid() in a loop until it returns 0 "no more tracees >>> to wait", then handle them all. It was NAKed a few years ago. >> >> It would be nice to have a look at that discussion. Do you have a >> reference? What was the rationale that time? > > I guess Roland saw it as "ugly" and "trying to work around ptrace API > which is a hopelessly bad API anyway". It is indeed ugly and hopeless. But perhaps Roland hoped we can have something new. Now that utrace is dead, unlikely this is possible. > On the technical note, it adds additional waitpid call per every > syscall entry/exit, and serializes strace even more: instead of > servicing and restarting a thread as fast as we can, we collect > other threads - keeping ready threads stuck a bit longer. But what else we can do to fix the problem? (lets not discuss the vectorized do_wait right now ;) Imho, you should reconsider your patch. > I looked into in and _maybe_ signalfd may help us noticeably. unlikely, because > for one, some SIGCHLDs can be lost. yes, SIGCHLD doesn't queue. Oleg. |
From: Dmitry V. L. <ld...@al...> - 2012-05-17 18:02:05
|
On Wed, May 16, 2012 at 12:47:14PM +0200, Denys Vlasenko wrote: > On 05/16/2012 11:48 AM, Dmitry V. Levin wrote: [...] > >> One solution is to run waitpid() in a loop until it returns 0 "no more tracees > >> to wait", then handle them all. It was NAKed a few years ago. > > > > It would be nice to have a look at that discussion. Do you have a > > reference? What was the rationale that time? > > I guess Roland saw it as "ugly" and "trying to work around ptrace API > which is a hopelessly bad API anyway". > > On the technical note, it adds additional waitpid call per every > syscall entry/exit, So its cost is quite noticeable, and therefore this fair scheduling is not always desirable. Is it possible to implement the feature as an option? -- ldv |