From: Jan S. <jst...@re...> - 2012-12-06 12:06:47
|
Hi, this testsuite has many testcases, where binary name is in format: [number]-[number].run-test This is not very helpful when trying to find out, which one actually crashed or which one is responsible for kernel oops/hung tasks. For example if I get in dmesg: [ 702.018706] 1-5.run-test[10140]: segfault at 7f84e724bed8 ip 0000000000402208 sp 00007f84e724bed0 error 6 in 1-5.run-test[400000+4000] it's not clear, which testcase is responsible. Does anyone have some ideas? I had these two: 1. rename all testcases - lot of work, they are also referenced in some .sh scripts 2. modify run-tests.sh to copy it under more verbose name and run that one diff --git a/testcases/open_posix_testsuite/bin/run-tests.sh b/testcases/open_posix_testsuite/bin/run-tests.sh index bd8ca14..ddb6e0b 100755 --- a/testcases/open_posix_testsuite/bin/run-tests.sh +++ b/testcases/open_posix_testsuite/bin/run-tests.sh @@ -46,8 +46,11 @@ run_test() { complog=$(basename $testname).log.$$ - sh -c "$SCRIPT_DIR/t0 $TIMEOUT_VAL ./$1 $(cat ./$(echo "$1" | sed 's,\.[^\.]*,,').args 2>/dev/null)" > $complog 2>&1 + # create a copy with more verbose name + new_testname=$(basename $TEST_PATH)-$1 + cp -f ./$1 $new_testname + sh -c "$SCRIPT_DIR/t0 $TIMEOUT_VAL ./$new_testname $(cat ./$(echo "$1" | sed 's,\.[^\.]*,,').args 2>/dev/null)" > $complog 2>&1 ret_code=$? if [ "$ret_code" = "0" ]; then Which makes logs a bit more helpful: [179468.329814] User process fault: interruption code 0x60011 in pthread_create-1-5.run-test[80000000+5000] Regards, Jan |
From: <ch...@su...> - 2012-12-06 14:13:47
|
Hi! > this testsuite has many testcases, where binary name is in format: [number]-[number].run-test > This is not very helpful when trying to find out, which one actually > crashed or which one is responsible for kernel oops/hung tasks. > For example if I get in dmesg: > [ 702.018706] 1-5.run-test[10140]: segfault at 7f84e724bed8 ip 0000000000402208 sp 00007f84e724bed0 error 6 in 1-5.run-test[400000+4000] > it's not clear, which testcase is responsible. Here in suse we use CTCS2 to run the testcases and we have a perl script that generates the the test names accordingly to the directory structure which doesn't help when looking into the kernel log but the segfault is logged into the test results anyway. And mostly because of that the current test execution framework both for LTP and Open Posix Testsuite is a little out of my radar, although I would like to have them fixed too. > Does anyone have some ideas? > > I had these two: > 1. rename all testcases - lot of work, they are also referenced in some .sh scripts > 2. modify run-tests.sh to copy it under more verbose name and run that one Well I don't think that the option one is that hard, at least I'm not aware of much cases of hardcoded names in the Open Posix Testsuite. On the other hand having better test driver that executes the tests and presents the result in more convinient way would be more appealing. Or we can do both of them. I was thinking of creating a relatively simply test driver that runs the tests, takes care of timeouts (and anything else) and logs information in some reasonable format (I used JSON in similar case). And then about some tools that could compare the results, print list of failed testcases and so on. Does that sound reasonable? And on more general note we surely need to decide how to do things there. There are some howtos in the Documentation directory none of them are up to date. > diff --git a/testcases/open_posix_testsuite/bin/run-tests.sh b/testcases/open_posix_testsuite/bin/run-tests.sh > index bd8ca14..ddb6e0b 100755 > --- a/testcases/open_posix_testsuite/bin/run-tests.sh > +++ b/testcases/open_posix_testsuite/bin/run-tests.sh > @@ -46,8 +46,11 @@ run_test() { > > complog=$(basename $testname).log.$$ > > - sh -c "$SCRIPT_DIR/t0 $TIMEOUT_VAL ./$1 $(cat ./$(echo "$1" | sed 's,\.[^\.]*,,').args 2>/dev/null)" > $complog 2>&1 > + # create a copy with more verbose name > + new_testname=$(basename $TEST_PATH)-$1 > + cp -f ./$1 $new_testname > > + sh -c "$SCRIPT_DIR/t0 $TIMEOUT_VAL ./$new_testname $(cat ./$(echo "$1" | sed 's,\.[^\.]*,,').args 2>/dev/null)" > $complog 2>&1 > ret_code=$? > > if [ "$ret_code" = "0" ]; then > > Which makes logs a bit more helpful: > [179468.329814] User process fault: interruption code 0x60011 in pthread_create-1-5.run-test[80000000+5000] It's rather crude hack, but I guess that it's better than nothing. And you should probably remove the file after the test has finished. -- Cyril Hrubis ch...@su... |
From: Jan S. <jst...@re...> - 2012-12-06 15:35:58
|
----- Original Message ----- > From: ch...@su... > To: "Jan Stancek" <jst...@re...> > Cc: ltp...@li... > Sent: Thursday, 6 December, 2012 3:13:36 PM > Subject: Re: [LTP] open posix testsuite and names of testcases > > Hi! > > this testsuite has many testcases, where binary name is in format: > > [number]-[number].run-test > > This is not very helpful when trying to find out, which one > > actually > > crashed or which one is responsible for kernel oops/hung tasks. > > For example if I get in dmesg: > > [ 702.018706] 1-5.run-test[10140]: segfault at 7f84e724bed8 ip > > 0000000000402208 sp 00007f84e724bed0 error 6 in > > 1-5.run-test[400000+4000] > > it's not clear, which testcase is responsible. > > Here in suse we use CTCS2 to run the testcases and we have a perl > script > that generates the the test names accordingly to the directory > structure > which doesn't help when looking into the kernel log but the segfault > is logged into the test results anyway. > > And mostly because of that the current test execution framework both > for > LTP and Open Posix Testsuite is a little out of my radar, although I > would like to have them fixed too. > > > Does anyone have some ideas? > > > > I had these two: > > 1. rename all testcases - lot of work, they are also referenced in > > some .sh scripts > > 2. modify run-tests.sh to copy it under more verbose name and run > > that one > > Well I don't think that the option one is that hard, at least I'm not > aware of much cases of hardcoded names in the Open Posix Testsuite. > On > the other hand having better test driver that executes the tests and > presents the result in more convinient way would be more appealing. > Or > we can do both of them. I think we should do the renaming anyway. Would the following format work for everyone? "testdirname-[number]-[number].c" For example: pthread_create-1-5.c It's unique, and it's easy to tell what old name was. > I was thinking of creating a relatively simply test driver that runs > the > tests, takes care of timeouts (and anything else) and logs > information > in some reasonable format (I used JSON in similar case). And then > about > some tools that could compare the results, print list of failed > testcases and so on. Does that sound reasonable? How would this be related to ltp-pan? Would this be only for open posix testsuite or also for LTP? Regards, Jan |
From: <ch...@su...> - 2012-12-06 16:35:46
|
Hi! > > Well I don't think that the option one is that hard, at least I'm not > > aware of much cases of hardcoded names in the Open Posix Testsuite. > > On > > the other hand having better test driver that executes the tests and > > presents the result in more convinient way would be more appealing. > > Or > > we can do both of them. > > I think we should do the renaming anyway. > Would the following format work for everyone? "testdirname-[number]-[number].c" > For example: pthread_create-1-5.c > It's unique, and it's easy to tell what old name was. I'm ok with that. > > I was thinking of creating a relatively simply test driver that runs > > the > > tests, takes care of timeouts (and anything else) and logs > > information > > in some reasonable format (I used JSON in similar case). And then > > about > > some tools that could compare the results, print list of failed > > testcases and so on. Does that sound reasonable? > > How would this be related to ltp-pan? Would this be only for open posix testsuite > or also for LTP? Well, that is a good question. The differencies between Open Posix Testsuite and the rest of the LTP aren't too big so it may be possible to create a test driver for both and share most of the code. On the other hand. To just run the tests you just need to export right VARIABLES and modify PATH accordingly. To collect the result you need to translate the test return values. On the other hand I'm not even sure about all the features that ltp-pan has and what subset of them is used by different people. But I will surely like to have more up to date test driver that produces more usefull data. There are couple possibilities what we can do. We may start creating something that could run the Open Posix Testsuite and then gradually extend it to work with the rest of the LTP adding all the features we would need while deprecating the ltp-pan and removing it once we have full replacement. Or we may look if there is something to fit our meets out there allready and modify it to meet our needs. Or anything that any of you may come up with. -- Cyril Hrubis ch...@su... |
From: Wanlong G. <gao...@cn...> - 2012-12-06 14:16:06
|
On 12/06/2012 08:06 PM, Jan Stancek wrote: > Hi, > > this testsuite has many testcases, where binary name is in format: [number]-[number].run-test > This is not very helpful when trying to find out, which one actually > crashed or which one is responsible for kernel oops/hung tasks. > For example if I get in dmesg: > [ 702.018706] 1-5.run-test[10140]: segfault at 7f84e724bed8 ip 0000000000402208 sp 00007f84e724bed0 error 6 in 1-5.run-test[400000+4000] > it's not clear, which testcase is responsible. I met the same problem before because running the open_posix on recent RHEL6.4alpha and beta will cause kernel panic... Thanks, Wanlong Gao > > Does anyone have some ideas? > > I had these two: > 1. rename all testcases - lot of work, they are also referenced in some .sh scripts > 2. modify run-tests.sh to copy it under more verbose name and run that one > > diff --git a/testcases/open_posix_testsuite/bin/run-tests.sh b/testcases/open_posix_testsuite/bin/run-tests.sh > index bd8ca14..ddb6e0b 100755 > --- a/testcases/open_posix_testsuite/bin/run-tests.sh > +++ b/testcases/open_posix_testsuite/bin/run-tests.sh > @@ -46,8 +46,11 @@ run_test() { > > complog=$(basename $testname).log.$$ > > - sh -c "$SCRIPT_DIR/t0 $TIMEOUT_VAL ./$1 $(cat ./$(echo "$1" | sed 's,\.[^\.]*,,').args 2>/dev/null)" > $complog 2>&1 > + # create a copy with more verbose name > + new_testname=$(basename $TEST_PATH)-$1 > + cp -f ./$1 $new_testname > > + sh -c "$SCRIPT_DIR/t0 $TIMEOUT_VAL ./$new_testname $(cat ./$(echo "$1" | sed 's,\.[^\.]*,,').args 2>/dev/null)" > $complog 2>&1 > ret_code=$? > > if [ "$ret_code" = "0" ]; then > > Which makes logs a bit more helpful: > [179468.329814] User process fault: interruption code 0x60011 in pthread_create-1-5.run-test[80000000+5000] > > Regards, > Jan > > ------------------------------------------------------------------------------ > LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial > Remotely access PCs and mobile devices and provide instant support > Improve your efficiency, and focus on delivering more value-add services > Discover what IT Professionals Know. Rescue delivers > http://p.sf.net/sfu/logmein_12329d2d > _______________________________________________ > Ltp-list mailing list > Ltp...@li... > https://lists.sourceforge.net/lists/listinfo/ltp-list > |
From: <ch...@su...> - 2012-12-06 14:27:40
|
Hi! > > this testsuite has many testcases, where binary name is in format: [number]-[number].run-test > > This is not very helpful when trying to find out, which one actually > > crashed or which one is responsible for kernel oops/hung tasks. > > For example if I get in dmesg: > > [ 702.018706] 1-5.run-test[10140]: segfault at 7f84e724bed8 ip 0000000000402208 sp 00007f84e724bed0 error 6 in 1-5.run-test[400000+4000] > > it's not clear, which testcase is responsible. > > I met the same problem before because running the open_posix on recent RHEL6.4alpha and beta > will cause kernel panic... Which could still be fixed by a testdriver that ensures that the logfile was written to the disk before each test is executed. But the renaming is still a good idea because there may be a rare case where the filesystem will be trashed. -- Cyril Hrubis ch...@su... |
From: Wanlong G. <gao...@cn...> - 2012-12-06 14:47:29
|
On 12/06/2012 10:27 PM, ch...@su... wrote: > Hi! >>> this testsuite has many testcases, where binary name is in format: [number]-[number].run-test >>> This is not very helpful when trying to find out, which one actually >>> crashed or which one is responsible for kernel oops/hung tasks. >>> For example if I get in dmesg: >>> [ 702.018706] 1-5.run-test[10140]: segfault at 7f84e724bed8 ip 0000000000402208 sp 00007f84e724bed0 error 6 in 1-5.run-test[400000+4000] >>> it's not clear, which testcase is responsible. >> >> I met the same problem before because running the open_posix on recent RHEL6.4alpha and beta >> will cause kernel panic... > > Which could still be fixed by a testdriver that ensures that the logfile > was written to the disk before each test is executed. But the renaming > is still a good idea because there may be a rare case where the > filesystem will be trashed. Yeah, I prefer the renaming idea because we may get just the kernel crash vmcore without the logfile sometimes. For example from customers. Thanks, Wanlong Gao > |
From: <ch...@su...> - 2013-02-11 14:38:11
|
Hi! Just recently I've found that one of the posix tests stays running after the testsuite has exited (is reparented to init and runs happily ever after). Which is another usecase for this rename. But the original patch was rather hack (and it doesn't work for me anyway). So what about renaming the sources itself? Do we agree on this? -- Cyril Hrubis ch...@su... |
From: Jan S. <jst...@re...> - 2013-02-11 15:26:41
|
----- Original Message ----- > From: ch...@su... > To: "Jan Stancek" <jst...@re...> > Cc: ltp...@li... > Sent: Monday, 11 February, 2013 3:38:38 PM > Subject: Re: [LTP] open posix testsuite and names of testcases > > Hi! > Just recently I've found that one of the posix tests stays running > after > the testsuite has exited (is reparented to init and runs happily ever > after). Which is another usecase for this rename. > > But the original patch was rather hack (and it doesn't work for me > anyway). > > So what about renaming the sources itself? Do we agree on this? I'm not against renaming sources, just thinking if there is less intrusive way to achieve same result. I believe all Makefiles are generated, so other option that comes to mind would be to modify scripts/generate-makefiles.sh. Any thoughts about this approach? Regards, Jan |
From: <ch...@su...> - 2013-02-11 15:50:46
|
Hi! > > Just recently I've found that one of the posix tests stays running > > after > > the testsuite has exited (is reparented to init and runs happily ever > > after). Which is another usecase for this rename. > > > > But the original patch was rather hack (and it doesn't work for me > > anyway). > > > > So what about renaming the sources itself? Do we agree on this? > > I'm not against renaming sources, just thinking if there is less > intrusive way to achieve same result. I believe all Makefiles are generated, > so other option that comes to mind would be to modify scripts/generate-makefiles.sh. > Any thoughts about this approach? We can hack the script to generate Makefiles that include the dir name into the binary name which would be less intrusive but on the other hand that would break the rule that foo.c generates foo.run_test... -- Cyril Hrubis ch...@su... |
From: Jan S. <jst...@re...> - 2013-02-12 12:22:57
|
----- Original Message ----- > From: ch...@su... > To: "Jan Stancek" <jst...@re...> > Cc: ltp...@li... > Sent: Monday, 11 February, 2013 4:51:11 PM > Subject: Re: [LTP] open posix testsuite and names of testcases > > Hi! > > > Just recently I've found that one of the posix tests stays > > > running > > > after > > > the testsuite has exited (is reparented to init and runs happily > > > ever > > > after). Which is another usecase for this rename. > > > > > > But the original patch was rather hack (and it doesn't work for > > > me > > > anyway). > > > > > > So what about renaming the sources itself? Do we agree on this? > > > > I'm not against renaming sources, just thinking if there is less > > intrusive way to achieve same result. I believe all Makefiles are > > generated, > > so other option that comes to mind would be to modify > > scripts/generate-makefiles.sh. > > Any thoughts about this approach? > > We can hack the script to generate Makefiles that include the dir > name > into the binary name which would be less intrusive but on the other > hand > that would break the rule that foo.c generates foo.run_test... Caspar, Wanlong, any preference / thoughts? |
From: Wanlong G. <gao...@cn...> - 2013-02-18 02:07:33
|
On 02/12/2013 08:22 PM, Jan Stancek wrote: > > > ----- Original Message ----- >> From: ch...@su... >> To: "Jan Stancek" <jst...@re...> >> Cc: ltp...@li... >> Sent: Monday, 11 February, 2013 4:51:11 PM >> Subject: Re: [LTP] open posix testsuite and names of testcases >> >> Hi! >>>> Just recently I've found that one of the posix tests stays >>>> running >>>> after >>>> the testsuite has exited (is reparented to init and runs happily >>>> ever >>>> after). Which is another usecase for this rename. >>>> >>>> But the original patch was rather hack (and it doesn't work for >>>> me >>>> anyway). >>>> >>>> So what about renaming the sources itself? Do we agree on this? >>> >>> I'm not against renaming sources, just thinking if there is less >>> intrusive way to achieve same result. I believe all Makefiles are >>> generated, >>> so other option that comes to mind would be to modify >>> scripts/generate-makefiles.sh. >>> Any thoughts about this approach? >> >> We can hack the script to generate Makefiles that include the dir >> name >> into the binary name which would be less intrusive but on the other >> hand >> that would break the rule that foo.c generates foo.run_test... > > Caspar, Wanlong, any preference / thoughts? I vote to changing the rule of generating the Makefiles, it's simple and the simple rule change is easy to be accepted. Agree? Thanks, Wanlong Gao > > |
From: <ch...@su...> - 2013-02-27 14:38:06
|
Hi! > >>> I'm not against renaming sources, just thinking if there is less > >>> intrusive way to achieve same result. I believe all Makefiles are > >>> generated, > >>> so other option that comes to mind would be to modify > >>> scripts/generate-makefiles.sh. > >>> Any thoughts about this approach? > >> > >> We can hack the script to generate Makefiles that include the dir > >> name > >> into the binary name which would be less intrusive but on the other > >> hand > >> that would break the rule that foo.c generates foo.run_test... > > > > Caspar, Wanlong, any preference / thoughts? > > I vote to changing the rule of generating the Makefiles, it's simple > and the simple rule change is easy to be accepted. Agree? Ok. I will look into this and hopefully commit a patch soon enough. -- Cyril Hrubis ch...@su... |
From: <ch...@su...> - 2013-02-27 17:10:39
|
Hi! > > I vote to changing the rule of generating the Makefiles, it's simple > > and the simple rule change is easy to be accepted. Agree? > > Ok. I will look into this and hopefully commit a patch soon enough. I tried to change the generate-makefiles.sh script, but there are some problems. Some tests are shell scripts that doesn't have build phase. I've workarounded them by adding build rules that just copy them to name_x-y.sh in the build phase, which unfortunatelly haven't fixed all the problems. We have quite quite a lot of tests (~270) that are shell scripts that prints "This is tested implicitly via assertion xyz" and returns zero. I'm quite tempted to remove them. These just increase tests run time, log sizes, etc. but doesn't do any actual work. Then there are more problematic ones, shell tests that calls one binary with different parameters. As in sigignore for example. The paths and names are obviously hardcoded, which makes them fail. The fix for such cases will depend case by case but what I will probably do is to make each test self-contained, but I haven't analyzed all the failures yet. -- Cyril Hrubis ch...@su... |
From: <ch...@su...> - 2013-04-23 13:47:51
|
> I tried to change the generate-makefiles.sh script, but there are some > problems. > > Some tests are shell scripts that doesn't have build phase. I've > workarounded them by adding build rules that just copy them to > name_x-y.sh in the build phase, which unfortunatelly haven't fixed all > the problems. > > We have quite quite a lot of tests (~270) that are shell scripts that > prints "This is tested implicitly via assertion xyz" and returns zero. > I'm quite tempted to remove them. These just increase tests run time, > log sizes, etc. but doesn't do any actual work. > > Then there are more problematic ones, shell tests that calls one binary > with different parameters. As in sigignore for example. The paths and > names are obviously hardcoded, which makes them fail. The fix for such > cases will depend case by case but what I will probably do is to make > each test self-contained, but I haven't analyzed all the failures yet. Good news everyone, I've just commited a patch that modifies the script that generates the Makefiles (since I've fixed all the issues that blocked this change). Now the test binaries and scripts are prefixed with the directory such as 'sigaction_'. I've did some testruns and everything seems fine. Please tests. -- Cyril Hrubis ch...@su... |
From: Jan S. <jst...@re...> - 2013-04-25 08:36:57
|
----- Original Message ----- > From: ch...@su... > To: "Wanlong Gao" <gao...@cn...> > Cc: ltp...@li... > Sent: Tuesday, 23 April, 2013 3:48:45 PM > Subject: Re: [LTP] open posix testsuite and names of testcases > > > I tried to change the generate-makefiles.sh script, but there are some > > problems. > > > > Some tests are shell scripts that doesn't have build phase. I've > > workarounded them by adding build rules that just copy them to > > name_x-y.sh in the build phase, which unfortunatelly haven't fixed all > > the problems. > > > > We have quite quite a lot of tests (~270) that are shell scripts that > > prints "This is tested implicitly via assertion xyz" and returns zero. > > I'm quite tempted to remove them. These just increase tests run time, > > log sizes, etc. but doesn't do any actual work. > > > > Then there are more problematic ones, shell tests that calls one binary > > with different parameters. As in sigignore for example. The paths and > > names are obviously hardcoded, which makes them fail. The fix for such > > cases will depend case by case but what I will probably do is to make > > each test self-contained, but I haven't analyzed all the failures yet. > > > Good news everyone, I've just commited a patch that modifies the script > that generates the Makefiles (since I've fixed all the issues that > blocked this change). > > Now the test binaries and scripts are prefixed with the directory such > as 'sigaction_'. I've did some testruns and everything seems fine. > > Please tests. I think I didn't see these ones failing before: Failed testcases: stress/mqueues/mqueues_multi_send_rev_1 stress/mqueues/mqueues_multi_send_rev_2 from logfile.stress-test: stress/mqueues/mqueues_multi_send_rev_1: execution: FAILED: Output: Usage: ./mqueues_multi_send_rev_1.run-test number_of_threads stress/mqueues/mqueues_multi_send_rev_2: execution: FAILED: Output: Usage: ./mqueues_multi_send_rev_2.run-test number_of_threads Tweaking bin/run-tests.sh or just renaming those ".args" files should do the trick. # find . | grep .args ./stress/mqueues/multi_send_rev_1.args ./stress/mqueues/multi_send_rev_2.args Regards, Jan > > -- > Cyril Hrubis > ch...@su... > > ------------------------------------------------------------------------------ > Try New Relic Now & We'll Send You this Cool Shirt > New Relic is the only SaaS-based application performance monitoring service > that delivers powerful full stack analytics. Optimize and monitor your > browser, app, & servers with just a few lines of code. Try New Relic > and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr > _______________________________________________ > Ltp-list mailing list > Ltp...@li... > https://lists.sourceforge.net/lists/listinfo/ltp-list > |
From: <ch...@su...> - 2013-04-25 10:41:45
|
Hi! > I think I didn't see these ones failing before: > Failed testcases: > stress/mqueues/mqueues_multi_send_rev_1 > stress/mqueues/mqueues_multi_send_rev_2 > > from logfile.stress-test: > stress/mqueues/mqueues_multi_send_rev_1: execution: FAILED: Output: > Usage: ./mqueues_multi_send_rev_1.run-test number_of_threads > stress/mqueues/mqueues_multi_send_rev_2: execution: FAILED: Output: > Usage: ./mqueues_multi_send_rev_2.run-test number_of_threads > > Tweaking bin/run-tests.sh or just renaming those ".args" files should do the trick. > > # find . | grep .args > ./stress/mqueues/multi_send_rev_1.args > ./stress/mqueues/multi_send_rev_2.args Thanks I've missed this. Given that these are only two occurencies I've commited a patch that just renames the arg files. -- Cyril Hrubis ch...@su... |