From: Mikael S. <mik...@ax...> - 2003-01-24 21:07:27
|
As Paul points out crash0* are intentionally random to create various nasty code and calls. I guess that these tests are usually not run in a daily regression test system (not in ours anyway). But there are also other tests like fsync02 that may generate more or less BROK depending on the random values (the test should of course return 0 BROK in a correct system). Our daily regression tests with LTP returns a different number of PASS every day (but BROK and FAIL are the same). This may be caused by errors in my testscripts or by randomness in the LTP tests. Please tell me if you have the same experience. If eveybody gets the same amount of PASS/FAIL/BROK for every run I know for sure that my scripts are broken. /Mikael -----Original Message----- From: Dan Kegel To: Mikael Starvik Cc: 'ltp...@li...' Sent: 2003-01-24 22:09 Subject: Re: [LTP] Re: results comparator? (Dan Kegel) Mikael Starvik wrote: > There are some testcases that are random in their behaviour > (e.g. crash01 and crash02). IIRC there are also some tests > that runs a random number of testcases and hence generates > different number of PASS/FAIL/BROK. > > It would sure be nice if it would be possible to disable > this randomness to make the results 100% reproducable > (assuming everything outside LTP is exaclty the same). Hey, good point. Sounds like something that will be needed (if it's not already there). - Dan -- Dan Kegel http://www.kegel.com http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045 |
From: William J. H. <wj...@us...> - 2003-01-24 21:09:58
|
the failures.list file simply lists tests which fail. The #'s are stripped out so it wouldn't really be prone to incorrect reporting due to #'s changing. I'd say simply do a run on your target machine and then move that directory to something like "baseline" then in ltp_check change the script to do a diff on what it just pulled down. something like; if ( -d "../baseline" ) { if (system(" cmp ../baseline/$FAILS_OUT $FAILS_OUT")) { print "Results different then baseline\n"; print `diff ../baseline/$FAILS_OUT $FAILS_OUT`,"\n"; } else { print "Test results unchanged\n"; } } else { print "No baseline run for this system yet\n"; print "To make this the baseline run rename this directory to \"baseline\"","\n"; } Something like that, my perl's a little rusty and you could of course make this a bit better if you remember the variables of your perl script. Anyway is this not something that would work for what you'd like? j Jay Huie wj...@us... zSeries Linux System Test Phone: 845-435-8164 |
From: Dan K. <da...@ke...> - 2003-01-24 21:53:00
|
William Jay Huie wrote: > [just use tools/ltp_check, it's a good enough results comparator; > maybe add a diff on the file failures.list that it produces] Hmm. Yeah, maybe ltp_check would do. It's kind of complicated, but then, anything real will be. Vasan, could you look at using ltp_check to fetch and postprocess the ltp results, and then do a diff on its failures.list output file against the baseline? - Dan -- Dan Kegel http://www.kegel.com http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045 |
From: V.R.Sundar <vr_...@at...> - 2003-01-24 23:19:20
|
On Fri, 2003-01-24 at 14:16, Dan Kegel wrote: > William Jay Huie wrote: > > [just use tools/ltp_check, it's a good enough results comparator; > > maybe add a diff on the file failures.list that it produces] > > Hmm. Yeah, maybe ltp_check would do. It's kind of complicated, > but then, anything real will be. > > Vasan, could you look at using ltp_check to fetch and postprocess > the ltp results, and then do a diff on its failures.list output > file against the baseline? yea, i started looking into them after William's last mail. One quick question. How do i create the log file that ltp_check uses. I noticed that ltp_check greps for "stat=" lines. runalltests -l ltp.log did not create a ltp.log anywhere ( i did a find from /) . And the stuff that it prints out to stdout don't contain stat= lines. Did i miss something? The stdout output contains termination_id= lines which i guess i can use for now. These look like the exit status of that particular testcase > - Dan -- V.R.Sundar <vr_...@at...> |
From: Mikael S. <mik...@ax...> - 2003-01-24 21:16:46
|
Sorry, I haven't investigated that further. I am happy as long as I get PASS :-) This also imply that I agree that the main goal for a tool should be to point out differences in BROK/FAIL. >Could that be caused by programs that crash without >reporting any result? I was going to consider lack >of 'PASS' as a different kind of FAIL, more or less. IIRC there are some tests that doesn't return a textual result but exits with an exitcode to indicate PASS/FAIL/BROK. I can't remember which tests that have this behaviour but I know that my testscripts checks for it. /Mikael -----Original Message----- From: Dan Kegel To: Mikael Starvik Cc: ''ltp...@li...' ' Sent: 2003-01-24 22:33 Subject: Re: [LTP] Re: results comparator? (Dan Kegel) Mikael Starvik wrote: > Our daily regression tests with LTP returns a different > number of PASS every day (but BROK and FAIL are the > same). This may be caused by errors in my testscripts > or by randomness in the LTP tests. Could that be caused by programs that crash without reporting any result? I was going to consider lack of 'PASS' as a different kind of FAIL, more or less. Which programs generate a varying number of PASS's in your experience? - Dan -- Dan Kegel http://www.kegel.com http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045 |
From: Paul L. <pl...@li...> - 2003-01-24 22:03:56
|
On Fri, 2003-01-24 at 15:16, Mikael Starvik wrote: > IIRC there are some tests that doesn't return a=20 > textual result but exits with an exitcode to > indicate PASS/FAIL/BROK. I can't remember which > tests that have this behaviour but I know that > my testscripts checks for it.=20 In every case, tests should be returning 0 for pass, anything else for fail, brok, etc. If this isn't happening, it's buggy. I think what you are referring to though, is the fact that some tests don't adhere to the LTP test harness api. So, there are a few tests where you won't ever actually see them report nice pretty messages like FAIL, PASS, BROK, etc. These should be fixed where possible because it would make things like this so much simpler. The ltpoutput script that I wrote just cared about detecting failures and looks at the output from ltp for places where the termination_id !=3D 0. This was the only safe way to detect failures. -Paul Larson |
From: William J. H. <wj...@us...> - 2003-01-27 18:27:37
|
There's a README file in think in the tools directory that kind of explains it w/ ascii drawings. Basically even if you don't use the ltp_master script to do the run you can "by hand" use "ltprun". I'd simply log into a system and do; nohup ./ltprun &> ltprun.out & Follow ltprun for a description of what happens but it does all the bulding and so forth (if I want to do repeat runs sometimes I just comment the fetching/building of a ltp tarball out since it's already been done). ltprun calls runalltests.sh with the appropriate parameters. ./runalltests.sh -l ~/ltp/$LTP_LOGFILE &> $LTP_RUNALL_OUT The real "magic" that needs to be done for ltpcheck is to create the tarball that gets processed. That tarball has the ltp-logfile (which is the 1 line rundown for each test) along with the output from runalltests.sh The output from ltprun (ltprun.out) gets included but it's not necessary for processing it's used mostly to diagnose something going wrong _before_ ltp runs. i.e. to see if ltp didn't build for example. j Jay Huie wj...@us... zSeries Linux System Test Phone: 845-435-8164 To: Dan Kegel <da...@ke...> cc: William Jay Huie/Poughkeepsie/IBM@IBMUS, ltp...@li... Subject: Re: [LTP] Re: results comparator? (Dan Kegel) On Fri, 2003-01-24 at 14:16, Dan Kegel wrote: > William Jay Huie wrote: > > [just use tools/ltp_check, it's a good enough results comparator; > > maybe add a diff on the file failures.list that it produces] > > Hmm. Yeah, maybe ltp_check would do. It's kind of complicated, > but then, anything real will be. > > Vasan, could you look at using ltp_check to fetch and postprocess > the ltp results, and then do a diff on its failures.list output > file against the baseline? yea, i started looking into them after William's last mail. One quick question. How do i create the log file that ltp_check uses. I noticed that ltp_check greps for "stat=" lines. runalltests -l ltp.log did not create a ltp.log anywhere ( i did a find from /) . And the stuff that it prints out to stdout don't contain stat= lines. Did i miss something? The stdout output contains termination_id= lines which i guess i can use for now. These look like the exit status of that particular testcase > - Dan -- V.R.Sundar <vr_...@at...> |
From: V.R.Sundar <vr_...@at...> - 2003-01-28 22:44:01
|
On Mon, 2003-01-27 at 10:27, William Jay Huie wrote: > <snip> > ltprun calls runalltests.sh with the appropriate parameters. > ./runalltests.sh -l ~/ltp/$LTP_LOGFILE &> > $LTP_RUNALL_OUT > <snip> Ok. Found the difference. I was running it as: ./runalltests.sh -l ltp.log The difference being you were giving a full path. The script runalltests.sh creates and moves to a temp directory before running pan and then deletes it at the end. So the log file was created there and then deleted. > > > To: Dan Kegel <da...@ke...> > cc: William Jay Huie/Poughkeepsie/IBM@IBMUS, > ltp...@li... > Subject: Re: [LTP] Re: results comparator? (Dan Kegel) > > > > On Fri, 2003-01-24 at 14:16, Dan Kegel wrote: > > William Jay Huie wrote: > > > [just use tools/ltp_check, it's a good enough results comparator; > > > maybe add a diff on the file failures.list that it produces] > > > > Hmm. Yeah, maybe ltp_check would do. It's kind of complicated, > > but then, anything real will be. > > > > Vasan, could you look at using ltp_check to fetch and postprocess > > the ltp results, and then do a diff on its failures.list output > > file against the baseline? > > yea, i started looking into them after William's last mail. > > One quick question. How do i create the log file that ltp_check uses. > I > noticed that ltp_check greps for "stat=" lines. runalltests -l ltp.log > did not create a ltp.log anywhere ( i did a find from /) . And the > stuff that it prints out to stdout don't contain stat= lines. Did i > miss > something? > > The stdout output contains termination_id= lines which i guess i can > use > for now. These look like the exit status of that particular testcase > > > - Dan > -- > V.R.Sundar <vr_...@at...> -- V.R.Sundar <vr_...@at...> |
From: Dan K. <da...@ke...> - 2003-01-28 22:57:03
|
V.R.Sundar wrote: > On Mon, 2003-01-27 at 10:27, William Jay Huie wrote: > >><snip> >> ltprun calls runalltests.sh with the appropriate parameters. >> ./runalltests.sh -l ~/ltp/$LTP_LOGFILE &> >>$LTP_RUNALL_OUT >><snip> > > > Ok. Found the difference. I was running it as: > ./runalltests.sh -l ltp.log > The difference being you were giving a full path. The script > runalltests.sh creates and moves to a temp directory before running pan > and then deletes it at the end. So the log file was created there and > then deleted. It might be helpful to add a check in runalltests.sh to abort if -l is given a nonabsolute path, e.g. case $log in */*) ;; *) echo "absolute path required"; exit 1 ;; esac (Alternately, to use crap like logdir=`dirname $log` logdir=`cd $logdir; pwd` log=$logdir/`basename $log` to convert the logfile name to an absolute path... that's going a bit too far, though...) -- Dan Kegel http://www.kegel.com http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045 |
From: Paul L. <pl...@li...> - 2003-01-28 23:27:10
|
On Tue, 2003-01-28 at 17:21, Dan Kegel wrote: > It might be helpful to add a check in runalltests.sh to abort if > -l is given a nonabsolute path, e.g. Good idea, I've seen a few people hit this before not realizing until after the run was over what they had forgotten. However the way you did it won't necessarily work. What if someone does a -l results/logfile? This will still be wrong, but that case statement will think it's ok.=20 Instead I just did a compare on the first character of $OPTARG. Thanks, Paul Larson |
From: V.R.Sundar <vr_...@at...> - 2003-01-28 23:40:40
|
On Tue, 2003-01-28 at 15:25, Paul Larson wrote: > On Tue, 2003-01-28 at 17:21, Dan Kegel wrote: > > > It might be helpful to add a check in runalltests.sh to abort if > > -l is given a nonabsolute path, e.g. > Good idea, I've seen a few people hit this before not realizing until > after the run was over what they had forgotten. However the way you did > it won't necessarily work. What if someone does a -l results/logfile? > This will still be wrong, but that case statement will think it's ok. > Instead I just did a compare on the first character of $OPTARG. probably also need the same for the -f option. Also it might be a good idea to specify this in the usage help in runalltest.sh. > > Thanks, > Paul Larson -- V.R.Sundar <vr_...@at...> |
From: William J. H. <wj...@us...> - 2003-01-28 23:34:21
|
Yea, It's tricky I found that out (thus the absolute path). I expect the more "rational" thing might be to have pan open the log file(s) before changing to it's tmp directory. j Jay Huie wj...@us... zSeries Linux System Test Phone: 845-435-8164 Sent by: ltp...@li... To: "V.R.Sundar" <vr_...@at...> cc: ltp...@li... Subject: Re: [LTP] Re: results comparator? (Dan Kegel) V.R.Sundar wrote: > On Mon, 2003-01-27 at 10:27, William Jay Huie wrote: > >><snip> >> ltprun calls runalltests.sh with the appropriate parameters. >> ./runalltests.sh -l ~/ltp/$LTP_LOGFILE &> >>$LTP_RUNALL_OUT >><snip> > > > Ok. Found the difference. I was running it as: > ./runalltests.sh -l ltp.log > The difference being you were giving a full path. The script > runalltests.sh creates and moves to a temp directory before running pan > and then deletes it at the end. So the log file was created there and > then deleted. It might be helpful to add a check in runalltests.sh to abort if -l is given a nonabsolute path, e.g. case $log in */*) ;; *) echo "absolute path required"; exit 1 ;; esac (Alternately, to use crap like logdir=`dirname $log` logdir=`cd $logdir; pwd` log=$logdir/`basename $log` to convert the logfile name to an absolute path... that's going a bit too far, though...) -- Dan Kegel http://www.kegel.com http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045 ------------------------------------------------------- This SF.NET email is sponsored by: SourceForge Enterprise Edition + IBM + LinuxWorld = Something 2 See! http://www.vasoftware.com _______________________________________________ Ltp-list mailing list Ltp...@li... https://lists.sourceforge.net/lists/listinfo/ltp-list |
From: V.R.Sundar <vr_...@at...> - 2003-01-29 00:22:22
Attachments:
compare.pl
|
Hi, Attached is a simple perl script I wrote yesterday with input from Dan for comparing two results. It uses the output from stdout. It looks for termination_id for a test, compares it with the base and prints out a list of tests that have changed. Generate the output something like this: $ ./runalltests.sh >baselog 2>&1 & Generate two outputs, one the baseline and the other the new output that you want to check and compare them like this: $./compare.pl baseline newoutput Below is the output from comparing a ltp run on my x86 with one on the ppc 405 system. Number of new fails: 201 access02 access04 access05 adjtimex02 ar chdir03 chdir04 chmod03 chmod04 chmod05 chmod06 chmod07 chown03 chown04 chroot01 chroot02 chroot03 chroot04 creat01 creat04 creat06 delete_module02 dio01 dio02 dio03 dio05 dio06 dio07 dio08 dio09 dio11 dio12 dio13 dio14 dio15 dio16 dio17 dio18 dio19 dio20 dio21 dio22 dio23 dio24 dio25 dio26 dio27 dio28 execve02 execve03 fchdir03 fchmod02 fchmod03 fchmod04 fchmod05 fchmod06 fchown03 fchown04 fcntl18 float_bessel float_exp_log float_power float_trigo fptest03 fs_inod01 fstat02 fstat04 fstat05 getgroups03 gethostid01 getresgid02 getresgid03 getresuid02 getresuid03 gf01 gf10 gf11 gf12 gf13 gf14 gf15 gf16 gf17 gf18 gf27 gf28 gf29 hangup01 inode02 ioctl01 ioctl02 kill05 ld ldd linker01 lstat02 lstat03 mem02 mkdir02 mkdir03 mkdir04 mkdir05 mknod02 mknod03 mknod04 mknod05 mknod07 mknod08 mlockall02 mm01 msgctl05 munlockall02 nice04 objdump open05 open08 prctl01 pth_str02 pty01 readlink01 readlink03 readlink04 readv02 reboot02 rename03 rename09 rename10 rename12 rmdir03 rwtest01 rwtest02 rwtest03 rwtest04 rwtest05 sched_setparam05 sched_setscheduler02 semctl02 semctl04 semctl06 semget02 semop02 sendfile02 sendfile03 setdomainname03 setgid02 setgid03 setgroups02 setgroups03 sethostname03 setpriority02 setpriority05 setregid02 setregid03 setregid04 setresgid01 setresgid02 setresgid03 setresuid01 setresuid02 setresuid03 setreuid02 setreuid03 setreuid04 setreuid05 setreuid06 setrlimit01 setrlimit02 settimeofday02 setuid03 shmat02 shmat03 shmctl02 shmctl03 shmget04 shmget05 sigsuspend01 sockioctl01 stat01 stat02 stat03 statfs02 statfs03 stime02 swapoff01 swapoff02 swapon01 swapon02 symlink03 sysctl03 syslog11 syslog12 tar truncate03 truncate04 utime02 utime03 utime05 utime06 vhangup01 waitpid09 write02 Number of new passes: 6 msgsnd04 msgsnd05 msgsnd06 personality01 pread02 pwrite02 On Tue, 2003-01-28 at 15:34, William Jay Huie wrote: > Yea, > It's tricky I found that out (thus the absolute path). I > expect the more "rational" thing might be to have pan open the log > file(s) before changing to it's tmp directory. > j > > > Jay Huie > wj...@us... > zSeries Linux System Test > Phone: 845-435-8164 > > > > Sent by: ltp...@li... > > To: "V.R.Sundar" <vr_...@at...> > cc: ltp...@li... > Subject: Re: [LTP] Re: results comparator? (Dan Kegel) > > > > V.R.Sundar wrote: > > On Mon, 2003-01-27 at 10:27, William Jay Huie wrote: > > > >><snip> > >> ltprun calls runalltests.sh with the appropriate parameters. > >> ./runalltests.sh -l ~/ltp/$LTP_LOGFILE &> > >>$LTP_RUNALL_OUT > >><snip> > > > > > > Ok. Found the difference. I was running it as: > > ./runalltests.sh -l ltp.log > > The difference being you were giving a full path. The script > > runalltests.sh creates and moves to a temp directory before running > pan > > and then deletes it at the end. So the log file was created there > and > > then deleted. > > It might be helpful to add a check in runalltests.sh to abort if > -l is given a nonabsolute path, e.g. > > case $log in > */*) ;; > *) echo "absolute path required"; exit 1 ;; > esac > > (Alternately, to use crap like > logdir=`dirname $log` > logdir=`cd $logdir; pwd` > log=$logdir/`basename $log` > to convert the logfile name to an absolute path... that's going a bit > too far, though...) > > -- > Dan Kegel > http://www.kegel.com > http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045 > > > > ------------------------------------------------------- > This SF.NET email is sponsored by: > SourceForge Enterprise Edition + IBM + LinuxWorld = Something 2 See! > http://www.vasoftware.com > _______________________________________________ > Ltp-list mailing list > Ltp...@li... https://lists.sourceforge.net/lists/listinfo/ltp-list -- V.R.Sundar <vr_...@at...> |
From: Paul L. <pl...@li...> - 2003-01-29 15:13:06
|
A couple of suggestions: First, have the whole summary at the top of the report, then the details about what the differences were at the bottom. something like: PASS FAIL BROK CONF ...... ---- ---- ---- ---- Run1 300 5 2 0 Run2 304 3 0 0 -------------------------------------------- Diff: +4 -2 -2 0 ...then list the details below it. Also, consider having some options for things like a name associated with those files you are comparing. So you could name the first one Run1 and the second Run2, or in this case something maybe like "debianx86" and "ppc2.4.16" or something. Another useful option might be to have it give you the detailed output lines for (separate options) passes, fails, broks, etc, or at least the ones that were different. Good work so far, I look forward to seeing the next version! Thanks, Paul Larson |
From: Dan K. <da...@ke...> - 2003-01-29 04:37:53
|
V.R.Sundar wrote: > Below is the output from comparing a ltp run on my x86 with one on the > ppc 405 system. BTW, the x86 was an up-to-date Debian, and the ppc was an ancient 2.4.16 kernel. Presumably the modern ppc kernels are happier. - Dan -- Dan Kegel http://www.kegel.com http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045 |
From: V.R.Sundar <vr_...@at...> - 2003-01-29 06:02:54
|
On Tue, 2003-01-28 at 21:02, Dan Kegel wrote: > V.R.Sundar wrote: > > Below is the output from comparing a ltp run on my x86 with one on the > > ppc 405 system. > > BTW, the x86 was an up-to-date Debian, and the ppc was an ancient > 2.4.16 kernel. Presumably the modern ppc kernels are happier. oh yea.. forgot about the kernel versions. actually the ppc seems to be the newer version. uname -a reports 2.4.14 on the x86 and 2.4.17 on the ppc. (btw i was wondering what the -v option meant. does anyone know?) > - Dan -- V.R.Sundar <vr_...@at...> |
From: Dan K. <da...@ke...> - 2003-01-24 21:10:22
|
Mikael Starvik wrote: > Our daily regression tests with LTP returns a different > number of PASS every day (but BROK and FAIL are the > same). This may be caused by errors in my testscripts > or by randomness in the LTP tests. Could that be caused by programs that crash without reporting any result? I was going to consider lack of 'PASS' as a different kind of FAIL, more or less. Which programs generate a varying number of PASS's in your experience? - Dan -- Dan Kegel http://www.kegel.com http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045 |