Menu ▾ ▴

#3346 Race condition in test suite

closed-fixed
Felix
None
other
5
2022-07-13
2022-03-08
Felix
No

There is a bug in the test suite that leads to failed tests. I tried [r13049].

Steps to reproduce (after compilation):

  1. $ cd support/regression
  2. remove .SILENT in Makefile (so we get useful output).
  3. $ make clean # just in case.
  4. make test-port PORT=host -j10 |& grep timeout

What should happen? tests on host should pass.
What happens? tests randomly fail. output can be random, but similar to

[..]
support/regression $ make test-port PORT=host -j10 |& grep timeout
gcc -pipe -ggdb -g -O2 ../../../support/regression/fwk/lib/timeout.c -o
gen/timeout
gcc -pipe -ggdb -g -O2 ../../../support/regression/fwk/lib/timeout.c -o
gen/timeout
gcc -pipe -ggdb -g -O2 ../../../support/regression/fwk/lib/timeout.c -o
gen/timeout
gcc -pipe -ggdb -g -O2 ../../../support/regression/fwk/lib/timeout.c -o
gen/timeout
gcc -pipe -ggdb -g -O2 ../../../support/regression/fwk/lib/timeout.c -o
gen/timeout
gcc -pipe -ggdb -g -O2 ../../../support/regression/fwk/lib/timeout.c -o
gen/timeout
gen/timeout 1 gen/host/abs/abs.bin > gen/host/abs/abs.out
/bin/sh: 1: gen/timeout: Permission denied
gen/timeout 1 gen/host/absolute/absolute_mem___xdata.bin >
gen/host/absolute/absolute_mem___xdata.out
/bin/sh: 1: gen/timeout: Permission denied
gen/timeout 1 gen/host/absolute/absolute_mem___code.bin >
gen/host/absolute/absolute_mem___code.out
/bin/sh: 1: gen/timeout: Permission denied
gen/timeout 1 gen/host/addrspace/addrspace.bin >
gen/host/addrspace/addrspace.out
gen/timeout 1 gen/host/and_survive/and_survive.bin >
gen/host/and_survive/and_survive.out
gen/timeout 1 gen/host/assert/assert.bin > gen/host/assert/assert.out
/bin/sh: 1: gen/timeout: Text file busy
[..]

There is more nondeterminism possibly due to broken rules. Some is harder to track down. I will tidy up the rules (hopefully fixing it all) and I intend to add more tests from other compilers soon.

Discussion

  • Benedikt Freisen

    Benedikt Freisen - 2022-03-08
    • Description has changed:

    Diff:

    --- old
+++ new
@@ -1,13 +1,16 @@
-There is a bug in the test suite that leads to failed tests. I tried r13049.
+There is a bug in the test suite that leads to failed tests. I tried [r13049].

 Steps to reproduce (after compilation):
-1. $ cd support/regression
-2. remove .SILENT in Makefile (so we get useful output).
-3.  $ make clean # just in case.
-4.  make test-port PORT=host -j10 |& grep timeout
+
+1. `$ cd support/regression`
+2. `remove .SILENT in Makefile (so we get useful output).`
+3.  `$ make clean # just in case.`
+4.  `make test-port PORT=host -j10 |& grep timeout`

 What should happen? tests on host should pass.
 What happens? tests randomly fail. output can be random, but similar to
+
+```text
 [..]
 support/regression $ make test-port PORT=host -j10 |& grep timeout
 gcc -pipe -ggdb -g -O2 ../../../support/regression/fwk/lib/timeout.c -o
@@ -37,5 +40,6 @@
 gen/timeout 1 gen/host/assert/assert.bin > gen/host/assert/assert.out
 /bin/sh: 1: gen/timeout: Text file busy
 [..]
+```

 There is more nondeterminism possibly due to broken rules. Some is harder to track down. I will  tidy up the rules (hopefully fixing it all) and I intend to add more tests from other compilers soon.
     

  • Felix

    Felix - 2022-03-16

    Another issue in the test rules: if a test fails to compile, then the summary is broken. I tried r13246.
    Is this (not) a bug?

     

    • Philipp Klaus Krause

      Philipp Klaus Krause - 2022-03-16

      We've had similar bugs before. Two ([bugs:#2824] and [bugs:#2728]) are still open, though I don't know the current state on them.

       

      Related

      Bugs: #2728
      Bugs: #2824

  • Felix

    Felix - 2022-07-13
    • status: open --> closed-fixed
    • assigned_to: Felix
     

  • Felix

    Felix - 2022-07-13

    The timeout race is fixed with the regtest rework. Will open another ticket if anything shows up...

     

Log in to post a comment.

Monday.com Logo