Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

#1916 Parallel regression tests

closed-fixed
Erik Petrich
other
5
2014-04-27
2012-01-22
No

When trying to run the regression tests in parallel using make -j 3 (or some other number higher than 1) I get errors like the following:

Running host regression tests
gen/host/extern1.o: file not recognized: File truncated
collect2: ld returned 1 exit status
make[3]: *** [gen/host/abs/abs.bin] Fehler 1
make[2]: *** [results/host/abs.out] Fehler 2
make[2]: *** Warte auf noch nicht beendete Prozesse...
absolute (f: 0, t: 0, c: 4, b: 0, t: 0)
addsub (f: 0, t: 60, c: 12, b: 0, t: 0)
make[1]: *** [test-port] Fehler 2
making huge library
make[2]: Warnung: Kein Jobserver verfügbar: -j1 wird gesetzt. »+« zur Ursprungsregel
hinzufügen.

or

/bin/sh: 1: cannot create gen/compare_2_n.c: Directory nonexistent
make[1]: *** [gen/compare_2_n.c] Fehler 2
make[1]: *** Warte auf noch nicht beendete Prozesse...
make: *** [test-ucgbz80] Fehler 2

The errors do not always occour though.

I suspect there's something wrong with the Makefile dependencies, introducing a race condition.

Since computers geet more cores and sdcc gets more regression tests, fixing this bug would help a lot in speeding up the regression testing for developers.

Philipp

P.S.: Here's one with LANG=C:

make -j 3
/bin/sh: 1: cannot create gen/compare_2_n.c: Directory nonexistent
make[1]: *** [gen/compare_2_n.c] Error 2
make[1]: *** Waiting for unfinished jobs....
Running hc08 regression tests

While it is not clear from the error message, the bug caused us to skip over the ds390 tests and proceed to hc08 here.

Discussion

  • Erik Petrich
    Erik Petrich
    2012-03-24

    I believe I have this fixed (parallel recursive makes attempting to simultaneously resolve the same dependency) with revision #7481. However, just because the regression tests run in parallel now on my computer without errors does not mean the Makefiles are error free. So instead of closing this item, I've set it to "Pending". It will automatically convert to "Closed" after awhile if no one adds a follow-up comment. Add a comment if you still see any problems with running regression tests in parallel.

     
  • Erik Petrich
    Erik Petrich
    2012-03-24

    • milestone: --> fixed
    • assigned_to: nobody --> epetrich
    • status: open --> pending-fixed
     
    • status: pending-fixed --> open-fixed
     
  • The situation is much better now. I can run parallel make practically all the time without issues, however, while they are much rarer, I just got one:

    philipp@phoinikas:~/sdcc/support/regression$ make clean
    philipp@phoinikas:~/sdcc/support/regression$ make -j 5 test-ucgbz80
    /bin/sh: 1: cannot create gen/compare_2_n.c: Directory nonexistent
    make[1]: *** [gen/compare_2_n.c] Fehler 2
    make[1]: *** Warte auf noch nicht beendete Prozesse...
    /usr/bin/ld: cannot open output file gen/timeout: No such file or directory
    collect2: ld returned 1 exit status
    make[1]: *** [gen/timeout] Fehler 1
    make: *** [test-ucgbz80] Fehler 2
    philipp@phoinikas:~/sdcc/support/regression$ make clean
    philipp@phoinikas:~/sdcc/support/regression$ make test-ucgbz80
    Running ucgbz80 regression tests
    abs (f: 0, t: 6, c: 1, b: 3957, t: 7379)
    absolute (f: 0, t: 20, c: 4, b: 8122, t: 20648

     
  • Maarten Brock
    Maarten Brock
    2012-03-29

    There is also a small issue with the final summary. Because it is not printed in one instruction the output is mixed with other results.

    Btw. Erik, how did you get the output to stop from being concatenated on a single line? Was it the split of 'cat in the Makefile?

     
  • Erik Petrich
    Erik Petrich
    2012-03-29

    In the old version of the Makefile the summary was generated by:
    echo Summary for \'$(PORT)\': `cat $(PORT_RESULTS) | python $(srcdir)/collate-results.py`
    The part quoted by the back-ticks is executed by the shell and any output is substituted back, except the newline characters are stripped. It's possible to do some pre- and post- filtering to hide the newlines from the shell and put them back in later, but it seemed like a really ugly solution. In the current version of the Makefile, collate-results.py's output is going directly to standard output rather than being filtered and substituted as parameters for echo.

    I've noticed that a few systems in the DCF don't understand echo -n, so I was planning on moving the "Summary for ..." into collate-results.py. This should also reduce the likelihood of mixed output in the middle of the summary message.
    .

     
  • Erik Petrich
    Erik Petrich
    2012-04-09

    • status: open-fixed --> pending-fixed
     
  • Erik Petrich
    Erik Petrich
    2012-04-09

    I think I've fixed the remaining problems in revision #7553; if you see anything to the contrary, add a comment.

     
  • Maarten Brock
    Maarten Brock
    2014-04-27

    Can this be closed now after 2 years?

     
  • I haven't encountered the problems since Erik's fix in revision #7553.

    Philipp

     
    • status: pending-fixed --> closed-fixed
    • Category: --> other