#10 Multithreaded RNG_test gets stuck

[Sebastiano Vigna]

I'm testing a number of generators using the "builtin" ("dummy") generator and options

-tf 2 -te 1 -tlmaxonly -multithreaded

After about five days of computation, out of ten processes only two are using CPU. The other eight are stuck at 0.3% CPU usage, and they are not outputting anything since four days. They last output is at 2TB, whereas the only two really running processes are at 16TB.

This appears to be a serious thread synchronization issue in the multithreaded code.

Seems likely. What OS was this on?

[Sebastiano Vigna]

Linux.

Here (our server):

Linux 4.10.14-200.fc25.x86_64 #1 SMP Wed May 3 22:52:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Intel(R) Xeon(R) CPU E7- 4870 @ 2.40GHz
gcc version 6.3.1 20161221 (Red Hat 6.3.1-1) (GCC)

And here (AWS instance):

Linux 4.9.51-10.52.amzn1.x86_64 #1 SMP Fri Sep 29 01:16:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
gcc version 6.4.1 20170727 (Red Hat 6.4.1-1) (GCC)

Although... how are you measuring the 16 TB thing?

The worker threads are supposed to never be able to get more than 128 MB away from each other in terms of how much data they've tested. However, that's measured against the base datastream, not the foldings. If the main thread has finished generating the next set of random numbers it should block waiting on a worker thread in TestThread::retire. If a worker thread has finished testing a set of random numbers, it should simply exit - worker threads should almost never block. If active testing is still occurring in at least one thread, that suggests that either one or more test is taking forever to handle just 128 MB, or the main thread had reason to believe that all tests (it considers an entire folding as a single test) had tested the same amount (measured against the unfolded data) at some point semi-recently.

So I'm wondering about other possibilities, like a test or two deciding to use infinite CPU time on a finite amount of data.