#22 sim4db: worker starvation when > 2 threads used

[Nathan Weeks]

NOTE: this bug report assumes that the patch to splice.C described in http://sourceforge.net/tracker/?func=detail&aid=3313766&group_id=185357&atid=912949 which allows more than two worker threads to be used, has been applied.

When running "sim4db -v -threads 46", the status stream reported to stderr showed alignments rate ramped up to about 7.5/s. After 16384 alignments had
been completed, the "queued for compute" value started to drop off, until it alternated between 2 and 3:
########################################
7.5/s - 17407 loaded; 1024 queued for compute; 16383 finished; 16212 written; 171 queued for output)
...
0.4/s - 36546 loaded; 3 queued for compute; 36543 finished; 36543 written; 0 queued for output)
########################################

This caused only 2-3 worker threads to be active at any one time, resulting in a significant performance degradation.

The problem was traced to sweatShop.C, which starts dynamically adjusting the queue size after 16384 alignments. Here are values of relevant variables in the "while(_writerP)" loop in sweatShop.C, lines 371-405, during a period of performance degradation:

########################################
(gdb) print readjustAt
$16 = 35111
(gdb) print _numberComputed
$17 = 39451
(gdb) print cpuPerSec
$18 = 0.35687335713397383
(gdb) print _loaderQueueSize
$20 = 1
########################################

As a workaround, the attached patch (sweatShop.C.patch) ensures that the _loaderQueueSize never falls below 2*_numberOfWorkers. While this may not be an ideal size in all cases, it was sufficient in this case to prevent worker-thread starvation. Perhaps the dynamic-queue-size-adjustment code should be revisited.

[Nathan Weeks]

patch to sweatShop.C (sim4db-r1896)