#3 load balancing

[Matthew Blythe]

Currently, distmake appears to use the hosts in "priority" order. That is, if it doesn't have a job dispatched to the first host in the list, it will launch a job there, otherwise, it will try the second host in the list, etc.

This breaks down in a multi-user environment if everyone uses the same host list. The first host in the list will be over-loaded, and the last hosts under-utilized. In my particular use case, it has forced us to create a secondary script that generates a semi-randomized hostlist (which is a subset of the hosts available) based on load averages at the time that we begin a build.

Honestly, doing intelligent load balancing is probably out of the scope of this project. The good news is that there are other open-source projects whose only purpose is to do load balancing (e.g. haproxy). The bad news is that distmake isn't coded up in a way that's particularly friendly to external load-balancers. Specifically, distmake establishes a single TCP connection to each build host that it uses for the duration of execution. Pairing distmake with a load-balancer would do no better than the "randomized hosts" workaround above.

My proposal is to modify distmake to (optionally) create a new TCP connection for each job executed. This would allow a load-balancer to do its job at a much finer grain, and hopefully keep build hosts more evenly loaded.