[BProc] parallel make using BProc?

[<ha...@no...>]

Is anybody working on parallel GNU make using BProc to offload
processes to slaves?

Should not be hard, GNU make is prepared for this and at least two
opensource projects go this way (just using different transport):

[1] qmake in Grid Engine - allocates certain number of nodes (like PVM
    or MPI does), then works just as GNU "make -j N" but transports
    processes to nodes

[2] ANTS uses normal make -j N, but Makefile is modified - commands 
    are prefixed with "rant" which acts like "bpsh FREENODE"


Of course there are problems, e.g.:

- parallel compilation can be slow because normal file cache cannot
  help as much as it can on one node

- Makefile should operate just on files in NFS-mounted /home

- certain Makefiles do not work with "make -j N"

- flaky NFS can make make's decisions based on existence of files just
  created elsewhere problematic

- there are better ways of resource management, see my (hanzl's) posts in [3]

but I still would have good suitable applications. Even hand-allocation
of free nodes like "bpmake --nodes 1,3,7-10 -- MAKE_OPTIONS" would be
often useful on small clusters and easy to install cause it requires
BProc and nothing else.

I suppose a student given sources of bpsh and qmake would be able to
write bpmake - I will try to start this project here if nobody else
did it yet.

Best Regards

Vaclav Hanzl


References:

[1] http://gridengine.sunsource.net/unbranded-source/browse/~checkout~/gridengine/doc/htmlman/htmlman1/qmake.html

[2] http://unthought.net/antsd/

[3] http://gridengine.sunsource.net/servlets/BrowseList?listName=dev&by=thread&from=85&to=85&first=1&count=7&JServSessionIdservlets=pdmvxixv01