[OctDev] parallel processing in Octave

[Jaroslav H. <hi...@gm...>]

hi all,

in case anyone is interested, I committed today into the "general"
package an initial m-file implementation of parcellfun.
parcellfun is supposed to be able to evaluate a given function for
multiple sets of input arguments using multiple processes.

Given N, the function spawns N subprocesses using fork (), and creates
2*N+1 pipes to communicate with them (actually the pipes come first,
but you knew that). Therefore, it should be (in theory) portable to
any Unix system. Most suitable for systems like GNU/Linux, where fork
() is efficient and pipes are a relatively cheap resource. (Dunno
about Windoze, for example).
The function also depends on two assistant compiled functions, fload
and fsave, that can save/load any Octave variable to/from a binary
stream.

a short demo follows (pseudoinversion of 100 400x400 matrices):

n = 400;
m = 100;
disp ("create 100 random 400x400 matrices");
a = rand (n, n, m);
a = mat2cell (a, n, n, ones (1, m));
a = a(:);
disp ("calculate pseudoinverses - uniprocess");
tic;
p = cellfun (@pinv, a, "UniformOutput", false);
toc
clear p
disp ("calculate pseudoinverses - multiprocess");
tic;
p = parcellfun (2, @pinv, a);
toc

on my Core 2 Duo @2.83 GHz machine, using two processes, I get:

create 100 random 400x400 matrices
calculate pseudoinverses - uniprocess
Elapsed time is 55.6268 seconds.
calculate pseudoinverses - multiprocess
parcellfun: 100/100 jobs done
Elapsed time is 30.3612 seconds.

which amounts to some 91% of theoretical peak scalability (and would
be quite nice if it always worked so).
If you have a multithreaded BLAS, remember to switch multithreading
off before testing.

The code uses dynamic scheduling of individual jobs corresponding to
single cells in the input cell arrays.
If the scheduling overhead is not negligible compared to processing a
single input, data should be partitioned
into bigger chunks in advance.

Feedback is much welcome.

enjoy

-- 
RNDr. Jaroslav Hajek
computing expert
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz