Re: [q-lang-users] nutty idea?

[Tim H. <q...@st...>]

Albert Graef <Dr....@t-...> writes:

> Tim Haynes wrote:
>
> > Been pondering. Over in the land of erlang, I gather they have ability
> > to distribute computation all over a cluster/network. Seems quite nice:
> 
> Hi Tim,
> 
> actually I've been tossing around the idea of an MPI module for a while
> now. So I don't think it's a nutty idea at all. Having taken a brief look
> at PVM, it seems even better suited for this kind of thing.

Ah, excellent :) 
 
> As Q already has all the stuff needed for communicating over sockets, you
> can even do something like this right now. See dgram.q in the core
> distribution for a (very simple) example. This uses Q's builtin
> text-based serialization (str/val) for communicating Q values over a
> socket. Of course this is not ideal; 

Yes, I was wondering about evolving something usable from a network-repl
bit of code upwards.

> it's ok for simple applications, but for passing around big chunks of Q
> data we certainly need a binary serialization protocol which uses network
> byte order for the numbers. Something along the lines of Peter Minten's
> binary marshalling library (written in Q), but implemented in C for
> maximum efficiency.

Oh, that's a good idea. Slipped the mind, oops..

> And then, as you point out below, we need a decent infrastructure for
> distributing the tasks. PVM might be the way to go there.

Ah, enlightenment dawns, I think: I'm comparatively used to PVM being
implemented a layer under what I'm trying to do - my experience is greatest
with POV-ray, where it uses PVM to say "you, go render the top-left block
of the image", and so on - but my interface to pvm-povray is just the same
as to povray itself, apart from setting up all the boxes to run the
utility. Hence I was thinking of PVM as taking typical chunks of Q
interpreter code - `run the parser over here' versus `evaluate it over
there'. Obviously, it doesn't need to be so :)

> > * a connectivity daemon, * network protocol for conveying requests &
> > results back & forth, * a system for identifying a node
> > (`tim@somebox'), * a shared-secret file (a la .Xauthority) for
> > security, * node-availability management, and * an easy syntax for
> > remoteifying expression-evaluation.
> 
> > Guess what: I'm about to ask how hard it would be to slam all this into
> > a world-dominating future release of Q. :)
> 
> Well, as I pointed out above, IMHO nothing needs to be changed inside the
> interpreter to implement all this (minus the special syntax, but I don't
> think that this is really needed). 

Well, over in erlang it looks like almost any other function-call, to me -
it just takes a node, a lambda and some parameters as its args and makes
the whole evaluation happen "over there". So it's not "special" in that
regard at all; FFI-ing in PVM with SWIG would be a good start.

> What we need is (1) a serialization module and (2) a wrapper for PVM.
> With SWIG (2) is probably quite easy. (1) might be a bit tricky, since we
> need a secure way to pass around function symbols between different
> scripts running on different machines.

Yup.

> Given these two things, the rest of the required functionality which is
> not provided by PVM (authentication, maybe?) could then also be
> programmed in Q.

PVM operates over rsh/ssh, so I doubt authentication will be a problem :)

> > I'm also wondering if PVM could help out, but unless the nature of Q
> > lends itself to chunks of interpretation being carried-out on different
> > machines at an internal level, I'd doubt it was that much use.
> 
> Why would you need that kind of special support from the interpreter?
> We're not talking about fine-grained parallelism here. 

As long as we're talking ability to take Q expressions, prefix them with a
function call citing a node on which to be evaluated and off it goes, this
is cool. Something like, perhaps:

    randnode = Ns ! (random mod #Ns) 
                 where Ns=remote_nodes;

    remotetakfp = spawn_link randnode takfp ;

    takfp (X, Y, Z) = Z if Y>=X;
                = takfp (
                   remotetakfp (X-1.0, Y, Z),
                   remotetakfp (Y-1.0, Z, X),
                   remotetakfp (Z-1.0, X, Y)) 
                     otherwise;

    takstart N = takfp (N*3.0, N*2.0, N*1.0);

I should probably consider it a mini-project to implement something like
the above in terms of dgram.q, I think. Data is simple enough, at least.

> As I imagine it, in Q you'd just call the PVM functions, like you'd do in
> a C program. You'd pass around serialized Q data in PVM buffers to send
> the jobs (e.g., unevaluated Q expressions) and receive the results (e.g.,
> the evaluated expressions). Pretty much like in the dgrams.q example.
> Using these primitives, it should be quite easy to implement any kind of
> remote_eval function that you can think of.

Sounds like it'll be a bit of work to abstract it up to the level above :)

> Any volunteers? ;-)

Now what have I got into? If it were down to me, it'd be yeeeeears :/ 
Help! :)

~Tim
-- 
<http://spodzone.org.uk/>