Re: [Queue-developers] New site at Savannah

[Koni <mh...@co...>]

Hi Bob, thanks for joining up and offering your help. Below is relevant
for the developer list so I am copying this there as well. 

> But reading through some of your initial design postings I see that
> you seem to be concentrating on a completely homogeneous design where
> all of the machines are identical.  That will definitely limit the
> usefulness to me personally.  That is not to say that it won't be
> useful to you or to someone else.  But most compute pools are
> heterogeneous and so this may be less useful to me.  We shall see.
> 

Mike also objected to my previous statements about that, I should
clarify my thinking some. 

The extant GNU queue didn't really do anything special to attempt to
support heterogenous environments or even attempt to be aware of
heterogeneity that I know of, what I have in mind will be no less
supportive of mixed setups than the old GNU queue. From what I see in
the old code GQ wouldn't have even handled distribution correctly
between a PPC and x86 system, because the communication between "queue"
and "queued" used binary formats.

I will at least deal with making sure that new GQ itself can handle
architecture differences when talking to itself across nodes, but
otherwise it seems to me that it invites trouble to the non-programmer
user unless the application being distributed is both programmed
properly to handle input/output generated on other architectures, and
installed correctly and locally on each system. 

Following up on that, we can make the new GQ system itself aware of what
architecture each of the compute nodes are, and allow a user to specify
which architecture(s) are acceptable for a job. That would allow
developers of distributed apps and competent users to use GQ to take
advantage of mixed environments where desired, provided their apps/jobs
can handle it. Beyond that, I flinch a bit at trying to make the
complexities of distributed computing on ad-hoc heterogeneous clusters
part of GQ's problem space. 

Personally, I see a decline in ad-hoc clusters formed from spare or idle
systems and a rise in small dedicated clusters where the systems are
purchased all at once. Thus, if the latter truly is an expanding market,
we need to have a release as soon as possible that can handle this
simpler case well enough to establish a user base. The extensions above
will be simple enough I think at that point.

How does this plan sound?

K