queue-developers

[Queue-developers] new design details

[Koni <mh...@co...>]

As promised before, here is some more details about the new system I
mentioned before. 

Werner: This is more or less identical to what I sent you previously. 



I envision 4 separate programs working together in this system:

qs: Users use this program (like "queue" or "qsh" in GNU Queue) to
submit jobs [ Presently not implemented at all ]

qm: the queue manager running on some central host. qs sends job
requests to qm. [ ~60% implemented ]

qd: daemon running on slave or "compute" nodes, possibly on the same
host as qm as well. More than one qd may run on any host, there may be
any number of them on any number of hosts. Only qm talks to qd's,
sending jobs as available. The distribution protocol works as a
offer/volunteer system, qm sends offers to multiple qd's at once for the
same job, willing qd's respond will a volunteer. qm assigns the job to
exactly one qd. qd may refuse at this point too (resets job to offer
stage), or commit and receive transfer of the job and begin execution.
Important point is qd's decide autonomously if they can spare resources
for the job. qm has some state information of the availability of the
qd's it knows about and does not send offers to qd's it knows are fully
committed, but qm does not need an accurate perception, it's qd's
decision. [ ~70% implemented ]

qe: Execution agent forked and exec'd by the qd process for running a
job. qe is responsible for setting the environment, calling back to
waiting qs if foreground mode selected (called interactive mode in GNU
Queue I think), validating and changing to the user of the job,
monitoring for the termination of the job and return code, etc. qe is
the only part of this system that needs to be setuid root. qd and qm may
need to start as root to read the system wide key file (see below) but
can drop privilege permanently after that. [ currently a trivial program
which just returns immediately is presently implemented ] 

Some design goals/choices:

NFS is not used for communication and distribution of the jobs. This was
a primary goal in the design for me. After getting into it, I have new
appreciation for the design of GNU Queue though. :)

Stateless UDP is used for communication for qm and qd, which results in
some complexity in the code due to the possibility of lost messages.
This is a design goal as persistent TCP connections consume file
descriptors limiting the number of qd's that can be connected to qm. I
would like this to scale well beyond typical limits for open file
descriptors. 

All messages between qd and qm are cryptographically signed [ this is
already fully implemented ] using key'd SHA-1. On connection, a
registration protocol verifies the authenticity of both qm and qd by
proving knowledge of a system-wide key. After registration, each qd is
assigned a session key used to sign messages after that. qs will
communicate username/password information (encrypted) to qm which will
ultimately be passed from qd to qe which will authenticate before
switching to the user requested. 

Much effort is being put into low latency distribution of jobs.
Experimenting with the version of GNU Queue I have, after making several
changes to get it to go and all, it takes a second or more between
submission of a job and onset of execution in an idle cluster. Much of
this I think is due to built-in deliberate delays to work-around NFS
race conditions, hence my interest in eliminating NFS as a communication
layer between submitting users and execution agents. My present system
is seemingly instantaneous on an idle cluster (but much is not
implemented yet), my goal is to have latency for executing say 1000 no-
op jobs on a system with a single qd agent comparable to that of a shell
script doing the same directly. 

Some drawbacks:

Security rests ultimately with the privacy of the system-wide key file,
which must be installed or accessible to both qm and all qd agents.

All systems running qd must have access to the same authentication
system for validating username/password for submitting users. NIS or
something equivalent is probably the easiest both for me as developer
and administrators at large who might use this thing. We can potentially
use a custom arrangement through PAM too.

NFS or other shared network filesystem still required for user jobs to
read/write input and output, unless they only want to use stdin/stdout
in which case qs can handle it. I don't consider this a problem really
for dedicated systems.

Job transfer takes place over a transient TCP connection, but I've
noticed this can cause a hiccup (qm pauses for several seconds but
eventually resumes rapid distribution of jobs) if the TCP SYN packet is
lost, which seems to happen after about 30,000 jobs have been sent and
executed as fast as possible. the TIME_WAIT state of a closed TCP
connection hogs the system resources on the qm host, potentially
blocking the opening of new connections until resources are available.
This is only a problem in the pathological case of >30,000 no-op jobs at
once, surely not a real-world problem. Presently the system will pause
if the SYN packet is dropped when forming a new connection, and will
wait until both enough TIME_WAIT old TCP connections are cleared and the
SYN retransmit timer expires, at which point connection is established
and distribution commences again.

This system has a central manager, qm, which the present GNU Queue does
not. Failure at qm will cause the whole cluster to stop executing jobs
after their present assignments. This does not happen with GNU Queue,
unless the NFS server goes down. However, when NFS comes back, provided
no corruption to the filesystem, everything continues. My system will
need some crash-recovery complexity for qm. qd's can die and come back
all they like.


Comments are welcome. If you want to peak at the code, reply back to
this list. If there is interest and no objections, I will post a copy of
the source as is to the list. It doesn't do much for the moment except
implement the qm -> qd -> qe chain of events and demonstrate the
distribution of jobs.

Cheers,
Koni

On Sun, 2005-05-01 at 19:38 -0400, Richard Stallman wrote:
>     Anyhow, I suggested in my email to Koni and Mike that
>     we wait a week or two for Mike to respond.
> 
> I think that is a reasonable plan.  The program needs a maintainer who
> will make releases, and more generally, who will give the program
> proper attention.
> 
> 					       At some
>     point, we'd post publically, and then wait about 30
>     days or some reasonable time. 
> 
> I don't understand that part.  Wait 30 days for what?
> 
>     What's the standard procedure for reclaiming an
>     abandoned GNU project?
> 
> I can appoint (and remove) maintainers at any time.
> So once the situation is clear, I can simply appoint
> a new maintainer for GNU Queue.

Re: [Queue-developers] new design details

[Jordi B. C. <jbu...@oc...>]

Hi Koni, all,

On Monday 09 May 2005 23:47, Koni wrote:
> Comments are welcome. If you want to peak at the code, reply back to
> this list. If there is interest and no objections, I will post a copy of
> the source as is to the list. It doesn't do much for the moment except
> implement the qm -> qd -> qe chain of events and demonstrate the
> distribution of jobs.

I would be very interested in having a look at the code. I tried to use GNU 
Queue a few weeks ago but found the same problems you did. Now I am 
considering Sun ONE Grid Engine as a batch system for our clusters, but would 
like to experiment with your new code.

Cheers,
jordi

Re: [Queue-developers] new design details

[wernerkrebs <wer...@ya...>]

Two comments.

1. Regarding the protocol, GQ's protocols largely
predated modern RPC standards, such as SOAP and XML. 

These are most suited for easy implementation in Java
or C#/Mono, where there are excellent development
environments (e.g., Eclipse, JBuilder, Visual Studio
.NET, etc.) that will almost write the code for you
automagically. Of these, Eclipse and Mono are Open
Source, so suitable for consideration here. SOAP does
fully support C,C++, even Perl, however, although
development is slightly more difficult IMHO.

I would think some of the current features of the GQ
TCP/IP protocol would be best done using some sort of
SOAP implementation. For example, aspects of the
initial authentication, and querying load information
would be best done using SOAP.

Also, since GQ was written, standard protocols for
this type of thing have emerged. Look at Apst/Apstd
system at SDSC (where, ironically, I used to work,
although not on that project):

http://grail.sdsc.edu/projects/apst/

Apst is a meta demon for cluster demons. It doesn't
currently support starting jobs using GQ, but does
support starting other (commerical) systems. GQ
support would be fairly trivial for them to add, if
they wanted to. SDSC (part of UCSD) receives grant
money from a firm that makes a GQ-like commercial
product, so it's not clear if that's a direction they
want to go in. They do support the commerical product.
However, the source code is available, so the
community is free to add support for GQ as well.

Apst will query each cluster manager (this would
similar to the qm program you are proposing) and
obtain load information via an XML file returned from
the cluster manager. It will then decide how many jobs
to start on that particular cluster (which it will
start using a crude ssh command-line protocol to
submit the jobs and scp to first transfer the relevant
files into place). It's up to the cluster manager to
then distribute the jobs to the cluster nodes.

Apst, which is C/C++ based (Apstd is available in
Java) is similar to Nimrod, which is Java-based.
Source code for all of these is available.

2. Regarding qm, a divison of the Texas Instruments
actually contributed a SQL-based qm in C++. (It would
require that an SQL database, preferably Open Source
and free such as Postgresql, be running on a server).

This is part of the GQ distribution, but is optional
and not compiled by default (due to C++ autoconf
problems at the time since resolved. Also, users wrote
to me explaining their preference for a small, simple
package with peer-to-peer behavior, rather than a
centralized package with a manager that might crash,
so the original behavior of GQ remained the default.)

Beforing writing a manager from scratch, you might
want to look at the manager code and documentation
that TI's subsidary contributed.
--
Werner G. Krebs, Ph.D.
Technical Specialist
Personal website: http://www.wernergkrebs.com


--- Koni <mh...@co...> wrote:

> 
> As promised before, here is some more details about
> the new system I
> mentioned before. 
> 
> Werner: This is more or less identical to what I
> sent you previously. 
> 
> 
> 
> I envision 4 separate programs working together in
> this system:
> 
> qs: Users use this program (like "queue" or "qsh" in
> GNU Queue) to
> submit jobs [ Presently not implemented at all ]
> 
> qm: the queue manager running on some central host.
> qs sends job
> requests to qm. [ ~60% implemented ]
> 
> qd: daemon running on slave or "compute" nodes,
> possibly on the same
> host as qm as well. More than one qd may run on any
> host, there may be
> any number of them on any number of hosts. Only qm
> talks to qd's,
> sending jobs as available. The distribution protocol
> works as a
> offer/volunteer system, qm sends offers to multiple
> qd's at once for the
> same job, willing qd's respond will a volunteer. qm
> assigns the job to
> exactly one qd. qd may refuse at this point too
> (resets job to offer
> stage), or commit and receive transfer of the job
> and begin execution.
> Important point is qd's decide autonomously if they
> can spare resources
> for the job. qm has some state information of the
> availability of the
> qd's it knows about and does not send offers to qd's
> it knows are fully
> committed, but qm does not need an accurate
> perception, it's qd's
> decision. [ ~70% implemented ]
> 
> qe: Execution agent forked and exec'd by the qd
> process for running a
> job. qe is responsible for setting the environment,
> calling back to
> waiting qs if foreground mode selected (called
> interactive mode in GNU
> Queue I think), validating and changing to the user
> of the job,
> monitoring for the termination of the job and return
> code, etc. qe is
> the only part of this system that needs to be setuid
> root. qd and qm may
> need to start as root to read the system wide key
> file (see below) but
> can drop privilege permanently after that. [
> currently a trivial program
> which just returns immediately is presently
> implemented ] 
> 
> Some design goals/choices:
> 
> NFS is not used for communication and distribution
> of the jobs. This was
> a primary goal in the design for me. After getting
> into it, I have new
> appreciation for the design of GNU Queue though. :)
> 
> Stateless UDP is used for communication for qm and
> qd, which results in
> some complexity in the code due to the possibility
> of lost messages.
> This is a design goal as persistent TCP connections
> consume file
> descriptors limiting the number of qd's that can be
> connected to qm. I
> would like this to scale well beyond typical limits
> for open file
> descriptors. 
> 
> All messages between qd and qm are cryptographically
> signed [ this is
> already fully implemented ] using key'd SHA-1. On
> connection, a
> registration protocol verifies the authenticity of
> both qm and qd by
> proving knowledge of a system-wide key. After
> registration, each qd is
> assigned a session key used to sign messages after
> that. qs will
> communicate username/password information
> (encrypted) to qm which will
> ultimately be passed from qd to qe which will
> authenticate before
> switching to the user requested. 
> 
> Much effort is being put into low latency
> distribution of jobs.
> Experimenting with the version of GNU Queue I have,
> after making several
> changes to get it to go and all, it takes a second
> or more between
> submission of a job and onset of execution in an
> idle cluster. Much of
> this I think is due to built-in deliberate delays to
> work-around NFS
> race conditions, hence my interest in eliminating
> NFS as a communication
> layer between submitting users and execution agents.
> My present system
> is seemingly instantaneous on an idle cluster (but
> much is not
> implemented yet), my goal is to have latency for
> executing say 1000 no-
> op jobs on a system with a single qd agent
> comparable to that of a shell
> script doing the same directly. 
> 
> Some drawbacks:
> 
> Security rests ultimately with the privacy of the
> system-wide key file,
> which must be installed or accessible to both qm and
> all qd agents.
> 
> All systems running qd must have access to the same
> authentication
> system for validating username/password for
> submitting users. NIS or
> something equivalent is probably the easiest both
> for me as developer
> and administrators at large who might use this
> thing. We can potentially
> use a custom arrangement through PAM too.
> 
> NFS or other shared network filesystem still
> required for user jobs to
> read/write input and output, unless they only want
> to use stdin/stdout
> in which case qs can handle it. I don't consider
> this a problem really
> for dedicated systems.
> 
> Job transfer takes place over a transient TCP
> connection, but I've
> noticed this can cause a hiccup (qm pauses for
> several seconds but
> eventually resumes rapid distribution of jobs) if
> the TCP SYN packet is
> lost, which seems to happen after about 30,000 jobs
> have been sent and
> executed as fast as possible. the TIME_WAIT state of
> a closed TCP
> connection hogs the system resources on the qm host,
> potentially
> blocking the opening of new connections until
> resources are available.
> This is only a problem in the pathological case of
> >30,000 no-op jobs at
> once, surely not a real-world problem. Presently the
> system will pause
> if the SYN packet is dropped when forming a new
> connection, and will
> wait until both enough TIME_WAIT old TCP connections
> are cleared and the
> SYN retransmit timer expires, at which point
> connection is established
> and distribution commences again.
> 
> This system has a central manager, qm, which the
> present GNU Queue does
> not. Failure at qm will cause the whole cluster to
> stop executing jobs
> after their present assignments. This does not
> happen with GNU Queue,
> unless the NFS server goes down. However, when NFS
> comes back, provided
> no corruption to the filesystem, everything
> continues. My system will
> need some crash-recovery complexity for qm. qd's can
> die and come back
> all they like.
> 
> 
> Comments are welcome. If you want to peak at the
> code, reply back to
> this list. If there is interest and no objections, I
> will post a copy of
> the source as is to the list. It doesn't do much for
> the moment except
> implement the qm -> qd -> qe chain of events and
> demonstrate the
> distribution of jobs.
> 
> Cheers,
> Koni
> 
> On Sun, 2005-05-01 at 19:38 -0400, Richard Stallman
> wrote:
> >     Anyhow, I suggested in my email to Koni and
> Mike that
> >     we wait a week or two for Mike to respond.
> > 
> > I think that is a reasonable plan.  The program
> needs a maintainer who
> > will make releases, and more generally, who will
> give the program
> > proper attention.
> > 
> > 					       At some
> >     point, we'd post publically, and then wait
> about 30
> >     days or some reasonable time. 
> > 
> > I don't understand that part.  Wait 30 days for
> what?
> > 
> >     What's the standard procedure for reclaiming
> an
> >     abandoned GNU project?
> > 
> > I can appoint (and remove) maintainers at any
> time.
> > So once the situation is clear, I can simply
> appoint
> > a new maintainer for GNU Queue.
> 
> 
> 
>
-------------------------------------------------------
> This SF.Net email is sponsored by: NEC IT Guy Games.
> Get your fingers limbered up and give it your best
> shot. 4 great events, 4
> opportunities to win big! Highest score wins.NEC IT
> Guy Games. Play to
> win an NEC 61 plasma display. Visit
> http://www.necitguy.com/?r=20
> _______________________________________________
> Queue-developers mailing list
> Que...@li...
> To unsubscribe, subscribe, or set options:
>
https://lists.sourceforge.net/lists/listinfo/queue-developers
> 

Re: [Queue-developers] new design details

[Koni <mh...@co...>]

On Tue, 2005-05-10 at 08:56 -0700, wernerkrebs wrote:
> 
> Two comments.
> 
> 1. Regarding the protocol, GQ's protocols largely
> predated modern RPC standards, such as SOAP and XML. 
> 

I'm not sure any of these things are worth their weight in a homogeneous
system. The communication between the GQ system as I have envisioned it
is pretty lightweight and there is very little structure to the
information. In this case, I think using XML or SOAP for a communication
layer adds complexity (in my mind) which is contrary to their purpose in
general. 

[snip]

> I would think some of the current features of the GQ
> TCP/IP protocol would be best done using some sort of
> SOAP implementation. For example, aspects of the
> initial authentication, and querying load information
> would be best done using SOAP.
> 

I don't think SOAP will do much for us regarding authentication. The
authentication stuff here is really simple (to me). Perhaps for load
information if a lot of detail is returned (like all the information ps
would return say). As for authentication, its already implemented as a
simple challenge handshake (initial authentication):

qd                              qm

auth/register request
(send nonce)          -------->

                                sign nonce with system key,
                      <-------  reply with our own nonce

verify response       --------> 
sign qm nonce

                      <--------  verify response, send session key


If either verification fails, the offended party stops the protocol.
Receipt of the session key indicates to qd that the challenge handshake
protocol completed successfully. After that, all communication between
the qd and qm come with simple signatures using that key. The complexity
of the generation of signatures and verification of them is already more
or less isolated from the logic if handling the message payload. 



> Also, since GQ was written, standard protocols for
> this type of thing have emerged. Look at Apst/Apstd
> system at SDSC (where, ironically, I used to work,
> although not on that project):
> 
> http://grail.sdsc.edu/projects/apst/
> 
> Apst is a meta demon for cluster demons. It doesn't
> currently support starting jobs using GQ, but does
> support starting other (commerical) systems. GQ
> support would be fairly trivial for them to add, if
> they wanted to. SDSC (part of UCSD) receives grant
> money from a firm that makes a GQ-like commercial
> product, so it's not clear if that's a direction they
> want to go in. They do support the commerical product.
> However, the source code is available, so the
> community is free to add support for GQ as well.
> 
> Apst will query each cluster manager (this would
> similar to the qm program you are proposing) and
> obtain load information via an XML file returned from
> the cluster manager. It will then decide how many jobs
> to start on that particular cluster (which it will
> start using a crude ssh command-line protocol to
> submit the jobs and scp to first transfer the relevant
> files into place). It's up to the cluster manager to
> then distribute the jobs to the cluster nodes.
> 
> Apst, which is C/C++ based (Apstd is available in
> Java) is similar to Nimrod, which is Java-based.
> Source code for all of these is available.

This sounds interesting. It would be great for GQ, whether GQ becomes my
new proposed implementation, remains as is, or something else
altogether, contributing a "driver" (so to speak) so that this meta
system can work with it would be cool and perhaps broaden the market for
us.

> 
> 2. Regarding qm, a divison of the Texas Instruments
> actually contributed a SQL-based qm in C++. (It would
> require that an SQL database, preferably Open Source
> and free such as Postgresql, be running on a server).
> 

Cool. I was first thinking about job information being managed by a
mysql (or postgres) backend, where the SQL engine would handle things
like atomicity and persistent state information across failure. Would
have been cake if I wrote qm in perl (I am very familiar with Perl-DBI).
The only thing I don't like about this is the potential high-latency --
one (or more) threads insert to the job table (qs) while some another
thread polls (qm) the table for new rows. Perhaps in postgres there is a
way to install a trigger or something so polling is unnecessary. I don't
think there is a way to do that in mysql. qm is actually unnecessary if
qd's can talk to the SQL engine directly. SQL can handle authentication
and atomicity and qd's can just compete for jobs. That's kind of nice.
Not sure it will scale well though. 1000 qd's each with persistent TCP
connection to mysql would create 1000 forked processes at the database
server. 


> This is part of the GQ distribution, but is optional
> and not compiled by default (due to C++ autoconf
> problems at the time since resolved. Also, users wrote
> to me explaining their preference for a small, simple
> package with peer-to-peer behavior, rather than a
> centralized package with a manager that might crash,
> so the original behavior of GQ remained the default.)
> 
> Beforing writing a manager from scratch, you might
> want to look at the manager code and documentation
> that TI's subsidary contributed.

OK, I'll try to have a look. The manager is almost already all written
though in my haste to flesh out ideas rolling around in my head. I shall
post a tarball of the code shortly. I want to add at least a rudimentary
support for actually submitting a job to the system and having it
execute. While I'm doing that, we can get a better feel for who is out
there reading this list and what interest there is. 

Thanks for your comments Werner, I appreciate your insights greatly.

Cheers,
Koni

Re: [Queue-developers] new design details

[wernerkrebs <wer...@ya...>]

Progamming is partially an artform as much as a
science, and it's usually best to try to use the most
modern techniques that everyone else is using. This
way, you can leverage off what other people in the
community are doing, and develop synergy with other
projects. GQ was modern for its time, but lots has
happened since it came out.


1.

In certain modern environments (Java or C#/Mono),
using SOAP/XML actually simplifies things. The
complexity gets shifted to the libraries.

Moreover, it adds the ability to then become a
"standard" that work together with other environments
--- open the possibility of easily making GQ work in
non-homogenous environments, something that was much
requested by the users.

Check out Ganglia, another open source project,
originally developed at UCB.
http://ganglia.sourceforge.net/

This is becoming one of the standards for determining
load averages on remote machines. Meta-clustering
systems, such as APST, recognize this as one of the
protocols for querying load averages and other
information.

So, it would be good if GQ supported a standard
monitoring protocol like ganglia. You'd have to run
ganglia somewhere anyway if you wanted to use a meta
system like apst, so GQ could feed off that, rather
than, or in addition to, its own load monitoring code.
Or, GQ could continue to use its own monitoring code,
but also support the ganglia protocol (it's open
source, after all), so there wouldn't be a need to run
two load monitoring daemons.

Ganglia exchanges using information using XML, of
course. 

2.

Regarding your concerns about SQL scalability: a lot
of work was gone into making SQL environments highly
scalable --- its a huge issue for corporations and
anyone trying to run a high-traffic, mission critical
website (been there). Remember the old commercials for
a certain computer services firm about the startup
that didn't consult with them, and therefore its
website wasn't scalable? mysql (which RMS wouldn't
want us to mention here because it is semi-commercial
--- he wants us to say 'postgresql' because that's
completely free), may not be one of them, but the
major two commercial SQL are designed to be highly
scalable in a cluster environment (this is why one
pays big bucks in support and fees for them instead of
mysql).

The other solution, commonly used in the website world
(J2EE and .NET/Mono), involves so-called 3-tiered
architectures.

Basically, the idea is to have another set of threads,
which can potentially run on another CPU or another
machine, handle all the actual communication with
client (web browser) and cache some of the SQL queries
(pre-compilation), and, in some cases, even cache the
results when possible. This takes a significant load
off the back-end SQL database, which can now handles
1000s more clients. 

This approach would work with a GQ mananger by
creating an intermediary gm daemon that would reply to
a large number of clients by caching and periodically
refreshing the results of a small number of SQL
queries.

SQL databases are at the leading edge of scalability
technology (although often in the commercial rather
than open source worlds) and have other benefits
(standardization, so other clients can interact, and
existing SQL database management tools can be used).

Still, I'll let you decide how you think it's best to
do this.

--- Koni <mh...@co...> wrote:

> On Tue, 2005-05-10 at 08:56 -0700, wernerkrebs
> wrote:
> > 
> > Two comments.
> > 
> > 1. Regarding the protocol, GQ's protocols largely
> > predated modern RPC standards, such as SOAP and
> XML. 
> > 
> 
> I'm not sure any of these things are worth their
> weight in a homogeneous
> system. The communication between the GQ system as I
> have envisioned it
> is pretty lightweight and there is very little
> structure to the
> information. In this case, I think using XML or SOAP
> for a communication
> layer adds complexity (in my mind) which is contrary
> to their purpose in
> general. 
> 
> [snip]
> 
> > I would think some of the current features of the
> GQ
> > TCP/IP protocol would be best done using some sort
> of
> > SOAP implementation. For example, aspects of the
> > initial authentication, and querying load
> information
> > would be best done using SOAP.
> > 
> 
> I don't think SOAP will do much for us regarding
> authentication. The
> authentication stuff here is really simple (to me).
> Perhaps for load
> information if a lot of detail is returned (like all
> the information ps
> would return say). As for authentication, its
> already implemented as a
> simple challenge handshake (initial authentication):
> 
> qd                              qm
> 
> auth/register request
> (send nonce)          -------->
> 
>                                 sign nonce with
> system key,
>                       <-------  reply with our own
> nonce
> 
> verify response       --------> 
> sign qm nonce
> 
>                       <--------  verify response,
> send session key
> 
> 
> If either verification fails, the offended party
> stops the protocol.
> Receipt of the session key indicates to qd that the
> challenge handshake
> protocol completed successfully. After that, all
> communication between
> the qd and qm come with simple signatures using that
> key. The complexity
> of the generation of signatures and verification of
> them is already more
> or less isolated from the logic if handling the
> message payload. 
> 
> 
> 
> > Also, since GQ was written, standard protocols for
> > this type of thing have emerged. Look at
> Apst/Apstd
> > system at SDSC (where, ironically, I used to work,
> > although not on that project):
> > 
> > http://grail.sdsc.edu/projects/apst/
> > 
> > Apst is a meta demon for cluster demons. It
> doesn't
> > currently support starting jobs using GQ, but does
> > support starting other (commerical) systems. GQ
> > support would be fairly trivial for them to add,
> if
> > they wanted to. SDSC (part of UCSD) receives grant
> > money from a firm that makes a GQ-like commercial
> > product, so it's not clear if that's a direction
> they
> > want to go in. They do support the commerical
> product.
> > However, the source code is available, so the
> > community is free to add support for GQ as well.
> > 
> > Apst will query each cluster manager (this would
> > similar to the qm program you are proposing) and
> > obtain load information via an XML file returned
> from
> > the cluster manager. It will then decide how many
> jobs
> > to start on that particular cluster (which it will
> > start using a crude ssh command-line protocol to
> > submit the jobs and scp to first transfer the
> relevant
> > files into place). It's up to the cluster manager
> to
> > then distribute the jobs to the cluster nodes.
> > 
> > Apst, which is C/C++ based (Apstd is available in
> > Java) is similar to Nimrod, which is Java-based.
> > Source code for all of these is available.
> 
> This sounds interesting. It would be great for GQ,
> whether GQ becomes my
> new proposed implementation, remains as is, or
> something else
> altogether, contributing a "driver" (so to speak) so
> that this meta
> system can work with it would be cool and perhaps
> broaden the market for
> us.
> 
> > 
> > 2. Regarding qm, a divison of the Texas
> Instruments
> > actually contributed a SQL-based qm in C++. (It
> would
> > require that an SQL database, preferably Open
> Source
> > and free such as Postgresql, be running on a
> server).
> > 
> 
> Cool. I was first thinking about job information
> being managed by a
> mysql (or postgres) backend, where the SQL engine
> would handle things
> like atomicity and persistent state information
> across failure. Would
> have been cake if I wrote qm in perl (I am very
> familiar with Perl-DBI).
> The only thing I don't like about this is the
> potential high-latency --
> one (or more) threads insert to the job table (qs)
> while some another
> thread polls (qm) the table for new rows. Perhaps in
> postgres there is a
> way to install a trigger or something so polling is
> unnecessary. I don't
> think there is a way to do that in mysql. qm is
> actually unnecessary if
> qd's can talk to the SQL engine directly. SQL can
> handle authentication
> and atomicity and qd's can just compete for jobs.
> That's kind of nice.
> Not sure it will scale well though. 1000 qd's each
> with persistent TCP
> connection to mysql would create 1000 forked
> processes at the database
> server. 
> 
> 
> > This is part of the GQ distribution, but is
> optional
> > and not compiled by default (due to C++ autoconf
> > problems at the time since resolved. Also, users
> wrote
> > to me explaining their preference for a small,
> simple
> > package with peer-to-peer behavior, rather than a
> > centralized package with a manager that might
> crash,
> > so the original behavior of GQ remained the
> default.)
> > 
> > Beforing writing a manager from scratch, you might
> > want to look at the manager code and documentation
> > that TI's subsidary contributed.
> 
> OK, I'll try to have a look. The manager is almost
> already all written
> though in my haste to flesh out ideas rolling around
> in my head. I shall
> post a tarball of the code shortly. I want to add at
> least a rudimentary
> support for actually submitting a job to the system
> and having it
> execute. While I'm doing that, we can get a better
> feel for who is out
> there reading this list and what interest there is. 
> 
> Thanks for your comments Werner, I appreciate your
> insights greatly.
> 
> Cheers,
> Koni
> 
> 
> 
>
-------------------------------------------------------
> This SF.Net email is sponsored by Oracle Space
> Sweepstakes
> Want to be the first software developer in space?
> Enter now for the Oracle Space Sweepstakes!
>
http://ads.osdn.com/?ad_id=7393&alloc_id=16281&op=click
> _______________________________________________
> Queue-developers mailing list
> Que...@li...
> To unsubscribe, subscribe, or set options:
>
https://lists.sourceforge.net/lists/listinfo/queue-developers
> 

Re: [Queue-developers] new design details

[Koni <mh...@co...>]

On Wed, 2005-05-11 at 22:01 -0700, wernerkrebs wrote:
> Progamming is partially an artform as much as a
> science, and it's usually best to try to use the most
> modern techniques that everyone else is using. This
> way, you can leverage off what other people in the
> community are doing, and develop synergy with other
> projects. GQ was modern for its time, but lots has
> happened since it came out.
> 

I agree here in principle, but some things that might be called modern
techniques are really just trendy to me. 

I think we (we as in all interested parties on this list) first need to
sort out a couple critical items:

First, do we wish or intend to subscribe firmly to the RMS point of view
of free software for this project? If GNU Queue remains "GNU", then I
think this is most relevant albeit restrictive. A firm commitment to
that would limit our discussions about mysql or high performance
commercial database backends and use of Java or C#/Mono. 

Second, I think we should define a limited scope of problems that GQ
will purport to solve and do that well. I am confused by the idea of
meta-clustering. It seems like it won't be much work on us to have some
meta-controller interface with GQ's controller, but why do people do
this? I can think of a few reasons, but are the needs of these
applications fitting within the problem space we want to address?

I see a growing market of small dedicated homogeneous systems with
simple distribution needs, driven by cheap computer prices. In time up
to present or perhaps into the future some more, interest heterogeneous
and non-dedicated systems I think is coming from interest in tapping the
resource of idle desktop CPUs. With that comes substantial complexity
that other projects like condor have embraced and seem to be doing
well. 

I don't think its beneficial to GQ to try to tackle some of the same
problems if condor already fills that niche. I also think many groups
with new interest in this area will have small dedicated systems and be
looking for something simple and easy. Thus, this seems like an
expanding niche that would be well filled by something that takes 2
minutes to setup and start using. GQ can do this, especially if it is
stand-alone with no backend database and no library dependencies.  

I think the ease of setup and use will make a big difference in the
adoption of GQ as a solution in the situations where it is applicable. 

Re: [Queue-developers] new design details

[wernerkrebs <wer...@ya...>]

--- Koni <mh...@co...> wrote:

> On Wed, 2005-05-11 at 22:01 -0700, wernerkrebs
> wrote:
> > Progamming is partially an artform as much as a
> > science, and it's usually best to try to use the
> most
> > modern techniques that everyone else is using.
> This
> > way, you can leverage off what other people in the
> > community are doing, and develop synergy with
> other
> > projects. GQ was modern for its time, but lots has
> > happened since it came out.
> > 
> 
> I agree here in principle, but some things that
> might be called modern
> techniques are really just trendy to me. 
> 
> I think we (we as in all interested parties on this
> list) first need to
> sort out a couple critical items:
> 
> First, do we wish or intend to subscribe firmly to
> the RMS point of view
> of free software for this project? If GNU Queue
> remains "GNU", then I
> think this is most relevant albeit restrictive. A
> firm commitment to
> that would limit our discussions about mysql or high
> performance
> commercial database backends and use of Java or
> C#/Mono. 

Up to you. >;->

> Second, I think we should define a limited scope of
> problems that GQ
> will purport to solve and do that well. I am
> confused by the idea of
> meta-clustering. It seems like it won't be much work
> on us to have some
> meta-controller interface with GQ's controller, but
> why do people do
> this? 

Lack of centralized administrative control of the
different clusters, which are spread out at different
institutions. 

Computational biology (specifically simulation and
modelling of protein molecules for rational drug
design) in support of the multi-billion dollar
pharmaceutical R&D budget is the current "killer app"
for "the grid", in part because the ageing population
of affulent countries is seen as creating the business
model to sell the next generation of supercomputers.
(GQ was originally written to help in econometrics
problems, but I apparently moved in comp. bio. with
the rest of the supercomputer market as I underwent my
doctoral studies.)

"The Grid" is the current buzzword that describes what
GQ does (aka "clustering", but on a wide-area,
multi-institution scale). You can read books on
Amazon.com on it; I know some of the authors
personally. It's a word that didn't even exist when GQ
was first released.

The idea is that multiple institutions in different
countries across the globe would loan their clusters
to form a meta-cluster, connected by a dedicated
TeraBit backbone network currently being built. Time
on this meta cluster might then be rented to
pharmaceutical companies. There is very serious
government money in this area. (It's also an area that
your institution, Cornell, has been heavily involved
in, and has it's dedicated own Linux clusters. The
person at Cornell is Prof. Harold Scheraga, whom I've
had the privilege of meeting.)

Your analysis may be dead on: it might not be the
thing for GQ, which maybe should focus on small
clusters and easy installation.

Still, it's something to keep in mind. If you want to
write a GQ manual someday, it will sell a lot more
copies if you advertise GQ as facilitating "The Grid"
or "The TeraGrid." Similarily, the GQ website would
pick up a lot more traffic if we added "Grid
Applications" or "TeraGrid Applications" somewhere on
the webpage. "Clustering" is the old-school term, I'm
afraid. >:->

> I can think of a few reasons, but are the
> needs of these
> applications fitting within the problem space we
> want to address?

Don't know; your call.

> I see a growing market of small dedicated
> homogeneous systems with
> simple distribution needs, driven by cheap computer
> prices. In time up
> to present or perhaps into the future some more,
> interest heterogeneous
> and non-dedicated systems I think is coming from
> interest in tapping the
> resource of idle desktop CPUs. With that comes
> substantial complexity
> that other projects like condor have embraced and
> seem to be doing
> well. 
> 
> I don't think its beneficial to GQ to try to tackle
> some of the same
> problems if condor already fills that niche. I also
> think many groups
> with new interest in this area will have small
> dedicated systems and be
> looking for something simple and easy. Thus, this
> seems like an
> expanding niche that would be well filled by
> something that takes 2
> minutes to setup and start using. GQ can do this,
> especially if it is
> stand-alone with no backend database and no library
> dependencies.  
> 
> I think the ease of setup and use will make a big
> difference in the
> adoption of GQ as a solution in the situations where
> it is applicable. 

Sounds good. >:->

> 
> 
> 
> 
>
-------------------------------------------------------
> This SF.Net email is sponsored by Oracle Space
> Sweepstakes
> Want to be the first software developer in space?
> Enter now for the Oracle Space Sweepstakes!
>
http://ads.osdn.com/?ad_id=7393&alloc_id=16281&op=click
> _______________________________________________
> Queue-developers mailing list
> Que...@li...
> To unsubscribe, subscribe, or set options:
>
https://lists.sourceforge.net/lists/listinfo/queue-developers
> 

[Queue-developers] Please welcome Mark Wright as new GQ maintainer

[wernerkrebs <wer...@ya...>]

Hi all --- 

This is the announcement some of you have been
expecting.

RMS has appointed Cornell graduate student Mark Wright
(aka Koni) <mh...@co...> as the new maintainer of
GNU Queue. 

Please welcome him!
--
Werner G. Krebs, Ph.D.
Technical Specialist

Personal homepage: http://www.wernergkrebs.com

Re: [Queue-developers] new design details

[<bo...@pr...>]

Koni wrote:
> I envision 4 separate programs working together in this system:
> 
> qs: Users use this program (like "queue" or "qsh" in GNU Queue) to
> submit jobs [ Presently not implemented at all ]

It would be bonus points to make this interoperable with the POSIX
standard.  As far as I can tell this is simply PBS.

  http://www.opengroup.org/onlinepubs/009695399/utilities/qsub.html

Most queue software is completely unique and people don't really
expect POSIX conformance at this point in history.  It just does not
seem to have the grip in the industry that conformance to other parts
of the POSIX spec have.  So I would not say this is really important
at this time.  But at least being aware of it would be good.

> Some design goals/choices:
> 
> NFS is not used for communication and distribution of the jobs. This was
> a primary goal in the design for me. After getting into it, I have new
> appreciation for the design of GNU Queue though. :)

I looked into GNU queue way back in the beginning.  But the integrated
NFS as an integral part of the design made it unsuitable for my use
with several thousand compute servers.  I have been a lurker on the
list every since because GNU queue had some nice features.

It would be nice if a project goal were to have code portable to a
wide range of platforms.  I would say GNU/Linux, HP-UX, Solaris, AIX,
SGI, Mac OS X, at a start.  What this really means is avoiding a lot
of heavy dependencies from weird libraries.  Trying to build a large
project with fifty library dependencies in a mix of C++ with heavy STL
and C# and Java from scratch on AIX is not an easy task.

> executed as fast as possible. the TIME_WAIT state of a closed TCP
> connection hogs the system resources on the qm host, potentially

I have actually seen this often on my network just doing normal TCP
activity.  I believe that flakey network hardware can raise the
likelyhood of this problem drastically.  I don't have a solution other
than waiting for the TIME_WAIT timeout to clear the network stack.  If
you are opening and closing a lot of network connections very rapidly
I think this could be a problem on bigger networks.  But if this is
only one connection per job then on my network that would not be a
problem as that is well within tolerable limits.

Bob