[Queue-developers] Re: Gnu queue

[W. G. K. <wer...@ya...>]

Hi Ole,

Quoting Ole W Saastad <ole...@ch...>:

> 
> Hi,
> first of all I would like to thank you for a good program.
> 
> I have installed it on a RH 6.2 box that will run quantum chemical
> calculation. At such a machine it is important to have a kind of batch
> processing system. Queue does this nicely.
> 
> However,
> there are no commands to list the queues and no commands to
> remove jobs from the queues.
> I have written a simple script that cats the queuestat file and counts
> the number of files in the CFDIR in order to print at least some info
> about the queue.

Please contribute it to que...@li.... If it a 
longer script (more than, say, 10 or 20 lines) make it copyright yourself 
and say that it is GPL'd somewhere in the comments. Thanks!

> I would like to have some tools for manipulation
> of the queues. Like lq (list queue) and rmq (remove from queue).
> lq -d Gaussian which should produce some information about the
> queue named Gaussian. And rmq -d Gaussian JobID which deletes
> the JobID from the queue, the option -f would terminate Job ID if
> it were running. Otherwise rmq -d Gaussian JobID will not stop
> currently running jobs.

I think Monica Lau is supposedly working on this. She submitted a set of C/C++ 
patches for GNU Queue that sets up a central server and may include additional 
tools. The patches and other stuff she submitted are available for downloading 
from Sourceforge; see the archives of queue-developers on www.gnuqueue.org to 
find out where this is.

I would like to incorporate her patches into the next release of GNU Queue, or 
at least put them on the CVS archive so that people can begin hacking on them 
and the resulting parallelization will speed up development. I'm trying to get 
her to either release the patches under her company's copyright (as GPL'd --- 
this is the fastest way) but I made the mistake of also suggesting the standard 
GNU/FSF spiel and suggested she consider transfering the copyright to the FSF. 
Of course she did this, and now presumably the whole thing is tied up in her 
company's legal department. (It may also be the FSF got the forms from her 
company and hasn't told me yet.) Right now, the program is in copyright limbo; 
she's distributed it to the net, but it's not clear who has copyright or 
whether it is GPL'd. And so it goes....

In any event, as someone suggested a few weeks ago in developers, Queue's tables 
could be put into SQL database, which could then be used as the central server 
for the system. The scripts to manipulate Queue would then simply be things like 
PHP3 or Perl DBI scripts that would interact with the database and control 
variables in table there. Counting the number of jobs running globally would 
then be a simple SQL statement.

I've been reluctant to move in this direction because it would make GNU Queue 
less reliable; if the database fails, the whole system goes down. Also, the 
Internet RFC people want the protocol to be Internet-wide scalable (or, we have 
to explictly advise people that it is not scalable and indicate what 
circumstances it would be appropriate to use GNU Queue.) If we stipulate central 
server, then protocol is inherently not scalable. (Currently, it can easily be 
made scalable with slight modifications to the host searching protocol, although 
you'd want cryptographic cookies before doing this.)

I'll probably leave the current mechanism in tact; that will take care of 
Internet-wide scalability and also the folks that don't want to go through the 
trouble of setting up a SQL database. But, I'll probably make Queue try to look 
for a SQL database (at least during configuration) and use that if it's 
available. The tools would then include PHP3 scripts to manipulate the SQL 
database (probably MySQL.) That would leave the issue of including a bunch of 
RPMS for configuring phplib to run with MySQL --- the IMP package currently 
includes a bunch of RPMs like this (http://www.horde.org.), and it can be a pain 
to set up, but goes relatively quickly if you are running the latest version of 
RedHat.

> I would also like to know the syntax of the options in the profile
> file. How do I specify a maximum of 15 min CPU time. A list of
> examples should be included in the manual. Apart from this the manual
> was good.

You put an "rlimitcpu time" as a line in the profile for the batch queue, where 
time is the time in minutes. This uses the rlimit() kernel facility. The other 
way to do this originally was to place a "setlimit cpu time" statement in a 
csh/tcsh script; this set the kernel rlimit(), which then got passed to Queue. 
However, I think the code that does this is commented out now because it caused 
problems on Solaris and some other systems.

All kernels have such anr limit() facility, but not all of them use it --- 
HP-UX, for example, ignores this limit for cpu. You can set it, but the kernel 
never stops the program. There's a package copyright Uwaterloo that's not GPL 
that's available for download that fixes this problem under HPUX. This is the 
#ifdef HAVE_UWATERLOO directive in queued.c .

Hope this helps.

> 
> 
> Best regards
> Ole
> 
> 
> 
> --
> Ole W Saastad, Dr.Scient.               Tel. 22067852
> SINTEF Kjemi. PB 124 Blindern
> 0314 Oslo
> 
>