clockwork-developers

[Clockwork-developers] Kicking around some requirements

[Joel L. <jo...@lo...>]

In doing a little research on the web by looking at commercial job schedulers,
it seems that there are very different approaches to the architecture of a
scheduler. But before I started thinking about that too much, I wanted to
build a list of the things I wanted to see in a scheduler.

Here's what I came up with:

* Cross-platform. It needs to run on lots of UNIX variants. This is an
  enterprise tool, and enterprises rarely have only one platform.

* Reliable. One process or node going down shouldn't hose the schedule. And
  recovering from failures should be easy.

* Management GUI. Of course you can do everything from the command line, but
  if you have a large enough schedule, a GUI might be the only way to
  manage your schedule effectively. The GUI would be the only component
  that might should be able to also run on Windows.

* Scalability. The scheduler needs to work with thousands upon thousands of
  jobs, without requiring heaps of resources or clever kludge-like solutions
  from its operators. I think that AutoSys fails miserably at this.

* Performance. Reactions from the scheduler should be quick. If you have
  enough jobs, AutoSys fails here, too.

* Security. Listening to the scheduler's traffic on the network shouldn't
  give away any secrets. And allowing a system to be managed by the scheduler
  shouldn't open it up to spoof job requests from malicious people on the
  network.

* Flexibility in management. The larger the shop, the more custom a solution
  they'll need for managing the schedule. Things should be configureable
  enough so that it's possible to easily have different groups that monitor
  different parts of the schedule, and different types of roles/access for
  managing them.

I'm trying not to think about implementation details, but just of the things
I would want to see in an enterprise job scheduler.

If you've got any thoughts along this line, please post them. I'm sure
we've all had enough experience with AutoSys to at least have an opinion
on what's important and what's not.

-- 
Joel

Re: [Clockwork-developers] Kicking around some requirements

[Shawn M. <smc...@ei...>]

This one time, at band camp, Joel Loudermilk wrote:
>=20
> * Management GUI. Of course you can do everything from the command line, =
but
>   if you have a large enough schedule, a GUI might be the only way to
>   manage your schedule effectively. The GUI would be the only component
>   that might should be able to also run on Windows.

Two thoughts:

1) I think we should at least look at the feasibility of the client
running on Windows.  Like it or not, most enterprises have some Windows
in production as well.  I don't see much point in making the server run
on Windows, since security and reliability are design goals.

2) It'd be nice if the GUI layout was subject to configuration, not just
completely auto-generated.  I.E., if you could drag a box from one place
to another, so that it makes more sense to the humans staring blankly at
the screen, and retain that setting across invocations of the GUI.  I'm=20
not sure how to do that scalably, but then I'm not sure how to do it=20
non-scalably either.  :-)

Re: [Clockwork-developers] Kicking around some requirements

[Charles B. H. <br...@do...>]

----- Original Message -----
From: "Joel Loudermilk" <jo...@lo...>
To: <clo...@li...>
Sent: Monday, December 10, 2001 9:35 PM
Subject: [Clockwork-developers] Kicking around some requirements


> In doing a little research on the web by looking at commercial job
schedulers,
> it seems that there are very different approaches to the architecture of a
> scheduler. But before I started thinking about that too much, I wanted to
> build a list of the things I wanted to see in a scheduler.
>
> Here's what I came up with:

<Snip>

> If you've got any thoughts along this line, please post them. I'm sure
> we've all had enough experience with AutoSys to at least have an opinion
> on what's important and what's not.

I have a couple of requirements I thought I'd throw out to see what everyone
thinks:

1) Dynamic evaluation of schedule:  There should be a capability within the
scheduler to evaluate certain values dynamically.  In other words,
definitions of jobs could include, say, some variable, whose value would not
be evaluated until runtime.  Obviously, certain fields would need to be
exempt from dynamic evaluation: job name, start time, etc., but things such
as user name, executable locations, etc., should be able to be evaluated by
the scheduler at runtime.  This is one feature that Autosys doesn't seem to
have.  It does support dynamic evaluation of variables on the target system
(i.e., you can include an environment variable in the executable name, to be
expanded by the shell when the job is run), but not as such in the
scheduler.  I think this feature would be quite valuable to schedule
administrators.

2) An efficient way of entering schedule changes:  It seems to me that there
should not only be a way to edit jobs directly, through, say, a GUI, but
there should also be a command-line method for entering changes.  Further,
this command-line method should support some syntax for making *changes* to
an existing job, rather than just supporting the replacing of one job
definition with a new one.  Perhaps a declarative language somewhat like SQL
could be used to interactively make changes to and display information about
existing jobs.

3) An efficient way of designing schedules:  Autosys lacks (big-time) a way
to design a schedule, and then translate that design into something the
scheduler can take as input.  A GUI application, perhaps, could allow the
user the capability to graphically design a schedule, and could then export
the schedule in some sensible format.  Much of the tedious part of working
in SchedEx must be the dual maintenance of spreadsheets and the actual
schedule.  This might be a difficult application to write, and it should
likely be placed in the somewhat-distant future as compared to other pieces
of the scheduler, but I think it's important.

4) Truly dynamic data storage:  This probably means that we need to
implement the schedule in some sort of database, but we can't use anything
that (for example) requires a HUP to be sent to daemons when changes are
made.  Direct file-based I/O might be okay too, provided we're interested in
developing and maintaining that code as well.  If we choose a database, I
propose that we make a (possibly feeble) attempt to support more than one
database platform.  Depending on what database features we need to use (e.g.
triggers), we might lock out some database platforms, but even so, our
database interactions ought to be standards (ANSI SQL) compliant if
possible.

--
C. Brian Hill
br...@do...