[Config4gnu-developer] Re: J&J reinvent the wheel: they rediscover db & oop :)

[Justin Y. <ju...@sk...>]

On Tue, 2002-11-12 at 12:54, Jason Long wrote:
> Hopefully it's not so much reinventing the wheel as using existing ideas....
> 
> i.e. using an existing database wouldn't help because we want the data to
> exist in the native configuration files. Using a database to store the XML
> would mean we have two copies of configuration-- the one that's in the
> database and the other that's in the original configuration files. However,
> this again brings up the question of caching the configuration, in which
> case using a database to store the configuration is appropriate.

Yes, since one of our goals is to maintain the existing formats and keep
the config files as simple text files which are easy to backup and more
or less 100% reliable, we can't very well store them in the database. 
We could perhaps use the berkley db or gdbm libraries to handle some of
our logging & othe things, although, I would vote for logs being
human-readable text files (just like all other *nix logs).

Something which might have been confusing about my bash script example
is the "commit" command.  This would be more useful if the script made
multiple configuration changes, then said "commit".  That way, unparsers
would not be re-run for *every* change, "Do this.  Ok, now do this.  Ok,
now this...".  Instead, at the end of all the changes, the user says
"Here's a list of things.  Ok, now do what I just said all at once", and
the unparsers and everything else is only run once.

> I do see the project as something along the lines of providing an
> object-oriented database _interface_ to the configuration data. But what
> should an object-oriented database interface look like?


That, I think really is the key.  We just want to make a fancy interface
to not so fancy, often weird, config files.  It follows that we *could*
create a fancy object-oriented way of storing data in some fast binary
format, or even just as raw XML, but this would be relegated to an
optional lower level, something of a very fancy parser/unparser.  I
guess we are doing something similar to a database when you really get
down to it.  Who isn't though? :)


> I agree that Java has a lot of capabilities that will be useful, but I think
> creating a dependency on a JVM would be a serious detriment to the
> acceptability of our project. JVM's are big, take a while to start up, and
> I'm not sure how available they are beyond the Win32, Solaris, and Linux
> environments.


I agree that requiring Java would prevent any sort of wide-spread
acceptance.  Perl does allow some object-oriented-ness with its modules,
and most object support that Java has is also available in Perl, and C
for that matter (if you use glib anyway).  

What has made me think the most about parsers is how important it is
that they are truly extendable.  I can't think of more than a dozen
types of config formats, maybe 2 or 3 dozen if you consider *all* of the
variants.  Still, thats hardly anything, and each parser can be written
using the new Perl module in about 10-20 lines of code, using regex's
and simple if statements.  *If* the (un)parsers only were responsible
for converting to and from native files and a few generic XML tags, I
would say that writing a new parser for each new config *format*
(non-application specific) is the easiest and best way to go.  It also
would be reasonably fast.  This is something like the question "Why
write parsers that only can handle CFGs (the other CFG) when they could
handle CSGs?".

I'm also not sure how to extend something which is basically a list of
regular expressions to try to match against a file.

However, if the parsers need to be smart enough to generate customized
XML tags, such as using <ipaddress> for an IP, and <color> for a
hexadecimal RGB color, then I'm not so sure.  My concept of the parsers
has been that the current "(un)parsers" are to handle file
non-application-specific *formats* (INI, Colon, Flatfile, Resolv, etc.)
which they do now, instead of handling specific config files (i.e., same
parser is used for Samba and PHP since they both use INI formatted
config files).

Now, there certainly needs to be application-specific stuff done shortly
after the parsers get done with things.  It seems that the most flexible
way to do this is to write a layer on top of the parsers that read *XML*
and perform validation, add data types, etc.  That way, a Samba
"validator" would not need to be re-written should Samba ever decide to
use an Apache-style config format.  

Additionally, (and more importantly, since Samba switching formats is a
silly example), similar applications, such as all of the various email
servers, could potentially share very similar validators if the
validators were made to validate the same generic XML.  If the XML is
valid, the validators could also perform the task of turning the generic
XML into more specific XML tags for each particular application's config
files.  This could either be done on a second pass, or on yet another
very thin layer sitting above the validators.

Part of the reason why I think parsers should be generic and only do the
simple task of XML conversion upon request is because currently if a
config file includes another config file, it is the job of the middle
layer to call the parser again on each included file.  That seems like
the correct way to do it.  If the parser did validation, in a single
layer, then the parser would have to read all of the included files
first, and then validate.  Maybe this isn't a huge deal one way or
another, but I'm not sure whether or not parsers & validators should be
merged into one.  We should probably lean toward modular where possible.


Jason, maybe it would be a good idea for us to set aside an hour or two
in the next few days to sit down and revise our current design/schematic
and other documentation to reflect the current situation as well as the
proposed semi-final design.  We can decide on what to do about the UIs
being generic XML editors (which I'm thinking more and more is a great
idea), what to do about the validation & data type adding questions, and
whatever else we can think of.  I hesitate to say we should do it via an
online chat room, because I think that would be slightly since we're
right next to each other.  Maybe we could just take some brief notes and
post what we decide on as a news item.  I vote for that over IRC/IM.

Justin

-- 
SkiingYAC.com
Custom Solutions
su...@sk...
http://www.SkiingYAC.com