Re: [Rest2web-develop] Re: ConfigObj 4 work

[Michael F. <mi...@pc...>]

Nicola Larosa wrote:
>>Ok - we have either a confusion or a difficulty ;-)
>>
>>Everything is written as a string - surprise, surprise. That means when
>>it's read back, it's a string again.
>>
>>It sounds to me like you want the validation process to also do type
>>conversion ? (So that a value you set as an integer - although written
>>and read back in as a string, becomes an integer after validation).
> 
> 
> That's correct.
> 
> 
> 
>>Currently validation checks whether your read values (strings) conform
>>to whatever standard you set. It currently *doesn't* allow the
>>validation process to alter the stored values.
>>
>>``validate.test`` returns a single True/False - pass/fail for each test.
>>
>>I could also write a transforming test that returns a tuple of : (pass,
>>newvalue)
>>
>>This would allow the validation process to also transform values for you.
>>It means two things :
>>
>>    a) You must be allowed to set non string values on a ConfigObj
>>    b) You need a stringify option to transform all non-string values
>>when writing out
> 
> 
> A bidirectional to-from-string typing system. I'm still vague about the
> schema that describes types. Maybe we could take a page from Python and let
> the types be dynamic, but then how do we recognize the type of a value in a
> string read from the config file?
> 

I don't think we need to specify the type of the value stored - it ought
to be unambiguous in the light of the *purpose* of the value. i.e. a
'port number' is always going to be an integer, a 'user name' is always
a string etc. This means that all the validation schema needs to do is
check that the stored value *makes sense* (can be sensibly converted to
the required type).

I think this is the stage to explain the 'schema' system - using
``validate.py``. You can decide whether this meets your needs - or can
be made to meet your needs.

The configspec is a file (list of lines, dictionary, whatever) that is
very like the config file. For each member of the config file, you
specify (effectively) a function to do the testing.

e.g. (from the tests)

        test1='range(30,50)'
        test2='lower()'
        test3='integer'
        test4='alphanumeric'

The function can be specified with or without parameters. If there are
no parameters then the '()' is optional. ``val.test`` then maps the
value in the configspec to a real function call. The function specified
is passed the value (as a string *or* a list of strings when read from
the config file) - *plus* any parameters specified in the config spec.

This means that, from the example above, the member 'test1' must be an
integer value between 30 and 50.

``lower`` checks the value is lowercase, ``integer`` just checks that
the value is an integer etc.

So I *don't* think we need to store any type information in the config
file. Once you have written a general purpose set of functions that
check the values make sense - all you need to do is write the
configspec. The validate method then checks each value individually.

One way of doing it would be to have the functions return the converted
value *or* raise an exception. They would look something like this :

def range(value, min, max):
    if isinstance(value, list):
        raise Exception
    if not value.isdigit():
        raise Exception
    value = int(value)
    if not min <= value <= max:
        raise Exception
    return value

If the member is missing, or the test raises an Exception, then it's a
fail. Otherwise the original value is replaced with the returned new value.

So we don't need to specify any type info in the config file - the type
should be obvious from context.

How does this sound ?
We can work together on a general set of functions that will cover most
of your needs (the syntax already allows for your configspec to pass
keyword arguments or specify multiple tests for a value).

A lot of are written already - but I think they could be clearer.

> 
> 
>>The sort of type checking I'm talking about is when your program has a
>>bug and you accidentally set a value to be ``None`` or to be a class
>>instance by mistake. When you come to write out the file - we crash
>>because the value isn't a string.
>>
>>If we *automatically* stringify non-string values, you won't see the
>>problem till much later.
> 
> 
> If the schema specifies the type for params, you catch the problem right
> away. The stringification should only happen later, when writing to file,
> and shouldn't be burdened with type checking concerns.
> 

Hmm.. I only envisaged using the schema (configspec) to check a file
once you have read it, although there's no reason you couldn't set the
values and then validate.

The schema isn't involved when you *set* individual values. You would
have to validate specifically afterwards.... (allowing validation of
individual values at the point when they are set would be messy - I was
only suggesting type checking in the context of all values being strings
for writing to a file).

> 
> 
>>If we allow the program to crash in the ``write`` method (as currently)
>>- this may actually be called from a totally different part of your
>>application that set the value - hard to track.
>>
>>If we do type checking when you set the value - the code that sets the
>>value is the code that raises the error - easier to trace.
> 
> 
> Indeed.
> 
> 
> 
>>My view is that a ConfigObj represents the *config file*, and so it
>>makes sense to keep values as strings. Validation allows you to check
>>that your string does represent an integer, or represent a bool, or
>>whatever - but you do the conversion yourself.
>>
>>You want (I think ?) the ConfigObj to represent the config data - so you
>>want the validate option to change the type of value from strings to
>>integers etc.
> 
> 
> The config file is just a kind of backend, an implementation detail. It
> could be a database connection, a link to cyberspace, or a time-space
> machine that reaches back to Pico de la Mirandola. ;-) What I'm interested
> in is the config *data*, in fact. The config data should be accurate, and
> consistently typed. Such concerns are better handled in the config system,
> instead of diluting them in the main program.
> 

The nice thing about using ``validate.py`` is that you can use the same
system to check values wherever they come from. Mark Andrews - who I
designed this with, wanted to use it to validate the same values stored
in a database and a file.


> This of course complicates the config system. ZConfig, for instance, goes
> to great lengths to embed the config schema and types in the config objects
> themselves, reaching unheard-of levels of complexity. :-|
> 

Ha....

> 
> 
>>I'm quite happy to add to validate so it can transform values, add
>>*optional* type checking, and add a ``stringify`` option that is *on* by
>>default. This means ConfigObj will work with either model.
>>
>>There will be a new keyword for the validate method - which will cause
>>it to *either* use ``val.test`` or ``val.transforming_test``.
> 
> 
> So, how do you specify what the value types should be? You have talked
> about this before, but it's still not clear to me.
> 


*Hopefully* I've answered this above.


> 
> 
[snip..]
> 
>>>>5) Should I attempt to preserve the amount of whitespace around the
>>>>divider, the type of quoting used, and the amount of whitespace before
>>>>the comment - and then re-use it in the write method ? I'd rather not....
> 
> 
>>>Absolutely not. Reformat mercilessly. After a while, the user will
>>>capitulate, and conform. ;-)
> 
> 
>>lol - OTOH their nicely lined up config files will become all jagged,
>>but unavoidable really.
> 
> 
> Why? I was thinking about the opposite outcome instead, a lesson in tidying
> up. :-)
> 

Well, because if you add white space to your values to line up the
comments etc.... then the write method will simply add a standard amount
of whitespace after the value and before the comment.

It will also write a standard amount of whitespace between value and
divider, rather than padding it so that they line up.

Best Regards,


Fuzzy
http://www.voidspace.org.uk/python