rest2web-develop — A discussion list for rest2web development.

Re: [Rest2web-develop] sections and configspec

[Michael F. <mi...@pc...>]

Peter TB Brett wrote:
> 
> Nicola Larosa wrote:
> 
> 
>>BTW, __repeated__ is a bit long, what about __many__?
>>
>>I also need nested repeated sections:
>>
>>[ cats ]
>>   [[ __many__ ]]
>>       color = string
>>       whiskers = int
>>       [[[ kittens ]]]
>>           [[[[ __many__ ]]]]
>>               age = string
>>[ dogs ]
>>   [[ __many__ ]]
>>       color = string
>>       trained = boolean
>>       [[[ puppies ]]]
>>          [[[[ __many__ ]]]]
>>              age = string
> 
> 
> 
> This is elegant: +1 vote from me.
> 
> BTW, hi everyone, I'm a rest2web user.
> 

Hello Peter,

I'm working on the features you requested for rest2web. Shouldn't take
*too* long I hope. The plugin system will also make it possible to
extend rest2web yourself. I might even make it possible to plugin
alternative templating systems.

The documentation for ConfigObj should be ready soon as well - and then
a beta release will follow. If you do any significant programming in
Python then ConfigObj ought to be useful :-)

I'm hoping Nicola will implement the solution suggested above for
repeated subsections. Nesting them will be interesting though.

Regards,

Fuzzy
http://www.voidspace.org.uk/python

> Peter
> 
> 
> 

Re: [Rest2web-develop] sections and configspec

[Peter TB B. <pe...@pe...>]

Nicola Larosa wrote:

> BTW, __repeated__ is a bit long, what about __many__?
>
> I also need nested repeated sections:
>
> [ cats ]
>    [[ __many__ ]]
>        color = string
>        whiskers = int
>        [[[ kittens ]]]
>            [[[[ __many__ ]]]]
>                age = string
> [ dogs ]
>    [[ __many__ ]]
>        color = string
>        trained = boolean
>        [[[ puppies ]]]
>           [[[[ __many__ ]]]]
>               age = string


This is elegant: +1 vote from me.

BTW, hi everyone, I'm a rest2web user.

Peter



-- 
If you are receiving this at your Hotmail account, please keep in mind
that you might not be able to receive it after November, when Microsoft
implements YABIS (Yet Another Broken Incompatible Standard).

You should switch to e.g. a GMail account or a Yahoo! account if you want
to continue receiving emails from non-Microsoft accounts.

Re: [Rest2web-develop] rest2web and ConfigObj

[Michael F. <mi...@pc...>]

Nicola Larosa wrote:
>>>I've implemented (and checked in) defaults as discussed. I've done one
>>>test - more appreciated ;-)
>>>
>>>It uses the 'default' keyword in checks, and adds a new attribute
>>>'defaults' on sections.
>>>
>>>If a keyword is missing but a default is supplied, then the default is
>>>used. (The default *is* still passed to the check so that type
>>>conversion can be done).
> 
> 
> Here, in validate.py, you were using the "pop" method of dicts. Alas, it
> was introduced in Python 2.3, so it broke compatibility with Python 2.2 .
> 
> Moreover, there was a naked except: clause in ConfigObj.validate that
> silenced the exception (*never* use naked except: clauses).
> 
> I fixed both problems, maybe not in the cleanest way, see the code. All
> tests pass once again on Python 2.2, 2.3 and 2.4 .
> 

Brilliant by the way..... Nice one.

Regards,

Fuzzy
http://www.voidspace.org.uk/python

Re: [Rest2web-develop] sections and configspec

[Nicola L. <ni...@te...>]

>> Or better, how do I specify that, when reading the config file, all
>> subsections of the "cats" section have to be validated against the "cat"
>> configspec, and that all subsections of the "dogs" section have to be
>> validated against the "dog" configspec?

> You want named types of configspec sections and to be able to apply
> those types in your config file.

Not necessarily "named". :-)


> In this particular example you are saying that all 'dogs' (sub-sections
> in the dogs section) should use the [[dog]] configspec and 'cats'
> (sub-sections in the cats section) should use the 'cat' configspec.

Yes.


> Two possible implementations spring to mind - named sections, or
> repeated sections. Repeated sections would meet your specific example
> here - but aren't a generic solution.

I don't care. I don't know if YAGNI, but surely INNIN (I'm Not Needing It
Now), where "it" is "named sections". Repeated sections are enough.


> I'll suggest an alternative implementation of named sections, that is
> less complicated than yesterdays.

But probably still too complicated. ;-)


> Named Sections
> ==============
> 
> The trouble is that with a hierarchical structure (rather than a flat
> one) there is no unique identifier per section (sections have non unique
> names).
> 
> The alternatives are either to allow sections to be named - a magic
> value (either '__name__' or '__type__') or to use something like a
> dotted syntax to reference sections 'section.sub-section.sub-sub-section'.

Dotted syntax would yield unique ids, so magic values are not needed, and
best avoided, since they're ugly. But even it is not needed anyway. :-)


> I think my previous explanation was overly complex. I favour this
> solution - using a '__type__' magic value in sections.

I don't, it's ugly.


> [ cats ]
>     [[ felix ]]
> 	__type__ = cat
>         color = black
>         whiskers = 5
>     [[ miao ]]
> 	__type__ = cat
>         color = white
>         whiskers = 7
> [ dogs ]
>     [[ fido ]]
> 	__type__ = dog
>         color = brown
>         trained = yes
>     [[ lassie ]]
> 	__type__ = dog
>         color = grey
>         trained = no
>     [[ rintintin ]]
> 	__type__ = dog
>         color = pink
>         trained = no

Magic values in the config file even, not just in the configspec? Oh, the
horrors. :-)


> When parsing a configspec any __type__ values are not treated as checks,
> but a *separate* dictionary of types is kept.
> 
> If the validate method encounters a section with a __type__ value it
> looks for the configspec for this section in the type dictionary.
> 
> There are a couple of implementation questions if you like this
> solution.....

I don't. :-) We don't really need to give users the ability to define
custom types. That way lies madness, and XML, and ZConfig. ;-)


> Repeated sections could solve this particular example without the use of
>  a magic value in the config file.
> 
> You could do something like
> 
> [ cats ]
>     [[ __repeated__ ]]
>         color = string
>         whiskers = int
> [ dogs ]
>     [[ __repeated__ ]]
> 	__type__ = dog
>         color = string
>         trained = boolean
> 
> Effectively saying that any subsection of 'cats' or 'dogs' has to meet
> the __repeated__ specification. This isn't useful if you want to be able
> to have a 'dog' somewhere else of course.....

I don't want to. :-)


> Using the dotted syntax allows you to avoid magic values in your
> configspec (which is less useful).

But it is enough, and less ugly.


> You could do :
> 
> [ dogs ]
>     [[ fido ]]
> 	__type__ = dogs.dog
>         color = brown
>         trained = yes
>     [[ lassie ]]
> 	__type__ = dogs.dog
>         color = grey
>         trained = no
>     [[ rintintin ]]
> 	__type__ = dogs.dog
>         color = pink
>         trained = no
> 
> I don't like this personally.....

Me neither. I don't want to pollute the config file this way. Types are not
the users' problem.

BTW, __repeated__ is a bit long, what about __many__?

I also need nested repeated sections:

[ cats ]
   [[ __many__ ]]
       color = string
       whiskers = int
       [[[ kittens ]]]
           [[[[ __many__ ]]]]
               age = string
[ dogs ]
   [[ __many__ ]]
       color = string
       trained = boolean
       [[[ puppies ]]]
          [[[[ __many__ ]]]]
              age = string

allowing one to write:

[ cats ]
    [[ felix ]]
        color = black
        whiskers = 5
        [[[ kittens ]]]
            [[[[ mia ]]]]
                age = 1
            [[[[ amy ]]]]
                age = 3
[ dogs ]
    [[ fido ]]
        color = brown
        trained = yes
        [[[ puppies ]]]
            [[[[ fuffy ]]]]
                age = 2
    [[ lassie ]]
        color = grey
        trained = no


Yes, I like this. All those brackets are a little obtrusive, though. What
if we only keep the closing ones?

[ cats ]
    [ felix ]]
        color = black
        whiskers = 5
        [ kittens ]]]
            [ mia ]]]]
                age = 1
            [ amy ]]]]
                age = 3
[ dogs ]
    [ fido ]]
        color = brown
        trained = yes
        [ puppies ]]]
            [ fuffy ]]]]
                age = 2
    [ lassie ]]
        color = grey
        trained = no

Nicer. They're not balanced anymore, but who cares? They just form tokens
here. Look, it's almost as if an invisible rope pulls and stretches them
rightward. :-)

-- 
Nicola Larosa - ni...@te...

When I thought life had some purpose - Then I thought I had some choice
  (I was running blind)
And I made some value judgments - In a self-important voice
  (I was outa line)
But then absurdity came over me - And I longed to lose control
  (into no mind)
Oh all I ever wanted - Was just to come in from the cold
 -- Joni Mitchell, Come in from the cold, Night Ride Home, 1991

Re: [Rest2web-develop] sections and configspec

[Michael F. <mi...@pc...>]

Nicola Larosa wrote:
>>>The only thing really missing now is having "lists of sections", for the
>>>cases when their number cannot be determined in advance. Maybe some easy
>>>workaround can be found.
> 
> 
>>I don't understand what you want. If you give me an example I'll think
>>about it. It sounds like you need a way of referencing (naming)
>>individual sections. Is this just for the programmer - or for the system
>>administrator (the user).
> 
> 
> My cousin has several cats and several dogs, and wants to "configure" them.
> (I know, peculiar cousin.) I don't exactly know how many they are, and
> moreover, their number could change with time.
> 
> All the cats have the same params among them, the same with dogs, but cats'
> params are slightly different from dogs' params.
> 
> I write this configspec:
> 
> [ cats ]
>     [[ cat ]]
>         color = string
>         whiskers = int
> [ dogs ]
>     [[ dog ]]
>         color = string
>         trained = boolean
> 
> Then I write this config file:
> 
> [ cats ]
>     [[ felix ]]
>         color = black
>         whiskers = 5
>     [[ miao ]]
>         color = white
>         whiskers = 7
> [ dogs ]
>     [[ fido ]]
>         color = brown
>         trained = yes
>     [[ lassie ]]
>         color = grey
>         trained = no
>     [[ rintintin ]]
>         color = pink
>         trained = no
> 
> How do I specify that the "cat" configspec has to be used for the "felix"
> and "miao" sections, while the "dog" configspec has to be used for the
> "fido", "lassie" and "rintintin" sections?
> 
> Or better, how do I specify that, when reading the config file, all
> subsections of the "cats" section have to be validated against the "cat"
> configspec, and that all subsections of the "dogs" section have to be
> validated against the "dog" configspec?
> 

You want named types of configspec sections and to be able to apply
those types in your config file.

In this particular example you are saying that all 'dogs' (sub-sections
in the dogs section) should use the [[dog]] configspec and 'cats'
(sub-sections in the cats section) should use the 'cat' configspec.

Two possible implementations spring to mind - named sections, or
repeated sections. Repeated sections would meet your specific example
here - but aren't a generic solution. I'll suggest an alternative
implementation of named sections, that is less complicated than yesterdays.

Named Sections
==============

The trouble is that with a hierarchical structure (rather than a flat
one) there is no unique identifier per section (sections have non unique
names).

The alternatives are either to allow sections to be named - a magic
value (either '__name__' or '__type__') or to use something like a
dotted syntax to reference sections 'section.sub-section.sub-sub-section'.

I think my previous explanation was overly complex. I favour this
solution - using a '__type__' magic value in sections. These can be
treated specially when parsing configspecs and are not treated specially
by the ``write`` method.

I write this configspec:

[ cats ]
    [[ cat ]]
	__type__ = cat
        color = string
        whiskers = int
[ dogs ]
    [[ dog ]]
	__type__ = dog
        color = string
        trained = boolean

Then I write this config file:

[ cats ]
    [[ felix ]]
	__type__ = cat
        color = black
        whiskers = 5
    [[ miao ]]
	__type__ = cat
        color = white
        whiskers = 7
[ dogs ]
    [[ fido ]]
	__type__ = dog
        color = brown
        trained = yes
    [[ lassie ]]
	__type__ = dog
        color = grey
        trained = no
    [[ rintintin ]]
	__type__ = dog
        color = pink
        trained = no

When parsing a configspec any __type__ values are not treated as checks,
but a *separate* dictionary of types is kept.

If the validate method encounters a section with a __type__ value it
looks for the configspec for this section in the type dictionary.

There are a couple of implementation questions if you like this
solution.....

Repeated sections could solve this particular example without the use of
 a magic value in the config file.

You could do something like

[ cats ]
    [[ __repeated__ ]]
        color = string
        whiskers = int
[ dogs ]
    [[ __repeated__ ]]
	__type__ = dog
        color = string
        trained = boolean

Effectively saying that any subsection of 'cats' or 'dogs' has to meet
the __repeated__ specification. This isn't useful if you want to be able
to have a 'dog' somewhere else of course.....

Using the dotted syntax allows you to avoid magic values in your
configspec (which is less useful).

You could do :

[ dogs ]
    [[ fido ]]
	__type__ = dogs.dog
        color = brown
        trained = yes
    [[ lassie ]]
	__type__ = dogs.dog
        color = grey
        trained = no
    [[ rintintin ]]
	__type__ = dogs.dog
        color = pink
        trained = no

I don't like this personally.....

Regards,

Fuzzy
http://www.voidspace.org.uk/python

Re: [Rest2web-develop] rest2web and ConfigObj

[Nicola L. <ni...@te...>]

>> I've implemented (and checked in) defaults as discussed. I've done one
>> test - more appreciated ;-)
>> 
>> It uses the 'default' keyword in checks, and adds a new attribute
>> 'defaults' on sections.
>>
>> If a keyword is missing but a default is supplied, then the default is
>> used. (The default *is* still passed to the check so that type
>> conversion can be done).

Here, in validate.py, you were using the "pop" method of dicts. Alas, it
was introduced in Python 2.3, so it broke compatibility with Python 2.2 .

Moreover, there was a naked except: clause in ConfigObj.validate that
silenced the exception (*never* use naked except: clauses).

I fixed both problems, maybe not in the cleanest way, see the code. All
tests pass once again on Python 2.2, 2.3 and 2.4 .

-- 
Nicola Larosa - ni...@te...

When I thought life had some purpose - Then I thought I had some choice
  (I was running blind)
And I made some value judgments - In a self-important voice
  (I was outa line)
But then absurdity came over me - And I longed to lose control
  (into no mind)
Oh all I ever wanted - Was just to come in from the cold
 -- Joni Mitchell, Come in from the cold, Night Ride Home, 1991

Re: [Rest2web-develop] rest2web and ConfigObj

[Nicola L. <ni...@te...>]

>> The only thing really missing now is having "lists of sections", for the
>> cases when their number cannot be determined in advance. Maybe some easy
>> workaround can be found.

> I don't understand what you want. If you give me an example I'll think
> about it. It sounds like you need a way of referencing (naming)
> individual sections. Is this just for the programmer - or for the system
> administrator (the user).

My cousin has several cats and several dogs, and wants to "configure" them.
(I know, peculiar cousin.) I don't exactly know how many they are, and
moreover, their number could change with time.

All the cats have the same params among them, the same with dogs, but cats'
params are slightly different from dogs' params.

I write this configspec:

[ cats ]
    [[ cat ]]
        color = string
        whiskers = int
[ dogs ]
    [[ dog ]]
        color = string
        trained = boolean

Then I write this config file:

[ cats ]
    [[ felix ]]
        color = black
        whiskers = 5
    [[ miao ]]
        color = white
        whiskers = 7
[ dogs ]
    [[ fido ]]
        color = brown
        trained = yes
    [[ lassie ]]
        color = grey
        trained = no
    [[ rintintin ]]
        color = pink
        trained = no

How do I specify that the "cat" configspec has to be used for the "felix"
and "miao" sections, while the "dog" configspec has to be used for the
"fido", "lassie" and "rintintin" sections?

Or better, how do I specify that, when reading the config file, all
subsections of the "cats" section have to be validated against the "cat"
configspec, and that all subsections of the "dogs" section have to be
validated against the "dog" configspec?

-- 
Nicola Larosa - ni...@te...

When I thought life had some purpose - Then I thought I had some choice
  (I was running blind)
And I made some value judgments - In a self-important voice
  (I was outa line)
But then absurdity came over me - And I longed to lose control
  (into no mind)
Oh all I ever wanted - Was just to come in from the cold
 -- Joni Mitchell, Come in from the cold, Night Ride Home, 1991

Re: [Rest2web-develop] rest2web and ConfigObj

[Michael F. <mi...@pc...>]

Nicola Larosa wrote:
> [Is it intended, this not going to the list?]
> 

No - a mistake. Sorry

> 
> 
>>>I've implemented (and checked in) defaults as discussed. I've done one
>>>test - more appreciated ;-)
>>>
>>>It uses the 'default' keyword in checks, and adds a new attribute
>>>'defaults' on sections.
>>>
>>>If a keyword is missing but a default is supplied, then the default is
>>>used. (The default *is* still passed to the check so that type
>>>conversion can be done).
>>>
>>>If a keyword is missing and no default is supplied then a
>>>ValidateMissingValue error is raised. (Check fails).
>>>
>>>The write method doesn't write out any values that come from defaults.
>>>If a whole section is missing the empty marker is still written out.
>>>
>>>Setting a value that was previously obtained from a default, removes it
>>>from the list of defaults - so it will be written out.
> 
> 
> Wonderful, I'll check this ASAP.
> 
> 
> 
>>>I *haven't* implemented a way of repeating whole sections in the configspec.
>>>
>>>If I *was* to do that I would probably do it by allowing you to name
>>>sections of a ConfigObj with the magic value ``'__name__' = name``
>>>
>>>You could then assign a section with (?) ``'__copy__' = name`` and it
>>>will copy the section (including recursive subsections ?).
>>>
>>>I don't like magic values though...... It's also a nuisance as you'd
>>>have to parse the whole thing and then go back through sorting out the
>>>magic values :-) What to do with the ``write`` method would also be
>>>interesting. (I'd probably *not* write out sections with '__copy__' as a
>>>value, and delete '__copy__' if you assign to a value.. not easy though
>>>in case someone sets a value in a *parent* section - have to walk up the
>>>parents deleting the '__copy__' value as well....).
> 
> 
> Yuck, too complicated. I'll first try your suggestion to manually assign
> configspecs to sections, even if it's already been used in another one.
> 

No problem - this *will* work. The validate method just uses the
'configspec' attribute for checks - so you can assign to it easily. You
still have to do it manually of course... :-)

> The only thing really missing now is having "lists of sections", for the
> cases when their number cannot be determined in advance. Maybe some easy
> workaround can be found.

I don't understand what you want. If you give me an example I'll think
about it. It sounds like you need a way of referencing (naming)
individual sections. Is this just for the programmer - or for the system
administrator (the user).

If you just want to obtain a list of all the sections in a config file -
then that would be quite easy to obtain....

Regards,


Fuzzy
http://www.voidspace.org.uk/python

> 
> --
> Nicola Larosa - ni...@te...
> 
> When I thought life had some purpose - Then I thought I had some choice
>   (I was running blind)
> And I made some value judgments - In a self-important voice
>   (I was outa line)
> But then absurdity came over me - And I longed to lose control
>   (into no mind)
> Oh all I ever wanted - Was just to come in from the cold
>  -- Joni Mitchell, Come in from the cold, Night Ride Home, 1991
> 

Re: [Rest2web-develop] rest2web and ConfigObj

[Nicola L. <ni...@te...>]

> I've also done a bugfix in ConfigObj - I hope this doesn't cross over
> with anything you've done Nicola. It was only one line I changed (line
> 805) :
> 
>     if os.path.isfile:
> 
> became :
> 
>     if os.path.isfile(self.filename):
> 
> Are you alright with the changes to validate/ConfigObj Nicola ? If not,
> I can do them. I'd like to get that released as well. The docs need
> updating - but it depends on the implementation details of the changes
> we discussed.........

Sorry for disappearing during the last few days, Real Life Intruding Again
(tm).

Yes, the changes to ConfigObj we discussed are fine with me, and it would
be great if you would be able to make them.

In the meanwhile I'll try to write doctests for odict.py .

-- 
Nicola Larosa - ni...@te...

Abstraction is layering ignorance on top of reality.
 -- rpg, cited by Richard P. Gabriel, November 2002

[Rest2web-develop] rest2web and ConfigObj

[Michael F. <mi...@pc...>]

Hello All,

Updates to rest2web. I'm working towards a new release.

    I've removed some image files
    Also cleaned up the gallery files
    More work on the gallery/plugin system
    py2exe-setup.py now works (builds rest2web as a windoze executable)
    make_dist.py now works - builds a distribution from the SVN
repository files

The gallery is nearly ready. I also have some thorny questions about
encodings and the plugin system to resolve.

Issues to resolve before next release :

    finish gallery todo list
    finish plugin system
    finish gallery and plugins docs
    implement 'file' keyword
    make 'section' an ordered dictionary
    add keyword to specify ordering of pages in section

I've also done a bugfix in ConfigObj - I hope this doesn't cross over
with anything you've done Nicola. It was only one line I changed (line
805) :

    if os.path.isfile:

became :

    if os.path.isfile(self.filename):

Are you alright with the changes to validate/ConfigObj Nicola ? If not,
I can do them. I'd like to get that released as well. The docs need
updating - but it depends on the implementation details of the changes
we discussed.........

Best Regards,


Fuzzyman
http://www.voidspace.org.uk/python

      

Re: [Rest2web-develop] Last Commit

[Michael F. <mi...@pc...>]

Hello Nicola,

-----Original Message-----
   >From: "Nicola Larosa"<ni...@te...>
   >Sent: 07/08/05 22:29:33
   >To: "res...@li..."<res...@li...>
   >Subject: Re: [Rest2web-develop] Last Commit
     >>> The first one: there is no default value. There should be a way to check
   >>> a param's value, when it is set in the config file, or else to fill in a
   >>> default value when it is not. As it is, one may leave the configspec
   >>> empty, meaning that the value may not be present: alas, by doing so one
   >>> loses any check on the value, and there's no default value anyway.
   >
   >> Easy - allow all the functions to take a keyword argument 'default'.
   >> Amend ConfigObj so that instead of automatically failing missing values
   >> it calls the tests with a 'missing=True' keyword.
   >>
   >> The test function can then decide if being missing is an automatic fail.
   >> If a default is available - then it's not.
   >
   >It sounds easy enough, I'll try it.
   >

Cool - you can implement the whole thing.
In ConfigObj.validate it currently *automatically* fails missing keys. Parsing the configspec will automatically create missing sections already (but they'llmbe empty).

Instead it should now merely pass a ``missing=True`` keyword to validate.check and ``None`` as the value. 

In Validator.check - if missing = True and no default keyword is supplied in the configspec - then an Exception should be raised (check fails).

Otherwise the default should be passed to the check function instead.

Create a  new section attribute ``_defaults``.  If a test passes where a value was missing (so the default was used) ConfigObj.validate should add the key to ``_defaults``. Section.__setitem__ should remove a key from ``_defaults`` if the key is altered other than by the validate method.

``write`` should then not write out keys in the ``_defaults`` list. Easy hey :-)

You still end up writing section markers for empty sections though.

[snip..]
    >>>> Hmmm... Well I want to get validate working as a Section attribute
   >>>> rather than an attribute just on ConfigObj...
   >
   >> In the version you now have, the configspec is a Section attribute - a
   >> dictionary of tests for that section.
   >
   >So I can already reuse the same configspec to instantiate a Section, and
   >attach it to a ConfigObj? Cool.
   >

Hmm.. Not quite I don't think. You can do :

c = ConfigObj(filename)
#
# suppose configspec is a dictionary of checks
c['section'] = {}
c['section'].configspec = configspec
c['section2'] = {}
c['section2'].configspec = configspec

Which is nearly what you said ?

Best Regards, 


Fuzzyman 
http://www.voidspace.org.uk/python 
   >-------------------------------------------------------
   >SF.Net email is Sponsored by the Better Software Conference & EXPO
   >September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
   >Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
   >Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
   >_______________________________________________
   >Rest2web-develop mailing list
   >Res...@li...
   >https://lists.sourceforge.net/lists/listinfo/rest2web-develop
   >
   >
   >

Re: [Rest2web-develop] Last Commit

[Nicola L. <ni...@te...>]

>> The first one: there is no default value. There should be a way to check
>> a param's value, when it is set in the config file, or else to fill in a
>> default value when it is not. As it is, one may leave the configspec
>> empty, meaning that the value may not be present: alas, by doing so one
>> loses any check on the value, and there's no default value anyway.

> Easy - allow all the functions to take a keyword argument 'default'.
> Amend ConfigObj so that instead of automatically failing missing values
> it calls the tests with a 'missing=True' keyword.
>
> The test function can then decide if being missing is an automatic fail.
> If a default is available - then it's not.

It sounds easy enough, I'll try it.


> We *may* run into trouble with quotes in default values here, as we're
> using ConfigObj to parse the configspec. If it is a problem we'll work on
> a special mode for ConfigObj when parsing the configspec.

A few tests will highlight any problems.


>> This default value business is very handy, because it allows to specify,
>> in the config file, just the values that differ from the defaults.

> So in the configspec :
> 
> Key =  integer
> 
> Would fail if key is missing. Whereas :
> 
> Key = integer(default=3)
> 
> Would substitute 3 for missing values.
> 
> *However* - this would cause the default value to be written out by the
> ``write`` method.

That's what my ZConfig extension currently does, and it's quite ungainly.


> (Unless we create a new private Section attribute for
> 'defaults' that aren't written out unless they get set from outside the
> ConfigObj).

It would be great to carry the defaults around, and only write values to
file when they are different from the defaults. This only applies to
scalars, not to sections, obviously.


>> The second problem is deeper: the configspec is monolitic, and I need
>> it to be modular. That is, some sections contain a number of
>> homogeneous subsections, and there's no way to know beforehand how many
>> they will be. They should reuse the same configspec, so should be
>> somehow declared of a certain "type", and validation should have a way
>> to pick the right configspec fragment.
>>
>> This may be related to what you wrote:
>>
>>> Hmmm... Well I want to get validate working as a Section attribute
>>> rather than an attribute just on ConfigObj...

> In the version you now have, the configspec is a Section attribute - a
> dictionary of tests for that section.

So I can already reuse the same configspec to instantiate a Section, and
attach it to a ConfigObj? Cool.


>> ZConfig does this by reusing section types in "multisections". Probably
>> we could do something similar by validating the configspec for
>> Sections, and then composing them in the complete configspec for the
>> ConfigObj.

> You can already manually add them to a Section. You could add syntax to
> the configspec for naming and repeating them...

I'll try that.


>> If we can solve these two problems, I think we can set the Configobj
>> status straight to "beta", from my POV at least. ;-)

> Cool. Let me know your response to my suggestion. It only means using
> the existing syntax of validate.py - using keyword arguments (so only
> changing the functions). I wonder if it is too clumsy for you though ?

Not at all, I find it convenient enough, and "explicit is better than
implicit" anyway. :-)

-- 
Nicola Larosa - ni...@te...

Abstraction is layering ignorance on top of reality.
 -- rpg, cited by Richard P. Gabriel, November 2002

Re: [Rest2web-develop] Last Commit

[Michael F. <mi...@pc...>]

Hello Nicola,

-----Original Message-----
   >From: "Nicola Larosa"<ni...@te...>
   >Sent: 07/08/05 19:21:19
   >To: "res...@li..."<res...@li...>
   >Subject: Re: [Rest2web-develop] Last Commit
   >
   >Some work done this weekend. I'll refer a few times to ZConfig, please bear
   >with me. ;-)
   >
   >First, I committed the files from your email.
   >
   >Then I fixed the doctests for both configobj.py and validate.py to make
   >them work on Python 2.2.3 . I even tried Python 2.1.3, but it would have
   >taken too many changes, so I dropped it. Only one test, for validate.py,
   >still fails on 2.2.3, for trivial reasons.
   >

Cool - nice one.

   >Finally I converted a rather elaborate config file from ZConfig format to
   >Configobj. All is well as far as values go, but I have two non-trivial
   >problems with validation.
   >
   >The first one: there is no default value. There should be a way to check a
   >param's value, when it is set in the config file, or else to fill in a
   >default value when it is not. As it is, one may leave the configspec empty,
   >meaning that the value may not be present: alas, by doing so one loses any
   >check on the value, and there's no default value anyway.
   >

Easy - allow all the functions to take a keyword argument 'default'. Amend ConfigObj so that instead of automatically failing missing values it calls the tests with a 'missing=True' keyword.

The test function can then decide if being missing is an automatic fail. If a default is available - then it's not.

We *may* run into trouble with quotes in default values here, as we're using ConfigObj to parse the configspec. If it is a problem we'll work on a special mode for ConfigObj when parsing the configspec.

   >This default value business is very handy, because it allows to specify, in
   >the config file, just the values that differ from the defaults.
   >

So in the configspec :

Key =  integer

Would fail if key is missing. Whereas :

Key = integer(default=3)

Would substitute 3 for missing values.

*However* - this would cause the default value to be written out by the ``write`` method. (Unless we create a new private Section attribute for 'defaults' that aren't written out unless they get set from outside the ConfigObj).


   >The second problem is deeper: the configspec is monolitic, and I need it to
   >be modular. That is, some sections contain a number of homogeneous
   >subsections, and there's no way to know beforehand how many they will be.
   >They should reuse the same configspec, so should be somehow declared of a
   >certain "type", and validation should have a way to pick the right
   >configspec fragment.
   >
   >This may be related to what you wrote:
   >
   >> Hmmm... Well I want to get validate working as a Section attribute rather
   >> than an attribute just on ConfigObj...
   >

In the version you now have, the configspec is a Section attribute - a dictionary of tests for that section.

   >ZConfig does this by reusing section types in "multisections". Probably we
   >could do something similar by validating the configspec for Sections, and
   >then composing them in the complete configspec for the ConfigObj.
   >

You can already manually add them to a Section. You could add syntax to the configspec for naming and repeating them...

   >By the way, maybe you could give a look to the ZConfig doc PDF, if you
   >didn't already. There may be some good ideas in there, amidst all the
   >formality ;-) :
   >
   >http://svn.zope.org/*checkout*/ZConfig/trunk/doc/zconfig.pdf?rev=30398
I will do.

   >
   >If we can solve these two problems, I think we can set the Configobj status
   >straight to "beta", from my POV at least. ;-)
   >

Cool. Let me know your response to my suggestion. It only means using the existing syntax of validate.py - using keyword arguments (so only changing the functions). I wonder if it is too clumsy for you though ?

Regards,


Fuzzy
http://www.voidspace.org.uk/python 

   >-- 
   >Nicola Larosa - ni...@te...
   >
   >Abstraction is layering ignorance on top of reality.
   > -- rpg, cited by Richard P. Gabriel, November 2002
   >
   >
   >
   >
   >-------------------------------------------------------
   >SF.Net email is Sponsored by the Better Software Conference & EXPO
   >September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
   >Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
   >Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
   >_______________________________________________
   >Rest2web-develop mailing list
   >Res...@li...
   >https://lists.sourceforge.net/lists/listinfo/rest2web-develop
   >
   >

Re: [Rest2web-develop] Last Commit

[Nicola L. <ni...@te...>]

Some work done this weekend. I'll refer a few times to ZConfig, please bear
with me. ;-)

First, I committed the files from your email.

Then I fixed the doctests for both configobj.py and validate.py to make
them work on Python 2.2.3 . I even tried Python 2.1.3, but it would have
taken too many changes, so I dropped it. Only one test, for validate.py,
still fails on 2.2.3, for trivial reasons.

Finally I converted a rather elaborate config file from ZConfig format to
Configobj. All is well as far as values go, but I have two non-trivial
problems with validation.

The first one: there is no default value. There should be a way to check a
param's value, when it is set in the config file, or else to fill in a
default value when it is not. As it is, one may leave the configspec empty,
meaning that the value may not be present: alas, by doing so one loses any
check on the value, and there's no default value anyway.

This default value business is very handy, because it allows to specify, in
the config file, just the values that differ from the defaults.

The second problem is deeper: the configspec is monolitic, and I need it to
be modular. That is, some sections contain a number of homogeneous
subsections, and there's no way to know beforehand how many they will be.
They should reuse the same configspec, so should be somehow declared of a
certain "type", and validation should have a way to pick the right
configspec fragment.

This may be related to what you wrote:

> Hmmm... Well I want to get validate working as a Section attribute rather
> than an attribute just on ConfigObj...

ZConfig does this by reusing section types in "multisections". Probably we
could do something similar by validating the configspec for Sections, and
then composing them in the complete configspec for the ConfigObj.

By the way, maybe you could give a look to the ZConfig doc PDF, if you
didn't already. There may be some good ideas in there, amidst all the
formality ;-) :

http://svn.zope.org/*checkout*/ZConfig/trunk/doc/zconfig.pdf?rev=30398

If we can solve these two problems, I think we can set the Configobj status
straight to "beta", from my POV at least. ;-)

-- 
Nicola Larosa - ni...@te...

Abstraction is layering ignorance on top of reality.
 -- rpg, cited by Richard P. Gabriel, November 2002

Re: [Rest2web-develop] Last Commit

[Michael F. <mi...@pc...>]

-----Original Message-----
   >From: "Nicola Larosa"<ni...@te...>
   >Sent: 04/08/05 17:16:54
   >To: "res...@li..."<res...@li...>
   >Subject: Re: [Rest2web-develop] Last Commit
   >
   >> I've improved ConfigObjError - it now has
   >> line number, message, and line attributes. This means that the
   >> ``errors`` list you get in the case of multiple errors is just a
   >> list of errors - not a list of tuples !
   >> 
   >> The comments test also passes again.
   >
   >Great work.
   >
   >
   >> Final commit until Monday. ... I *might* get  enough time to finish
   >> ConfigObj and work on cleaning up the docs. We can then do an alpha
   >> release.
   >
   >I was hoping I got the chance to hack on it a bit this weekend, but looks
   >like you want to keep that option open for yourself.
   >

Hmmm... Well I want to get validate working as a Section attribute rather than an attribute just on ConfigObj - but I can restrict myself to that. That should make it easy to merge changes.

Oh yes, and the two Interpolation errors need to inherit from a single ``InterpolationError`` base class.  Should this be a subclass of ConfigObError ?

My other priority is the docs - so I promise to leave the rest of ConfigObj alone.... (or clear up the mess afterwards myself).

   >Oh well, at the very least I'll try using it to substitute ZConfig in my
   >job's main app, that should be a good usage test. I'll let you know.
   >

That would be very good - probably the best way of uncovering those lurking bugs.

I'm deleting ConfigObj 3 from my path and *forcing* all my scripts and programs to use the new version....

Regards,

Fuzz

   >-- 
   >Nicola Larosa - ni...@te...
   >
   >One of the things I routinely tell people is that if it's in the news,
   >don't worry about it. By definition, "news" means that it hardly ever
   >
   >

[Message truncated. Tap Edit->Mark for Download to get remaining portion.]

Re: [Rest2web-develop] Last Commit

[Nicola L. <ni...@te...>]

> I've improved ConfigObjError - it now has
> line number, message, and line attributes. This means that the
> ``errors`` list you get in the case of multiple errors is just a
> list of errors - not a list of tuples !
> 
> The comments test also passes again.

Great work.


> Final commit until Monday. ... I *might* get  enough time to finish
> ConfigObj and work on cleaning up the docs. We can then do an alpha
> release.

I was hoping I got the chance to hack on it a bit this weekend, but looks
like you want to keep that option open for yourself.

Oh well, at the very least I'll try using it to substitute ZConfig in my
job's main app, that should be a good usage test. I'll let you know.

-- 
Nicola Larosa - ni...@te...

One of the things I routinely tell people is that if it's in the news,
don't worry about it. By definition, "news" means that it hardly ever
happens. If a risk is in the news, then it's probably not worth
worrying about. When something is no longer reported - automobile
deaths, domestic violence - when it's so common that it's not news,
then you should start worrying. -- Bruce Schneier, May 2005

[Rest2web-develop] Last Commit

[Michael F. <mi...@pc...>]

Hello Nicola,

Final commit until Monday. I've improved ConfigObjError - it now has
line number, message, and line attributes. This means that the
``errors`` list you get in the case of multiple errors is just a list of
errors - not a list of tuples !

The comments test also passes again.

Over the weekend I have a big timber show on that I'm helping with. I
*might* get  enough time to finish ConfigObj and work on cleaning up the
docs. We can then do an alpha release.

Regards,


Fuzzy
http://www.voidspace.org.uk/python

Re: [Rest2web-develop] Down with writein

[Nicola L. <ni...@te...>]

> Part of the point is that the comments *are* associated with the config
> items. That way if you're constructing config files programattically,
> you can say - this comment belongs with this value.

You're right, I wasn't keeping that in mind.


> If you don't care about the association (or how it's done internally) -
> then the current implementation 'just works'.

Yes.


> Interleaving comments and the other values sounds like a nightmare !

Probably it's not simple, and gives little gain over what you already did,
so let's just forget about it. :-)

-- 
Nicola Larosa - ni...@te...

If there are a few shark attacks in Florida - and a graphic movie -
suddenly every swimmer is worried. More people are killed every year by
pigs than by sharks, which shows you how good we are at evaluating risk.
 -- Bruce Schneier, May 2005

Re: [Rest2web-develop] Down with writein

[Michael F. <mi...@pc...>]

Nicola Larosa wrote:
[snip..]
>>I thought this could end up associating comments with the wrong value.
>>As a side effect it means that there is no way of providing a comment
>>for the *first* key/section. It will all be swallowed up by initial_comment.
> 
> 
> Why don't you do away with the notions of initial_comment, final_comment,
> and associating comments with config items, and instead keep all non-data
> lines just as they are?
> 
> You could do this by using a third sequence, say "self.comments", and
> interleaving its elements with self.scalars and self.sections somehow.
> 

Part of the point is that the comments *are* associated with the config
items. That way if you're constructing config files programattically,
you can say - this comment belongs with this value.

If you don't care about the association (or how it's done internally) -
then the current implementation 'just works'.

Interleaving comments and the other values sounds like a nightmare !

Best Regards,

Fuzzy
http://www.voidspace.org.uk/python

[Rest2web-develop] ConfigObj and Comments

[Michael F. <mi...@pc...>]

Hello Nicola,

There  is an updated version of configobj in SVN.

It now preserves all comments in config files. (and has no writein method).
It's not yet documented - I've broken the test in ``_handle_comments``
and done no new tests for the new functionality :-)

These things will come.

As a side effect - you can now have multiple lines of comments above a
section marker or key. Any comments above the *first* key/section will
be put in ``initial_comment though`` !

I want to improve error handling (give the exceptions line number and
original line attributes) and slightly change the way validation works.
When that's done - it's just the documentation and it's ready....
(slowly getting there).

Best Regards,

Fuzzyman
http://www.voidspace.org.uk/python

Re: [Rest2web-develop] Down with writein

[Nicola L. <ni...@te...>]

> The main use case was to allow the creation of a ConfigObj that
> represented part of a file. It meant making an arbitrary decision about
> what to do if the ConfigObj was changed (the keys in the object no
> longer match the keys in the file). On the basis of YAGNI I figured I
> could lose it...

Go ahead, lighter is better. ;-)


> Would you prefer it if I just kept all comments, even if separated by
> blank lines ?

Yes. :-) Best if you could keep blank lines too. :-D


> I thought this could end up associating comments with the wrong value.
> As a side effect it means that there is no way of providing a comment
> for the *first* key/section. It will all be swallowed up by initial_comment.

Why don't you do away with the notions of initial_comment, final_comment,
and associating comments with config items, and instead keep all non-data
lines just as they are?

You could do this by using a third sequence, say "self.comments", and
interleaving its elements with self.scalars and self.sections somehow.

-- 
Nicola Larosa - ni...@te...

If there are a few shark attacks in Florida - and a graphic movie -
suddenly every swimmer is worried. More people are killed every year by
pigs than by sharks, which shows you how good we are at evaluating risk.
 -- Bruce Schneier, May 2005

Re: [Rest2web-develop] Down with writein

[Michael F. <mi...@pc...>]

Nicola Larosa wrote:
>>I've decided to scrap the writein method for ConfigObj - and instead
>>preserve comments above keywords.
>>
>>I intend to preserve inline comments as well - so each Section will have
>>an ``inline_comments`` and a ``comments`` dictionary attached. Each
>>member of ``comments`` will be a *list* of comment lines. I can do this
>>with minor changes to the current implementation.
>>
>>The file format will look like :
>>
>># initial_comment
>># preserved
>>
>># lost
>>
>># preserved with section marker
>>[section] # also preserved
>># lost
>>
>># preserved with key
>>Key = value # also preserved
>># lost
>>
>># final_comment
>># preserved
>>
>>
>>What do you think ?
> 
> 
> I think that's great. I didn't quite like the idea of interspersing changes
> in an already existing file. This way, the user does not have to worry
> which write method has to be used.
> 

The main use case was to allow the creation of a ConfigObj that
represented part of a file. It meant making an arbitrary decision about
what to do if the ConfigObj was changed (the keys in the object no
longer match the keys in the file). On the basis of YAGNI I figured I
could lose it...

> The loss of isolated comment lines is unfortunate, and seems a rather
> arbitrary thing to do, from the user point of view. But these drawbacks are
> surely minor, and the gained simplicity is worth them. Maybe they could be
> alleviated by emitting some kind of warning during file parsing.
> 

The implementation is easy because we already keep track of comment
lines for final_comment and initial_comment.

The current version keeps all empty lines and comment lines. I was
intending it to only keep continuous comment lines. (Hard to explain).

Current implementation for initial_comment (needs extending) :

# kept

# kept as well
key = value

My suggested implementation :

# dropped

# kept
key = value


Would you prefer it if I just kept all comments, even if separated by
blank lines ?
I thought this could end up associating comments with the wrong value.
As a side effect it means that there is no way of providing a comment
for the *first* key/section. It will all be swallowed up by initial_comment.

Which way should I jump ?

All the best,

Fuzzy
http://www.voidspace.org.uk/python

Re: [Rest2web-develop] Latest Commit to pythonutils

[Michael F. <mi...@pc...>]

Hello Nicola,

Nicola Larosa wrote:
...
> Great.
> 
> A minor gripe: is it really necessary to have six megs of JPEG files inot
> the repository, just for testing? :-)
> 

You mean the gallery files :-)
The test site for the gallery *is* (or at least will be) part of the
distribution. *But* there's no need for it to be so bulky, I'll change
it to just a few images.

Regards,

Fuzzy
http://www.voidspace.org.uk/python

Re: [Rest2web-develop] Latest Commit to pythonutils

[Nicola L. <ni...@te...>]

> I've just done a new commit to the pythonutils branch. It adds all the
> pythonutils files into the branch, and also adds urlpath.py and odict.py.

Maybe it would be better if pythonutils had their own branch, instead of
being tucked away inside configobj4 .


> There are also the following minor changes to configobj.py :
> 
>     Fixed bug in Section.pop (now doesn't raise KeyError if a default value
>     is specified)
> 
>     Replaced ``basestring`` with ``types.StringTypes``
> 
>     Removed the ``writein`` method
>    
>     Added __version__

Great.

A minor gripe: is it really necessary to have six megs of JPEG files inot
the repository, just for testing? :-)

-- 
Nicola Larosa - ni...@te...

If there are a few shark attacks in Florida - and a graphic movie -
suddenly every swimmer is worried. More people are killed every year by
pigs than by sharks, which shows you how good we are at evaluating risk.
 -- Bruce Schneier, May 2005

[Rest2web-develop] Latest Commit to pythonutils

[Michael F. <mi...@pc...>]

Hello Nicola,

I've just done a new commit to the pythonutils branch. It adds all the
pythonutils files into the branch, and also adds urlpath.py and odict.py.

There are also the following minor changes to configobj.py :

    Fixed bug in Section.pop (now doesn't raise KeyError if a default value
    is specified)

    Replaced ``basestring`` with ``types.StringTypes``

    Removed the ``writein`` method
   
    Added __version__

Regards,

Fuzzy