configobj-develop — Support and development of ConfigObj

Re: [Configobj-develop] Validate - List Default Values

[Nicola L. <ni...@te...>]

> The alternative would be to indicate list values by surrounding with
> square brackets :
> 
>     checkname(default="[ 'val 1', 'val 2', 'val 3']")

What about using a pseudo-constructor?

     checkname(default=list('val 1', 'val 2', 'val 3'))

-- 
Nicola Larosa - ni...@te...

The only thing the meek inherit is their ass being handed to them.
 -- Elaine "hfb" Ashton, October 2005

[Configobj-develop] Re: Validate - List Default Values

[j v. <jgv...@co...>]

I think indicating list values by surrounding with square brackets :

    checkname(default="[ 'val 1', 'val 2', 'val 3']")

is preferable.

-- jv


On Tue, 2005-12-13 at 10:14 +0000, Fuzzyman wrote:
> Hello all,
> 
> There is a bug in the current validate code.
> 
> It doesn't correctly handle lists as default values - how did that get
> past the tests ?
> 
> In fact commas in default values *seem* to break it completely. No worry
> - I can fix this, so far only with a *truly horrible* set of regular
> expressions, but I'm sure they can be optimised.
> 
> The question is how do we want to specify list values as defaults in the
> configspec ?
> 
> The current rationale (which doesn't work) uses the same syntax as
> ConfigObj. This might sound logical - but it has an unfortunate side effect.
> 
> ConfigObj syntax is to treat any value with commas in as a list - 
> unless it is quoted. Because configspec values are part of an argument
> list they need to be quoted anyway. this means the current syntax for a
> normal value with commas is
> 
>     checkname(default="'a value, with commas'")
> 
> Notice the *double* quotes ! Truly horrible IMO.
> 
> The alternative would be to indicate list values by surrounding with
> square brackets :
> 
>     checkname(default="[ 'val 1', 'val 2', 'val 3']")
> 
> The advantage is that it means you don't need to double quote default
> values with commas. The disadvantage is that we now have a different
> syntax for lists in the configspec and in the ConfigObj.
> 
> I don't think this is such a problem.
> 
> I wouldn't (initially) add any escape values - this would mean you
> couldn't use quotes or square brackets inside list members.
> 
> The alternative is to build a 'full scale' parser for the configspec -
> which I am reluctant to do. An alternative way of specifying defaults is
> to use merge (the recursive update which will be in the new release of
> ConfigObj).
> 
> Feedback appreciated - this changes the syntax of the configspec, so
> it's a (relatively) important decision.
> 
> Thanks
> 
> Fuzzyman
> http://www.voidspace.org.uk/python/index.shtml

[Configobj-develop] Validate - List Default Values

[Fuzzyman <fuz...@vo...>]

Hello all,

There is a bug in the current validate code.

It doesn't correctly handle lists as default values - how did that get
past the tests ?

In fact commas in default values *seem* to break it completely. No worry
- I can fix this, so far only with a *truly horrible* set of regular
expressions, but I'm sure they can be optimised.

The question is how do we want to specify list values as defaults in the
configspec ?

The current rationale (which doesn't work) uses the same syntax as
ConfigObj. This might sound logical - but it has an unfortunate side effect.

ConfigObj syntax is to treat any value with commas in as a list - 
unless it is quoted. Because configspec values are part of an argument
list they need to be quoted anyway. this means the current syntax for a
normal value with commas is

    checkname(default="'a value, with commas'")

Notice the *double* quotes ! Truly horrible IMO.

The alternative would be to indicate list values by surrounding with
square brackets :

    checkname(default="[ 'val 1', 'val 2', 'val 3']")

The advantage is that it means you don't need to double quote default
values with commas. The disadvantage is that we now have a different
syntax for lists in the configspec and in the ConfigObj.

I don't think this is such a problem.

I wouldn't (initially) add any escape values - this would mean you
couldn't use quotes or square brackets inside list members.

The alternative is to build a 'full scale' parser for the configspec -
which I am reluctant to do. An alternative way of specifying defaults is
to use merge (the recursive update which will be in the new release of
ConfigObj).

Feedback appreciated - this changes the syntax of the configspec, so
it's a (relatively) important decision.

Thanks

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

[Configobj-develop] Re: New Config File Format

[Fuzzyman <fuz...@vo...>]

Holt, Patrick wrote:
> Michael,
> 
> Finally found your email address. 

I prefer to use the fuz...@vo... address. The other one has
 become unreliable.

The configobj-develop mailing list is a good way to get hold of me as well.

> We've integrated configObj into our program and it works great.
> 

Fantastic.

Is it public ? Anything I can link to (or quote) in the ConfigObj use
cases (even if the project isn't public) ?

I'm not offended if the answer is no. ;-)

> One question; Are there any escape characters I can use to 
> get configOby to ignore certain characters i.e. the # sign.
> I use this in a comment I store as part of a multi line header 
> but configObj complains when it picks up the second line, does not like
> it.  
> I've tried quotes around it but treats it as one long string instead of
> a list of strings.  Thoughts?
> 
> Regards,
> 
> Patrick
> 
> Ie.
> 
>     headerTemplate =
> ["#*************************************************",
> 		       "# ",
> 		       "#    SVE Automatic configuration file",
> 		       "# ",
> 		       "# System   : %s",
> 		       "# Version  : %s",
> 		       "# ",
> 		       "# Input file name : %s",
> 		       "# ",
>                      "# Generation Date  : %s",
>                       "# ",
>                       "# Notes: ",
>                       "# ",
>                       "# ",
>  
> "#*************************************************",
>                       " " ]
> 

I don't really understand I'm afraid.

ConfigObj *already* treats lines that begin with '#' as a comment.

Using your example :

>>> headerTemplate = ["#*************************************************",
... 		       "# ",
... 		       "#    SVE Automatic configuration file",
... 		       "# ",
... 		       "# System   : %s",
... 		       "# Version  : %s",
... 		       "# ",
... 		       "# Input file name : %s",
... 		       "# ",
...                      "# Generation Date  : %s",
...                       "# ",
...                       "# Notes: ",
...                       "# ",
...                       "# ",
...
... "#*************************************************",
...                       " " ]
>>> a = ConfigObj(headerTemplate)
>>> a
{}
>>> a.initial_comment
['#*************************************************', '# ', '#    SVE
Automatic configuration file', '# ', '# System   : %s', '# Version  :
%s', '# ', '# Input file name : %s', '# ', '# Generation Date  : %s', '#
', '# Notes: ', '# ', '# ',
'#*************************************************', ' ']
>>>

ConfigObj parses the lines above as all being comment, and preserves it
as the ``initial_comment`` attribute.

Note that this example caused me to uncover a (very minor) bug - in
versions of ConfigObj before 4.0.3 (which is about to be released). A
config file which is *all* comments (like the one above) would have the
comment as ``final_comment`` rather than ``initial_comment``. :-)

Does this answer your question, Patrick ? It's possible I still don't
understand your need.

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

> 
> -----Original Message-----
> From: Michael Foord [mailto:mi...@pc...] 
> Sent: Monday, August 01, 2005 4:20 AM
> To: Holt, Patrick
> Subject: New Config File Format
> 
> Hello Patrick,
> 
> I hope you had a good weekend.
> 
> Nicola Larosa has made the next set of  changes to ConfigObj - so the
> nested section format has now changes. The tests and the docs have been
> updated to reflect this.
> 
> You can pull the updated module and docs out of SVN at :
> http://svn.rest2web.python-hosting.com/branches/configobj4/
> 
> If you have any further questions, feel free to ask.
> 
> Best Regards,
> 
> Michael Foord
> http://www.voidspace.org.uk/python
> 
> 
> 

Re: [Configobj-develop] Merging configs (setting default values)

[Fuzzyman <fuz...@vo...>]

# configobj.py
# A config file reader/writer that supports nested sections in config files.
# Copyright (C) 2005 Michael Foord, Nicola Larosa
# E-mail: fuzzyman AT voidspace DOT org DOT uk
#         nico AT tekNico DOT net

# ConfigObj 4

# Released subject to the BSD License
# Please see http://www.voidspace.org.uk/python/license.shtml

# Scripts maintained at http://www.voidspace.org.uk/python/index.shtml
# For information about bugfixes, updates and support, please join the
# ConfigObj mailing list:
# http://lists.sourceforge.net/lists/listinfo/configobj-develop
# Comments, suggestions and bug reports welcome.

"""
    >>> z = ConfigObj()
    >>> z['a'] = 'a'
    >>> z['sect'] = {
    ...    'subsect': {
    ...         'a': 'fish',
    ...         'b': 'wobble',
    ...     },
    ...     'member': 'value',
    ... }
    >>> x = ConfigObj(z.write())
    >>> z == x
    1
"""

import sys
INTP_VER = sys.version_info[:2]
if INTP_VER < (2, 2):
    raise RuntimeError("Python v.2.2 or later needed")

import os, re
from types import StringTypes

# the UTF8 BOM - from codecs module
BOM_UTF8 = '\xef\xbb\xbf'

__version__ = '4.0.3'

__revision__ = '$Id: configobj.py 147 2005-11-08 12:08:49Z fuzzyman $'

__docformat__ = "restructuredtext en"

__all__ = (
    '__version__',
    'BOM_UTF8',
    'DEFAULT_INDENT_TYPE',
    'NUM_INDENT_SPACES',
    'MAX_INTERPOL_DEPTH',
    'ConfigObjError',
    'NestingError',
    'ParseError',
    'DuplicateError',
    'ConfigspecError',
    'ConfigObj',
    'SimpleVal',
    'InterpolationError',
    'InterpolationDepthError',
    'MissingInterpolationOption',
    'RepeatSectionError',
    '__docformat__',
    'flatten_errors',
)

DEFAULT_INDENT_TYPE = ' '
NUM_INDENT_SPACES = 4
MAX_INTERPOL_DEPTH = 10

OPTION_DEFAULTS = {
    'interpolation': True,
    'raise_errors': False,
    'list_values': True,
    'create_empty': False,
    'file_error': False,
    'configspec': None,
    'stringify': True,
    # option may be set to one of ('', ' ', '\t')
    'indent_type': None,
}

class ConfigObjError(SyntaxError):
    """
    This is the base class for all errors that ConfigObj raises.
    It is a subclass of SyntaxError.
    
    >>> raise ConfigObjError
    Traceback (most recent call last):
    ConfigObjError
    """
    def __init__(self, message='', line_number=None, line=''):
        self.line = line
        self.line_number = line_number
        self.message = message
        SyntaxError.__init__(self, message)

class NestingError(ConfigObjError):
    """
    This error indicates a level of nesting that doesn't match.
    
    >>> raise NestingError
    Traceback (most recent call last):
    NestingError
    """

class ParseError(ConfigObjError):
    """
    This error indicates that a line is badly written.
    It is neither a valid ``key = value`` line,
    nor a valid section marker line.
    
    >>> raise ParseError
    Traceback (most recent call last):
    ParseError
    """

class DuplicateError(ConfigObjError):
    """
    The keyword or section specified already exists.
    
    >>> raise DuplicateError
    Traceback (most recent call last):
    DuplicateError
    """

class ConfigspecError(ConfigObjError):
    """
    An error occured whilst parsing a configspec.
    
    >>> raise ConfigspecError
    Traceback (most recent call last):
    ConfigspecError
    """

class InterpolationError(ConfigObjError):
    """Base class for the two interpolation errors."""

class InterpolationDepthError(InterpolationError):
    """Maximum interpolation depth exceeded in string interpolation."""

    def __init__(self, option):
        """
        >>> raise InterpolationDepthError('yoda')
        Traceback (most recent call last):
        InterpolationDepthError: max interpolation depth exceeded in value "yoda".
        """
        InterpolationError.__init__(
            self,
            'max interpolation depth exceeded in value "%s".' % option)

class RepeatSectionError(ConfigObjError):
    """
    This error indicates additional sections in a section with a
    ``__many__`` (repeated) section.
    
    >>> raise RepeatSectionError
    Traceback (most recent call last):
    RepeatSectionError
    """

class MissingInterpolationOption(InterpolationError):
    """A value specified for interpolation was missing."""

    def __init__(self, option):
        """
        >>> raise MissingInterpolationOption('yoda')
        Traceback (most recent call last):
        MissingInterpolationOption: missing option "yoda" in interpolation.
        """
        InterpolationError.__init__(
            self,
            'missing option "%s" in interpolation.' % option)

class Section(dict):
    """
    A dictionary-like object that represents a section in a config file.
    
    It does string interpolation if the 'interpolate' attribute
    of the 'main' object is set to True.
    
    Interpolation is tried first from the 'DEFAULT' section of this object,
    next from the 'DEFAULT' section of the parent, lastly the main object.
    
    A Section will behave like an ordered dictionary - following the
    order of the ``scalars`` and ``sections`` attributes.
    You can use this to change the order of members.
    
    Iteration follows the order: scalars, then sections.
    """

    _KEYCRE = re.compile(r"%\(([^)]*)\)s|.")

    def __init__(self, parent, depth, main, indict=None, name=None):
        """
        * parent is the section above
        * depth is the depth level of this section
        * main is the main ConfigObj
        * indict is a dictionary to initialise the section with
        """
        if indict is None:
            indict = {}
        dict.__init__(self)
        # used for nesting level *and* interpolation
        self.parent = parent
        # used for the interpolation attribute
        self.main = main
        # level of nesting depth of this Section
        self.depth = depth
        # the sequence of scalar values in this Section
        self.scalars = []
        # the sequence of sections in this Section
        self.sections = []
        # purely for information
        self.name = name
        # for comments :-)
        self.comments = {}
        self.inline_comments = {}
        # for the configspec
        self.configspec = {}
        # for defaults
        self.defaults = []
        #
        # we do this explicitly so that __setitem__ is used properly
        # (rather than just passing to ``dict.__init__``)
        for entry in indict:
            self[entry] = indict[entry]

    def _interpolate(self, value):
        """Nicked from ConfigParser."""
        depth = MAX_INTERPOL_DEPTH
        # loop through this until it's done
        while depth:
            depth -= 1
            if value.find("%(") != -1:
                value = self._KEYCRE.sub(self._interpolation_replace, value)
            else:
                break
        else:
            raise InterpolationDepthError(value)
        return value

    def _interpolation_replace(self, match):
        """ """
        s = match.group(1)
        if s is None:
            return match.group()
        else:
            # switch off interpolation before we try and fetch anything !
            self.main.interpolation = False
            # try the 'DEFAULT' member of *this section* first
            val = self.get('DEFAULT', {}).get(s)
            # try the 'DEFAULT' member of the *parent section* next
            if val is None:
                val = self.parent.get('DEFAULT', {}).get(s)
            # last, try the 'DEFAULT' member of the *main section*
            if val is None:
                val = self.main.get('DEFAULT', {}).get(s)
            self.main.interpolation = True
            if val is None:
                raise MissingInterpolationOption(s)
            return val

    def __getitem__(self, key):
        """Fetch the item and do string interpolation."""
        val = dict.__getitem__(self, key)
        if self.main.interpolation and isinstance(val, StringTypes):
            return self._interpolate(val)
        return val

    def __setitem__(self, key, value):
        """
        Correctly set a value.
        
        Making dictionary values Section instances.
        (We have to special case 'Section' instances - which are also dicts)
        
        Keys must be strings.
        Values need only be strings (or lists of strings) if
        ``main.stringify`` is set.
        """
        if not isinstance(key, StringTypes):
            raise ValueError, 'The key "%s" is not a string.' % key
        # add the comment
        if not self.comments.has_key(key):
            self.comments[key] = []
            self.inline_comments[key] = ''
        # remove the entry from defaults
        if key in self.defaults:
            self.defaults.remove(key)
        #
        if isinstance(value, Section):
            if not self.has_key(key):
                self.sections.append(key)
            dict.__setitem__(self, key, value)
        elif isinstance(value, dict):
            # First create the new depth level,
            # then create the section
            if not self.has_key(key):
                self.sections.append(key)
            new_depth = self.depth + 1
            dict.__setitem__(
                self,
                key,
                Section(
                    self,
                    new_depth,
                    self.main,
                    indict=value,
                    name=key))
        else:
            if not self.has_key(key):
                self.scalars.append(key)
            if not self.main.stringify:
                if isinstance(value, StringTypes):
                    pass
                elif isinstance(value, (list, tuple)):
                    for entry in value:
                        if not isinstance(entry, StringTypes):
                            raise TypeError, (
                                'Value is not a string "%s".' % entry)
                else:
                    raise TypeError, 'Value is not a string "%s".' % value
            dict.__setitem__(self, key, value)

    def __delitem__(self, key):
        """Remove items from the sequence when deleting."""
        dict. __delitem__(self, key)
        if key in self.scalars:
            self.scalars.remove(key)
        else:
            self.sections.remove(key)
        del self.comments[key]
        del self.inline_comments[key]

    def get(self, key, default=None):
        """A version of ``get`` that doesn't bypass string interpolation."""
        try:
            return self[key]
        except KeyError:
            return default

    def update(self, indict):
        """
        A version of update that uses our ``__setitem__``.
        """
        for entry in indict:
            self[entry] = indict[entry]


    def pop(self, key, *args):
        """ """
        val = dict.pop(self, key, *args)
        if key in self.scalars:
            del self.comments[key]
            del self.inline_comments[key]
            self.scalars.remove(key)
        elif key in self.sections:
            del self.comments[key]
            del self.inline_comments[key]
            self.sections.remove(key)
        if self.main.interpolation and isinstance(val, StringTypes):
            return self._interpolate(val)
        return val

    def popitem(self):
        """Pops the first (key,val)"""
        sequence = (self.scalars + self.sections)
        if not sequence:
            raise KeyError, ": 'popitem(): dictionary is empty'"
        key = sequence[0]
        val =  self[key]
        del self[key]
        return key, val

    def clear(self):
        """
        A version of clear that also affects scalars/sections
        Also clears comments and configspec.
        
        Leaves other attributes alone :
            depth/main/parent are not affected
        """
        dict.clear(self)
        self.scalars = []
        self.sections = []
        self.comments = {}
        self.inline_comments = {}
        self.configspec = {}

    def setdefault(self, key, default=None):
        """A version of setdefault that sets sequence if appropriate."""
        try:
            return self[key]
        except KeyError:
            self[key] = default
            return self[key]

    def items(self):
        """ """
        return zip((self.scalars + self.sections), self.values())

    def keys(self):
        """ """
        return (self.scalars + self.sections)

    def values(self):
        """ """
        return [self[key] for key in (self.scalars + self.sections)]

    def iteritems(self):
        """ """
        return iter(self.items())

    def iterkeys(self):
        """ """
        return iter((self.scalars + self.sections))

    __iter__ = iterkeys

    def itervalues(self):
        """ """
        return iter(self.values())

    def __repr__(self):
        return '{%s}' % ', '.join([('%s: %s' % (repr(key), repr(self[key])))
            for key in (self.scalars + self.sections)])

    __str__ = __repr__

    # Extra methods - not in a normal dictionary

    def dict(self):
        """
        Return a deepcopy of self as a dictionary.
        
        All members that are ``Section`` instances are recursively turned to
        ordinary dictionaries - by calling their ``dict`` method.
        
        >>> n = a.dict()
        >>> n == a
        1
        >>> n is a
        0
        """
        newdict = {}
        for entry in self:
            this_entry = self[entry]
            if isinstance(this_entry, Section):
                this_entry = this_entry.dict()
            elif isinstance(this_entry, (list, tuple)):
                # create a copy rather than a reference
                this_entry = list(this_entry)
            newdict[entry] = this_entry
        return newdict

    def merge(self, indict):
        """
        A recursive update - useful for merging config files.
        
        >>> a = '''[section1]
        ...     option1 = True
        ...     [[subsection]]
        ...     more_options = False
        ...     # end of file'''.splitlines()
        >>> b = '''# File is user.ini
        ...     [section1]
        ...     option1 = False
        ...     # end of file'''.splitlines()
        >>> c1 = ConfigObj(b)
        >>> c2 = ConfigObj(a)
        >>> c2.merge(c1)
        >>> c2
        {'section1': {'option1': 'False', 'subsection': {'more_options': 'False'}}}
        """
        for key, val in indict.items():
            if (key in self and isinstance(self[key], dict) and
                                isinstance(val, dict)):
                self[key].merge(val)
            else:   
                self[key] = val

    def rename(self, oldkey, newkey):
        """
        Change a keyname to another, without changing position in sequence.
        
        Implemented so that transformations can be made on keys,
        as well as on values. (used by encode and decode)
        
        Also renames comments.
        """
        if oldkey in self.scalars:
            the_list = self.scalars
        elif oldkey in self.sections:
            the_list = self.sections
        else:
            raise KeyError, 'Key "%s" not found.' % oldkey
        pos = the_list.index(oldkey)
        #
        val = self[oldkey]
        dict.__delitem__(self, oldkey)
        dict.__setitem__(self, newkey, val)
        the_list.remove(oldkey)
        the_list.insert(pos, newkey)
        comm = self.comments[oldkey]
        inline_comment = self.inline_comments[oldkey]
        del self.comments[oldkey]
        del self.inline_comments[oldkey]
        self.comments[newkey] = comm
        self.inline_comments[newkey] = inline_comment

    def walk(self, function, raise_errors=True,
            call_on_sections=False, **keywargs):
        """
        Walk every member and call a function on the keyword and value.
        
        Return a dictionary of the return values
        
        If the function raises an exception, raise the errror
        unless ``raise_errors=False``, in which case set the return value to
        ``False``.
        
        Any unrecognised keyword arguments you pass to walk, will be pased on
        to the function you pass in.
        
        Note: if ``call_on_sections`` is ``True`` then - on encountering a
        subsection, *first* the function is called for the *whole* subsection,
        and then recurses into it's members. This means your function must be
        able to handle strings, dictionaries and lists. This allows you
        to change the key of subsections as well as for ordinary members. The
        return value when called on the whole subsection has to be discarded.
        
        See  the encode and decode methods for examples, including functions.
        
        .. caution::
        
            You can use ``walk`` to transform the names of members of a section
            but you mustn't add or delete members.
        
        >>> config = '''[XXXXsection]
        ... XXXXkey = XXXXvalue'''.splitlines()
        >>> cfg = ConfigObj(config)
        >>> cfg
        {'XXXXsection': {'XXXXkey': 'XXXXvalue'}}
        >>> def transform(section, key):
        ...     val = section[key]
        ...     newkey = key.replace('XXXX', 'CLIENT1')
        ...     section.rename(key, newkey)
        ...     if isinstance(val, (tuple, list, dict)):
        ...         pass
        ...     else:
        ...         val = val.replace('XXXX', 'CLIENT1')
        ...         section[newkey] = val
        >>> cfg.walk(transform, call_on_sections=True)
        {'CLIENT1section': {'CLIENT1key': None}}
        >>> cfg
        {'CLIENT1section': {'CLIENT1key': 'CLIENT1value'}}
        """
        out = {}
        # scalars first
        for i in range(len(self.scalars)):
            entry = self.scalars[i]
            try:
                val = function(self, entry, **keywargs)
                # bound again in case name has changed
                entry = self.scalars[i]
                out[entry] = val
            except Exception:
                if raise_errors:
                    raise
                else:
                    entry = self.scalars[i]
                    out[entry] = False
        # then sections
        for i in range(len(self.sections)):
            entry = self.sections[i]
            if call_on_sections:
                try:
                    function(self, entry, **keywargs)
                except Exception:
                    if raise_errors:
                        raise
                    else:
                        entry = self.sections[i]
                        out[entry] = False
                # bound again in case name has changed
                entry = self.sections[i]
            # previous result is discarded
            out[entry] = self[entry].walk(
                function,
                raise_errors=raise_errors,
                call_on_sections=call_on_sections,
                **keywargs)
        return out

    def decode(self, encoding):
        """
        Decode all strings and values to unicode, using the specified encoding.
        
        Works with subsections and list values.
        
        Uses the ``walk`` method.
        
        Testing ``encode`` and ``decode``.
        >>> m = ConfigObj(a)
        >>> m.decode('ascii')
        >>> def testuni(val):
        ...     for entry in val:
        ...         if not isinstance(entry, unicode):
        ...             print >> sys.stderr, type(entry)
        ...             raise AssertionError, 'decode failed.'
        ...         if isinstance(val[entry], dict):
        ...             testuni(val[entry])
        ...         elif not isinstance(val[entry], unicode):
        ...             raise AssertionError, 'decode failed.'
        >>> testuni(m)
        >>> m.encode('ascii')
        >>> a == m
        1
        """
        def decode(section, key, encoding=encoding):
            """ """
            val = section[key]
            if isinstance(val, (list, tuple)):
                newval = []
                for entry in val:
                    newval.append(entry.decode(encoding))
            elif isinstance(val, dict):
                newval = val
            else:
                newval = val.decode(encoding)
            newkey = key.decode(encoding)
            section.rename(key, newkey)
            section[newkey] = newval
        # using ``call_on_sections`` allows us to modify section names
        self.walk(decode, call_on_sections=True)

    def encode(self, encoding):
        """
        Encode all strings and values from unicode,
        using the specified encoding.
        
        Works with subsections and list values.
        Uses the ``walk`` method.
        """
        def encode(section, key, encoding=encoding):
            """ """
            val = section[key]
            if isinstance(val, (list, tuple)):
                newval = []
                for entry in val:
                    newval.append(entry.encode(encoding))
            elif isinstance(val, dict):
                newval = val
            else:
                newval = val.encode(encoding)
            newkey = key.encode(encoding)
            section.rename(key, newkey)
            section[newkey] = newval
        self.walk(encode, call_on_sections=True)

    def istrue(self, key):
        """
        Accepts a key as input. The corresponding value must be a string or
        the objects (``True`` or 1) or (``False`` or 0). We allow 0 and 1 to
        retain compatibility with Python 2.2.
        
        If the string is one of  ``True``, ``On``, ``Yes``, or ``1`` it returns 
        ``True``.
        
        If the string is one of  ``False``, ``Off``, ``No``, or ``0`` it returns 
        ``False``.
        
        ``istrue`` is not case sensitive.
        
        Any other input will raise a ``ValueError``. 
        """
        val = self[key]
        if val == True:
            return True
        elif val == False:
            return False
        else:
            try:
                if not isinstance(val, StringTypes):
                    raise KeyError
                else:
                    return self.main._bools[val.lower()]
            except KeyError:
                raise ValueError('Value "%s" is neither True nor False' % val)

class ConfigObj(Section):
    """
    An object to read, create, and write config files.
    
    Testing with duplicate keys and sections.
    
    >>> c = '''
    ... [hello]
    ... member = value
    ... [hello again]
    ... member = value
    ... [ "hello" ]
    ... member = value
    ... '''
    >>> ConfigObj(c.split('\\n'), raise_errors = True)
    Traceback (most recent call last):
    DuplicateError: Duplicate section name at line 5.
    
    >>> d = '''
    ... [hello]
    ... member = value
    ... [hello again]
    ... member1 = value
    ... member2 = value
    ... 'member1' = value
    ... [ "and again" ]
    ... member = value
    ... '''
    >>> ConfigObj(d.split('\\n'), raise_errors = True)
    Traceback (most recent call last):
    DuplicateError: Duplicate keyword name at line 6.
    """

    _keyword = re.compile(r'''^ # line start
        (\s*)                   # indentation
        (                       # keyword
            (?:".*?")|          # double quotes
            (?:'.*?')|          # single quotes
            (?:[^'"=].*?)       # no quotes
        )
        \s*=\s*                 # divider
        (.*)                    # value (including list values and comments)
        $   # line end
        ''',
        re.VERBOSE)

    _sectionmarker = re.compile(r'''^
        (\s*)                     # 1: indentation
        ((?:\[\s*)+)              # 2: section marker open
        (                         # 3: section name open
            (?:"\s*\S.*?\s*")|    # at least one non-space with double quotes
            (?:'\s*\S.*?\s*')|    # at least one non-space with single quotes
            (?:[^'"\s].*?)        # at least one non-space unquoted
        )                         # section name close
        ((?:\s*\])+)              # 4: section marker close
        \s*(\#.*)?                # 5: optional comment
        $''',
        re.VERBOSE)

    # this regexp pulls list values out as a single string
    # or single values and comments
    _valueexp = re.compile(r'''^
        (?:
            (?:
                (
                    (?:
                        (?:
                            (?:".*?")|              # double quotes
                            (?:'.*?')|              # single quotes
                            (?:[^'",\#][^,\#]*?)       # unquoted
                        )
                        \s*,\s*                     # comma
                    )*      # match all list items ending in a comma (if any)
                )
                (
                    (?:".*?")|                      # double quotes
                    (?:'.*?')|                      # single quotes
                    (?:[^'",\#\s][^,]*?)             # unquoted
                )?          # last item in a list - or string value
            )|
            (,)             # alternatively a single comma - empty list
        )
        \s*(\#.*)?          # optional comment
        $''',
        re.VERBOSE)

    # use findall to get the members of a list value
    _listvalueexp = re.compile(r'''
        (
            (?:".*?")|          # double quotes
            (?:'.*?')|          # single quotes
            (?:[^'",\#].*?)       # unquoted
        )
        \s*,\s*                 # comma
        ''',
        re.VERBOSE)

    # this regexp is used for the value
    # when lists are switched off
    _nolistvalue = re.compile(r'''^
        (
            (?:".*?")|          # double quotes
            (?:'.*?')|          # single quotes
            (?:[^'"\#].*?)      # unquoted
        )
        \s*(\#.*)?              # optional comment
        $''',
        re.VERBOSE)

    # regexes for finding triple quoted values on one line
    _single_line_single = re.compile(r"^'''(.*?)'''\s*(#.*)?$")
    _single_line_double = re.compile(r'^"""(.*?)"""\s*(#.*)?$')
    _multi_line_single = re.compile(r"^(.*?)'''\s*(#.*)?$")
    _multi_line_double = re.compile(r'^(.*?)"""\s*(#.*)?$')

    _triple_quote = {
        "'''": (_single_line_single, _multi_line_single),
        '"""': (_single_line_double, _multi_line_double),
    }

    # Used by the ``istrue`` Section method
    _bools = {
        'yes': True, 'no': False,
        'on': True, 'off': False,
        '1': True, '0': False,
        'true': True, 'false': False,
        }

    def __init__(self, infile=None, options=None, **kwargs):
        """
        Parse or create a config file object.
        
        ``ConfigObj(infile=None, options=None, **kwargs)``
        """
        if infile is None:
            infile = []
        if options is None:
            options = {}
        # keyword arguments take precedence over an options dictionary
        options.update(kwargs)
        # init the superclass
        Section.__init__(self, self, 0, self)
        #
        defaults = OPTION_DEFAULTS.copy()
        for entry in options.keys():
            if entry not in defaults.keys():
                raise TypeError, 'Unrecognised option "%s".' % entry
        # TODO: check the values too
        # add the explicit options to the defaults
        defaults.update(options)
        #
        # initialise a few variables
        self._errors = []
        self.raise_errors = defaults['raise_errors']
        self.interpolation = defaults['interpolation']
        self.list_values = defaults['list_values']
        self.create_empty = defaults['create_empty']
        self.file_error = defaults['file_error']
        self.stringify = defaults['stringify']
        self.indent_type = defaults['indent_type']
        # used by the write method
        self.BOM = None
        #
        self.initial_comment = []
        self.final_comment = []
        #
        if isinstance(infile, StringTypes):
            self.filename = os.path.abspath(infile)
            if os.path.isfile(self.filename):
                infile = open(self.filename).readlines()
            elif self.file_error:
                # raise an error if the file doesn't exist
                raise IOError, 'Config file not found: "%s".' % self.filename
            else:
                # file doesn't already exist
                if self.create_empty:
                    # this is a good test that the filename specified
                    # isn't impossible - like on a non existent device
                    h = open(self.filename, 'w')
                    h.write('')
                    h.close()
                infile = []
        elif isinstance(infile, (list, tuple)):
            self.filename = None
        elif isinstance(infile, dict):
            # initialise self
            # the Section class handles creating subsections
            if isinstance(infile, ConfigObj):
                # get a copy of our ConfigObj
                infile = infile.dict()
            for entry in infile:
                self[entry] = infile[entry]
            self.filename = None
            del self._errors
            if defaults['configspec'] is not None:
                self._handle_configspec(defaults['configspec'])
            else:
                self.configspec = None
            return
        elif hasattr(infile, 'seek'):
            # this supports StringIO instances and even file objects
            self.filename = infile
            infile.seek(0)
            infile = infile.readlines()
            self.filename.seek(0)
        else:
            raise TypeError, ('infile must be a filename,'
                ' StringIO instance, or a file as a list.')
        #
        # strip trailing '\n' from lines
        infile = [line.rstrip('\n') for line in infile]
        #
        # remove the UTF8 BOM if it is there
        # FIXME: support other BOM
        if infile and infile[0].startswith(BOM_UTF8):
            infile[0] = infile[0][3:]
            self.BOM = BOM_UTF8
        else:
            self.BOM = None
        #
        self._parse(infile)
        # if we had any errors, now is the time to raise them
        if self._errors:
            error = ConfigObjError("Parsing failed.")
            # set the errors attribute; it's a list of tuples:
            # (error_type, message, line_number)
            error.errors = self._errors
            # set the config attribute
            error.config = self
            raise error
        # delete private attributes
        del self._errors
        #
        if defaults['configspec'] is None:
            self.configspec = None
        else:
            self._handle_configspec(defaults['configspec'])

    def _parse(self, infile):
        """
        Actually parse the config file
        
        Testing Interpolation
        
        >>> c = ConfigObj()
        >>> c['DEFAULT'] = {
        ...     'b': 'goodbye',
        ...     'userdir': 'c:\\\\home',
        ...     'c': '%(d)s',
        ...     'd': '%(c)s'
        ... }
        >>> c['section'] = {
        ...     'a': '%(datadir)s\\\\some path\\\\file.py',
        ...     'b': '%(userdir)s\\\\some path\\\\file.py',
        ...     'c': 'Yo %(a)s',
        ...     'd': '%(not_here)s',
        ...     'e': '%(c)s',
        ... }
        >>> c['section']['DEFAULT'] = {
        ...     'datadir': 'c:\\\\silly_test',
        ...     'a': 'hello - %(b)s',
        ... }
        >>> c['section']['a'] == 'c:\\\\silly_test\\\\some path\\\\file.py'
        1
        >>> c['section']['b'] == 'c:\\\\home\\\\some path\\\\file.py'
        1
        >>> c['section']['c'] == 'Yo hello - goodbye'
        1
        
        Switching Interpolation Off
        
        >>> c.interpolation = False
        >>> c['section']['a'] == '%(datadir)s\\\\some path\\\\file.py'
        1
        >>> c['section']['b'] == '%(userdir)s\\\\some path\\\\file.py'
        1
        >>> c['section']['c'] == 'Yo %(a)s'
        1
        
        Testing the interpolation errors.
        
        >>> c.interpolation = True
        >>> c['section']['d']
        Traceback (most recent call last):
        MissingInterpolationOption: missing option "not_here" in interpolation.
        >>> c['section']['e']
        Traceback (most recent call last):
        InterpolationDepthError: max interpolation depth exceeded in value "%(c)s".
        
        Testing our quoting.
        
        >>> i._quote('\"""\'\'\'')
        Traceback (most recent call last):
        SyntaxError: EOF while scanning triple-quoted string
        >>> try:
        ...     i._quote('\\n', multiline=False)
        ... except ConfigObjError, e:
        ...    e.msg
        'Value "\\n" cannot be safely quoted.'
        >>> k._quote(' "\' ', multiline=False)
        Traceback (most recent call last):
        SyntaxError: EOL while scanning single-quoted string
        
        Testing with "stringify" off.
        >>> c.stringify = False
        >>> c['test'] = 1
        Traceback (most recent call last):
        TypeError: Value is not a string "1".
        """
        comment_list = []
        done_start = False
        this_section = self
        maxline = len(infile) - 1
        cur_index = -1
        reset_comment = False
        while cur_index < maxline:
            if reset_comment:
                comment_list = []
            cur_index += 1
            line = infile[cur_index]
            sline = line.strip()
            # do we have anything on the line ?
            if not sline or sline.startswith('#'):
                reset_comment = False
                comment_list.append(line)
                continue
            if not done_start:
                # preserve initial comment
                self.initial_comment = comment_list
                comment_list = []
                done_start = True
            reset_comment = True
            # first we check if it's a section marker
            mat = self._sectionmarker.match(line)
##            print >> sys.stderr, sline, mat
            if mat is not None:
                # is a section line
                (indent, sect_open, sect_name, sect_close, comment) = (
                    mat.groups())
                if indent and (self.indent_type is None):
                    self.indent_type = indent[0]
                cur_depth = sect_open.count('[')
                if cur_depth != sect_close.count(']'):
                    self._handle_error(
                        "Cannot compute the section depth at line %s.",
                        NestingError, infile, cur_index)
                    continue
                if cur_depth < this_section.depth:
                    # the new section is dropping back to a previous level
                    try:
                        parent = self._match_depth(
                            this_section,
                            cur_depth).parent
                    except SyntaxError:
                        self._handle_error(
                            "Cannot compute nesting level at line %s.",
                            NestingError, infile, cur_index)
                        continue
                elif cur_depth == this_section.depth:
                    # the new section is a sibling of the current section
                    parent = this_section.parent
                elif cur_depth == this_section.depth + 1:
                    # the new section is a child the current section
                    parent = this_section
                else:
                    self._handle_error(
                        "Section too nested at line %s.",
                        NestingError, infile, cur_index)
                #
                sect_name = self._unquote(sect_name)
                if parent.has_key(sect_name):
##                    print >> sys.stderr, sect_name
                    self._handle_error(
                        'Duplicate section name at line %s.',
                        DuplicateError, infile, cur_index)
                    continue
                # create the new section
                this_section = Section(
                    parent,
                    cur_depth,
                    self,
                    name=sect_name)
                parent[sect_name] = this_section
                parent.inline_comments[sect_name] = comment
                parent.comments[sect_name] = comment_list
##                print >> sys.stderr, parent[sect_name] is this_section
                continue
            #
            # it's not a section marker,
            # so it should be a valid ``key = value`` line
            mat = self._keyword.match(line)
##            print >> sys.stderr, sline, mat
            if mat is not None:
                # is a keyword value
                # value will include any inline comment
                (indent, key, value) = mat.groups()
                if indent and (self.indent_type is None):
                    self.indent_type = indent[0]
                # check for a multiline value
                if value[:3] in ['"""', "'''"]:
                    try:
                        (value, comment, cur_index) = self._multiline(
                            value, infile, cur_index, maxline)
                    except SyntaxError:
                        self._handle_error(
                            'Parse error in value at line %s.',
                            ParseError, infile, cur_index)
                        continue
                else:
                    # extract comment and lists
                    try:
                        (value, comment) = self._handle_value(value)
                    except SyntaxError:
                        self._handle_error(
                            'Parse error in value at line %s.',
                            ParseError, infile, cur_index)
                        continue
                #
##                print >> sys.stderr, sline
                key = self._unquote(key)
                if this_section.has_key(key):
                    self._handle_error(
                        'Duplicate keyword name at line %s.',
                        DuplicateError, infile, cur_index)
                    continue
                # add the key
##                print >> sys.stderr, this_section.name
                this_section[key] = value
                this_section.inline_comments[key] = comment
                this_section.comments[key] = comment_list
##                print >> sys.stderr, key, this_section[key]
##                if this_section.name is not None:
##                    print >> sys.stderr, this_section
##                    print >> sys.stderr, this_section.parent
##                    print >> sys.stderr, this_section.parent[this_section.name]
                continue
            #
            # it neither matched as a keyword
            # or a section marker
            self._handle_error(
                'Invalid line at line "%s".',
                ParseError, infile, cur_index)
        if self.indent_type is None:
            # no indentation used, set the type accordingly
            self.indent_type = ''
        # preserve the final comment
        self.final_comment = comment_list

    def _match_depth(self, sect, depth):
        """
        Given a section and a depth level, walk back through the sections
        parents to see if the depth level matches a previous section.
        
        Return a reference to the right section,
        or raise a SyntaxError.
        """
        while depth < sect.depth:
            if sect is sect.parent:
                # we've reached the top level already
                raise SyntaxError
            sect = sect.parent
        if sect.depth == depth:
            return sect
        # shouldn't get here
        raise SyntaxError

    def _handle_error(self, text, ErrorClass, infile, cur_index):
        """
        Handle an error according to the error settings.
        
        Either raise the error or store it.
        The error will have occured at ``cur_index``
        """
        line = infile[cur_index]
        message = text % cur_index
        error = ErrorClass(message, cur_index, line)
        if self.raise_errors:
            # raise the error - parsing stops here
            raise error
        # store the error
        # reraise when parsing has finished
        self._errors.append(error)

    def _unquote(self, value):
        """Return an unquoted version of a value"""
        if (value[0] == value[-1]) and (value[0] in ('"', "'")):
            value = value[1:-1]
        return value

    def _quote(self, value, multiline=True):
        """
        Return a safely quoted version of a value.
        
        Raise a ConfigObjError if the value cannot be safely quoted.
        If multiline is ``True`` (default) then use triple quotes
        if necessary.
        
        Don't quote values that don't need it.
        Recursively quote members of a list and return a comma joined list.
        Multiline is ``False`` for lists.
        Obey list syntax for empty and single member lists.
        
        If ``list_values=False`` then the value is only quoted if it contains
        a ``\n`` (is multiline).
        """
        if isinstance(value, (list, tuple)):
            if not value:
                return ','
            elif len(value) == 1:
                return self._quote(value[0], multiline=False) + ','
            return ', '.join([self._quote(val, multiline=False)
                for val in value])
        if not isinstance(value, StringTypes):
            if self.stringify:
                value = str(value)
            else:
                raise TypeError, 'Value "%s" is not a string.' % value
        squot = "'%s'"
        dquot = '"%s"'
        noquot = "%s"
        wspace_plus = ' \r\t\n\v\t\'"'
        tsquot = '"""%s"""'
        tdquot = "'''%s'''"
        if not value:
            return '""'
        if (not self.list_values and '\n' not in value) or not (multiline and
                ((("'" in value) and ('"' in value)) or ('\n' in value))):
            if not self.list_values:
                # we don't quote if ``list_values=False``
                quot = noquot
            # for normal values either single or double quotes will do
            elif '\n' in value:
                # will only happen if multiline is off - e.g. '\n' in key
                raise ConfigObjError, ('Value "%s" cannot be safely quoted.' %
                    value)
            elif ((value[0] not in wspace_plus) and
                    (value[-1] not in wspace_plus) and
                    (',' not in value)):
                quot = noquot
            else:
                if ("'" in value) and ('"' in value):
                    raise ConfigObjError, (
                        'Value "%s" cannot be safely quoted.' % value)
                elif '"' in value:
                    quot = squot
                else:
                    quot = dquot
        else:
            # if value has '\n' or "'" *and* '"', it will need triple quotes
            if (value.find('"""') != -1) and (value.find("'''") != -1):
                raise ConfigObjError, (
                    'Value "%s" cannot be safely quoted.' % value)
            if value.find('"""') == -1:
                quot = tdquot
            else:
                quot = tsquot
        return quot % value

    def _handle_value(self, value):
        """
        Given a value string, unquote, remove comment,
        handle lists. (including empty and single member lists)
        
        Testing list values.
        
        >>> testconfig3 = '''
        ... a = ,
        ... b = test,
        ... c = test1, test2   , test3
        ... d = test1, test2, test3,
        ... '''
        >>> d = ConfigObj(testconfig3.split('\\n'), raise_errors=True)
        >>> d['a'] == []
        1
        >>> d['b'] == ['test']
        1
        >>> d['c'] == ['test1', 'test2', 'test3']
        1
        >>> d['d'] == ['test1', 'test2', 'test3']
        1
        
        Testing with list values off.
        
        >>> e = ConfigObj(
        ...     testconfig3.split('\\n'),
        ...     raise_errors=True,
        ...     list_values=False)
        >>> e['a'] == ','
        1
        >>> e['b'] == 'test,'
        1
        >>> e['c'] == 'test1, test2   , test3'
        1
        >>> e['d'] == 'test1, test2, test3,'
        1
        
        Testing creating from a dictionary.
        
        >>> f = {
        ...     'key1': 'val1',
        ...     'key2': 'val2',
        ...     'section 1': {
        ...         'key1': 'val1',
        ...         'key2': 'val2',
        ...         'section 1b': {
        ...             'key1': 'val1',
        ...             'key2': 'val2',
        ...         },
        ...     },
        ...     'section 2': {
        ...         'key1': 'val1',
        ...         'key2': 'val2',
        ...         'section 2b': {
        ...             'key1': 'val1',
        ...             'key2': 'val2',
        ...         },
        ...     },
        ...      'key3': 'val3',
        ... }
        >>> g = ConfigObj(f)
        >>> f == g
        1
        
        Testing we correctly detect badly built list values (4 of them).
        
        >>> testconfig4 = '''
        ... config = 3,4,,
        ... test = 3,,4
        ... fish = ,,
        ... dummy = ,,hello, goodbye
        ... '''
        >>> try:
        ...     ConfigObj(testconfig4.split('\\n'))
        ... except ConfigObjError, e:
        ...     len(e.errors)
        4
        
        Testing we correctly detect badly quoted values (4 of them).
        
        >>> testconfig5 = '''
        ... config = "hello   # comment
        ... test = 'goodbye
        ... fish = 'goodbye   # comment
        ... dummy = "hello again
        ... '''
        >>> try:
        ...     ConfigObj(testconfig5.split('\\n'))
        ... except ConfigObjError, e:
        ...     len(e.errors)
        4
        """
        # do we look for lists in values ?
        if not self.list_values:
            mat = self._nolistvalue.match(value)
            if mat is None:
                raise SyntaxError
            (value, comment) = mat.groups()
            # NOTE: we don't unquote here
            return (value, comment)
        mat = self._valueexp.match(value)
        if mat is None:
            # the value is badly constructed, probably badly quoted,
            # or an invalid list
            raise SyntaxError
        (list_values, single, empty_list, comment) = mat.groups()
        if (list_values == '') and (single is None):
            # change this if you want to accept empty values
            raise SyntaxError
        # NOTE: note there is no error handling from here if the regex
        # is wrong: then incorrect values will slip through
        if empty_list is not None:
            # the single comma - meaning an empty list
            return ([], comment)
        if single is not None:
            single = self._unquote(single)
        if list_values == '':
            # not a list value
            return (single, comment)
        the_list = self._listvalueexp.findall(list_values)
        the_list = [self._unquote(val) for val in the_list]
        if single is not None:
            the_list += [single]
        return (the_list, comment)

    def _multiline(self, value, infile, cur_index, maxline):
        """
        Extract the value, where we are in a multiline situation
        
        Testing multiline values.
        
        >>> i == {
        ...     'name4': ' another single line value ',
        ...     'multi section': {
        ...         'name4': '\\n        Well, this is a\\n        multiline '
        ...             'value\\n        ',
        ...         'name2': '\\n        Well, this is a\\n        multiline '
        ...             'value\\n        ',
        ...         'name3': '\\n        Well, this is a\\n        multiline '
        ...             'value\\n        ',
        ...         'name1': '\\n        Well, this is a\\n        multiline '
        ...             'value\\n        ',
        ...     },
        ...     'name2': ' another single line value ',
        ...     'name3': ' a single line value ',
        ...     'name1': ' a single line value ',
        ... }
        1
        """
        quot = value[:3]
        newvalue = value[3:]
        single_line = self._triple_quote[quot][0]
        multi_line = self._triple_quote[quot][1]
        mat = single_line.match(value)
        if mat is not None:
            retval = list(mat.groups())
            retval.append(cur_index)
            return retval
        elif newvalue.find(quot) != -1:
            # somehow the triple quote is missing
            raise SyntaxError
        #
        while cur_index < maxline:
            cur_index += 1
            newvalue += '\n'
            line = infile[cur_index]
            if line.find(quot) == -1:
                newvalue += line
            else:
                # end of multiline, process it
                break
        else:
            # we've got to the end of the config, oops...
            raise SyntaxError
        mat = multi_line.match(line)
        if mat is None:
            # a badly formed line
            raise SyntaxError
        (value, comment) = mat.groups()
        return (newvalue + value, comment, cur_index)

    def _handle_configspec(self, configspec):
        """Parse the configspec."""
        try:
            configspec = ConfigObj(
                configspec,
                raise_errors=True,
                file_error=True,
                list_values=False)
        except ConfigObjError, e:
            # FIXME: Should these errors have a reference
            # to the already parsed ConfigObj ?
            raise ConfigspecError('Parsing configspec failed: %s' % e)
        except IOError, e:
            raise IOError('Reading configspec failed: %s' % e)
        self._set_configspec_value(configspec, self)

    def _set_configspec_value(self, configspec, section):
        """Used to recursively set configspec values."""
        if '__many__' in configspec.sections:
            section.configspec['__many__'] = configspec['__many__']
            if len(configspec.sections) > 1:
                # FIXME: can we supply any useful information here ?
                raise RepeatSectionError
        for entry in configspec.scalars:
            section.configspec[entry] = configspec[entry]
        for entry in configspec.sections:
            if entry == '__many__':
                continue
            if not section.has_key(entry):
                section[entry] = {}
            self._set_configspec_value(configspec[entry], section[entry])

    def _handle_repeat(self, section, configspec):
        """Dynamically assign configspec for repeated section."""
        try:
            section_keys = configspec.sections
            scalar_keys = configspec.scalars
        except AttributeError:
            section_keys = [entry for entry in configspec 
                                if isinstance(configspec[entry], dict)]
            scalar_keys = [entry for entry in configspec 
                                if not isinstance(configspec[entry], dict)]
        if '__many__' in section_keys and len(section_keys) > 1:
            # FIXME: can we supply any useful information here ?
            raise RepeatSectionError
        scalars = {}
        sections = {}
        for entry in scalar_keys:
            val = configspec[entry]
            scalars[entry] = val
        for entry in section_keys:
            val = configspec[entry]
            if entry == '__many__':
                scalars[entry] = val
                continue
            sections[entry] = val
        #
        section.configspec = scalars
        for entry in sections:
            if not section.has_key(entry):
                section[entry] = {}
            self._handle_repeat(section[entry], sections[entry])

    def _write_line(self, indent_string, entry, this_entry, comment):
        """Write an individual line, for the write method"""
        return '%s%s = %s%s' % (
            indent_string,
            self._quote(entry, multiline=False),
            self._quote(this_entry),
            comment)

    def _write_marker(self, indent_string, depth, entry, comment):
        """Write a section marker line"""
        return '%s%s%s%s%s' % (
            indent_string,
            '[' * depth,
            self._quote(entry, multiline=False),
            ']' * depth,
            comment)

    def _handle_comment(self, comment):
        """
        Deal with a comment.
        
        >>> filename = a.filename
        >>> a.filename = None
        >>> values = a.write()
        >>> index = 0
        >>> while index < 23:
        ...     index += 1
        ...     line = values[index-1]
        ...     assert line.endswith('# comment ' + str(index))
        >>> a.filename = filename
        
        >>> start_comment = ['# Initial Comment', '', '#']
        >>> end_comment = ['', '#', '# Final Comment']
        >>> newconfig = start_comment + testconfig1.split('\\n') + end_comment
        >>> nc = ConfigObj(newconfig)
        >>> nc.initial_comment
        ['# Initial Comment', '', '#']
        >>> nc.final_comment
        ['', '#', '# Final Comment']
        >>> nc.initial_comment == start_comment
        1
        >>> nc.final_comment == end_comment
        1
        """
        if not comment:
            return ''
        if self.indent_type == '\t':
            start = '\t'
        else:
            start = ' ' * NUM_INDENT_SPACES
        if not comment.startswith('#'):
            start += '# '
        return (start + comment)

    def _compute_indent_string(self, depth):
        """
        Compute the indent string, according to current indent_type and depth
        """
        if self.indent_type == '':
            # no indentation at all
            return ''
        if self.indent_type == '\t':
            return '\t' * depth
        if self.indent_type == ' ':
            return ' ' * NUM_INDENT_SPACES * depth
        raise SyntaxError

    # Public methods

    def write(self, section=None):
        """
        Write the current ConfigObj as a file
        
        tekNico: FIXME: use StringIO instead of real files
        
        >>> filename = a.filename
        >>> a.filename = 'test.ini'
        >>> a.write()
        >>> a.filename = filename
        >>> a == ConfigObj('test.ini', raise_errors=True)
        1
        >>> os.remove('test.ini')
        >>> b.filename = 'test.ini'
        >>> b.write()
        >>> b == ConfigObj('test.ini', raise_errors=True)
        1
        >>> os.remove('test.ini')
        >>> i.filename = 'test.ini'
        >>> i.write()
        >>> i == ConfigObj('test.ini', raise_errors=True)
        1
        >>> os.remove('test.ini')
        >>> a = ConfigObj()
        >>> a['DEFAULT'] = {'a' : 'fish'}
        >>> a['a'] = '%(a)s'
        >>> a.write()
        ['a = %(a)s', '[DEFAULT]', 'a = fish']
        """
        int_val = 'test'
        if self.indent_type is None:
            # this can be true if initialised from a dictionary
            self.indent_type = DEFAULT_INDENT_TYPE
        #
        out = []
        return_list = True
        if section is None:
            int_val = self.interpolation
            self.interpolation = False
            section = self
            return_list = False
            for line in self.initial_comment:
                stripped_line = line.strip()
                if stripped_line and not stripped_line.startswith('#'):
                    line = '# ' + line
                out.append(line)
        #
        indent_string = self._compute_indent_string(section.depth)
        for entry in (section.scalars + section.sections):
            if entry in section.defaults:
                # don't write out default values
                continue
            for comment_line in section.comments[entry]:
                comment_line = comment_line.lstrip()
                if comment_line and not comment_line.startswith('#'):
                    comment_line = '#' + comment_line
                out.append(indent_string + comment_line)
            this_entry = section[entry]
            comment = self._handle_comment(section.inline_comments[entry])
            #
            if isinstance(this_entry, dict):
                # a section
                out.append(self._write_marker(
                    indent_string,
                    this_entry.depth,
                    entry,
                    comment))
                out.extend(self.write(this_entry))
            else:
                out.append(self._write_line(
                    indent_string,
                    entry,
                    this_entry,
                    comment))
        #
        if not return_list:
            for line in self.final_comment:
                stripped_line = line.strip()
                if stripped_line and not stripped_line.startswith('#'):
                    line = '# ' + line
                out.append(line)
        #
        if int_val != 'test':
            self.interpolation = int_val
        #
        if (return_list) or (self.filename is None):
            return out
        #
        if isinstance(self.filename, StringTypes):
            h = open(self.filename, 'w')
            h.write(self.BOM or '')
            h.write('\n'.join(out))
            h.close()
        else:
            self.filename.seek(0)
            self.filename.write(self.BOM or '')
            self.filename.write('\n'.join(out))
            # if we have a stored file object (or StringIO)
            # we *don't* close it

    def validate(self, validator, section=None, preserve_errors=False):
        """
        Test the ConfigObj against a configspec.
        
        It uses the ``validator`` object from *validate.py*.
        
        To run ``validate`` on the current ConfigObj, call: ::
        
            test = config.validate(validator)
        
        (Normally having previously passed in the configspec when the ConfigObj
        was created - you can dynamically assign a dictionary of checks to the
        ``configspec`` attribute of a section though).
        
        It returns ``True`` if everything passes, or a dictionary of
        pass/fails (True/False). If every member of a subsection passes, it
        will just have the value ``True``. (It also returns ``False`` if all
        members fail).
        
        In addition, it converts the values from strings to their native
        types if their checks pass (and ``stringify`` is set).
        
        If ``preserve_errors`` is ``True`` (``False`` is default) then instead
        of a marking a fail with a ``False``, it will preserve the actual
        exception object. This can contain info about the reason for failure.
        For example the ``VdtValueTooSmallError`` indeicates that the value
        supplied was too small. If a value (or section) is missing it will
        still be marked as ``False``.
        
        You can then use the ``flatten_errors`` function to turn your nested
        results dictionary into a flattened list of failures - useful for
        displaying meaningful error messages.
        
        >>> try:
        ...     from validate import Validator
        ... except ImportError:
        ...     print >> sys.stderr, 'Cannot import the Validator object, skipping the related tests'
        ... else:
        ...     config = '''
        ...     test1=40
        ...     test2=hello
        ...     test3=3
        ...     test4=5.0
        ...     [section]
        ...         test1=40
        ...         test2=hello
        ...         test3=3
        ...         test4=5.0
        ...         [[sub section]]
        ...             test1=40
        ...             test2=hello
        ...             test3=3
        ...             test4=5.0
        ... '''.split('\\n')
        ...     configspec = '''
        ...     test1= integer(30,50)
        ...     test2= string
        ...     test3=integer
        ...     test4=float(6.0)
        ...     [section ]
        ...         test1=integer(30,50)
        ...         test2=string
        ...         test3=intege...
 
[truncated message content]

Re: [Configobj-develop] Merging configs (setting default values)

[Matthew B. <mat...@gm...>]

Hi,

>  You can only update from a dictionary (b must be a dictionary or a
> ConfigObj).

Ah yes, sorry, I was comparing your method to my function, and forgot
that.  Doh.

>  Because all subsections are instances of Section - the new method *is*
> recursive (it uses Section.update).

Yes, of course - but the standard dictionary update isn't:

>>> a =3D {'one': 1, 'two': {'as_field': 3}}
>>> b =3D {b =3D {'two': {'bs_field': 4}}
>>> a.update(b)
>>> a
>>> {'two': {'bs_field': 4}, 'one': 1}

So, for example, with the current code

a.update(b)

could give a different answer from

c =3D a.copy() # which gives a dict
c.update(b)

- or more generally, if d was a dictionary with the same (dictionary)
contents as a, then d.update(b) will not necessarily be the same as
a.update(b).

How about renaming the method to .recursive_update or something?

Best,

Matthew

Re: [Configobj-develop] Merging configs (setting default values)

[Fuzzyman <fuz...@vo...>]

Matthew Brett wrote:

>Hi,
>
>  
>
>>         for key, val in indict.items():
>>             if key in self and isinstance(self[key], dict) and
>>                                 isinstance(val, dict):
>>                 self[key].update(val)
>>             else:
>>                 self[key] = val
>>
>>
>> Just in case you try to overwrite a sub-section with a scalar value.
>>    
>>
>
>So, for
>
>a.update(b)
>
>if a is a dictionary (subsection), and b is not, then won't a be
>overwritten with the non-dictionary value?
>
>  
>

I don't understand the question.

You can only update from a dictionary (b must be a dictionary or a
ConfigObj).

If *inside* your dictionary a, one of the values is a dictionary (a
subsection) - that can be overwritten if dictionary b has a scalar for
that value.

This should be the same result as for a dictionary.

Can you show me an example (using the new code) that doesn't behave how
you would expect ?

>Does it matter that the update function works differently for the
>configobj object and a dictionary?  For example
>
>a.update(b)
>
>can give a different result for the same dictionary contents if a is
>of type dict or type configobj, because the normal update method is
>not recursive.
>
>  
>
Because all subsections are instances of Section - the new method *is*
recursive (it uses Section.update).

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

>Best,
>
>Matthew
>
>  
>

Re: [Configobj-develop] Merging configs (setting default values)

[Fuzzyman <fuz...@vo...>]

Fuzzyman wrote:
[snip..]

> This is correct - and it's a bug.
>
> Replace ``Section.update`` with :
>
>     def update(self, indict):
>         """A version of update that uses our ``__setitem__``."""
>         for key, val in indict.items():
>             if key in self and isinstance(self[key], dict):
>                 self[key].update(val)
>             else:
>                 self[key] = val
>
> and it should work.
>
Hmmm.. that had probably better be :

    def update(self, indict):
        """
        A version of update that uses our ``__setitem__``.
    >>> a = '''[section1]
    ...     option1 = True
    ...     [[subsection]]
    ...     more_options = False
    ...     # end of file'''.splitlines()
    >>> b = '''# File is user.ini
    ...     [section1]
    ...     option1 = False
    ...     # end of file'''.splitlines()
    >>> c1 = ConfigObj(b)
    >>> c2 = ConfigObj(a)
    >>> c2.update(c1)
    >>> c2
    {'section1': {'option1': 'False', 'subsection': {'more_options':
'False'}}}
        """
        for key, val in indict.items():
            if key in self and isinstance(self[key], dict) and
                                isinstance(val, dict):
                self[key].update(val)
            else:  
                self[key] = val


Just in case you try to overwrite a sub-section with a scalar value.

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

 

> Update to follow. :-)
>
> Thanks
>
> Fuzzyman
> http://www.voidspace.org.uk/python/index.shtml
>
>>>*However* - validation includes a system for setting default values. You
>>>do this by including the default in the configspec.
>>>    
>>>
>>
>>I think I can't use the configspec defaults, because this will mean
>>that the defaults are filled in for every file I load, so I won't be
>>able to tell in my merge whether the values came from the file, or
>>from the defaults - but very happy to be corrected if that's not the
>>case...
>>
>>Thanks again,
>>
>>Matthew
>>
>>  
>>
>

Re: [Configobj-develop] Merging configs (setting default values)

[Fuzzyman <fuz...@vo...>]

Matthew Brett wrote:

>Hi,
>
>  
>
>>First off you could create an external config file (or dictionary) with
>>all your default values.
>>
>>Create the initial config file from the defaults. Then *update* with the
>>user one.
>>    
>>
>
>I _think_ there's a problem with just an update though.  Let's say I
>have this in the defaults:
>
># File is default.ini
>[section1]
>option1 = True
>[[subsection]]
>more_options = False
># end of file
>
>and I have this in my user config file:
>
># File is user.ini
>[section1]
>option1 = False
># end of file
>
>If I do:
>
>a = ConfigObj('defaults.ini')
>b = ConfigObj('user.ini')
>c = a.update(b)
>
>- doesn't this end up wiping out my subsection, because update is not recursive?
>
>  
>
This is correct - and it's a bug.

Replace ``Section.update`` with :

    def update(self, indict):
        """A version of update that uses our ``__setitem__``."""
        for key, val in indict.items():
            if key in self and isinstance(self[key], dict):
                self[key].update(val)
            else:
                self[key] = val

and it should work.

Update to follow. :-)

Thanks

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

>>*However* - validation includes a system for setting default values. You
>>do this by including the default in the configspec.
>>    
>>
>
>I think I can't use the configspec defaults, because this will mean
>that the defaults are filled in for every file I load, so I won't be
>able to tell in my merge whether the values came from the file, or
>from the defaults - but very happy to be corrected if that's not the
>case...
>
>Thanks again,
>
>Matthew
>
>  
>

Re: [Configobj-develop] Merging configs (setting default values)

[Matthew B. <mat...@gm...>]

Hi,

> First off you could create an external config file (or dictionary) with
> all your default values.
>
> Create the initial config file from the defaults. Then *update* with the
> user one.

I _think_ there's a problem with just an update though.  Let's say I
have this in the defaults:

# File is default.ini
[section1]
option1 =3D True
[[subsection]]
more_options =3D False
# end of file

and I have this in my user config file:

# File is user.ini
[section1]
option1 =3D False
# end of file

If I do:

a =3D ConfigObj('defaults.ini')
b =3D ConfigObj('user.ini')
c =3D a.update(b)

- doesn't this end up wiping out my subsection, because update is not recur=
sive?

> *However* - validation includes a system for setting default values. You
> do this by including the default in the configspec.

I think I can't use the configspec defaults, because this will mean
that the defaults are filled in for every file I load, so I won't be
able to tell in my merge whether the values came from the file, or
from the defaults - but very happy to be corrected if that's not the
case...

Thanks again,

Matthew

Re: [Configobj-develop] Merging configs (setting default values)

[Fuzzyman <fuz...@vo...>]

Fuzzyman wrote:

>Setting defaults.
>
>There are two easy ways that spring to mind to set default values in a
>config file. (Sorry about the top post - but it seemed the most
>appropriate way to respond).
>
>  
>
I see (from properly reading your email :oops:) that you are already
aware of default values in configspecs.

Even for reading config values from multiple sources (a hierarchy) - the
``update`` approach should work. Note that you can also update
individual sections - not just the whole config.

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

Re: [Configobj-develop] Merging configs (setting default values)

[Fuzzyman <fuz...@vo...>]

Setting defaults.

There are two easy ways that spring to mind to set default values in a
config file. (Sorry about the top post - but it seemed the most
appropriate way to respond).

First off you could create an external config file (or dictionary) with
all your default values.

Create the initial config file from the defaults. Then *update* with the
user one. Only values et in the user config will over write the values
from the default one. You may wish to change the filename of the
resulting config : ::

    cfg = ConfigObj(default_filename) # filename or dictionary
    cfg2 = ConfigObj(user_config)

    cfg.update(cfg2)
    cfg.filename = cfg2.filename

That *probably* achieves what you want.

*However* - validation includes a system for setting default values. You
do this by including the default in the configspec.

See :
   
http://www.voidspace.org.uk/python/configobj.html#mentioning-default-values
    http://www.voidspace.org.uk/python/validate.html#default-values

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

Matthew Brett wrote:

>Hi,
>
>Just another thought.  I have been browsing the different suggested
>replacements for ConfigParser listed on the shootout page:
>
>http://wiki.python.org/moin/ConfigParserShootout
>
>ConfigObj certainly seems to get me most of the way there, but one
>feature that I really miss is the ability to merge different
>configuration files, while stlll falling back to defaults for settings
>not found in the files. This is one of the desirable features listed
>for the ConfigParser replacement:
>
>http://wiki.python.org/moin/ConfigParserShootout#head-a1bf82289c05f5749e5b0c6bea97f6b1d35c05cd
>
>For example, let us say I have a system configuration file, a local
>user configuration file and a file in the current working directory:
>
>cfg_files = [sys_cfg, user_cfg, pwd_cfg]
>
>I would like to be able to take any settings from pwd_cfg, then get
>any settings missing in pwd_cfg from user_cfg, then any still missing
>from sys_cfg, and any remaining options from my hardcoded defaults.
>
>In fact, this is very nice feature of cfgparse:
>
>http://cfgparse.sourceforge.net/
>
> - you can do something like:
>
>p = cfgparse.ConfigParser()
>p.add_option('some_option', default=True)
># etc
>for f in cfg_files:
>    if os.path.exists(_f):
>        p.add_file(_f)
>opts = p. parse()
>
>I would really like to be able to do something similar with ConfigObj,
>but my current approach is a little cludgy.  At the moment, I think I
>have a) avoid setting defaults in my configspec b) make a separate
>options string list with my defaults, and c) apply my own merge
>function (see code snippet below).  Is there a better way to do this?
>
>Thanks again,
>
>Matthew
>
>
># Code snippet:
>
>def soft_merge(a, b):
>    """
>    Fills values recursively in dictionary a from values in dictionary
>    b.  Any fields missing or None in a but present in b will be
>    filled from b. Fields which are dictionaries (objects with
>    iteritems method) will be filled recursively.  We prefer a
>    dictionary field in either a or b to a non-dictionary.
>
>    """
>    if a is None: return b
>    try:
>        b_iter = b.iteritems()
>    except:
>        return a
>
>    try:
>        a_iter = a.iteritems()
>    except:
>        return b
>
>    for (k,v) in b_iter:
>        if a.has_key(k):
>            a[k] = soft_merge(a[k], v)
>        else:
>            a[k] = v
>
>    return a
>
>cfg_spec = '''
>[my_section]
>setting1 = boolean()
>setting2 = boolean()
>""".split('\n')
>
>default_cfg = """
>[my_section]
>setting1 = True
>setting2 = False
>""".split('\n')
>
>cfg_obj = configobj.ConfigObj;
>opts = cfg_obj(None, configspec=cfg_spec)
>cfg_files.append(default_cfg)
>for f in cfg_files:
>    opts = soft_merge(opts, cfg_obj(f))
>
>
>-------------------------------------------------------
>This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
>for problems?  Stop!  Download the new AJAX search engine that makes
>searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
>http://ads.osdn.com/?ad_idv37&alloc_id865&op=click
>_______________________________________________
>Configobj-develop mailing list
>Con...@li...
>https://lists.sourceforge.net/lists/listinfo/configobj-develop
>
>  
>

[Configobj-develop] Merging configs

[Matthew B. <mat...@gm...>]

Hi,

Just another thought.  I have been browsing the different suggested
replacements for ConfigParser listed on the shootout page:

http://wiki.python.org/moin/ConfigParserShootout

ConfigObj certainly seems to get me most of the way there, but one
feature that I really miss is the ability to merge different
configuration files, while stlll falling back to defaults for settings
not found in the files. This is one of the desirable features listed
for the ConfigParser replacement:

http://wiki.python.org/moin/ConfigParserShootout#head-a1bf82289c05f5749e5b0=
c6bea97f6b1d35c05cd

For example, let us say I have a system configuration file, a local
user configuration file and a file in the current working directory:

cfg_files =3D [sys_cfg, user_cfg, pwd_cfg]

I would like to be able to take any settings from pwd_cfg, then get
any settings missing in pwd_cfg from user_cfg, then any still missing
from sys_cfg, and any remaining options from my hardcoded defaults.

In fact, this is very nice feature of cfgparse:

http://cfgparse.sourceforge.net/

 - you can do something like:

p =3D cfgparse.ConfigParser()
p.add_option('some_option', default=3DTrue)
# etc
for f in cfg_files:
    if os.path.exists(_f):
        p.add_file(_f)
opts =3D p. parse()

I would really like to be able to do something similar with ConfigObj,
but my current approach is a little cludgy.  At the moment, I think I
have a) avoid setting defaults in my configspec b) make a separate
options string list with my defaults, and c) apply my own merge
function (see code snippet below).  Is there a better way to do this?

Thanks again,

Matthew


# Code snippet:

def soft_merge(a, b):
    """
    Fills values recursively in dictionary a from values in dictionary
    b.  Any fields missing or None in a but present in b will be
    filled from b. Fields which are dictionaries (objects with
    iteritems method) will be filled recursively.  We prefer a
    dictionary field in either a or b to a non-dictionary.

    """
    if a is None: return b
    try:
        b_iter =3D b.iteritems()
    except:
        return a

    try:
        a_iter =3D a.iteritems()
    except:
        return b

    for (k,v) in b_iter:
        if a.has_key(k):
            a[k] =3D soft_merge(a[k], v)
        else:
            a[k] =3D v

    return a

cfg_spec =3D '''
[my_section]
setting1 =3D boolean()
setting2 =3D boolean()
""".split('\n')

default_cfg =3D """
[my_section]
setting1 =3D True
setting2 =3D False
""".split('\n')

cfg_obj =3D configobj.ConfigObj;
opts =3D cfg_obj(None, configspec=3Dcfg_spec)
cfg_files.append(default_cfg)
for f in cfg_files:
    opts =3D soft_merge(opts, cfg_obj(f))

Re: [Configobj-develop] Getting those nice validation error messages out

[Fuzzyman <fuz...@vo...>]

Matthew Brett wrote:

>Hi,
>
>  
>
>> You would still have to produce your own error messages from these of
>>course - would it be useful if I showed you a (recursive) function to
>>produce a list like this ?
>>    
>>
>
>I ended up with something like this:
>
># Check / processing functions for options
>def check_result(sct, res, err_strs=[], levels=['[root]']):
>    if res is True: return
>    if res is False: res = dict(zip(sct.keys(), [res]*len(sct)))
>
>    for (k, v) in sct.items():
>        if res[k] is True: continue
>        if isinstance(v, dict):
>            # Go down one level
>            levels.append(k)
>            check_result(v, res[k], err_strs, levels)
>            continue
>        try:
>            vtor.check(sct.configspec[k], v)
>        except validate.ValidateError, err:
>            err_strs.append("%s, %s: %s" % (", ".join(levels), k, err))
>
>    # Go up one level
>    if levels: levels.pop()
>
>    return err_strs
>
># Run checks
>val_res = opts.validate(vtor)
>if val_res is not True:
>    print "\n".join(check_result(opts, val_res))
>
>I hope this is efficient, I am rather new to python. Of course I was
>wondering if I was missing some way of avoiding redoing the error
>checks, but it sounds like I haven't, and this works fine for me.  I
>wonder if there is any interest in a patch to the configobj.py
>funciton, on the lines of:
>
>def validate(self, validator, section=None, error_strs=[], levels=['base'])
>
>then
>
>val_res = opts.validate(vtor, errror_strs = my_list)?
>  
>
Right, I see.

The validate method does actually lose information - the check method
*does* give you a more useful error message (like value too large, too
small etc). The  validate method just turns this into True/False.

So currently - the only option is to re-run the test to retrieve the
actual error.

Yes - it sounds like an extra option to validate  would be a good idea.
I don't think I'd return it as a flattened list by default - maybe just
replace the True/False with the error messages, and maybe include a
function like yours as an example.

ConfigObj 4.0.3 anyone :-) (I'll probably implement this in the next few
days - it's a good idea).

Your function looks fine by the way.

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

>Thanks a lot.
>
>Matthew
>
>  
>

Re: [Configobj-develop] Getting those nice validation error messages out

[Matthew B. <mat...@gm...>]

Hi,

>  You would still have to produce your own error messages from these of
> course - would it be useful if I showed you a (recursive) function to
> produce a list like this ?

I ended up with something like this:

# Check / processing functions for options
def check_result(sct, res, err_strs=3D[], levels=3D['[root]']):
    if res is True: return
    if res is False: res =3D dict(zip(sct.keys(), [res]*len(sct)))

    for (k, v) in sct.items():
        if res[k] is True: continue
        if isinstance(v, dict):
            # Go down one level
            levels.append(k)
            check_result(v, res[k], err_strs, levels)
            continue
        try:
            vtor.check(sct.configspec[k], v)
        except validate.ValidateError, err:
            err_strs.append("%s, %s: %s" % (", ".join(levels), k, err))

    # Go up one level
    if levels: levels.pop()

    return err_strs

# Run checks
val_res =3D opts.validate(vtor)
if val_res is not True:
    print "\n".join(check_result(opts, val_res))

I hope this is efficient, I am rather new to python. Of course I was
wondering if I was missing some way of avoiding redoing the error
checks, but it sounds like I haven't, and this works fine for me.  I
wonder if there is any interest in a patch to the configobj.py
funciton, on the lines of:

def validate(self, validator, section=3DNone, error_strs=3D[], levels=3D['b=
ase'])

then

val_res =3D opts.validate(vtor, errror_strs =3D my_list)?

Thanks a lot.

Matthew

Re: [Configobj-develop] Getting those nice validation error messages out

[Fuzzyman <fuz...@vo...>]

Matthew Brett wrote:

>No problem, thanks very much for letting me know...
>
>  
>
Hello Matthew,

I don't think my previous answer addressed all the issues you were
asking about.

I realise that you also want to include in your list *why* the test
failed. Again the question is "in what form do you want these error
messages" ?

Obviously the information you have to go off is the original test (or if
the entry is just plain missing) - if you had a flattened list of
failures (of the sort I discussed) you could have each entry as a tuple :

    (entry_name, original_test)

You would still have to produce your own error messages from these of
course - would it be useful if I showed you a (recursive) function to
produce a list like this ?

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml


>On 12/3/05, Fuzzyman <fuz...@vo...> wrote:
>  
>
>>Hello Matthew,
>>
>>I'm very short of time - I'll do you a proper reply on Monday, sorry
>>about that.
>>
>>Michael
>>
>>Matthew Brett wrote:
>>
>>    
>>
>>>Hi,
>>>
>>>I am very sorry if I missed something simple, but is there an easy way
>>>to get a list of informative validation error messages from the
>>>ConfigObj.validate interface?
>>>
>>>I have run:
>>>
>>>res = config_obj.validate(vtor)
>>>
>>>and have a nested list of boolean values - but I want to be able to
>>>print out a list of items that failed validation and why.  I am just
>>>thinking about reiterating through the option dictionary, running the
>>>validation by hand (.check) for each False in the 'res' dictionary,
>>>but this seems very cludgy - is there a better way?
>>>
>>>Thanks a lot,
>>>
>>>Matthew
>>>
>>>
>>>-------------------------------------------------------
>>>This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
>>>for problems?  Stop!  Download the new AJAX search engine that makes
>>>searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
>>>http://ads.osdn.com/?ad_idv37&alloc_id865&op=click
>>>_______________________________________________
>>>Configobj-develop mailing list
>>>Con...@li...
>>>https://lists.sourceforge.net/lists/listinfo/configobj-develop
>>>
>>>
>>>
>>>      
>>>
>>    
>>
>
>  
>

[Configobj-develop] Re: Validation errors

[Fuzzyman <fuz...@vo...>]

Hello Matthew,

> Matthew Brett wrote:
>
> >Hi,
> >
> >I am very sorry if I missed something simple, but is there an easy way
> >to get a list of informative validation error messages from the
> >ConfigObj.validate interface?
> >
> >I have run:
> >
> >res = config_obj.validate(vtor)
> >
> >and have a nested list of boolean values - but I want to be able to

res is a nested *dictionary*. It has the same structure as your configobj.


> >print out a list of items that failed validation and why.  I am just

So how do you want the failures represented ?
You want your nested dictionary of True/False flattened to a list of failures ?

Simple enough... *but*, how do you want nested keys represented in this list ?

Suppose you have a config file that looks like :

Key1 = value1
Key2 = value2
[section]
    Key1 = value1
    Key2 = value2
    [sub-section]
        Key1 = value1
        Key2 = value2

Suppose Key1 in the top level section *and* Key1 in 'sub-section' fail, how do you want to show that ?

I can suggest two options :

['Key1', 'section/sub-section/Key1']
[['Key1'], [['section', ['sub-section'], ['Key1']] ]

The first is 'flatter', the second is arguably more 'accurate'.

It should be easy enough to write a function that iterates over either the res dictionary or your ConfigObj *and* the res dictionary - flattening it to a list.

Would this do what you want ?

All the best,


Fuzzyman
Http://www.voidspace.org.uk/python/index.shtml

> >thinking about reiterating through the option dictionary, running the
> >validation by hand (.check) for each False in the 'res' dictionary,
> >but this seems very cludgy - is there a better way?
> >
> >Thanks a lot,
> >
> >Matthew

[Configobj-develop] ConfigObj 4.0.2 and pythonutils

[Fuzzyman <fuz...@vo...>]

Hello all,

ConfigObj 4.0.2 is now available.

This is a minor bugfix release - fixing the bug in ``create_empty``. Blog entry and announcement to ``comp.lang.python.announce`` to follow.

As ever, the homepage is http://www.voidspace.org.uk/python/configobj.html

There are also new versions of pathutils and cgiutils. See :
Http://www.voidspace.org.uk/python/recipebook.shtml#utils

Tomorrow I release an updated version of odict the Ordered Dictionary. If this proves stable, I will tie these together with a new pythonutils release.

All the best,

Fuzzyman
Http://www.voidspace.org.uk/python/index.shtml

[Configobj-develop] Getting those nice validation error messages out

[Matthew B. <mat...@gm...>]

Hi,

I am very sorry if I missed something simple, but is there an easy way
to get a list of informative validation error messages from the
ConfigObj.validate interface?

I have run:

res =3D config_obj.validate(vtor)

and have a nested list of boolean values - but I want to be able to
print out a list of items that failed validation and why.  I am just
thinking about reiterating through the option dictionary, running the
validation by hand (.check) for each False in the 'res' dictionary,
but this seems very cludgy - is there a better way?

Thanks a lot,

Matthew

[Configobj-develop] Re: create_empty fails

[Fuzzyman <fuz...@vo...>]

Sorry about the late reply to the "create_empty" patch.

My subscription to this list was up the wall.

The patch is noted and will be in ConfigObj 4.0.2

Note that the current version is ConfigObj 4.0.1 - which has a few
bugfixes since the b5 version.

Thanks

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

[Configobj-develop] validatation of ConfigObj strings with embedded commas

[Jim V. <Jim...@no...>]

Hello,

I am a new user of the pythonutils package (which looks to be very 
nice), but I am not able to get a config file entry, with embedded 
commas, to be converted to a string via the validate() procedure.  
Attached is a script that demonstrates the behavior.  What am I missing?

Thanks, -- jv

P.S.
Here is the output of the attached script when run on my computer:

sys.version:             2.4.1 (#65, Mar 30 2005, 09:13:57) [MSC v.1310 
32 bit (Intel)]
sys.getwindowsversion(): (5, 1, 2600, 2, 'Service Pack 2')
sys.platform:            win32
sys.winver:              2.4

configobj.__version__: 4.0.0
validate.__version__:  0.2.0

*** validation succeeded ***
Traceback (most recent call last):
  File "C:\Documents and Settings\jim.vickroy\My 
Documents\Projects\_experimental_\pythonutils\configobj\test-reader.py", 
line 48, in ?
    assert facts['a string with commas'] == 'a, b, c', facts['a string 
with commas'] # this one fails !
AssertionError: ['a', 'b', 'c']


Notice how the string ('a, b, c') with embedded commas is being 
converted to a list of strings.

Re: [Configobj-develop] Ordered Dictionary, keys and values

[Nicola L. <ni...@te...>]

[Sorry for the lateness, it's been a busy week, and it's not over yet.]


> What about this suggestion. It sounds a little esoteric to me - but if
> you think it's worth the effort I'll do it.

I am not sure I understand it (did not yet read the whole comp.lang.python
thread). Can you give a few usage examples?


> We could implement the ``keys`` and ``values`` methods as custom objects
> with all the sequence methods. (With error checking so you can't mutate
> the keys so that they don't reflect the internal contents of the odict).

Do you mean that the ``keys`` and ``values`` methods *return* custom objects?


> We would also implement ``__getitem__``, ``__setitem__``, and
> ``__call__`` such that you can directly index (slice, delete, etc)
> members as if it was a list.

Do you mean as these?

k4 = d1.keys()[4]
d1.values()[4] = v4


> You can also call it as if it was a method
> - including passing it a sequence to replace the current keys with.
> 
> We would presumably have to implement ``__setattr__`` on the main
> OrderedDict so that :
> 
> d1 = OrderedDict(sequence_of_tuples)
> d1.keys = a_list
> 
> is the same as :
> 
> d1 = OrderedDict(sequence_of_tuples)
> d1.keys(a_list)

If the two forms are equivalent, I'd say just do the first...

> What should ``d1.keys(a_list)`` return ? (The same as d1.keys() I guess
> - except with the new key set).

...so that we dodge this question. :-)


> We (I) would still implement slicing, insert, reverse, and sort for the
> OrderedDict class itself.

What is there to sort? Isn't it already sorted? :-)


> Fredrik Lundh pointed out the performance impact of adding to many
> layers above the C implementation. I have no idea how relevant that is
> to the suggestion I've just made. :-) (How many method calls would
> indexing or assigning to the ``keys`` object actually result in 'under
> the hood' ?)

I'm not sure I grasp the last sentence. Performance is good, and I
generally like *some* optimization, but in this context convenience and
versatility, without excess, do override performance concerns.

-- 
Nicola Larosa - ni...@te...

She was up the learning curve like a mountain goat.
 -- WasterDave on Slashdot, October 2005

[Configobj-develop] Ordered Dictionary, keys and values

[Fuzzyman <fuz...@vo...>]

Hello Nicola, (et al)

This is still vaguely on topic for this list. After completing the (next
phase of) OrderedDict improvements I need to decide how many of those
improvements to move into ConfigObj.

(And yes Nicola, I *do* remember your suggestion that I derive ConfigObj
from OrderedDict - and I *still* don't want ConfigObj to depend on it).

What about this suggestion. It sounds a little esoteric to me - but if
you think it's worth the effort I'll do it.

We could implement the ``keys`` and ``values`` methods as custom objects
with all the sequence methods. (With error checking so you can't mutate
the keys so that they don't reflect the internal contents of the odict).

We would also implement ``__getitem__``, ``__setitem__``, and
``__call__`` such that you can directly index (slice, delete, etc)
members as if it was a list. You can also call it as if it was a method
- including passing it a sequence to replace the current keys with.

We would presumably have to implement ``__setattr__`` on the main
OrderedDict so that :

d1 = OrderedDict(sequence_of_tuples)
d1.keys = a_list

is the same as :

d1 = OrderedDict(sequence_of_tuples)
d1.keys(a_list)

What should ``d1.keys(a_list)`` return ? (The same as d1.keys() I guess
- except with the new key set).

We (I) would still implement slicing, insert, reverse, and sort for the
OrderedDict class itself.

Fredrik Lundh pointed out the performance impact of adding to many
layers above the C implementation. I have no idea how relevant that is
to the suggestion I've just made. :-) (How many method calls would
indexing or assigning to the ``keys`` object actually result in 'under
the hood' ?)

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

[Configobj-develop] odict (and SVN)

[Fuzzyman <fuz...@vo...>]

Hello all,

We're about to make changes to odict (OrderedDictionary) module.

pythonutils doesn't yet have it's own mailing list - it *probably* needs
it's own sourceforge project with mailing list. *sigh*

I temporarily don't have easy access to committing to SVN - so I'll have
to get Nicola to commit the changes.

The changes will add sequence methods (including slicing) and improve
performance.

It will also break backwards compatibility by hiding the ``sequence``
attribute and providing access to it through the keys and values methods.

Any objections should be raised now, before it's too late. :-)

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

[Configobj-develop] create_empty fails

[Paul J. <pj...@pl...>]

create_empty fails to accomplish its goal (at least, version 4.0.0b5 does)
without the following patch:

@@ -772,7 +772,7 @@
                 if self.create_empty:
                     # this is a good test that the filename specified
                     # isn't impossible - like on a non existent device
-                    h = open(self.filename)
+                    h = open(self.filename,'w')
                     h.write('')
                     h.close()
                 infile = []