configobj-develop

[Configobj-develop] Merging configs

[Matthew B. <mat...@gm...>]

Hi,

Just another thought.  I have been browsing the different suggested
replacements for ConfigParser listed on the shootout page:

http://wiki.python.org/moin/ConfigParserShootout

ConfigObj certainly seems to get me most of the way there, but one
feature that I really miss is the ability to merge different
configuration files, while stlll falling back to defaults for settings
not found in the files. This is one of the desirable features listed
for the ConfigParser replacement:

http://wiki.python.org/moin/ConfigParserShootout#head-a1bf82289c05f5749e5b0=
c6bea97f6b1d35c05cd

For example, let us say I have a system configuration file, a local
user configuration file and a file in the current working directory:

cfg_files =3D [sys_cfg, user_cfg, pwd_cfg]

I would like to be able to take any settings from pwd_cfg, then get
any settings missing in pwd_cfg from user_cfg, then any still missing
from sys_cfg, and any remaining options from my hardcoded defaults.

In fact, this is very nice feature of cfgparse:

http://cfgparse.sourceforge.net/

 - you can do something like:

p =3D cfgparse.ConfigParser()
p.add_option('some_option', default=3DTrue)
# etc
for f in cfg_files:
    if os.path.exists(_f):
        p.add_file(_f)
opts =3D p. parse()

I would really like to be able to do something similar with ConfigObj,
but my current approach is a little cludgy.  At the moment, I think I
have a) avoid setting defaults in my configspec b) make a separate
options string list with my defaults, and c) apply my own merge
function (see code snippet below).  Is there a better way to do this?

Thanks again,

Matthew


# Code snippet:

def soft_merge(a, b):
    """
    Fills values recursively in dictionary a from values in dictionary
    b.  Any fields missing or None in a but present in b will be
    filled from b. Fields which are dictionaries (objects with
    iteritems method) will be filled recursively.  We prefer a
    dictionary field in either a or b to a non-dictionary.

    """
    if a is None: return b
    try:
        b_iter =3D b.iteritems()
    except:
        return a

    try:
        a_iter =3D a.iteritems()
    except:
        return b

    for (k,v) in b_iter:
        if a.has_key(k):
            a[k] =3D soft_merge(a[k], v)
        else:
            a[k] =3D v

    return a

cfg_spec =3D '''
[my_section]
setting1 =3D boolean()
setting2 =3D boolean()
""".split('\n')

default_cfg =3D """
[my_section]
setting1 =3D True
setting2 =3D False
""".split('\n')

cfg_obj =3D configobj.ConfigObj;
opts =3D cfg_obj(None, configspec=3Dcfg_spec)
cfg_files.append(default_cfg)
for f in cfg_files:
    opts =3D soft_merge(opts, cfg_obj(f))

Re: [Configobj-develop] Merging configs (setting default values)

[Fuzzyman <fuz...@vo...>]

Setting defaults.

There are two easy ways that spring to mind to set default values in a
config file. (Sorry about the top post - but it seemed the most
appropriate way to respond).

First off you could create an external config file (or dictionary) with
all your default values.

Create the initial config file from the defaults. Then *update* with the
user one. Only values et in the user config will over write the values
from the default one. You may wish to change the filename of the
resulting config : ::

    cfg = ConfigObj(default_filename) # filename or dictionary
    cfg2 = ConfigObj(user_config)

    cfg.update(cfg2)
    cfg.filename = cfg2.filename

That *probably* achieves what you want.

*However* - validation includes a system for setting default values. You
do this by including the default in the configspec.

See :
   
http://www.voidspace.org.uk/python/configobj.html#mentioning-default-values
    http://www.voidspace.org.uk/python/validate.html#default-values

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

Matthew Brett wrote:

>Hi,
>
>Just another thought.  I have been browsing the different suggested
>replacements for ConfigParser listed on the shootout page:
>
>http://wiki.python.org/moin/ConfigParserShootout
>
>ConfigObj certainly seems to get me most of the way there, but one
>feature that I really miss is the ability to merge different
>configuration files, while stlll falling back to defaults for settings
>not found in the files. This is one of the desirable features listed
>for the ConfigParser replacement:
>
>http://wiki.python.org/moin/ConfigParserShootout#head-a1bf82289c05f5749e5b0c6bea97f6b1d35c05cd
>
>For example, let us say I have a system configuration file, a local
>user configuration file and a file in the current working directory:
>
>cfg_files = [sys_cfg, user_cfg, pwd_cfg]
>
>I would like to be able to take any settings from pwd_cfg, then get
>any settings missing in pwd_cfg from user_cfg, then any still missing
>from sys_cfg, and any remaining options from my hardcoded defaults.
>
>In fact, this is very nice feature of cfgparse:
>
>http://cfgparse.sourceforge.net/
>
> - you can do something like:
>
>p = cfgparse.ConfigParser()
>p.add_option('some_option', default=True)
># etc
>for f in cfg_files:
>    if os.path.exists(_f):
>        p.add_file(_f)
>opts = p. parse()
>
>I would really like to be able to do something similar with ConfigObj,
>but my current approach is a little cludgy.  At the moment, I think I
>have a) avoid setting defaults in my configspec b) make a separate
>options string list with my defaults, and c) apply my own merge
>function (see code snippet below).  Is there a better way to do this?
>
>Thanks again,
>
>Matthew
>
>
># Code snippet:
>
>def soft_merge(a, b):
>    """
>    Fills values recursively in dictionary a from values in dictionary
>    b.  Any fields missing or None in a but present in b will be
>    filled from b. Fields which are dictionaries (objects with
>    iteritems method) will be filled recursively.  We prefer a
>    dictionary field in either a or b to a non-dictionary.
>
>    """
>    if a is None: return b
>    try:
>        b_iter = b.iteritems()
>    except:
>        return a
>
>    try:
>        a_iter = a.iteritems()
>    except:
>        return b
>
>    for (k,v) in b_iter:
>        if a.has_key(k):
>            a[k] = soft_merge(a[k], v)
>        else:
>            a[k] = v
>
>    return a
>
>cfg_spec = '''
>[my_section]
>setting1 = boolean()
>setting2 = boolean()
>""".split('\n')
>
>default_cfg = """
>[my_section]
>setting1 = True
>setting2 = False
>""".split('\n')
>
>cfg_obj = configobj.ConfigObj;
>opts = cfg_obj(None, configspec=cfg_spec)
>cfg_files.append(default_cfg)
>for f in cfg_files:
>    opts = soft_merge(opts, cfg_obj(f))
>
>
>-------------------------------------------------------
>This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
>for problems?  Stop!  Download the new AJAX search engine that makes
>searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
>http://ads.osdn.com/?ad_idv37&alloc_id865&op=click
>_______________________________________________
>Configobj-develop mailing list
>Con...@li...
>https://lists.sourceforge.net/lists/listinfo/configobj-develop
>
>  
>

Re: [Configobj-develop] Merging configs (setting default values)

[Fuzzyman <fuz...@vo...>]

Fuzzyman wrote:

>Setting defaults.
>
>There are two easy ways that spring to mind to set default values in a
>config file. (Sorry about the top post - but it seemed the most
>appropriate way to respond).
>
>  
>
I see (from properly reading your email :oops:) that you are already
aware of default values in configspecs.

Even for reading config values from multiple sources (a hierarchy) - the
``update`` approach should work. Note that you can also update
individual sections - not just the whole config.

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

Re: [Configobj-develop] Merging configs (setting default values)

[Matthew B. <mat...@gm...>]

Hi,

> First off you could create an external config file (or dictionary) with
> all your default values.
>
> Create the initial config file from the defaults. Then *update* with the
> user one.

I _think_ there's a problem with just an update though.  Let's say I
have this in the defaults:

# File is default.ini
[section1]
option1 =3D True
[[subsection]]
more_options =3D False
# end of file

and I have this in my user config file:

# File is user.ini
[section1]
option1 =3D False
# end of file

If I do:

a =3D ConfigObj('defaults.ini')
b =3D ConfigObj('user.ini')
c =3D a.update(b)

- doesn't this end up wiping out my subsection, because update is not recur=
sive?

> *However* - validation includes a system for setting default values. You
> do this by including the default in the configspec.

I think I can't use the configspec defaults, because this will mean
that the defaults are filled in for every file I load, so I won't be
able to tell in my merge whether the values came from the file, or
from the defaults - but very happy to be corrected if that's not the
case...

Thanks again,

Matthew

Re: [Configobj-develop] Merging configs (setting default values)

[Fuzzyman <fuz...@vo...>]

Matthew Brett wrote:

>Hi,
>
>  
>
>>First off you could create an external config file (or dictionary) with
>>all your default values.
>>
>>Create the initial config file from the defaults. Then *update* with the
>>user one.
>>    
>>
>
>I _think_ there's a problem with just an update though.  Let's say I
>have this in the defaults:
>
># File is default.ini
>[section1]
>option1 = True
>[[subsection]]
>more_options = False
># end of file
>
>and I have this in my user config file:
>
># File is user.ini
>[section1]
>option1 = False
># end of file
>
>If I do:
>
>a = ConfigObj('defaults.ini')
>b = ConfigObj('user.ini')
>c = a.update(b)
>
>- doesn't this end up wiping out my subsection, because update is not recursive?
>
>  
>
This is correct - and it's a bug.

Replace ``Section.update`` with :

    def update(self, indict):
        """A version of update that uses our ``__setitem__``."""
        for key, val in indict.items():
            if key in self and isinstance(self[key], dict):
                self[key].update(val)
            else:
                self[key] = val

and it should work.

Update to follow. :-)

Thanks

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

>>*However* - validation includes a system for setting default values. You
>>do this by including the default in the configspec.
>>    
>>
>
>I think I can't use the configspec defaults, because this will mean
>that the defaults are filled in for every file I load, so I won't be
>able to tell in my merge whether the values came from the file, or
>from the defaults - but very happy to be corrected if that's not the
>case...
>
>Thanks again,
>
>Matthew
>
>  
>

Re: [Configobj-develop] Merging configs (setting default values)

[Fuzzyman <fuz...@vo...>]

Fuzzyman wrote:
[snip..]

> This is correct - and it's a bug.
>
> Replace ``Section.update`` with :
>
>     def update(self, indict):
>         """A version of update that uses our ``__setitem__``."""
>         for key, val in indict.items():
>             if key in self and isinstance(self[key], dict):
>                 self[key].update(val)
>             else:
>                 self[key] = val
>
> and it should work.
>
Hmmm.. that had probably better be :

    def update(self, indict):
        """
        A version of update that uses our ``__setitem__``.
    >>> a = '''[section1]
    ...     option1 = True
    ...     [[subsection]]
    ...     more_options = False
    ...     # end of file'''.splitlines()
    >>> b = '''# File is user.ini
    ...     [section1]
    ...     option1 = False
    ...     # end of file'''.splitlines()
    >>> c1 = ConfigObj(b)
    >>> c2 = ConfigObj(a)
    >>> c2.update(c1)
    >>> c2
    {'section1': {'option1': 'False', 'subsection': {'more_options':
'False'}}}
        """
        for key, val in indict.items():
            if key in self and isinstance(self[key], dict) and
                                isinstance(val, dict):
                self[key].update(val)
            else:  
                self[key] = val


Just in case you try to overwrite a sub-section with a scalar value.

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

 

> Update to follow. :-)
>
> Thanks
>
> Fuzzyman
> http://www.voidspace.org.uk/python/index.shtml
>
>>>*However* - validation includes a system for setting default values. You
>>>do this by including the default in the configspec.
>>>    
>>>
>>
>>I think I can't use the configspec defaults, because this will mean
>>that the defaults are filled in for every file I load, so I won't be
>>able to tell in my merge whether the values came from the file, or
>>from the defaults - but very happy to be corrected if that's not the
>>case...
>>
>>Thanks again,
>>
>>Matthew
>>
>>  
>>
>

Re: [Configobj-develop] Merging configs (setting default values)

[Fuzzyman <fuz...@vo...>]

Matthew Brett wrote:

>Hi,
>
>  
>
>>         for key, val in indict.items():
>>             if key in self and isinstance(self[key], dict) and
>>                                 isinstance(val, dict):
>>                 self[key].update(val)
>>             else:
>>                 self[key] = val
>>
>>
>> Just in case you try to overwrite a sub-section with a scalar value.
>>    
>>
>
>So, for
>
>a.update(b)
>
>if a is a dictionary (subsection), and b is not, then won't a be
>overwritten with the non-dictionary value?
>
>  
>

I don't understand the question.

You can only update from a dictionary (b must be a dictionary or a
ConfigObj).

If *inside* your dictionary a, one of the values is a dictionary (a
subsection) - that can be overwritten if dictionary b has a scalar for
that value.

This should be the same result as for a dictionary.

Can you show me an example (using the new code) that doesn't behave how
you would expect ?

>Does it matter that the update function works differently for the
>configobj object and a dictionary?  For example
>
>a.update(b)
>
>can give a different result for the same dictionary contents if a is
>of type dict or type configobj, because the normal update method is
>not recursive.
>
>  
>
Because all subsections are instances of Section - the new method *is*
recursive (it uses Section.update).

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

>Best,
>
>Matthew
>
>  
>

Re: [Configobj-develop] Merging configs (setting default values)

[Matthew B. <mat...@gm...>]

Hi,

>  You can only update from a dictionary (b must be a dictionary or a
> ConfigObj).

Ah yes, sorry, I was comparing your method to my function, and forgot
that.  Doh.

>  Because all subsections are instances of Section - the new method *is*
> recursive (it uses Section.update).

Yes, of course - but the standard dictionary update isn't:

>>> a =3D {'one': 1, 'two': {'as_field': 3}}
>>> b =3D {b =3D {'two': {'bs_field': 4}}
>>> a.update(b)
>>> a
>>> {'two': {'bs_field': 4}, 'one': 1}

So, for example, with the current code

a.update(b)

could give a different answer from

c =3D a.copy() # which gives a dict
c.update(b)

- or more generally, if d was a dictionary with the same (dictionary)
contents as a, then d.update(b) will not necessarily be the same as
a.update(b).

How about renaming the method to .recursive_update or something?

Best,

Matthew

Re: [Configobj-develop] Merging configs (setting default values)

[Fuzzyman <fuz...@vo...>]

# configobj.py
# A config file reader/writer that supports nested sections in config files.
# Copyright (C) 2005 Michael Foord, Nicola Larosa
# E-mail: fuzzyman AT voidspace DOT org DOT uk
#         nico AT tekNico DOT net

# ConfigObj 4

# Released subject to the BSD License
# Please see http://www.voidspace.org.uk/python/license.shtml

# Scripts maintained at http://www.voidspace.org.uk/python/index.shtml
# For information about bugfixes, updates and support, please join the
# ConfigObj mailing list:
# http://lists.sourceforge.net/lists/listinfo/configobj-develop
# Comments, suggestions and bug reports welcome.

"""
    >>> z = ConfigObj()
    >>> z['a'] = 'a'
    >>> z['sect'] = {
    ...    'subsect': {
    ...         'a': 'fish',
    ...         'b': 'wobble',
    ...     },
    ...     'member': 'value',
    ... }
    >>> x = ConfigObj(z.write())
    >>> z == x
    1
"""

import sys
INTP_VER = sys.version_info[:2]
if INTP_VER < (2, 2):
    raise RuntimeError("Python v.2.2 or later needed")

import os, re
from types import StringTypes

# the UTF8 BOM - from codecs module
BOM_UTF8 = '\xef\xbb\xbf'

__version__ = '4.0.3'

__revision__ = '$Id: configobj.py 147 2005-11-08 12:08:49Z fuzzyman $'

__docformat__ = "restructuredtext en"

__all__ = (
    '__version__',
    'BOM_UTF8',
    'DEFAULT_INDENT_TYPE',
    'NUM_INDENT_SPACES',
    'MAX_INTERPOL_DEPTH',
    'ConfigObjError',
    'NestingError',
    'ParseError',
    'DuplicateError',
    'ConfigspecError',
    'ConfigObj',
    'SimpleVal',
    'InterpolationError',
    'InterpolationDepthError',
    'MissingInterpolationOption',
    'RepeatSectionError',
    '__docformat__',
    'flatten_errors',
)

DEFAULT_INDENT_TYPE = ' '
NUM_INDENT_SPACES = 4
MAX_INTERPOL_DEPTH = 10

OPTION_DEFAULTS = {
    'interpolation': True,
    'raise_errors': False,
    'list_values': True,
    'create_empty': False,
    'file_error': False,
    'configspec': None,
    'stringify': True,
    # option may be set to one of ('', ' ', '\t')
    'indent_type': None,
}

class ConfigObjError(SyntaxError):
    """
    This is the base class for all errors that ConfigObj raises.
    It is a subclass of SyntaxError.
    
    >>> raise ConfigObjError
    Traceback (most recent call last):
    ConfigObjError
    """
    def __init__(self, message='', line_number=None, line=''):
        self.line = line
        self.line_number = line_number
        self.message = message
        SyntaxError.__init__(self, message)

class NestingError(ConfigObjError):
    """
    This error indicates a level of nesting that doesn't match.
    
    >>> raise NestingError
    Traceback (most recent call last):
    NestingError
    """

class ParseError(ConfigObjError):
    """
    This error indicates that a line is badly written.
    It is neither a valid ``key = value`` line,
    nor a valid section marker line.
    
    >>> raise ParseError
    Traceback (most recent call last):
    ParseError
    """

class DuplicateError(ConfigObjError):
    """
    The keyword or section specified already exists.
    
    >>> raise DuplicateError
    Traceback (most recent call last):
    DuplicateError
    """

class ConfigspecError(ConfigObjError):
    """
    An error occured whilst parsing a configspec.
    
    >>> raise ConfigspecError
    Traceback (most recent call last):
    ConfigspecError
    """

class InterpolationError(ConfigObjError):
    """Base class for the two interpolation errors."""

class InterpolationDepthError(InterpolationError):
    """Maximum interpolation depth exceeded in string interpolation."""

    def __init__(self, option):
        """
        >>> raise InterpolationDepthError('yoda')
        Traceback (most recent call last):
        InterpolationDepthError: max interpolation depth exceeded in value "yoda".
        """
        InterpolationError.__init__(
            self,
            'max interpolation depth exceeded in value "%s".' % option)

class RepeatSectionError(ConfigObjError):
    """
    This error indicates additional sections in a section with a
    ``__many__`` (repeated) section.
    
    >>> raise RepeatSectionError
    Traceback (most recent call last):
    RepeatSectionError
    """

class MissingInterpolationOption(InterpolationError):
    """A value specified for interpolation was missing."""

    def __init__(self, option):
        """
        >>> raise MissingInterpolationOption('yoda')
        Traceback (most recent call last):
        MissingInterpolationOption: missing option "yoda" in interpolation.
        """
        InterpolationError.__init__(
            self,
            'missing option "%s" in interpolation.' % option)

class Section(dict):
    """
    A dictionary-like object that represents a section in a config file.
    
    It does string interpolation if the 'interpolate' attribute
    of the 'main' object is set to True.
    
    Interpolation is tried first from the 'DEFAULT' section of this object,
    next from the 'DEFAULT' section of the parent, lastly the main object.
    
    A Section will behave like an ordered dictionary - following the
    order of the ``scalars`` and ``sections`` attributes.
    You can use this to change the order of members.
    
    Iteration follows the order: scalars, then sections.
    """

    _KEYCRE = re.compile(r"%\(([^)]*)\)s|.")

    def __init__(self, parent, depth, main, indict=None, name=None):
        """
        * parent is the section above
        * depth is the depth level of this section
        * main is the main ConfigObj
        * indict is a dictionary to initialise the section with
        """
        if indict is None:
            indict = {}
        dict.__init__(self)
        # used for nesting level *and* interpolation
        self.parent = parent
        # used for the interpolation attribute
        self.main = main
        # level of nesting depth of this Section
        self.depth = depth
        # the sequence of scalar values in this Section
        self.scalars = []
        # the sequence of sections in this Section
        self.sections = []
        # purely for information
        self.name = name
        # for comments :-)
        self.comments = {}
        self.inline_comments = {}
        # for the configspec
        self.configspec = {}
        # for defaults
        self.defaults = []
        #
        # we do this explicitly so that __setitem__ is used properly
        # (rather than just passing to ``dict.__init__``)
        for entry in indict:
            self[entry] = indict[entry]

    def _interpolate(self, value):
        """Nicked from ConfigParser."""
        depth = MAX_INTERPOL_DEPTH
        # loop through this until it's done
        while depth:
            depth -= 1
            if value.find("%(") != -1:
                value = self._KEYCRE.sub(self._interpolation_replace, value)
            else:
                break
        else:
            raise InterpolationDepthError(value)
        return value

    def _interpolation_replace(self, match):
        """ """
        s = match.group(1)
        if s is None:
            return match.group()
        else:
            # switch off interpolation before we try and fetch anything !
            self.main.interpolation = False
            # try the 'DEFAULT' member of *this section* first
            val = self.get('DEFAULT', {}).get(s)
            # try the 'DEFAULT' member of the *parent section* next
            if val is None:
                val = self.parent.get('DEFAULT', {}).get(s)
            # last, try the 'DEFAULT' member of the *main section*
            if val is None:
                val = self.main.get('DEFAULT', {}).get(s)
            self.main.interpolation = True
            if val is None:
                raise MissingInterpolationOption(s)
            return val

    def __getitem__(self, key):
        """Fetch the item and do string interpolation."""
        val = dict.__getitem__(self, key)
        if self.main.interpolation and isinstance(val, StringTypes):
            return self._interpolate(val)
        return val

    def __setitem__(self, key, value):
        """
        Correctly set a value.
        
        Making dictionary values Section instances.
        (We have to special case 'Section' instances - which are also dicts)
        
        Keys must be strings.
        Values need only be strings (or lists of strings) if
        ``main.stringify`` is set.
        """
        if not isinstance(key, StringTypes):
            raise ValueError, 'The key "%s" is not a string.' % key
        # add the comment
        if not self.comments.has_key(key):
            self.comments[key] = []
            self.inline_comments[key] = ''
        # remove the entry from defaults
        if key in self.defaults:
            self.defaults.remove(key)
        #
        if isinstance(value, Section):
            if not self.has_key(key):
                self.sections.append(key)
            dict.__setitem__(self, key, value)
        elif isinstance(value, dict):
            # First create the new depth level,
            # then create the section
            if not self.has_key(key):
                self.sections.append(key)
            new_depth = self.depth + 1
            dict.__setitem__(
                self,
                key,
                Section(
                    self,
                    new_depth,
                    self.main,
                    indict=value,
                    name=key))
        else:
            if not self.has_key(key):
                self.scalars.append(key)
            if not self.main.stringify:
                if isinstance(value, StringTypes):
                    pass
                elif isinstance(value, (list, tuple)):
                    for entry in value:
                        if not isinstance(entry, StringTypes):
                            raise TypeError, (
                                'Value is not a string "%s".' % entry)
                else:
                    raise TypeError, 'Value is not a string "%s".' % value
            dict.__setitem__(self, key, value)

    def __delitem__(self, key):
        """Remove items from the sequence when deleting."""
        dict. __delitem__(self, key)
        if key in self.scalars:
            self.scalars.remove(key)
        else:
            self.sections.remove(key)
        del self.comments[key]
        del self.inline_comments[key]

    def get(self, key, default=None):
        """A version of ``get`` that doesn't bypass string interpolation."""
        try:
            return self[key]
        except KeyError:
            return default

    def update(self, indict):
        """
        A version of update that uses our ``__setitem__``.
        """
        for entry in indict:
            self[entry] = indict[entry]


    def pop(self, key, *args):
        """ """
        val = dict.pop(self, key, *args)
        if key in self.scalars:
            del self.comments[key]
            del self.inline_comments[key]
            self.scalars.remove(key)
        elif key in self.sections:
            del self.comments[key]
            del self.inline_comments[key]
            self.sections.remove(key)
        if self.main.interpolation and isinstance(val, StringTypes):
            return self._interpolate(val)
        return val

    def popitem(self):
        """Pops the first (key,val)"""
        sequence = (self.scalars + self.sections)
        if not sequence:
            raise KeyError, ": 'popitem(): dictionary is empty'"
        key = sequence[0]
        val =  self[key]
        del self[key]
        return key, val

    def clear(self):
        """
        A version of clear that also affects scalars/sections
        Also clears comments and configspec.
        
        Leaves other attributes alone :
            depth/main/parent are not affected
        """
        dict.clear(self)
        self.scalars = []
        self.sections = []
        self.comments = {}
        self.inline_comments = {}
        self.configspec = {}

    def setdefault(self, key, default=None):
        """A version of setdefault that sets sequence if appropriate."""
        try:
            return self[key]
        except KeyError:
            self[key] = default
            return self[key]

    def items(self):
        """ """
        return zip((self.scalars + self.sections), self.values())

    def keys(self):
        """ """
        return (self.scalars + self.sections)

    def values(self):
        """ """
        return [self[key] for key in (self.scalars + self.sections)]

    def iteritems(self):
        """ """
        return iter(self.items())

    def iterkeys(self):
        """ """
        return iter((self.scalars + self.sections))

    __iter__ = iterkeys

    def itervalues(self):
        """ """
        return iter(self.values())

    def __repr__(self):
        return '{%s}' % ', '.join([('%s: %s' % (repr(key), repr(self[key])))
            for key in (self.scalars + self.sections)])

    __str__ = __repr__

    # Extra methods - not in a normal dictionary

    def dict(self):
        """
        Return a deepcopy of self as a dictionary.
        
        All members that are ``Section`` instances are recursively turned to
        ordinary dictionaries - by calling their ``dict`` method.
        
        >>> n = a.dict()
        >>> n == a
        1
        >>> n is a
        0
        """
        newdict = {}
        for entry in self:
            this_entry = self[entry]
            if isinstance(this_entry, Section):
                this_entry = this_entry.dict()
            elif isinstance(this_entry, (list, tuple)):
                # create a copy rather than a reference
                this_entry = list(this_entry)
            newdict[entry] = this_entry
        return newdict

    def merge(self, indict):
        """
        A recursive update - useful for merging config files.
        
        >>> a = '''[section1]
        ...     option1 = True
        ...     [[subsection]]
        ...     more_options = False
        ...     # end of file'''.splitlines()
        >>> b = '''# File is user.ini
        ...     [section1]
        ...     option1 = False
        ...     # end of file'''.splitlines()
        >>> c1 = ConfigObj(b)
        >>> c2 = ConfigObj(a)
        >>> c2.merge(c1)
        >>> c2
        {'section1': {'option1': 'False', 'subsection': {'more_options': 'False'}}}
        """
        for key, val in indict.items():
            if (key in self and isinstance(self[key], dict) and
                                isinstance(val, dict)):
                self[key].merge(val)
            else:   
                self[key] = val

    def rename(self, oldkey, newkey):
        """
        Change a keyname to another, without changing position in sequence.
        
        Implemented so that transformations can be made on keys,
        as well as on values. (used by encode and decode)
        
        Also renames comments.
        """
        if oldkey in self.scalars:
            the_list = self.scalars
        elif oldkey in self.sections:
            the_list = self.sections
        else:
            raise KeyError, 'Key "%s" not found.' % oldkey
        pos = the_list.index(oldkey)
        #
        val = self[oldkey]
        dict.__delitem__(self, oldkey)
        dict.__setitem__(self, newkey, val)
        the_list.remove(oldkey)
        the_list.insert(pos, newkey)
        comm = self.comments[oldkey]
        inline_comment = self.inline_comments[oldkey]
        del self.comments[oldkey]
        del self.inline_comments[oldkey]
        self.comments[newkey] = comm
        self.inline_comments[newkey] = inline_comment

    def walk(self, function, raise_errors=True,
            call_on_sections=False, **keywargs):
        """
        Walk every member and call a function on the keyword and value.
        
        Return a dictionary of the return values
        
        If the function raises an exception, raise the errror
        unless ``raise_errors=False``, in which case set the return value to
        ``False``.
        
        Any unrecognised keyword arguments you pass to walk, will be pased on
        to the function you pass in.
        
        Note: if ``call_on_sections`` is ``True`` then - on encountering a
        subsection, *first* the function is called for the *whole* subsection,
        and then recurses into it's members. This means your function must be
        able to handle strings, dictionaries and lists. This allows you
        to change the key of subsections as well as for ordinary members. The
        return value when called on the whole subsection has to be discarded.
        
        See  the encode and decode methods for examples, including functions.
        
        .. caution::
        
            You can use ``walk`` to transform the names of members of a section
            but you mustn't add or delete members.
        
        >>> config = '''[XXXXsection]
        ... XXXXkey = XXXXvalue'''.splitlines()
        >>> cfg = ConfigObj(config)
        >>> cfg
        {'XXXXsection': {'XXXXkey': 'XXXXvalue'}}
        >>> def transform(section, key):
        ...     val = section[key]
        ...     newkey = key.replace('XXXX', 'CLIENT1')
        ...     section.rename(key, newkey)
        ...     if isinstance(val, (tuple, list, dict)):
        ...         pass
        ...     else:
        ...         val = val.replace('XXXX', 'CLIENT1')
        ...         section[newkey] = val
        >>> cfg.walk(transform, call_on_sections=True)
        {'CLIENT1section': {'CLIENT1key': None}}
        >>> cfg
        {'CLIENT1section': {'CLIENT1key': 'CLIENT1value'}}
        """
        out = {}
        # scalars first
        for i in range(len(self.scalars)):
            entry = self.scalars[i]
            try:
                val = function(self, entry, **keywargs)
                # bound again in case name has changed
                entry = self.scalars[i]
                out[entry] = val
            except Exception:
                if raise_errors:
                    raise
                else:
                    entry = self.scalars[i]
                    out[entry] = False
        # then sections
        for i in range(len(self.sections)):
            entry = self.sections[i]
            if call_on_sections:
                try:
                    function(self, entry, **keywargs)
                except Exception:
                    if raise_errors:
                        raise
                    else:
                        entry = self.sections[i]
                        out[entry] = False
                # bound again in case name has changed
                entry = self.sections[i]
            # previous result is discarded
            out[entry] = self[entry].walk(
                function,
                raise_errors=raise_errors,
                call_on_sections=call_on_sections,
                **keywargs)
        return out

    def decode(self, encoding):
        """
        Decode all strings and values to unicode, using the specified encoding.
        
        Works with subsections and list values.
        
        Uses the ``walk`` method.
        
        Testing ``encode`` and ``decode``.
        >>> m = ConfigObj(a)
        >>> m.decode('ascii')
        >>> def testuni(val):
        ...     for entry in val:
        ...         if not isinstance(entry, unicode):
        ...             print >> sys.stderr, type(entry)
        ...             raise AssertionError, 'decode failed.'
        ...         if isinstance(val[entry], dict):
        ...             testuni(val[entry])
        ...         elif not isinstance(val[entry], unicode):
        ...             raise AssertionError, 'decode failed.'
        >>> testuni(m)
        >>> m.encode('ascii')
        >>> a == m
        1
        """
        def decode(section, key, encoding=encoding):
            """ """
            val = section[key]
            if isinstance(val, (list, tuple)):
                newval = []
                for entry in val:
                    newval.append(entry.decode(encoding))
            elif isinstance(val, dict):
                newval = val
            else:
                newval = val.decode(encoding)
            newkey = key.decode(encoding)
            section.rename(key, newkey)
            section[newkey] = newval
        # using ``call_on_sections`` allows us to modify section names
        self.walk(decode, call_on_sections=True)

    def encode(self, encoding):
        """
        Encode all strings and values from unicode,
        using the specified encoding.
        
        Works with subsections and list values.
        Uses the ``walk`` method.
        """
        def encode(section, key, encoding=encoding):
            """ """
            val = section[key]
            if isinstance(val, (list, tuple)):
                newval = []
                for entry in val:
                    newval.append(entry.encode(encoding))
            elif isinstance(val, dict):
                newval = val
            else:
                newval = val.encode(encoding)
            newkey = key.encode(encoding)
            section.rename(key, newkey)
            section[newkey] = newval
        self.walk(encode, call_on_sections=True)

    def istrue(self, key):
        """
        Accepts a key as input. The corresponding value must be a string or
        the objects (``True`` or 1) or (``False`` or 0). We allow 0 and 1 to
        retain compatibility with Python 2.2.
        
        If the string is one of  ``True``, ``On``, ``Yes``, or ``1`` it returns 
        ``True``.
        
        If the string is one of  ``False``, ``Off``, ``No``, or ``0`` it returns 
        ``False``.
        
        ``istrue`` is not case sensitive.
        
        Any other input will raise a ``ValueError``. 
        """
        val = self[key]
        if val == True:
            return True
        elif val == False:
            return False
        else:
            try:
                if not isinstance(val, StringTypes):
                    raise KeyError
                else:
                    return self.main._bools[val.lower()]
            except KeyError:
                raise ValueError('Value "%s" is neither True nor False' % val)

class ConfigObj(Section):
    """
    An object to read, create, and write config files.
    
    Testing with duplicate keys and sections.
    
    >>> c = '''
    ... [hello]
    ... member = value
    ... [hello again]
    ... member = value
    ... [ "hello" ]
    ... member = value
    ... '''
    >>> ConfigObj(c.split('\\n'), raise_errors = True)
    Traceback (most recent call last):
    DuplicateError: Duplicate section name at line 5.
    
    >>> d = '''
    ... [hello]
    ... member = value
    ... [hello again]
    ... member1 = value
    ... member2 = value
    ... 'member1' = value
    ... [ "and again" ]
    ... member = value
    ... '''
    >>> ConfigObj(d.split('\\n'), raise_errors = True)
    Traceback (most recent call last):
    DuplicateError: Duplicate keyword name at line 6.
    """

    _keyword = re.compile(r'''^ # line start
        (\s*)                   # indentation
        (                       # keyword
            (?:".*?")|          # double quotes
            (?:'.*?')|          # single quotes
            (?:[^'"=].*?)       # no quotes
        )
        \s*=\s*                 # divider
        (.*)                    # value (including list values and comments)
        $   # line end
        ''',
        re.VERBOSE)

    _sectionmarker = re.compile(r'''^
        (\s*)                     # 1: indentation
        ((?:\[\s*)+)              # 2: section marker open
        (                         # 3: section name open
            (?:"\s*\S.*?\s*")|    # at least one non-space with double quotes
            (?:'\s*\S.*?\s*')|    # at least one non-space with single quotes
            (?:[^'"\s].*?)        # at least one non-space unquoted
        )                         # section name close
        ((?:\s*\])+)              # 4: section marker close
        \s*(\#.*)?                # 5: optional comment
        $''',
        re.VERBOSE)

    # this regexp pulls list values out as a single string
    # or single values and comments
    _valueexp = re.compile(r'''^
        (?:
            (?:
                (
                    (?:
                        (?:
                            (?:".*?")|              # double quotes
                            (?:'.*?')|              # single quotes
                            (?:[^'",\#][^,\#]*?)       # unquoted
                        )
                        \s*,\s*                     # comma
                    )*      # match all list items ending in a comma (if any)
                )
                (
                    (?:".*?")|                      # double quotes
                    (?:'.*?')|                      # single quotes
                    (?:[^'",\#\s][^,]*?)             # unquoted
                )?          # last item in a list - or string value
            )|
            (,)             # alternatively a single comma - empty list
        )
        \s*(\#.*)?          # optional comment
        $''',
        re.VERBOSE)

    # use findall to get the members of a list value
    _listvalueexp = re.compile(r'''
        (
            (?:".*?")|          # double quotes
            (?:'.*?')|          # single quotes
            (?:[^'",\#].*?)       # unquoted
        )
        \s*,\s*                 # comma
        ''',
        re.VERBOSE)

    # this regexp is used for the value
    # when lists are switched off
    _nolistvalue = re.compile(r'''^
        (
            (?:".*?")|          # double quotes
            (?:'.*?')|          # single quotes
            (?:[^'"\#].*?)      # unquoted
        )
        \s*(\#.*)?              # optional comment
        $''',
        re.VERBOSE)

    # regexes for finding triple quoted values on one line
    _single_line_single = re.compile(r"^'''(.*?)'''\s*(#.*)?$")
    _single_line_double = re.compile(r'^"""(.*?)"""\s*(#.*)?$')
    _multi_line_single = re.compile(r"^(.*?)'''\s*(#.*)?$")
    _multi_line_double = re.compile(r'^(.*?)"""\s*(#.*)?$')

    _triple_quote = {
        "'''": (_single_line_single, _multi_line_single),
        '"""': (_single_line_double, _multi_line_double),
    }

    # Used by the ``istrue`` Section method
    _bools = {
        'yes': True, 'no': False,
        'on': True, 'off': False,
        '1': True, '0': False,
        'true': True, 'false': False,
        }

    def __init__(self, infile=None, options=None, **kwargs):
        """
        Parse or create a config file object.
        
        ``ConfigObj(infile=None, options=None, **kwargs)``
        """
        if infile is None:
            infile = []
        if options is None:
            options = {}
        # keyword arguments take precedence over an options dictionary
        options.update(kwargs)
        # init the superclass
        Section.__init__(self, self, 0, self)
        #
        defaults = OPTION_DEFAULTS.copy()
        for entry in options.keys():
            if entry not in defaults.keys():
                raise TypeError, 'Unrecognised option "%s".' % entry
        # TODO: check the values too
        # add the explicit options to the defaults
        defaults.update(options)
        #
        # initialise a few variables
        self._errors = []
        self.raise_errors = defaults['raise_errors']
        self.interpolation = defaults['interpolation']
        self.list_values = defaults['list_values']
        self.create_empty = defaults['create_empty']
        self.file_error = defaults['file_error']
        self.stringify = defaults['stringify']
        self.indent_type = defaults['indent_type']
        # used by the write method
        self.BOM = None
        #
        self.initial_comment = []
        self.final_comment = []
        #
        if isinstance(infile, StringTypes):
            self.filename = os.path.abspath(infile)
            if os.path.isfile(self.filename):
                infile = open(self.filename).readlines()
            elif self.file_error:
                # raise an error if the file doesn't exist
                raise IOError, 'Config file not found: "%s".' % self.filename
            else:
                # file doesn't already exist
                if self.create_empty:
                    # this is a good test that the filename specified
                    # isn't impossible - like on a non existent device
                    h = open(self.filename, 'w')
                    h.write('')
                    h.close()
                infile = []
        elif isinstance(infile, (list, tuple)):
            self.filename = None
        elif isinstance(infile, dict):
            # initialise self
            # the Section class handles creating subsections
            if isinstance(infile, ConfigObj):
                # get a copy of our ConfigObj
                infile = infile.dict()
            for entry in infile:
                self[entry] = infile[entry]
            self.filename = None
            del self._errors
            if defaults['configspec'] is not None:
                self._handle_configspec(defaults['configspec'])
            else:
                self.configspec = None
            return
        elif hasattr(infile, 'seek'):
            # this supports StringIO instances and even file objects
            self.filename = infile
            infile.seek(0)
            infile = infile.readlines()
            self.filename.seek(0)
        else:
            raise TypeError, ('infile must be a filename,'
                ' StringIO instance, or a file as a list.')
        #
        # strip trailing '\n' from lines
        infile = [line.rstrip('\n') for line in infile]
        #
        # remove the UTF8 BOM if it is there
        # FIXME: support other BOM
        if infile and infile[0].startswith(BOM_UTF8):
            infile[0] = infile[0][3:]
            self.BOM = BOM_UTF8
        else:
            self.BOM = None
        #
        self._parse(infile)
        # if we had ...
 
[truncated message content]