Re: [Configobj-develop] String Substitution

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Robin Munn wrote:
> Here's the patch. This one implements a *proper* recursive,
> lazy-loading interpolation approach, and no longer suffers from the
> "stop when a $$ is seen" problem. If anything stops interpolation of
> one key, the rest of the keys in the string will not have their
> interpolation stopped. Thus, "$foo + $bar + $baz" will follow "$baz"
> all the way down the interpolation tree even if "$foo" immediately
> evaluates to "$$99.95" (which becomes "$99.95" and stops interpolation
> of "$foo").
>

Thanks for this Robin.

I'm going to have to take some time to understand this and work out what 
is really going on. :-)

It sounds good though, appreciated.

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml
> Infinite-recursion loops will be detected as soon as any value is
> entered for the second time, and an exception will be thrown at that
> point. This has the added advantage of making the MAX_INTERPOL_DEPTH
> value completely unnecessary -- with my patch applied,
> MAX_INTERPOL_DEPTH can still be set, but it has no effect whatsoever.
> The recursion will continue down as deep as necessary as long as there
> are no loops.
>
> I've tested this code against all the scenarios I could come up with,
> and it's worked properly every time.
>
> This version of the patch makes no change to the "which section to
> look at" behavior: it only looks at sections named DEFAULT. Next up: a
> patch that changes this lookup order to the following:
>
> 1) Current section
> 2) Current section's DEFAULT
> 3) current = current.parent; repeat from 1
>
> ------------------------------------------------------------------------
>
> === docs/configobj.txt
> ==================================================================
> --- docs/configobj.txt	(revision 20140)
> +++ docs/configobj.txt	(local)
> @@ -786,13 +786,13 @@
>  * create_empty
>  * file_error
>  
> -interpolate
> -~~~~~~~~~~~
> +interpolation
> +~~~~~~~~~~~~~
>  
>  ConfigObj can perform string interpolation in a *similar* way to
>  ``ConfigParser``. See the interpolation_ section for full details.
>  
> -If ``interpolate`` is set to ``False``, then interpolation is *not* done when
> +If ``interpolation`` is set to ``False``, then interpolation is *not* done when
>  you fetch values.
>  
>  stringify
> @@ -1957,9 +1957,29 @@
>  =============
>  
>  ConfigObj allows string interpolation *similar* to the way ``ConfigParser``
> +or ``string.Template`` work. The value of the ``interpolation`` attribute
> +determines which style of interpolation you want to use. Valid values are
> +"ConfigParser" or "Template" (case-insensitive, so "configparser" and
> +"template" will also work). For backwards compatibility reasons, the value
> +``True`` is also a valid value for the ``interpolation`` attribute, and
> +will select ``ConfigParser``-style interpolation. At some undetermined point
> +in the future, that default *may* change to ``Template``-style interpolation.
>  
> -You specify a value to be substituted by including ``%(name)s`` in the value.
> +For ``ConfigParser``-style interpolation, you specify a value to be
> +substituted by including ``%(name)s`` in the value.
>  
> +For ``Template``-style interpolation, you specify a value to be substituted
> +by including ``${name}`` in the value. Alternately, if 'name' is a valid
> +Python identifier (i.e., is composed of nothing but alphanumeric characters,
> +plus the underscore character), then the braces are optional and the value
> +can be written as ``$name``.
> +
> +Note that ``ConfigParser``-style interpolation and ``Template``-style
> +interpolation are mutually exclusive; you cannot have a configuration file
> +that's a mix of one or the other. Pick one and stick to it. ``Template``-style
> +interpolation is simpler to read and write by hand, and is recommended if
> +you don't have a particular reason to use ``ConfigParser``-style.
> +
>  Interpolation checks first the 'DEFAULT' sub-section of the current section to
>  see if ``name`` is the key to a value. ('name' is case sensitive).
>  
> === pythonutils/configobj.py
> ==================================================================
> --- pythonutils/configobj.py	(revision 20140)
> +++ pythonutils/configobj.py	(local)
> @@ -102,6 +102,7 @@
>  __all__ = (
>      '__version__',
>      'DEFAULT_INDENT_TYPE',
> +    'DEFAULT_INTERPOLATION',
>      'NUM_INDENT_SPACES',
>      'MAX_INTERPOL_DEPTH',
>      'ConfigObjError',
> @@ -112,7 +113,7 @@
>      'ConfigObj',
>      'SimpleVal',
>      'InterpolationError',
> -    'InterpolationDepthError',
> +    'InterpolationLoopError',
>      'MissingInterpolationOption',
>      'RepeatSectionError',
>      'UnreprError',
> @@ -121,6 +122,7 @@
>      'flatten_errors',
>  )
>  
> +DEFAULT_INTERPOLATION = 'configparser'
>  DEFAULT_INDENT_TYPE = ' '
>  NUM_INDENT_SPACES = 4
>  MAX_INTERPOL_DEPTH = 10
> @@ -250,13 +252,13 @@
>  class InterpolationError(ConfigObjError):
>      """Base class for the two interpolation errors."""
>  
> -class InterpolationDepthError(InterpolationError):
> +class InterpolationLoopError(InterpolationError):
>      """Maximum interpolation depth exceeded in string interpolation."""
>  
>      def __init__(self, option):
>          InterpolationError.__init__(
>              self,
> -            'max interpolation depth exceeded in value "%s".' % option)
> +            'interpolation loop detected in value "%s".' % option)
>  
>  class RepeatSectionError(ConfigObjError):
>      """
> @@ -276,11 +278,157 @@
>      """An error parsing in unrepr mode."""
>  
>  
> +class InterpolationEngine(object):
> +    """
> +    A helper class to help perform string interpolation.
> +
> +    This class is an abstract base class; its descendants perform
> +    the actual work.
> +    """
> +
> +    # compiled regexp to use in self.interpolate()
> +    _KEYCRE = re.compile(r"%\(([^)]*)\)s")
> +
> +    def __init__(self, section):
> +        # the Section instance that "owns" this engine
> +        self.section = section
> +
> +    def interpolate(self, key, value):
> +        def recursive_interpolate(key, value, section, backtrail):
> +            """The function that does the actual work.
> +
> +            ``value``: the string we're trying to interpolate.
> +            ``section``: the section in which that string was found
> +            ``backtrail``: a dict to keep track of where we've been,
> +            to detect and prevent infinite recursion loops
> +
> +            This is similar to a depth-first-search algorithm.
> +            """
> +            # Have we been here already?
> +            if backtrail.has_key((key, section.name)):
> +                # Yes - infinite loop detected
> +                raise InterpolationLoopError(key)
> +            # Place a marker on our backtrail so we won't come back here again
> +            backtrail[(key, section.name)] = 1
> +
> +            # Now start the actual work
> +            match = self._KEYCRE.search(value)
> +            while match:
> +                # The actual parsing of the match is implementation-dependent,
> +                # so delegate to our helper function
> +                k, v, s = self._parse_match(match)
> +                if k is None:
> +                    # That's the signal that no further interpolation is needed
> +                    replacement = v
> +                else:
> +                    # Further interpolation may be needed to obtain final value
> +                    replacement = recursive_interpolate(k, v, s, backtrail)
> +                # Replace the matched string with its final value
> +                start, end = match.span()
> +                value = ''.join((value[:start], replacement, value[end:]))
> +                new_search_start = start + len(replacement)
> +                # Pick up the next interpolation key, if any, for next time
> +                # through the while loop
> +                match = self._KEYCRE.search(value, new_search_start)
> +
> +            # Now safe to come back here again; remove marker from backtrail
> +            del backtrail[(key, section.name)]
> +
> +            return value
> +
> +        # Back in interpolate(), all we have to do is kick off the recursive
> +        # function with appropriate starting values
> +        value = recursive_interpolate(key, value, self.section, {})
> +        return value
> +
> +    def _fetch(self, key):
> +        """Helper function to fetch values from owning section.
> +
> +        Returns a 2-tuple: the value, and the section where it was found.
> +        """
> +        # switch off interpolation before we try and fetch anything !
> +        save_interp = self.section.main.interpolation
> +        self.section.main.interpolation = False
> +        # try the 'DEFAULT' member of owning section first
> +        current_section = self.section
> +        val = current_section.get('DEFAULT', {}).get(key)
> +        # try the 'DEFAULT' member of owner's parent section next
> +        if val is None:
> +            current_section = self.section.parent
> +            val = current_section.get('DEFAULT', {}).get(key)
> +        # last, try the 'DEFAULT' member of the main section
> +        if val is None:
> +            current_section = self.section.main
> +            val = current_section.get('DEFAULT', {}).get(key)
> +        # restore interpolation to previous value before returning
> +        self.section.main.interpolation = save_interp
> +        if val is None:
> +            raise MissingInterpolationOption(key)
> +        return val, current_section
> +
> +    def _parse_match(self, match):
> +        """Implementation-dependent helper function.
> +
> +        Will be passed a match object corresponding to the interpolation
> +        key we just found (e.g., "%(foo)s" or "$foo"). Should look up that
> +        key in the appropriate config file section (using the ``_fetch()``
> +        helper function) and return a 3-tuple: (key, value, section)
> +
> +        ``key`` is the name of the key we're looking for
> +        ``value`` is the value found for that key
> +        ``section`` is a reference to the section where it was found
> +
> +        ``key`` and ``section`` should be None if no further
> +        interpolation should be performed on the resulting value
> +        (e.g., if we interpolated "$$" and returned "$").
> +        """
> +        raise NotImplementedError
> +    
> +
> +class ConfigParserInterpolation(InterpolationEngine):
> +    """Behaves like ConfigParser."""
> +    _KEYCRE = re.compile(r"%\(([^)]*)\)s")
> +
> +    def _parse_match(self, match):
> +        key = match.group(1)
> +        value, section = self._fetch(key)
> +        return key, value, section
> +
> +
> +class TemplateInterpolation(InterpolationEngine):
> +    """Behaves like string.Template."""
> +    _delimiter = '$'
> +    _KEYCRE = re.compile(r"""
> +        \$(?:
> +          (?P<escaped>\$)              |   # Two $ signs
> +          (?P<named>[_a-z][_a-z0-9]*)  |   # $name format
> +          {(?P<braced>[^}]*)}              # ${name} format
> +        )
> +        """, re.IGNORECASE | re.VERBOSE)
> +
> +    def _parse_match(self, match):
> +        # Valid name (in or out of braces): fetch value from section
> +        key = match.group('named') or match.group('braced')
> +        if key is not None:
> +            value, section = self._fetch(key)
> +            return key, value, section
> +        # Escaped delimiter (e.g., $$): return single delimiter
> +        if match.group('escaped') is not None:
> +            # Return None for key and section to indicate it's time to stop
> +            return None, self._delimiter, None
> +        # Anything else: ignore completely, just return it unchanged
> +        return None, match.group(), None
> +
> +interpolation_engines = {
> +    'configparser': ConfigParserInterpolation,
> +    'template': TemplateInterpolation,
> +}
> +
>  class Section(dict):
>      """
>      A dictionary-like object that represents a section in a config file.
>      
> -    It does string interpolation if the 'interpolate' attribute
> +    It does string interpolation if the 'interpolation' attribute
>      of the 'main' object is set to True.
>      
>      Interpolation is tried first from the 'DEFAULT' section of this object,
> @@ -293,8 +441,6 @@
>      Iteration follows the order: scalars, then sections.
>      """
>  
> -    _KEYCRE = re.compile(r"%\(([^)]*)\)s|.")
> -
>      def __init__(self, parent, depth, main, indict=None, name=None):
>          """
>          * parent is the section above
> @@ -335,46 +481,33 @@
>          for entry in indict:
>              self[entry] = indict[entry]
>  
> -    def _interpolate(self, value):
> -        """Nicked from ConfigParser."""
> -        depth = MAX_INTERPOL_DEPTH
> -        # loop through this until it's done
> -        while depth:
> -            depth -= 1
> -            if value.find("%(") != -1:
> -                value = self._KEYCRE.sub(self._interpolation_replace, value)
> +    def _interpolate(self, key, value):
> +        try:
> +            # do we already have an interpolation engine?
> +            engine = self._interpolation_engine
> +        except AttributeError:
> +            # not yet: first time running _interpolate(), so pick the engine
> +            name = self.main.interpolation
> +            if name == True:  # note that "if name:" would be incorrect here
> +                # backwards-compatibility: interpolation=True means use default
> +                name = DEFAULT_INTERPOLATION
> +            name = name.lower()  # so that "Template", "template", etc. all work
> +            klass = interpolation_engines.get(name, None)
> +            if klass is None:
> +                # invalid value for self.main.interpolation
> +                self.main.interpolation = False
> +                return value
>              else:
> -                break
> -        else:
> -            raise InterpolationDepthError(value)
> -        return value
> +                # save reference to engine so we don't have to do this again
> +                engine = self._interpolation_engine = klass(self)
> +        # let the engine do the actual work
> +        return engine.interpolate(key, value)
>  
> -    def _interpolation_replace(self, match):
> -        """ """
> -        s = match.group(1)
> -        if s is None:
> -            return match.group()
> -        else:
> -            # switch off interpolation before we try and fetch anything !
> -            self.main.interpolation = False
> -            # try the 'DEFAULT' member of *this section* first
> -            val = self.get('DEFAULT', {}).get(s)
> -            # try the 'DEFAULT' member of the *parent section* next
> -            if val is None:
> -                val = self.parent.get('DEFAULT', {}).get(s)
> -            # last, try the 'DEFAULT' member of the *main section*
> -            if val is None:
> -                val = self.main.get('DEFAULT', {}).get(s)
> -            self.main.interpolation = True
> -            if val is None:
> -                raise MissingInterpolationOption(s)
> -            return val
> -
>      def __getitem__(self, key):
>          """Fetch the item and do string interpolation."""
>          val = dict.__getitem__(self, key)
>          if self.main.interpolation and isinstance(val, StringTypes):
> -            return self._interpolate(val)
> +            return self._interpolate(key, val)
>          return val
>  
>      def __setitem__(self, key, value, unrepr=False):
> @@ -471,7 +604,7 @@
>              del self.inline_comments[key]
>              self.sections.remove(key)
>          if self.main.interpolation and isinstance(val, StringTypes):
> -            return self._interpolate(val)
> +            return self._interpolate(key, val)
>          return val
>  
>      def popitem(self):
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> Configobj-develop mailing list
> Con...@li...
> https://lists.sourceforge.net/lists/listinfo/configobj-develop
>   