#453 CommentMap.insert causes merged keys to be duplicated when dumping

[Alex Miller]

Inserting into a map which merges from another map causes all of the merged keys to be duplicated. I'm using ruamel.yaml 0.17.21. I've worked out a minimal repro for this:

from ruamel.yaml import YAML
import sys

yaml = YAML()

yamldoc = yaml.load("""
to-merge: &anchor
  merge-key: should not be duplicated

usage:
  <<: *anchor
  usage-key: usage-value
""")

yamldoc['usage'].insert(0, 'insert-key', 'insert-value')

yaml.dump(yamldoc, sys.stdout)

Shows:

to-merge: &anchor
  merge-key: should not be duplicated

usage:
  <<: *anchor
  insert-key: insert-value
  usage-key: usage-value
  merge-key: should not be duplicated

I think this is because CommentedMap.insert is a bit too conservative? Annotations added as #! comments:

    def insert(self, pos, key, value, comment=None):
        # type: (Any, Any, Any, Optional[Any]) -> None
        """insert key value into given position
        attach comment if provided
        """
        #! self.keys() is all keys, those which are owned keys or not
        keys = list(self.keys()) + [key]
        ordereddict.insert(self, pos, key, value)
        for keytmp in keys:
            #! And this line then takes ownership of all non-owned keys
            self._ok.add(keytmp)
        # ...

Surrounding the insert() call with manual fixing of _ok leads to a correct output:

ok = yamldoc['usage']._ok.copy()
ok.add('insert-key')
yamldoc['usage'].insert(0, 'insert-key', 'insert-value')
print(yamldoc['usage']._ok)
yamldoc['usage']._ok = ok
print(ok)

with the prints yielding:

{'merge-key', 'usage-key', 'insert-key'}
{'usage-key', 'insert-key'}

[Anthon van der Neut]

The actual error was in the insert routine grafted onto Collections.OrderedDict (in compat.py). That did not look at merge information copying all keys as you would see by iterating over the yamldoc['usage']. It would also move any merge key to the first key position. I changed this to no longer include the unwanted keys and .insert(0, ....) resulting in a key before the << key. This will be in 0.17.24.

Thanks for reporting

[Anthon van der Neut]

The actual error was in the insert routine grafted onto Collections.OrderedDict (in compat.py). That did not look at merge information copying all keys as you would see by iterating over the yamldoc['usage']. It would also move any merge key to the first key position. I changed this to no longer include the unwanted keys and .insert(0, ....) resulting in a key before the << key. This will be in 0.17.24.

Thanks for reporting