#550 guidance for referencing anchors (broken after 0.18.15)

[xdu]

hi!

I need to create a yaml file from a script using ruamel and I'd love to know what's the best way to create anchor references. the goal is to reduce duplication to make the generated yaml file more human-user friendly.

the implementation (slightly simplified for this post) I've been using is:

default_metadata = CommentedMap()
default_metadata["foo"] = "bar"
default_metadata.yaml_set_anchor("default_metadata", always_dump=True)

config_root["default"] = CommentedMap()
config_root["default"]["metadata"] = default_metadata

instances = []
for i in some_range:
    m = CommentedMap()
    m['name'] = i
    metadata = CommentedMap()
    setattr(metadata, ruamel.yaml.comments.merge_attrib, [(0, default_metadata)])
    m["metadata"] = metadata
    instances.append(m)

config_root['instances']=instances

yaml = YAML()
yaml.indent(mapping=2, sequence=2, offset=0)
yaml.default_flow_style = False
yaml.preserve_quotes = True
yaml.dump(config_root, file)

where the expected yaml file would look like:

default:
  metadata: &default_metadata
    foo: bar
instances:

- name: name0
  metadata:
    <<: *default_metadata
- name: name1
  metadata:
    <<: *default_metadata
...

in version 0.18.14 this implementation worked. attempted a few other options (update, merge_attrib, insert) and << was considered as a special key and got quoted "<<": *default_metadata or '<<': *default_metadata), regardless of the value of yaml.preserve_quotes

when upgrading ruamel to 0.18.15+ (tested both 0.18.15 and 0.18.16), this implementation stopped working with the following error:

  File "myproj/./myscript.py", line 317, in write_file_contents
    yaml.dump(config_root, file)
  File "myproj/.venv/lib/python3.10/site-packages/ruamel/yaml/main.py", line 597, in dump
    return self.dump_all([data], stream, transform=transform)
  File "myproj/.venv/lib/python3.10/site-packages/ruamel/yaml/main.py", line 607, in dump_all
    self._context_manager.dump(data)
  File "myproj/.venv/lib/python3.10/site-packages/ruamel/yaml/main.py", line 956, in dump
    self._yaml.representer.represent(data)
  File "myproj/.venv/lib/python3.10/site-packages/ruamel/yaml/representer.py", line 83, in represent
    node = self.represent_data(data)
  File "myproj/.venv/lib/python3.10/site-packages/ruamel/yaml/representer.py", line 105, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "myproj/.venv/lib/python3.10/site-packages/ruamel/yaml/representer.py", line 1016, in represent_dict
    return self.represent_mapping(tag, data)
  File "myproj/.venv/lib/python3.10/site-packages/ruamel/yaml/representer.py", line 846, in represent_mapping
    node_value = self.represent_data(item_value)
  File "myproj/.venv/lib/python3.10/site-packages/ruamel/yaml/representer.py", line 105, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "myproj/.venv/lib/python3.10/site-packages/ruamel/yaml/representer.py", line 309, in represent_list
    return self.represent_sequence('tag:yaml.org,2002:seq', data)
  File "myproj/.venv/lib/python3.10/site-packages/ruamel/yaml/representer.py", line 750, in represent_sequence
    node_item = self.represent_data(item)
  File "myproj/.venv/lib/python3.10/site-packages/ruamel/yaml/representer.py", line 105, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "myproj/.venv/lib/python3.10/site-packages/ruamel/yaml/representer.py", line 1016, in represent_dict
    return self.represent_mapping(tag, data)
  File "myproj/.venv/lib/python3.10/site-packages/ruamel/yaml/representer.py", line 846, in represent_mapping
    node_value = self.represent_data(item_value)
  File "myproj/.venv/lib/python3.10/site-packages/ruamel/yaml/representer.py", line 105, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "myproj/.venv/lib/python3.10/site-packages/ruamel/yaml/representer.py", line 1016, in represent_dict
    return self.represent_mapping(tag, data)
  File "myproj/.venv/lib/python3.10/site-packages/ruamel/yaml/representer.py", line 873, in represent_mapping
    if merge_value.sequence is None:  # type: ignore
AttributeError: 'list' object has no attribute 'sequence'

I've tried a few different things based on research results but couldn't find a good fix.

would love to ask 2 questions:

1. with the goal and the expected yaml in mind, what would be the best way to implement my script? I'd love to follow the guidance on best practices (lm if the anchor references are expected to be deprecated)
2. if the above example is close to the recommended implementation, what are the necessary changes that need to be applied in latest version of ruamel to get it working? I can live with a version pin for now but I'd like to know the effort to enable version upgrade in the future.

Thank you in advance!!

[Anthon van der Neut]

I cannot run your code as it is incomplete, so what is below is only partly tested. And this is probably better asked on StackOverflow (it worked before, but you are using internals. IMO it is not a bug per se), more people will see this.

1: load the expected output and dump it to make sure the output can (still) be generated from ruamel.yaml

import sys
import ruamel.yaml

yaml_str = """\
default:
  metadata: &default_metadata
    foo: bar
instances:

- name: name0
  metadata:
    <<: *default_metadata
- name: name1
  metadata:
    <<: *default_metadata
"""

yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True

data = yaml.load(yaml_str)
yaml.dump(data, sys.stdout)

2: the above nicely roundtrips , so now inspect the merge attribute

mv = getattr(data['instances'][0]['metadata'], ruamel.yaml.comments.merge_attrib)
print(type(mv))

which will give you the output <class 'ruamel.yaml.mergevalue.MergeValue'>
so what happened is that the old usage of a list of dictionaries to merge you now need an instance of MergeValue e.g. mv = MergeValue() and then use mv.append(default_metadata) to get the one dict reference

3 set mv as the attribute instead of the [(0, default_metadata)] list

[xdu]

thank you for the quick response!

following your suggestion I debugged locally a bit. the solution is:

mv = MergeValue()
mv.append(default_metadata)
mv.merge_pos = 0
setattr(metadata, ruamel.yaml.comments.merge_attrib, mv)

merge_pos is optional and default to None: makes sense. Should it be set to 0 (and keep an internal counter to track the right value) when append is called?

I'd also love to know if this implementation is following best practices. would love to get the guidance