#464 Parser error when using explicit end markers

[Mariusz Rusiniak]

Hi,

Firsdt of all I'd like to thank you for maintaining this project, you are doing a great job here!

I found a bug in the latest release of ruamel.yaml==0.17.27. Minimum reproducible example below:

import ruamel.yaml

a = """---
a: True
..."""
 ruamel.yaml.load(a)

It seems that starting from this version parser has issues with analysis of the explicit end markers:

Traceback:

ParserError                               Traceback (most recent call last)
Cell In[3], line 1
----> 1 ruamel.yaml.load(a)

File ~/github/tempo-syslib/venv/lib/python3.9/site-packages/ruamel/yaml/main.py:1063, in load(stream, Loader, version, preserve_quotes)
   1061 loader = Loader(stream, version, preserve_quotes=preserve_quotes)  # type: Any
   1062 try:
-> 1063     return loader._constructor.get_single_data()
   1064 finally:
   1065     loader._parser.dispose()

File ~/github/tempo-syslib/venv/lib/python3.9/site-packages/ruamel/yaml/constructor.py:111, in BaseConstructor.get_single_data(self)
    109 def get_single_data(self) -> Any:
    110     # Ensure that the stream contains a single document and construct it.
--> 111     node = self.composer.get_single_node()
    112     if node is not None:
    113         return self.construct_document(node)

File ~/github/tempo-syslib/venv/lib/python3.9/site-packages/ruamel/yaml/composer.py:69, in Composer.get_single_node(self)
     67 document: Any = None
     68 if not self.parser.check_event(StreamEndEvent):
---> 69     document = self.compose_document()
     71 # Ensure that the stream contains no more documents.
     72 if not self.parser.check_event(StreamEndEvent):

File ~/github/tempo-syslib/venv/lib/python3.9/site-packages/ruamel/yaml/composer.py:94, in Composer.compose_document(self)
     91 node = self.compose_node(None, None)
     93 # Drop the DOCUMENT-END event.
---> 94 self.parser.get_event()
     96 self.anchors = {}
     97 return node

File ~/github/tempo-syslib/venv/lib/python3.9/site-packages/ruamel/yaml/parser.py:158, in Parser.get_event(self)
    156 if self.current_event is None:
    157     if self.state:
--> 158         self.current_event = self.state()
    159 # assert self.current_event is not None
    160 # if self.current_event.end_mark.line != self.peek_event().start_mark.line:
    161 xprintf('get_event', repr(self.current_event), self.peek_event().start_mark.line)

File ~/github/tempo-syslib/venv/lib/python3.9/site-packages/ruamel/yaml/parser.py:248, in Parser.parse_document_end(self)
    246 # if token.end_mark.line != self.peek_event().start_mark.line:
    247 if token.end_mark.line == self.scanner.peek_token().start_mark.line:
--> 248     raise ParserError(
    249         None,
    250         None,
    251         'found non-comment content after document end marker, '
    252         f'{self.scanner.peek_token().id,!r}',
    253         self.scanner.peek_token().start_mark,
    254     )
    255 end_mark = token.end_mark
    256 explicit = True

ParserError: found non-comment content after document end marker, ('<stream end>',)
  in "<unicode string>", line 4, column 4:
    ...
       ^ (line: 4)

Environment: Python 3.9.9, MacOs 13.3.1.

Thanks!
Mariusz

[Anthon van der Neut]

Thanks for reporting this.
I made some changes there and it seems it requires the doc ends in a newline after the end of document marker. I'll look into that.

Please note that the load function has been deprecated and that you should use the ruamel.yaml.YAML.load() method

[Anthon van der Neut]

Fixed in 0.17.28

[Mariusz Rusiniak]

That was fast! Thank you! Thanks for pointing out that load is deprecated!

[Anthon van der Neut]

I remembered what I changed ( to get an error on non comment content on the line after the end-of-document marker), so it was easy to add an extra test case and change two lines to solve the issue.

As long as the PTB don't change Python packaging tools (again), then it is just one command to run flake, mypy, check the Readme-changelog, run all the tests for each Python version supported and if nothing fails, commit, tag and push the new version. I have 300+ packages (only part of which are open source) so being able to do that efficiently is necessary.

But there have been periods, I spent more time on being able to make releases, than on actually making the changes (recenty with setuptools changes, but especially on my open source software, in the year bitbucket dropped support for mercurial).