#8 TypeError: a bytes-like object is required, not 'str' while dumping binary streams

[Vit Zikmund]

Hello Anton, thanks for this great yaml parser! ;)
Alas, I found that the dumpers using the C implementation can't dump to a binary stream/file, whether it's a file opened as wb or a io.BytesIO stream. I hope that's not to be expected.

Reproduction:

• x86_64-linux-gnu
• Python 3.9.12
• ruamel.yaml 0.17.21

from io import BytesIO
from ruamel.yaml import YAML

yaml = YAML(typ='safe', pure=False)
with BytesIO() as stream:
    yaml.dump({'key':'value'}, stream)
    print(stream.getvalue().decode())

# from pathlib import Path
# with Path('/tmp/testfile.yaml').open('wb') as file:
#   yaml.dump('test', file)

Running the code raises this exception:

Traceback (most recent call last):
  File "scratch_7.py", line 8, in <module>
    yaml.dump({'key':'value'}, file)
  File "<venv>/lib/python3.9/site-packages/ruamel/yaml/main.py", line 574, in dump
    return self.dump_all([data], stream, transform=transform)
  File "<venv>/lib/python3.9/site-packages/ruamel/yaml/main.py", line 583, in dump_all
    self._context_manager.dump(data)
  File "<venv>/lib/python3.9/site-packages/ruamel/yaml/main.py", line 915, in dump
    self._yaml.representer.represent(data)
  File "<venv>/lib/python3.9/site-packages/ruamel/yaml/representer.py", line 81, in represent
    self.serializer.serialize(node)
  File "_ruamel_yaml.pyx", line 1349, in _ruamel_yaml.CEmitter.serialize
  File "_ruamel_yaml.pyx", line 1524, in _ruamel_yaml.output_handler
TypeError: a bytes-like object is required, not 'str'

This has been observed for typ in ['safe', 'unsafe']. I believe rt is still pure only, so the problem doesn't apply there. If pure=True everything works as expected and the simple dict is properly dumped.

[Anthon van der Neut]

AFAICT that is a restriction of ruamel.yaml.clib (whre the erroring _ruamel_yaml.pyx code lives).

You'll have to use a StringIO buffer, even if you are going to dump UTF-8.

[Anthon van der Neut]

Ticket moved from /p/ruamel-yaml/tickets/430/

[Vit Zikmund]

Hi @anthon,
I spent a couple hours learning about the yaml code, cython & unicode and after that I see the problem might be that there's no way how to pass the desired encoding to the CEmitter. No code I saw passes on the encoding argument to CEmitter, so by default it's None, which means in CEmitter.open() (sourceforge.net), the CEmitter.dump_unicode always seems to end up being 1. This makes the output_handler dump unicode (thus Python 3's str) while we would need the other option that dumps str (i.e. Python3's bytes).

TBH the whole route of the encoding down the very end seems rather unclear.
I was able to successfully dump to bytes using the supported encodings: utf-16-le and utf-16-be and even the desired utf-8, when I passed that value in the encoding argument in the XDumper creation. Maybe it could get the encoding attribute from the main YAML class?

            encoding=self.encoding,

Wasn't that forgotten by any chance? :)

This doesn't break dumping to StringIO, because that object/stream contains an encoding attribute, which makes the CEmitter always use dump_unicode = 1, so it might be a proper fix after all! :O

With the proposed simple change above the testing code starts to behave predictably:

from io import BytesIO, StringIO
from ruamel.yaml import YAML

dict_to_dump = {'key': 'Ř'}

for pure in [False, True]:
    print(f"pure={pure}")
    for encoding in ['utf-8', 'utf-16-le', 'utf-16-le']:
        print(f"encoding={encoding}")
        yaml = YAML(typ='safe', pure=pure)
        yaml.encoding = encoding

        with BytesIO() as bytes_stream:
            yaml.dump(dict_to_dump, bytes_stream)
            print(bytes_stream.getvalue().decode(encoding), end='')

        with StringIO() as str_stream:
            yaml.dump(dict_to_dump, str_stream)
            print(str_stream.getvalue(), end='')

LMK if this would be a good PR candidate.

[Anthon van der Neut]

@Vit thanks for looking into this. I might have screwed this up when I introduced the YAML() class. (Before that any extra parameter meant updating multiple in the aggragate XXXLoaders as well as the stage actually needing the extra data, which was cumbersome. The calling of th CEmitter gives a bit of taste of that).

So it looks like I indeed missed copying that line in ruamel.yaml.main.py. Unfortunately the pre-18 code has changed a lot, so I don't know if I can merge a PR for that. The current diff for main.py alone is 1388 lines. Is it ok, when I change this and mention you in the README for finding and fixing this?

[Vit Zikmund]

@anthon The greatest satisfaction will be this being fixed. The big fame as a PR author can still wait :) A readme mention will be just fine. Thank you ;)

[Anthon van der Neut]

I finally got around in adding this in 0.18.0.