The YAML parser will only expand \x80 if it was inside double quotes. My example did not use double quotes:
foo: !badstr BadName\x80
Which is different from:
foo: !badstr "BadName\x80"
In the former case, the scalar is passed as-is (well, subject to plain scalar processing such as folding, which do not include expanding escape codes) to the application - specifically, by using whatever rules the badstr tag has set up. Just like the int tag knows how to convert the characters 1 0 into an integer with the decimal value 10, a badstr tag would know to convert the characters \ x 8 0 to a single byte with hexadecimal value 0x80. In both these cases, this isn't done by the YAML parser itself. Escape sequences inside double quotes are handled by the YAML parser itself, outside the application tag control.
It is interesting that Plan9 also does not enforce validity. FWIW, I think that Windows allowing invalid UTF-16 in file names is a travesty, but that's just me (I also think that windows should have made the space character equivalent to underscore in file names, the same way that it makes capital and lower case letters equivalent, so I could easily type in /.../program_files/... in a command line argument. But I digress). This is neither here nor there.
As you correctly point out, most systems use an actual "must be valid str" internal representation for the data type "string"; they tend to throw exceptions and so on if seeing stuff like invalid high+low surrogate pairs and so on. !!str is meant to be loaded into exactly that - simple UTF-16 or UTF-8 internal "string" representation, which is rather fussy about validity in many systems. As you point out, for your use case you need a different internal representation that can handle all valid Unicode data point + some invalid ones. I suggested (a bit tongue-in-cheek) to tag this other, not-quite-a-normal-string internal representation "badstr". A better name would be "binstr" - A string that also contains "binary data" (arbitrary bytes).
Using an explicit binstr tag indicates to the YAML parser to "please load this into a String-like internal data type that also supports the additional invalid code points". This may or may not be the same internal data type that normal strings are loaded into - depending on how fussy your system's string data type is. Perhaps in Plan9, these might be the same internal data type, while in Java they might be different data types.
If we allowed all normal strings to contain arbitrary bytes, this means that we wouldn't be able to load them into the normal fussy-about-validity internal string type. We would be forced to load everything into the less-usual, not-what-you'd-expect less-fussy string-like data type; this isn't a defensible design choice for languages whose standard string type is fussy about validity - YAML strings would lose interoperability with the rest of the system.
On output, I assume that an application would strive to use !!str rather than !!binstr as much as possible. But that's really an application decision, and not the YAML library's choice. E.g., a hypothetical Windows-file-names dumping application would only resort to emitting !!binstr if the filename was, indeed, a string-with-weird-characters as opposed to a normal string. This would allow it to be safely loaded into (say) a Java application without causing an exception, while minimizing the pain to the human YAML reader.,