#1219 KeePass prepending Unicode BOM to created clipboard contents

[Tilman Blumenbach]

I'm using KeePass 2.25 on Arch Linux (Mono 3.2.3). I have xsel 1.2.0 installed to work around clipboard bugs (not using xsel is not an option since I need to be able to use KeePass' clipboard features).

Without setting LC_CTYPE=C before starting KeePass, KeePass (or mono?) prepends a UTF-8 byte order mark (BOM) to all clipboard contents it creates -- and probably to even more strings since I also cannot login using KeeFox and the KeePassRPC plugin when LC_CTYPE is not set to "C".

To rule out plugin errors, I have tried to run KeePass without the KeePassRPC plugin, but that didn't help. The BOM is still being prepended to all clipboard contents (created by using the copy username/copy password feature of KeePass). The same issue occurrs when I use the official portable, binary KeePass distribution.

While setting LC_CTYPE=C is a working workaround, a proper fix would be nice. Any ideas? My locale is "de_DE.utf8".

[Dominik Reichl]

KeePass does not append any BOM to in-memory strings. Given the fact that I cannot reproduce the problem using KeePass 2.25 under Mono 2.10.8.1 on Kubuntu 13.10 (here no BOM is prepended), this must be a bug outside KeePass, most likely in Mono.

Unfortunately I don't really see how a workaround could look like. The method to run another executable accepts strings only (and there's nothing I could to if Mono converts this to a byte array and prepends a BOM). Changing LC_CTYPE does not feel like a good solution; it might lead to other problems, and I'm not sure whether anything would change if KeePass would set LC_CTYPE while running (if Mono evaluates this only at the start, letting KeePass change it afterwards has no effect). As the problem will affect many other applications, I hope the Mono developers fix it soon.

Best regards,
Dominik

[Tilman Blumenbach]

I think I have found a proper solution for this. Patch for KeePassLib.Native.NativeLib.RunConsoleApp() attached.

The issue seems to be with the StreamWriter that is being used to write data to the stdin of the xsel process: On my system, it by default apparently uses an UTF8Encoding that emits a BOM. Since for some reason, one cannot specify the stdin encoding using ProcessStartInfo, the solution is to create an UTF8Encoding that does not emit a BOM and manually convert the data that is going to be sent to stdin using that Encoding.

Please test the attached patch and consider merging it into upstream. Thanks!

[Tilman Blumenbach]

Here's a smaller version of my patch which uses the existing KeePassLib.Utility.StrUtil.Utf8 object instead of creating its own.

[Dominik Reichl]

Thanks a lot for your analysis!

There's one issue with your suggested patch: on Windows (and maybe in some cases under Mono, too) when closing the StreamWriter, it writes the BOM without checking the base stream (i.e. we get the data that we wrote to the base stream and an appended BOM). Closing the base stream before closing the StreamWriter is not possible (otherwise the StreamWriter throws an exception).

A slightly different workaround approach is to make the StreamWriter think that the BOM has already been written. This can be done by setting its private member variable 'preamble_done' to true.

As I currently don't have a Linux system for testing this (all my Linux boxes use an older version of Mono, where the bug wasn't present yet), it would be great if you could test it.

Here's the latest development snapshot, including the modified NativeLib.cs file:
http://keepass.info/filepool/KeePass_140331.zip

Thanks again, best regards,
Dominik

[Dominik Reichl]

If the Encoding does not specify a BOM, the StreamWriter does not emit one either. I've now added a check for this.

Here's the latest development snapshot for testing, including the new NativeLib.cs file:
http://keepass.info/filepool/KeePass_140331b.zip

Thanks and best regards,
Dominik

[Tilman Blumenbach]

Interesting. :-)

Anyway, the second snapshot fixes the issue for me -- no BOM visible. Thanks!

[Michał Kępień]

This issue reemerged for me when I upgraded to Mono 4.x in Arch (I'm using KeePass 2.29). A quick investigation revealed that Mono switched to using text stream classes from Microsoft's reference sources back in February. The version of StreamWriter contained in the reference sources has no preamble_done property any more, but uses a property called haveWrittenPreamble for the same purpose, as far as I can tell. Any chance you can get this fixed?

[Dominik Reichl]

Thanks a lot for reporting this! I've now enhanced the Mono workaround to additionally support the StreamWriter from Microsoft's reference source.

Here's the latest development snapshot for testing:
http://keepass.info/filepool/KeePass_150523.zip

Best regards,
Dominik

[Michał Kępień]

Unfortunately, the problem still persists in the development snapshot you posted a link to above. I did some more digging and it turns out setting haveWrittenPreamble to true is not enough for Mono 4.x.

The problem lies in Process.Start() as it sets the AutoFlush property of the spawned process' standard input to true. This in turn causes an initial flush of the stream. While all the above holds true for both Mono 3.x and 4.x, flushing implementation differs between the two (see: 3.x, 4.x). Specifically, 3.x doesn't write anything (including the preamble) until the first Write() is issued while 4.x writes the preamble immediately after setting AutoFlush to true. This renders the EnsureNoBom() call void as by the time it happens, the BOM has already been written.