#158 UTF-8-BOM has rendering / diff artifacts

[Benjamin Schroeder]

This is occuring on Windows XP x86 and Windows 7 x86 using either 'Auto Detect Unicode' in the Regional Settings or explicitly setting the File Encoding to UTF-8-BOM.

Do a diff of the attached file with a copy of itself you will see three question marks at the start of line three.

Using a hex editor (or other editor that does not modify line endings / BOM / etc.) remove the first line leaving the BOM intact
Do a diff of this file with the original and the modified file will now render correctly.
However, the multibyte character will now also incorrectly show as a difference.

Although the attached file is a little contrived, the important thing to note is the offset of the multibyte character.
It appears to be due to the multibyte character being clipped by a buffer, as it also occurs at multiples of that offset.
ie. The multibyte character in the example is offset 0x3FFE from the end of the BOM. It also occurs at offsets 0x3FFF, 0x13FFE, 0x13FFF, 0x23FFE, etc.

[Benjamin Schroeder]