#121 Auto-detect utf-8 without signature (BOM)

open
nobody
None
5
2014-12-08
2011-09-15
tav
No

It would be nice if kdiff could auto-detect utf-8 for files without explicit 3-byte order mark (and when in regional settings specified non-utf-8 encoding, and checkbox "Auto Detect Unicode" is set).
Current verision of kdiff (0.9.96) for such files show message box on open (and displays them incorrectly):
"Some input characters could not be converted to valid unicode.
You might be using the wrong codec. (e.g. UTF-8 for non UTF-8 files)."

To check if a file is a utf-8 encoded any simple heuristic algorithm can be used, for example this perl expression:

$field =~
m/\A(
[\x09\x0A\x0D\x20-\x7E] # ASCII
| [\xC2-\xDF][\x80-\xBF] # non-overlong 2-byte
| \xE0[\xA0-\xBF][\x80-\xBF] # excluding overlongs
| [\xE1-\xEC\xEE\xEF][\x80-\xBF]{2} # straight 3-byte
| \xED[\x80-\x9F][\x80-\xBF] # excluding surrogates
| \xF0[\x90-\xBF][\x80-\xBF]{2} # planes 1-3
| [\xF1-\xF3][\x80-\xBF]{3} # planes 4-15
| \xF4[\x80-\x8F][\x80-\xBF]{2} # plane 16
)*\z/x;

Discussion


Log in to post a comment.