Re: [Configobj-develop] Re: ConfigObj unicode Support in SVN

[Fuzzyman <fuz...@vo...>]

Aaron Bentley wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Michael Foord wrote:
> | I would really appreciate it if you could check this out and try and
> | break it. Unicode is notoriously fiddly to get right !
>
> This all looks good.  I will try converting bzr to the latest version
> soon, and let you know.
>
Great.

> | There is still a question about automatic support for UTF8 files with a
> | BOM. The new position is that UTF8 files with a BOM will be
> | automatically decoded to unicode.
> |
> | I have also proposed a *better* way of handling the BOM.
>
> I like the idea of BOM being a boolean, if that's what you mean.
>
Ok, it's actually simpler than what I've currently implemented and is 
more sensible really.

The other question is, if ConfigObj is passed a UTF8 file with a BOM, 
should it automatically decode to unicode ?

The behaviour of ConfigObj 4.1.0 (and previous) was to preserve the BOM 
(UTF8 only) but not decode. On balance it is probably better *not* to 
decode (ConfigObj doesn't attempt to detect the encoding of other 
files). For UTF16 files it must automatically decode - otherwise it 
splits characters on byte boundaries which mangles them.

So if BOM is True, but no encoding is specified, then a UTF8 BOM will be 
written out. Otherwise a BOM will only be written out if BOM is True and 
the encoding is UTF8 or UTF16.


All the best,


Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

> Aaron
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.1 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
>
> iD8DBQFD4YK10F+nu1YWqI0RApP8AJ0S+tbZ3V+oYUXL1+MrKwnQQpEenQCghLVJ
> 7TmgjVEsz55qNapITDocMD4=
> =UHuk
> -----END PGP SIGNATURE-----
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log 
> files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
> _______________________________________________
> Configobj-develop mailing list
> Con...@li...
> https://lists.sourceforge.net/lists/listinfo/configobj-develop
>