wwwsearch

On Thu, Mar 09, 2006 at 01:29:33PM -0800, Titus Brown wrote:
-> Hi all,
-> 
-> I'm facing a minor revolt on the twill list over the default use of
-> 'latin-1' for form parsing.  I'm going to be switching to UTF-8, because
-> that appears to be a sensible default:
-> 
-> 	http://www.alanwood.net/unicode/htmlunicode.html
-> 
-> The true situation seems to be confused, as usual:
-> 
-> 	http://www.w3.org/TR/REC-html40/charset.html
-> 
-> (I'd try to reference older wwwsearch-general posts, but right now the
-> search on SF is broken!)

p.s. It'd be nice to have a single variable in ClientForm.py,
'DEFAULT_ENCODING', that we could patch if we wanted to change this.

On 3/9/06, Titus Brown <titus@...> wrote:
> Hi all,
>
> I'm facing a minor revolt on the twill list over the default use of
> 'latin-1' for form parsing.  I'm going to be switching to UTF-8, because
> that appears to be a sensible default:
>
>         http://www.alanwood.net/unicode/htmlunicode.html
>
> The true situation seems to be confused, as usual:
>
>         http://www.w3.org/TR/REC-html40/charset.html
>
> (I'd try to reference older wwwsearch-general posts, but right now the
> search on SF is broken!)

In my experience, pages in the western world are normally declared to
be latin-1, and are actually iso-8859-15 (latin-1 plus the euro sign),
or more usually cp1252. The rationale for this is the fact that the
standard Apache config will automatically add the default character
set of latin-1 to web headers unless explicitly modified either by the
dynamic script running the website (which many programmers seem not to
bother doing), or in the appropriate .htaccess file or apache config
file in the case of static files.

I believe the various RFCs and TSs end up specifying that latin-1
should be the default encoding for HTML if none is specified in the
HTTP headers or in the page's META tag, unless a unicode encoding's
Byte Order Mark is encountered. The default encoding for XML (and
hence XHTML) is, I believe, UTF-8.

Moof

On Fri, Mar 10, 2006 at 10:46:56AM +0100, Moof wrote:
-> On 3/9/06, Titus Brown <titus@...> wrote:
-> > Hi all,
-> >
-> > I'm facing a minor revolt on the twill list over the default use of
-> > 'latin-1' for form parsing.  I'm going to be switching to UTF-8, because
-> > that appears to be a sensible default:
-> >
-> >         http://www.alanwood.net/unicode/htmlunicode.html
-> >
-> > The true situation seems to be confused, as usual:
-> >
-> >         http://www.w3.org/TR/REC-html40/charset.html
-> >
-> > (I'd try to reference older wwwsearch-general posts, but right now the
-> > search on SF is broken!)
-> 
-> In my experience, pages in the western world are normally declared to
-> be latin-1, and are actually iso-8859-15 (latin-1 plus the euro sign),
-> or more usually cp1252. The rationale for this is the fact that the
-> standard Apache config will automatically add the default character
-> set of latin-1 to web headers unless explicitly modified either by the
-> dynamic script running the website (which many programmers seem not to
-> bother doing), or in the appropriate .htaccess file or apache config
-> file in the case of static files.
-> 
-> I believe the various RFCs and TSs end up specifying that latin-1
-> should be the default encoding for HTML if none is specified in the
-> HTTP headers or in the page's META tag, unless a unicode encoding's
-> Byte Order Mark is encountered. The default encoding for XML (and
-> hence XHTML) is, I believe, UTF-8.

Here's something from http://www.alanwood.net/unicode/htmlunicode.html:

"""
The only detail about character encodings that a writer needs to know is
that some character encodings (for example UTF-8) allow any of the
characters in the document character set to be included, while others
(for example ISO-8859-1 or SHIFT_JIS) only allow for subsets. However,
characters that are not allowed for in a character encoding can still be
included in an HTML document by using character references. UTF-8 is the
normal character encoding for any HTML file that contains text in two or
more non-Latin scripts, but it can be used for any document.
"""

This appears to be the problem twill users (4 separate ones...) are
running into: latin-1 simply can't encode the characters in some of
the documents.  A simple example is a page containing only
this:

"""
&rsaquo;
"""

If you try to parse the forms on a page including that character, you'll
get a unicode error.

cheers,
--titus

On Fri, 10 Mar 2006, Moof wrote:

> In my experience, pages in the western world are normally declared to
> be latin-1, and are actually iso-8859-15 (latin-1 plus the euro sign),
> or more usually cp1252. The rationale for this is the fact that the
> standard Apache config will automatically add the default character
> set of latin-1 to web headers unless explicitly modified either by the
> dynamic script running the website (which many programmers seem not to
> bother doing), or in the appropriate .htaccess file or apache config
> file in the case of static files.

Perhaps best solution there is to document and add a functional test 
showing how to modify the Content-Type header to fix bad encoding 
declarations?

[...]
> Byte Order Mark is encountered. The default encoding for XML (and
> hence XHTML) is, I believe, UTF-8.

I'm certainly not handling XML correctly.  Probably that's evil of me and 
I should completely disable XML support by default.


John

On Fri, 10 Mar 2006, Titus Brown wrote:
[...]
> This appears to be the problem twill users (4 separate ones...) are
> running into: latin-1 simply can't encode the characters in some of
> the documents.  A simple example is a page containing only
> this:
>
> """
> ›
> """
>
> If you try to parse the forms on a page including that character, you'll
> get a unicode error.

I explicitly considered that when I was fixing encoding issues recently, 
and thought I'd tested it, so it's a bug if SVN does that.  Will check 
later.  The intent was to leave entity refs that cannot be represented in 
the encoding intact (yes, using unicode objects throughout would be 
better).


John

> """
> The only detail about character encodings that a writer needs to know is
> that some character encodings (for example UTF-8) allow any of the
> characters in the document character set to be included, while others
> (for example ISO-8859-1 or SHIFT_JIS) only allow for subsets. However,
> characters that are not allowed for in a character encoding can still be
> included in an HTML document by using character references. UTF-8 is the
> normal character encoding for any HTML file that contains text in two or
> more non-Latin scripts, but it can be used for any document.
> """
>
> This appears to be the problem twill users (4 separate ones...) are
> running into: latin-1 simply can't encode the characters in some of
> the documents.  A simple example is a page containing only
> this:
>
> """
> ›
> """
>
> If you try to parse the forms on a page including that character, you'll
> get a unicode error.

Having just discovered it today, have you tried:
http://chardet.feedparser.org/ ?

Moof

On Mon, 13 Mar 2006, John J Lee wrote:

> On Fri, 10 Mar 2006, Titus Brown wrote:
> [...]
>>  This appears to be the problem twill users (4 separate ones...) are
>>  running into: latin-1 simply can't encode the characters in some of
>>  the documents.  A simple example is a page containing only
>>  this:
>>
>>  """
>>  ›
>>  """
>>
>>  If you try to parse the forms on a page including that character, you'll
>>  get a unicode error.
>
> I explicitly considered that when I was fixing encoding issues recently, and 
> thought I'd tested it, so it's a bug if SVN does that.  Will check later. 
> The intent was to leave entity refs that cannot be represented in the 
> encoding intact (yes, using unicode objects throughout would be better).

OK, I checked, and I can't trigger a unicode error.  Can you provide a 
script that demonstrates the problem?


John

On Tue, 14 Mar 2006, Moof wrote:
[...]
> Having just discovered it today, have you tried:
> http://chardet.feedparser.org/ ?

That looks useful, thanks.  Probably I'll make it an option to use this 
package.


John

On Thu, 9 Mar 2006, Titus Brown wrote:
[...]
> p.s. It'd be nice to have a single variable in ClientForm.py,
> 'DEFAULT_ENCODING', that we could patch if we wanted to change this.

Done.


John

On Tue, 14 Mar 2006, John J Lee wrote:

> On Tue, 14 Mar 2006, Moof wrote:
> [...]
>>  Having just discovered it today, have you tried:
>>  http://chardet.feedparser.org/ ?
>
> That looks useful, thanks.  Probably I'll make it an option to use this 
> package.

Strangely, I had occasion to try this out today at work, and it failed to 
detect a UTF-16 file (LE with BOM) for me.  Maybe I messed something up...


John