[Htmlparser-developer] charset

[Derrick O. <Der...@ro...>]

g'day,

I've dropped code to handle charset parameters, but this raised a couple 
of issues. First an explanation of the changes.

HTMLParser constructors were added that take a URLConnection, so that 
the input stream could be reacquired as needed.
For consistency the 'from file' handling is identical to a URL now.

The HTTP header is now examined for the charset parameter and the input 
stream is converted to a reader with that encoding.
However, a lot of sites lie. They don't specify the charset in the HTTP 
header, even though the HTML is encoded with a non-standard encoding.
So, we have to pre-read the header portion (meta tags) and restart if 
the charset in the meta tags is different than in the HTTP header.

Ya, ya, pre-reading is fraught with peril, and that's what I'm on about 
here...

1) Why aren't the standard scanners registered by the HTMLParser 
constructor? It would seem that proper parsing of HTML would require 
these always be registered. Unless there's a set for HTML 3.2 and one 
for HTML 4.1 or something. Some of the tests rely on this 'empty scanner 
list' behaviour to return the expected number of nodes.

2) Mark and reset are still supported somewhat. If you mark before 
calling elements() and reset() after exhausting the input stream, all is 
well. But mark() in the midst of parsing is useless and dangerous 
because of the BufferedReader. Perhaps the documentation should reflect 
this, or a way to do it properly worked out.

Aside notes:

Nobody's checking that the contents of the document are "text/html", 
perhaps this check should be added.

These changes still leave the 'Reader' constructors as a separate group 
which is used by a majority of the test cases. The 'new testing 
framework' should try to use the same code path as most of the  
real-world use cases.  Perhaps, test cases could be loaded from 
htmlparser.sourceforge.net like HTMLParserTest.testURLWithSpaces() does.

Derrick