[Htmlparser-developer] new i/o subsystem

[Derrick O. <Der...@ro...>]

Marc, James, Somik, Joshua, Amit, et. al.

I've just dropped some speed fixes to the lexer package, the new low 
level i/o subsystem I've been working on.
It now appears to be 10% to 50% faster at getting raw nodes than the 
NodeReader/parserHelpers were.
It's not complete:
    - it needs an EndNode class for speed and memory reasons
    - I backed off multi-threading for speed
    - character set detection isn't really working yet
    - there's no constructor taking a file name
But the next logical step is probably integration into the real parser 
to run against real test cases.
However, I think this will cause a *lot* of unit tests to fail.
There are a number of reasons for this:
    - attributes will have case preserved, I think I've gotten around 
this temporarily with a switch in the ParserTestCase class
    - whitespace is preserved, a lot of this has to do with the 
different line endings handling
    - the order of attributes in tags is preserved, so toHtml() output 
is completely different
    - the count of nodes may be altered by the whitespace nodes, this 
may require changing the ParserTestCase counting strategy
    - remark nodes store all the text, even the dashes
    - I mostly only paid attention to the HTML specification, real HTML 
is somewhat more exotic
All these failing tests will need labour intensive manual attention to 
detail to get the tests correct again.
In other words, once this is integrated there's no turning back.
As with any animal that's having it's spine replaced, there's bound to 
be a bit of pain.
So, before that happens, the code should go through a period of severe 
code review.
That's what open source is about right?
So if you have some time. please go over the lexer package with a fine 
tooth comb.
Add more test cases to the lexerTests package.
Take a look at the toString() output (see testReal in LexerTests for 
example).
Optimize the hell out of it.
Bounce it around and see what methods would make you happy. Then add them.
I'm thinking, two weeks minimum, so this period would span at least two 
integration builds.
The first one will be August 24th, so if you don't have CVS access 
you'll need to start with that.

OK, let's have at 'er folks!

Derrick