htmlparser-developer

[Htmlparser-developer] Re: [Htmlparser-user] new problem

[Somik R. <so...@ya...>]

Hi Annette,
    Regarding your second problem, the parsing error occurs because -=20
=20
<div align=3D"center"><font face=3D"Arial,"helvetica," =
sans-serif=3D"sans-serif" size=3D"2" color=3D"#FFFFFF"><a =
href=3D"/index.html" link=3D"#000000" vlink=3D"#000000"><font=20

In the above - font face=3D"Arial,"helvetica,"  -- note the erroneoue =
extra " in front of helvetica. Remove it and the parsing is fine. Now of =
course you cant remove it, bcos this site is not yours :). So, we do =
have to support this kind of dirty html. Thank you so much for bringing =
it to our notice. I have written a test case to reproduce this bug, and =
am working to resolve this.

Regards,
Somik
  =20

  <div align=3D"center"><font face=3D"Arial,"helvetica," =
sans-serif=3D"sans-serif" size=3D"2" color=3D"#FFFFFF"><a =
href=3D"/index.html" link=3D"#000000" vlink=3D"#000000"><font =
color=3D"#FFFFFF">Home</font></a>=20

          | <a href=3D"/cia/notices.html" link=3D"#000000" =
vlink=3D"#000000"><font color=3D"#FFFFFF">Notices</font></a>=20

          | <a href=3D"/cia/notices.html#priv" link=3D"#000000" =
vlink=3D"#000000"><font color=3D"#FFFFFF">Privacy</font></a>=20

          | <a href=3D"/cia/notices.html#sec" link=3D"#000000" =
vlink=3D"#000000"><font color=3D"#FFFFFF">Security</font></a>=20

          | <a href=3D"/cia/contact.htm" link=3D"#000000" =
vlink=3D"#000000"><font color=3D"#FFFFFF">Contact Us</font></a>

          | <a href=3D"/cia/sitemap.html" link=3D"#000000" =
vlink=3D"#000000"><font color=3D"#FFFFFF">Site Map</font></a>

          | <a href=3D"/cia/siteindex.html" link=3D"#000000" =
vlink=3D"#000000"><font color=3D"#FFFFFF">Index</font></a>

          | <a href=3D"/search" link=3D"#000000" vlink=3D"#000000"><font =
color=3D"#FFFFFF">Search</font></a>=20

          </font></div>

  =20

  Stops at=20

  TAG LINE FOUND      <div align=3D"center"><font =
face=3D"Arial,"helvetica," sans-serif=3D"sans-serif" size=3D"2" =
color=3D"#FFFFFF"><a href=3D"/index.html" link=3D"#000000" =
vlink=3D"#000000"><font color=3D"#FFFFFF">Home</font></a>=20

  LINE is       <div align=3D"center"><font face=3D"Arial,"helvetica," =
sans-serif=3D"sans-serif" size=3D"2" color=3D"#FFFFFF"><a =
href=3D"/index.html" link=3D"#000000" vlink=3D"#000000"><font =
color=3D"#FFFFFF">Home</font></a>=20

  POSITION IS 26

  TAGLINE 197

  Process completed.

  =20

  Annette Doyle

  =20

[Htmlparser-developer] Annette's bug difficult to fix - need help (AI approaches ??)

[Somik R. <so...@ya...>]

Hi Folks,
    If you've been following the latest exchange on htmlparser-user, =
Annette has shown us a crazy example of dirty html, which works in the =
browser, but crashes the parser.
    The site is http://www.cia.gov  =20
    Search for this string - <font face=3D"Arial,"helvetica,"
    and you will find it in the html. Now this erroneous inverted comma =
in front of helvetica should not be there.=20

    This has been captured in a test case in HTMLTagTest.java (you can =
get it from CVS), and this test fails (testParsing()).
    The problem is - the core parsing mechanism ignores anything within =
inverted commas. This is critical so as to be able to accept angular =
brackets in inverted commas. If we remove this feature from the parser =
other tests will break.
   =20
    So I need some suggestions on how we might modify our parsing - how =
do we intelligently understand that this is an error (how easy it is for =
us humans to figure this out) ? Looks like linear approaches wouldnt =
work anymore... Maybe we need to associate some intelligence - that if =
its a font tag, then this kind of stuff is most definitely an error. =
Whereas if its a jsp tag, we can get more strict with our parsing. This =
will probably cause a fundamental shift in our core parsing technique.

Regards,
Somik