htmlparser-user — The user mailing list for users of the htmlparser library

Re: [Htmlparser-user] Not able to catch ParserException

[Martin S. <mst...@gm...>]

Hi,

While I'm not a developer of HTMLParser, I think it is very unlikely that
somewhere in the code a ParseException is printed to standard out. I use
StringBean rather intensively in my application, but did not have this
problem.

The only thing I can think of is that your logger is printing the exception
to standard out or something (you are passing the exception to the error
method). Does the problem occur when you comment out the logger.error("Coul=
d
not parse url", e); ?

--
Martin

2007/1/11, =D8ystein Lervik Larsen <oys...@me...>:
>
> Martin Sturm wrote:
> > Hi,
>
> Good morning, thanks for your reply!
>
> > Are you sure you are running your own program?
>
> Not quite sure what you mean there...
>
> > StringExtractor is only a sample application for the StringBean, so it
> > would be better to use the StringBean directly instead of
> StringExtractor.
>
> I tried that too and the same thing happened, see below.
>
> > I did not experience your problem (I use StringBean rather intensively)
> > and it would mean that the ParserException is catched somewhere in the
> > HTMLParser codebase and printed directly to standard out. This is not
> > very likely I think.
>
> That is exactly what I'm suspecting. Though unlikely - is it possible?
>
> > Maybe you can try to use StringBean directly insttead of
> StringExtractor:
> >
> > String content =3D "";
> > try {
> >         StringBean sb;
> >
> >         sb =3D new StringBean ();
> >         sb.setLinks (false);
> >         sb.setURL (url);
> >         content =3D sb.getStrings();
> >         System.out.println(content);
> > } catch (ParserException e) {
> >      logger.error("Could not parse url", e);
> >      System.out.println("test");
> > }
>
> I modified my code according to your example but removed the try/catch
> and the exception was still written to the console the same way as it
> used to. When I'm not catching the exception I guess it get caught some
> place else?
>
> The exception occurs when the web server responds with http error code
> 401 Unauthorized.
>
> I'm developing a web application using Tomcat and Spring framework if
> that's relevant.
>
>
> -=D8ystein
>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share
> your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D=
DEVDEV
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>

Re: [Htmlparser-user] Adding elements to the html

[Martin S. <mst...@gm...>]

Hi,

I did a quick test, and I think you forgot to add the endtag to the toCreate
object.
You should add:

    toCreate.setEndTag(new Span());

before you final println statement.

The following code snippet works for me:
         TagNode toCreate = new Span();
         toCreate.setAttribute("key", "Test", '"');
         NodeList nl = new NodeList();
         nl.add(new TextNode("Test2"));
         toCreate.setChildren(nl);
         toCreate.setEndTag(new Span());
         System.out.println(toCreate.toHtml());

Result: <SPAN key="Test">Test2<SPAN>

Hope this will help you.

-- Martin


2007/1/10, Joel <jo...@ha...>:
>
> I want to wrap text string with a span tag. I've tried the folowing, but
> I'm running into a problem, that the tag's children aren't being
> displayed.
>
>         //New <span key="x">some text here</span>
>         TagNode toCreate = new Span();
>         toCreate.setAttribute("key", getKey(str), '"');
>         NodeList nl = new NodeList();
>         nl.add(new TextNode(str));
>         toCreate.setChildren(nl);
>         System.out.println(toCreate.toHtml());
>
> This ends up showing <span key="x"> without the text node and end tag,
> what am I doing wrong?
>
> Joel
>
>
>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share
> your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>

Re: [Htmlparser-user] Not able to catch ParserException

[<oys...@me...>]

Martin Sturm wrote:
> Hi,

Good morning, thanks for your reply!

> Are you sure you are running your own program?

Not quite sure what you mean there...

> StringExtractor is only a sample application for the StringBean, so it 
> would be better to use the StringBean directly instead of StringExtractor.

I tried that too and the same thing happened, see below.

> I did not experience your problem (I use StringBean rather intensively) 
> and it would mean that the ParserException is catched somewhere in the 
> HTMLParser codebase and printed directly to standard out. This is not 
> very likely I think.

That is exactly what I'm suspecting. Though unlikely - is it possible?

> Maybe you can try to use StringBean directly insttead of StringExtractor:
> 
> String content = "";
> try {
>         StringBean sb;
> 
>         sb = new StringBean ();
>         sb.setLinks (false);
>         sb.setURL (url);
>         content = sb.getStrings();
>         System.out.println(content);
> } catch (ParserException e) {
>      logger.error("Could not parse url", e);
>      System.out.println("test");
> }

I modified my code according to your example but removed the try/catch 
and the exception was still written to the console the same way as it 
used to. When I'm not catching the exception I guess it get caught some 
place else?

The exception occurs when the web server responds with http error code 
401 Unauthorized.

I'm developing a web application using Tomcat and Spring framework if 
that's relevant.


-Øystein

[Htmlparser-user] Adding elements to the html

[Joel <jo...@ha...>]

I want to wrap text string with a span tag. I've tried the folowing, but 
I'm running into a problem, that the tag's children aren't being displayed.

        //New <span key="x">some text here</span>
        TagNode toCreate = new Span();
        toCreate.setAttribute("key", getKey(str), '"');
        NodeList nl = new NodeList();
        nl.add(new TextNode(str));
        toCreate.setChildren(nl);
        System.out.println(toCreate.toHtml());

This ends up showing <span key="x"> without the text node and end tag, 
what am I doing wrong?
       
Joel

Re: [Htmlparser-user] Parsing quoted strings seems to be broken in 1.6

[sebb <se...@gm...>]

FYI:

I've now tested the parser using

ScriptScanner.STRICT=false

and that solved the "problem".

Thanks again.

On 08/01/07, sebb <se...@gm...> wrote:
> Sorry, my bad - I've now read the document referenced in the scanner
> source, and I see that "</" acts as the terminator unless suitably
> hidden.
>
> S.
> On 08/01/07, sebb <se...@gm...> wrote:
> > Thanks for the quick reply. I'll give it a try.
> >
> > However, I'm not sure why the script example is bad.
> >
> > It is not enclosed in "<!--" and "// -->", but AIUI those are only
> > needed as a work-round for older browsers that did not understand the
> > <script> tag.
> >
> >
> > On 08/01/07, Derrick Oswald <Der...@ro...> wrote:
> > >
> > > For parsing bad script like this you probably want to set the static
> > > boolean value org.htmlparser.scanners.ScriptScanner.STRICT to false. See
> > > the explanation in the ScriptScanner.java file.
> > >
> > > sebb wrote:
> > >
> > > >The sample script:
> > > >
> > > ><HTML>
> > > >  <body>
> > > >  <script>
> > > >  fred = "<img src='a.gif'></img>"
> > > >  </script>
> > > >  </body>
> > > ></HTML>
> > > >
> > > >generates the following output from parser.cmd:
> > > >
> > > >Tag (0[0,0],6[0,6]): HTML
> > > >  Txt (6[0,6],10[1,2]): \n
> > > >  Tag (10[1,2],16[1,8]): body
> > > >    Txt (16[1,8],20[2,2]): \n
> > > >    Tag (20[2,2],28[2,10]): script
> > > >      Txt (28[2,10],57[3,27]): \n  fred = "<img src='a.gif'>
> > > >      End (57[3,27],57[3,27]): /script
> > > >    End (57[3,27],63[3,33]): /img
> > > >    Txt (63[3,33],68[4,2]): "\n
> > > >    End (68[4,2],77[4,11]): /script
> > > >    Txt (77[4,11],81[5,2]): \n
> > > >    End (81[5,2],88[5,9]): /body
> > > >  Txt (88[5,9],90[6,0]): \n
> > > >  End (90[6,0],97[6,7]): /HTML
> > > >Txt (97[6,7],101[8,0]): \n\n
> > > >
> > > >It looks like the closing tag is being recognised - though the opening
> > > >tag is not.
> > > >
> > > >Is this a bug, or have I misunderstood something?
> > > >
> > > >-------------------------------------------------------------------------
> > > >Take Surveys. Earn Cash. Influence the Future of IT
> > > >Join SourceForge.net's Techsay panel and you'll get the chance to share your
> > > >opinions on IT & business topics through brief surveys - and earn cash
> > > >http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> > > >_______________________________________________
> > > >Htmlparser-user mailing list
> > > >Htm...@li...
> > > >https://lists.sourceforge.net/lists/listinfo/htmlparser-user
> > > >
> > > >
> > > >
> > >
> > >
> > > -------------------------------------------------------------------------
> > > Take Surveys. Earn Cash. Influence the Future of IT
> > > Join SourceForge.net's Techsay panel and you'll get the chance to share your
> > > opinions on IT & business topics through brief surveys - and earn cash
> > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> > > _______________________________________________
> > > Htmlparser-user mailing list
> > > Htm...@li...
> > > https://lists.sourceforge.net/lists/listinfo/htmlparser-user
> > >
> >
>

Re: [Htmlparser-user] Not able to catch ParserException

[Martin S. <mst...@gm...>]

Hi,

Are you sure you are running your own program?
StringExtractor is only a sample application for the StringBean, so it woul=
d
be better to use the StringBean directly instead of StringExtractor.
I did not experience your problem (I use StringBean rather intensively) and
it would mean that the ParserException is catched somewhere in the
HTMLParser codebase and printed directly to standard out. This is not very
likely I think.

However, the main() method of StringExtractor does catch ParserException an=
d
print it to standard out.

Maybe you can try to use StringBean directly insttead of StringExtractor:

String content =3D "";
try {
        StringBean sb;

        sb =3D new StringBean ();
        sb.setLinks (false);
        sb.setURL (url);
        content =3D sb.getStrings();
        System.out.println(content);
} catch (ParserException e) {
     logger.error("Could not parse url", e);
     System.out.println("test");
}

--
Martin Sturm

2007/1/10, =D8ystein Lervik Larsen <oys...@me...>:
>
> Hi list! :-]
>
> I seem to have a problem catching the ParserException when using
> StringExtractor.
>
> My console says:
> org.htmlparser.util.ParserException: Exception getting input stream from
> (and so on...)
>
> ...but my logger (log4j) does not print and the string "test" is not
> written to the console.
>
>
> String content =3D "";
>    try {
>      StringExtractor se =3D new StringExtractor(url);
>      content =3D se.extractStrings(false);
>      System.out.println(content);
>    }
>    catch(ParserException e){
>      logger.error("Could not parse url",e);
>      System.out.println("test");
> }
>
>
> Could the exception be handled some place else that I'm not aware of?
> The string "content" is sometimes empty due to a 401 error.
>
> Thanks in advance for any reply!
>
> Best regards,
> =D8ystein
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share
> your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D=
DEVDEV
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>

[Htmlparser-user] Not able to catch ParserException

[<oys...@me...>]

Hi list! :-]

I seem to have a problem catching the ParserException when using 
StringExtractor.

My console says:
org.htmlparser.util.ParserException: Exception getting input stream from 
(and so on...)

...but my logger (log4j) does not print and the string "test" is not 
written to the console.


String content = "";
   try {
     StringExtractor se = new StringExtractor(url);
     content = se.extractStrings(false);
     System.out.println(content);
   }
   catch(ParserException e){
     logger.error("Could not parse url",e);
     System.out.println("test");
}


Could the exception be handled some place else that I'm not aware of?
The string "content" is sometimes empty due to a 401 error.

Thanks in advance for any reply!

Best regards,
Øystein

[Htmlparser-user] Encoding problem when parsing website

[Martin S. <mst...@gm...>]

SGVsbG8sCgpJJ20gdXNpbmcgSFRNTFBhcnNlciBmb3IgZXh0cmFjdGluZyB0ZXh0IGZyb20gYSBI
VE1MIHBhZ2UgaW4gb3JkZXIgdG8gaW5kZXgKaXQgdXNpbmcgYSBmdWxsIHRleHQgc2VhcmNoIGVu
Z2luZS4KRHVyaW5nIHRoZSB0ZXN0aW5nIHBoYXNlLCBJIGRpc2NvdmVyZWQgdGhhdCBzb21lIHdl
YiBwYWdlcyBhcmUgbm90IHBhcnNlZApjb3JyZWN0bHkgYnkgSFRNTFBhcnNlci4gT25lIG9mIHRo
ZXNlIHdlYnBhZ2VzIGlzIGZvciBleGFtcGxlCmh0dHA6Ly93d3cubWljcm9zb2Z0LmNvbS4KSSB0
aGluayB0aGUgcHJvYmxlbSBpcyB0aGF0IGFjY29yZGluZyB0byB0aGUgSFRUUCBoZWFkZXJzLCB0
aGUgZW5jb2RpbmcgaXMKaW4gVVRGLTgsIGJ1dCBpbiBIVE1MIE1FVEEgdGFncyB0aGlzIGlzIGNo
YW5nZWQgdG8gVVRGLTE2LiBUaGlzIGNhbiBiZQpoYW5kbGVkIGJ5IGNhdGNoaW5nIHRoZSBFbmNv
ZGluZ0NoYW5nZUV4Y2VwdGlvbiwgYnV0IHRoaXMgZG9lc24ndCBwcmV2ZW50CnRoZSB0ZXh0dWFs
IGNvbnRlbnQgb2YgdGhlIHNpdGUgaW50ZXJwcmV0ZWQgaW5jb3JyZWN0bHkuCgpBIGNvbmNyZXRl
IGV4YW1wbGUgdG8gc2VlIHRoZSBwcm9ibGVtOgoKICAgICAgICBTdHJpbmdCZWFuIHNiID0gbmV3
IFN0cmluZ0JlYW4oKTsKICAgICAgICBzYi5zZXRVUkwoImh0dHA6Ly93d3cubWljcm9zb2Z0LmNv
bSIpOwogICAgICAgIFN5c3RlbS5vdXQucHJpbnRsbihzYi5nZXRTdHJpbmdzKCkpOwoKVGhlIG91
dHB1dCBvZiB0aGUgYWJvdmUgY29kZSBzbmlwcGV0IGlzOgrkkY/kjZTlpZDklKDmobTmtazigZDl
lYLksYnkjKDiiK3ivK/lnLPkjK/ivYTlkYTigYjlkY3ksKDjkK7jgKDlkbLmha7njannkanmva4g
Li4uLgoKTm90IHJlYWxseSB3aGF0IEkgd2FzIGV4cGVjdGluZy4KCkFtIEkgbWlzc2luZyBzb21l
dGhpbmcsIG9yIGlzIHRoaXMgYSBidWcgaW4gdGhlIEhUTUxQYXJzZXI/CgpNYXJ0aW4gU3R1cm0K

Re: [Htmlparser-user] Parsing quoted strings seems to be broken in 1.6

[sebb <se...@gm...>]

Sorry, my bad - I've now read the document referenced in the scanner
source, and I see that "</" acts as the terminator unless suitably
hidden.

S.
On 08/01/07, sebb <se...@gm...> wrote:
> Thanks for the quick reply. I'll give it a try.
>
> However, I'm not sure why the script example is bad.
>
> It is not enclosed in "<!--" and "// -->", but AIUI those are only
> needed as a work-round for older browsers that did not understand the
> <script> tag.
>
>
> On 08/01/07, Derrick Oswald <Der...@ro...> wrote:
> >
> > For parsing bad script like this you probably want to set the static
> > boolean value org.htmlparser.scanners.ScriptScanner.STRICT to false. See
> > the explanation in the ScriptScanner.java file.
> >
> > sebb wrote:
> >
> > >The sample script:
> > >
> > ><HTML>
> > >  <body>
> > >  <script>
> > >  fred = "<img src='a.gif'></img>"
> > >  </script>
> > >  </body>
> > ></HTML>
> > >
> > >generates the following output from parser.cmd:
> > >
> > >Tag (0[0,0],6[0,6]): HTML
> > >  Txt (6[0,6],10[1,2]): \n
> > >  Tag (10[1,2],16[1,8]): body
> > >    Txt (16[1,8],20[2,2]): \n
> > >    Tag (20[2,2],28[2,10]): script
> > >      Txt (28[2,10],57[3,27]): \n  fred = "<img src='a.gif'>
> > >      End (57[3,27],57[3,27]): /script
> > >    End (57[3,27],63[3,33]): /img
> > >    Txt (63[3,33],68[4,2]): "\n
> > >    End (68[4,2],77[4,11]): /script
> > >    Txt (77[4,11],81[5,2]): \n
> > >    End (81[5,2],88[5,9]): /body
> > >  Txt (88[5,9],90[6,0]): \n
> > >  End (90[6,0],97[6,7]): /HTML
> > >Txt (97[6,7],101[8,0]): \n\n
> > >
> > >It looks like the closing tag is being recognised - though the opening
> > >tag is not.
> > >
> > >Is this a bug, or have I misunderstood something?
> > >
> > >-------------------------------------------------------------------------
> > >Take Surveys. Earn Cash. Influence the Future of IT
> > >Join SourceForge.net's Techsay panel and you'll get the chance to share your
> > >opinions on IT & business topics through brief surveys - and earn cash
> > >http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> > >_______________________________________________
> > >Htmlparser-user mailing list
> > >Htm...@li...
> > >https://lists.sourceforge.net/lists/listinfo/htmlparser-user
> > >
> > >
> > >
> >
> >
> > -------------------------------------------------------------------------
> > Take Surveys. Earn Cash. Influence the Future of IT
> > Join SourceForge.net's Techsay panel and you'll get the chance to share your
> > opinions on IT & business topics through brief surveys - and earn cash
> > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> > _______________________________________________
> > Htmlparser-user mailing list
> > Htm...@li...
> > https://lists.sourceforge.net/lists/listinfo/htmlparser-user
> >
>

Re: [Htmlparser-user] Parsing quoted strings seems to be broken in 1.6

[sebb <se...@gm...>]

Thanks for the quick reply. I'll give it a try.

However, I'm not sure why the script example is bad.

It is not enclosed in "<!--" and "// -->", but AIUI those are only
needed as a work-round for older browsers that did not understand the
<script> tag.


On 08/01/07, Derrick Oswald <Der...@ro...> wrote:
>
> For parsing bad script like this you probably want to set the static
> boolean value org.htmlparser.scanners.ScriptScanner.STRICT to false. See
> the explanation in the ScriptScanner.java file.
>
> sebb wrote:
>
> >The sample script:
> >
> ><HTML>
> >  <body>
> >  <script>
> >  fred = "<img src='a.gif'></img>"
> >  </script>
> >  </body>
> ></HTML>
> >
> >generates the following output from parser.cmd:
> >
> >Tag (0[0,0],6[0,6]): HTML
> >  Txt (6[0,6],10[1,2]): \n
> >  Tag (10[1,2],16[1,8]): body
> >    Txt (16[1,8],20[2,2]): \n
> >    Tag (20[2,2],28[2,10]): script
> >      Txt (28[2,10],57[3,27]): \n  fred = "<img src='a.gif'>
> >      End (57[3,27],57[3,27]): /script
> >    End (57[3,27],63[3,33]): /img
> >    Txt (63[3,33],68[4,2]): "\n
> >    End (68[4,2],77[4,11]): /script
> >    Txt (77[4,11],81[5,2]): \n
> >    End (81[5,2],88[5,9]): /body
> >  Txt (88[5,9],90[6,0]): \n
> >  End (90[6,0],97[6,7]): /HTML
> >Txt (97[6,7],101[8,0]): \n\n
> >
> >It looks like the closing tag is being recognised - though the opening
> >tag is not.
> >
> >Is this a bug, or have I misunderstood something?
> >
> >-------------------------------------------------------------------------
> >Take Surveys. Earn Cash. Influence the Future of IT
> >Join SourceForge.net's Techsay panel and you'll get the chance to share your
> >opinions on IT & business topics through brief surveys - and earn cash
> >http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> >_______________________________________________
> >Htmlparser-user mailing list
> >Htm...@li...
> >https://lists.sourceforge.net/lists/listinfo/htmlparser-user
> >
> >
> >
>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>

Re: [Htmlparser-user] Parsing quoted strings seems to be broken in 1.6

[Derrick O. <Der...@Ro...>]

For parsing bad script like this you probably want to set the static 
boolean value org.htmlparser.scanners.ScriptScanner.STRICT to false. See 
the explanation in the ScriptScanner.java file.

sebb wrote:

>The sample script:
>
><HTML>
>  <body>
>  <script>
>  fred = "<img src='a.gif'></img>"
>  </script>
>  </body>
></HTML>
>
>generates the following output from parser.cmd:
>
>Tag (0[0,0],6[0,6]): HTML
>  Txt (6[0,6],10[1,2]): \n
>  Tag (10[1,2],16[1,8]): body
>    Txt (16[1,8],20[2,2]): \n
>    Tag (20[2,2],28[2,10]): script
>      Txt (28[2,10],57[3,27]): \n  fred = "<img src='a.gif'>
>      End (57[3,27],57[3,27]): /script
>    End (57[3,27],63[3,33]): /img
>    Txt (63[3,33],68[4,2]): "\n
>    End (68[4,2],77[4,11]): /script
>    Txt (77[4,11],81[5,2]): \n
>    End (81[5,2],88[5,9]): /body
>  Txt (88[5,9],90[6,0]): \n
>  End (90[6,0],97[6,7]): /HTML
>Txt (97[6,7],101[8,0]): \n\n
>
>It looks like the closing tag is being recognised - though the opening
>tag is not.
>
>Is this a bug, or have I misunderstood something?
>
>-------------------------------------------------------------------------
>Take Surveys. Earn Cash. Influence the Future of IT
>Join SourceForge.net's Techsay panel and you'll get the chance to share your
>opinions on IT & business topics through brief surveys - and earn cash
>http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
>_______________________________________________
>Htmlparser-user mailing list
>Htm...@li...
>https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>
>  
>

[Htmlparser-user] Parsing quoted strings seems to be broken in 1.6

[sebb <se...@gm...>]

The sample script:

<HTML>
  <body>
  <script>
  fred = "<img src='a.gif'></img>"
  </script>
  </body>
</HTML>

generates the following output from parser.cmd:

Tag (0[0,0],6[0,6]): HTML
  Txt (6[0,6],10[1,2]): \n
  Tag (10[1,2],16[1,8]): body
    Txt (16[1,8],20[2,2]): \n
    Tag (20[2,2],28[2,10]): script
      Txt (28[2,10],57[3,27]): \n  fred = "<img src='a.gif'>
      End (57[3,27],57[3,27]): /script
    End (57[3,27],63[3,33]): /img
    Txt (63[3,33],68[4,2]): "\n
    End (68[4,2],77[4,11]): /script
    Txt (77[4,11],81[5,2]): \n
    End (81[5,2],88[5,9]): /body
  Txt (88[5,9],90[6,0]): \n
  End (90[6,0],97[6,7]): /HTML
Txt (97[6,7],101[8,0]): \n\n

It looks like the closing tag is being recognised - though the opening
tag is not.

Is this a bug, or have I misunderstood something?

Re: [Htmlparser-user] running StringBean on Node

[Jeffrey B. <jb...@cs...>]

> > I'd like to be able to run StringBean on a given Node and have it give
> > me all the text from that Node and from its descendants on down the
> > DOM tree.  I can do something similar on the whole page by using a
> > Parser to first get a NodeList of all of the nodes in the tree and
> > then run the following:
> >
> >                 StringExtractor sb = new StringExtractor();
> >                 all_nodes.visitAllNodesWith(sb);
> >
> > Is there a way to either get all of the descendants of a given Node or
> > to otherwise get just the text from all the descendants of a given
> > Node?  Worst case, I can write my own recursive function that will
> > gather up all the Nodes and their children and their descendants - I'm
> > just thinking that there is probably an existing way to do this.
> >
> > Thanks!
> > Jeff
>
> If the Node is a CompositeTag, you can (I think) use CompositeTag.accept(sb).

This worked great - thanks for pointing it out!

I'm sure that accept somehow makes sense as an intuitive name, but it
was lost on me.

-Jeff




> Ian
>
> On 1/7/07, Jeffrey Bigham <jb...@cs...> wrote:
> > Hi,
> >
> >
> > -------------------------------------------------------------------------
> > Take Surveys. Earn Cash. Influence the Future of IT
> > Join SourceForge.net's Techsay panel and you'll get the chance to share your
> > opinions on IT & business topics through brief surveys - and earn cash
> > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> > _______________________________________________
> > Htmlparser-user mailing list
> > Htm...@li...
> > https://lists.sourceforge.net/lists/listinfo/htmlparser-user
> >
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>

Re: [Htmlparser-user] running StringBean on Node

[Ian M. <ian...@gm...>]

For a recursive function that walks the HTMLParser DOM-like structure,
you can use the NodeTreeWalker class and just keep the text nodes.
That's pretty simple.

If the Node is a CompositeTag, you can (I think) use CompositeTag.accept(sb).

Ian

On 1/7/07, Jeffrey Bigham <jb...@cs...> wrote:
> Hi,
>
> I'd like to be able to run StringBean on a given Node and have it give
> me all the text from that Node and from its descendants on down the
> DOM tree.  I can do something similar on the whole page by using a
> Parser to first get a NodeList of all of the nodes in the tree and
> then run the following:
>
>                 StringExtractor sb = new StringExtractor();
>                 all_nodes.visitAllNodesWith(sb);
>
> Is there a way to either get all of the descendants of a given Node or
> to otherwise get just the text from all the descendants of a given
> Node?  Worst case, I can write my own recursive function that will
> gather up all the Nodes and their children and their descendants - I'm
> just thinking that there is probably an existing way to do this.
>
> Thanks!
> Jeff
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>

[Htmlparser-user] running StringBean on Node

[Jeffrey B. <jb...@cs...>]

Hi,

I'd like to be able to run StringBean on a given Node and have it give
me all the text from that Node and from its descendants on down the
DOM tree.  I can do something similar on the whole page by using a
Parser to first get a NodeList of all of the nodes in the tree and
then run the following:

		StringExtractor sb = new StringExtractor();
		all_nodes.visitAllNodesWith(sb);

Is there a way to either get all of the descendants of a given Node or
to otherwise get just the text from all the descendants of a given
Node?  Worst case, I can write my own recursive function that will
gather up all the Nodes and their children and their descendants - I'm
just thinking that there is probably an existing way to do this.

Thanks!
Jeff

Re: [Htmlparser-user] unable to parse a page from ncbi

[Ian M. <ian...@gm...>]

The XML you see in your browser isn't actually XML - it's HTML-encoded
XML. Therefore it's actually text. So:

- Parse the document in HTML Parser, look for that div, then look for
the text nodes within the div.
- You now have the XML as HTML-encoded text, and you have to convert
it into XML. You can convert it in a number of ways, but the easiest
would be to just replace the strings %lt; and &gt; with < and >.
-You'll now have XML, use an XML parser. HTMLParser might be able to
handle it - what you could do is register the various XML tags in
there as CompositeTags in the PrototypicalNodeFactory to make it
easier to deal with.

Ian

On 1/4/07, Jay Bhavsar <kin...@gm...> wrote:
> Hey guys,
>   I have looked through all the examples and javadocs but I am still
> unsuccessful.  Here is want I am tying to do
>
> I would like to follow a link like the following.
>
> http://www.ncbi.nlm.nih.gov/entrez/viewer.fcgi?db=protein&qty=1&c_start=1&list_uids=43366978&uids=&dopt=xml&dispmax=5&sendto=&from=begin&to=end&extrafeatpresent=1&ef_CDD=8&ef_MGC=16&ef_HPRD=32&ef_STS=64&ef_tRNA=128&ef_microRNA=256
>
> This displays a XML format of a report.  I want to parse the XML
> section out of the web page and then parse 3-4 tag from the XML
> section.  The text in XML format is in between <div
> class='recordbody'> ... </div> tag.  I was using
>
> ------------------
> NodeList divs = list.extractAllNodesThatMatch (new TagNameFilter ("TITLE"));
>
>                 NodeIterator i = divs.elements();
>
>                 while (i.hasMoreNodes()){
>                      System.out.println("has more nodes");
>                      processMyNodes(i.nextNode());
>                 }
> --------------------
>
> based on the example from the javadocs.  But anything other than
> HTML in TagNameFilter returns nothing in divs.elements().  (It never
> prints "has more nodes")
>
> Can anyone help extract the XML part from this web page? or is there a
> way I can directly extract what I need from this site without saving
> the XML part first and then using SAX XMLParser to extract it?
>
> Note everything between <div class="recordbody">...</div> is text , it
> not a xml document.  If you view the source you will see what I mean,
> (sorry I don't mean to insult any one's intelligence but I just want
> to be through about my problem.)
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>

[Htmlparser-user] unable to parse a page from ncbi

[Jay B. <kin...@gm...>]

Hey guys,
  I have looked through all the examples and javadocs but I am still
unsuccessful.  Here is want I am tying to do

I would like to follow a link like the following.

http://www.ncbi.nlm.nih.gov/entrez/viewer.fcgi?db=protein&qty=1&c_start=1&list_uids=43366978&uids=&dopt=xml&dispmax=5&sendto=&from=begin&to=end&extrafeatpresent=1&ef_CDD=8&ef_MGC=16&ef_HPRD=32&ef_STS=64&ef_tRNA=128&ef_microRNA=256

This displays a XML format of a report.  I want to parse the XML
section out of the web page and then parse 3-4 tag from the XML
section.  The text in XML format is in between <div
class='recordbody'> ... </div> tag.  I was using

------------------
NodeList divs = list.extractAllNodesThatMatch (new TagNameFilter ("TITLE"));

                NodeIterator i = divs.elements();

                while (i.hasMoreNodes()){
                     System.out.println("has more nodes");
                     processMyNodes(i.nextNode());
                }
--------------------

based on the example from the javadocs.  But anything other than
HTML in TagNameFilter returns nothing in divs.elements().  (It never
prints "has more nodes")

Can anyone help extract the XML part from this web page? or is there a
way I can directly extract what I need from this site without saving
the XML part first and then using SAX XMLParser to extract it?

Note everything between <div class="recordbody">...</div> is text , it
not a xml document.  If you view the source you will see what I mean,
(sorry I don't mean to insult any one's intelligence but I just want
to be through about my problem.)

Re: [Htmlparser-user] extracting text from paragraph tag<p>

[walterwkh <wal...@ho...>]

how about this :
=20
         NodeFilter filter =3D  new NodeClassFilter(ParagraphTag.class)
         list =3D parser.extractAllNodesThatMatch(filter);

-----Original Message-----
From: htm...@li...
[mailto:htm...@li...] On Behalf Of =
Derrick
Oswald
Sent: Thursday, December 14, 2006 12:02 AM
To: htmlparser user list
Subject: Re: [Htmlparser-user] extracting text from paragraph tag<p>


Did you try the StringBean?


----- Original Message ----
From: Khan Khurram Ali <khu...@ya...>
To: htm...@li...
Sent: Wednesday, December 13, 2006 7:11:03 AM
Subject: [Htmlparser-user] extracting text from paragraph tag<p>


hi there !=20
I am new to usin htmlparser , and I need to extract
pure textual content of the webpage like paragraphs in
the webpage. I tried to use paragraph  tag class but
cannot able to work.=20
Is there any easy way. and one  thing the doucments
api is extreamly complicated as I can not understand
it .=20
please do hellp  me out with this..... I just need to
extract the pure text of web page that is inside <p>
tag


thanking for your anticpation in advacnce.

Khan,Khurram=20



_________________________________________________________________________=
___
________
Want to start your own business?
Learn how on Yahoo! Small Business.
HYPERLINK "http://smallbusiness.yahoo.com/r-index"
\nhttp://smallbusiness.yahoo.com/r-index

-------------------------------------------------------------------------=

Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share =
your
opinions on IT & business topics through brief surveys - and earn cash
HYPERLINK
"http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D=
DEVDEV"
\nhttp://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D=
DEVDEV
_______________________________________________
Htmlparser-user mailing list
Htm...@li...
HYPERLINK "https://lists.sourceforge.net/lists/listinfo/htmlparser-user"
\nhttps://lists.sourceforge.net/lists/listinfo/htmlparser-user




--=20
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.5.432 / Virus Database: 268.15.18/584 - Release Date: =
12/12/2006
23:17
=20

Re: [Htmlparser-user] extracting text from paragraph tag<p>

[Derrick O. <der...@ro...>]

Did you try the StringBean?=0A=0A----- Original Message ----=0AFrom: Khan K=
hurram Ali <khu...@ya...>=0ATo: htm...@li...urceforge=
.net=0ASent: Wednesday, December 13, 2006 7:11:03 AM=0ASubject: [Htmlparser=
-user] extracting text from paragraph tag<p>=0A=0Ahi there ! =0AI am new to=
 usin htmlparser , and I need to extract=0Apure textual content of the webp=
age like paragraphs in=0Athe webpage. I tried to use paragraph  tag class b=
ut=0Acannot able to work. =0AIs there any easy way. and one  thing the douc=
ments=0Aapi is extreamly complicated as I can not understand=0Ait . =0Aplea=
se do hellp  me out with this..... I just need to=0Aextract the pure text o=
f web page that is inside <p>=0Atag=0A=0A=0Athanking for your anticpation i=
n advacnce.=0A=0AKhan,Khurram =0A=0A=0A =0A________________________________=
____________________________________________________=0AWant to start your o=
wn business?=0ALearn how on Yahoo! Small Business.=0Ahttp://smallbusiness.y=
ahoo.com/r-index=0A=0A-----------------------------------------------------=
--------------------=0ATake Surveys. Earn Cash. Influence the Future of IT=
=0AJoin SourceForge.net's Techsay panel and you'll get the chance to share =
your=0Aopinions on IT & business topics through brief surveys - and earn ca=
sh=0Ahttp://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=
=3DDEVDEV=0A_______________________________________________=0AHtmlparser-us=
er mailing list=0AH...@li...=0Ahttps://lists.sou=
rceforge.net/lists/listinfo/htmlparser-user=0A=0A=0A=0A=0A

[Htmlparser-user] extracting text from paragraph tag<p>

[Khan K. A. <khu...@ya...>]

hi there ! 
I am new to usin htmlparser , and I need to extract
pure textual content of the webpage like paragraphs in
the webpage. I tried to use paragraph  tag class but
cannot able to work. 
Is there any easy way. and one  thing the doucments
api is extreamly complicated as I can not understand
it . 
please do hellp  me out with this..... I just need to
extract the pure text of web page that is inside <p>
tag


thanking for your anticpation in advacnce.

Khan,Khurram 


 
____________________________________________________________________________________
Want to start your own business?
Learn how on Yahoo! Small Business.
http://smallbusiness.yahoo.com/r-index

[Htmlparser-user] It ready

[Agnes M. <gra...@ab...>]

News Alert!

Fueled by the possibility of an upcoming merger, Wild Brush 
Energy (WBRS) is gearing up for an explosion.  Tension is 
building and soon the scramble to take a position will push 
this one off the charts.

Wild Brush Energy
Symbol: WBRS
Current Price: $0.05
Short Term Target: $0.32
Long Term Target: $0.80

WBRS is engaged in some of the most lucrative gas regions in North 
America.  Major discoveries are happening all the time and WBRS is in 
the thick of it.

With the array of drilling projects Wild Brush has going on at the moment 
tension is building.  As the drilling gets closer to completion insiders are 
accumulating ahead of that major discovery announcement.

Finally the market is ready for explosion
Wednesday December 13 2006. will be a huge growth of WBRS at 1.00 am
Get ready to make some cash today!

[Htmlparser-user] htmlparser and maven

[<ger...@gm...>]

Hello everybody,
I'm trying to move my project under maven 2.0 but I can't find any pom file
for HTML Parser. Should I build it myself ?.... It surely exists one,
doesn't it ?

Best regards

G.D.

Re: [Htmlparser-user] CDATA in script breakes parser?

[Eugeny N D. <bo...@re...>]

> Hi there, I found page: http://www.katzenfinch.com/
> This page contains several links, but HtmlParser does not follow them - in
> general after parsing items it has only head and meta tags available - no body
> tag with links, tables etc.
> 
> Looks like CDATA item inside JavaScript breakes things?
> Could somebody please advice?

I tried to use this code:


import java.io.InputStream;
import java.util.LinkedList;

import org.apache.log4j.Logger;
import org.xml.sax.Attributes;
import org.xml.sax.ErrorHandler;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;
import org.xml.sax.SAXParseException;
import org.xml.sax.helpers.DefaultHandler;

public class SAXHTMLParser extends DefaultHandler {

    private static Logger log = Logger.getLogger(SAXHTMLParser.class);

    public LinkedList parseDocument(InputStream document, String encoding) {
        try {
            org.xml.sax.XMLReader reader = org.xml.sax.helpers.XMLReaderFactory
                    .createXMLReader("org.htmlparser.sax.XMLReader");
            reader.setContentHandler(this);
            reader.setErrorHandler(new MyErrorHandler());
            reader.parse(new InputSource(document));
        } catch (Exception e) {
            log.error(e, e);
        }
        return new LinkedList();
    }

    /**
     *@see org.xml.sax.helpers.DefaultHandler#startElement(java.lang.String, java.lang.String, java.lang.String, org.xml.sax.Attributes)
     */
    public void startElement(String uri, String localName, String qName,
            Attributes attrs) throws SAXException {
        //        if ("img".equalsIgnoreCase(qName) || "a".equalsIgnoreCase(qName)
        //                || "frame".equalsIgnoreCase(qName)
        //                || "title".equalsIgnoreCase(qName)
        //                || "base".equalsIgnoreCase(qName))
        log.debug(localName);
    }

    class MyErrorHandler implements ErrorHandler {

        /**
         *@see org.xml.sax.ErrorHandler#error(org.xml.sax.SAXParseException)
         */
        public void error(SAXParseException arg0) throws SAXException {
            log.error(arg0);
        }

        /**
         *@see org.xml.sax.ErrorHandler#fatalError(org.xml.sax.SAXParseException)
         */
        public void fatalError(SAXParseException arg0) throws SAXException {
            log.error(arg0);
        }

        /**
         *@see org.xml.sax.ErrorHandler#warning(org.xml.sax.SAXParseException)
         */
        public void warning(SAXParseException arg0) throws SAXException {
            log.error(arg0);
        }
        
    }
    
}



and results were

[main] DEBUG SAXHTMLParser - !DOCTYPE
[main] DEBUG SAXHTMLParser - HTML
[main] DEBUG SAXHTMLParser - HEAD
[main] DEBUG SAXHTMLParser - TITLE
[main] DEBUG SAXHTMLParser - META
[main] DEBUG SAXHTMLParser - META
[main] DEBUG SAXHTMLParser - STYLE
[main] DEBUG SAXHTMLParser - SCRIPT



but if I switch to another SAX parser for HTML 

    org.xml.sax.XMLReader reader = org.xml.sax.helpers.XMLReaderFactory
		    .createXMLReader("org.ccil.cowan.tagsoup.Parser");
    reader.setContentHandler(this);
    reader.setErrorHandler(new MyErrorHandler());
    reader.parse(new InputSource(document));

I see this:

 [main] DEBUG .SAXHTMLParser  - html
 [main] DEBUG .SAXHTMLParser  - head
 [main] DEBUG .SAXHTMLParser  - title
 [main] DEBUG .SAXHTMLParser  - meta
 [main] DEBUG .SAXHTMLParser  - meta
 [main] DEBUG .SAXHTMLParser  - style
 [main] DEBUG .SAXHTMLParser  - script
 [main] DEBUG .SAXHTMLParser  - body
 [main] DEBUG .SAXHTMLParser  - div
 [main] DEBUG .SAXHTMLParser  - table
 [main] DEBUG .SAXHTMLParser  - tr
 [main] DEBUG .SAXHTMLParser  - td
 [main] DEBUG .SAXHTMLParser  - td
 [main] DEBUG .SAXHTMLParser  - tr
 [main] DEBUG .SAXHTMLParser  - td
 [main] DEBUG .SAXHTMLParser  - td
 [main] DEBUG .SAXHTMLParser  - p
 [main] DEBUG .SAXHTMLParser  - strong
 [main] DEBUG .SAXHTMLParser  - br
 [main] DEBUG .SAXHTMLParser  - span
 [main] DEBUG .SAXHTMLParser  - td
 [main] DEBUG .SAXHTMLParser  - tr
 [main] DEBUG .SAXHTMLParser  - td
 [main] DEBUG .SAXHTMLParser  - img
 [main] DEBUG .SAXHTMLParser  - td
 [main] DEBUG .SAXHTMLParser  - img
 [main] DEBUG .SAXHTMLParser  - td
 [main] DEBUG .SAXHTMLParser  - img
 [main] DEBUG .SAXHTMLParser  - div
 [main] DEBUG .SAXHTMLParser  - img
 [main] DEBUG .SAXHTMLParser  - div
 [main] DEBUG .SAXHTMLParser  - div
 [main] DEBUG .SAXHTMLParser  - a
 [main] DEBUG .SAXHTMLParser  - img
 [main] DEBUG .SAXHTMLParser  - div
 [main] DEBUG .SAXHTMLParser  - noscript
 [main] DEBUG .SAXHTMLParser  - div
 [main] DEBUG .SAXHTMLParser  - a
 [main] DEBUG .SAXHTMLParser  - img
 [main] DEBUG .SAXHTMLParser  - script
 [main] DEBUG .SAXHTMLParser  - div
 [main] DEBUG .SAXHTMLParser  - a
 [main] DEBUG .SAXHTMLParser  - img
 [main] DEBUG .SAXHTMLParser  - div
 [main] DEBUG .SAXHTMLParser  - a
 [main] DEBUG .SAXHTMLParser  - img
 [main] DEBUG .SAXHTMLParser  - div
 [main] DEBUG .SAXHTMLParser  - a
 [main] DEBUG .SAXHTMLParser  - img
 [main] DEBUG .SAXHTMLParser  - div
 [main] DEBUG .SAXHTMLParser  - a
 [main] DEBUG .SAXHTMLParser  - img
 [main] DEBUG .SAXHTMLParser  - div
 [main] DEBUG .SAXHTMLParser  - a
 [main] DEBUG .SAXHTMLParser  - img
 [main] DEBUG .SAXHTMLParser  - div
 [main] DEBUG .SAXHTMLParser  - a
 [main] DEBUG .SAXHTMLParser  - img
 [main] DEBUG .SAXHTMLParser  - div
 [main] DEBUG .SAXHTMLParser  - a
 [main] DEBUG .SAXHTMLParser  - img
 [main] DEBUG .SAXHTMLParser  - div
 [main] DEBUG .SAXHTMLParser  - a
 [main] DEBUG .SAXHTMLParser  - img
 [main] DEBUG .SAXHTMLParser  - script

So looks like implementation of  a SAX parser in the HtmlParser is a bit
buggy? Is it possible to provide custom SAX parser for HTMLParser library
somehow?

-- 
Eugene N Dzhurinsky

[Htmlparser-user] CDATA in script breakes parser?

[Eugeny N D. <bo...@re...>]

Hi there, I found page: http://www.katzenfinch.com/
This page contains several links, but HtmlParser does not follow them - in
general after parsing items it has only head and meta tags available - no body
tag with links, tables etc.

Looks like CDATA item inside JavaScript breakes things?
Could somebody please advice?

-- 
Eugene N Dzhurinsky

Re: [Htmlparser-user] is htmlparser used only once?

[Derrick O. <Der...@Ro...>]

Collect everything in a NodeList using parse(null), i.e. no filter.
Then filter the NodeList each time using 
NodeList.extractAllNodesThatMatch().


Dave wrote:

> Hi,
>  
> Parser parser = new Parser();
> parser.setResource(http://web-site);
>  
> ...
> NodeList nodes = parser.extractAllNodesThatMatch(filter);
>  
> NodeList nodes1 = parser.extractAllNodesThatMatch(filter);
>  
> The first call is correct, having the right node list.
> but the second call with the same filter returned null.
>  
> I need to use the same parser multiple times without re-parsing the 
> same page.
> parser.reset() will re-parse the same page. What should I do?
>  
> Thanks for help.
>  
> david
>  
>
> ------------------------------------------------------------------------
> Check out the New Yahoo! Mail 
> <http://us.rd.yahoo.com/evt=43257/*http://advision.webevents.yahoo.com/mailbeta>- 
> Fire up a more powerful email and get things done faster.
>
>------------------------------------------------------------------------
>
>-------------------------------------------------------------------------
>Using Tomcat but need to do more? Need to support web services, security?
>Get stuff done quickly with pre-integrated technology to make your job easier
>Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Htmlparser-user mailing list
>Htm...@li...
>https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>  
>