htmlparser-user — The user mailing list for users of the htmlparser library

Re: [Htmlparser-user] Htmlparser-user Digest, Vol 18, Issue 3

[Derrick O. <der...@ro...>]

You could make your own as described here:
http://htmlparser.sourceforge.net/faq.html#composite

----- Original Message ----
From: HAMMER_SHI <dmr...@gm...>
To: htm...@li...
Sent: Sunday, January 6, 2008 8:54:35 PM
Subject: Re: [Htmlparser-user] Htmlparser-user Digest, Vol 18, Issue 3

 hi,all

I want to extract html page Font tags,But I cannot find Find any Font or FontTag in <B>org.htmlparser.tags</B> package.How Can I do.

thanks

Re: [Htmlparser-user] Htmlparser-user Digest, Vol 18, Issue 3

[HAMMER_SHI <dmr...@gm...>]

 hi,all

I want to extract html page Font tags,But I cannot find Find any Font or
FontTag in <B>org.htmlparser.tags</B> package.How Can I do.

thanks

Re: [Htmlparser-user] How if source is in byte form

[Derrick O. <der...@ro...>]

The Page class has a constructor taking an InputStream and an encoding.
You can make an InputStream from a byte array for example.
You need to have stored the encoding somewhere to reconstitute the bytes correctly.
The Parser constructor taking a Lexer constructed from a Page would be what you want.

----- Original Message ----
From: cash cash <ca...@ya...>
To: htmlparser user list <htm...@li...>
Sent: Sunday, January 6, 2008 2:19:23 AM
Subject: [Htmlparser-user] How if source is in byte form

Dear HTMLParser community,

We adapted a web crawler which stores crawled web
pages in byte form. Can HTMLParser take byte as input
and do filtering on HTML tags?

Thank you.




    
  ____________________________________________________________________________________
Never miss a thing.  Make Yahoo your home page. 
http://www.yahoo.com/r/hs

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Htmlparser-user mailing list
Htm...@li...
https://lists.sourceforge.net/lists/listinfo/htmlparser-user

[Htmlparser-user] How if source is in byte form

[cash c. <ca...@ya...>]

Dear HTMLParser community,

We adapted a web crawler which stores crawled web
pages in byte form. Can HTMLParser take byte as input
and do filtering on HTML tags?

Thank you.




      ____________________________________________________________________________________
Never miss a thing.  Make Yahoo your home page. 
http://www.yahoo.com/r/hs

Re: [Htmlparser-user] Javascript

[Derrick O. <der...@ro...>]

No, there is no support for evaluating Javascript.

----- Original Message ----
From: Adi's Gmail <adi...@gm...>
To: htm...@li...
Sent: Sunday, December 30, 2007 12:16:56 PM
Subject: [Htmlparser-user] Javascript





 



Dear HTMLParser 

community,


 


We know that Javascript can be used 

to dynamically load and modify a web page. Suppose I have such code in a 

HTML file:


 


    this is a 

<script>document.write("text");</script>


 


Could the full text "this is a text" 

be extracted instead of "this is a document.write("text");" or "this is a" 

?


This means that Javascript is 

executed first and then the text is extracted from the page. Does HTMLParser 

support this operation?


 


I have tried searching HTMLParser 

documentation but none mention about this. I would really appreciate your 

reply. Thank you.

[Htmlparser-user] Javascript

[Adi's G. <adi...@gm...>]

RGVhciBIVE1MUGFyc2VyIGNvbW11bml0eSwNCg0KV2Uga25vdyB0aGF0IEphdmFzY3JpcHQgY2Fu
IGJlIHVzZWQgdG8gZHluYW1pY2FsbHkgbG9hZCBhbmQgbW9kaWZ5IGEgd2ViIHBhZ2UuIFN1cHBv
c2UgSSBoYXZlIHN1Y2ggY29kZSBpbiBhIEhUTUwgZmlsZToNCg0KICAgIHRoaXMgaXMgYSA8c2Ny
aXB0PmRvY3VtZW50LndyaXRlKCJ0ZXh0Iik7PC9zY3JpcHQ+DQoNCkNvdWxkIHRoZSBmdWxsIHRl
eHQgInRoaXMgaXMgYSB0ZXh0IiBiZSBleHRyYWN0ZWQgaW5zdGVhZCBvZiAidGhpcyBpcyBhIGRv
Y3VtZW50LndyaXRlKCJ0ZXh0Iik7IiBvciAidGhpcyBpcyBhIiA/DQpUaGlzIG1lYW5zIHRoYXQg
SmF2YXNjcmlwdCBpcyBleGVjdXRlZCBmaXJzdCBhbmQgdGhlbiB0aGUgdGV4dCBpcyBleHRyYWN0
ZWQgZnJvbSB0aGUgcGFnZS4gRG9lcyBIVE1MUGFyc2VyIHN1cHBvcnQgdGhpcyBvcGVyYXRpb24/
DQoNCkkgaGF2ZSB0cmllZCBzZWFyY2hpbmcgSFRNTFBhcnNlciBkb2N1bWVudGF0aW9uIGJ1dCBu
b25lIG1lbnRpb24gYWJvdXQgdGhpcy4gSSB3b3VsZCByZWFsbHkgYXBwcmVjaWF0ZSB5b3VyIHJl
cGx5LiBUaGFuayB5b3Uu

Re: [Htmlparser-user] Non-charset=utf-8 Characters in Parsed Text?

[Jeffery B. <jef...@gm...>]

Thank you Derrick,

That worked perfectly.


On Dec 12, 2007 10:52 PM, Derrick Oswald <der...@ro...> wrote:

> You can create a class extending org.htmlparser.tags.MetaTag and
> overriding doSemanticAction () to do nothing.
> Register this with a org.htmlparser.PrototypicalNodeFactory you assign to
> your parser as described here<http://htmlparser.sourceforge.net/faq.html#composite>
> .
>
>
> ----- Original Message ----
> From: Jeffery Brewer <jef...@gm...>
> To: htmlparser user list <htm...@li...>
> Sent: Wednesday, December 12, 2007 9:31:16 PM
> Subject: Re: [Htmlparser-user] Non-charset=utf-8 Characters in Parsed
> Text?
>
> Thanks Karsten,
>
> I have now read the FAQ and have spent some time trying to solve my
> problem. I'm learning a lot more about the parser but haven't solved my
> problem yet.
>
> The pages I'm trying to read have a meta tag setting the encoding to
> UTF-8...
>     <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
>
> but they are obviously using a different character set (they shouldn't
> be!).
>
> If I copy the page and modify the tag for windows-1252 encoding...
>
>     <meta http-equiv="Content-Type" content="text/html;
> charset=windows-1252">
>
> and parse the page, I can recover the characters and convert them.
>
> Likewise, if I omit that meta tag and set the parser for windows-1252
> encoding I can also recover the characters and convert them.
>
> But if I set the parser for windows-1252 encoding and then have it parse
> the page from the website, the parser reads the utf-8 encoding tag and and
> automatically parses the page using utf-8 encoding.
>
> In other words, if I do this...
>
>      Parser parser = new Parser ("http://www.examiner.com/a-1097821~County_plan_could_double_neighborhood_enforcement.html
>
> <http://www.examiner.com/a-1097821%7ECounty_plan_could_double_neighborhood_enforcement.html>
> ");
>      parser.setEncoding("windows-1252");
>      System.out.println("encoding=" + parser.getEncoding());
>      NodeList divNodeList = parser.parse(new HasAttributeFilter("id",
> "article_main"));
>      System.out.println("encoding=" + parser.getEncoding());
>
> it prints out
> encoding=windows-1252
> encoding=UTF-8
>
> I wonder if it's possible to have the parser ignore the meta tag, or if
> it's somehow possible to alter or delete the meta tag before the site is
> parsed or if there is a better approach?
>
>
>
> On Dec 12, 2007 12:28 AM, Karsten Ohme <wid...@t-...> wrote:
>
> > Jeffery Brewer schrieb:
> > > I'm running into an issue where I'm getting question mark characters
> > in
> > > place of quotes, apostrophes, hyphens, etc.
> >
> > Have you read the FAQ?
> >
> > http://htmlparser.sourceforge.net/faq.html
> >
> > The "Why am I getting an EncodingChangeException?" should be helpful how
> > to handle character encoding issues. If the web page does not contain an
> >
> > encoding hint, let the parser fetch the web site for you, maybe the HTTP
> > header contains the correct encoding. So it is used. If the web site is
> > offline, set the correct encoding in the parser. Does this help?
> >
> > Regards,
> > Karsten
> >
> > >
> > > I know this has to do with the website using characters outside those
> > > defined by the specification. Is there a way to correct this in the
> > > htmlparser? I started trying to do a simple character replacement on
> > the
> > > parsed text, but whenever I do an "(int) string.charAt(n)" for any
> > special
> > > character I'm getting a 65533, and if I do a "
> > Character.getNumericValue(
> > > string.charAt(n))" I'm getting a -1, so I'm assuming I'm far to far
> > > "downstream" to fix the problem.
> > >
> > > Also I've just been using the Parser.parse method to return nodelists
> > and
> > > have been working my way through the documents that way rather than
> > try any
> > > of the other htmlparser features (which may already account for
> > this??).
> > >
> > > Thanks in advance for any help. I'm really enjoying working with the
> > parser
> > > and thanks to everyone who built this thing.
> > >
> > > Jeff
> > >
> > >
> > >
> > >
> > ------------------------------------------------------------------------
> > >
> > >
> > -------------------------------------------------------------------------
> > > SF.Net email is sponsored by:
> > > Check out the new SourceForge.net Marketplace.
> > > It's the best place to buy or sell services for
> > > just about anything Open Source.
> > > http://sourceforge.net/services/buy/index.php
> > >
> > >
> > >
> > ------------------------------------------------------------------------
> > >
> > > _______________________________________________
> > > Htmlparser-user mailing list
> > > Htm...@li...
> > > https://lists.sourceforge.net/lists/listinfo/htmlparser-user
> >
> >
> >
> > -------------------------------------------------------------------------
> > SF.Net email is sponsored by:
> > Check out the new SourceForge.net Marketplace.
> > It's the best place to buy or sell services for
> > just about anything Open Source.
> > http://sourceforge.net/services/buy/index.php
> > _______________________________________________
> > Htmlparser-user mailing list
> > Htm...@li...
> > https://lists.sourceforge.net/lists/listinfo/htmlparser-user
> >
>
>
>
> -------------------------------------------------------------------------
> SF.Net email is sponsored by:
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services
> for just about anything Open Source.
>
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>
>

Re: [Htmlparser-user] Non-charset=utf-8 Characters in Parsed Text?

[Derrick O. <der...@ro...>]

You can create a class extending org.htmlparser.tags.MetaTag and overriding doSemanticAction () to do nothing.
Register this with a org.htmlparser.PrototypicalNodeFactory  you assign to your parser as described here.

----- Original Message ----
From: Jeffery Brewer <jef...@gm...>
To: htmlparser user list <htm...@li...>
Sent: Wednesday, December 12, 2007 9:31:16 PM
Subject: Re: [Htmlparser-user] Non-charset=utf-8 Characters in Parsed Text?

Thanks Karsten,

I have now read the FAQ and have spent some time trying to solve my problem. I'm learning a lot more about the parser but haven't solved my problem yet.

The pages I'm trying to read have a meta tag setting the encoding to UTF-8...

    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">

but they are obviously using a different character set (they shouldn't be!).

If I copy the page and modify the tag for windows-1252 encoding...



    <meta http-equiv="Content-Type" content="text/html; charset=windows-1252">

and parse the page, I can recover the characters and convert them.

Likewise, if I omit that meta tag and set the parser for windows-1252 encoding I can also recover the characters and convert them.


But if I set the parser for windows-1252 encoding and then have it parse the page from the website, the parser reads the utf-8 encoding tag and and automatically parses the page using utf-8 encoding. 

In other words, if I do this...


     Parser parser = new Parser ("http://www.examiner.com/a-1097821~County_plan_could_double_neighborhood_enforcement.html
");
     parser.setEncoding("windows-1252");
     System.out.println("encoding=" + parser.getEncoding());
     NodeList divNodeList = parser.parse(new HasAttributeFilter("id", "article_main"));

     System.out.println("encoding=" + parser.getEncoding());    

it prints out
encoding=windows-1252
encoding=UTF-8

I wonder if it's possible to have the parser ignore the meta tag, or if it's somehow possible to alter or delete the meta tag before the site is parsed or if there is a better approach?




On Dec 12, 2007 12:28 AM, Karsten Ohme <wid...@t-...> wrote:

Jeffery Brewer schrieb:
> I'm running into an issue where I'm getting question mark characters in
> place of quotes, apostrophes, hyphens, etc.


Have you read the FAQ?


http://htmlparser.sourceforge.net/faq.html

The "Why am I getting an EncodingChangeException?" should be helpful how
to handle character encoding issues. If the web page does not contain an

encoding hint, let the parser fetch the web site for you, maybe the HTTP
header contains the correct encoding. So it is used. If the web site is
offline, set the correct encoding in the parser. Does this help?


Regards,
Karsten

>
> I know this has to do with the website using characters outside those
> defined by the specification. Is there a way to correct this in the
> htmlparser? I started trying to do a simple character replacement on the

> parsed text, but whenever I do an "(int) string.charAt(n)" for any special
> character I'm getting a 65533, and if I do a "Character.getNumericValue(
> string.charAt(n))" I'm getting a -1, so I'm assuming I'm far to far

> "downstream" to fix the problem.
>
> Also I've just been using the Parser.parse method to return nodelists and
> have been working my way through the documents that way rather than try any

> of the other htmlparser features (which may already account for this??).
>
> Thanks in advance for any help. I'm really enjoying working with the parser
> and thanks to everyone who built this thing.

>
> Jeff
>
>
>

> ------------------------------------------------------------------------
>
> -------------------------------------------------------------------------

> SF.Net email is sponsored by:
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> 
http://sourceforge.net/services/buy/index.php
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Htmlparser-user mailing list

> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user



-------------------------------------------------------------------------
SF.Net email is sponsored by:
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for

just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Htmlparser-user mailing list

Htm...@li...
https://lists.sourceforge.net/lists/listinfo/htmlparser-user

Re: [Htmlparser-user] Non-charset=utf-8 Characters in Parsed Text?

[Jeffery B. <jef...@gm...>]

Thanks Karsten,

I have now read the FAQ and have spent some time trying to solve my problem.
I'm learning a lot more about the parser but haven't solved my problem yet.

The pages I'm trying to read have a meta tag setting the encoding to
UTF-8...
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">

but they are obviously using a different character set (they shouldn't be!).

If I copy the page and modify the tag for windows-1252 encoding...

    <meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">

and parse the page, I can recover the characters and convert them.

Likewise, if I omit that meta tag and set the parser for windows-1252
encoding I can also recover the characters and convert them.

But if I set the parser for windows-1252 encoding and then have it parse the
page from the website, the parser reads the utf-8 encoding tag and and
automatically parses the page using utf-8 encoding.

In other words, if I do this...

     Parser parser = new Parser ("
http://www.examiner.com/a-1097821~County_plan_could_double_neighborhood_enforcement.html
");
     parser.setEncoding("windows-1252");
     System.out.println("encoding=" + parser.getEncoding());
     NodeList divNodeList = parser.parse(new HasAttributeFilter("id",
"article_main"));
     System.out.println("encoding=" + parser.getEncoding());

it prints out
encoding=windows-1252
encoding=UTF-8

I wonder if it's possible to have the parser ignore the meta tag, or if it's
somehow possible to alter or delete the meta tag before the site is parsed
or if there is a better approach?



On Dec 12, 2007 12:28 AM, Karsten Ohme <wid...@t-...> wrote:

> Jeffery Brewer schrieb:
> > I'm running into an issue where I'm getting question mark characters in
> > place of quotes, apostrophes, hyphens, etc.
>
> Have you read the FAQ?
>
> http://htmlparser.sourceforge.net/faq.html
>
> The "Why am I getting an EncodingChangeException?" should be helpful how
> to handle character encoding issues. If the web page does not contain an
> encoding hint, let the parser fetch the web site for you, maybe the HTTP
> header contains the correct encoding. So it is used. If the web site is
> offline, set the correct encoding in the parser. Does this help?
>
> Regards,
> Karsten
>
> >
> > I know this has to do with the website using characters outside those
> > defined by the specification. Is there a way to correct this in the
> > htmlparser? I started trying to do a simple character replacement on the
> > parsed text, but whenever I do an "(int) string.charAt(n)" for any
> special
> > character I'm getting a 65533, and if I do a "Character.getNumericValue(
> > string.charAt(n))" I'm getting a -1, so I'm assuming I'm far to far
> > "downstream" to fix the problem.
> >
> > Also I've just been using the Parser.parse method to return nodelists
> and
> > have been working my way through the documents that way rather than try
> any
> > of the other htmlparser features (which may already account for this??).
> >
> > Thanks in advance for any help. I'm really enjoying working with the
> parser
> > and thanks to everyone who built this thing.
> >
> > Jeff
> >
> >
> >
> > ------------------------------------------------------------------------
> >
> >
> -------------------------------------------------------------------------
> > SF.Net email is sponsored by:
> > Check out the new SourceForge.net Marketplace.
> > It's the best place to buy or sell services for
> > just about anything Open Source.
> > http://sourceforge.net/services/buy/index.php
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Htmlparser-user mailing list
> > Htm...@li...
> > https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>
>
> -------------------------------------------------------------------------
> SF.Net email is sponsored by:
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>

Re: [Htmlparser-user] Non-charset=utf-8 Characters in Parsed Text?

[Karsten O. <wid...@t-...>]

Jeffery Brewer schrieb:
> I'm running into an issue where I'm getting question mark characters in
> place of quotes, apostrophes, hyphens, etc.

Have you read the FAQ?

http://htmlparser.sourceforge.net/faq.html

The "Why am I getting an EncodingChangeException?" should be helpful how
to handle character encoding issues. If the web page does not contain an
encoding hint, let the parser fetch the web site for you, maybe the HTTP
header contains the correct encoding. So it is used. If the web site is
offline, set the correct encoding in the parser. Does this help?

Regards,
Karsten

> 
> I know this has to do with the website using characters outside those
> defined by the specification. Is there a way to correct this in the
> htmlparser? I started trying to do a simple character replacement on the
> parsed text, but whenever I do an "(int) string.charAt(n)" for any special
> character I'm getting a 65533, and if I do a "Character.getNumericValue(
> string.charAt(n))" I'm getting a -1, so I'm assuming I'm far to far
> "downstream" to fix the problem.
> 
> Also I've just been using the Parser.parse method to return nodelists and
> have been working my way through the documents that way rather than try any
> of the other htmlparser features (which may already account for this??).
> 
> Thanks in advance for any help. I'm really enjoying working with the parser
> and thanks to everyone who built this thing.
> 
> Jeff
> 
> 
> 
> ------------------------------------------------------------------------
> 
> -------------------------------------------------------------------------
> SF.Net email is sponsored by: 
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user

[Htmlparser-user] Non-charset=utf-8 Characters in Parsed Text?

[Jeffery B. <jef...@gm...>]

I'm running into an issue where I'm getting question mark characters in
place of quotes, apostrophes, hyphens, etc.

I know this has to do with the website using characters outside those
defined by the specification. Is there a way to correct this in the
htmlparser? I started trying to do a simple character replacement on the
parsed text, but whenever I do an "(int) string.charAt(n)" for any special
character I'm getting a 65533, and if I do a "Character.getNumericValue(
string.charAt(n))" I'm getting a -1, so I'm assuming I'm far to far
"downstream" to fix the problem.

Also I've just been using the Parser.parse method to return nodelists and
have been working my way through the documents that way rather than try any
of the other htmlparser features (which may already account for this??).

Thanks in advance for any help. I'm really enjoying working with the parser
and thanks to everyone who built this thing.

Jeff

Re: [Htmlparser-user] scanning / parsing bug?

[Subramanya S. <sa...@cs...>]

Thank you very much!  That fixed it!

Is there a list of such flags and what they do?  This is the second such 
flag I had to set in recent times to handle the wild HTML that exists 
out there.  I might as well figure out what they do and set as many as 
needed right now since I am parsing all kinds of HTML found on the web.

-S.
> I believe you want to set the static member for strict remark parsing 
> to false:
>   org.htmlparser.lexer.Lexer.STRICT_REMARKS = false;
>

Re: [Htmlparser-user] scanning / parsing bug?

[Derrick O. <der...@ro...>]

I believe you want to set the static member for strict remark parsing to false:
  org.htmlparser.lexer.Lexer.STRICT_REMARKS = false;

----- Original Message ----
From: Subramanya Sastry <sa...@cs...>
To: htmlparser user list <htm...@li...>
Sent: Tuesday, December 11, 2007 5:02:08 PM
Subject: [Htmlparser-user] scanning / parsing bug?

For this url, 
http://www.washingtonpost.com/wp-dyn/content/article/2007/12/10/AR2007121001600.html 
(and maybe other washington post urls), I wonder if HTML Parser is 
running into a bug.

The HTML source for this page has the following block of HTML in the 
middle ..

    <!---------------- End New Comments Box ------------------>
    <div class="sidebarhack"><b></b></div>
    ....
    ....
    </div>
    <!-- sphereit end -->
    <br clear="all">

The parser is ignoring all content from the start of the line 'End New 
Comments Box' till 'sphereit end' ... I wonder if this is because of
 the 
lack of a space before the '-->' closing comment string in the first 
line ... I tested the code by adding a space manually at that point,
 and 
sure enough, the block of HTML in the middle is correctly recognized.

Is there a workaround for this?  I am also willing to download the 
source code and incorporate a fix, if necessary.

Thanks,
Subbu.

-------------------------------------------------------------------------
SF.Net email is sponsored by: 
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Htmlparser-user mailing list
Htm...@li...
https://lists.sourceforge.net/lists/listinfo/htmlparser-user

[Htmlparser-user] scanning / parsing bug?

[Subramanya S. <sa...@cs...>]

For this url, 
http://www.washingtonpost.com/wp-dyn/content/article/2007/12/10/AR2007121001600.html 
(and maybe other washington post urls), I wonder if HTML Parser is 
running into a bug.

The HTML source for this page has the following block of HTML in the 
middle ..

    <!---------------- End New Comments Box ------------------>
    <div class="sidebarhack"><b></b></div>
    ....
    ....
    </div>
    <!-- sphereit end -->
    <br clear="all">

The parser is ignoring all content from the start of the line 'End New 
Comments Box' till 'sphereit end' ... I wonder if this is because of the 
lack of a space before the '-->' closing comment string in the first 
line ... I tested the code by adding a space manually at that point, and 
sure enough, the block of HTML in the middle is correctly recognized.

Is there a workaround for this?  I am also willing to download the 
source code and incorporate a fix, if necessary.

Thanks,
Subbu.

Re: [Htmlparser-user] TagNameFilter

[Derrick O. <der...@ro...>]

The accept() method is used by the visitor pattern, not the filter paradigm.

It's not clear what string you are trying to 'do not match'.
The <p> tag has no string. Maybe you mean the string between <p> tags,
or if you've made the <p> tag composite, then maybe it's children.

You should probably just add more filtering clauses, e.g.
  parser.extractAllNodesThatMatch(new AndFilter (new TagNameFilter("p"), ...));

I would suggest you try the FilterBuilder application to build up your filter.

----- Original Message ----
From: "at...@gm..." <at...@gm...>
To: htmlparser user list <htm...@li...>
Sent: Thursday, December 6, 2007 2:09:07 PM
Subject: [Htmlparser-user] TagNameFilter

Hi,

i need some help with the TagNameFilter.

I have a function to get all the p tags out a html document.


NodeList nl = parser.extractAllNodesThatMatch(new TagNameFilter("p"));

But now i want to filter from the NodeList all entries that do not
 match a  
special string.
I guess the key would be the "accept() function" but im unsure how to  
implement it(well the string compare etc is clear but the usage of the
  
accept() + Tag.class ).
And furthermore i have problems with doubled entries because of nested
 p  
tags.
Thanks
Alex

-------------------------------------------------------------------------
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell.  From the desktop to the data center, Linux is going
mainstream.  Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
_______________________________________________
Htmlparser-user mailing list
Htm...@li...
https://lists.sourceforge.net/lists/listinfo/htmlparser-user

Re: [Htmlparser-user] Transformation limitations?

[Tom H. <thj...@ri...>]

Hi Derrick,

That doesn't sound too bad at all to me.  I created a single utility 
function that handles that for any tag I need to inject:

    void setEndTag(Tag tag)
    {
        TagNode endTag = new TagNode();
        endTag.setTagName("/" + tag.getTagName());
        tag.setEndTag(endTag);
    }

That works like a charm!

Thanks,
Tom

Derrick Oswald wrote:
> You will need to add your own end tag to the script tag you are 
> injecting. I believe it's something like this:
>   TagNode end = new TagNode ();
>   end.setTagName ("/SCRIPT");
>   script.setEndTag (end)
>
> I guess this could be made much easier.
>
> ----- Original Message ----
> From: Tom Hjellming <thj...@ri...>
> To: htm...@li...
> Sent: Thursday, December 6, 2007 3:19:22 AM
> Subject: [Htmlparser-user] Transformation limitations?
>
> I'm experimenting with the HtmlParser library to see if I can use it to
> transform webpages.  One thing I'm trying is to see if I can inject some
> javascript into the HTML page.
>
> My test app uses the PrototypicalNodeFactory to register some overridden
> tags like MyHeadTag and MyBodyTag (which derive from the HeadTag and
> BodyTag classes respectively) and then I run the parser.  I then locate
> the MyHeadTag object found during the parsing and do the following:
>
>   ScriptTag script = new ScriptTag();
>   script.setAttribute("SRC", "blah.js");
>   script.setLanguage("javascript");
>
>   NodeList childNodes = headTag.getChildren();
>   childNodes.add(script);
>
> I then loop through the parser-generated listHtmlNodes calling toHtml()
> on each node and appending the result in a StringBuffer.
>
> But looking at the resulting StringBuffer contents, I see that the
> <script> tag is not terminated with a </script>:
>
> <html>
> <head>
> <title>Testing...</title>
> <SCRIPT LANGUAGE=javascript SRC="blah.js">
> </head>
> <body>
> <p>Testing...</p>
> </body>
> </html>
>
> All the other tags that were in the original HTML file with end tags are
> fine.  It is just the newly injected ScriptTag that is not properly
> terminated.
>
> This happens with any "container" tag I try to insert into the
> parser-generated "DOM" tree.
>
> Does anyone know why?  Any hints on how to fix this?  Is this an
> unreasonable thing to do with HtmlParser?
>
> thanks,
> Tom
>
>
> -------------------------------------------------------------------------
> SF.Net email is sponsored by: The Future of Linux Business White Paper
> from Novell.  From the desktop to the data center, Linux is going
> mainstream.  Let it simplify your IT future.
> http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li... 
> <mailto:Htm...@li...>
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>
> ------------------------------------------------------------------------
>
> -------------------------------------------------------------------------
> SF.Net email is sponsored by: The Future of Linux Business White Paper
> from Novell.  From the desktop to the data center, Linux is going
> mainstream.  Let it simplify your IT future.
> http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
> ------------------------------------------------------------------------
>
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>   

[Htmlparser-user] TagNameFilter

[<at...@gm...>]

Hi,

i need some help with the TagNameFilter.

I have a function to get all the p tags out a html document.


NodeList nl =3D parser.extractAllNodesThatMatch(new TagNameFilter("p"));=


But now i want to filter from the NodeList all entries that do not match=
 a  =

special string.
I guess the key would be the "accept() function" but im unsure how to  =

implement it(well the string compare etc is clear but the usage of the  =

accept() + Tag.class ).
And furthermore i have problems with doubled entries because of nested p=
  =

tags.
Thanks
Alex

Re: [Htmlparser-user] Transformation limitations?

[Derrick O. <der...@ro...>]

You will need to add your own end tag to the script tag you are injecting. I believe it's something like this:
  TagNode end = new TagNode ();
  end.setTagName ("/SCRIPT");
  script.setEndTag (end)

I guess this could be made much easier.

----- Original Message ----
From: Tom Hjellming <thj...@ri...>
To: htm...@li...
Sent: Thursday, December 6, 2007 3:19:22 AM
Subject: [Htmlparser-user] Transformation limitations?

I'm experimenting with the HtmlParser library to see if I can use it to
 
transform webpages.  One thing I'm trying is to see if I can inject
 some 
javascript into the HTML page.

My test app uses the PrototypicalNodeFactory to register some
 overridden 
tags like MyHeadTag and MyBodyTag (which derive from the HeadTag and 
BodyTag classes respectively) and then I run the parser.  I then locate
 
the MyHeadTag object found during the parsing and do the following:

   ScriptTag script = new ScriptTag();
   script.setAttribute("SRC", "blah.js");
   script.setLanguage("javascript");

   NodeList childNodes = headTag.getChildren();
   childNodes.add(script);

I then loop through the parser-generated listHtmlNodes calling toHtml()
 
on each node and appending the result in a StringBuffer.

But looking at the resulting StringBuffer contents, I see that the 
<script> tag is not terminated with a </script>:

<html>
<head>
<title>Testing...</title>
<SCRIPT LANGUAGE=javascript SRC="blah.js">
</head>
<body>
<p>Testing...</p>
</body>
</html>

All the other tags that were in the original HTML file with end tags
 are 
fine.  It is just the newly injected ScriptTag that is not properly 
terminated.

This happens with any "container" tag I try to insert into the 
parser-generated "DOM" tree.

Does anyone know why?  Any hints on how to fix this?  Is this an 
unreasonable thing to do with HtmlParser?

thanks,
Tom


-------------------------------------------------------------------------
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell.  From the desktop to the data center, Linux is going
mainstream.  Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
_______________________________________________
Htmlparser-user mailing list
Htm...@li...
https://lists.sourceforge.net/lists/listinfo/htmlparser-user

[Htmlparser-user] Transformation limitations?

[Tom H. <thj...@ri...>]

I'm experimenting with the HtmlParser library to see if I can use it to 
transform webpages.  One thing I'm trying is to see if I can inject some 
javascript into the HTML page.

My test app uses the PrototypicalNodeFactory to register some overridden 
tags like MyHeadTag and MyBodyTag (which derive from the HeadTag and 
BodyTag classes respectively) and then I run the parser.  I then locate 
the MyHeadTag object found during the parsing and do the following:

   ScriptTag script = new ScriptTag();
   script.setAttribute("SRC", "blah.js");
   script.setLanguage("javascript");

   NodeList childNodes = headTag.getChildren();
   childNodes.add(script);

I then loop through the parser-generated listHtmlNodes calling toHtml() 
on each node and appending the result in a StringBuffer.

But looking at the resulting StringBuffer contents, I see that the 
<script> tag is not terminated with a </script>:

<html>
<head>
<title>Testing...</title>
<SCRIPT LANGUAGE=javascript SRC="blah.js">
</head>
<body>
<p>Testing...</p>
</body>
</html>

All the other tags that were in the original HTML file with end tags are 
fine.  It is just the newly injected ScriptTag that is not properly 
terminated.

This happens with any "container" tag I try to insert into the 
parser-generated "DOM" tree.

Does anyone know why?  Any hints on how to fix this?  Is this an 
unreasonable thing to do with HtmlParser?

thanks,
Tom

Re: [Htmlparser-user] does someone have a simple example of printing out all nodes of a document

[Derrick O. <der...@ro...>]

Printing out the top level node generates the entire html again.
NodeLists also understand toHtml().

So for a parse like,
  NodeList list = parser.parse (null);
the entire page is printed out with:
  System.out.println (list.toHtml ());
or
  for (int i = 0; i < list.size(); i++)
    System.out.println (list.elementAt(i).toHtml ());

You should look at the toHtml() method of CompositeTag if you don't want it to print the nested tags.

----- Original Message ----
From: Randy Paries <rtp...@gm...>
To: htm...@li...
Sent: Friday, November 23, 2007 3:55:06 PM
Subject: [Htmlparser-user] does someone have a simple example of printing out all nodes of a document

hello,
this should not be as hard as i am making this, but i have a brain
 lock.

I need a simple function to parse a html file and print out each of its
 nodes

i am espesiailly having problems with nested nodes. Like tables within
divs within divs.

it needs to prints everything so, i could take the output and display
the same html page.

but it needs to go to the level where there are no children. I hope
this make sense.

thanks for any help someone can give me
randy

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Htmlparser-user mailing list
Htm...@li...
https://lists.sourceforge.net/lists/listinfo/htmlparser-user

Re: [Htmlparser-user] does someone have a simple example of printing out all nodes of a document

[Randy P. <rtp...@gm...>]

More Details.

Using the lexer will not help me cause i need fully formed tags.

thanks again

On Nov 23, 2007 5:55 PM, Randy Paries <rtp...@gm...> wrote:
> hello,
> this should not be as hard as i am making this, but i have a brain lock.
>
> I need a simple function to parse a html file and print out each of its nodes
>
> i am espesiailly having problems with nested nodes. Like tables within
> divs within divs.
>
> it needs to prints everything so, i could take the output and display
> the same html page.
>
> but it needs to go to the level where there are no children. I hope
> this make sense.
>
> thanks for any help someone can give me
> randy
>

[Htmlparser-user] does someone have a simple example of printing out all nodes of a document

[Randy P. <rtp...@gm...>]

hello,
this should not be as hard as i am making this, but i have a brain lock.

I need a simple function to parse a html file and print out each of its nodes

i am espesiailly having problems with nested nodes. Like tables within
divs within divs.

it needs to prints everything so, i could take the output and display
the same html page.

but it needs to go to the level where there are no children. I hope
this make sense.

thanks for any help someone can give me
randy

Re: [Htmlparser-user] Attribute of a tag

[Derrick O. <der...@ro...>]

string value = tag.getAttribute("<name>");

----- Original Message ----
From: Ali <to...@ya...>
To: htm...@li...
Sent: Friday, November 23, 2007 11:29:07 AM
Subject: [Htmlparser-user] Attribute of a tag

hi, how can i get a attribute of tage?
thanks



    
  ____________________________________________________________________________________
Be a better pen pal. 
Text or chat with friends inside Yahoo! Mail. See how.
  http://overview.mail.yahoo.com/

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Htmlparser-user mailing list
Htm...@li...
https://lists.sourceforge.net/lists/listinfo/htmlparser-user

[Htmlparser-user] Attribute of a tag

[Ali <to...@ya...>]

hi, how can i get a attribute of tage?
thanks



      ____________________________________________________________________________________
Be a better pen pal. 
Text or chat with friends inside Yahoo! Mail. See how.  http://overview.mail.yahoo.com/

Re: [Htmlparser-user] Link Location resolving

[Jurgen V. <ri...@pl...>]

Thanks that worked.

Jurgen

Derrick Oswald wrote:
>
> You should be able to use the Page.setBaseUrl (string base) method to 
> set the URL used as a prefix for relative links, i.e.
>   parser.getLexer ().getPage ().setBaseUrl ("http://yadda.yadda");
>
>
> ----- Original Message ----
> From: Jurgen Voorneveld <j.e...@st...>
> To: htm...@li...
> Sent: Friday, November 23, 2007 11:13:33 AM
> Subject: [Htmlparser-user] Link Location resolving
>
> List,
>
> I've recently started using htmlparser as part of a webspidering tool
> that I have written and I've run into a small problem.
> My spider downloads files from webservers using HttpClient from the
> Apache Commons project. These files are then stored locally in a
> temporary location. If a file contains HTML it is then parsed by 
> htmlparser.
> During parsing the parser resolves relative links to other files by
> adding the location of the file to the relative link. Which of course
> completely screws up the links. Is there any way to turn this feature
> off or some way of telling the parser that the location of the data is
> not where it gets the data from.
>
> thanks
> Jurgen Voorneveld
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li... 
> <mailto:Htm...@li...>
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>
> ------------------------------------------------------------------------
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> ------------------------------------------------------------------------
>
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>