htmlparser-user — The user mailing list for users of the htmlparser library

[Htmlparser-user] NoClassDefFoundError: sun/security/action/GetPropertyAction

[Lou-yahoo <vas...@ya...>]

Hi folks,

I'd like to use the parser API for some pretty straightforward html file 
parsing, but when I try your simple example (although I did have to change 
Enumeration to HTMLEnumeration)...

	public static void main(String[] args) {
		try{
			System.out.println("testpoint 0");
			HTMLParser parser = 
				new HTMLParser("http://www.yahoo.com");
			//HTMLParser parser = 
			//	new HTMLParser("http://www.yahoo.com",new DefaultHTMLParserFeedback());
			System.out.println("testpoint 1");
			System.exit(0);
			// In this example, we are registering all the common scanners
			parser.registerScanners(); 
			for (HTMLEnumeration e = parser.elements();e.hasMoreNodes();) {
			    HTMLNode node = (HTMLNode)e.nextHTMLNode();
  	      	    	node.print();
			}
		}		
		catch (Exception e) {
			System.out.println("Exception Thrown:" + e.getMessage() + " : " + e);
			e.printStackTrace();
		}
		System.out.println("Done! yay");
	}

I get 

NoClassDefFoundError: sun/security/action/GetPropertyAction

What am I doin wrong?


Thanks,
Lou

[Htmlparser-user] Added to Users

[Somik R. <so...@ya...>]

Leslie and Derrick have been added to the list of users in cvs. Claude, =
Kaarle and Dhaval are also on this list.
You can add cvs watches by doing :
cvs watch add * (I think), and you will be automatically emailed when =
anyone does a cvs update.

Regards,
Somik

[Htmlparser-user] Candidate 5 is out

[Somik R. <so...@ya...>]

Hi Folks,
    This week's release is Candidate 5. We've had talented developers
joining us over the weekend, hence, you can expect improvements in quality
in the coming weeks. Hopefully, we should have our production release ready
by New Year's...

From the change log :
Integration Build 1.2 - 20021208
---------------------------------
[1] Fixed bug in base href scanner - would always expect href
[2] Refactored HTMLFormScanner
[3] Refactored HTMLRenderer to use the Visitor pattern- enabling
connections with links and images
[4] HTMLStringNode returns a blank string in toPlainTextString()
[5] HTMLFormTag returns string information in toPlainTextString()

#5 is an important fix as now, we wont lose any meaningful string info
contained inside forms when we issue calls like node.toPlainTextString().

Get the latest release from http://htmlparser.sourceforge.net

The site update is continuing at an even pace. There is a new section on
writing tests for HTMLParser. We're also trying to introduce a philosophy
called "Communicate with TestCases". If you've found a bug, write a testcase
for it, and submit that in your report. Of course, you dont have to do this,
but if you do, we'd be able to make the fix much faster (and motivated to
make the fix). Writing a testcase for the parser is super simple - you can
check the philosophy and an example on the documentation page.

http://htmlparser.sourceforge.net/design/index.html

Regards,
Somik

Re: [Htmlparser-user] how to deal with form tag following table tag

[Somik R. <so...@ya...>]

Hi Leslie,
    Indeed, the <form> tag is a nightmare to work with. At one point, we had
removed it from the basic set of scanners. We put it back in after our
exception handling mechanism was in place - so now, if things get messy, you
should get an exception. We can't possibly handle every bit of screwed html
:), although we try really hard to.

> it would be better if the end-form tag could be 'assumed' so that the
> file could at least be parsed.  that would mirror the behavior of
commercial
> browsers.

I had spent some time on this, tried it and failed miserably. It turned out
to be almost impossible to predict where a form tag should end, bcos of its
expanse, and particularly bcos of its intermingling with <table>.

Its quite possible I missed something - so if you have any innovative
suggestions, it would be really helpful. I think we're dealing on the realm
of AI here :). Remember, there is a big constraint, we have a streaming,
real-time parser and not a DOM style parser where we have all of it and can
go back and forth. A good heuristic that really works will make our day.

Bytway, maybe this discussion could better happen on the dev list.. Feel
free to join us as a dev (send me your sourceforge id).

Regards,
Somik
----- Original Message -----
From: "Leslie Rohde" <le...@op...>
To: <htm...@li...>
Sent: Thursday, December 05, 2002 5:25 PM
Subject: Re: [Htmlparser-user] how to deal with form tag following table tag


> actually, there are two problems in the case at hand, and i am not at all
> sure that the <table><form> construction is the worst of them.
>
> not only does hotbot produce this invalid sequence, but they also
> failed to close the form tag.  it looks like HTMLFormScanner
> simply falls out of the loop at lines 136-154 looking for the end tag and
> throws an exception when not found.
>
> it would be better if the end-form tag could be 'assumed' so that the
> file could at least be parsed.  that would mirror the behavior of
commercial
> browsers.
>
> Leslie Rohde wrote:
>
> > the construction <table ...><form...> is not allowed in spec, but it
> > does occur in such places as the hotbot search engine results page.
> > currently, htmlparser delivers a flood errors and exceptions when
> > parsing a hotbot results page.
> >
> > how best to handle this?
> >
>
> --
> Leslie Rohde
> mailto:le...@op...
> http://www.optitext.com
>
>
>
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user

Re: [Htmlparser-user] how to deal with form tag following table tag

[Leslie R. <le...@op...>]

actually, there are two problems in the case at hand, and i am not at all
sure that the <table><form> construction is the worst of them.

not only does hotbot produce this invalid sequence, but they also
failed to close the form tag.  it looks like HTMLFormScanner
simply falls out of the loop at lines 136-154 looking for the end tag and
throws an exception when not found.

it would be better if the end-form tag could be 'assumed' so that the
file could at least be parsed.  that would mirror the behavior of commercial
browsers.

Leslie Rohde wrote:

> the construction <table ...><form...> is not allowed in spec, but it
> does occur in such places as the hotbot search engine results page.
> currently, htmlparser delivers a flood errors and exceptions when
> parsing a hotbot results page.
>
> how best to handle this?
>

-- 
Leslie Rohde
mailto:le...@op...
http://www.optitext.com

[Htmlparser-user] how to deal with form tag following table tag

[Leslie R. <le...@op...>]

the construction <table ...><form...> is not allowed in spec, but it
does occur in such places as the hotbot search engine results page.
currently, htmlparser delivers a flood errors and exceptions when
parsing a hotbot results page.

how best to handle this?

-- 
Leslie Rohde
mailto:le...@op...
http://www.optitext.com

RE: [Htmlparser-user] Extracting Images

[<dha...@or...>]

Navid,

You can easily get around this problem by removing the HTMLFormScanner
from the scan list. The HTMLFormScanner is automatically registered when
u use HTMLParser.registerScanners(). Use the removeScanner function to
remove this scanner and then proceed with your code.

Regards,

Dhaval Udani
Senior Analyst
M-Line, QPEG
OrbiTech Solutions Ltd.
+91-22-8290019 Extn. 1457



-----Original Message-----
From: nav911t [mailto:na...@ya...]
Sent: Thursday, December 05, 2002 10:24 AM
To: htmlparser-user
Cc: nav911t
Subject: Re: [Htmlparser-user] Extracting Images


Hi Somik,
Thank you. I these examples are helpful. But the
problem is, that the image tags between <form> and
</form>, get skipped, I can not extract them! Once I
remove <form> and </form>, I can extract all images
tags, but it is not so when I put them back.

Navid
--- Somik Raha <so...@ya...> wrote:
> Hi Navid,
>     If you want to extract images - you need to read
> this :
>
http://htmlparser.sourceforge.net/samples/imageslinks.html
>     If you want to extract text b/w two images,
> thats something quite
> different. Use toPlainTextString() to get the data
> out.
> 
> Regards
> Somik
> ----- Original Message -----
> From: "Navid H.Langaroudi" <na...@ya...>
> To: <htm...@li...>
> Sent: Wednesday, December 04, 2002 4:33 PM
> Subject: Re: [Htmlparser-user] Candidate Release 4
> is out
> 
> 
> > Hi Somik,
> > I am trying to extract all images of a page. I can
> > only extract part of it. I do not know what I am
> > missing.
> > Actually what I am doing is, trying to extract all
> > text that appears between two images.
> > Any suggestion?
> >
> > Thanks,
> > Navid
> > --- Somik Raha <so...@ya...> wrote:
> > > Hi Folks,
> > >     Candidate Release 4 is out. This actually
> > > contains a few minor API changes which wont
> affect
> > > your application, but have been done to improve
> the
> > > OO design of the system. HTMLFormScanner has
> been
> > > improved. The major work in this release went in
> > > refactoring 201 testcases - so as to make it
> more
> > > readable, and follow the Once-And-Only-Once
> > > paradigm. Well, the package size dropped about
> 12KB
> > > (after zipping), so you can estimate how much
> > > refactoring was done.. All tests are passing.
> > >
> > >     From the Change Log,
> > >
> > > Integration Build 1.2 - 20021201
> > > --------------------------------
> > > [1] Refactored HTMLNode, API improved, now
> HTMLNode
> > > stores
> > > nodeBegin and nodeEnd.
> > > [2] Refactored Testing framework - to reduce the
> > > code size substantially.
> > > [3] HTMLFormScanner improved to include
> > > Input,TextArea, Select and Option scanners
> within
> > >
> > > You can get it from
> > > http://htmlparser.sourceforge.net
> > > There's an all-new Contributors Page (linked
> from
> > > the main site). Just in case I missed anybody,
> or
> > > you have info to add, pls let me know.
> > >
> > > Regards,
> > > Somik
> >
> >
> > __________________________________________________
> > Do you Yahoo!?
> > Yahoo! Mail Plus - Powerful. Affordable. Sign up
> now.
> > http://mailplus.yahoo.com
> >
> >
> >
>
-------------------------------------------------------
> > This SF.net email is sponsored by: Microsoft
> Visual Studio.NET
> > comprehensive development tool, built to increase
> your
> > productivity. Try a free online hosted session at:
> >
>
http://ads.sourceforge.net/cgi-bin/redirect.pl?micr0003en
> > _______________________________________________
> > Htmlparser-user mailing list
> > Htm...@li...
> >
>
https://lists.sourceforge.net/lists/listinfo/htmlparser-user
> 
> 
> 
>
-------------------------------------------------------
> This SF.net email is sponsored by: Microsoft Visual
> Studio.NET 
> comprehensive development tool, built to increase
> your 
> productivity. Try a free online hosted session at:
>
http://ads.sourceforge.net/cgi-bin/redirect.pl?micr0003en
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
>
https://lists.sourceforge.net/lists/listinfo/htmlparser-user


__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com


-------------------------------------------------------
This SF.net email is sponsored by: Microsoft Visual Studio.NET 
comprehensive development tool, built to increase your 
productivity. Try a free online hosted session at:
http://ads.sourceforge.net/cgi-bin/redirect.pl?micr0003en
_______________________________________________
Htmlparser-user mailing list
Htm...@li...
https://lists.sourceforge.net/lists/listinfo/htmlparser-user

Re: [Htmlparser-user] How to parse a forms nodes

[Navid H.L. <na...@ya...>]

Thank you.

--- Somik Raha <so...@ya...> wrote:
> for (Enumeration
>
e=formTag.getAllNodesVector().elements();e.hasMoreElements();)
> {
>     HTMLNode node = (HTMLNode)e.nextElement();
>     if (node instanceof HTMLImageTag) {
>         // ...
>     }
> }
> ----- Original Message -----
> From: "Navid H.Langaroudi" <na...@ya...>
> To: <htm...@li...>
> Sent: Wednesday, December 04, 2002 6:15 PM
> Subject: [Htmlparser-user] How to parse a forms
> nodes
> 
> 
> > Hi Somik,
> > I know that HTMLFormTag has a method
> > getAllNodesVector()  that returns all nodes of
> form
> > tag. But how can I use this vector to extract
> Tags,
> > like image tags.
> >
> > Thanks
> > Navid
> > --- Somik Raha <so...@ya...> wrote:
> > > Hi Folks,
> > >     Candidate Release 4 is out. This actually
> > > contains a few minor API changes which wont
> affect
> > > your application, but have been done to improve
> the
> > > OO design of the system. HTMLFormScanner has
> been
> > > improved. The major work in this release went in
> > > refactoring 201 testcases - so as to make it
> more
> > > readable, and follow the Once-And-Only-Once
> > > paradigm. Well, the package size dropped about
> 12KB
> > > (after zipping), so you can estimate how much
> > > refactoring was done.. All tests are passing.
> > >
> > >     From the Change Log,
> > >
> > > Integration Build 1.2 - 20021201
> > > --------------------------------
> > > [1] Refactored HTMLNode, API improved, now
> HTMLNode
> > > stores
> > > nodeBegin and nodeEnd.
> > > [2] Refactored Testing framework - to reduce the
> > > code size substantially.
> > > [3] HTMLFormScanner improved to include
> > > Input,TextArea, Select and Option scanners
> within
> > >
> > > You can get it from
> > > http://htmlparser.sourceforge.net
> > > There's an all-new Contributors Page (linked
> from
> > > the main site). Just in case I missed anybody,
> or
> > > you have info to add, pls let me know.
> > >
> > > Regards,
> > > Somik
> >
> >
> > __________________________________________________
> > Do you Yahoo!?
> > Yahoo! Mail Plus - Powerful. Affordable. Sign up
> now.
> > http://mailplus.yahoo.com
> >
> >
> >
>
-------------------------------------------------------
> > This SF.net email is sponsored by: Microsoft
> Visual Studio.NET
> > comprehensive development tool, built to increase
> your
> > productivity. Try a free online hosted session at:
> >
>
http://ads.sourceforge.net/cgi-bin/redirect.pl?micr0003en
> > _______________________________________________
> > Htmlparser-user mailing list
> > Htm...@li...
> >
>
https://lists.sourceforge.net/lists/listinfo/htmlparser-user
> 
> 
> 
>
-------------------------------------------------------
> This SF.net email is sponsored by: Microsoft Visual
> Studio.NET 
> comprehensive development tool, built to increase
> your 
> productivity. Try a free online hosted session at:
>
http://ads.sourceforge.net/cgi-bin/redirect.pl?micr0003en
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
>
https://lists.sourceforge.net/lists/listinfo/htmlparser-user


__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com

Re: [Htmlparser-user] Extracting Images

[Navid H.L. <na...@ya...>]

Hi Somik,
Thank you. I these examples are helpful. But the
problem is, that the image tags between <form> and
</form>, get skipped, I can not extract them! Once I
remove <form> and </form>, I can extract all images
tags, but it is not so when I put them back.

Navid
--- Somik Raha <so...@ya...> wrote:
> Hi Navid,
>     If you want to extract images - you need to read
> this :
>
http://htmlparser.sourceforge.net/samples/imageslinks.html
>     If you want to extract text b/w two images,
> thats something quite
> different. Use toPlainTextString() to get the data
> out.
> 
> Regards
> Somik
> ----- Original Message -----
> From: "Navid H.Langaroudi" <na...@ya...>
> To: <htm...@li...>
> Sent: Wednesday, December 04, 2002 4:33 PM
> Subject: Re: [Htmlparser-user] Candidate Release 4
> is out
> 
> 
> > Hi Somik,
> > I am trying to extract all images of a page. I can
> > only extract part of it. I do not know what I am
> > missing.
> > Actually what I am doing is, trying to extract all
> > text that appears between two images.
> > Any suggestion?
> >
> > Thanks,
> > Navid
> > --- Somik Raha <so...@ya...> wrote:
> > > Hi Folks,
> > >     Candidate Release 4 is out. This actually
> > > contains a few minor API changes which wont
> affect
> > > your application, but have been done to improve
> the
> > > OO design of the system. HTMLFormScanner has
> been
> > > improved. The major work in this release went in
> > > refactoring 201 testcases - so as to make it
> more
> > > readable, and follow the Once-And-Only-Once
> > > paradigm. Well, the package size dropped about
> 12KB
> > > (after zipping), so you can estimate how much
> > > refactoring was done.. All tests are passing.
> > >
> > >     From the Change Log,
> > >
> > > Integration Build 1.2 - 20021201
> > > --------------------------------
> > > [1] Refactored HTMLNode, API improved, now
> HTMLNode
> > > stores
> > > nodeBegin and nodeEnd.
> > > [2] Refactored Testing framework - to reduce the
> > > code size substantially.
> > > [3] HTMLFormScanner improved to include
> > > Input,TextArea, Select and Option scanners
> within
> > >
> > > You can get it from
> > > http://htmlparser.sourceforge.net
> > > There's an all-new Contributors Page (linked
> from
> > > the main site). Just in case I missed anybody,
> or
> > > you have info to add, pls let me know.
> > >
> > > Regards,
> > > Somik
> >
> >
> > __________________________________________________
> > Do you Yahoo!?
> > Yahoo! Mail Plus - Powerful. Affordable. Sign up
> now.
> > http://mailplus.yahoo.com
> >
> >
> >
>
-------------------------------------------------------
> > This SF.net email is sponsored by: Microsoft
> Visual Studio.NET
> > comprehensive development tool, built to increase
> your
> > productivity. Try a free online hosted session at:
> >
>
http://ads.sourceforge.net/cgi-bin/redirect.pl?micr0003en
> > _______________________________________________
> > Htmlparser-user mailing list
> > Htm...@li...
> >
>
https://lists.sourceforge.net/lists/listinfo/htmlparser-user
> 
> 
> 
>
-------------------------------------------------------
> This SF.net email is sponsored by: Microsoft Visual
> Studio.NET 
> comprehensive development tool, built to increase
> your 
> productivity. Try a free online hosted session at:
>
http://ads.sourceforge.net/cgi-bin/redirect.pl?micr0003en
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
>
https://lists.sourceforge.net/lists/listinfo/htmlparser-user


__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com

Re: [Htmlparser-user] How to parse a forms nodes

[Somik R. <so...@ya...>]

for (Enumeration
e=formTag.getAllNodesVector().elements();e.hasMoreElements();) {
    HTMLNode node = (HTMLNode)e.nextElement();
    if (node instanceof HTMLImageTag) {
        // ...
    }
}
----- Original Message -----
From: "Navid H.Langaroudi" <na...@ya...>
To: <htm...@li...>
Sent: Wednesday, December 04, 2002 6:15 PM
Subject: [Htmlparser-user] How to parse a forms nodes


> Hi Somik,
> I know that HTMLFormTag has a method
> getAllNodesVector()  that returns all nodes of form
> tag. But how can I use this vector to extract Tags,
> like image tags.
>
> Thanks
> Navid
> --- Somik Raha <so...@ya...> wrote:
> > Hi Folks,
> >     Candidate Release 4 is out. This actually
> > contains a few minor API changes which wont affect
> > your application, but have been done to improve the
> > OO design of the system. HTMLFormScanner has been
> > improved. The major work in this release went in
> > refactoring 201 testcases - so as to make it more
> > readable, and follow the Once-And-Only-Once
> > paradigm. Well, the package size dropped about 12KB
> > (after zipping), so you can estimate how much
> > refactoring was done.. All tests are passing.
> >
> >     From the Change Log,
> >
> > Integration Build 1.2 - 20021201
> > --------------------------------
> > [1] Refactored HTMLNode, API improved, now HTMLNode
> > stores
> > nodeBegin and nodeEnd.
> > [2] Refactored Testing framework - to reduce the
> > code size substantially.
> > [3] HTMLFormScanner improved to include
> > Input,TextArea, Select and Option scanners within
> >
> > You can get it from
> > http://htmlparser.sourceforge.net
> > There's an all-new Contributors Page (linked from
> > the main site). Just in case I missed anybody, or
> > you have info to add, pls let me know.
> >
> > Regards,
> > Somik
>
>
> __________________________________________________
> Do you Yahoo!?
> Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
> http://mailplus.yahoo.com
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Microsoft Visual Studio.NET
> comprehensive development tool, built to increase your
> productivity. Try a free online hosted session at:
> http://ads.sourceforge.net/cgi-bin/redirect.pl?micr0003en
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user

Re: [Htmlparser-user] Extracting Images

[Somik R. <so...@ya...>]

Hi Navid,
    If you want to extract images - you need to read this :
http://htmlparser.sourceforge.net/samples/imageslinks.html
    If you want to extract text b/w two images, thats something quite
different. Use toPlainTextString() to get the data out.

Regards
Somik
----- Original Message -----
From: "Navid H.Langaroudi" <na...@ya...>
To: <htm...@li...>
Sent: Wednesday, December 04, 2002 4:33 PM
Subject: Re: [Htmlparser-user] Candidate Release 4 is out


> Hi Somik,
> I am trying to extract all images of a page. I can
> only extract part of it. I do not know what I am
> missing.
> Actually what I am doing is, trying to extract all
> text that appears between two images.
> Any suggestion?
>
> Thanks,
> Navid
> --- Somik Raha <so...@ya...> wrote:
> > Hi Folks,
> >     Candidate Release 4 is out. This actually
> > contains a few minor API changes which wont affect
> > your application, but have been done to improve the
> > OO design of the system. HTMLFormScanner has been
> > improved. The major work in this release went in
> > refactoring 201 testcases - so as to make it more
> > readable, and follow the Once-And-Only-Once
> > paradigm. Well, the package size dropped about 12KB
> > (after zipping), so you can estimate how much
> > refactoring was done.. All tests are passing.
> >
> >     From the Change Log,
> >
> > Integration Build 1.2 - 20021201
> > --------------------------------
> > [1] Refactored HTMLNode, API improved, now HTMLNode
> > stores
> > nodeBegin and nodeEnd.
> > [2] Refactored Testing framework - to reduce the
> > code size substantially.
> > [3] HTMLFormScanner improved to include
> > Input,TextArea, Select and Option scanners within
> >
> > You can get it from
> > http://htmlparser.sourceforge.net
> > There's an all-new Contributors Page (linked from
> > the main site). Just in case I missed anybody, or
> > you have info to add, pls let me know.
> >
> > Regards,
> > Somik
>
>
> __________________________________________________
> Do you Yahoo!?
> Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
> http://mailplus.yahoo.com
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Microsoft Visual Studio.NET
> comprehensive development tool, built to increase your
> productivity. Try a free online hosted session at:
> http://ads.sourceforge.net/cgi-bin/redirect.pl?micr0003en
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user

[Htmlparser-user] How to parse a forms nodes

[Navid H.L. <na...@ya...>]

Hi Somik,
I know that HTMLFormTag has a method
getAllNodesVector()  that returns all nodes of form
tag. But how can I use this vector to extract Tags,
like image tags.

Thanks
Navid
--- Somik Raha <so...@ya...> wrote:
> Hi Folks,
>     Candidate Release 4 is out. This actually
> contains a few minor API changes which wont affect
> your application, but have been done to improve the
> OO design of the system. HTMLFormScanner has been
> improved. The major work in this release went in
> refactoring 201 testcases - so as to make it more
> readable, and follow the Once-And-Only-Once
> paradigm. Well, the package size dropped about 12KB
> (after zipping), so you can estimate how much
> refactoring was done.. All tests are passing.
> 
>     From the Change Log, 
> 
> Integration Build 1.2 - 20021201
> --------------------------------
> [1] Refactored HTMLNode, API improved, now HTMLNode
> stores
> nodeBegin and nodeEnd.
> [2] Refactored Testing framework - to reduce the
> code size substantially.
> [3] HTMLFormScanner improved to include
> Input,TextArea, Select and Option scanners within
> 
> You can get it from
> http://htmlparser.sourceforge.net
> There's an all-new Contributors Page (linked from
> the main site). Just in case I missed anybody, or
> you have info to add, pls let me know.
> 
> Regards,
> Somik


__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com

Re: [Htmlparser-user] Candidate Release 4 is out

[Navid H.L. <na...@ya...>]

Hi Somik,
I am trying to extract all images of a page. I can
only extract part of it. I do not know what I am
missing.
Actually what I am doing is, trying to extract all
text that appears between two images. 
Any suggestion?

Thanks,
Navid
--- Somik Raha <so...@ya...> wrote:
> Hi Folks,
>     Candidate Release 4 is out. This actually
> contains a few minor API changes which wont affect
> your application, but have been done to improve the
> OO design of the system. HTMLFormScanner has been
> improved. The major work in this release went in
> refactoring 201 testcases - so as to make it more
> readable, and follow the Once-And-Only-Once
> paradigm. Well, the package size dropped about 12KB
> (after zipping), so you can estimate how much
> refactoring was done.. All tests are passing.
> 
>     From the Change Log, 
> 
> Integration Build 1.2 - 20021201
> --------------------------------
> [1] Refactored HTMLNode, API improved, now HTMLNode
> stores
> nodeBegin and nodeEnd.
> [2] Refactored Testing framework - to reduce the
> code size substantially.
> [3] HTMLFormScanner improved to include
> Input,TextArea, Select and Option scanners within
> 
> You can get it from
> http://htmlparser.sourceforge.net
> There's an all-new Contributors Page (linked from
> the main site). Just in case I missed anybody, or
> you have info to add, pls let me know.
> 
> Regards,
> Somik


__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com

Re: [Htmlparser-user] Base tag problem

[Somik R. <so...@ya...>]

Hi Marcin
    Thanks for the bug report - it has been reproduced and fixed. Bytway,
when submitting reports, could you login at sourceforge- else, the automatic
response mechanism of the bug reporting system will not be able to initimate
you of our responses.
    You can expect this fix in the next release (coming week).

Regards,
Somik
----- Original Message -----
From: "Marcin Pionnier" <mar...@so...>
To: <htm...@li...>
Sent: Monday, December 02, 2002 5:08 AM
Subject: [Htmlparser-user] Base tag problem


> Hi
>
> When I was parsing pages from google.com directory I found a problem:
> Inside a document there is such line:
> <base target="_top">
>
> but HtmlBaseHREFScanner assumes that there is HREF paramater:
> (Line 68) HTMLBaseHREFScanner.java)
> String baseUrl = (String)tag.getParameter("HREF");
> String absoluteBaseUrl = removeLastSlash(baseUrl.trim());
>
> so I got null pointer exception (of course it is very easy to fix it)
>
> Is that line from google malformed HTML or it is a bug in parser?
>
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user

Re: [Htmlparser-user] Base tag problem

[Somik R. <so...@ya...>]

That sounds like a bug. Can you enter it in at
http://htmlparser.sourceforge.net ?

Thanks a lot.

Regards
Somik
--- Marcin Pionnier <mar...@so...>
wrote:
> Hi
> 
> When I was parsing pages from google.com directory I
> found a problem:
> Inside a document there is such line:
> <base target="_top">
> 
> but HtmlBaseHREFScanner assumes that there is HREF
> paramater:
> (Line 68) HTMLBaseHREFScanner.java)
> String baseUrl = (String)tag.getParameter("HREF");
> String absoluteBaseUrl =
> removeLastSlash(baseUrl.trim());
> 
> so I got null pointer exception (of course it is
> very easy to fix it) 
> 
> Is that line from google malformed HTML or it is a
> bug in parser?
> 
> 
> 
>
-------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
>
https://lists.sourceforge.net/lists/listinfo/htmlparser-user


__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com

[Htmlparser-user] Base tag problem

[Marcin P. <mar...@so...>]

Hi

When I was parsing pages from google.com directory I found a problem:
Inside a document there is such line:
<base target="_top">

but HtmlBaseHREFScanner assumes that there is HREF paramater:
(Line 68) HTMLBaseHREFScanner.java)
String baseUrl = (String)tag.getParameter("HREF");
String absoluteBaseUrl = removeLastSlash(baseUrl.trim());

so I got null pointer exception (of course it is very easy to fix it) 

Is that line from google malformed HTML or it is a bug in parser?

[Htmlparser-user] Candidate Release 4 is out

[Somik R. <so...@ya...>]

Hi Folks,
    Candidate Release 4 is out. This actually contains a few minor API =
changes which wont affect your application, but have been done to =
improve the OO design of the system. HTMLFormScanner has been improved. =
The major work in this release went in refactoring 201 testcases - so as =
to make it more readable, and follow the Once-And-Only-Once paradigm. =
Well, the package size dropped about 12KB (after zipping), so you can =
estimate how much refactoring was done.. All tests are passing.

    From the Change Log,=20

Integration Build 1.2 - 20021201
--------------------------------
[1] Refactored HTMLNode, API improved, now HTMLNode stores
nodeBegin and nodeEnd.
[2] Refactored Testing framework - to reduce the code size =
substantially.
[3] HTMLFormScanner improved to include Input,TextArea, Select and =
Option scanners within

You can get it from http://htmlparser.sourceforge.net
There's an all-new Contributors Page (linked from the main site). Just =
in case I missed anybody, or you have info to add, pls let me know.

Regards,
Somik

Re: [Htmlparser-user] How to use it.

[Amit R. <ami...@ya...>]

> > Well, now I reached to the real difficult part. I
> am
> > trying to extract meaningful data from some sites,
> > like a site products names, discription and
> keywords,
> > which is not always in meta tags.
> > Do have any suggestion?
> 
> You might do well to use Artificial Intelligence
> (whatever that is). :)
> If you define your goal more clearly, it would be
> easier to help you. 

i used Regular expressions for a similar job.

__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com

Re: [Htmlparser-user] How to use it.

[Somik R. <so...@ya...>]

> Well, now I reached to the real difficult part. I am
> trying to extract meaningful data from some sites,
> like a site products names, discription and keywords,
> which is not always in meta tags.
> Do have any suggestion?

You might do well to use Artificial Intelligence (whatever that is). :)
If you define your goal more clearly, it would be easier to help you. 

Regards,
Somik

Re: [Htmlparser-user] How to use it.

[Navid H.L. <na...@ya...>]

Thank you Somik, It worked as I wanted. 
Also using the parser classes, I could extract each
part of page data separatly.
Well, now I reached to the real difficult part. I am
trying to extract meaningful data from some sites,
like a site products names, discription and keywords,
which is not always in meta tags.
Do have any suggestion?

Navid

--- Somik Raha <so...@ya...> wrote:
> Hi Navid,
>   I ran the program, and it does exactly what I
> expected.
>   But I see your doubt now. You want to suppress the
> exception messages. These are happening because of
> DefaultHMLParserFeedback(). Pls write your own
> NullHTMLParserFeedback(), that does not print
> anything
> when it encounters an error, and use that to
> initialize the parser. Read the javadoc of
> HTMLParser.java carefully.
> 
>  I have written the modified program for you :
> public void	testNullUrl(){
>     try {
>         parser = new
> HTMLParser("http://www.yahooeeeeee.com",new
> HTMLParserFeedback() {
> 			/**
> 			 * @see
> org.htmlparser.util.HTMLParserFeedback#info(String)
> 			 */
> 			public void info(String message) {
> 			}
> 
> 			/**
> 			 * @see
>
org.htmlparser.util.HTMLParserFeedback#warning(String)
> 			 */
> 			public void warning(String message) {
> 			}
> 
> 			/**
> 			 * @see
> org.htmlparser.util.HTMLParserFeedback#error(String,
> HTMLParserException)
> 			 */
> 			public void error(String message,
> HTMLParserException e) {
> 			}
> 		});
>         //assertTrue("Should have thrown an
> exception!",false);
> 		parser.registerScanners();
> 		parser.addScanner(new HTMLLinkScanner("-l"));
> 
>     }
>     catch (HTMLParserException e) {
> 	System.out.println("Can not connect the URL!");
>     }
> }
> 
> Try this- it should give you what you want now.
> 
> Regards
> Somik
> 
> (The earlier mail went before I could complete it..)
> 
> __________________________________________________
> Do you Yahoo!?
> Yahoo! Mail Plus – Powerful. Affordable. Sign up
> now.
> http://mailplus.yahoo.com
> 
> 
>
-------------------------------------------------------
> This SF.net email is sponsored by: Get the new Palm
> Tungsten T 
> handheld. Power & Color in a compact size! 
>
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0002en
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
>
https://lists.sourceforge.net/lists/listinfo/htmlparser-user


__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com

[Htmlparser-user] Candidate 3 is out

[Somik R. <so...@ya...>]

Hi Folks,
     Candidate 3 is out. You can get it from
http://htmlparser.sourceforge.net
    The website is getting an overhaul, though this is in progress. You =
will
find a new samples page.

    If anyone wishes to contribute a simple program to add to the =
catalog,
please feel free to come forward.

From the change log, in this release :

Integration Build 1.2 - 20021125
--------------------------------
[1] Incorporated Bug Fix for HTMLLinkProcessor to parse dynamic urls
[2] Refactored package names to org.htmlparser
[3] Added documentation
[4] Can handle url with spaces in it
[5] Fixed bug 643352 - going into infinite loop on bad img within link
[6] Refactored HTMLLinkTag - unnecessary boolean variables removed

Developers --> can you send me a brief bio, with a pic - I'd like to =
acknowledge everyone who has contributed to this project.=20

Regards,
Somik

Re: [Htmlparser-user] How to use it.

[Somik R. <so...@ya...>]

Hi Navid,
  I ran the program, and it does exactly what I
expected.
  But I see your doubt now. You want to suppress the
exception messages. These are happening because of
DefaultHMLParserFeedback(). Pls write your own
NullHTMLParserFeedback(), that does not print anything
when it encounters an error, and use that to
initialize the parser. Read the javadoc of
HTMLParser.java carefully.

 I have written the modified program for you :
public void	testNullUrl(){
    try {
        parser = new
HTMLParser("http://www.yahooeeeeee.com",new
HTMLParserFeedback() {
			/**
			 * @see
org.htmlparser.util.HTMLParserFeedback#info(String)
			 */
			public void info(String message) {
			}

			/**
			 * @see
org.htmlparser.util.HTMLParserFeedback#warning(String)
			 */
			public void warning(String message) {
			}

			/**
			 * @see
org.htmlparser.util.HTMLParserFeedback#error(String,
HTMLParserException)
			 */
			public void error(String message,
HTMLParserException e) {
			}
		});
        //assertTrue("Should have thrown an
exception!",false);
		parser.registerScanners();
		parser.addScanner(new HTMLLinkScanner("-l"));

    }
    catch (HTMLParserException e) {
	System.out.println("Can not connect the URL!");
    }
}

Try this- it should give you what you want now.

Regards
Somik

(The earlier mail went before I could complete it..)

__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus – Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com

Re: [Htmlparser-user] How to use it.

[Somik R. <so...@ya...>]

Hi Navid,
  I ran the program, and it does exactly what I
expected.
  But I see your doubt now. You want to suppress the
exception messages. These are happening because of
DefaultHMLParserFeedback(). Pls write your own
NullHTMLParserFeedback(), that does not print anything
when it encounters an error, and use that to
initialize the parser. Read the javadoc of
HTMLParser.java carefully.



> --- Somik Raha <so...@ya...> wrote:
> > Hi Navid,
> > 
> > 
> > > I still need your help.
> > > I am getting some exception errors in my
> program,
> > it
> > > happens when it tried to open a url to a non
> > existing
> > > page.
> > > I tried some try/catch, but still can't catch
> this
> > > one.
> > > It says
> > > Error: HTMLParser.oprnURLConnection(): Error in
> > > opening a URL connection to
> > > http://www.somenoneexistingurl.com .......
> > >
> > > How can I skip this, my program reads all urls
> of
> > a
> > > site and tries to go to next page, and if the
> url
> > does
> > > not exist or is wrong, it terminats, I think I
> > should
> > > control this in order to let the program carry
> on
> > on
> > > correct links.
> > 
> > I wrote a testcase for this - in
> > HTMLParserTest.java. This test proves that
> > there is no bug in the parser. You can add this
> > snippet and verify for
> > yourself.
> > 
> > public void testNullUrl() {
> >     HTMLParser parser;
> >     try {
> >         parser = new
> > HTMLParser("http://someoneexisting.com");
> >         assertTrue("Should have thrown an
> > exception!",false);
> >     }
> >     catch (HTMLParserException e) {
> >     }
> > }
> > 
> > I can guess what you might be doing wrong though
> ..
> > are you sure you are
> > trying to trap HTMLParserException ?
> > 
> > Regards,
> > Somik
> > 
> > 
> > 
> >
>
-------------------------------------------------------
> > This sf.net email is sponsored by:ThinkGeek
> > Welcome to geek heaven.
> > http://thinkgeek.com/sf
> > _______________________________________________
> > Htmlparser-user mailing list
> > Htm...@li...
> >
>
https://lists.sourceforge.net/lists/listinfo/htmlparser-user
> 
> 
> 
> __________________________________________________
> Do you Yahoo!?
> Yahoo! Mail Plus – Powerful. Affordable. Sign up
> now.
> http://mailplus.yahoo.com


__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus – Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com

Re: [Htmlparser-user] How to use it.

[Navid H.L. <na...@ya...>]

Hi Somik,
I am still getting this error.
Here I send a sample code, which if you run, you will
get the error.
Please see the attachment.

Thanks
Navid

--- Somik Raha <so...@ya...> wrote:
> Hi Navid,
> 
> 
> > I still need your help.
> > I am getting some exception errors in my program,
> it
> > happens when it tried to open a url to a non
> existing
> > page.
> > I tried some try/catch, but still can't catch this
> > one.
> > It says
> > Error: HTMLParser.oprnURLConnection(): Error in
> > opening a URL connection to
> > http://www.somenoneexistingurl.com .......
> >
> > How can I skip this, my program reads all urls of
> a
> > site and tries to go to next page, and if the url
> does
> > not exist or is wrong, it terminats, I think I
> should
> > control this in order to let the program carry on
> on
> > correct links.
> 
> I wrote a testcase for this - in
> HTMLParserTest.java. This test proves that
> there is no bug in the parser. You can add this
> snippet and verify for
> yourself.
> 
> public void testNullUrl() {
>     HTMLParser parser;
>     try {
>         parser = new
> HTMLParser("http://someoneexisting.com");
>         assertTrue("Should have thrown an
> exception!",false);
>     }
>     catch (HTMLParserException e) {
>     }
> }
> 
> I can guess what you might be doing wrong though ..
> are you sure you are
> trying to trap HTMLParserException ?
> 
> Regards,
> Somik
> 
> 
> 
>
-------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
>
https://lists.sourceforge.net/lists/listinfo/htmlparser-user



__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus – Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com

[Htmlparser-user] Re: Another oddity

[Somik R. <so...@ya...>]

Hi Steve,
    Im taking this to the list..
    You cant use URLEncoder bcos URLEncoder replaces spaces with +. =
However, what works on the browser is %20. OTOH, there are still =
problems with replacing spaces with %20 when you come to mailto links.=20
    For your case though, the parser now checks if a url has spaces, and =
if so, it performs the correct encoding.=20
    The latest codebase is in CVS.=20

Cheers,
Somik
  ----- Original Message -----=20
  From: Stephen J. Harrington=20
  To: Somik Raha=20
  Sent: Thursday, November 21, 2002 10:23 AM
  Subject: Another oddity


  Somik,=20
  Some of the URL's we parse have spaces in them, e.g =
http://www.cnn.com/This is a test.html=20

  I know this in non-standard, but I just parse what I am told to =
parse.....=20

  My first response was to URLEncode the URL (URLEncoder.encode()) and =
pass it to the htmlparser.  This didn't work!?!=20

  So I then wrote a quick utility to replace all spaces with the correct =
URL encoded character and this worked.  Any idea why the parser would =
not like a URLEncoded URL?=20

  --stephen=20