htmlparser-user — The user mailing list for users of the htmlparser library

[Htmlparser-user] wrong link encoding?

[Eugeny N D. <bo...@re...>]

Hello!
I'm trying to parse this page and extract all links there: 
http://www.vu.lt/lt/naujienos/337/

for some reason the link to PDF file looks like: 
http://www.vu.lt/site_files/InfS/Naujienos/istorik??%20dienos.pdf

which is wrong. It seems like some wrong charset was used?

Here is part of my code which does the parsing:

public LinkedList parseDocument(InputStream document, String encoding) {
    try {
	Lexer lexer = new Lexer(new Page(document, encoding));
	String href;
	try {
	    lexer.reset();
	    if (banner != null)
		validateBanner(lexer);
	    lexer.reset();
	    Parser parser = new Parser(lexer);
	    NodeList list = null;
	    try {
		list = parser
			.extractAllNodesThatMatch(new InterestedTagsFilter());
	    } catch (EncodingChangeException e) {
		log.warn(e);
		lexer.reset();
		lexer.getPage().setEncoding(parser.getEncoding());
		list = parser
			.extractAllNodesThatMatch(new InterestedTagsFilter());
	    }
	    for (SimpleNodeIterator it = list.elements(); it.hasMoreNodes();) {
		TagNode node = (TagNode) it.nextNode();
		href = null;
		if (LinkTag.class.equals(node.getClass())
			&& validateLink((LinkTag) node)) {
		    href = ((LinkTag) node).getLink();
		} else if (ImageTag.class.equals(node.getClass())
			|| FrameTag.class.equals(node.getClass())) {
		    href = node.getAttribute("src");
		} else if (TitleTag.class.equals(node.getClass())) {
		    title = ((TitleTag) node).getTitle();
		} else if (BaseHrefTag.class.equals(node.getClass())) {
		    try {
			baseTag = getBaseURL(new URI(((BaseHrefTag) node)
				.getBaseUrl(), false));
		    } catch (URIException e2) {
		    }
		} else if (MetaTag.class.equals(node.getClass())
			&& "refresh".equalsIgnoreCase(((MetaTag) node)
				.getHttpEquiv())) {
		    String URL = ((MetaTag) node).getMetaContent();
		    if (URL != null && URL.length() > 0) {
			String arr[] = URL.split("URL=");
			if (arr != null && arr.length == 2)
			    href = arr[1];
		    }
		}
		if (href != null && href.length() > 0) {
		    if (log.isDebugEnabled())
------->		log.debug(href);		<-----------
		    results.add(getURL(StringEscapeUtils
			    .unescapeHtml(getEscapedURL(href.trim()))));
		}
	    }
	    this.encoding = parser.getEncoding();
	    if (log.isDebugEnabled())
		log.debug(this.encoding);
	} catch (ParserException e1) {
	    log.error(e1, e1);
	}
    } catch (UnsupportedEncodingException e) {
	log.error(e, e);
    }
    return results;
}

And on marked line application logs
/site_files/InfS/Naujienos/istorik??%20dienos.pdf

what could be wrong there?

-- 
Eugene N Dzhurinsky

[Htmlparser-user] Issue for lexer modification

[Xue-Feng Y. <jus...@ya...>]

I am trying to modify for the TextNodes in a lexer by 

TextNode.setText(String). Then I tried to print the
lexer by

 Page toPage=lexer.getPage();
 String toString=toPage.getText();
 System.out.println(toString);

The page was unchanged.
 

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [Htmlparser-user] finding meta data

[kavorka <the...@gm...>]

Hi Oswald,
I have another question. In HTMLPARSER, is it possible to extract only the
text in the webpage. In the stringextractor program, it extract also link
text in the page, i want to extract "pure" text. can i do it?
thanks
Murat


On 7/25/06, kavorka <the...@gm...> wrote:
>
>  Hi Oswald,
>
> Thanks a lot for your help.
>
> Murat
>
>
>  On 7/24/06, Derrick Oswald <Der...@ro...> wrote:
> >
> > Kavorka,
> >
> > This should give you the meta tag, from which you can get the
> > information you want:
> >
> > NodeList nodes = parser.parse (null);
> > NodeList metas = nodes.extractAllNodesThatMatch (new TagNameFilter
> > ("META"));
> > MetaTag meta = (MetaTag)metas.elementAt (0);
> > System.out.println (meta);
> >
> > Derrick
> >
> > kavorka wrote:
> >
> > > Hi all,
> > > I'm new to HTML-parser. I used sample programs to understand how can i
> > > find the meta data of the page but i could't use it. Do you have any
> > > code samples that finds meta data of the page using HTMLparser.
> > > Thank you
> > > best regards
> > >
> >
> >
> >
> > -------------------------------------------------------------------------
> > Take Surveys. Earn Cash. Influence the Future of IT
> > Join SourceForge.net's Techsay panel and you'll get the chance to share
> > your
> > opinions on IT & business topics through brief surveys -- and earn cash
> > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> >
> > _______________________________________________
> > Htmlparser-user mailing list
> > Htm...@li...
> > https://lists.sourceforge.net/lists/listinfo/htmlparser-user
> >
>
>

[Htmlparser-user] Issue for lexer modification

[Xue-Feng Y. <jus...@ya...>]

I am trying to modify for the TextNodes in a lexer by 
TextNode.setText(String). Then I tried to print the
lexer by

 Page toPage=lexer.getPage();
 String toString=toPage.getText();
 System.out.println(toString);

The page was unchanged.
 
Does any one have idea how to modify a lexer or simply
a html page?

Thanks,

Xue-Feng

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [Htmlparser-user] finding meta data

[kavorka <the...@gm...>]

Hi Oswald,

Thanks a lot for your help.

Murat


On 7/24/06, Derrick Oswald <Der...@ro...> wrote:
>
> Kavorka,
>
> This should give you the meta tag, from which you can get the
> information you want:
>
> NodeList nodes = parser.parse (null);
> NodeList metas = nodes.extractAllNodesThatMatch (new TagNameFilter
> ("META"));
> MetaTag meta = (MetaTag)metas.elementAt (0);
> System.out.println (meta);
>
> Derrick
>
> kavorka wrote:
>
> > Hi all,
> > I'm new to HTML-parser. I used sample programs to understand how can i
> > find the meta data of the page but i could't use it. Do you have any
> > code samples that finds meta data of the page using HTMLparser.
> > Thank you
> > best regards
> >
>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share
> your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>

Re: [Htmlparser-user] finding meta data

[Derrick O. <Der...@Ro...>]

Kavorka,

This should give you the meta tag, from which you can get the 
information you want:

NodeList nodes = parser.parse (null);
NodeList metas = nodes.extractAllNodesThatMatch (new TagNameFilter 
("META"));
MetaTag meta = (MetaTag)metas.elementAt (0);
System.out.println (meta);

Derrick

kavorka wrote:

> Hi all,
> I'm new to HTML-parser. I used sample programs to understand how can i 
> find the meta data of the page but i could't use it. Do you have any 
> code samples that finds meta data of the page using HTMLparser.
> Thank you
> best regards
>

Re: [Htmlparser-user] Searching HTML code in a page

[Derrick O. <Der...@Ro...>]

Eigeny,

In general, you probably want to look at the filter package.
Try running the filterbuilder application (startup script is in the bin 
directory) and read the help and tutorial.
Using this application you can create a Java program that selects only 
the 'sometext' you want.

Derrick

Eugeny N Dzhurinsky wrote:

>Hello!
>I need to search for HTML code in a page, for instance the code to search
>looks like this:
>
><div class="someclass"><a href="somelocation" ><img src="image/here"
>border="0"></a></div><span style="style2">sometext</span>
>
>This code could be placed as single line or formatted somehow, containing one
>or more linebreaks.
>
>I need also to track situation while this code is commented out, or placed
>outside <body> section.
>
>For now I created a Lexer instance for document and for this code, comparing
>them token by token, but may be there is some better way?
>
>  
>

Re: [Htmlparser-user] String With only a DataTable

[Ian M. <ian...@gm...>]

HTMLParser is usually capable of parsing just an HTML fragment.

Parser.setInputHTML("html") and then Parser.parse(null).

Ian

On 7/14/06, Dennis Gesker <ge...@al...> wrote:
> Since it was just a string I added html and body tags and it seems I'm
> on my way.
>
> str = "<head><body> + str + "</head><;body>";
>
> --Dennis
>
> Dennis Gesker wrote:
> > I would like to parse a portion of html that I have in a buffer
> > (String), that is to say not a complete page. The string contains an
> > html table only.
> >
> > Could someone point to or provide some sample code for how to parse just
> > a fragment of html?
> >
> > Dennis
> >
> >
> >
>
> --
> Dennis R. Gesker
> email: de...@al...
> Key Id: 0xEFA10A51
>
>
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>

[Htmlparser-user] finding meta data

[kavorka <the...@gm...>]

Hi all,
I'm new to HTML-parser. I used sample programs to understand how can i find
the meta data of the page but i could't use it. Do you have any code samples
that finds meta data of the page using HTMLparser.
Thank you
best regards

[Htmlparser-user] Searching HTML code in a page

[Eugeny N D. <bo...@re...>]

Hello!
I need to search for HTML code in a page, for instance the code to search
looks like this:

<div class="someclass"><a href="somelocation" ><img src="image/here"
border="0"></a></div><span style="style2">sometext</span>

This code could be placed as single line or formatted somehow, containing one
or more linebreaks.

I need also to track situation while this code is commented out, or placed
outside <body> section.

For now I created a Lexer instance for document and for this code, comparing
them token by token, but may be there is some better way?

-- 
Eugene N Dzhurinsky

Re: [Htmlparser-user] String With only a DataTable

[Dennis G. <ge...@al...>]

Since it was just a string I added html and body tags and it seems I'm 
on my way.

str = "<head><body> + str + "</head><;body>";

--Dennis

Dennis Gesker wrote:
> I would like to parse a portion of html that I have in a buffer 
> (String), that is to say not a complete page. The string contains an 
> html table only.
>
> Could someone point to or provide some sample code for how to parse just 
> a fragment of html?
>
> Dennis
>
>
>   

-- 
Dennis R. Gesker
email: de...@al...
Key Id: 0xEFA10A51

[Htmlparser-user] String With only a DataTable

[Dennis G. <ge...@al...>]

I would like to parse a portion of html that I have in a buffer 
(String), that is to say not a complete page. The string contains an 
html table only.

Could someone point to or provide some sample code for how to parse just 
a fragment of html?

Dennis


-- 
Dennis R. Gesker
email: de...@al...
Key Id: 0xEFA10A51

Re: [Htmlparser-user] query on HTML Parser

[Derrick O. <Der...@Ro...>]

This should give you the "Content":

NodeList nodes = parser.parse (null);
NodeList metas = nodes.extractAllNodesThatMatch (new TagNameFilter 
("META"));
System.out.println (metas.elementAt (0).getMetaContent ());

vasantha reddy wrote:

> Hi,
>  
>                     I am using HTML parser in my project.The HTML 
> parser doesn't give the contents of meta tag as its output.I need the 
> content of the meta tag.Is there any method that I can use to get the 
> content of a particular tag by giving the tag name as input?
>
> Thank you,
> Regards,
> Vasantha
>
>

[Htmlparser-user] query on HTML Parser

[vasantha r. <hi_...@ya...>]

Hi,
   
                      I am using HTML parser in my project.The HTML parser doesn't give the contents of meta tag as its output.I need the content of the meta tag.Is there any method that I can use to get the content of a particular tag by giving the tag name as input?
  
Thank you,
  Regards,
Vasantha

 				
---------------------------------
 Yahoo! India Answers: Share what you know. Learn something new Click here
Catch all the FIFA World Cup 2006 action on Yahoo! India Click here

Re: [Htmlparser-user] How to implement NodeFilter?

[Derrick O. <Der...@Ro...>]

Using the NotFilter directly like that probably won't help you.
As written it would return a nodelist of top level nodes (no Remarks of 
course), but these would include Remarks as children.
What you probably want to do is override RemarkNode to return nothing 
from toHtml() and set this as the default node for Remarks on the 
PrototypicalNodeFactory via setRemarkPrototype().
Then when you issue the toHtml() on the NodeList returned from a 
straight parse, the contents will have the Remarks removed.

Mark Stark wrote:

>hello,
>
>can anyone explain how to usw a NodeFilter?
>
>NodeList l = p.parse(new NotFilter());
>
>i would like to parse all nodes/tags but RemarkTags/RemarkNodes
>
>thanks a lot
>
>
>
>  
>

[Htmlparser-user] How to implement NodeFilter?

[Mark S. <htm...@ey...>]

hello,

can anyone explain how to usw a NodeFilter?

NodeList l = p.parse(new NotFilter());

i would like to parse all nodes/tags but RemarkTags/RemarkNodes

thanks a lot

Re: [Htmlparser-user] Rebuild HTML File

[Mark S. <htm...@ey...>]

Thanks, it works

Derrick Oswald schrieb:
> I think you need something like this...
> 
> Parser parser = new Parser ();
> parser.setInputHtml (html);
> NodeList list = parser.parse (null);
> list.visitAllNodesWith (new SegmentReplacingVisitor ());
> System.out.println (list.toHtml ());
> 
> ...assuming the SegmentReplacingVisitor does the right stuff to the 
> right nodes.
> 
> Mark Stark wrote:
> 
>> I dont understand. I've got a html file, manipulate some tags with my
>> visitor, perhaps change an attribute value. and then? serialize the
>> toHtml() output into a file and thats it?
>>
>> Ian Macfarlane schrieb:
>>  
>>
>>> If you do Parser.parse() you get a NodeList, and then if you call the
>>> toHtml() method it will return a String of the reparsed code.
>>>
>>> Ian
>>>
>>> On 6/26/06, Mark Stark <htm...@ey...> wrote:
>>>    
>>>
>>>> hi,
>>>>
>>>> is it possible to change certain stirngs in a html file, without
>>>> "rebuilding" it? I would like to change some string in a html file, like
>>>> a dictionary lookup, but i dont want to rebuild the html file like its
>>>> done in URLModifyingVisitor.
>>>>
>>>> Can anybody give me a hint? I've still got
>>>>
>>>> Parser p = new Parser();
>>>> p.setInputHTML(html);
>>>> SegmentReplacingVisitor s = new SegmentReplacingVisitor();
>>>> p.visitAllNodesWith(s);
>>>>
>>>> and do some text.setText() within the SegmentReplacingVisitor each time
>>>> i visit a StringNode. How to save it into a translated.htm file without
>>>> rebuilding it?
>>>>
>>>> thanks a lot
>>>>
>>>>
>>>> Using Tomcat but need to do more? Need to support web services, security?
>>>> Get stuff done quickly with pre-integrated technology to make your job easier
>>>> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>>>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>>>> _______________________________________________
>>>> Htmlparser-user mailing list
>>>> Htm...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>>>>
>>>>      
>>>>
>>> Using Tomcat but need to do more? Need to support web services, security?
>>> Get stuff done quickly with pre-integrated technology to make your job easier
>>> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>>> _______________________________________________
>>> Htmlparser-user mailing list
>>> Htm...@li...
>>> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>>>
>>>
>>>    
>>>
>>
>>
>> Using Tomcat but need to do more? Need to support web services, security?
>> Get stuff done quickly with pre-integrated technology to make your job easier
>> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>> _______________________________________________
>> Htmlparser-user mailing list
>> Htm...@li...
>> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>>
>>  
>>
> 
> 
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
> 
> 

Re: [Htmlparser-user] Rebuild HTML File

[Derrick O. <Der...@Ro...>]

I think you need something like this...

Parser parser = new Parser ();
parser.setInputHtml (html);
NodeList list = parser.parse (null);
list.visitAllNodesWith (new SegmentReplacingVisitor ());
System.out.println (list.toHtml ());

...assuming the SegmentReplacingVisitor does the right stuff to the 
right nodes.

Mark Stark wrote:

>I dont understand. I've got a html file, manipulate some tags with my
>visitor, perhaps change an attribute value. and then? serialize the
>toHtml() output into a file and thats it?
>
>Ian Macfarlane schrieb:
>  
>
>>If you do Parser.parse() you get a NodeList, and then if you call the
>>toHtml() method it will return a String of the reparsed code.
>>
>>Ian
>>
>>On 6/26/06, Mark Stark <htm...@ey...> wrote:
>>    
>>
>>>hi,
>>>
>>>is it possible to change certain stirngs in a html file, without
>>>"rebuilding" it? I would like to change some string in a html file, like
>>>a dictionary lookup, but i dont want to rebuild the html file like its
>>>done in URLModifyingVisitor.
>>>
>>>Can anybody give me a hint? I've still got
>>>
>>>Parser p = new Parser();
>>>p.setInputHTML(html);
>>>SegmentReplacingVisitor s = new SegmentReplacingVisitor();
>>>p.visitAllNodesWith(s);
>>>
>>>and do some text.setText() within the SegmentReplacingVisitor each time
>>>i visit a StringNode. How to save it into a translated.htm file without
>>>rebuilding it?
>>>
>>>thanks a lot
>>>
>>>
>>>Using Tomcat but need to do more? Need to support web services, security?
>>>Get stuff done quickly with pre-integrated technology to make your job easier
>>>Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>>>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>>>_______________________________________________
>>>Htmlparser-user mailing list
>>>Htm...@li...
>>>https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>>>
>>>      
>>>
>>Using Tomcat but need to do more? Need to support web services, security?
>>Get stuff done quickly with pre-integrated technology to make your job easier
>>Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>>_______________________________________________
>>Htmlparser-user mailing list
>>Htm...@li...
>>https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>>
>>
>>    
>>
>
>
>
>Using Tomcat but need to do more? Need to support web services, security?
>Get stuff done quickly with pre-integrated technology to make your job easier
>Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>_______________________________________________
>Htmlparser-user mailing list
>Htm...@li...
>https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>
>  
>

Re: [Htmlparser-user] Rebuild HTML File

[Mark S. <htm...@ey...>]

I tried to use it this way:

	Parser p = new Parser();
	p.setInputHTML(normalHTML);
	System.out.println(">>>>>>> "+p.parse(null).toHtml());
	TestVisitor s = new TestVisitor();
	p.parse(null).visitAllNodesWith(s);
	System.out.println("<<<<<<<" +p.parse(null).toHtml());


In TestVisitor

    public void visitStringNode (Text string)
    {	
    	string.setText("XXX");
    }

The first System.out prints the original html, second System.out prints
out nothing but the "<<<<". Have you any suggestions? :)

Thanks a lot

Ian Macfarlane schrieb:
> If you do Parser.parse() you get a NodeList, and then if you call the
> toHtml() method it will return a String of the reparsed code.
> 
> Ian
> 
> On 6/26/06, Mark Stark <htm...@ey...> wrote:
>> hi,
>>
>> is it possible to change certain stirngs in a html file, without
>> "rebuilding" it? I would like to change some string in a html file, like
>> a dictionary lookup, but i dont want to rebuild the html file like its
>> done in URLModifyingVisitor.
>>
>> Can anybody give me a hint? I've still got
>>
>> Parser p = new Parser();
>> p.setInputHTML(html);
>> SegmentReplacingVisitor s = new SegmentReplacingVisitor();
>> p.visitAllNodesWith(s);
>>
>> and do some text.setText() within the SegmentReplacingVisitor each time
>> i visit a StringNode. How to save it into a translated.htm file without
>> rebuilding it?
>>
>> thanks a lot
>>
>>
>> Using Tomcat but need to do more? Need to support web services, security?
>> Get stuff done quickly with pre-integrated technology to make your job easier
>> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>> _______________________________________________
>> Htmlparser-user mailing list
>> Htm...@li...
>> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>>
> 
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
> 
> 

Re: [Htmlparser-user] Rebuild HTML File

[Mark S. <htm...@ey...>]

I dont understand. I've got a html file, manipulate some tags with my
visitor, perhaps change an attribute value. and then? serialize the
toHtml() output into a file and thats it?

Ian Macfarlane schrieb:
> If you do Parser.parse() you get a NodeList, and then if you call the
> toHtml() method it will return a String of the reparsed code.
> 
> Ian
> 
> On 6/26/06, Mark Stark <htm...@ey...> wrote:
>> hi,
>>
>> is it possible to change certain stirngs in a html file, without
>> "rebuilding" it? I would like to change some string in a html file, like
>> a dictionary lookup, but i dont want to rebuild the html file like its
>> done in URLModifyingVisitor.
>>
>> Can anybody give me a hint? I've still got
>>
>> Parser p = new Parser();
>> p.setInputHTML(html);
>> SegmentReplacingVisitor s = new SegmentReplacingVisitor();
>> p.visitAllNodesWith(s);
>>
>> and do some text.setText() within the SegmentReplacingVisitor each time
>> i visit a StringNode. How to save it into a translated.htm file without
>> rebuilding it?
>>
>> thanks a lot
>>
>>
>> Using Tomcat but need to do more? Need to support web services, security?
>> Get stuff done quickly with pre-integrated technology to make your job easier
>> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>> _______________________________________________
>> Htmlparser-user mailing list
>> Htm...@li...
>> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>>
> 
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
> 
> 

Re: [Htmlparser-user] Rebuild HTML File

[Ian M. <ian...@gm...>]

If you do Parser.parse() you get a NodeList, and then if you call the
toHtml() method it will return a String of the reparsed code.

Ian

On 6/26/06, Mark Stark <htm...@ey...> wrote:
> hi,
>
> is it possible to change certain stirngs in a html file, without
> "rebuilding" it? I would like to change some string in a html file, like
> a dictionary lookup, but i dont want to rebuild the html file like its
> done in URLModifyingVisitor.
>
> Can anybody give me a hint? I've still got
>
> Parser p = new Parser();
> p.setInputHTML(html);
> SegmentReplacingVisitor s = new SegmentReplacingVisitor();
> p.visitAllNodesWith(s);
>
> and do some text.setText() within the SegmentReplacingVisitor each time
> i visit a StringNode. How to save it into a translated.htm file without
> rebuilding it?
>
> thanks a lot
>
>
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>

[Htmlparser-user] Rebuild HTML File

[Mark S. <htm...@ey...>]

hi,

is it possible to change certain stirngs in a html file, without
"rebuilding" it? I would like to change some string in a html file, like
a dictionary lookup, but i dont want to rebuild the html file like its
done in URLModifyingVisitor.

Can anybody give me a hint? I've still got

Parser p = new Parser();
p.setInputHTML(html);
SegmentReplacingVisitor s = new SegmentReplacingVisitor();
p.visitAllNodesWith(s);

and do some text.setText() within the SegmentReplacingVisitor each time
i visit a StringNode. How to save it into a translated.htm file without
rebuilding it?

thanks a lot

Re: [Htmlparser-user] HasAttributeFilter

[Ian M. <ian...@gm...>]

Dear Ryan,

This was discussed, and even the constructors/methods were hammered out in
this dev mailing list about a month or two ago. However I've not had the
time to implement it yet.

Please feel free to implement the agreed-upon design (or comment on it).
Unfortunately it looks like the SourceForge message archives are down for
now, so I'll forward you the entire thread to your own address (no need to
copy it all here).

Best wishes

Ian Macfarlane

On 6/22/06, Ryan Smith <rs...@li...> wrote:
>
> <div a=foobar b=aybabtu></div>
> <div a=foodbar b=aybabtu></div>
>
>
> Is there a way to use HasAttributeFilter("a","foo")
>
> But have the value match *NOT EXACT* ?
>
> Meaning, it would be nice if the value could be a regular expression, so
> i could match all attributes with the word "foo"
> instead of having to match thte attribute EXACTLY
>
> Is this possible? can i add a patch?
>
>
> Also, while im at it, the HtmlParserUtils has nice methods like trim
> tags etc...
> But i use a custom parser with my own tags defined, and the util class
> provides no way to give your own parser, (it called new Parser() :( )
> Can support be added to this?  ( i have a patch if anyone wants it )
>
>
> Thanks
> -Ryan Smith
> Software Developer
> Foreclosure.com
>
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job
> easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>

[Htmlparser-user] HasAttributeFilter

[Ryan S. <rs...@li...>]

<div a=foobar b=aybabtu></div>
<div a=foodbar b=aybabtu></div>


Is there a way to use HasAttributeFilter("a","foo")

But have the value match *NOT EXACT* ?

Meaning, it would be nice if the value could be a regular expression, so 
i could match all attributes with the word "foo"
instead of having to match thte attribute EXACTLY

Is this possible? can i add a patch?


Also, while im at it, the HtmlParserUtils has nice methods like trim 
tags etc...
But i use a custom parser with my own tags defined, and the util class 
provides no way to give your own parser, (it called new Parser() :( )
Can support be added to this?  ( i have a patch if anyone wants it )


Thanks
-Ryan Smith
Software Developer
Foreclosure.com

Re: [Htmlparser-user] Charset and multiple reparsing questions

[Ian M. <ian...@gm...>]

Yup, internationalization issues can be pretty confusing :)

I've written a test file using windows-1251 source file and
windows-specific characters, then changing meta content and headers to
find out what happens in my web browser. The case is as follows:

- With no charset specified in headers or meta tags, it can't work out
the charset
- With no charset specified in headers but charset specified in meta
tag, the meta tag one is used
- When the charset is specified in the headers, it overrides anything
specified in the meta tag

I'm going to eventaully have to emulate that. I'm pretty sure that the
Apache HTTPClient just uses the charset specified in the http headers.
It does have a method to get the response charset specified in the
headers, but unfortunately it looks like if none is specified it
defaults to iso-8859-1 (or whichever is set as default), which means I
can't really tell if it's had one set or not.

So I think, for now, I'm going to ignore it and see if it turns out to
be a problem or not :)

Ian

On 6/9/06, Derrick Oswald <Der...@ro...> wrote:
> Ian,
>
> Don't you just hate Windows Search - completely broken and it's been
> that way for a half dozen years.
> But if you complain it doesn't get you anywhere...
>
> Correct so far.
>
> - the interpretation of the bytes from the input stream follows the META
> tag after that's encountered,
>   a Java String doesn't really have a charset as far as I can tell -
> it's Unicode UTF-16 (I may be wrong here),
>   so the answer is: it will have the 'correct' charset - whatever set it
> last, header or META.
>   The regenerated toHtml() String will have the 'correct' charset
> because it's coming from an array of char
>   which (as far as I can tell) covers most of the possible charsets.
>   [I say most because there is a move afoot to make chars int32 in size
> to accommodate many more Chinese glyphs etc.
>   and I'm not sure if that's in Mustang (Java 1.5) or not.]
>   Now if you want to write an array of bytes on disk or pass a string to
> another program with 8 bit chars,
>   you need to choose an encoding that can accommodate your charset...
> whole new ballgame.
>   Bottom line is the encoding only matters if it's converted to bytes (I
> think).
>   Check out the 'Save As Unicode' option in NotePad, it doesn't ask for
> a charset,
>   (but then again, it may *know* the charset from user settings)
>   but that sets the encoding (Unicode UTF-8 I think) for the file on disk.
>
> - in a number of places this is exactly the processing used, reset()
> followed by reparse,
>   see for example StringBean.setStrings () - the point being that the
> client *must* rehandle
>   nodes it was given, usually by starting from scratch - I don't know
> any other way - because
>   what it was given was erroneous. In the case of a String as input,
> reparse won't yield any
>   different characters (they just come from the String via charAt() and
> that won't change
>   because the String is immutable) so the reset is redundant, except
> that the StringSource
>   will have it's encoding (member variable) set correctly the second
> time so the hiccup
>   won't happen twice.  But the conversion from the byte stream to a
> String has to have
>   been correct regardless of what the HTTP header says, otherwise you're
> pimped.
>   So if the HttpClient gives you a String you have to ask:
>   - did it look at the META tag?
>   - is the META tag correct?
>
> If it sounds confused, that's because it probably is - in my own mind.
>
> Derrick
>
> Ian Macfarlane wrote:
>
> >That will teach me to rely on windows search. Bleh.
> >
> >Ok, so if the headers kick the file out as one charset, then the meta
> >tag states that it is a different one, I assume (based on the W3C
> >recommendations and a quick peek at InputStreamSource) if the new
> >encoding is compatible (characters parsed so far are the same) it will
> >just reparse the rest of the page with the new charset, otherwise it
> >will throw an EncodingChangeException. Am I right so far?
> >
> >Now if I walk through these two potential paths:
> >
> >- If the exception is not thrown, is the parsed document encoded with
> >the charset specified in the headers or in the meta tag? I.e. if I
> >convert it back to a String from a Nodelist etc, will it have the
> >correct charset from the meta tag still?
> >
> >- If the exception is thrown, can I reparse the entire document from
> >the original String or would I have to go back to the orignal byte[]
> >to do this?
> >
> >Thanks,
> >
> >Ian
> >
> >On 6/7/06, Derrick Oswald <der...@ro...> wrote:
> >
> >
> >>Its thrown in
> >>  org.htmlparser.lexer.InputStreamSource.setEncoding
> >>(String)
> >>
> >>
> >>
> >>Ian Macfarlane <ian...@gm...> wrote:
> >>
> >> Derrick,
> >>
> >>I can't see anywhere EncodingChangeException is thrown in the code,
> >>perhaps this is not implemented yet?
> >>
> >>Ian
> >>
> >>On 6/5/06, Derrick Oswald wrote:
> >>
> >>
> >>>Ian,
> >>>
> >>>If you have a String in Java, it's Unicode encoded in UTF-16 - no?
> >>>(the trick of course, is in how it got to be a String, or how the String
> >>>gets saved to a Stream)
> >>>so I don't think you *need* to specify the encoding if you are passing
> >>>in a String.
> >>>Looking at the StringSource.java code, the encoding which may be passed
> >>>in the constructor is just stored as a property.
> >>>It doesn't appear to be used. But if set properly on the constructor it
> >>>would avoid a retrace when the META tag is encountered.
> >>>You would do something like this:
> >>>new Parser (new Lexer (new Page (my_string, my_encoding)))
> >>>
> >>>There is code in MetaTag.doSemanticAction() to set the page encoding
> >>>based on the META tag.
> >>>This mechanism wouldn't do anything under the hood if the input is a
> >>>String (based on the the fact the StringSource just stores the encoding).
> >>>But, if the HttpClient incorrectly converted the stream to a String
> >>>based on the HTTP header content type and the META tag actually has the
> >>>correct encoding you have a problem (this is the reason for the
> >>>EncodingChangeException thrown by the parser).
> >>>
> >>>Conversion from the parse tree to a String actually just regurgitates
> >>>the characters read in, so the charset and encoding don't enter into it
> >>>here.
> >>>
> >>>Submitting the String to be parsed again brings up the same issues as
> >>>the first time.
> >>>
> >>>Derrick
> >>>
> >>>Ian Macfarlane wrote:
> >>>
> >>>
> >>>
> >>>>I have a few questions regarding the best way to perform multiple
> >>>>parsing to and from HTML stored as a String and HTMLParser parsed
> >>>>(tree) format.
> >>>>
> >>>>1) Firstly, when first parsing (using Parser not Lexer, I need a
> >>>>tree), is there a way to pass it the charset (e.g. UTF-8) that was
> >>>>specified in the HTTP headers? Do I need to do this if it is already
> >>>>encoded correctly? (I'm using Apache HTTPClient which can convert into
> >>>>a Byte[] or a correctly encoded String using the headers found, and
> >>>>I'm using the latter option).
> >>>>
> >>>>2) Once I have done this, I'd want it to be overridden if the Meta
> >>>>http-equiv Content-Type gives me a different one. Can the parser
> >>>>automatically do this? Or do I have to attempt to read it myself?
> >>>>
> >>>>3) Now I've got the body tag, and a charset specified either by the
> >>>>headers or the meta tag (or if none, a sensible default), I want to
> >>>>convert the document back into a String again. Do I need to be
> >>>>concerned about the charset again here, or do the Node/NodeList
> >>>>toString methods handle this?
> >>>>
> >>>>4) Finally, once I have a String that's a product of the above, and I
> >>>>want to again convert it into an HTMLParser tree, do I need to specify
> >>>>the charset again here?
> >>>>
> >>>>Thanks
> >>>>
> >>>>Ian
> >>>>
> >>>>
> >>>>
> >
> >
>
>
>
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>