As I see, your example works fine - it does the job.
html-to-xml itself cannot recover missing < or >.
Probably regular expressions are best choice for this kind of text processing.
When specifiying replace="true" in regexp processor, only matching values for the pattern (in your case <tbody* are replaced with the value from regexp-result, the rest is copied from regexp-source as is.
I have tried an data example with your config and it work fine.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Upon closer examination it looked like I needed to use Jtidy to preprocess the html because it could better resolve the <TBODY tag. I finally put a simple step to call JTidy before the tagsoup call. I could not get the http client to be happy w/Jtidy output (even w/xmlout=true, hence the inline jtidy call prior to the tagsoup call. ... probably not very efficient but I am learning! The following trivial code needs some care and feeding but maybe it will help someone with similar problem.
Html cleaning is the weakest part of Web-Harvest. I'm really not heppy with TagSoup - it has some very serious weaknesses. In some cases it simply doesn't work well. For example if there are DIV elements inside A, TagSoup pushes them out, so the original structure is changes. Furthermore it produces strange sequences of formatting tags (B, EM, I...) in some cases.
I have tried some others like JTidy and NekoHtml. None of them is satisfiable. JTidy has problems with scripts and as I saw on the site it is not maintained any more. Last version is from 2001!
So I've chosen TagSoup as the laast bad solution, but I'm seriously thinking about writing new html-to-xhtml utility.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Is there any way to use a defined variable for the regexp-pattern element? I don't know if there is a mechanism to preprocess the config file so the variable reference is not interpreted as the actual regulare expression pattern.
Exception in thread "main" java.util.regex.PatternSyntaxException: Illegal repetition near index 0
${my_reg_exp_var} Note I changed the name of the var here to match my example above
^
thanks,
-Pete
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
hello
Great package. Thank you.
I need to pre-process the output from <html-to-xml> with a regular expression to fix a missing ">" on a <TBODY> tag.
Is it possible to do this with the regex functionality (or alternative)?
It is unclear to me how to use the regexp-result/template to accomplish this.
Could someone please explain this in more detail?
Thanks,
-Pete
<file action="write" path="replacedTbody.txt">
<regexp replace="true">
<regexp-pattern>(<tbody)([^>])</regexp-pattern>
<regexp-source>
<text>
<html-to-xml>
<http url= "${url_start}" />
</html-to-xml>
</text>
</regexp-source>
<regexp-result>
<template>${_1}>${_2}</template>
</regexp-result>
</regexp>
As I see, your example works fine - it does the job.
html-to-xml itself cannot recover missing < or >.
Probably regular expressions are best choice for this kind of text processing.
When specifiying replace="true" in regexp processor, only matching values for the pattern (in your case <tbody* are replaced with the value from regexp-result, the rest is copied from regexp-source as is.
I have tried an data example with your config and it work fine.
Thanks for the quick response!
Upon closer examination it looked like I needed to use Jtidy to preprocess the html because it could better resolve the <TBODY tag. I finally put a simple step to call JTidy before the tagsoup call. I could not get the http client to be happy w/Jtidy output (even w/xmlout=true, hence the inline jtidy call prior to the tagsoup call. ... probably not very efficient but I am learning! The following trivial code needs some care and feeding but maybe it will help someone with similar problem.
package org.webharvest.runtime.html;
import java.io.*;
import org.webharvest.exception.ParserException;
import org.webharvest.utils.XMLWriter;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;
//import org.w3c.tidy.Tidy
public class JTidyProcessor implements IXHtmlProcessor {
}
Html cleaning is the weakest part of Web-Harvest. I'm really not heppy with TagSoup - it has some very serious weaknesses. In some cases it simply doesn't work well. For example if there are DIV elements inside A, TagSoup pushes them out, so the original structure is changes. Furthermore it produces strange sequences of formatting tags (B, EM, I...) in some cases.
I have tried some others like JTidy and NekoHtml. None of them is satisfiable. JTidy has problems with scripts and as I saw on the site it is not maintained any more. Last version is from 2001!
So I've chosen TagSoup as the laast bad solution, but I'm seriously thinking about writing new html-to-xhtml utility.
Is there any way to use a defined variable for the regexp-pattern element? I don't know if there is a mechanism to preprocess the config file so the variable reference is not interpreted as the actual regulare expression pattern.
for example
<var-def name="my_reg_exp_var">
<template><td>${passedParameterValue.toString()}<td></template>
</var-def>
<regexp-pattern>${my_reg_exp_var}</regexp-pattern>
<regexp-pattern><td>This text is known to caller and could be a parameter </td></regexp-pattern>
<regexp-pattern>${my_reg_exp_var}</regexp-pattern>
Exception in thread "main" java.util.regex.PatternSyntaxException: Illegal repetition near index 0
${my_reg_exp_var} Note I changed the name of the var here to match my example above
^
thanks,
-Pete
DUH : <regexp-pattern><template>${my_reg_exp_var}</template></regexp-pattern>
solves my problem!