htmlparser-user

[Htmlparser-user] Parsing ISO-8859-1 --> storing in UTF-8?

[Jan H. <jan...@gm...>]

Hi guys,

I'm trying to parse a website which is encoded in ISO-8859-1. I need to 
store extracted link-texts in UTF-8 format.

My code looks like this:

<code>
Parser myParser = new Parser();
myParser.setURL(url);

// I created a filter named "myLinkFilter" which filters LinkNodes
NodeList myLinkNodeList = myParser.parse(myLinkFilter);

Node myLinkNode = myLinkNodeList.elementAt(0);

LinkTag linkTag = (LinkTag) myLinkNode;

String linkText = linkTag.getLinkText();
</code>

The problem now is, that certain characters (like the lower quotation 
marks: „Quote“) are converted to question marks.

So I tried a coding like this:

<code>

String isoString = linkTag.getLinkText();
String utf8String = null;

try
{
    byte[] stringBytesISO = isoString.getBytes("ISO-8859-1");
    utf8String = new String(stringBytesISO, "UTF-8");
}
catch (UnsupportedEncodingException e)
{
    // do something...
}
</code>

But this still returns question marks in the utf8String.
Any ideas what I need to change?

Thanks and regards
Jan Hempel

Re: [Htmlparser-user] Parsing ISO-8859-1 --> storing in UTF-8?

[Derrick O. <Der...@Ro...>]

Jan,

It may be that the site is lying (in the HTTP header or even in the META 
tag of the page) and it really is in another encoding - maybe UTF-8 already.
Try setEncoding() on the Parser before asking for nodes or filtering.

Derrick

Jan Hempel wrote:

>Hi guys,
>
>I'm trying to parse a website which is encoded in ISO-8859-1. I need to 
>store extracted link-texts in UTF-8 format.
>
>My code looks like this:
>
><code>
>Parser myParser = new Parser();
>myParser.setURL(url);
>
>// I created a filter named "myLinkFilter" which filters LinkNodes
>NodeList myLinkNodeList = myParser.parse(myLinkFilter);
>
>Node myLinkNode = myLinkNodeList.elementAt(0);
>
>LinkTag linkTag = (LinkTag) myLinkNode;
>
>String linkText = linkTag.getLinkText();
></code>
>
>The problem now is, that certain characters (like the lower quotation 
>marks: „Quote“) are converted to question marks.
>
>So I tried a coding like this:
>
><code>
>
>String isoString = linkTag.getLinkText();
>String utf8String = null;
>
>try
>{
>    byte[] stringBytesISO = isoString.getBytes("ISO-8859-1");
>    utf8String = new String(stringBytesISO, "UTF-8");
>}
>catch (UnsupportedEncodingException e)
>{
>    // do something...
>}
></code>
>
>But this still returns question marks in the utf8String.
>Any ideas what I need to change?
>
>Thanks and regards
>Jan Hempel
>
>
>
>
>
>-------------------------------------------------------------------------
>Using Tomcat but need to do more? Need to support web services, security?
>Get stuff done quickly with pre-integrated technology to make your job easier
>Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>_______________________________________________
>Htmlparser-user mailing list
>Htm...@li...
>https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>
>  
>

Re: [Htmlparser-user] Parsing ISO-8859-1 --> storing in UTF-8?

[Jan H. <jan...@gm...>]

Derrick,

thanks for Your advise! I tried setEncoding(), but then I get 
ParserExceptions.

I also tried my code with all kinds of public pages, all using 
ISO-8859-1, but whenever they have characters specific to ISO-8859-1 (as 
I mentioned, for example the lower and upper quotation marks) I have 
problems.

I debugged the code in eclipse. This is how I retrieve the link-text:

<code>

LinkTag linkTag = (LinkTag) linkNode;
String linktext = linkTag.getLinkText();

</code>

The method "linkTag.getLinkText()()" returns the text with little 
"boxes" instad of the quotation-marks (I can't put them in this 
plain-text mail).

So it seems like the getLinkText() method does return these characters 
wrongly encoded?

Thanks and regards
Jan Hempel


Derrick Oswald wrote:
> Jan,
> 
> It may be that the site is lying (in the HTTP header or even in the META 
> tag of the page) and it really is in another encoding - maybe UTF-8 already.
> Try setEncoding() on the Parser before asking for nodes or filtering.
> 
> Derrick
> 
> Jan Hempel wrote:
> 
>> Hi guys,
>>
>> I'm trying to parse a website which is encoded in ISO-8859-1. I need to 
>> store extracted link-texts in UTF-8 format.
>>
>> My code looks like this:
>>
>> <code>
>> Parser myParser = new Parser();
>> myParser.setURL(url);
>>
>> // I created a filter named "myLinkFilter" which filters LinkNodes
>> NodeList myLinkNodeList = myParser.parse(myLinkFilter);
>>
>> Node myLinkNode = myLinkNodeList.elementAt(0);
>>
>> LinkTag linkTag = (LinkTag) myLinkNode;
>>
>> String linkText = linkTag.getLinkText();
>> </code>
>>
>> The problem now is, that certain characters (like the lower quotation 
>> marks: „Quote“) are converted to question marks.
>>
>> So I tried a coding like this:
>>
>> <code>
>>
>> String isoString = linkTag.getLinkText();
>> String utf8String = null;
>>
>> try
>> {
>>    byte[] stringBytesISO = isoString.getBytes("ISO-8859-1");
>>    utf8String = new String(stringBytesISO, "UTF-8");
>> }
>> catch (UnsupportedEncodingException e)
>> {
>>    // do something...
>> }
>> </code>
>>
>> But this still returns question marks in the utf8String.
>> Any ideas what I need to change?
>>
>> Thanks and regards
>> Jan Hempel
>>
>>
>>
>>
>>
>> -------------------------------------------------------------------------
>> Using Tomcat but need to do more? Need to support web services, security?
>> Get stuff done quickly with pre-integrated technology to make your job easier
>> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>> _______________________________________________
>> Htmlparser-user mailing list
>> Htm...@li...
>> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>>
>>  
>>
> 
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
> 
>