#1564 HTML entity parsing of certain entity results in uncaught IllegalArgumentException

[Sebastian Cato]

Hello,

    @Test
    public void entityRefTest() throws Exception {
        final String html = "<html><body>"

            + "&#x1b3d6e;"
            + "</body></html>";
         loadPage(html);
    }

java.lang.IllegalArgumentException
at java.lang.Character.toChars(Character.java:2583)
at org.cyberneko.html.HTMLScanner.appendChar(HTMLScanner.java:1675)
at org.cyberneko.html.HTMLScanner.scanEntityRef(HTMLScanner.java:1384)
at org.cyberneko.html.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2049)
at org.cyberneko.html.HTMLScanner.scanDocument(HTMLScanner.java:918)
at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:499)
at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:452)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)

Cheers,
Sebastian Cato

[Ahmed Ashour]

• Description has changed:

Diff:

--- old
+++ new
@@ -1,6 +1,7 @@
 Hello,

 Test case:
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     @Test
     public void entityRefTest() throws Exception {
         final String html = "<html><body>"
@@ -9,10 +10,11 @@

         loadPage(html);
     }    
-
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


 Stack trace:
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 java.lang.IllegalArgumentException
    at java.lang.Character.toChars(Character.java:2583)
    at org.cyberneko.html.HTMLScanner.appendChar(HTMLScanner.java:1675)
@@ -31,6 +33,6 @@
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:331)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:396)

-
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Cheers, 
 Sebastian Cato

[Ahmed Ashour]

• Description has changed:

Diff:

--- old
+++ new
@@ -1,20 +1,17 @@
 Hello,

-Test case:
+
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     @Test
     public void entityRefTest() throws Exception {
         final String html = "<html><body>"

            + "&#x1b3d6e;"
             + "</body></html>";
-
-        loadPage(html);
-    }    
+    }
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


-Stack trace:
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 java.lang.IllegalArgumentException
    at java.lang.Character.toChars(Character.java:2583)
    at org.cyberneko.html.HTMLScanner.appendChar(HTMLScanner.java:1675)
@@ -24,15 +21,6 @@
    at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:499)
    at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:452)
    at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)

-   at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.parse(HTMLParser.java:902)
-   at com.gargoylesoftware.htmlunit.html.HTMLParser.parse(HTMLParser.java:244)
-   at com.gargoylesoftware.htmlunit.html.HTMLParser.parseHtml(HTMLParser.java:190)
-   at com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:268)
-   at com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:156)
-   at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:457)
-   at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:331)
-   at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:396)

-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Cheers, 
 Sebastian Cato

[Sebastian Cato]

I appreciate the markup. The call to loadPage is however significant to the test case and should not be removed. To clarify, entityRefTest was written to be a part of one of the classes inheriting from SimpleWebTestCase in the test suite for HtmlUnit (I used WebClientTest), and loadPage refers to SimpleWebTestCase#loadPage(String).

[Ahmed Ashour]

There error is obvious, and it is due to non handling of "�" entities by NekoHtml.

I searched about any 6-digits hexadecimals for entity, but couldn't find none.

Can you provide any 6-digits value handled by real browsers as a visible character? Because otherwise, we will let NekoHtml team decide what to put (ignored or trimmed character).

[Ahmed Ashour]

• Description has changed:

Diff:

--- old
+++ new
@@ -7,6 +7,7 @@
         final String html = "<html><body>"

            + "&#x1b3d6e;"
             + "</body></html>";
+         loadPage(html);
     }
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[Sebastian Cato]

we will let NekoHtml team decide what to put (ignored or trimmed character)

That sounds right to me, in the documentation for Character#toChars it says that IllegalArgumentException is thrown if the specified codePoint is not a valid Unicode code point, so I would say catching that exception straight away would be the thing to do.

[Marc Guillemot]

Now fixed in SVN.

Thanks for the tiny test case (adapted version is now in build: MalformedHtmlTest.entityWithInvalidUTF16Code).