Double encoding prevents fetch of URLs
An open source search engine with RESTFul API and crawlers
Brought to you by:
emmanuel_keller
Menu
▾
▴
#201 Double encoding prevents fetch of URLs
Server delivers HTML page with encoded href, i.e. %20 for space:
<a href="path%20one/index.html.">Index/</a>
In the URL browser this would be re-encoded with %25 for the percent char to:
path%2520one/index.html
So the resulting decoded URL is not reachable and the document is never indexed:
path%20one/index.html
see also hints from HTML tidy:
A space should not be contained in a URI (even if it works in all browsers…). This is detailed in RFC1738; look for the word “unsafe”.
Solution:
If the URI contains impermissible characters, replace the characters with permissible ones or encode them using hexadecimal format. In the case of URIs, hexadecimal format uses a percentage character followed by a combination of two letters and digits (a-f and 0-9). The notation for the space character is %20.
If the URI is missing a quotation mark delimiter, add the character.
...
Workaround: change the HTML Parser attribute from Best Score to HtmlUnit solves the problem.
(Schema→Parser list→HTML parser→Edit→HTML parser)