Hello,
I really appreciate your work on jspider. I reported this small bug
in order to improve this nice project.
When parsing broken robots.txt like e.g.
User-Agent:
Disallow:
it leads to
java.lang.StringIndexOutOfBoundsException: String index out
of range: -1
at java.lang.String.substring(String.java:1480)
at java.lang.String.substring(String.java:1447)
at
net.javacoding.jspider.core.util.html.RobotsTXTLineSet.findUserAgent(Unknown
Source)
Regards
Vojtech
Logged In: YES
user_id=363928
Bug is fixed.
JUnit testcase for this one.
Diff: