google-kongulo / Bugs / #1 uncaught exception causes webpage links not to be downloaded

#1 uncaught exception causes webpage links not to be downloaded

Status: open

Owner: nobody

Labels: None

Priority: 5

Updated: 2005-07-21

Created: 2005-07-21

Private: No

Our local intranet has a webpage which kongulo could
not download. I tracked it down to the processing a
link called
http://sfo-verity1.micromuse.com:9990/mmuse/pages/support/init.jsp?ReferringPage=%2Fmmuse%2Fpages%2Fsearch%2Fadvanced.jsp&Action=Login
What would happen is this:- somewhere within GetRules a
exception was thrown and as it was never caught it
propergated up until it was caught as an IOError during
Crawl (nolinks error message).

This would mean no links would be downloaded at all for
that webpage, rather than just the offending one being
abandoned.

Putting a try/except around the call to GetRules (with
IsCrawlable returning false if an exception was caught)
fixed the problem.