Heritrix: Internet Archive Web Crawler

Hello,

I just noticed that the latest modification of
ExtractorHTML in CVS HEAD causes probably
unsubstantiated alerts, for example:

> Problem java.lang.NullPointerException occured when
trying to process
'http://www.nthposition.com/author.php?authid=281' at
step ABOUT_TO_BEGIN_PROCESSOR in ExtractorHTML:
>
> Associated Throwable: java.lang.NullPointerException
>
> Stacktrace:
> java.lang.NullPointerException
> at
org.archive.crawler.extractor.ExtractorHTML.innerProcess(ExtractorHTML.java
:352)
> at
org.archive.crawler.framework.Processor.process(Processor.java:102)
> at
org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:273)

> at
org.archive.crawler.framework.ToeThread.run(ToeThread.java:143)

This is caused by the fact that curi.getHttpRecorder()
may return null (javadoc says so), thus an assignment
like cs =
curi.getHttpRecorder().getReplayCharSequence()
(ExtractorHTML, line 352) is not guaranteed to be
evaluated without throwing a NullPointerException. This
exception is no longer caught directly, so it is thrown
up too far, triggering the reported alert.

A patch fixing this behaviour is attached.


Christian Kohlschütter