Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-413 -- please add further
comments at that location.

Logged In: YES
user_id=144912

Fixed by clearing up synchronization problems in
StatisticsTracker. Commit comment:

Fix for [ 1204931 ] NPE when viewing crawl report
Partial fix for [ 1221570 ] reports (web ui and to disk)
don't scale
* StatisticsTracker.java
synchronization cleanup
updated for writer-based Reporter interface
avoid composing giant Strings for disk-based report writing

Assigning to QA for verification.

Logged In: YES
user_id=924942

Tom is having same issue. I upped the priority.

Tom Emerson wrote:

> Monitoring an ongoing crawl I was looking at the crawl job
> report. Part way through the "Hosts" section I get an NPE:
>
> [...]
> www.alanwar.com 1208 1184 KB
24s144ms
> www.bentqatar.com 1205 31 MB 15ms
> www.alwaraq.com 1194 6 MB
2m26s581ms
> An error occured
>
> java.lang.NullPointerException
>
> java.lang.NullPointerException
> at
org.archive.crawler.jspc.admin.reports.crawljob_jsp._jspService(Unknown
Source)
> at
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:137)
> at
javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
> at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:358)
> at
org.mortbay.jetty.servlet.WebApplicationHandler$Chain.doFilter(WebApplicationHandler.java:342)
> at
org.archive.crawler.admin.ui.RootFilter.doFilter(RootFilter.java:67)
> at
org.mortbay.jetty.servlet.WebApplicationHandler$Chain.doFilter(WebApplicationHandler.java:334)
> at
org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:286)
> at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
> at
org.mortbay.http.HttpContext.handle(HttpContext.java:1807)
> at
org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:525)
> at
org.mortbay.http.HttpContext.handle(HttpContext.java:1757)
> at
org.mortbay.http.HttpServer.service(HttpServer.java:879)
> at
org.mortbay.http.HttpConnection.service(HttpConnection.java:789)
> at
org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:960)
> at
org.mortbay.http.HttpConnection.handle(HttpConnection.java:806)
> at
org.mortbay.http.SocketListener.handleConnection(SocketListener.java:218)
> at
org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:300)
> at
org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:511)
>
> You may be able to recover by going back
>
>
> I've gotten this on other admin pages too. It seems to be
spurious,
> insofar as I can go back and return to the page, only to
get a bit
> further in the list before it happens again.
>
> Details:
>
> Mac OS X 10.4.1
> (0) tree% java -version
> java version "1.4.2_07"
> Java(TM) 2 Runtime Environment, Standard Edition (build
1.4.2_07-215)
> Java HotSpot(TM) Client VM (build 1.4.2-50, mixed mode)
>
> Heritrix 1.4 binary distro.
>
> Order contains some 2300 seeds, SURT prefix scope,
inferring prefixes
> from seeds (i.e., no external SURT prefix file), 100
toe-threads, 15
> ARC writers.
>
> Filters:
>
> 1) URI regexp exclude filter, with my big "HTML-only" regexp
>
> 2) Midfetch filter to only grab content that's text/html.