Heritrix: Internet Archive Web Crawler

From Karl:

Example:

Sending the Service bean alert=o gives:

tcl 11:34:13 /home/webcrawl/tests/archiveit/Interaction
53 % ask crawling024 9097 Heritrix alert=0
Exception in thread "main"
javax.management.RuntimeMBeanException:
RuntimeException thrown by the invoke method of the
Dynamic MBean
at
com.sun.jmx.mbeanserver.DynamicMetaDataImpl.invoke(DynamicMetaDataImpl.java
:227)
at
com.sun.jmx.mbeanserver.MetaDataImpl.invoke(MetaDataImpl.java:220)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanSe
rverInterceptor.java:815)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:784)
at
com.sun.jmx.remote.security.MBeanServerAccessController.invoke(MBeanServerA
ccessController.java:429)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl
.java:1408)
at
javax.management.remote.rmi.RMIConnectionImpl.access$100(RMIConnectionImpl.
java:81)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMICo
nnectionImpl.java:1245)
at java.security.AccessController.doPrivileged(Native
Method)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConn
ectionImpl.java:1348)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java
:782)
at
sun.reflect.GeneratedMethodAccessor22.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp
l.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
at sun.rmi.transport.Transport$1.run(Transport.java:153)
at java.security.AccessController.doPrivileged(Native
Method)
at
sun.rmi.transport.Transport.serviceCall(Transport.java:149)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:
701)
at java.lang.Thread.run(Thread.java:595)
at
sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemote
Call.java:247)
at
sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:223)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:126)
at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
at
javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown
Source)
at
javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke
(RMIConnector.java:969)
at Heritrix.doBeanOperation(Client.java:566)
at org.archive.jmx.Client.doSubCommand(Client.java:348)
at org.archive.jmx.Client.doBean(Client.java:299)
at org.archive.jmx.Client.doBeans(Client.java:259)
at org.archive.jmx.Client.execute(Client.java:233)
at org.archive.jmx.Client.main(Client.java:159)
Caused by: java.lang.ClassCastException: java.lang.Integer
at org.archive.crawler.Heritrix.invoke(Heritrix.java:2078)
at
com.sun.jmx.mbeanserver.DynamicMetaDataImpl.invoke(DynamicMetaDataImpl.java
:213)
at
com.sun.jmx.mbeanserver.MetaDataImpl.invoke(MetaDataImpl.java:220)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanSe
rverInterceptor.java:815)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:784)
at
com.sun.jmx.remote.security.MBeanServerAccessController.invoke(MBeanServerA
ccessController.java:429)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl
.java:1408)
at
javax.management.remote.rmi.RMIConnectionImpl.access$100(RMIConnectionImpl.
java:81)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMICo
nnectionImpl.java:1245)
at java.security.AccessController.doPrivileged(Native
Method)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConn
ectionImpl.java:1348)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java
:782)
at
sun.reflect.GeneratedMethodAccessor22.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp
l.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
at sun.rmi.transport.Transport$1.run(Transport.java:153)
at java.security.AccessController.doPrivileged(Native
Method)
at
sun.rmi.transport.Transport.serviceCall(Transport.java:149)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:
701)
at java.lang.Thread.run(Thread.java:595)

But the web interface reads:

Time: Jan. 27, 2006 18:35:08 GMT
Level: SEVERE
Message:

Problem java.lang.NegativeArraySizeException occured
when trying to process
'http://www2.synapse.ne.jp/garappa/bbstick.swf' at step
ABOUT_TO_BEGIN_PROCESSOR in ExtractorSWF
(in processor ExtractorSWF)

Exception:

java.lang.NegativeArraySizeException
Stacktrace: java.lang.NegativeArraySizeException
at
com.anotherbigidea.io.InStream.read(InStream.java:164)
at
org.archive.crawler.extractor.ExtractorSWF$1.readOneTag(ExtractorSWF.java:1
23)
at
com.anotherbigidea.flash.readers.SWFReader.readTags(SWFReader.java:102)
at
com.anotherbigidea.flash.readers.SWFReader.readFile(SWFReader.java:92)
at
org.archive.crawler.extractor.ExtractorSWF.extract(ExtractorSWF.java:129)
at
org.archive.crawler.extractor.Extractor.innerProcess(Extractor.java:67)
at
org.archive.crawler.framework.Processor.process(Processor.java:103)
at
org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:306)

at
org.archive.crawler.framework.ToeThread.run(ToeThread.java:153)


I suspect the problem is the ClassCastException, maybe
a badly
formatted conversion from string to int? I'll file a
bug if you like.

Let me know if I you need more info,
--K.