Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-689 -- please add further
comments at that location.

Logged In: YES
user_id=924942

Closing. Igor has at least tried it. Just added autostart
of the JMX Agent by Heritrix if on SUN JDK 1.5.0.

Logged In: YES
user_id=924942

Needs testing by others than I. Then I'll close this.

Logged In: YES
user_id=924942

Made a HeritrixMBean and added cmdline option to start MBean
Server. Added a JMXCLient to run Heritrix advertised
functions. Can 'start' and 'stop'. Need to add more.
Changed category.

Logged In: YES
user_id=924942

Another issue was opened and closed as a duplicate of this
one. Here is text from it:

Feature Requests item #1066441, was opened at 2004-11-14 19:49
Message generated for change (Tracker Item Submitted) made
by Item Submitter
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=539102&aid=1066441&group_id=73833

Category: Configuration
Group: None
Status: Open
Priority: 5
[ 1066441 ] command-line options to trigger recrawls (eg
from scripts)

Submitted By: Gordon Mohr (gojomo)
Assigned to: Nobody/Anonymous (nobody)
Summary: command-line options to trigger recrawls (eg from
scripts)

Initial Comment:
From NLA workshop/Matt: Ability to pipe commands
(starting/configuring previous crawl profiles) from
scripts, at specific times.

Logged In: YES
user_id=924942

I've started a wiki page for diiscussing JMX soln. to thie
prob:
http://crawler.archive.org/cgi-bin/wiki.pl?JmxControlStatus

Logged In: YES
user_id=924942

A call for cmd-line shutdown has been made on the list:
http://groups.yahoo.com/group/archive-crawler/message/547.

If we put up a JMX server on crawler start, it should be easy
enough sending shutdown with a JMX client.

Logged In: YES
user_id=924942

Lowering priority because its been open a while without
getting attention.

Aside from the symmetrical start/stop from the command-line
so our server works like any other server in this regard,
examples of why we'd want to control the crawler from the
command-line might include a cron job to start and stop the
crawler to snapshot a site on a period.

As to the server being unreachable, at least on the
command-line we have the kill -9 fallback. We should do as
the other servers do packaging behind a script the finding
of the pid and the sending of the kill.

Logged In: YES
user_id=892643

As I have no intention of getting into the whole
"RMI/SOAP/Remote JMX/JMS/HTTP POST/etc" I'm reassigning this
to Michael.

Personally I'm content with only providing graceful shutdown
via WUI. Odds are that if that isn't responding then RMI and
similar wouldn't be either. And no one has given me a really
good example of why a script would want to do this.

Logged In: YES
user_id=924942

Signal handling is JVM particular.

From the cmdline, we can't do any more than a "kill -9 `echo
heritirx.pid`". For a graceful shutdown we need to write a
java RMI/SOAP/Remote JMX/JMS/HTTP POST/etc. client to send
the message w/ appropriate authorizations.

Here's a list of cmd-line tasks such a client might do:

+ Shutdown
+ Graceful shutdown (Checkpoint?, shutdown of webserver,
stop of all threads and sys.exit()).
+ (Emergency) checkpoint

What else would we ever want the client to do?

Logged In: YES
user_id=144912

Investigate: is there an alternate signal that can be caught
in Java and trigger a graceful shutdown (or, when available,
a checkpoint & shutdown)

Logged In: YES
user_id=892643

I've implemented this for the UI. Option availible under the
Console.

Some steps are taken to terminate ongoing crawls in an
orderly manner.