Heritrix: Internet Archive Web Crawler

Below is a note from Kris:

"...What I'm thinking is that when the user changes the
setting any new ARC files created after that point will
be written to the new location but any currently open
ARCs will be completed where they are.

"The reason for this is that a crawl here was a bit
misconfigured and is downloading ARCs to a disk with
insufficient space for the amount of data we are
gathering. Even moving the ARCs away the crawler will
exhaust available space overnight..."

This will be awkward to do because the arc dir is set
once on initialization of ARCWriterProcessor. The
arcdir location is passed into the ARCWriterPool which
in turn passes it to ARCWriters as they are created.

One thing to look at is having ARCWriterProcessor
monitor for arcdir changes. It could then update
ARCWriterPool which could then pass on the message to
all members of the ARCWriter pool (If the arcdir
directory were a static it'd make things a little easier).