Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-149 -- please add further
comments at that location.

Logged In: YES
user_id=144912

Applied immediate fix: removed the
weakreference/softreference flushable cache mechanism.
Closing; ServerCache growth will be revisited by [ 1020779 ]
Never OOM.

Fix for [ 958096 ] Flushing CrawlServers problematic
* ServerCache.java
Revert back to strong/direct references, because risk of
OOMs is better, for now, than hysteresis.

Logged In: YES
user_id=144912

At the end of the ~2 week umich crawl, definitely saw
redundant refetches of dns and robots of remaining sites,
sometimes sooner than 1 minute after last fetch. Suspect
that flushing of CrawlServer instances while queues were
briefly snoozed is the culprit.

CrawlServer cache should probably be a BDB-backed
collection, fronted by both a soft-reference cache (that
gets flushed in low-mem conditions), and perhaps also a hard
MRU cache of the last N*(# of toethreads) used CrawlServer
instances, so that nothing that's 'active' gets churned to
disk.

Logged In: YES
user_id=144912

Should be fixed for 1.2.

Logged In: YES
user_id=144912

This enhancement has problems...

- In a broad BdbFrontier crawl starting from
directory.google.com, I've seen what appears to be
dns-flushing-hysteresis: by the time a non-DNS URI comes up,
the DNS info has been lost to a soft-reference flush,
triggering a new DNS lookup. But the info doesn't stick
around long enough to be there for the first non-DNS URI,
causing a repeat.

- HostQueuesFrontier/KeyedQueue attempts to prevent this by
having any active KeyedQueue hold a reference to the
matching CrawlServer, thus preventing its GC. However, since
the CrawlServer <-> KeyedQueue mapping is not always
one-to-one, this will eventually show problems, too.

A fix could be for BdbFrontier and HostQueuesFrontier to
hold sets of CrawlServers against GCing; better though would
probably be to convert eh CrawlServer cache into a
disk-persisted collection like those offered by BDBJE.

Logged In: YES
user_id=144912

ServerCache has been changed in HEAD to use soft references
-- all CrawlServers (and CrawlHosts) not strongly referenced
can be GCd in low-mem conditions.

Logged In: YES
user_id=144912

Should be easy to achieve this by making the ServerCache use
SoftReferences and a WeakDictionary.

To keep 'active' CrawlServers from ever being discarded,
KeyedQueue could hold a CrawlServer while ACTIVE.

One harm to discarding a CrawlServer would be needing to
redo the IP and Robot lookups... but that's minor, and the
info could be persisted to disk if necessary.

Another open issue: would losing the credentials avatars
when a CrawlServer is discarded be any concern? (Or woudl
they just be refreshed when needed?)