Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-933 -- please add further
comments at that location.

Logged In: YES
user_id=924942

Committed. Message below. Closing.

(Thanks Christian).

[debord 648] heritrix > more /tmp/diff.txt
Apply patch '[ 1207898 ] [contrib] WorkQueueFrontier: Store
allQueues in RAM
if poss.'.
Contributed by Christian Kohlschuetter. Tested and reviewed
by St.Ack

Here is comment on the below from Christian:

"Currently, the "allQueues" Map is always backed by a
Bdb database (using a CachedBdbBigMap).

In fact, this is not necessary in all cases, especially
when the BucketQueueAssignmentPolicy is in use (it has
a fixed number of queues).

Instead, allQueues can be stored in a HashMap, if the
assigned QueueAssignmentPolicy instance has a maximum
number of queues lower than a certain number (is 3000
ok?) and if the frontier implementation stores its
WorkQueue payload on harddisk.

This patch provides hooks to retrieve this information
(QueueAssignmentPolicy.maximumNumberOfKeys() and
WorkQueueFrontier.workQueueDataOnDisk()), so it should
be straightforward to apply this to non-standard
implementations, too."

* src/java/org/archive/crawler/frontier/BdbFrontier.java
*
src/java/org/archive/crawler/frontier/BucketQueueAssignmentPolicy.java
(workQueueDataOnDisk): Added.
*
src/java/org/archive/crawler/frontier/QueueAssignmentPolicy.java
(maximumNumberOfKeys): Added default implementation.
* src/java/org/archive/crawler/frontier/WorkQueueFrontier.java
Added test that overrides can influence that allows
determining whether
or not to use BigMap for allQueues.
(workQueueDataOnDisk): Added abstract method that
overrides must implement.