Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-862 -- please add further
comments at that location.

Logged In: YES
user_id=144912

Core implementation completed. Commit comment:

Implementation (essentially complete) for [ 1078016 ]
'Economic' frontier which defers low-value URIs
* BdbFrontier.java
Add tracking of total/lifetime budget and expenditures
per queue;
Add concept of 'retired' queue, which still has items
but is over total budget, and so permanently inactive
(unless the operator intervenes)
Clean up terminology to better distinguish 'budget'
(limit/threshold) and 'balance' (running total of units left
in current activation)
Improve per-queue reporting to show retired queues,
budgetting figures

Logged In: YES
user_id=144912

Implemented zero-cost policy as way to eliminate budgetting
on/off toggle -- instead can disable budgetting by makingall
URIs zero-cost. COmmit comment:

Support for [ 1078016 ] 'Economic' frontier which defers
low-value URIs
* ZeroCostAssignmentPolicy.java
No-op costing policy; allows elimination of
'use-budgetting' setting, instead just set URIs to have zero
cost, and budget mechanisms never have effect.

Logged In: YES
user_id=144912

Commit:

Work (in progress) towards [ 1078016 ] 'Economic' frontier
which defers low-value URIs
* BdbFrontier.java
Make budget-driven deactivation (and thus rotation
between queues) configurable: whether it is in effect at
all, and how large the budget should be on each
reactivation. Current cost of all URIs is '1'.

More notes:
* Using hold-queues=false and use-budgetted-rotation=false
retains classic behavior, where all queues ever created are
round-robined
* Using hold-queues=true and use-budgetted-rotation=false
mimics 'site-first' behavior: a queue remains active until
it is exhausted of URIs. Inactive queues only become active
when the crawler has nothing on active queues ready to
crawl. A site in endless trap-junk that's active will stay
active, starving other sites of attention.
* Using hold-queues=true and use-budgetted-rotation=true
will cause a site that has been active for a while (until
its 'budget' is used up) to be deactivated and placed at the
back of the inactive queues list. This increases the chances
the crawler will need to activate queues from the front of
the inactive queues list. The intended effect is to
concentrate on a site to a degree, but then even if it's not
finished, let other sites have a chance to catch up.

The current 'budget' per activation is 3000, and each
attempt at processing a URI (even attempts that fail or
trigger prerequisite tries) costs 1 budget unit. So queues
rotate out of the active positions about every 3000 URIs.
Configurable URI costing is coming up next.

Logged In: YES
user_id=144912

First step committed:

Work towards [ 1078016 ] 'Economic' frontier which defers
low-value URIs
* BdbFrontier.java
Add 'hold-queues' option (site-first) to keep queues
'inactive' until they are needed to keep crawler threads
fully busy