Heritrix: Internet Archive Web Crawler

Igor reports and I've verified that increasing
queue-total-budget does not have the desired effect of
unretiring queues and increasing their effective budgets.

The queues are momentarily unretired, but retain their
original budget, and thus are immediately retired again.

The problem appears to be that changes to the
total-budget setting are only noticed in
WorkQueueFrontier.noteAboutToEmit(). Meanwhile, the
temporary unretiring of queues (from
WorkQueueFrontier.kickUpdate) results in them being
inactive, but then when their chance at activation
comes up, no CrawlURIs are emitted so the old
total-budget is never updated from settings.

The reason the settings are not consulted every time
the total budget is needed for a check is that usually,
during the queue-juggling steps where budget is
important, there is no current CrawlURI to serve as the
settings context. In fact, in replenishSessionBalance,
this is worked around by peeking for a temporary
CrawlURI to use as the context.

(This could cause other confusing behavior in the
future -- if CrawlURIs subject to different overrides
share the same queue (if IP politeness or queue-forcing
of subdomains into one queue), the settings in effect
for the queue would be somewhat random, depending on
which CrawlURI was in position when the queue was
determining its working values.)

I thought this had worked previously, but looking at
the relevant code back a few revisions before 1.6
doesn't show a recent change that would break it, nor
do I recall a change, and I don't see anything in the
current code suggesting it was disrupted from a
previously working condition.

A potential fix would be to reset the total budget at
the same time as the session balances are replenished
-- this would give any queue coming into active status
a chance to operate under its new budget parameters. It
would still leave the potentially confusing behavior
that a total-budget change would not affect an active
queue until after it goes inactive (by going over its
session or total budget) and then active again... but
that seems a minor concern.