Heritrix: Internet Archive Web Crawler

Especially when a URI connection fails, we may not want
to use a constant-length retry-delay, but rather one
that grows with each failure. This could be achived by
providing a factor which is raised to the power of the
number of retries, that is multiplied by the 'initial'
retry-delay.

That is, delay = base_delay *
(backoff_factor^consecutive_retries).

A typical value for base_delay might be 1 minute; a
typical factor for backoff_factor might be 2. Then
retries would occur after 1 minute, then another 2,
then another 4, then another 8, etc.

When a site is really down for many hours (or a day or
more), this sort of patience could be better than
considering a URI a failure after only an hour or two
of retrying.