Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-400 -- please add further
comments at that location.

Logged In: YES
user_id=144912

The OOM guard of a simple threshold is in place; Stack
further made it property-configurable. (Property:
org.archive.crawler.datamodel.CrawlURI.maxOutLinks).
Closing; further configurability or robustness against
arbitrary numbers of legit outlinks should be considered as
part of new RFE [ 1242766 ] Massive outlinks: make threshold
optional, configurable

Logged In: YES
user_id=144912

Taking and upping priority as this is a possible cause of
as-yet-unexplained OOMs in giant-heap crawls.

Committing initial truncation of outlinks at 6000 and
crawlURI annotation for crawl.log that this has been done:

First fix for [ 1192029 ] OOME guard against pages of
thousands of links
* CrawlURI.java
enforce a max number of outlinks; discard overflow,
noting count of discarded, and annotating CrawlURI so it's
identifiable for further investigation in the crawl.log

Logged In: YES
user_id=144912

I don't know what a reasonable default max-links-to-extract
would be. Maybe 5,000? Perhaps the Extractors could remember
the most extreme counts they encounter for the Processors
report, to help us notice the outliers.

That extraction was cut-off for a certain resource should be
noted somewhere -- an alert or at the very least a crawl.log
annotation.

If we thought any such giant link collections were
legitimate, or wanted protection against the worst-case
scenario of several max-links pages being processed at once,
we could make the link collections overflow to disk as with
other crawler datastructures. At thsi point, I think that
step would be overkill.