Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-999 -- please add further
comments at that location.

Logged In: YES
user_id=1269624

Verified in Archive-It testing. Closing.

Logged In: YES
user_id=924942

Implemented. Commit msg. below. Assigning Karl w/ upped
priority so gets into pending release:

TO TEST:

Run crawl with 'source-tag-seeds' == FALSE. Ensure all works
as before and that no 'source' report when crawl is done.

Then run new crawl with 'source-tag-seeds' set to TRUE. Let
it run a while. Check new report is present at crawl
termination. Check out the content to see it makes some sense.


Implement "[ 1445970 ] New "seed source report" of # of URLs
per host per source"
Patch from Dan Avery (minor changes by St.Ack. Removed custom
serialization/deserialization of Hashtable. Let bdbje do
this for us).
If 'source-tag-seeds' is true, keep a (costly looking) count
per host per seed and
at crawl end, emit a new sources-report.

* src/java/org/archive/crawler/admin/StatisticsTracker.java
(sourceHostDistribution): Added new BigMap.
(saveSourceStats, writeSourceReportTo): Added.