Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-415 -- please add further
comments at that location.

Logged In: YES
user_id=1269624

Test in harness, verified bug existence, verified fix. Closing.

Logged In: YES
user_id=144912

Added rough feature to make any future inadvertently ignored
seed-file items (non-comment whitespace-trimmed lines)
prominent in the UI's seeds report, so that problems can be
corrected. Commit comment:

feature suggested by [ 1207378 ] seeds listed without
scheme, but with path, being ignored
* webapps/admin/reports/seeds.jsp
If any 'ignored seed items' were recorded the last time
seeds were imported to the frontier, report them on the
seeds report for operator review
* CrawlJob.java
add convenience method for getting the ignored seed
items (if any) as a string
* CrawlScope.java
adjust seedsIterator to allow optional passed-in writer
for reporting ignored seed items
* AbstractFrontier.java
extend loadSeeds to either save any ignored seeds, or
delete older saved ignored seeds if none on most recent scan
* AdaptiveRevisitFrontier.java, HostQueuesFrontier.java
mimic AbstractFrontier's saving of any ignored seeds
* SeedFileIterator.java
allow ignoredWriter to be any Writer (not just
BufferedWriter)

==
Bug believed fixed and extra insurance added against further
problems. Assigning to Karl for final bug disposition.

Logged In: YES
user_id=144912

Immediate fix committed. Commit comment:

Fix for [ 1207378 ] seeds listed without scheme, but with
path, being ignored
* SeedFileIterator.java
be more tolerant of abbriviated URI formats: try adding
"http://" to the front of any entry-line that doesn't
already have a URI scheme
also: new facility to report ignored entry lines to
optionally supplied writer (not yet used elsewhere)
also: close reader/writer when done
* SeedFileIteratorTest.java
unit test for SeedFileIterator, including pattern which
triggers referenced bug in unfixed code
* TransformingIteratorWrapper.java
new hook for cleanup (such as IO closing) when iterator
exhausted
* DomainScope.java, HostScope.java, PathScope.java, CrawlScope
add hooks for 'just-in-case' closing of SeedFileIterator
instances (if used)

===
Leaving open to add new section to seeds-report,
highlighting any non-comment entries in a seeds file that
were ignored during seed parsing (in progress).