Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-887 -- please add further
comments at that location.

Logged In: YES
user_id=144912

Added doc of "+" syntax for SurtPrefixScope. Commit comment:

More doc for [ 1108520 ] SURT needs facelift
* user_manual.xml
added 2 paragraphs explaining '+' syntax for specifying
SURT prefixes in seeds box

With previous doc work and general broader
discussion/understanding of SURTs, closing this issue as
completed.

Logged In: YES
user_id=144912

Taking Andy's suggestion. Commit comment:

Work for [ 1108520 ] SURT needs facelift : accept Andy's
suggestion of SURT-specification in seeds box
* SurtPrefixScope.java
interpret seeds list as source of seeds and directives;
also ensure earlier readPrefixes at initialization() time
* SurtPrefixSet.java
add method for reading mised seeds & '+' directives from
file

SURT doc update forthcoming.

Logged In: YES
user_id=144912

Response from Andy @ LOC on list:
Andy Boyko wrote:
> Gordon Mohr (Internet Archive) wrote:
>
>
>>We'd like feedback...
>> - Are SURT options understandable?
>> - How could they be made easier to use/understand/configure?
>>
>>
>
> It'd be nice to be able to supply auxiliary SURTs for
scoping directly
> in the webapp UI, analogously to how seeds are supplied,
without having
> to use the somewhat-awkward surts-source-file. In
simplest form, just
> an auxiliary textarea, similar to the seed entry box,
would do.
>
> A next step might be to augment the seed entry box so
that, if you're
> converting seeds-as-surt-prefixes, to be able to flag
certain supplied
> seed URLs to be used only as SURT filters rather than as
seeds. For
> example, borrowing syntax I think I've seen IA use, a seed
list:
> http://a.com/
> +http://b.com/images/
> would crawl starting from a.com, and include the b.com
path as an
> additional allowed scope. Functionally, no different than
putting
> http://a.com/ in the seeds and putting a SURT-ified
> http://(com,b,)/images/ in the surts-source-file, but
obviously simpler
> to configure.
>
> That leads to the idea of an analogous "-" prefix, but
then you start
> getting into precedence and all the other stuff that the
NewScopingModel
> is for...
>
> -Andy
> aboy@loc.gov

Logged In: YES
user_id=144912

Expand doc. Solicit input from list about understanding of
SURTs, and whether SurtPrefixScope (with auto
seed-to-surting) should become new default.

Logged In: YES
user_id=924942

If there were a surt scheme, would it be possible to do away
with the hard-to-find options 'surts-source-file' and
'seeds-as-surt-prefixes' -- simplifying surt config (Notion
of mixing surt scheme and normal scheme in scope is an
interesting idea -- would it be hard to do?).

Another option taken from reverse dns would be to add a
suffix to the domain part when its a surt as in:
http://org.archive.www.surt/ or http://com.surt-form/doc/. I
ain't sure I like this one too much -- would have to be some
unique charsequence other than 'surt' or 'surt-form'.

For sure, you can add documentation to compensate for
awkward config (surt is lacking here) but this RFE is about
usability, making it so resort to doc. is minimized.

Logged In: YES
user_id=144912

I think making SURTs look any more like URIs would risk more
confusion and bugs: the 'cognitive dissonance' (and parse
errors) caused by commas and parentheses specifically
protect against misinterpreting SURTs and SURT prefixes.

(Which brings up an important clarification: the SURT form
of a URI will always have matched parenthesis... it's only
the truncated prefix, used for scoping, that will have
fragments of hostnames and open parens. The terminiology may
need cleanup.)

A special scheme, like: 'surt:http://org.archive.www./'
might work, but still has a higher risk of
misinterpretation, and could encourage the idea that regular
URIs and SURT-form URIs can be mixed in the same lists...
which while an interesting possibility, opens other issues
we'd have to tackle.

I prefer to fix any problems here with more documentation,
and clarifying the terminology/uses.