Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-991 -- please add further
comments at that location.

Logged In: YES
user_id=924942

Applied with below message. Closing.

Implementing '[ 1388275 ] [contrib] Preselector
ATTR_ALLOW_BY_REGEXP'.
Contribution by Karl Wright of Metacarta (kwright at
metacarta dot com).
From email with Karl:
>> >> This contribution allows one to set a curi inclusion
regexp in the
>> >> preselector. This turns out to be very handy if your
crawl should
>> >> grab only an identified set of mime types. I set it
up to pretty
>> >> carefully match the exclusion regexp already in place.
>> >> Once again this is based on the 1.5 branch head.
> >
> > Patch looks good. Thanks for the contrib.
> >
> > Here's a question on motivation. So your regex looks at
file endings to
> > figure mimetypes? And you couldn't make this work using
the
> > ATTR_BLOCK_BY_REGEXP? (Negative regexes -- regexes for
things that do not
> > match an expression -- are awkward to write. This is
probably sufficent
> > reason for including your patch.)
> >
Right- we had a messy regexp, and making it negative was
well-nigh impossible.

* src/java/org/archive/crawler/prefetch/Preselector.java
Add attribute ATTR_ALLOW_BY_REGEXP.
(innerProcess): Test CrawlURI.toString against
ATTR_ALLOW_BY_REGEXP if
present.

Logged In: YES
user_id=924942

More on this patch. Below quote is email from me to Karl:

> Patch looks good. Thanks for the contrib.
>
> Here's a question on motivation. So your regex looks at
file endings to figure mimetypes? And you couldn't make
this work using the ATTR_BLOCK_BY_REGEXP? (Negative regexes
-- regexes for things that do not match an expression -- are
awkward to write. This is probably sufficent reason for
including your patch.)
>

Right- we had a messy regexp, and making it negative was
well-nigh impossible.