Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-267 -- please add further
comments at that location.

Logged In: YES
user_id=144912

Improvement is best we'll do unless we do a comprehensive
revisit of regexp-based extractors. Closing as fixed.

Logged In: YES
user_id=924942

Tested 5 random pages as well as the two listed above. The
new regex is way faster and extracts same set of links (Had
trouble with arcihve.org; the page was changing on me so
downloaded page local to run the compares).

Logged In: YES
user_id=144912

Improvement committed. Comment:

Improvement for [ 1051072 ] ExtractorHTML takes forever on
worst-case HTML
* ExtractorHTML.java
Tighten up RELEVANT_TAG_EXTRACTOR, by using possessive
qualifiers and [^>] instead of .*, to improve performance on
tricky HTML.

For reference, old and new versions:
// version w/ problems with unclosed script tags
// static final String RELEVANT_TAG_EXTRACTOR =
//
"(?is)<(?:((script.*?)>.*?</script)|((style.*?)>.*?</style)|(((meta)|(?:\\w+))\\s+.*?)|(!--.*?--))>";

// version w/ less unnecessary backtracking
static final String RELEVANT_TAG_EXTRACTOR =
"(?is)<(?:((script[^>]*+)>.*?</script)|((style[^>]*+)>[^<]*+</style)|(((meta)|(?:\\w+))\\s+[^>]*+)|(!--.*?--))>";

Logged In: YES
user_id=144912

May be related to/ duplicate of

[ 951607 ] STUCK in extractorHTML(But works usually!)
https://sourceforge.net/tracker/index.php?func=detail&aid=951607&group_id=73833&atid=539099