Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-528 -- please add further
comments at that location.

Logged In: YES
user_id=144912

Considering fixed. Ideally will become moot when a
comprehensive settings redo is in place.

Logged In: YES
user_id=144912

Applying pandae's suggested fix as simple, low-risk fix for
1.8. (Thanks!) Commit comment:

Initial fix for [ 1379040 ] regex for midfetch filter not
being stored in crawl order
* FetchHTTP.java
make midfetchfilters a non-expert setting, working
around problem with non-expert settings nested inside expert
settings

--
Should re-investigate post-1.8.

Logged In: YES
user_id=962291

proposed fix:

file to edit:
src/java/org/archive/crawler/fetcher/FetchHTTP.java

line to remove:
this.midfetchfilters.setExpertSetting(true);

Logged In: YES
user_id=962291

I further analyzed the issue and it seems like I found the
cause.
The settings for the midfetch-filters will only get saved in
"expert settings" mode. I think this behaviour is
unintentional because those settings are also displayed in
non-expert mode.

So I think there are two possible solutions. Either make
sure that those settings will only be seen in expert mode
(which will require to not only mark the midfetch-filter as
"expert mode" but also all its child settings) - or the
solution I would prefer simply not making this filter an
expert setting.

Logged In: NO

I am experiencing the same issue using 1.7.3 CVS code
(checked out yesterday).