Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-563 -- please add further
comments at that location.

Logged In: YES
user_id=924942

G and I agreed that instead we should commit this:

Index: heritrix
===================================================================
RCS file:
/cvsroot/archive-crawler/ArchiveOpenCrawler/src/scripts/heritrix,v
retrieving revision 1.20
diff -u -r1.20 heritrix
--- heritrix 21 Feb 2006 19:56:17 -0000 1.20
+++ heritrix 5 May 2006 22:25:35 -0000
@@ -75,7 +75,9 @@
if [ -z "$JAVACMD" ]
then
# It may be defined in env - including flags!!
- JAVACMD=$JAVA_HOME/bin/java
+ # See '[ 1482761 ] BDB Adler32 gc-lock OOME risk' for
why we include the
+ # 'je.disable.java.adler32'.
+ JAVACMD="$JAVA_HOME/bin/java -Dje.disable.java.adler32=true"
fi

I committed it. Giving issue back to G. Below is message:

* src/scripts/heritrix
Fix for '[ 1482761 ] BDB Adler32 gc-lock OOME risk'
Add system property that invokes the bdbje adler32
implementation rather than use the gc-locking JVMs
version. This commit was reviewed by Gordon.

Logged In: YES
user_id=144912

Suggested fix is attached for review: add the necessary BDB
JE property flag to our default Java flags in the heritrix
launch script. Also will be adding a note to release notes
for people who set their own JAVA_FLAGS.

Igor has run with this in his IT test crawl; it restores the
same BDB behavior as was the default in je-2.0.90, or any
time a pre-1.5 JVM is detected -- so it should be a safe
change.