Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-812 -- please add further
comments at that location.

Logged In: YES
user_id=924942

JProfiler gave us licenses.

Logged In: YES
user_id=924942

Ok. This JProfiler looks pretty good. The profiled
application will crash afer a while for the more austere
profiling configs (i.e. Full Instrumentation) but runs long
enough to get a picture.

The value of the profiiler will come with familiarity and
frequent use; we'll be able to look at the graphs and see if
a change made things better or worse but we need to be used
to how the profiler to be able to detect change.

I'll write to JProfiler to see if they'll give us a copy for
free or a price break.

Meantime, spending time trying to make sense of the info.

Logged In: YES
user_id=924942

Can't get another optimizeit eval license without making up
some address somewhere. I wanted to confirm it does a
better job Besides says requiremens are red hat 7.3.
Tried starting it on debian and gave out message about
installing rpms.

Was going to give up on jprofiler for getting a better
picture of CPU use. and then I used the sampling rather than
recording of exact use and now I get a good picture of CPU
use (Don't know whether I should believe it or not, but it
looks good). Will attach it to the performance wiki page
soon as I can get it out (First attempt crashed). Picture
is of a broad crawl with 200 threads crawling 50 porn seeds.
Its pretty.

Logged In: YES
user_id=924942

Tried jprofiler to see if I could get better reporting on
CPU usage. $500. Very nice -- fancy wizards, beautiful
views -- but remote app crashes when I try to disable the
sun and apache filters (The CPU usage shows ToeThread#run at
90% but not what its doing in #run. In YJP could infer it
calling processCrawlURI by looking at percentages). Remote
app is using jdk1.5.0 because its ops03 machine with 2.6
kernel and NPTL.

Trying on a 1.4.2 crawl machine, still can't get any more
granularity than ToeThread.#run. Nothing in help nor
configuration seems to speak to this. Giving up on this
profiler.

Here is start script:

stack@ops03:~$ more jprofiler.sh
#!/bin/sh
JAVA_HOME=/usr/local/jdk1.5.0/
PATH=$JAVA_HOME/bin:$PATH
JPROFILER_HOME=$HOME/jprofiler3/
LIB=$JPROFILER_HOME/bin/linux-x86
export LD_LIBRARY_PATH=$LIB:$LD_LIBRARY_PATH
export JAVA_OPTS="-Xmx1024m -Xint -Xshare:off
-Xrunjprofiler:port=8849
-Xbootclasspath/a:$JPROFILER_HOME/bin/agent.jar"
$HOME/heritrix/bin/foreground_heritrix

Logged In: YES
user_id=924942

yjp is annoying in that I can't export the reports as HTML.

Here is start script I used on remote side to get profiling
to work:

stack@ops03:~$ more yjp.sh
#!/bin/sh
LIB=$HOME/yjp-2.5.2-build316/bin/
export LD_LIBRARY_PATH=$LIB:$LD_LIBRARY_PATH
export JAVA_OPTS="-Xmx1024m -Xrunyjpagent:cpu=sampling "
$HOME/heritrix/bin/foreground_heritrix

Logged In: YES
user_id=924942

The yourkit profiler is very basic. Takes cpu and memory
snapshots. You have to copy them from remote machine to
local and load it up into the viewing tool. The cpu
rendering lists percenage usage by package down to class
level and then to method, but it does not make connection
between classes so you'll see that the HTTPFetcher consumes
80% of CPU in its innerProcess method but it won't say which
of the methods called by innerProcess are responsible; you
have to infer by looking at other places where 80% or
sub-80% are consumed and assume thats where innerProcess
went next (In other profilers, they'll follow the usage
till you can drill down no more).

Logged In: YES
user_id=924942

Looking at yourkit profiler. Its 300 euros.
http://www.yourkit.com/download/index.jsp.

Logged In: YES
user_id=924942

jMechanic is no longer actively supported. Suggests using
http://www.eclipse.org/hyades/ instead. I tried this but it
has explicit libc version requirements. I ain't up for
moving my libc version back a bunch of versions to make this
profiler work.

JProfiler is 500 dollars per single-user.

ejp seems to be dead since january.

JMP pages won't load. Server must be down (The debian
packages for it are broken; can't resolve dependencies. One
of the packages is missing).

Logged In: YES
user_id=924942

Running locally profiliing CPU, soon after startup it says
frontier#next is responsible for up to 90% of cpu time but
tells me no more than that. I try to set it up to debug
threads and it looks like it sees the two PoolThreads only
and all of the other thread events are dump out on console.

Giving up on this profiler.

Logged In: NO

I looked at commercial profilers: Optimizeit is 700 dollars
plus for single user and 2400 for the enterprise edition.