[htdig-dev] 3.2.0b4 slowing down of dig?

[Budd, S. <s....@ic...>]

Hello.

 Question..  what is the cause of the slowing down of a dig.  When dig start
I get more than 200 pages/minute  and nearly  90 % cpu usage  but near the
end, I am seeing  20 pages/minute and at most 10% cpu usage.

If I fetch the pages from the site being dug at the end, the page comes in
very fast ( 100Mb line ) so access to the system is ok.  

 I have set the db cache to 1.3 Gbyte but htdig seems to only use about
260Mbyte.

I am putting the databases into /tmp i.e into memory.  still plenty of space
in memory.

du -k /tmp
8 /tmp/.pcmcia
8 /tmp/.X11-unix
8 /tmp/.X11-pipe
8 /tmp/root
8 /tmp/kshroot
8 /tmp/ppp
1351528 /tmp/college
1351624 /tmp



here is a top.

last pid:  3903;  load averages:  0.05,  0.06,  0.07
10:43:05
45 processes:  44 sleeping, 1 on cpu
CPU states: 99.2% idle,  0.2% user,  0.6% kernel,  0.0% iowait,  0.0% swap
Memory: 2048M real, 351M free, 1613M swap in use, 2029M swap free

   PID USERNAME THR PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
  3607 ppp        1  18    4  265M  263M sleep  762:28  5.17% htdig
  3900 ppp        1  58    0 2584K 1704K cpu      0:05  0.15% top
   306 root      12  49    0 3472K 2456K sleep    1:54  0.00% mibiisa

a -v  log ...............
185822:170323:11:http://www.cc.ic.ac.uk/college/onlinedocs/SASOnlineDocV8/sa
sdoc/sashtml/gref/zeration.htm: ******* size = 3599
185823:172065:11:http://www.cc.ic.ac.uk/college/onlinedocs/SASOnlineDocV8/sa
sdoc/sashtml/accdb/z1258182.htm: ******* size = 3228
185824:172066:11:http://www.cc.ic.ac.uk/college/onlinedocs/SASOnlineDocV8/sa
sdoc/sashtml/accdb/z-dbnull.htm: ******* size = 4636