Re: [htdig] Help with incremental mode

[Andreas M. <an...@an...>]

Hello!

Aha, now I whats going on. The original databases created with rundig
looked like this:
-rw-r--r-- 1 root root   84787200 2006-08-30 07:04 db.docdb
-rw-r--r-- 1 root root   21094400 2006-08-30 07:04 db.docs.index
-rw-r--r-- 1 root root  400564224 2006-08-30 06:56 db.excerpts
-rw-r--r-- 1 root root 1097656320 2006-08-30 09:38 db.words.db
-rw-r--r-- 1 root root      16384 2006-08-30 09:38 db.words.db_weakcmpr

Now with htdig -vv running it seems to do what I want:
-rw-r--r-- 1 root root   84787200 2006-09-06 21:05 db.docdb
-rw-r--r-- 1 root root   21094400 2006-09-06 21:05 db.docs.index
-rw-r--r-- 1 root root  400596992 2006-09-06 21:04 db.excerpts
-rw-r--r-- 1 root root 1100663808 2006-09-06 21:05 db.words.db
-rw-r--r-- 1 root root      16384 2006-08-30 09:38 db.words.db_weakcmpr

Great! Everything seems to work. I will see how long it takes to update
the databases.

Thank you very much for your help!

Kind regards

  Andreas Meyer


<mic...@bt...> schrieb:

>  Andreas,
> Just realised that I sent that message direct, and not to the list as
> intended!
> 
> It sounds like you may be mis-interpreting the   -v  responses - it
> might help to move up to -vv or -vvv.  
> When in 'update' mode, htdig still needs to make a 'head' request for
> every page on the site, expecting to recieve a 403 'Not Modified'
> response for most pages. Only those that have been modified will be
> downloaded in full.
> 
> Does that explain what you are seeing?
> 
> Mike
> 
> 
> > -----Original Message-----
> > From: Andreas Meyer [mailto:an...@an...] 
> > Sent: Monday, September 04, 2006 7:03 PM
> > To: Brockington,MJ,Michael,JPGA4Y D
> > Subject: Re: [htdig] Help with incremental mode
> > 
> > Hi!
> > 
> > <mic...@bt...> schrieb:
> > 
> > > Apart from -v,  what options are you calling htdig with? 
> > 
> > I didn't call htdig with any other option than -v.
> > 
> > > Have you read  http://www.htdig.org/rundig.html   
> > particularly the last
> > > section?
> > 
> > OK, did that. I didn't use the -a option. The only thing I
> > changed in the rundig is DBDIR=/daten1/htdig/db.
> > I thought after the rundig completet after four days,
> > running htdig -v would incremental add the newly added files.
> > 
> > This is the corresponding line in /etc/htdig/htdig.conf
> > database_dir:           /daten1/htdig/db
> > 
> > hm?