[htdig] a 1.6 gigabyte database that contains no words

[Rick U. <dis...@up...>]

Hello,

I have a 1.6 gigabyte database that is the result of 3 or 4 days worth
of indexing 3 websites defined by the
/usr/local/share/htdig/common/start.url file.

I used the /usr/local/bin/rundig file which seems to have worked.
However, I did not pipe the output to a file : (

-rw-r--r--    1 root     wheel        1.6G Dec 23 20:39 db.docdb
-rw-r--r--    1 root     wheel        2.0k Dec 23 18:55 db.docs.index
-rw-r--r--    1 root     wheel        163M Dec 23 18:55 db.wordlist
-rw-r--r--    1 root     wheel        105M Dec 23 18:55 db.words.db

The htsearch in cgi-bin loads up fine but I cannot find any words at all
whatsoever, no matter what I search for.

I have now increased the size of the following defaults, but not yet 
re-run the rundig script because I know it is going to take days....

#max_head_length:   10000
max_head_length:    100000

#max_doc_size:      200000
max_doc_size:       2000000

but I seem to recall that the last time I attempted to index these
sites, (.jsp driven) that helped -- maybe??.

The problem is that is going to take days to run and might not help.

So, my question is "WTF is in that 1.6 gig database and how do I know in
advance what the current problem is so that I can fix it?"

Thanks,

Rick