[htdig-dev] Re: Feedback on configuring ht://Dig?

[Jonathan B. <ba...@ps...>]

On 06/06/03 23:30, Lachlan Andrew wrote:
>Regarding the time taken to create your  htdig.conf  file, I
>realise that this is a problem.  Can you suggest what would
>have speeded up the process?

Not really.  But here's the story.  I maintain what seems to be a
very large site: http://finzi.psych.upenn.edu.  The search
function there searches about 4 years of a VERY active mailing
list, plus other stuff.

I started off using rundig.  It would crash, sometimes after 3
hours of running.  Then I would remove some attachments from the
mail archives and start over, and then usually it worked.  (It
might have also worked if I had recited an incantation and just
started over.)  But finally I gave up this procedure, and for
each monthly update I now use htdig instead of rundig.  No more
crashes, even with all the attachments.  But this take 5 hours.

I just looked at the manual again and realized that I can
probably make separate databases for new stuff and old stuff.  I
had seen that before but I was afraid to try it.  But that was
before I gave up rundig.  So now I will try that next month and
see if it works.  If it does, I'm in good shape.

One thing that isn't clear to me is whether I can keep using
htmerge repeatedly, merging each new month's mail with the result
of the last merge.  This is not the whole thing.  The site also
deals with "functions," and I have to re-do these every month
because many of the files are updated.  So I would merge the mail
into a mail file, then merge that with the new functions file.
But probably this will work.  So I will try it.

Aside from this, it took me forever to figure out how to get my
index to stop including stuff outside of my local site.  I think
I've finally succeeded in that, but it was a lot of trial and
error.  I don't have time now to read through the manual to
figure out why it took me so long.

This is all just for your information.  Probably if I had just
RTFM, over and over, I would have found things faster.

I must say that this site I run is much used and much
appreciated, so I intend to maintain it.  And I think it is some
of the features of htdig that make it so useful.

Jon