I'm running htdig 3.1.6 and indexing a site that has 20,000+ pages. I
had previously done this with 3.2b5 and it took over 30 hours to
finish; before I start working on getting incremental indexing working
I thought I'd see if 3.1.6 was any faster. So far results are mixed.
Judging from the last modification times on the db files, it took just
under an hour for htdig and htmerge to run. The next step is htnotify,
and that's been going (consuming most of the CPU) for almost 24 hours.
Something is clearly amiss.
I found the "prevent htnotify from looping endlessly" patch but it
claims to be for a Solaris 8 bug. And I'm really wondering whether I
need htnotify to run at all. I don't really need to be notified of out
of date pages if I re-index the entire site every couple of days.
So I have two questions - first, are there any reasons I'm overlooking
to use htnotify, and if there are then two, has anyone had this problem
on Linux and found that the Solaris patch fixed it? I'm not sure
exactly what version of Linux is on this server; I know it's RedHat
and I think it's ES 3 but I'm not positive.