Hi all,


A couple of years ago we’ve implemented a website for one of our clients. The website serves as an information portal for regular visitors seeking information, but it has also workgroups for specialists which share information about different subjects. When we created the website we used htdig as a search engine to spider the public information as well as the information in the workgroups. Workgroups contain news and agenda, forums, webmail and documents.


The problem is that since a few months the load on the server has increased. During indexing the serverload gets up to and above 3.00 – 4.00 average. The duration of the indexing: up to 12 hours! Now I know there is a lot of information on the website plus 600+ workgroups to index, so I’d expect htdig to consume some time. In the end we end up with the usual htdig files: docdb = 206 MB, docs.index = 11 MB, wordlist = 222 MB and words db = 169 MB. Comparing these filesizes with the filesizes of an other (and a lot smaller) website index: not much different. Indexing the smaller website gives us a docdb file of 195 MB, a wordlist file of 195 MB and a words db file of 140 MB. The indexing takes about 2 hours from start to finish.


I’ve tried to set up htdig to index incremental but that doesn’t work well. After a while htdig stops indexing with an error: the work files get past the 2GB boundary. Cleaning up the workfiles before the indexing works, but then it’s no longer an incremental search, is it? :-) Besides the indexing still take about 5-6 hours when doing an incremental search.


Can anyone help me with this problem? Or does anyone have an idea what the problem might be? I’ve added serverinformation at the bottom of the e-mail followed by the configuration file.


Thanks in advance!




Marco Houtman

Ecommany B.V.




PS: Serverinformation


SuSE Linux (I believe it’s 8.2, but our service provider installed it for us and I do not have enough privileges to login to the server and look up the exact version)

Htdig 3.1.6 has been installed from source (I can see the sourcefile an there’s no RPM with the name htdig installed afaik).

Server hardware is about 3 years old now. I can’t tell you exactly what the components are but I know it was state of the art equipment back then :-)


PS2: configuration


# common

root_dir:                        /home/www/domain.nl/htdig


common_dir:                 ${root_dir}/common

database_dir:                 ${root_dir}/db

template_dir:                 ${root_dir}/templates


# htdig

bad_extensions:            .wav .gz .z .sit .au .zip .tar .hqx .exe .com .gif \

.jpg .jpeg .aiff .class .map .ram .tgz .bin .rpm \

.mpg .mov .avi .css .js .inc

bad_word_list:               ${common_dir}/bad_words

create_url_list:               yes

exclude_urls:                 /bestel/ /zoek/ /download/ /pdf/ /uploaded/ \

selectie= regio= type= letter= auteur= rubriek= start= \

tumor= uitgever= sort= regtoev= meth= status= fase= \

aktie= thesaurus

external_parsers:           application/msword /usr/local/bin/parse_doc.pl \

application/pdf /usr/local/bin/parse_pdf.pl

limit_urls_to:                  http://www.domain.nl/

maintainer:                    mye-mail@domain.nl

max_doc_size:              5000000

max_head_length:          10000

max_hop_count:            10

start_url:                       http://www.domain.nl/index.php

user_agent:                   domain-digger


# htmerge


# htdump


# htload


# htfuzzy

endings_affix_file:           ${common_dir}/nederlands.aff

endings_dictionary:        ${common_dir}/nederlands.0


# htnotify


# htsearch

max_prefix_matches:     100

minimum_prefix_length:  2

no_excerpt_show_top:    true

nothing_found_file:         ${template_dir}/nomatch.html

prefix_match_character: *

search_algorithm:          exact:1 prefix:0.5 endings:0.1

search_results_footer:    ${template_dir}/footer.html

search_results_header:  ${template_dir}/header.html

syntax_error_file:           ${template_dir}/syntax.html

sort:                              score

template_map:               Long builtin-long builtin-long \

Short builtin-short builtin-short \

Website website ${template_dir}/website.html

template_name:             website