The ht://Dig system is a complete indexing and searching system for a domain or intranet. This system is not meant to replace the need for powerful internet-wide search systems like Lycos, Infoseek, Google and AltaVista.
While htDig was great 10 years ago, the project now seems to be abandoned :-( There has been no new official release for 6 years, no progress on the outstanding bugs and no solution for critical missing features like full unicode support and building indexes larger than 2GB. There is also no help available via the (now very quiet) official mailing list. As a result the code base has not keep up with new operating system releases and modern development environments. So it is now very hard to compile and use. If you need site search, possible replacements for htDig are, Nutch / Lucene is a good candidate, but requires a Java environment (apache/tomcat server). Sphider is OK if you are alright with setting up SQL as a backend and limited support. Otherwise there is also reasonably priced commercial packages, like the Zoom Search Engine (which doesn't need Java, nor SQL).