[htdig-dev] Cygwin words.db Compression

[Steve E. <Ste...@ch...>]

I'm compiling htdig-3.2.0b4-20090928 under Cygwin 1.5.5 using gcc 3.3.1, on both Windows XP Pro SP1 and Windows 2000 Server SP4. Compiling and installation is not a problem. But db.words.db is always a zero length file after running htdig with the compression flags at their default values. After some profiling I also noticed that it wasn't creating the "db.words.db.work_weakcmpr" file during the dig. When compiled under Cygwin 1.3.22 using "gcc-3.2 20020927 (prerelease)", the work file *is* created during the dig and db.words.db has size to it afterwards. However, I am not able to htdb_dump that file or use htsearch against it. It's corrupt or something. The other db files seem to get created fine under both sets of binaries, although I didn't try to dump them. And the same version related behavior occurs under both XP and 2000 OS's.

After reading all the SF posts about compression and db issues, I decided to disable compression and see what happens:

wordlist_compress: false
wordlist_compress_zlib: false
compression_level: 0

With those settings, everything appears to work fine for both sets of binaries: I can dig pages and run htsearch. I haven't modified any of the code to try and address the problem yet, but it looks like others are having similar issues on other platforms? Is anybody else having trouble with db compression on Windows? I have tried different settings for compression_level with no success.

Also, my initial attempts at changing the compression flag values failed with error messages from htdig while trying to read the configuration file. It seems that the htdig.conf parser doesn't like CR (ASCII=13) characters. Notepad and Wordpad are obvious choices for editing this file on Windows, but those don't work because both insert CRLF pairs to terminate lines in the file (e.g. DOS format). And then the parser apparently won't see flags at the bottom of the CRLF file. The solution was a simple JavaScript program to modify htdig.conf by removing all CR characters *before* running htdig. Is anybody else seeing this on Cygwin builds?

Sorry for the long post :)

PS - I'm running 3.1.6 in production on Windows at http://www.childrenshc.org/Search/ and it rocks!!

Thanx
-Steve
__________________________________

Confidentiality Statement:
This email/fax, including attachments, may include confidential and/or proprietary information and may be used only by the person or entity to which it is addressed. If the reader of this email/fax is not the intended recipient or his or her agent, the reader is hereby notified that any dissemination, distribution or copying of this email/fax is prohibited. If you have received this email/fax in error, please notify the sender by replying to this message and deleting this email or destroying this facsimile immediately.