[htdig-dev] Report on Native Win32 & memory leak fixes

[Neal R. <ne...@ri...>]

Hey,
	In looking at the amount of changes to the tree to be checked in
for the Native Win32 port I decided to NOT check in incrementally through
the week and do it all at once.

	I wrote up a report down below:

	There is pretty minimal impact for Unix code.  The question is do
you want me to sit on this or go ahead and check it in?

	I also looked over my fixes for memory leaks, these are separate
from the native Win32 changes, and will be committing those ASAP..
 early next week.

	I will also commit the latest version of libhtdig & libhtdigphp.
These have independent makefiles from the rest of htdig and won't affect
the new beta.

	I also have an efficiency improvement to WordDB.cc that uses the
STL.  This is probably inappropriate for the upcoming beta, but I'd like
to add it soon.

	After I get this stuff added and I'll get back to thinking about
cool new features like Porter Stemming in the worddb, improvements to
worddb storage format and researchy AI stuff.

Thanks!

Neal Richter

REPORT:

	There are 22 new files (10 of them WIN32 Makefiles), and nearly
1100 lines changed in the code base.  Not much considering the size of the
code base.

	I have made every effort to make all changes within #ifdef
_MSC_VER and these changes should largely not effect the main body of code.

	Here are the exceptions:

db/mp_alloc.c:

I added two new functions CDB_get_mp_dirty_level & CDB_set_mp_dirty_level
and made the CDB___mp_dirty_level static to db/mp_alloc.c.  This is
slightly cleaner that a true global variable and prevented some
dllexporting win32 crap.

htcommon/conf_lexer.lxx:
bcopy() -> memcopy()

htcommon/defaults.cc:
I used "/quotes to divide up some of the very long strings to prevent
MSVC++ from barfing on long lines.  This won't affect gcc/g++ since

"Hello"
"World"

is equivalent to

"Hello \
World"

in C.


htdig/Document.h:

added a one line ContentType()    {return contentType.get();}
for use with libhtdig

htdig/Retriever.cc:

Various changes to IsValidURL() to return an actual error code rather then
FALSE for every error.

More usage of the 'urls_seen' file.  There were a few cases where a URL
was not put into this file.

htfuzzy/EndingsDB.cc & htfuzzy/Synonym.cc:

I am using a new file_copy routine which is very well tested and works in
Unix and Win32 rather than a 'system' call to copy files.

This code is #ifdefed for use ONLY with  Native Win32 &
libhtdig/libhtdigphp usage and uses the old system("MV xx yy") for all
others.

I recommend at some point we do away with the 'system' call.

/htlib/HtDateTime.cc:

global var _strtime to my_strtime to make MSVC++ happy.

There are a few new things added for Win32 which may prove useful later:

local version of GNU regex (LGPL) for win32
Some POSIX-like directory routines for <dirent.h> for win32 (written by RightNow)
local version of getopt  (public domain version)

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485