htdig-dev — Developer Discussion for the ht://Dig project

[htdig-dev] Re: extra semicolon on endif

[Lachlan A. <lh...@us...>]

Greetings Andy,

On Fri, 20 Jun 2003 07:46, And...@wi... wrote:
> I'm seeing a complaint about extra junk after an "endif" in
> htnet/SSLConnection.h line 48:
> #endif;

Thanks, I'll fix that (once sourceforge lets me commit...)

> Interesting file sizes (indexing the same site)
> Sol 2.8
> 45027328      db.words.db
>
> Sol 2.7
> 222810112   db.words.db
>
> Redhat
> 224599040   db.words.db

Interesting, and worrying!  If you search for  *  does it give you the=20
same results in each case?

> all three show HAVE_LIBZ AND HAVE_ZLIB_H defined (confusing switch
> there).

Yes, the problem is that autoconf automatically sets these flags.  If=20
the authors of ZLib called their header file "libz" then it would=20
make the switches more logical.

Cheers,
Lachlan

--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

[htdig-dev] Re: htdig-3.2.0b4-20030615 on Sol 2.8

[Lachlan A. <lh...@us...>]

Greetings,

Thank you for pointing out that bug.

It is in lines  10283  and  10288  of configure.  Replace the *four*=20
occurrences of  CFLAGS  with  CPPFLAGS.

Cheers,
Lachlan

On Fri, 20 Jun 2003 03:18, And...@wi... wrote:
> Got it to compile reasonably cleanly and work for non-https
> connections on x86 Solaris 2.8, gcc 2.95.3.  It still failed to set
> HAVE_SSL_H and so didn't do the ssl
> (despite the --with-ssl=3D/usr/local/ssl configure flag) though it
> did add the:
> LD_FLAG =3D -L/usr/local/ssl/lib

--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

[htdig-dev] Re: Re: b5 make fails on Solaris with Sun c++

[Lachlan A. <lh...@us...>]

Greetings Neil,

I should have read the error message more carefully.  Try the attached=20
patch (again, applied to the clean package).

Thanks yet again for this,
Lachlan

On Fri, 20 Jun 2003 02:27, Neil Kohl wrote:
> No complaints about ResultMatch. Still getting 2 errors in
> search.cc:
>
> "search.cc", line 2301: Error: An integer constant expression is
> required within the array subscript operator.
>
> Almost there...

--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

Re: [htdig-dev] 3.2.0b5 Next progress check :)

[Lachlan A. <lh...@us...>]

Greetings Jesse,

I don't think Geoff was saying we *shouldn't* use 2.7.
It's just that we haven't actually *tested* it under 2.7.
If (when :) you can confirm that it still works under 2.8,
that's great!

Cheers,
Lachlan

On Fri, 20 Jun 2003 00:45, J. op den Brouw wrote:
> Hmmm, i'm still compiling htdig with 2.8.1 since my
> admin will not update for now (too much work...)
>
> Any reason why compiling under 2.7 is discouraged?

--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

[htdig-dev] Report on Native Win32 & memory leak fixes

[Neal R. <ne...@ri...>]

Hey,
	In looking at the amount of changes to the tree to be checked in
for the Native Win32 port I decided to NOT check in incrementally through
the week and do it all at once.

	I wrote up a report down below:

	There is pretty minimal impact for Unix code.  The question is do
you want me to sit on this or go ahead and check it in?

	I also looked over my fixes for memory leaks, these are separate
from the native Win32 changes, and will be committing those ASAP..
 early next week.

	I will also commit the latest version of libhtdig & libhtdigphp.
These have independent makefiles from the rest of htdig and won't affect
the new beta.

	I also have an efficiency improvement to WordDB.cc that uses the
STL.  This is probably inappropriate for the upcoming beta, but I'd like
to add it soon.

	After I get this stuff added and I'll get back to thinking about
cool new features like Porter Stemming in the worddb, improvements to
worddb storage format and researchy AI stuff.

Thanks!

Neal Richter

REPORT:

	There are 22 new files (10 of them WIN32 Makefiles), and nearly
1100 lines changed in the code base.  Not much considering the size of the
code base.

	I have made every effort to make all changes within #ifdef
_MSC_VER and these changes should largely not effect the main body of code.

	Here are the exceptions:

db/mp_alloc.c:

I added two new functions CDB_get_mp_dirty_level & CDB_set_mp_dirty_level
and made the CDB___mp_dirty_level static to db/mp_alloc.c.  This is
slightly cleaner that a true global variable and prevented some
dllexporting win32 crap.

htcommon/conf_lexer.lxx:
bcopy() -> memcopy()

htcommon/defaults.cc:
I used "/quotes to divide up some of the very long strings to prevent
MSVC++ from barfing on long lines.  This won't affect gcc/g++ since

"Hello"
"World"

is equivalent to

"Hello \
World"

in C.


htdig/Document.h:

added a one line ContentType()    {return contentType.get();}
for use with libhtdig

htdig/Retriever.cc:

Various changes to IsValidURL() to return an actual error code rather then
FALSE for every error.

More usage of the 'urls_seen' file.  There were a few cases where a URL
was not put into this file.

htfuzzy/EndingsDB.cc & htfuzzy/Synonym.cc:

I am using a new file_copy routine which is very well tested and works in
Unix and Win32 rather than a 'system' call to copy files.

This code is #ifdefed for use ONLY with  Native Win32 &
libhtdig/libhtdigphp usage and uses the old system("MV xx yy") for all
others.

I recommend at some point we do away with the 'system' call.

/htlib/HtDateTime.cc:

global var _strtime to my_strtime to make MSVC++ happy.

There are a few new things added for Win32 which may prove useful later:

local version of GNU regex (LGPL) for win32
Some POSIX-like directory routines for <dirent.h> for win32 (written by RightNow)
local version of getopt  (public domain version)

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

[htdig-dev] extra semicolon on endif

[<And...@wi...>]

I'm seeing a complaint about extra junk after an "endif" in 
htnet/SSLConnection.h line 48:
#endif;

I also compiled the 3.2.0b4-20030615 snapshot on x86 solaris 2.7/gcc 
2.95.3 and redhat enterprise 1.1, gcc 3.1 (both have the same 
HAVE_SSL_H/include troubles).

Interesting file sizes (indexing the same site)
Sol 2.8
344064          db.docdb
180224          db.docs.index
1138688        db.excerpts
45027328      db.words.db
16384             db.words.db_weakcmpr

Sol 2.7
344064          db.docdb
172032          db.docs.index
3694592        db.excerpts
222810112   db.words.db
16384            db.words.db_weakcmpr


Redhat
327680         db.docdb
172032         db.docs.index
3637248       db.excerpts
224599040   db.words.db
16384             db.words.db_weakcmpr

all three show HAVE_LIBZ AND HAVE_ZLIB_H defined (confusing switch there).

a

Andy Bach, Sys. Mangler
Internet: and...@wi... 
VOICE: (608) 261-5738  FAX 264-5030

"We are either doing something, or we are not. 'Talking about' is a 
subset of 'not'."            -- Mike Sphar in alt.sysadmin.recovery

[htdig-dev] htdig-3.2.0b4-20030615 on Sol 2.8

[<And...@wi...>]

Got it to compile reasonably cleanly and work for non-https connections on 
x86 Solaris 2.8, gcc 2.95.3.  It still failed to set HAVE_SSL_H and so 
didn't do the ssl 
(despite the --with-ssl=/usr/local/ssl configure flag) though it did add 
the:
LD_FLAG = -L/usr/local/ssl/lib

however, changing the include/htconfig.h to define HAVE_SSL_H failed until 
I added:
LOCAL_DEFINES = -I /usr/local/ssl

Looking through the code, m4 files etc. it appears there's a 
HAVE_SSL/HAVE_SSL_H disconnect, for one. Configure set CFLAGS to have 
-DHAVE_SSL but, some of this is .cc files so ...

I'm over my head as far as the make/m4 stuff but I note that only in htlib 
Makefile does the CFLAGS show -DHAVE_SSL while the 2 dirs that need 
HAVE_SSL_H are htdig (Document.cc) and htnet (HtHTTPSecure.cc and .h).

I tried adding
LOCAL_DEFINES="$LOCAL_DEFINES -I$ssldir/include/openssl -DHAVE_SSL_H";
and:
LOCAL_DEFINES="$LOCAL_DEFINES -I$ssldir/include -DHAVE_SSL_H";

first to aclocal.m4 and then to acinclude.m4 in the ssl section, to no 
effect.  I then added AC_DEFINE(HAVE_SSL_H), no effect.  I believe this is 
the right idea, I'm just not able to get it to 'take'

a

Andy Bach, Sys. Mangler
Internet: and...@wi... 
VOICE: (608) 261-5738  FAX 264-5030

"We are either doing something, or we are not. 'Talking about' is a 
subset of 'not'."            -- Mike Sphar in alt.sysadmin.recovery

[htdig-dev] Re: b5 make fails on Solaris with Sun c++

[Neil K. <nk...@ma...>]

No complaints about ResultMatch. Still getting 2 errors in search.cc:

"search.cc", line 2301: Error: An integer constant expression is required =
within the array subscript operator.
"search.cc", line 2331: Error: An integer constant expression is required =
within the array subscript operator.

Almost there...


Neil Kohl
Manager, ACP Online             =20
American College of Physicians
nk...@ac...              215.351.2638, 800.523.1546 x2638


>>> Lachlan Andrew <lh...@us...> 06/19/03 09:51AM >>>
Thanks for your continued help on this.

The attached patch (which contains the previous one) should fix those=20
errors.  I'm not sure about the one in ResultMatch.cc.  If it still=20
complains, could you please try putting
  ResultMatch::
at the start of line 231?

Once again, your help is greatly appreciated!
Lachlan

On Thu, 19 Jun 2003 00:54, Neil Kohl wrote:
> Lachlan,
>
> I applied the patch but it still won't compile. Following are the
> Error: lines from output of 'make -k check'. I have the entire make
> output saved if that would be helpful.
>
> "HtFTP.cc", line 175: Error: Cannot use const char* to initialize
> char*. "htdb_stat.cc", line 83: Error: Multiple declaration for
> internal. "htdb_stat.cc", line 108: Error: The operand "*internal"
> cannot be assigned to. "htdb_stat.cc", line 129: Error: The operand
> "*internal" cannot be assigned to. "htdb_stat.cc", line 548: Error:
> Formal argument 2 of type char* in call to
> CDB___lock_dump_region(__db_env*, char*, __FILE*) is being passed
> std::ios_base&(*)(std::ios_base&). "htdb_stat.cc", line 635: Error:
> Formal argument 2 of type char* in call to
> CDB___memp_dump_region(__db_env*, char*, __FILE*) is being passed
> std::ios_base&(*)(std::ios_base&). "ResultMatch.cc", line 231:
> Error: ResultMatch::SortType is not accessible from struct.
> "search.cc", line 2301: Error: An integer constant expression is
> required within the array subscript operator. "search.cc", line
> 2331: Error: An integer constant expression is required within the
> array subscript operator.
>
>
> Neil Kohl
> Manager, ACP Online
> American College of Physicians
> nk...@ac...              215.351.2638, 800.523.1546 x2638
>
> >>> Lachlan Andrew <lh...@us...> 06/18/03 08:17AM >>>
>
> Thanks for that, Neil.
>
> That problem should be fixed in the attached patch.  Could you try
> "make -k" so that it will report as many errors at once as
> possible?
>
> Thanks again,
> Lachlan
>
> On Wed, 18 Jun 2003 02:15, Neil Kohl wrote:
> > The make of b5 on Solaris 9 with Sun c++ failed. Gory details
> > follow. I've got the complete configure and make output saved if
> > you need it.

--=20
lh...@us...=20
ht://Dig developer DownUnder  (http://www.htdig.org)

Re: [htdig-dev] 3.2.0b5 Next progress check :)

[J. op d. B. <ht...@op...>]

Hmmm, i'm still compiling htdig with 2.8.1 since my
admin will not update for now (too much work...)

Any reason why compiling under 2.7 is discouraged?

Lachlan Andrew wrote:
> 
> Thanks for the tip about the address -- I was forgetting that these
> pages are intended for the BBB (Big Bad WeB).
> 
> We don't claim that we compile under gcc-2.7.  The old systems are
> commented out.  (Just my fear of deleting...)

--jesse

[htdig-dev] Re: HAVE_ZLIB vs HAVE_LIBZ bug?

[Lachlan A. <lh...@us...>]

Greetings,

Well spotted.  It shouldn't matter, though.  The comments talk about=20
"default" page size, so it could well be overridden anyway...  Notice=20
that it is set to at most 16k a few lines above, so it isn't likely=20
to cause a 64k limit.

Cheers,
Lachlan

On Thu, 19 Jun 2003 04:00, Neal Richter wrote:
> Hey,
> =09I think I have found something here.. no idea of importance yet.
>
> In db/db.c line 1049 we have an #ifdef:
>
>     /*
>      * If compression is on, the minimum page size must be
> multiplied * by the compression factor.
>      */
> #ifdef HAVE_ZLIB
>     if(F_ISSET(dbp, DB_AM_CMPR)) {
>       if(iopsize < DB_CMPR_MULTIPLY(dbenv, DB_MIN_PGSIZE))
>         iopsize =3D DB_CMPR_MULTIPLY(dbenv, DB_MIN_PGSIZE);
>     }
> #endif /* HAVE_ZLIB */
>
>
> This was added to db.c 3 years ago when Geoff merged Loic's code
> from milfuz.
>
> The problem is that it needs to be HAVE_LIBZ rather than HAVE_ZLIB!
>  Our db_config.h contains HAVE_LIBZ & HAVE_ZLIB_H .. no HAVE_ZLIB.
>
> I am not sure yet how this affects us yet... it looks like a range
> check and may be part of why db-compression doesn't work when I try
> and set the wordlist_page_size to 64K.
>
> I also checked and this is the only occurance of this screwup.
>
> Obviously it must not be too important since we do have functional
> libz-based DB compression.
>
> Thanks.
>
> Neal Richter
> Knowledgebase Developer
> RightNow Technologies, Inc.
> Customer Service for Every Web Site
> Office: 406-522-1485

--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

Re: [htdig-dev] 3.2.0b5 Next progress check :)

[Lachlan A. <lh...@us...>]

Thanks for the tip about the address -- I was forgetting that these=20
pages are intended for the BBB (Big Bad WeB).

We don't claim that we compile under gcc-2.7.  The old systems are=20
commented out.  (Just my fear of deleting...)

Cheers,
Lachlan

On Thu, 19 Jun 2003 15:30, Geoff Hutchison wrote:
> On Wednesday, June 18, 2003, at 07:43  AM, Lachlan Andrew wrote:
> > The supported platforms are currently in htdoc/requirements.html.
> > Is there really a need for a new document?
>
> No, but they do need updating, even with your recent changes. For
> example, we can't really be sure we compile with gcc-2.7 anymore,
> right?
>
> (BTW, I wouldn't suggest putting an e-mail address in the page even
> as a comment. SPAMbots seem to harvest these too.)
>
> -Geoff

--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

[htdig-dev] Re: b5 make fails on Solaris with Sun c++

[Lachlan A. <lh...@us...>]

Thanks for your continued help on this.

The attached patch (which contains the previous one) should fix those=20
errors.  I'm not sure about the one in ResultMatch.cc.  If it still=20
complains, could you please try putting
  ResultMatch::
at the start of line 231?

Once again, your help is greatly appreciated!
Lachlan

On Thu, 19 Jun 2003 00:54, Neil Kohl wrote:
> Lachlan,
>
> I applied the patch but it still won't compile. Following are the
> Error: lines from output of 'make -k check'. I have the entire make
> output saved if that would be helpful.
>
> "HtFTP.cc", line 175: Error: Cannot use const char* to initialize
> char*. "htdb_stat.cc", line 83: Error: Multiple declaration for
> internal. "htdb_stat.cc", line 108: Error: The operand "*internal"
> cannot be assigned to. "htdb_stat.cc", line 129: Error: The operand
> "*internal" cannot be assigned to. "htdb_stat.cc", line 548: Error:
> Formal argument 2 of type char* in call to
> CDB___lock_dump_region(__db_env*, char*, __FILE*) is being passed
> std::ios_base&(*)(std::ios_base&). "htdb_stat.cc", line 635: Error:
> Formal argument 2 of type char* in call to
> CDB___memp_dump_region(__db_env*, char*, __FILE*) is being passed
> std::ios_base&(*)(std::ios_base&). "ResultMatch.cc", line 231:
> Error: ResultMatch::SortType is not accessible from struct.
> "search.cc", line 2301: Error: An integer constant expression is
> required within the array subscript operator. "search.cc", line
> 2331: Error: An integer constant expression is required within the
> array subscript operator.
>
>
> Neil Kohl
> Manager, ACP Online
> American College of Physicians
> nk...@ac...              215.351.2638, 800.523.1546 x2638
>
> >>> Lachlan Andrew <lh...@us...> 06/18/03 08:17AM >>>
>
> Thanks for that, Neil.
>
> That problem should be fixed in the attached patch.  Could you try
> "make -k" so that it will report as many errors at once as
> possible?
>
> Thanks again,
> Lachlan
>
> On Wed, 18 Jun 2003 02:15, Neil Kohl wrote:
> > The make of b5 on Solaris 9 with Sun c++ failed. Gory details
> > follow. I've got the complete configure and make output saved if
> > you need it.

--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

[htdig-dev] Re: select() problem on UX 10.20

[Lachlan A. <lh...@us...>]

Good work.  It looks like you're making good progress.  I'll patch the=20
configure script soon.  It will be good to see the error messages=20
from  make.

The reason the compiler complains the first time is that  select  is=20
declared in  limits.h,  which  2.57  includes, but  2.13  doesn't. =20
(The C program is so simple that it doesn't matter what type  select=20
is declared as, as long as it hasn't been declared with a conflicting=20
type in one of the headers.  The aim is to make the program not=20
depend on any headers which may be absent, but in this case it=20
backfires...)

Cheers,
Lachlan

On Thu, 19 Jun 2003 19:08, J. op den Brouw wrote:

> I'm trying to locate/fix the select() problem on HP-UX 10.20

> | /* System header to define __stub macros and hopefully few
> | prototypes, which can conflict with char select (); below.
> |     Prefer <limits.h> to <assert.h> if __STDC__ is defined, since
> |     <limits.h> exists even on freestanding compilers.  */
> | #ifdef __STDC__
> | # include <limits.h>
> | #else
> | # include <assert.h>
> | #endif

> So, autoconf 2.57 generated code breaks, autoconf 2.13 generated
> code is okay... Baffled me is....
>
> Maybe it's just my system.....
>
> Oh, after that, make breaks on some htdig file...
> more on that later...
--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

[htdig-dev] select() problem on UX 10.20

[J. op d. B. <ht...@op...>]

Hi all,

I'm trying to locate/fix the select() problem
on HP-UX 10.20

Sysinfo
-------
uname -a
   HP-UX chaos B.10.20 U 9000/800 2004964319 unlimited-user license
gcc -v
   Reading specs from
/opt/gcc/lib/gcc-lib/hppa1.1-hp-hpux10.20/2.8.1/specs
   gcc version 2.8.1
make -v
   GNU Make version 3.78, by Richard Stallman and Roland McGrath.
htdig
   htdig-3.2.0b4-20030601

Problem
-------
When running ./configure with numerous option in db directory of htdig,
./configure stops complaining about select() not found:

  ...
  checking for qsort... yes
  checking for select... no
  configure: error: No select library function.
  configure: error: /bin/sh './configure' failed for db
  [msql@chaos htdig-3.2.0b4-20030601]$

After some digging(!), it came clear that the gcc compiler
finds two prototypes for select(), one in an include
file and the other, as stated by the config.log file
(I added the -v flag on gcc):

...
configure:12688: checking for select
configure:12741: gcc -v -v -o conftest -g -O2 -Wall -D_REENTRANT 
-I/opt/htdig/lib/zlib/include -L/opt/htdig/lib/zlib/lib conftest.c -lz 
-w >&5
Reading specs from /opt/gcc/lib/gcc-lib/hppa1.1-hp-hpux10.20/2.8.1/specs
gcc version 2.8.1
 /opt/gcc/lib/gcc-lib/hppa1.1-hp-hpux10.20/2.8.1/cpp -lang-c -v
-I/opt/htdig/lib/zlib/include -undef -D__GNUC__=2 -D__GNUC_MIN
OR__=8 -Dhppa -Dhp9000s800 -D__hp9000s800 -Dhp9k8 -DPWB -Dhpux -Dunix
-D__hppa__ -D__hp9000s800__ -D__hp9000s800 -D__hp9k8__ -
D__PWB__ -D__hpux__ -D__unix__ -D__hppa -D__hp9000s800 -D__hp9k8 -D__PWB
-D__hpux -D__unix -Asystem(unix) -Asystem(hpux) -Acpu
(hppa) -Amachine(hppa) -D__OPTIMIZE__ -g -Wall -w -D__hp9000s700
-D_PA_RISC1_1 -D_HPUX_SOURCE -D_HIUX_SOURCE -D_REENTRANT conf
test.c /var/tmp/cca05375.i
GNU CPP version 2.8.1 (hppa)
#include "..." search starts here:
#include <...> search starts here:
 /opt/htdig/lib/zlib/include
 /usr/local/include
 /opt/gcc/hppa1.1-hp-hpux10.20/include
 /opt/gcc/lib/gcc-lib/hppa1.1-hp-hpux10.20/2.8.1/include
 /usr/include
End of search list.
 /opt/gcc/lib/gcc-lib/hppa1.1-hp-hpux10.20/2.8.1/cc1 /var/tmp/cca05375.i
-quiet -dumpbase conftest.c -g -O2 -Wall -w -version 
-o /var/tmp/cca05375.s
GNU C version 2.8.1 (hppa1.1-hp-hpux10.20) compiled by CC.
configure:12743: conflicting types for `select'
/usr/include/sys/time.h:512: previous declaration of `select'
configure:12744: $? = 1
configure: failed program was:
| #line 12693 "configure"
| /* confdefs.h.  */
| 
| #define PACKAGE_NAME ""
| #define PACKAGE_TARNAME ""
| #define PACKAGE_VERSION ""
| #define PACKAGE_STRING ""
| #define PACKAGE_BUGREPORT ""
| #define STDC_HEADERS 1
| #define HAVE_SYS_TYPES_H 1
| #define HAVE_SYS_STAT_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_STRING_H 1
| #define HAVE_MEMORY_H 1
| #define HAVE_STRINGS_H 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_UNISTD_H 1
| #define HAVE_LIBZ 1
| #define WORDS_BIGENDIAN 1
| #define TIME_WITH_SYS_TIME 1
| #define HAVE_STRUCT_STAT_ST_BLKSIZE 1
| #define HAVE_ST_BLKSIZE 1
| #define HAVE_DIRENT_H 1
| #define HAVE_SYS_TIME_H 1
| #define HAVE_MUTEX_THREADS 1
| #define HAVE_MUTEX_TAS 1
| #define HAVE_MUTEX_HPPA_MSEM_INIT 1
| #define HAVE_GETUID 1
| #define HAVE_PSTAT_GETDYNAMIC 1
| #define HAVE_SYSCONF 1
| #define HAVE_SCHED_YIELD 1
| #define HAVE_SIGFILLSET 1
| #define HAVE_FCNTL_F_SETFD 1
| #define HAVE_QSORT 1
| /* end confdefs.h.  */
| /* System header to define __stub macros and hopefully few prototypes,
|     which can conflict with char select (); below.
|     Prefer <limits.h> to <assert.h> if __STDC__ is defined, since
|     <limits.h> exists even on freestanding compilers.  */
| #ifdef __STDC__
| # include <limits.h>
| #else
| # include <assert.h>
| #endif
| /* Override any gcc2 internal prototype to avoid an error.  */
| #ifdef __cplusplus
| extern "C"
| {
| #endif
| /* We use char because int might match the return type of a gcc2
|    builtin and then its argument prototype would still apply.  */
| char select ();
| /* The GNU C library defines this for functions which it implements
|     to always fail with ENOSYS.  Some functions are actually named
|     something starting with __ and the normal name is an alias.  */
| #if defined (__stub_select) || defined (__stub___select)
| choke me
| #else
| char (*f) () = select;
| #endif
| #ifdef __cplusplus
| }
| #endif
| 
| int
| main ()
| {
| return f != select;
|   ;
|   return 0;
| }

It seems that this C-source is defining select() again as char but the
file
Original failing C-source in configure (by autoconf 2.57) but the file
/usr/include/sys/time.h defines it as:
extern int select(int, fd_set *, fd_set *, fd_set *, struct timeval *);

Okay, I pulled the configure.in through autoconf 2.13, cut out only
the select() routines and pasted it over the 2.57 select() routine.

Now the C-source generated bij autoconf 2.13 is (below) and
here select is also prototyped (again) as char but this
time the compiler is not complaining....

| #line 2048 "configure"
| #include "confdefs.h"
| /* System header to define __stub macros and hopefully few prototypes,
|     which can conflict with char select(); below.  */
| #include <assert.h>
| /* Override any gcc2 internal prototype to avoid an error.  */
| /* We use char because int might match the return type of a gcc2
|     builtin and then its argument prototype would still apply.  */
| char select();
| 
| int main() {
| 
| /* The GNU C library defines this for functions which it implements
|     to always fail with ENOSYS.  Some functions are actually named
|     something starting with __ and the normal name is an alias.  */
| #if defined (__stub_select) || defined (__stub___select)
| choke me
| #else
| select();
| #endif
| 
| ; return 0; }

So, autoconf 2.57 generated code breaks, autoconf 2.13 generated code
is okay... Baffled me is....

Maybe it's just my system.....

Oh, after that, make breaks on some htdig file...
more on that later...

Re: [htdig-dev] 3.2.0b5 Next progress check :)

[Geoff H. <ghu...@ws...>]

On Wednesday, June 18, 2003, at 07:43  AM, Lachlan Andrew wrote:

> The supported platforms are currently in htdoc/requirements.html.
> Is there really a need for a new document?

No, but they do need updating, even with your recent changes. For 
example, we can't really be sure we compile with gcc-2.7 anymore, right?

(BTW, I wouldn't suggest putting an e-mail address in the page even as 
a comment. SPAMbots seem to harvest these too.)

-Geoff

[htdig-dev] HAVE_ZLIB vs HAVE_LIBZ bug?

[Neal R. <ne...@ri...>]

Hey,
	I think I have found something here.. no idea of importance yet.

In db/db.c line 1049 we have an #ifdef:

    /*
     * If compression is on, the minimum page size must be multiplied
     * by the compression factor.
     */
#ifdef HAVE_ZLIB
    if(F_ISSET(dbp, DB_AM_CMPR)) {
      if(iopsize < DB_CMPR_MULTIPLY(dbenv, DB_MIN_PGSIZE))
        iopsize = DB_CMPR_MULTIPLY(dbenv, DB_MIN_PGSIZE);
    }
#endif /* HAVE_ZLIB */


This was added to db.c 3 years ago when Geoff merged Loic's code from
milfuz.

The problem is that it needs to be HAVE_LIBZ rather than HAVE_ZLIB!  Our
db_config.h contains HAVE_LIBZ & HAVE_ZLIB_H .. no HAVE_ZLIB.

I am not sure yet how this affects us yet... it looks like a range check
and may be part of why db-compression doesn't work when I try and set the
wordlist_page_size to 64K.

I also checked and this is the only occurance of this screwup.

Obviously it must not be too important since we do have functional
libz-based DB compression.

Thanks.

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

[htdig-dev] Re: b5 make fails on Solaris with Sun c++

[Neil K. <nk...@ma...>]

Lachlan,

I applied the patch but it still won't compile. Following are the Error: =
lines from output of 'make -k check'. I have the entire make output saved =
if that would be helpful.

"HtFTP.cc", line 175: Error: Cannot use const char* to initialize char*.
"htdb_stat.cc", line 83: Error: Multiple declaration for internal.
"htdb_stat.cc", line 108: Error: The operand "*internal" cannot be =
assigned to.
"htdb_stat.cc", line 129: Error: The operand "*internal" cannot be =
assigned to.
"htdb_stat.cc", line 548: Error: Formal argument 2 of type char* in call =
to CDB___lock_dump_region(__db_env*, char*, __FILE*) is being passed =
std::ios_base&(*)(std::ios_base&).
"htdb_stat.cc", line 635: Error: Formal argument 2 of type char* in call =
to CDB___memp_dump_region(__db_env*, char*, __FILE*) is being passed =
std::ios_base&(*)(std::ios_base&).
"ResultMatch.cc", line 231: Error: ResultMatch::SortType is not accessible =
from struct.
"search.cc", line 2301: Error: An integer constant expression is required =
within the array subscript operator.
"search.cc", line 2331: Error: An integer constant expression is required =
within the array subscript operator.


Neil Kohl
Manager, ACP Online             =20
American College of Physicians
nk...@ac...              215.351.2638, 800.523.1546 x2638


>>> Lachlan Andrew <lh...@us...> 06/18/03 08:17AM >>>
Thanks for that, Neil.

That problem should be fixed in the attached patch.  Could you try=20
"make -k" so that it will report as many errors at once as possible?

Thanks again,
Lachlan

On Wed, 18 Jun 2003 02:15, Neil Kohl wrote:
> The make of b5 on Solaris 9 with Sun c++ failed. Gory details
> follow. I've got the complete configure and make output saved if
> you need it.

--=20
lh...@us...=20
ht://Dig developer DownUnder  (http://www.htdig.org)

Re: [htdig-dev] 3.2.0b5 Next progress check :)

[Lachlan A. <lh...@us...>]

On Wed, 18 Jun 2003 05:32, J. op den Brouw wrote:
> I'm still having trouble getting the Berkeley database ./configure
> finding my select() function. I'm compiling a big post with
> lots of output from various programs/files, but I may not
> be able to produce a fix the planned release.

That sounds good.  It will be good to see the output.
Don't worry too much about the release schedule -- as I just posted,=20
I've found another database bug :(

> Add a (new) document containing the Unix/Windows flavors
> htdig runs on + more information and hints on how to
> compile and install htdig on those various flavors.
> (HP-UX needs a huge list of parameters to get it
> compiled anyway).

The supported platforms are currently in htdoc/requirements.html.
Is there really a need for a new document?

The compile hints belong in htdoc/install.html.
Could you post the compilation sequence for HP-UX?

Cheers,
Lachlan
--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

[htdig-dev] Bug :(

[Lachlan A. <lh...@us...>]

Greetings all,

I've just come across a database bug :(  It was reporting
  WordKey::Compare: key length for a or b < info.num_length
repeatedly when I ran a large dig without  -i.

I haven't tried repeating it yet, because the dig that produced it=20
takes three days!!  (It uses a rather inefficient external_transport)
I'll try to replicate it using a more manageable data set.

Regards,
Lachlan
--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

[htdig-dev] Re: b5 make fails on Solaris with Sun c++

[Lachlan A. <lh...@us...>]

Thanks for that, Neil.

That problem should be fixed in the attached patch.  Could you try=20
"make -k" so that it will report as many errors at once as possible?

Thanks again,
Lachlan

On Wed, 18 Jun 2003 02:15, Neil Kohl wrote:
> The make of b5 on Solaris 9 with Sun c++ failed. Gory details
> follow. I've got the complete configure and make output saved if
> you need it.

--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

Re: [htdig-dev] 3.2.0b5 Next progress check :)

[J. op d. B. <ht...@op...>]

Hi,
----- Original Message ----- 
From: "Lachlan Andrew" <lh...@us...>
To: <htd...@li...>
Sent: Monday, June 16, 2003 4:19 PM
Subject: [htdig-dev] 3.2.0b5 Next progress check :)


> Greetings all,
> 
> Are people happy with the tentative timetable I mailed a week ago?  If 
> so, the Unix code should be essentially complete now, and 3.2.0b5 
> will be released in one week's time.  Neal and Jess, do you still 
> plan to make any changes?

I'm still having trouble getting the Berkeley database ./configure
finding my select() function. I'm compiling a big post with
lots of output from various programs/files, but I may not
be able to produce a fix the planned release.

> Geoff, could you please post a to-do list for the release docs?  All I 
> can think of is
> - Replace all references to 3.2.0b4 as current with 3.2.0b5
>   (Can this be automated at all?)
> - Remove checklist and solved issues from STATUS
> - Update copyright (if possible -- I don't see this as critical...)

Add a (new) document containing the Unix/Windows flavors
htdig runs on + more information and hints on how to
compile and install htdig on those various flavors.
(HP-UX needs a huge list of parameters to get it
compiled anyway).
 
> Please let me know what I can do to help the release.
> 
> Thanks,
> Lachlan

--Jesse

Re: [htdig-dev] Re: Feedback on configuring ht://Dig?

[J. op d. B. <ht...@op...>]

Hi,

----- Original Message ----- 
From: "Jonathan Baron" <ba...@ps...>
To: "Lachlan Andrew" <lh...@us...>

(deleted lot of text)

> 
> I did notice that, in the source RPM for the new one, but not the
> old one, there was a single file called htdig.conf, consisting of

I think this is an Apache 2.0 plugin config file. Apache 2 can
load files that end with .conf (and sitting is some
predestined directory) together with the httpd.conf so
it's not the htdig config file.

> one line: Alias /htdig /usr/share/htdig
> But this should not affect anything, right?  This file was just
> sitting there.  It was not part of any directory, but it was in
> the root directory when you unpack the source RPM.  I don't know
> enough about RPMs to say.
(snipped text)
> 

--Jesse

[htdig-dev] b5 make fails on Solaris with Sun c++

[Neil K. <nk...@ma...>]

Lachlan,

The make of b5 on Solaris 9 with Sun c++ failed. Gory details follow. I've =
got the=20
complete configure and make output saved if you need it.

--=20
Neil Kohl
Manager, ACP Online
nk...@ma...=20


Hardware: Sun 220

uname -a:
SunOS acpsun01 5.9 Generic sun4u sparc SUNW,Ultra-60

c++ version (CC -V):
CC: WorkShop Compilers 5.0 98/12/15 C++ 5.0

Relevant output from 'make check':

rm -f .libs/Transport.lo
CC -DHAVE_CONFIG_H -I. -I. -I../include -DDEFAULT_CONFIG_FILE=3D\"/opt/www/=
conf/htdig.conf\" -I../include -I../htlib -I../htnet -I../htcommon =
-I../htword -I../db -I../db -g -c Transport.cc  -KPIC -DPIC -o Transport.o
"../htcommon/URL.h", line 94: Warning: String literal converted to char* =
in initialization.
"Connection.h", line 69: Warning: String literal converted to char* in =
initialization.
"Connection.h", line 70: Warning: String literal converted to char* in =
initialization.
"Connection.h", line 71: Warning: String literal converted to char* in =
initialization.
"Connection.h", line 73: Warning: String literal converted to char* in =
initialization.
"Transport.cc", line 423: Error: Cannot assign const char* to char*.
1 Error(s) and 5 Warning(s) detected.
*** Error code 1
make: Fatal error: Command failed for target `Transport.lo'
Current working directory /home/neilk/src/htdig-3.2.0b4-20030615/htnet
*** Error code 1
make: Fatal error: Command failed for target `check-recursive'


>>> Lachlan Andrew <lh...@us...> 06/17/03 09:06AM >>>
Thanks, Neil!  The latest snapshot is at

<http://www.htdig.org/files/snapshots/htdig-3.2.0b4-20030615.tar.gz>

The standard tests we run are just
(a) ./configure --enable-tests && make check
(b) Add "robotstxt_name: master-htdig" to  htdig.conf  and do a dig of
    http://www.htdig.org=20
The second test produces a database of around 400MB.

Thanks again,
Lachlan

On Tue, 17 Jun 2003 04:34, Neil Kohl wrote:
> I've been on vacation for the last week so I just saw this message
> on the htdig list.
>
> Our production htdig was compiled with gcc, but I have Sun C++ on
> one of our Solaris boxes.  I'd be happy to give b5 a whirl. Where's
> the tarball?
>


--=20
lh...@us...=20
ht://Dig developer DownUnder  (http://www.htdig.org)

Re: [htdig-dev] 3.2.0b5 Next progress check :)

[Neal R. <ne...@ri...>]

On Tue, 17 Jun 2003, Lachlan Andrew wrote:

> That sounds great, Neal.
>
> Were you also planning to update the copyright dates to 2003 as well
> as the COPYING file itself?  (I'm not asking you to do it, just
> asking if you're planning to :)

   Sure!  I'll update the copyright notices as well.


> It will be great having your Win32 patch in.  That will help a lot of
> people.
>
> Thanks for all your work!
> Lachlan
>
> On Tue, 17 Jun 2003 02:06, Neal Richter wrote:
> > On Tue, 17 Jun 2003, Lachlan Andrew wrote:
> > > Neal and Jess, do you still plan to make any changes?
> >
> >   Yes.  I'll do some more with memory-leak fixes and the Win32
> > port.
> >
> >   I'm in the process of comparing my tree with the CVS tree and
> > moving changes over to CVS. I plan to commit code frequently this
> > week and get my tree synched.
> >
> >   I will also take care of updating the GPL notifications to LGPL.
> >
> >   Thanks!
> >
> > > Geoff, could you please post a to-do list for the release docs?
> > > All I can think of is
> > > - Replace all references to 3.2.0b4 as current with 3.2.0b5
> > >   (Can this be automated at all?)
> > > - Remove checklist and solved issues from STATUS
> > > - Update copyright (if possible -- I don't see this as
> > > critical...)
> > >
> > > Please let me know what I can do to help the release.
> > >
> > > Thanks,
> > > Lachlan
> > >
> > > On Sun, 8 Jun 2003 15:20, Lachlan Andrew wrote:
> > > > Thank you to everybody for your work towards getting 3.2.0b5
> > > > out. Geoff:  Did you manage the copyright update?  If you have
> > > > written the sed script, but haven't had time to apply/commit,
> > > > could you please post it?  I should have some time this week to
> > > > do that.
> > > >
> > > > Neal: It's good to hear that you're coming along with the Win32
> > > > port. From where I'm standing, the highest priority is applying
> > > > the memory-leak patch, so that the rest of us can test it
> > > > thoroughly. Could you please do that some time this week?
> > > >
> > > > Jess:  Any luck with the HP-UX configuration?  If you post the
> > > > config.log then one of the rest of us might have an idea why it
> > > > is failing.
> > > >
> > > > Does the following timeline sound feasible?
> > > > This time, I think the showstoppers are gone, so we can afford
> > > > to set a firm timeline (although not necessarily this one :)
> > > >
> > > > Sun 8 - Fri 13: Finalise Unix code / configuration
> > > > 		Update copyright
> > > > 		Other documentation updates?
> > > > Weekend 14-15: Unix code freeze, and re-test all available
> > > >                platforms
> > > >		 Ask "non-team" volunteers to "install/make check" the
> > > >		 snapshot
> > > > Mon 16-Fri 20: Finish Win32, or decide to postpone until
> > >
> > > 		 3.2.0b6 Further documentation updates?
> > >
> > > > 		 Testing by non-team volunteers
> > > > Weekend 21-22: Code freeze
> > > > 		 Test Win32 (if finished)
> > > > 		 Rename 3.2.0b4->3.2.0b5
> > > > Mon 23:        BETA RELEASE
> > > >
> > > > What docs need to be updated?
> > >
> > > --
> > > lh...@us...
> > > ht://Dig developer DownUnder  (http://www.htdig.org)
> > >
> > >
> > > -------------------------------------------------------
> > > This SF.NET email is sponsored by: eBay
> > > Great deals on office technology -- on eBay now! Click here:
> > > http://adfarm.mediaplex.com/ad/ck/711-11697-6916-5
> > > _______________________________________________
> > > htdig-dev mailing list
> > > htd...@li...
> > > https://lists.sourceforge.net/lists/listinfo/htdig-dev
> >
> > Neal Richter
> > Knowledgebase Developer
> > RightNow Technologies, Inc.
> > Customer Service for Every Web Site
> > Office: 406-522-1485
>
> --
> lh...@us...
> ht://Dig developer DownUnder  (http://www.htdig.org)
>

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] 3.2.0b5 Next progress check :)

[Lachlan A. <lh...@us...>]

That sounds great, Neal.

Were you also planning to update the copyright dates to 2003 as well=20
as the COPYING file itself?  (I'm not asking you to do it, just=20
asking if you're planning to :)

It will be great having your Win32 patch in.  That will help a lot of=20
people.

Thanks for all your work!
Lachlan
=20
On Tue, 17 Jun 2003 02:06, Neal Richter wrote:
> On Tue, 17 Jun 2003, Lachlan Andrew wrote:
> > Neal and Jess, do you still plan to make any changes?
>
>   Yes.  I'll do some more with memory-leak fixes and the Win32
> port.
>
>   I'm in the process of comparing my tree with the CVS tree and
> moving changes over to CVS. I plan to commit code frequently this
> week and get my tree synched.
>
>   I will also take care of updating the GPL notifications to LGPL.
>
>   Thanks!
>
> > Geoff, could you please post a to-do list for the release docs?=20
> > All I can think of is
> > - Replace all references to 3.2.0b4 as current with 3.2.0b5
> >   (Can this be automated at all?)
> > - Remove checklist and solved issues from STATUS
> > - Update copyright (if possible -- I don't see this as
> > critical...)
> >
> > Please let me know what I can do to help the release.
> >
> > Thanks,
> > Lachlan
> >
> > On Sun, 8 Jun 2003 15:20, Lachlan Andrew wrote:
> > > Thank you to everybody for your work towards getting 3.2.0b5
> > > out. Geoff:  Did you manage the copyright update?  If you have
> > > written the sed script, but haven't had time to apply/commit,
> > > could you please post it?  I should have some time this week to
> > > do that.
> > >
> > > Neal: It's good to hear that you're coming along with the Win32
> > > port. From where I'm standing, the highest priority is applying
> > > the memory-leak patch, so that the rest of us can test it
> > > thoroughly. Could you please do that some time this week?
> > >
> > > Jess:  Any luck with the HP-UX configuration?  If you post the
> > > config.log then one of the rest of us might have an idea why it
> > > is failing.
> > >
> > > Does the following timeline sound feasible?
> > > This time, I think the showstoppers are gone, so we can afford
> > > to set a firm timeline (although not necessarily this one :)
> > >
> > > Sun 8 - Fri 13: Finalise Unix code / configuration
> > > =09=09Update copyright
> > > =09=09Other documentation updates?
> > > Weekend 14-15: Unix code freeze, and re-test all available
> > >                platforms
> > >=09=09 Ask "non-team" volunteers to "install/make check" the
> > >=09=09 snapshot
> > > Mon 16-Fri 20: Finish Win32, or decide to postpone until
> >
> > =09=09 3.2.0b6 Further documentation updates?
> >
> > > =09=09 Testing by non-team volunteers
> > > Weekend 21-22: Code freeze
> > > =09=09 Test Win32 (if finished)
> > > =09=09 Rename 3.2.0b4->3.2.0b5
> > > Mon 23:        BETA RELEASE
> > >
> > > What docs need to be updated?
> >
> > --
> > lh...@us...
> > ht://Dig developer DownUnder  (http://www.htdig.org)
> >
> >
> > -------------------------------------------------------
> > This SF.NET email is sponsored by: eBay
> > Great deals on office technology -- on eBay now! Click here:
> > http://adfarm.mediaplex.com/ad/ck/711-11697-6916-5
> > _______________________________________________
> > htdig-dev mailing list
> > htd...@li...
> > https://lists.sourceforge.net/lists/listinfo/htdig-dev
>
> Neal Richter
> Knowledgebase Developer
> RightNow Technologies, Inc.
> Customer Service for Every Web Site
> Office: 406-522-1485

--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)