htdig-dev — Developer Discussion for the ht://Dig project

[htdig-dev] MIME Types in Document index & update in libhtdig

[Neal R. <ne...@ri...>]

Hey,
	I'm reading through various pieces of code and I have a question.

	Has there been any thought to storing the MIME-type of a document
in the index?  An integer would do just fine for this..  I can't seem to
find anything in classes like DocumentRef to indicate that anyone
has
done this before.

FYI:
	I'll be committing a new directory libhtdig & libhtdigphp to the
main CVS server this week.  It won't affect anyone since this new code
doesn't build as part of the normal make-process.

	Let me know if you have any concerns or questions.

Thanks!

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

[htdig-dev] Re: Database error with small data set

[<And...@wi...>]

I used redhat
2.2.16-22enterprise #1 SMP Tue Aug 22 16:29:32 EDT 2000 i686
gcc -v
Reading specs from /usr/local/lib/gcc-lib/i686-pc-linux-gnu/3.1/specs
Configured with: ./configure
Thread model: single
gcc version 3.1:

locate libz.so
/usr/lib/libz.so.1
/usr/lib/libz.so.1.1.3

ld -v
GNU ld version 2.10.90 (with BFD 2.10.0.18)

I got the overflow error, but just once, haven't seen it since.

a

Andy Bach, Sys. Mangler
Internet: and...@wi... 
VOICE: (608) 261-5738  FAX 264-5030

"If you don't say anything, you won't be called on to repeat it" 
Calvin Coolidge

[htdig-dev] [ htdig-Patches-708700 ] Making htdig language sensitive

[SourceForge.net <no...@so...>]

Patches item #708700, was opened at 2003-03-24 08:59
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=304593&aid=708700&group_id=4593

Category: stable
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Rick van Rein (vanrein)
Assigned to: Nobody/Anonymous (nobody)
Summary: Making htdig language sensitive

Initial Comment:
Hello people,

I had feature request #707489 in search for language-
dependent support.  This is useful for sites that use 
content negotiation using the Accept-Language HTTP 
header.

I embellished htdig to process a setting accept_language 
in the config file, by sending it along with the query.  The 
patch is attached to the feature request.

In addition, I made a wrapper around htsearch to make it 
look for different languages; this could be done more 
elegantly, but I do not want to change so much to ht://Dig.  
Simplicity rules, especially for new features!

If you want to see it in action, try the search engine on 
openfortress.nl; it recognises Dutch and English, and the 
pages that are already translated are discriminated.  Try 
the terms "notaris", "notary" and "lawyer" for example, while 
shuffling your browser's language settings and reloading.

This now works well.  Let me know if you like it.

Cheers,

Rick van Rein,
OpenFortress.


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=304593&aid=708700&group_id=4593

[htdig-dev] Current Status as of snapshot 3.2.0b4-20030323

[Geoff H. <ghu...@us...>]

STATUS of ht://Dig branch 3-2-x

RELEASES:
   3.2.0b5: Next release, First quarter 2003???
   3.2.0b4: "In progress" -- snapshots called "3.2.0b4" until prerelease.
   3.2.0b3: Released:  22 Feb 2001.
   3.2.0b2: Released:  11 Apr 2000.
   3.2.0b1: Released:   4 Feb 2000.

(Please note that everything added here should have a tracker PR# so
we can be sure they're fixed. Geoff is currently trying to add PR#s for
what's currently here.)

SHOWSTOPPERS:
* Mifluz database errors are a severe problem (PR#428295)
-- Does Neal's new zlib patch solve this for now?

KNOWN BUGS:
* Odd behavior with $(MODIFIED) and scores not working with
   wordlist_compress set but work fine without wordlist_compress.
   (the date is definitely stored correctly, even with compression on
    so this must be some sort of weird htsearch bug) PR#618737.
* META descriptions are somehow added to the database as FLAG_TITLE,
   not FLAG_DESCRIPTION. (PR#618738)
   Can anyone reproduce this?  I can't! -- Lachlan

PENDING PATCHES (available but need work):
* Additional support for Win32.
* Memory improvements to htmerge. (Backed out b/c htword API changed.)
* Mifluz merge.

NEEDED FEATURES:
* Quim's new htsearch/qtest query parser framework.
* File/Database locking. PR#405764.

TESTING:
* httools programs: 
  (htload a test file, check a few characteristics, htdump and compare)
* Tests for new config file parser
* Duplicate document detection while indexing
* Major revisions to ExternalParser.cc, including fork/exec instead of popen,
  argument handling for parser/converter, allowing binary output from an
  external converter.
* ExternalTransport needs testing of changes similar to ExternalParser.

DOCUMENTATION:
* List of supported platforms/compilers is ancient. (PR#405279)
* Document all of htsearch's mappings of input parameters to config attributes
   to template variables. (Relates to PR#405278.)
  Should we make sure these config attributes are all documented in
  defaults.cc, even if they're only set by input parameters and never
  in the config file?
* Split attrs.html into categories for faster loading.
* Turn defaults.cc into an XML file for generating documentation and 
  defaults.cc.
* require.html is not updated to list new features and disk space
   requirements of 3.2.x (e.g. regex matching, database compression.)
   PRs# 405280 #405281.
* TODO.html has not been updated for current TODO list and
   completions.
   I've tried.  Someone "official" please check and remove this -- Lachlan
* Htfuzzy could use more documentation on what each fuzzy algorithm
   does. PR#405714.
* Document the list of all installed files and default
   locations. PR#405715.

OTHER ISSUES:
* Can htsearch actually search while an index is being created?
* The code needs a security audit, esp. htsearch. PR#405765.

[htdig-dev] Re: Database error with small data set

[Neal R. <ne...@ri...>]

I can't get duplication on Mandrake 8.1, RedHat 8.0, and RedHat 7.3
machines.

The RedHat machines don't make it through ./configure --enable_tests,
various errors happen.  I'm reading the configure script to figure out
why.

The Mandrake 8.1 machine passes all tests repeatedly with no errors in
htmerge.

Thanks

Neal.


On Sat, 22 Mar 2003, Lachlan Andrew wrote:

> Thanks for looking at this, Neal.
>
> I get the panic using Mandrake 8.2 with
>
> % gcc -v
> Reading specs from /usr/lib/gcc-lib/i586-mandrake-linux-gnu/3.2/specs
> Configured with: ../configure --prefix=/usr --libdir=/usr/lib
> --with-slibdir=/lib --mandir=/usr/share/man --infodir=/usr/share/info
> --enable-shared --enable-threads=posix --disable-checking
> --enable-long-long --enable-__cxa_atexit
> --enable-languages=c,c++,ada,f77,objc,java
> --host=i586-mandrake-linux-gnu --with-system-zlib
> Thread model: posix
> gcc version 3.2 (Mandrake Linux 9.0 3.2-1mdk)
>
> % locate libz.so
> /usr/lib/libz.so
> /usr/lib/libz.so.1.1.4
> /lib/libz.so.1
> /lib/libz.so.1.1.4
>
> % ld -v
> GNU ld version 2.12.90.0.15 20020717
>
>
> Andy Bach has also reported this problem.  Andy, what system are you
> using?
>
> When I put in debugging code to see what was happening, the error
> started manifesting itself in  t_htdig_local  instead, but it was
> still page 3.  The problem is that the free list for  _weakcmpr
> seems to be getting corrupted.  Page 3 overflows into a chain
> 3->71->72, but then another page overflows into page 71, corrupting
> page 3.
>
> Do you know how to display the free list?  I haven't even been able to
> find where new pages are allocated -- every scent I follow leads back
> to *memory* page allocation, rather than *file* page allocation :(
> The file  db_vrfy.c  has code to print the free list, but the file
> seems to have been copied in from a later version of BDB, and never
> integrated properly.
>
> Thanks,
> Lachlan
>
> On Tuesday 18 March 2003 09:51, Neal Richter wrote:
> > >     ./configure --enable-tests
> > >     make
> > >     make check
> > >     make check
> > Hmm.  I can't get this to duplicate!!!
> > ARG!  What kind of Linux system do you have running?
>

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

[htdig-dev] Re: Database error with small data set

[Lachlan A. <lh...@us...>]

Thanks for looking at this, Neal.

I get the panic using Mandrake 8.2 with

% gcc -v
Reading specs from /usr/lib/gcc-lib/i586-mandrake-linux-gnu/3.2/specs
Configured with: ../configure --prefix=3D/usr --libdir=3D/usr/lib=20
--with-slibdir=3D/lib --mandir=3D/usr/share/man --infodir=3D/usr/share/in=
fo=20
--enable-shared --enable-threads=3Dposix --disable-checking=20
--enable-long-long --enable-__cxa_atexit=20
--enable-languages=3Dc,c++,ada,f77,objc,java=20
--host=3Di586-mandrake-linux-gnu --with-system-zlib
Thread model: posix
gcc version 3.2 (Mandrake Linux 9.0 3.2-1mdk)

% locate libz.so
/usr/lib/libz.so
/usr/lib/libz.so.1.1.4
/lib/libz.so.1
/lib/libz.so.1.1.4

% ld -v
GNU ld version 2.12.90.0.15 20020717


Andy Bach has also reported this problem.  Andy, what system are you=20
using?

When I put in debugging code to see what was happening, the error=20
started manifesting itself in  t_htdig_local  instead, but it was=20
still page 3.  The problem is that the free list for  _weakcmpr =20
seems to be getting corrupted.  Page 3 overflows into a chain=20
3->71->72, but then another page overflows into page 71, corrupting=20
page 3.

Do you know how to display the free list?  I haven't even been able to=20
find where new pages are allocated -- every scent I follow leads back=20
to *memory* page allocation, rather than *file* page allocation :( =20
The file  db_vrfy.c  has code to print the free list, but the file=20
seems to have been copied in from a later version of BDB, and never=20
integrated properly.

Thanks,
Lachlan

On Tuesday 18 March 2003 09:51, Neal Richter wrote:
> >     ./configure --enable-tests
> >     make
> >     make check
> >     make check
> Hmm.  I can't get this to duplicate!!!
> ARG!  What kind of Linux system do you have running?

Re: [htdig-dev] ssl configure troubles

[<tro...@ju...>]

There is a newer version of 'check_ssl' macro available that handles
CXXFLAGS too (at least it was submitted about a month or so ago) - 
essentially exactly as you mention...
As for HAVE_SSL vs. HAVE_SSL_H - welp yer on yer own!!
    Mark

On Mon, 17 Mar 2003, And...@wi... wrote:

> > .. but it's usually faster to check the configure.in, aclocal.m4 
> > and acinclude.m4 files for the appropriate sections. 
> 
> A little easier ;-> But, somebody smarter will still have to the correct 
> decisions for me, er htdig.  acinclude.m4 (which is a shorter version of 
> aclocal.m4?? at least the SSL section is the same) works on HAVE_SSL but 
> it appears that's not used in the code, HAVE_SSL_H is. As importantly 
> htnet/SSL*.cc is where the ssl work is needed and that uses C++ and so 
> CXXFLAGS.  Finally, the #includes in the htnet/SSL*.cc code are all:
> #ifdef HAVE_SSL_H
> #include <openssl/crypto.h>
> #include <openssl/x509.h>
> #include <openssl/pem.h>
> #include <openssl/ssl.h>
> #include <openssl/err.h>
> 
> which means they're assuming openssl and want an include path of 
> /usr/local/ssl.  The m4 routine:
> AC_DEFUN([CHECK_SSL],
> [AC_MSG_CHECKING(if ssl is wanted)
> AC_ARG_WITH(ssl,
> [  --with-ssl enable ssl [will check /usr/local/ssl
>                             /usr/lib/ssl /usr/ssl /usr/pkg /usr/local /usr 
> ]
> ],
> [   AC_MSG_RESULT(yes)
>     for dir in $withval /usr/local/ssl /usr/lib/ssl /usr/ssl /usr/pkg 
> /usr/local
>  /usr; do
>         ssldir="$dir"
>         if test -f "$dir/include/openssl/ssl.h"; then
>             found_ssl="yes";
>             CFLAGS="$CFLAGS -I$ssldir/include/openssl -DHAVE_SSL";
>             break;
>         fi
>         if test -f "$dir/include/ssl.h"; then
>             found_ssl="yes";
>             CFLAGS="$CFLAGS -I$ssldir/include/ -DHAVE_SSL";
>             break
>         fi
>     done
>     if test x_$found_ssl != x_yes; then
>         AC_MSG_ERROR(Cannot find ssl libraries)
>     else
>         printf "OpenSSL found in $ssldir\n";
>         LIBS="$LIBS -lssl -lcrypto";
>         LDFLAGS="$LDFLAGS -L$ssldir/lib";
>         HAVE_SSL=yes
>     fi
>     AC_SUBST(HAVE_SSL)
> ],
> [
>     AC_MSG_RESULT(no)
> ])
> ])
> 
> 
> so change HAVE_SSL to HAVE_SSL_H and change:
>             CFLAGS="$CFLAGS -I$ssldir/include/ -DHAVE_SSL";
> to:
>             CXXFLAGS="$CXXFLAGS -I$ssldir/include/ -DHAVE_SSL_H";
> 
> though its probably better to change the htnet/SSL*cc stuff to use:
> #include <ssl.h>
> 
> if you want to keep the -I$ssldir/include/ vs -I$ssldir/include/openssl
> 
> or is that a six/half dozen thing?
> 
> 
> > make distclean 
> 
> This would be an automake bug. What do the top of the Makefile.in files
> say? (I want to be sure it's the original CVS version and not regenerated
> on your system.)
> 
> # Makefile.in generated by automake 1.6.3 from Makefile.am.
> # @configure_input@
> 
> here's the chunk from Makefile.in
> #
> # If --enable-tests is not specified, should remove
> # the test/Makefile anyway
> #
> distclean-local:
>         if "X$(TESTDIR)" = "Xtest" ; \
>         then \
>                 rm -f test/Makefile test/test_functions ; \
>         fi
> 
> It seems to be missing the 'test' sh cmd (and 'X' is an executable prog):
> #
> # If --enable-tests is not specified, should remove
> # the test/Makefile anyway
> #
> distclean-local:
>         if test "X$(TESTDIR)" = "Xtest" ; \
>         then \
>                 rm -f test/Makefile test/test_functions ; \
>         fi
> 
> 
> 

Re: [htdig-dev] Database error with small data set

[Neal R. <ne...@ri...>]

> I've found a (much) smaller data set that gives me errors.
> What do other people get with:
>     ./configure --enable-tests
>     make
>     make check
>     make check
>
> I get  t_htmerge  failing the second time, with a read error on page 3
> (if I remove the redirection of  stderr).

Hmm.  I can't get this to duplicate!!!

ARG!  What kind of Linux system do you have running?

Thanks!

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] ssl configure troubles

[<And...@wi...>]

> .. but it's usually faster to check the configure.in, aclocal.m4 
> and acinclude.m4 files for the appropriate sections. 

A little easier ;-> But, somebody smarter will still have to the correct 
decisions for me, er htdig.  acinclude.m4 (which is a shorter version of 
aclocal.m4?? at least the SSL section is the same) works on HAVE_SSL but 
it appears that's not used in the code, HAVE_SSL_H is. As importantly 
htnet/SSL*.cc is where the ssl work is needed and that uses C++ and so 
CXXFLAGS.  Finally, the #includes in the htnet/SSL*.cc code are all:
#ifdef HAVE_SSL_H
#include <openssl/crypto.h>
#include <openssl/x509.h>
#include <openssl/pem.h>
#include <openssl/ssl.h>
#include <openssl/err.h>

which means they're assuming openssl and want an include path of 
/usr/local/ssl.  The m4 routine:
AC_DEFUN([CHECK_SSL],
[AC_MSG_CHECKING(if ssl is wanted)
AC_ARG_WITH(ssl,
[  --with-ssl enable ssl [will check /usr/local/ssl
                            /usr/lib/ssl /usr/ssl /usr/pkg /usr/local /usr 
]
],
[   AC_MSG_RESULT(yes)
    for dir in $withval /usr/local/ssl /usr/lib/ssl /usr/ssl /usr/pkg 
/usr/local
 /usr; do
        ssldir="$dir"
        if test -f "$dir/include/openssl/ssl.h"; then
            found_ssl="yes";
            CFLAGS="$CFLAGS -I$ssldir/include/openssl -DHAVE_SSL";
            break;
        fi
        if test -f "$dir/include/ssl.h"; then
            found_ssl="yes";
            CFLAGS="$CFLAGS -I$ssldir/include/ -DHAVE_SSL";
            break
        fi
    done
    if test x_$found_ssl != x_yes; then
        AC_MSG_ERROR(Cannot find ssl libraries)
    else
        printf "OpenSSL found in $ssldir\n";
        LIBS="$LIBS -lssl -lcrypto";
        LDFLAGS="$LDFLAGS -L$ssldir/lib";
        HAVE_SSL=yes
    fi
    AC_SUBST(HAVE_SSL)
],
[
    AC_MSG_RESULT(no)
])
])


so change HAVE_SSL to HAVE_SSL_H and change:
            CFLAGS="$CFLAGS -I$ssldir/include/ -DHAVE_SSL";
to:
            CXXFLAGS="$CXXFLAGS -I$ssldir/include/ -DHAVE_SSL_H";

though its probably better to change the htnet/SSL*cc stuff to use:
#include <ssl.h>

if you want to keep the -I$ssldir/include/ vs -I$ssldir/include/openssl

or is that a six/half dozen thing?


> make distclean 

This would be an automake bug. What do the top of the Makefile.in files
say? (I want to be sure it's the original CVS version and not regenerated
on your system.)

# Makefile.in generated by automake 1.6.3 from Makefile.am.
# @configure_input@

here's the chunk from Makefile.in
#
# If --enable-tests is not specified, should remove
# the test/Makefile anyway
#
distclean-local:
        if "X$(TESTDIR)" = "Xtest" ; \
        then \
                rm -f test/Makefile test/test_functions ; \
        fi

It seems to be missing the 'test' sh cmd (and 'X' is an executable prog):
#
# If --enable-tests is not specified, should remove
# the test/Makefile anyway
#
distclean-local:
        if test "X$(TESTDIR)" = "Xtest" ; \
        then \
                rm -f test/Makefile test/test_functions ; \
        fi

Re: [htdig-dev] ssl configure troubles

[Geoff H. <ghu...@ws...>]

On Mon, 17 Mar 2003 And...@wi... wrote:

> first seems due to configure's:

OK... First gentle suggestion. Don't try reading/debugging a configure
script. I'm impressed, but it's usually faster to check the configure.in,
aclocal.m4 and acinclude.m4 files for the appropriate sections. Easier to
debug and it's easier for us to fix since configure scripts are generated
from these.

> which resets CFLAGS regardless of OLD_FLAGS being set?  This line should 
> be moved into the if section:
>  CFLAGS=$OLD_CFLAGS

Certainly a good fix.

> And, it appears the problem is that CXXFLAGS needs to have the SSL include 
> path added to it.  I munged things to do that, but I'm sure there's a 
> cleaner way.

Yes, please take a look at the acinclude/aclocal files which have the
CHECK_SSL macro. I think it's supposed to be in the acinclude file.

> make distclean 

This would be an automake bug. What do the top of the Makefile.in files
say? (I want to be sure it's the original CVS version and not regenerated
on your system.)

Thanks,
-Geoff

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

[htdig-dev] ssl configure troubles

[<And...@wi...>]

Hey.

Retrying this 2/17's cvs and, again, it ignored --with-ssl=/usr/local/ssl, 
neither setting HAVE_SSL_H nor including -I/usr/local/ssl/include in 
CFLAGS (actually, I mean CXXFLAGS - though CFLAGS may be being reset). The 
first seems due to configure's:
if test "${with_ssl+set}" = set; then
  withval="$with_ssl"
     echo "$as_me:$LINENO: result: yes" >&5
echo "${ECHO_T}yes" >&6
    for dir in $withval /usr/local/ssl /usr/lib/ssl /usr/ssl /usr/pkg 
/usr/local
 /usr; do
        ssldir="$dir"
        if test -f "$dir/include/openssl/ssl.h"; then
            found_ssl="yes";
            CFLAGS="$CFLAGS -I$ssldir/include/openssl -DHAVE_SSL";
            break;
        fi
        if test -f "$dir/include/ssl.h"; then
            found_ssl="yes";
            CFLAGS="$CFLAGS -I$ssldir/include/ -DHAVE_SSL";
            break
        fi
    done
    if test x_$found_ssl != x_yes; then
        { { echo "$as_me:$LINENO: error: Cannot find ssl libraries" >&5
echo "$as_me: error: Cannot find ssl libraries" >&2;}
   { (exit 1); exit 1; }; }
    else
        printf "OpenSSL found in $ssldir\n";
        LIBS="$LIBS -lssl -lcrypto";
        LDFLAGS="$LDFLAGS -L$ssldir/lib";
        HAVE_SSL=yes
    fi


which sets HAVE_SSL, and not HAVE_SSL_H.  I added
            CXXFLAGS="$CXXFLAGS -I$ssldir/include/openssl -DHAVE_SSL";

but that's doesn' work as the SSL routines include
#ifdef HAVE_SSL_H
#include <openssl/crypto.h>
#include <openssl/x509.h>
#include <openssl/pem.h>
#include <openssl/ssl.h>
#include <openssl/err.h>
so:
            CXXFLAGS="$CXXFLAGS -I$ssldir/include/ -DHAVE_SSL";

and then (adding HAVE_SSL_H to the flags list too):
HAVE_SSL_H=yes

The 2nd seems to be:
if test $WITH_RX = no; then
 echo "$as_me:$LINENO: checking if we should use the included regex?" >&5
echo $ECHO_N "checking if we should use the included regex?... $ECHO_C" 
>&6
 OLD_CFLAGS=$CFLAGS
 CFLAGS="$CFLAGS -I${srcdir}/htlib"
 if test "$cross_compiling" = yes; then
  echo "$as_me:$LINENO: result: unknown" >&5
echo "${ECHO_T}unknown" >&6;cat >>confdefs.h <<\_ACEOF
#define HAVE_BROKEN_REGEX 1
_ACEOF

else
  cat >conftest.$ac_ext <<_ACEOF
#line $LINENO "configure"
...
rm -f core core.* *.core gmon.out bb.out conftest$ac_exeext 
conftest.$ac_objext
conftest.$ac_ext
fi           # if test $WITH_RX=no
 CFLAGS=$OLD_CFLAGS
fi            # ?????

which resets CFLAGS regardless of OLD_FLAGS being set?  This line should 
be moved into the if section:
 CFLAGS=$OLD_CFLAGS

And, it appears the problem is that CXXFLAGS needs to have the SSL include 
path added to it.  I munged things to do that, but I'm sure there's a 
cleaner way.

and finally,
make distclean 
gets:

if "X" = "Xtest" ; \
then \
        rm -f test/Makefile test/test_functions ; \
fi
Unrecognized option: =
use: X [:<display>] [option]

from:
distclean-local:
        if "X$(TESTDIR)" = "Xtest" ; \
        then \
                rm -f test/Makefile test/test_functions ; \
        fi

s/b ??
distclean-local:
        if test "X$(TESTDIR)" = "Xtest"  ; \
        then \
                rm -f test/Makefile test/test_functions ; \
        fi


Andy Bach, Sys. Mangler
Internet: and...@wi... 
VOICE: (608) 261-5738  FAX 264-5030

"If you don't say anything, you won't be called on to repeat it" 
Calvin Coolidge

[htdig-dev] Current Status as of snapshot 3.2.0b4-20030316

[Geoff H. <ghu...@us...>]

STATUS of ht://Dig branch 3-2-x

RELEASES:
   3.2.0b5: Next release, First quarter 2003???
   3.2.0b4: "In progress" -- snapshots called "3.2.0b4" until prerelease.
   3.2.0b3: Released:  22 Feb 2001.
   3.2.0b2: Released:  11 Apr 2000.
   3.2.0b1: Released:   4 Feb 2000.

(Please note that everything added here should have a tracker PR# so
we can be sure they're fixed. Geoff is currently trying to add PR#s for
what's currently here.)

SHOWSTOPPERS:
* Mifluz database errors are a severe problem (PR#428295)
-- Does Neal's new zlib patch solve this for now?

KNOWN BUGS:
* Odd behavior with $(MODIFIED) and scores not working with
   wordlist_compress set but work fine without wordlist_compress.
   (the date is definitely stored correctly, even with compression on
    so this must be some sort of weird htsearch bug) PR#618737.
* META descriptions are somehow added to the database as FLAG_TITLE,
   not FLAG_DESCRIPTION. (PR#618738)
   Can anyone reproduce this?  I can't! -- Lachlan

PENDING PATCHES (available but need work):
* Additional support for Win32.
* Memory improvements to htmerge. (Backed out b/c htword API changed.)
* Mifluz merge.

NEEDED FEATURES:
* Quim's new htsearch/qtest query parser framework.
* File/Database locking. PR#405764.

TESTING:
* httools programs: 
  (htload a test file, check a few characteristics, htdump and compare)
* Tests for new config file parser
* Duplicate document detection while indexing
* Major revisions to ExternalParser.cc, including fork/exec instead of popen,
  argument handling for parser/converter, allowing binary output from an
  external converter.
* ExternalTransport needs testing of changes similar to ExternalParser.

DOCUMENTATION:
* List of supported platforms/compilers is ancient. (PR#405279)
* Document all of htsearch's mappings of input parameters to config attributes
   to template variables. (Relates to PR#405278.)
  Should we make sure these config attributes are all documented in
  defaults.cc, even if they're only set by input parameters and never
  in the config file?
* Split attrs.html into categories for faster loading.
* Turn defaults.cc into an XML file for generating documentation and 
  defaults.cc.
* require.html is not updated to list new features and disk space
   requirements of 3.2.x (e.g. regex matching, database compression.)
   PRs# 405280 #405281.
* TODO.html has not been updated for current TODO list and
   completions.
   I've tried.  Someone "official" please check and remove this -- Lachlan
* Htfuzzy could use more documentation on what each fuzzy algorithm
   does. PR#405714.
* Document the list of all installed files and default
   locations. PR#405715.

OTHER ISSUES:
* Can htsearch actually search while an index is being created?
* The code needs a security audit, esp. htsearch. PR#405765.

[htdig-dev] db_vrfy.c

[Lachlan A. <lh...@us...>]

Greetings,

The routines in  db/db_vrfy.c  look very useful, but the current  db.h =20
and  db_int.h  don't support its compilation.  Does anyone know how=20
to compile it (and why it is currently not supported)?  Are they=20
regenerated by  configure,  and do I need to modify  Makefile.am  and=20
run  automake  etc?

Thanks,
Lachlan

[htdig-dev] Re: panic

[Lachlan A. <lh...@us...>]

Thanks for the feedback.  We're currently having some trouble with the=20
zlib compression.  You can avoid the panic by either turning=20
compression off entirely or switching to non-zlib compression (which=20
has its own set of bugs...).

Cheers,
Lachlan

On Saturday 15 March 2003 02:12, And...@wi... wrote:

> Again from this week's cvs tree, a run on Wed:
> WordDB: CDB___memp_cmpr_read: unable to uncompress page at pgno =3D
> 89 WordDB: PANIC: Input/output error

[htdig-dev] panic

[<And...@wi...>]

Hey, 

Again from this week's cvs tree, a run on Wed:
WordDB: CDB___memp_cmpr_read: unable to uncompress page at pgno = 89
WordDB: PANIC: Input/output error
[ 100s of times (all at pgno 89), interspersed w/ bad pdf msgs from 
conv_doc.pl  until, finally]
WordDB: CDB___memp_cmpr_read: unable to uncompress page at pgno = 31
WordDB: PANIC: Input/output error
WordDBCursor::Get(17) failed DB_RUNRECOVERY: Fatal error, run database 
recovery

Er, its not a big search, though it is https.  The next day, its seems to 
have worked fine.

Please let know what I can do to supply useful debug etc. info,, if you 
want it.

a

Andy Bach, Sys. Mangler
Internet: and...@wi... 
VOICE: (608) 261-5738  FAX 264-5030

"If you don't say anything, you won't be called on to repeat it" 
Calvin Coolidge

Re: [htdig-dev] automake problems installing from CVS

[Geoff H. <ghu...@ws...>]

First off, you don't need automake or autoconf unless you're modifying the
Makefiles or configure script. If you're just editing existing files, you
shouldn't need them.

> RedHat 7.3 system, with (generally) all official RedHat updates.
> make-3.79.1-8
> gcc-c++-2.96-113
> libstdc++-2.96-113
> automake-1.4p5-4
> autoconf-2.13-17

These may be the "official RedHat updates" to 7.3, but that doesn't mean
they're the latest versions of anything. Remember that updates are
typically only bug-fixes, not new versions.

> It says:
> configure.in: 1283: required file `./_AM_File,.in' not found
> configure.in: 1283: required file `./[$2],.in' not found
> configure.in: 1283: required file `./[$3])].in' not found
...
> Is this a bug, or do I need a newer version of Automake ?

I'm currently using automake-1.7.1 and autoconf-2.56. You can get them
from ftp://ftp.gnu.org/ or mirrors.

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

[htdig-dev] cvs build

[<And...@wi...>]

Hey.

For another data point, I built from cvs yesterday on a Redhat 7 box. 
Using gcc 2.96, I couldn't get it to find libstdc++, and saw:
configure: WARNING: fcntl.h: present but cannot be compiled
configure: WARNING: fcntl.h: check for missing prerequisite headers?
configure: WARNING: fcntl.h: proceeding with the preprocessor's result
configure: WARNING:     ## ------------------------------------ ##
configure: WARNING:     ## Report this to bug...@gn.... ##

for limits.h, malloc.h sys/file.h ioctl.h sys/time.h getopt.h sys/wait.h 
wait.h and alloc.h, then:
configure: error: To compile, you will need a C++ library. Try installing 
libstdc++

Upgradding to gcc 3.1 got past that, but it failed to recognize ssl, 
despite --with-ssl=/usr/local/ssl. I changed include/htconfig.h to define 
HAVE_SSL_H and then added -I /usr/local/ssl/include to cppflags to get it 
to work.  I did get all the 'deprecated' warnings
                 from DB2_db.cc:23:
/usr/local/include/g++-v3/backward/backward_warning.h:32:2: warning: 
#warning This file includes at least one deprecated or antiquated header. 
Please consider using one of the 32 headers found in section 17.4.1.2 of 
the C++ standard. Examples include substituting the <X> header for the 
<X.h> header for C++ includes, or <sstream> instead of the deprecated 
header <strstream.h>. To disable this warning use -Wno-deprecated.

Autoconf version 2.13

a

Andy Bach, Sys. Mangler
Internet: and...@wi... 
VOICE: (608) 261-5738  FAX 264-5030

"If you don't say anything, you won't be called on to repeat it" 
Calvin Coolidge

[htdig-dev] Re: Re: Residual database errors

[Lachlan A. <lh...@us...>]

Just a thought, but may the problem be related to:
=09General Access Method Changes:
=096. Fix a bug in which DB-managed memory returned by a DB->get or
=09   DB->put call may be corrupted by a later cursor call. [#3576]
?  My suspicion comes since  mp_cmpr_alloc  calls  get()  and Jim's=20
example shows that it can recursively burrow further than we'd like.

It is still taking me forever to understand all the code/changes...

Cheers,
Lachlan

On Monday 10 March 2003 17:57, Neal Richter wrote:

> I did however look at over snapshots of BDB
> (http://www.sleepycat.com/download/patchlogs.shtml)

[htdig-dev] new htdig mirror

[FTPAdmin @ R. I. B. <ftp...@ia...>]

 Hy,

 I've setup a mirror for the htdig files (not the web site) on our ftp
server. The mirror details are:

	institution: 	Romanian Education Network, Iasi Branch
	country:	Romania
	city:		Iasi
	ftp access:     ftp://ftp.iasi.roedu.net/pub/mirrors/www.htdig.org/
	http access:	http://ftp.iasi.roedu.net/mirrors/www.htdig.org
	bandwidth:	34Mbps
	updated:	daily
	contact:	ftp...@ia...



 If you think our mirror is ok, then please add our mirror to you list.
Thanks.

Re: [htdig-dev] Re: Residual database errors

[Geoff H. <ghu...@ws...>]

On Tuesday, March 11, 2003, at 07:25  AM, Lachlan Andrew wrote:

> The new releases of BDB don't seem to have  mp_cmpr.c  at all.  Is
> that an add-on from another project?  Mifluz?

While Loic offered lots of the improvements and bug-fixes to the 
Sleepycat folks, I don't think they took any. In particular, they felt 
the database compression feature didn't make sense because disk space 
is cheap.

In any case, the core Berkeley DB code gets hammered pretty hard since 
it's used in lots of places.

-Geoff

[htdig-dev] Re: Residual database errors

[Lachlan A. <lh...@us...>]

Thanks for your email.

I think Jim's infinite recursion problem was fairly separate from the=20
database corruption I was having.  The recursion is fairly simple.=20
(1) The allocation routine selects a dirty cache page to flush
(2) When the page is written, it is "compressed" on the fly, but it is=20
actually slightly expanded and needs to be stored in two pages
(3) When the compression routine tries to allocate a new page, it=20
recursively calls the same allocation routine, and the same dirty=20
cache page is selected...

Ideally it would be nice to fix  mp_cmpr.c  so that the "weakcmpr"=20
page was allocated by a completely different mechanism (on the=20
stack?), but that has the potential to introduce lots more bugs.

The new releases of BDB don't seem to have  mp_cmpr.c  at all.  Is=20
that an add-on from another project?  Mifluz?

Cheers,
Lachlan

On Monday 10 March 2003 17:57, Neal Richter wrote:

> Interesting patch.. I need to read more code around it.
>
> My feeling at this point is that the bug is caused by a problem in
> the mp_cmpr.c code

Re: [htdig-dev] HTDig updates...is this group stale?

[Geoff H. <ghu...@ws...>]

> Is the HTDig development team still active?  Last
> update on the htdig.org site is dated Feb, 2002.

So? Just because we don't have mammoth news doesn't mean the website 
isn't updated, the lists are dead, or nothing's happening in the CVS. 
Heck, if you check the htdig-3.2.0b4 snapshots, you'd see that there's 
still quite a bit going on.

> Anybody there?  Has everyone left to work on another
> search engine product?

Nope. We're trying to hammer out one last showstopper before releasing 
what will be 3.2.0b5.

Still going strong. Want to join us?
Cheers,
-Geoff

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

[htdig-dev] HTDig updates...is this group stale?

[Lorenso, D. <dan...@at...>]

Is the HTDig development team still active?  Last
update on the htdig.org site is dated Feb, 2002.
Anybody there?  Has everyone left to work on another
search engine product?

Dante

[htdig-dev] automake problems installing from CVS

[Andrew D. <an...@da...>]

OK, so this is my first time trying to install something from CVS instead
of a tarball. But I don't think it's all "finger trouble"..

RedHat 7.3 system, with (generally) all official RedHat updates.
make-3.79.1-8
gcc-c++-2.96-113
libstdc++-2.96-113
automake-1.4p5-4
autoconf-2.13-17


First I did
cvs -d:pserver:ano...@cv...:/cvsroot/htdig login

cvs  -d:pserver:ano...@cv...:/cvsroot/htdig \
  checkout htdig

I get a lot of
 cvs [login aborted]: recv() from server
   cvs.sourceforge.net: Connection reset by peer
 cvs [update aborted]: recv() from server cvs.sourceforge.net: EOF
but eventually it worked.


htdoc/install.html says if I am going to change sources I can do
"automake".

It says:
configure.in: 1283: required file `./_AM_File,.in' not found
configure.in: 1283: required file `./[$2],.in' not found
configure.in: 1283: required file `./[$3])].in' not found

If I subsequently run ./configure, a subsequent make
fails with
Makefile:200: warning: overriding commands for target `_AM_File'
Makefile:190: warning: ignoring old commands for target `_AM_File'
Makefile:219: warning: overriding commands for target `[]'
Makefile:209: warning: ignoring old commands for target `[]'
make: *** No rule to make target `_AM_File,', needed by
`all-recursive-am'.  Stop.


Is this a bug, or do I need a newer version of Automake ?


-- 
Andrew Daviel

Re: [htdig-dev] Re: Residual database errors

[Neal R. <ne...@ri...>]

I tried using "make check" on a RedHat 8.0 machine.. no joy.  There seems
to be fundamental difference between
htdig-3.2.0b4-20030302/test/conf/httpd.conf wants and what RedHat 8.0
provides.

I'll try it on a different machine at work tommorow.

Interesting patch.. I need to read more code around it.

I did however look at over snapshots of BDB
(http://www.sleepycat.com/download/patchlogs.shtml)

There are a large number of changes to mp_alloc.c from
3.0.55 to 3.1.14, but that all seem to be superficial changes (function
names etc).

The diff from 3.0.55 to 3.3.11 is more interesting.. but nothing along the
lines of your patch that I saw...

My feeling at this point is that the bug is caused by a problem in the
mp_cmpr.c code, and since you've found a nice small test case it will
hopefully be easier to fix..

I'll report back tommorow.

Thanks

Neal

On Sun, 9 Mar 2003, Lachlan Andrew wrote:

> Greetings Jim,
>
> Attached is a hack which explicitly stops the recursion in OS X.  Does
> it work?  (Neal, would it be better in one of the other functions in
> the loop?)
>
> I don't know why a different OS should crash in a different place.
> Does OS X support  pread?  Type  man pread.
>
> Are you having any luck with the other errors in  'make check'?
>
> Thanks!
> Lachlan
>
> On Friday 28 February 2003 11:23, Jim Cole wrote:
> > Hi - Just a follow up on the issue of zlib version.  I installed
> > the 1.1.4 version of zlib available via Fink and rebuilt
> > everything. Using the newer version of zlib, I encounter the same
> > problem (i.e. a segfault from htdig with a very deep stack trace).
> > I did perform a distclean and verified the use of the 1.1.4 version
> > libz via otool.
> >
> > Jim
> >
> > On Thursday, February 27, 2003, at 04:41 AM, Lachlan Andrew wrote:
> > > On Thursday 27 February 2003 11:16, Jim Cole wrote:
> > >> The backtrace is attached. The problem does not occur if I turn
> > >> off compression.
> > >
> > > Thanks.  My guess is that (part of) the reason for the very deep
> > > recursion is that it's trying to allocate a block of  len=8247
> > > bytes, when the page size is only 8192:
> > > #3244 0x00070958 in CDB___memp_alloc (dbmp=0xa98c30,
> > > memreg=0xa99f60, mfp=0xc84e98, len=8247, offsetp=0x0,
> > > retp=0xbfffd900) at mp_alloc.c:88
> > >
> > > I used to get the error
> > >     Unable to allocate %lu bytes from mpool shared region
> > > at some stage too, which is generated inside  CDB___memp_alloc.
> > > From memory, that was when I was using 1.1.3.
> > >
> > > If that is really the problem, it can be fixed by testing
> > > explicitly whether  len>pagesize  (if the pagesize is available
> > > somewhere...).
> >
> > -------------------------------------------------------
> > This sf.net email is sponsored by:ThinkGeek
> > Welcome to geek heaven.
> > http://thinkgeek.com/sf
> > _______________________________________________
> > htdig-dev mailing list
> > htd...@li...
> > https://lists.sourceforge.net/lists/listinfo/htdig-dev
>

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485