htdig-dev — Developer Discussion for the ht://Dig project

[htdig-dev] Strange behaviour in gdb / xxdgb

[Lachlan A. <lh...@us...>]

Greetings all,

In trying to track down the problems using Sun's  cc,  I've run into=20
something new:  When I run
  VERBOSE=3D-vv make TESTS=3Dt_wordlist check
it reports an error: page 1 doesn't exist.  Fine.  When I run it once=20
within  gdb (in xxgdb)  it does the same thing.  Fine.  When I run it=20
a *second* time within the same session of  gdb  it repeatably passes=20
that point with no problems (and crashes later, but that's another=20
story...).

Any ideas?

For what it's worth, this problem occurs both with and *without*=20
compression.

Cheers,
Lachlan

[htdig-dev] Current Status as of snapshot 3.2.0b4-20030525

[Geoff H. <ghu...@us...>]

STATUS of ht://Dig branch 3-2-x

CHECKLIST FOR 3.2.0b5:
* Add more items to checklist :-)
* Must be able to
	(a) make check  and
	(b) index  www.htdig.org  using  "robotstxt_name: master-htdig"
  on all systems listed as "supported".
  Systems tested so far:
    - Mandrake 8.2, gcc 3.2 (lha, 21 May)
    - FreeBSD 4.6, gcc 2.95.3 (lha, 23 May)
    - Debian, Linux kernel 2.2.19, gcc 2.95.4 (lha, 23 May)
  Partly tested:
    - SunOS 5.8 = Solaris 2.8, gcc 3.1 (lha.  Makes check.  Big dig not tested)
    - SunOS 5.8 = Solaris 2.8, Sun cc with g++ 3.1 (lha. Compiles, fails check)
  To be tested:
    - RedHat anyone?
    - OS X anyone?
* Latin-encodings (Gilles)
* Check bugs listed in bug-tracker...
* Polish release docs (Geoff)





RELEASES:
   3.2.0b5: Next release, June 2003???
   3.2.0b4: "In progress" -- snapshots called "3.2.0b4" until prerelease.
   3.2.0b3: Released:  22 Feb 2001.
   3.2.0b2: Released:  11 Apr 2000.
   3.2.0b1: Released:   4 Feb 2000.

(Please note that everything added here should have a tracker PR# so
we can be sure they're fixed. Geoff is currently trying to add PR#s for
what's currently here.)

SHOWSTOPPERS:
* Mifluz database errors are a severe problem (PR#428295)
-- Does Neal's new zlib patch solve this for now?

KNOWN BUGS:
* Odd behavior with $(MODIFIED) and scores not working with
   wordlist_compress set but work fine without wordlist_compress.
   (the date is definitely stored correctly, even with compression on
    so this must be some sort of weird htsearch bug) PR#618737.
* META descriptions are somehow added to the database as FLAG_TITLE,
   not FLAG_DESCRIPTION. (PR#618738)
   Can anyone reproduce this?  I can't! -- Lachlan

PENDING PATCHES (available but need work):
* Additional support for Win32.
* Memory improvements to htmerge. (Backed out b/c htword API changed.)
* Mifluz merge.

NEEDED FEATURES:
* Quim's new htsearch/qtest query parser framework.
* File/Database locking. PR#405764.

TESTING:
* httools programs: 
  (htload a test file, check a few characteristics, htdump and compare)
* Tests for new config file parser
* Duplicate document detection while indexing
* Major revisions to ExternalParser.cc, including fork/exec instead of popen,
  argument handling for parser/converter, allowing binary output from an
  external converter.
* ExternalTransport needs testing of changes similar to ExternalParser.

DOCUMENTATION:
* List of supported platforms/compilers is ancient. (PR#405279)
* Document all of htsearch's mappings of input parameters to config attributes
   to template variables. (Relates to PR#405278.)
  Should we make sure these config attributes are all documented in
  defaults.cc, even if they're only set by input parameters and never
  in the config file?
* Split attrs.html into categories for faster loading.
* Turn defaults.cc into an XML file for generating documentation and 
  defaults.cc.
* require.html is not updated to list new features and disk space
   requirements of 3.2.x (e.g. regex matching, database compression.)
   PRs# 405280 #405281.
* TODO.html has not been updated for current TODO list and
   completions.
   I've tried.  Someone "official" please check and remove this -- Lachlan
* Htfuzzy could use more documentation on what each fuzzy algorithm
   does. PR#405714.
* Document the list of all installed files and default
   locations. PR#405715.

OTHER ISSUES:
* Can htsearch actually search while an index is being created?
* The code needs a security audit, esp. htsearch. PR#405765.

[htdig-dev] Re: Release schedule for 3.2.0b5

[Lachlan A. <lh...@us...>]

Greetings all,

A brief update on the testing:

I've fixed the  configure  process for FreeBSD 4.6, Mandrake 8.2,=20
Debian ?.? and SunOS 5.8 with gcc.  It is still quite broken for=20
Sun's compiler.  (It finally compiles, but fails most tests, and=20
hangs during t_htdig.)

What other platforms need testing?  RedHat and OS X are definites.
(RedHat on the compile farm doesn't seem to have Apache, which is=20
needed for  make check.  Also, I don't want to use it for big digs.)
Does anyone have access to Cygwin?  SuSE?  Non-x86 Linux?

I've incorporated a patch, based on Richard Monroe's excellent sketch,=20
to use -Wno-deprecated  when needed.

Jesse, Sun's  cc  also had problems with  select,  but in the main=20
directory so it's probably just a coincidence.

Finally, I still haven't got  autoconf  working, so most of the=20
changes I made manually at "all levels" (configure, configure.in,=20
aclocal.m4, acinclude.m4, etc.).  Before the 3.2.0b5 is released,=20
we'll have to make sure the automatic process works...

Cheers,
Lachlan

On Sun, 18 May 2003 00:32, J.E.J. op den Brouw wrote:

> There is a ./configure issue on HP-UX 10.20 about
> select(2) not found in de db tree.

Re: [htdig-dev] Re: Release schedule for 3.2.0b5?

[Gilles D. <gr...@sc...>]

According to Geoff Hutchison:
> On Sunday, May 18, 2003, at 12:15  AM, Lachlan Andrew wrote:
> > Thanks for your prompt reply, Geoff, and I apologise for sounding
> > impatient (yet again ):
> 
> I think we all want to see 3.2.0b5 out ASAP. :-)

I agree.  I know you've been looking to Geoff and me to provide some
leadership here, and I have to apologize that I haven't been in a
position lately to provide much.  However, despite being a bit out
of the loop lately, it does sound from the discussions here that the
cvs tree is in a better state than any prior beta release of 3.2.  So,
whether the remaining problems of occasional database corruption (still
not 100% sure that's licked, are we?) and OS X portability issues, are
showstoppers or not really becomes a judgement call, and I'd rather defer
to the judgement of those actively pursuing those particular problems.
When you guys feel the code is ready to release, I say go for it.

In the meantime, I'll try to find a bit of time this week or next to get
in the two fixes I promised to handle a long time ago (translate_latin1
attribute and htdig -m handling).  The latter is maybe dispensible, but
I think translate_latin1 is a must for our eastern European friends
(and possibly others who use 8-bit, non-latin1 character encodings),
as the current behaviour forces latin1 translations with no option to
turn it off.

> >  (d) chmod 755 t_url
> 
> Because the CVS tree at SourceForge horribly mangled a lot of 
> permissions, I wrote a "htdig-fix.sh" script that needs to be run 
> before a genuine release. I'll make sure that t_url is set 
> appropriately.
> 
> (I'm actually wondering at some point if we shouldn't just change 
> directory names in the SF CVS since the problem seems to be fixed with 
> another project I have there.)

If you can resolve the mode bit corruption, more power to you, Geoff!
If not, we may want to make the Makefiles and scripts somehow less
dependent on file modes in the source tree, and make sure modes are set
correctly in a make install.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] Autoconf question

[Lachlan A. <lh...@us...>]

Greetings Gabriele,

I wasn't running any of the other tools.  I just thought that, being=20
the last step in the process you recommended, running  autoreconf  by=20
itself should give the same files back.  When I run it twice in a=20
row, it gives the the same  configure  file each time, but that file=20
is different from the one in CVS.  Does that mean you didn't run =20
autoreconf  when you updated the files three months ago?

Lachlan

On Mon, 19 May 2003 21:44, Gabriele Bartolini wrote:
> Il lun, 2003-05-19 alle 12:41, Lachlan Andrew ha scritto:
> > Am I right in thinking that running
> >     autoreconf
> > twice in the last step should make no difference?  When I run
> > autoreconf (from Autoconf 2.57) the  configure  file is very
> > different from the one in CVS.
>
> However, are u sure you did not change the config.sub and
> config.guess files? What about aclocal.m4?
>
> So ... try to run the other tools separately. Which versions are
> you using of autoconf (2.57, right?), automake and libtool?

Re: [htdig-dev] Autoconf question

[Gabriele B. <g.b...@co...>]

Ciao Lachlan!

Il lun, 2003-05-19 alle 12:41, Lachlan Andrew ha scritto:
> Thanks for your help with the  auto*  process, Gabriele.

No worries. :-P

> Am I right in thinking that running
>     autoreconf
> twice in the last step should make no difference?  When I run  
> autoreconf (from Autoconf 2.57) the  configure  file is very 
> different from the one in CVS.  What's more, the one I generate 
> doesn't work :(  (I think there is a bug in the g++ compatability 
> library which it trips up on.)
> 
> Any ideas?

I have always tried to avoid to autoreconf tool, as I don't trust things
that do things in 'behalf of you'.

However, are u sure you did not change the config.sub and config.guess
files? What about aclocal.m4?

Also, autoreconf recalls autoheader, which can be pretty dangerous IMHO.

So ... try to run the other tools separately. Which versions are you
using of autoconf (2.57, right?), automake and libtool?

-Gabriele
-- 
Gabriele Bartolini - Web Programmer
Comune di Prato - Prato - Tuscany - Italy
g.b...@co... | http://www.comune.prato.it
> find bin/laden -name osama -exec rm {} ;

[htdig-dev] Autoconf question

[Lachlan A. <lh...@us...>]

Thanks for your help with the  auto*  process, Gabriele.

Am I right in thinking that running
    autoreconf
twice in the last step should make no difference?  When I run =20
autoreconf (from Autoconf 2.57) the  configure  file is very=20
different from the one in CVS.  What's more, the one I generate=20
doesn't work :(  (I think there is a bug in the g++ compatability=20
library which it trips up on.)

Any ideas?

Thanks,
Lachlan

On Fri, 2 May 2003 05:55, Gabriele Bartolini wrote:
> I don't think you need to run autoconf or automake either. However,
> in general, here are the steps I usually follow with ht://Check and
> - every death of pope (as we say in Italy - that is to say very
> rarely :-D ) - with ht://Dig.

> Step 5
          autoreconf

Re: [htdig-dev] Re: Release schedule for 3.2.0b5?

[Ted Stresen-R. <ted...@ma...>]

Being fortunate enough to know someone who actually works at Apple, I 
asked if he, or someone he knew, could take a look at this particular 
problem. He was too busy (not surprising) but he did point me to these 
two mailing lists. If someone has the time, perhaps he/she can 
summarize the problems we're having and post to one of these lists 
requesting help.

1) http://lists.apple.com/mailman/listinfo/darwin-development
2) http://lists.apple.com/mailman/listinfo/unix-porting (This list 
would be my second choice)

To quote my connection:
"I strongly urge you the developer who is actually having the build 
problem to send the appropriate details to this list.  Frankly, that 
kind of problem sounds very familiar, perhaps they should search the 
archives.  It definitely sounds like an environmental issue (and or 
build settings issue)."

He also suggested contacting the people on the Fink project, that they 
might have some hints/tips.

Ted Stresen-Reuter


On Sunday, May 18, 2003, at 02:14  PM, Jim Cole wrote:

> On Saturday, May 17, 2003, at 11:15 PM, Lachlan Andrew wrote:
>
>> I have managed to run  make check  on OS X, after a bit of manual
>> configure/making.  In case it helps:
>>  (a) libhtword.a  must be linked *after*  libhtdb.a  and/or  libht.a
>>      so when  make  failed with  WordType::instance  undefined, I
>>      re-ran the g++ command it had just issued, putting the full path
>>      to  libhtword.a  at the end.
>
> Seems to work for me also. Looks like libhtword.a needs to be linked 
> after libht.a. I tried placing libhtdb.a before and after and it 
> didn't make any difference.
>
> Jim
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: If flattening out C++ or Java
> code to make your application fit in a relational database is painful, 
> don't do it! Check out ObjectStore. Now part of Progress Software.
> http://www.objectstore.net/sourceforge
> _______________________________________________
> htdig-dev mailing list
> htd...@li...
> https://lists.sourceforge.net/lists/listinfo/htdig-dev
>

Re: [htdig-dev] Re: Release schedule for 3.2.0b5?

[Jim C. <li...@yg...>]

On Saturday, May 17, 2003, at 11:15 PM, Lachlan Andrew wrote:

> I have managed to run  make check  on OS X, after a bit of manual
> configure/making.  In case it helps:
>  (a) libhtword.a  must be linked *after*  libhtdb.a  and/or  libht.a
>      so when  make  failed with  WordType::instance  undefined, I
>      re-ran the g++ command it had just issued, putting the full path
>      to  libhtword.a  at the end.

Seems to work for me also. Looks like libhtword.a needs to be linked 
after libht.a. I tried placing libhtdb.a before and after and it didn't 
make any difference.

Jim

[htdig-dev] Re: Release schedule for 3.2.0b5?

[Geoff H. <ghu...@ws...>]

On Sunday, May 18, 2003, at 12:15  AM, Lachlan Andrew wrote:

> Thanks for your prompt reply, Geoff, and I apologise for sounding
> impatient (yet again ):

I think we all want to see 3.2.0b5 out ASAP. :-)

>  (d) chmod 755 t_url

Because the CVS tree at SourceForge horribly mangled a lot of 
permissions, I wrote a "htdig-fix.sh" script that needs to be run 
before a genuine release. I'll make sure that t_url is set 
appropriately.

(I'm actually wondering at some point if we shouldn't just change 
directory names in the SF CVS since the problem seems to be fixed with 
another project I have there.)

> Very good idea.  I can see the Solaris hack in  configure  and
> aclocal.m4,  but I haven't yet found which input file it comes

It's not from an input file. As far as I could tell, the only way to 
get at the libtool configuration was by hacking aclocal.m4. You're not 
supposed to do that, since your changes will disappear. But it needed 
to be done.

-Geoff

[htdig-dev] Current Status as of snapshot 3.2.0b4-20030518

[Geoff H. <ghu...@us...>]

STATUS of ht://Dig branch 3-2-x

RELEASES:
   3.2.0b5: Next release, First quarter 2003???
   3.2.0b4: "In progress" -- snapshots called "3.2.0b4" until prerelease.
   3.2.0b3: Released:  22 Feb 2001.
   3.2.0b2: Released:  11 Apr 2000.
   3.2.0b1: Released:   4 Feb 2000.

(Please note that everything added here should have a tracker PR# so
we can be sure they're fixed. Geoff is currently trying to add PR#s for
what's currently here.)

SHOWSTOPPERS:
* Mifluz database errors are a severe problem (PR#428295)
-- Does Neal's new zlib patch solve this for now?

KNOWN BUGS:
* Odd behavior with $(MODIFIED) and scores not working with
   wordlist_compress set but work fine without wordlist_compress.
   (the date is definitely stored correctly, even with compression on
    so this must be some sort of weird htsearch bug) PR#618737.
* META descriptions are somehow added to the database as FLAG_TITLE,
   not FLAG_DESCRIPTION. (PR#618738)
   Can anyone reproduce this?  I can't! -- Lachlan

PENDING PATCHES (available but need work):
* Additional support for Win32.
* Memory improvements to htmerge. (Backed out b/c htword API changed.)
* Mifluz merge.

NEEDED FEATURES:
* Quim's new htsearch/qtest query parser framework.
* File/Database locking. PR#405764.

TESTING:
* httools programs: 
  (htload a test file, check a few characteristics, htdump and compare)
* Tests for new config file parser
* Duplicate document detection while indexing
* Major revisions to ExternalParser.cc, including fork/exec instead of popen,
  argument handling for parser/converter, allowing binary output from an
  external converter.
* ExternalTransport needs testing of changes similar to ExternalParser.

DOCUMENTATION:
* List of supported platforms/compilers is ancient. (PR#405279)
* Document all of htsearch's mappings of input parameters to config attributes
   to template variables. (Relates to PR#405278.)
  Should we make sure these config attributes are all documented in
  defaults.cc, even if they're only set by input parameters and never
  in the config file?
* Split attrs.html into categories for faster loading.
* Turn defaults.cc into an XML file for generating documentation and 
  defaults.cc.
* require.html is not updated to list new features and disk space
   requirements of 3.2.x (e.g. regex matching, database compression.)
   PRs# 405280 #405281.
* TODO.html has not been updated for current TODO list and
   completions.
   I've tried.  Someone "official" please check and remove this -- Lachlan
* Htfuzzy could use more documentation on what each fuzzy algorithm
   does. PR#405714.
* Document the list of all installed files and default
   locations. PR#405715.

OTHER ISSUES:
* Can htsearch actually search while an index is being created?
* The code needs a security audit, esp. htsearch. PR#405765.

[htdig-dev] Re: Release schedule for 3.2.0b5?

[Lachlan A. <lh...@us...>]

Thanks for your prompt reply, Geoff, and I apologise for sounding=20
impatient (yet again ):

I have managed to run  make check  on OS X, after a bit of manual=20
configure/making.  In case it helps:
 (a) libhtword.a  must be linked *after*  libhtdb.a  and/or  libht.a
     so when  make  failed with  WordType::instance  undefined, I
     re-ran the g++ command it had just issued, putting the full path
     to  libhtword.a  at the end.
     With my infinite-recursion patch,  libhtdb.a  also needs to be
     libhtcommon.a, so I just repeated the full list of libraries.
     Will recreating the  libtool  files fix that?
 (b) The _MODULES_ macro in  test/conf/httpd.conf.in  was empty, so I
     put the path  /usr/libexec/httpd/  at the start of each module
 (c) Some modules weren't found, so I commented out the  LoadModule
     and  AddModule  lines for:
=09agent_log_module
=09referer_log_module
=09db_auth_module
     Are these needed on any platform, or can they be commented out by
     default?  It works without them on  Mandrake 8.2...
 (d) chmod 755 t_url

On Sun, 18 May 2003 03:29, Geoff Hutchison wrote:

> We can probably
> even hack configure to default to static libs on OS X and throw up
> a warning if you use --enable-shared. (I put in such a hack a while
> back when Solaris had problems with shared libs.)

Very good idea.  I can see the Solaris hack in  configure  and =20
aclocal.m4,  but I haven't yet found which input file it comes=20
from...  (Sorry, but I'm still rather bamboozled by the configuration=20
process :)  I'll keep looking.

Thanks again for your help (and patience with my impatience...)!
Lachlan

Re: [htdig-dev] Re: Release schedule for 3.2.0b5?

[J.E.J. op d. B. <ht...@op...>]

Hi all,

There is a ./configure issue on HP-UX 10.20 about
select(2) not found in de db tree. I'm trying to get the
problem fixed, but 'm not sure how long it will take.

----- Original Message ----- 
From: "Lachlan Andrew" <lh...@us...>
To: "Geoff Hutchison" <ghu...@ws...>
Cc: <htd...@li...>
Sent: Saturday, May 17, 2003 7:30 AM
Subject: [htdig-dev] Re: Release schedule for 3.2.0b5?


> Greetings all,
> 
> I've lost track, but my current understanding is:
> 
> 1. The bug causing a database crash doing the *second* dig using -i
>    has been fixed.
> 2. Other than under  OS X,  no other database bugs have been seen.
>    Anybody, ** PLEASE CORRECT ME IF THAT IS WRONG **
> 3. OS X  is a basket case (at least on the compile farm) because
>    libtool fails to link the C++ dynamic libraries (which aren't even
>    there on the compile farm!).
> 
> Geoff, what should we do to test/prove that the "showstopper" is fixed 
> (on all systems on which ht://Dig links)?  Will someone who knows OSX 
> try to fix that issue, or can we release 3.2.0b5 for Linux, FreeBSD, 
> Solaris, ...?
> 
> If there *is* a residual database eror, the sooner we release  3.2.0b5  
> and get a large number of users running it, the sooner we'll be able 
> to find it and get 3.2R out.
> 
> Geoff, Gilles, even if you don't have time for testing/debugging, 
> could I be bold enough to ask for some more hands-on leadership?  I 
> feel lost...
> 
> Thanks!!
> Lachlan

--jesse 

Re: [htdig-dev] Re: Release schedule for 3.2.0b5?

[Jim C. <li...@yg...>]

On Friday, May 16, 2003, at 11:30 PM, Lachlan Andrew wrote:

> 2. Other than under  OS X,  no other database bugs have been seen.
>    Anybody, ** PLEASE CORRECT ME IF THAT IS WRONG **

The patch you provided seemed to correct the last of the consistently 
repeatable database problems under OS X. I ran into one database 
problem after applying that patch but was never able to reproduce it a 
second time.

> Geoff, what should we do to test/prove that the "showstopper" is fixed
> (on all systems on which ht://Dig links)?  Will someone who knows OSX
> try to fix that issue, or can we release 3.2.0b5 for Linux, FreeBSD,
> Solaris, ...?

The problem with building under OS X is limited to dynamic builds. Even 
without correcting that problem, a fully functional install under OS X 
is still possible with a static build. As such, I don't personally see 
this issue as being any sort of show stopper. If it becomes a common 
issue, we can always add a FAQ. There is also still an issue with 'make 
check' under OS X, but again that doesn't seem like it should be a show 
stopper. I will continue to look at the OS X problems as I find time.

Jim

[htdig-dev] Re: Release schedule for 3.2.0b5?

[Lachlan A. <lh...@us...>]

Greetings all,

I've lost track, but my current understanding is:

1. The bug causing a database crash doing the *second* dig using -i
   has been fixed.
2. Other than under  OS X,  no other database bugs have been seen.
   Anybody, ** PLEASE CORRECT ME IF THAT IS WRONG **
3. OS X  is a basket case (at least on the compile farm) because
   libtool fails to link the C++ dynamic libraries (which aren't even
   there on the compile farm!).

Geoff, what should we do to test/prove that the "showstopper" is fixed=20
(on all systems on which ht://Dig links)?  Will someone who knows OSX=20
try to fix that issue, or can we release 3.2.0b5 for Linux, FreeBSD,=20
Solaris, ...?

If there *is* a residual database eror, the sooner we release  3.2.0b5 =20
and get a large number of users running it, the sooner we'll be able=20
to find it and get 3.2R out.

Geoff, Gilles, even if you don't have time for testing/debugging,=20
could I be bold enough to ask for some more hands-on leadership?  I=20
feel lost...

Thanks!!
Lachlan

On Sat, 22 Feb 2003 16:19, Geoff Hutchison wrote:
> >> SHOWSTOPPER:
> >> * Still need thorough testing of the database, with Neal's zlib
> >> patch Any ideas how we can test this thoroughly?
>
> As far as a large-scale test, I'd be glad to tar up the htdig.org
> website including mail archives. >:-)
>
> >> Fri 28 Feb: Code freeze.  Testing and documenting only.
> >> =09    Update version number throughout.
> >> Fri  7 Mar: Release.
>
> This may be ambitious. I've never knowingly sent out a release that
> had a database bug. So we obviously want to kill that SHOWSTOPPER
> soon.

[htdig-dev] ht:/Dig: files and htdig-patches mirror

[<j.s...@at...>]

Hi,

I'm pleased to announce that in addition to our mirror setup for ht:/Dig web 
site found at http://mirrors.atn.ro/htdig/maindocs/
we have set-up the mirrors for the Files and Patch Archive.
The HTTP address for these mirrors are:
http://mirrors.atn.ro/htdig/files/
http://mirrors.atn.ro/htdig/htdig-patches/

The rest of the Mirror information is unchanged:
Country: Romania
Location: Oradea
Provider: Atlas Telecom Network Romania <http://www.atlastelecom.ro>
Contact : Mirror Admin <mailto:mi...@at...>


Best Regards,
Jozsef Szilagyi
Atlas Telecom Network Romania

Re: [htdig-dev] Re: Undefined symbols in OS X

[Jim C. <li...@yg...>]

On Monday, May 12, 2003, at 12:40 PM, Lachlan Andrew wrote:

> Unfortunately,  locate libstdc++  on the SourceForge compile farm only
> finds  .a  files:
>
>     lha@ucf-cf-ppc-macosx-2:~/htdig$ locate libstdc++
>     /usr/lib/gcc/darwin/2.95.2/libstdc++.a
>     /usr/lib/gcc/darwin/3.1/libstdc++.a
>     /usr/lib/libstdc++.a
>     lha@ucf-cf-ppc-macosx-2:~/htdig$
>
> That would certainly explain why it doesn't find the symbols!  Where
> is the file on your system, Jim?  Does anyone know why it isn't on
> the compile farm??

On my system it is located in /usr/lib/gcc/darwin/3.1/private/A. Also, 
there is a dynamic link in /usr/lib/gcc/darwin/3.1/private/.

   /usr/lib/gcc/darwin/2.95.2/libstdc++.a
   /usr/lib/gcc/darwin/3.1/libstdc++.a
   /usr/lib/gcc/darwin/3.1/private/A/libstdc++.dylib
   /usr/lib/gcc/darwin/3.1/private/libstdc++.dylib
   /usr/lib/libstdc++.a

This may be a relatively recent addition as I don't recall the .dylib 
files being present at this location the last time I dug into the 
problem of unresolved symbols. But it has been a while, so I might be 
mistaken. Then again, if they are not present on the compile farm... 
Perhaps there is a difference in OS X version or installed Developer 
Tool updates. I am using 10.2 (Jaguar) and the Developer Tools that 
came with that release, including the October and December updates.

[magni:~] greyleaf% uname -a
Darwin magni.yggdrasill.net 6.6 Darwin Kernel Version 6.6: Thu May  1 
21:48:54 PDT 2003; root:xnu/xnu-344.34.obj~1/RELEASE_PPC  Power 
Macintosh powerpc

[magni:~] greyleaf% gcc --version
gcc (GCC) 3.1 20020420 (prerelease)
Copyright (C) 2002 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is 
NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR 
PURPOSE.


Jim

[htdig-dev] Re: Undefined symbols in OS X

[Lachlan A. <lh...@us...>]

Good work, Jim!

Unfortunately,  locate libstdc++  on the SourceForge compile farm only=20
finds  .a  files:

    lha@ucf-cf-ppc-macosx-2:~/htdig$ locate libstdc++
    /usr/lib/gcc/darwin/2.95.2/libstdc++.a
    /usr/lib/gcc/darwin/3.1/libstdc++.a
    /usr/lib/libstdc++.a
    lha@ucf-cf-ppc-macosx-2:~/htdig$

That would certainly explain why it doesn't find the symbols!  Where=20
is the file on your system, Jim?  Does anyone know why it isn't on=20
the compile farm??

Cheers,
Lachlan

On Mon, 5 May 2003 12:43, Jim Cole wrote:
> I have more information regarding the C++ related
> undefined symbols. The symbol names match
> those in libstdc++. So it looks like it is an issue
> of things not being found, rather than anything that is truly
> missing.
>
> Additionally, I discovered that setting DYLD_INSERT_LIBRARIES
> to the full path to libstdc++.dylib seems to solve the problem.

[htdig-dev] Current Status as of snapshot 3.2.0b4-20030511

[Geoff H. <ghu...@us...>]

STATUS of ht://Dig branch 3-2-x

RELEASES:
   3.2.0b5: Next release, First quarter 2003???
   3.2.0b4: "In progress" -- snapshots called "3.2.0b4" until prerelease.
   3.2.0b3: Released:  22 Feb 2001.
   3.2.0b2: Released:  11 Apr 2000.
   3.2.0b1: Released:   4 Feb 2000.

(Please note that everything added here should have a tracker PR# so
we can be sure they're fixed. Geoff is currently trying to add PR#s for
what's currently here.)

SHOWSTOPPERS:
* Mifluz database errors are a severe problem (PR#428295)
-- Does Neal's new zlib patch solve this for now?

KNOWN BUGS:
* Odd behavior with $(MODIFIED) and scores not working with
   wordlist_compress set but work fine without wordlist_compress.
   (the date is definitely stored correctly, even with compression on
    so this must be some sort of weird htsearch bug) PR#618737.
* META descriptions are somehow added to the database as FLAG_TITLE,
   not FLAG_DESCRIPTION. (PR#618738)
   Can anyone reproduce this?  I can't! -- Lachlan

PENDING PATCHES (available but need work):
* Additional support for Win32.
* Memory improvements to htmerge. (Backed out b/c htword API changed.)
* Mifluz merge.

NEEDED FEATURES:
* Quim's new htsearch/qtest query parser framework.
* File/Database locking. PR#405764.

TESTING:
* httools programs: 
  (htload a test file, check a few characteristics, htdump and compare)
* Tests for new config file parser
* Duplicate document detection while indexing
* Major revisions to ExternalParser.cc, including fork/exec instead of popen,
  argument handling for parser/converter, allowing binary output from an
  external converter.
* ExternalTransport needs testing of changes similar to ExternalParser.

DOCUMENTATION:
* List of supported platforms/compilers is ancient. (PR#405279)
* Document all of htsearch's mappings of input parameters to config attributes
   to template variables. (Relates to PR#405278.)
  Should we make sure these config attributes are all documented in
  defaults.cc, even if they're only set by input parameters and never
  in the config file?
* Split attrs.html into categories for faster loading.
* Turn defaults.cc into an XML file for generating documentation and 
  defaults.cc.
* require.html is not updated to list new features and disk space
   requirements of 3.2.x (e.g. regex matching, database compression.)
   PRs# 405280 #405281.
* TODO.html has not been updated for current TODO list and
   completions.
   I've tried.  Someone "official" please check and remove this -- Lachlan
* Htfuzzy could use more documentation on what each fuzzy algorithm
   does. PR#405714.
* Document the list of all installed files and default
   locations. PR#405715.

OTHER ISSUES:
* Can htsearch actually search while an index is being created?
* The code needs a security audit, esp. htsearch. PR#405765.

Re: [htdig-dev] Re: New patch [was: new release of htdig]

[Jim C. <li...@yg...>]

Hi - With wordlist_page_size set to 32768 I was unable to reproduce the 
problem. However I also failed to reproduce the problem when I again 
ran without the setting this morning. I am going to start another run 
without the attribute tonight. As for platform, I am currently working 
with OS X running on a dual G4. The OS X version is 10.2.5.

Darwin magni 6.5 Darwin Kernel Version 6.5: Mon Apr  7 17:05:38 PDT 
2003; root:xnu/xnu-344.32.obj~1/RELEASE_PPC  Power Macintosh powerpc

Jim

On Monday, May 5, 2003, at 10:08 AM, Neal Richter wrote:

>
> Jim,
> 	I'm ultimately responsible for this bug, as I re-enabled the zlib
> WordDB compression feature... I'd like to get it it fixed.. but I have
> benn totally unable to duplicate it!
>
> Please try adding this line to your .conf file and re-run your index.
>
> wordlist_page_size:    32768
>
> Also please include complete information on your platform.. Linux 
> distro
> version, CPU type..
>
> Thanks.
>
>
> On Sun, 4 May 2003, Jim Cole wrote:
>
>> Hi - I am still having problems with this patch applied. The dig makes
>> it much farther than it did last time around, but compression still
>> appears to be an issue. I receive the following a few times toward the
>> end of the dig.
>>
>> WordDB: CDB___memp_cmpr_read: unable to uncompress page at pgno = 
>> 246532
>> WordDB: PANIC: Input/output error
>> WordDB: CDB___memp_cmpr_read: unable to uncompress page at pgno = 
>> 246532
>> WordDB: PANIC: Input/output error
>>
>> Then at the end of the dig, I see the following.
>>
>> WordDB: CDB___memp_cmpr_read: unable to uncompress page at pgno = 
>> 246532
>> WordDB: PANIC: Input/output error
>> WordDBCursor::Get(17) failed DB_RUNRECOVERY: Fatal error, run database
>> recovery
>>
>> The referenced page number is the same for every error message. For
>> this test, the default value was used for wordlist_page_size. Both
>> wordlist_compress and wordlist_compress_zlib were set to true, and
>> compression_level was set to 8. I used a fresh copy of the CVS code 
>> (as
>> of yesterday) with dbase.patch2 applied.
>>
>> Jim
>>
>> On Sunday, April 27, 2003, at 02:27 AM, Lachlan Andrew wrote:
>>
>>> OK, here is another patch...
>>>
>>> The problem was that I was using the counts of clean and dirty cache
>>> pages, but they were not recorded correctly in the old BDB 2.x
>>> (although that didn't show up in my initial tests...).  This patch
>>> contains the fix, copied from BDB 3.3.11.
>>>
>>> Thanks again for your help :)
>>>
>>> Lachlan
>>>
>>> On Sun, 27 Apr 2003 12:54, Jim Cole wrote:
>>>> I am still running into fatal problems with OS X. I no longer get
>>>> the segfault, but instead see the output shown below.
>>> <dbase.patch2>
>>
>>
>>
>> -------------------------------------------------------
>> This sf.net email is sponsored by:ThinkGeek
>> Welcome to geek heaven.
>> http://thinkgeek.com/sf
>> _______________________________________________
>> htdig-dev mailing list
>> htd...@li...
>> https://lists.sourceforge.net/lists/listinfo/htdig-dev
>>
>
> Neal Richter
> Knowledgebase Developer
> RightNow Technologies, Inc.
> Customer Service for Every Web Site
> Office: 406-522-1485
>
>

Re: [htdig-dev] Re: New patch [was: new release of htdig]

[Neal R. <ne...@ri...>]

Jim,
	I'm ultimately responsible for this bug, as I re-enabled the zlib
WordDB compression feature... I'd like to get it it fixed.. but I have
benn totally unable to duplicate it!

Please try adding this line to your .conf file and re-run your index.

wordlist_page_size:    32768

Also please include complete information on your platform.. Linux distro
version, CPU type..

Thanks.


On Sun, 4 May 2003, Jim Cole wrote:

> Hi - I am still having problems with this patch applied. The dig makes
> it much farther than it did last time around, but compression still
> appears to be an issue. I receive the following a few times toward the
> end of the dig.
>
> WordDB: CDB___memp_cmpr_read: unable to uncompress page at pgno = 246532
> WordDB: PANIC: Input/output error
> WordDB: CDB___memp_cmpr_read: unable to uncompress page at pgno = 246532
> WordDB: PANIC: Input/output error
>
> Then at the end of the dig, I see the following.
>
> WordDB: CDB___memp_cmpr_read: unable to uncompress page at pgno = 246532
> WordDB: PANIC: Input/output error
> WordDBCursor::Get(17) failed DB_RUNRECOVERY: Fatal error, run database
> recovery
>
> The referenced page number is the same for every error message. For
> this test, the default value was used for wordlist_page_size. Both
> wordlist_compress and wordlist_compress_zlib were set to true, and
> compression_level was set to 8. I used a fresh copy of the CVS code (as
> of yesterday) with dbase.patch2 applied.
>
> Jim
>
> On Sunday, April 27, 2003, at 02:27 AM, Lachlan Andrew wrote:
>
> > OK, here is another patch...
> >
> > The problem was that I was using the counts of clean and dirty cache
> > pages, but they were not recorded correctly in the old BDB 2.x
> > (although that didn't show up in my initial tests...).  This patch
> > contains the fix, copied from BDB 3.3.11.
> >
> > Thanks again for your help :)
> >
> > Lachlan
> >
> > On Sun, 27 Apr 2003 12:54, Jim Cole wrote:
> >> I am still running into fatal problems with OS X. I no longer get
> >> the segfault, but instead see the output shown below.
> > <dbase.patch2>
>
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> htdig-dev mailing list
> htd...@li...
> https://lists.sourceforge.net/lists/listinfo/htdig-dev
>

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

[htdig-dev] Regex patch for 3.1.6 - ready

[Gabriele B. <g.b...@co...>]

Hi guys,

   I got to make it work today ... If some of you could please put in
the patch archive ... :-)

   Basically it is possibile to specify a regular expression in the
restrict and exclude attribute, as:

restrict: [^http://www.comune.prato.it/\$]

Ciao ciao
-Gabriele
-- 
Gabriele Bartolini - Web Programmer
Comune di Prato - Prato - Tuscany - Italy
g.b...@co... | http://www.comune.prato.it
> find bin/laden -name osama -exec rm {} ;

[htdig-dev] Undefined symbols in OS X

[Jim C. <li...@yg...>]

Hi - I have a bit more information regarding the C++ related undefined 
symbols. The symbol names that show up as undefined match exactly those 
in OS X's libstdc++. So it looks like it is an issue of things not 
being found, rather than anything that is truly missing.

Additionally, I discovered that setting the DYLD_INSERT_LIBRARIES 
environment variable to the full path to libstdc++.dylib before running 
an ht://Dig program seems to solve the problem of undefined symbols. 
This variable is used to specify dynamic libraries that should be 
loaded before any specified directly by a program. I think it is 
primarily intended for testing, but it does seem to identify the nature 
of the problem. Apparently the current build system somehow fails to 
record a libstdc++.dylib dependency, resulting the library not being 
loaded at runtime.

All of the above holds regardless of the version of libtool used. Or at 
least for the current version and 1.5.

Jim

Re: [htdig-dev] Re: libtool... (Was: New patch - feedback)

[Jim C. <li...@yg...>]

On Wednesday, April 30, 2003, at 03:45 PM, Lachlan Andrew wrote:

> Looking through the ChangeLog for libtool 1.5 (Released 14 April):
>
>  2003-03-21  Peter O'Gorman  <pe...@po...>
>  * libtool.m4 (darwin): Check compiler is apple gcc, add
> -single_module support so that dyloading c++ shared libraries will
> work.
>
> That sounds like just the patch we need!  If someone tells me the
> steps (autoconf? automake? ???), I'll download and run 1.5 on the
> weekend.

I tried using version 1.5 this afternoon with no joy. I am still 
getting undefined symbols for everything that is supposed to be coming 
from the C++ library. However I am not sure that I really know what I 
am doing ;) I built the standard GNU Libtool 1.5 distribution and ran 
libtoolize --copy --force in the top level ht://Dig directory. I tested 
with and without rerunning aclocal, which libtoolize implied might be 
necessary. Neither approach helped.

If anyone has suggestions on other thing to try in addition to simply 
updating libtool, I would be happy to give them a try.

Jim

Re: [htdig-dev] Re: New patch [was: new release of htdig]

[Ted Stresen-R. <ted...@ma...>]

Hi,

Might this behavior have something to do with the size of the site you 
are indexing? I am indexing four sites on my local machine (not very 
much to index, in truth) and it occurred to me that perhaps, in order 
to trigger the bug, you have to index a larger amount of information.

I'll give it a whirl indexing Yahoo! to see what happens...

just kidding... but I will try another site that is larger than what I 
have on my hard drive to see if that makes a difference. Also, if you 
have a chance, could post the config file you are using so we are all 
using the same attribute values?

Ted Stresen-Reuter
http://www.tedmasterweb.com/

On Sunday, May 4, 2003, at 01:31  PM, Jim Cole wrote:

> Hi - I am still having problems with this patch applied. The dig makes 
> it much farther than it did last time around, but compression still 
> appears to be an issue. I receive the following a few times toward the 
> end of the dig.
>
> WordDB: CDB___memp_cmpr_read: unable to uncompress page at pgno = 
> 246532
> WordDB: PANIC: Input/output error
> WordDB: CDB___memp_cmpr_read: unable to uncompress page at pgno = 
> 246532
> WordDB: PANIC: Input/output error
>
> Then at the end of the dig, I see the following.
>
> WordDB: CDB___memp_cmpr_read: unable to uncompress page at pgno = 
> 246532
> WordDB: PANIC: Input/output error
> WordDBCursor::Get(17) failed DB_RUNRECOVERY: Fatal error, run database 
> recovery
>
> The referenced page number is the same for every error message. For 
> this test, the default value was used for wordlist_page_size. Both 
> wordlist_compress and wordlist_compress_zlib were set to true, and 
> compression_level was set to 8. I used a fresh copy of the CVS code 
> (as of yesterday) with dbase.patch2 applied.
>
> Jim
>
> On Sunday, April 27, 2003, at 02:27 AM, Lachlan Andrew wrote:
>
>> OK, here is another patch...
>>
>> The problem was that I was using the counts of clean and dirty cache
>> pages, but they were not recorded correctly in the old BDB 2.x
>> (although that didn't show up in my initial tests...).  This patch
>> contains the fix, copied from BDB 3.3.11.
>>
>> Thanks again for your help :)
>>
>> Lachlan
>>
>> On Sun, 27 Apr 2003 12:54, Jim Cole wrote:
>>> I am still running into fatal problems with OS X. I no longer get
>>> the segfault, but instead see the output shown below.
>> <dbase.patch2>
>
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> htdig-dev mailing list
> htd...@li...
> https://lists.sourceforge.net/lists/listinfo/htdig-dev
>