htdig-dev — Developer Discussion for the ht://Dig project

Re: [htdig-dev] Configuration patch

[Jim C. <li...@yg...>]

On Wednesday, July 30, 2003, at 12:20 PM, Gabriele Bartolini wrote:

> That's what I believe too. AFAIK autotools are needed only if you want 
> to rebuild configure scripts and Makefile.in files. I have not run on 
> this problem on my laptop (where I generated it all). But ... I came 
> into this problem when trying with an older Red Hat distribution. I 
> found the problem with the 'db' subdirectory to be exact.

Well the messages involving automake appeared that they might be more 
of a warning nature than an outright error. However the autoconf 
version was definitely resulting in errors; the problem seemed to 
originate from the processing of aclocal.m4. At this point I don't know 
enough about the auto* tools to know if/why aclocal.m4 should be 
processed under normal circumstances.

What problem exactly did you run into with the 'db' directory? I did a 
make distclean intending to chase the problem I ran into a little 
further, but then ended up in a state where make fails in the 'db' 
directory due to there being no rule for target 'all' (I did rerun 
configure with no obvious errors).

Jim

Re: [htdig-dev] Configuration patch

[Gabriele B. <bar...@in...>]

Ciao guys,

    I am sorry if the patch is causing problems. But ... with the word 
'test' I meant exactly this. Try it and discover some problems.

>I don't know if it's still the case now, but in the past the auto*
>tools were only needed when you needed to develop/update the Makefiles,
>and weren't needed just to compile the code.

That's what I believe too. AFAIK autotools are needed only if you want to 
rebuild configure scripts and Makefile.in files. I have not run on this 
problem on my laptop (where I generated it all). But ... I came into this 
problem when trying with an older Red Hat distribution. I found the problem 
with the 'db' subdirectory to be exact.

>Can anyone with more automake/autoconf experience comment on the
>feasibility of this?  It would be nice if installation of ht://Dig
>continued not to require the auto* tools.

That's a *must* Gilles. And my (and Marco's) patch was in this direction.

-Gabriele
--
Gabriele Bartolini: Web Programmer, ht://Dig & IWA/HWG Member, ht://Check 
maintainer
Current Location: Prato, Tuscany, Italy
bar...@in... | http://www.prato.linux.it/~gbartolini | ICQ#129221447
 > "Leave every hope, ye who enter!", Dante Alighieri, Divine Comedy, The 
Inferno

Re: [htdig-dev] Configuration patch

[Gilles D. <gr...@sc...>]

According to Jim Cole:
> Hi - As someone already mentioned with regard to their Red Hat install, 
> the configuration changes to the 3.2 code base require auto* tools more 
> recent than those provided with the distribution. The same is true for 
> OS X, which currently provides autoconf 2.52 and automake 1.6.1. It 
> looks like the minimum requirements are currently autoconf 2.54 and 
> automake 1.7.
> 
> For the time being at least, I am unable to fully test the changes. It 
> appears that the required version of automake is even ahead of what is 
> available via fink, so it would have to be installed manually. Not to 
> say that installing it manually is any great chore, but I do wonder if 
> we are getting ahead of ourselves here. If upgrading auto* tools is 
> going to become a prerequisite for the typical ht://Dig install, that 
> is probably a bad thing.
> 
> Are the relevant enhancements only available via the newer auto* tools?

I don't know if it's still the case now, but in the past the auto*
tools were only needed when you needed to develop/update the Makefiles,
and weren't needed just to compile the code.  Now I think that with the
snapshots, the modtimes of the files aren't always set correctly so make
thinks it needs to rerun some of the auto* tools.

If things are set up correctly, you should normally be able to compile
and install an htdig distribution without using the auto* tools, unless
the packaged Makefiles don't work on your system.  It may be, though, that
with the new auto* tools the Makefiles end up being more system dependent
than in the past.

Can anyone with more automake/autoconf experience comment on the
feasibility of this?  It would be nice if installation of ht://Dig
continued not to require the auto* tools.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

[htdig-dev] Configuration patch

[Jim C. <li...@yg...>]

Hi - As someone already mentioned with regard to their Red Hat install, 
the configuration changes to the 3.2 code base require auto* tools more 
recent than those provided with the distribution. The same is true for 
OS X, which currently provides autoconf 2.52 and automake 1.6.1. It 
looks like the minimum requirements are currently autoconf 2.54 and 
automake 1.7.

For the time being at least, I am unable to fully test the changes. It 
appears that the required version of automake is even ahead of what is 
available via fink, so it would have to be installed manually. Not to 
say that installing it manually is any great chore, but I do wonder if 
we are getting ahead of ourselves here. If upgrading auto* tools is 
going to become a prerequisite for the typical ht://Dig install, that 
is probably a bad thing.

Are the relevant enhancements only available via the newer auto* tools?

Jim

[htdig-dev] Re: [htdig] Problems indexing javascript pages.

[Gabriele B. <g.b...@co...>]

Il mar, 2003-07-29 alle 14:19, Marco Antonio de Souza ha scritto:
> Hi all,
>=20
> I=B4m having problems with the htdig for indexing my site with javascript=
. The
> htdig don't index pages that are called by Javascript. I'm trying to conf=
ig
> the option url_rewrite_rules for solve my problem. I'm using the followin=
g
> syntax in htdig.conf.
>=20

Try and give a look at here:
http://www.htdig.org/FAQ.html#q5.18

Ciao
-Gabriele
--=20
Gabriele Bartolini - Web Programmer
Comune di Prato - Prato - Tuscany - Italy
g.b...@co... | http://www.comune.prato.it
> find bin/laden -name osama -exec rm {} ;

[htdig-dev] HTDig on Itanium

[David B. <D.B...@vp...>]

Hi folks,

Several (!) weeks ago I promised to test the current htdig beta on an
Itanium AS box I'm using. Sorry I've been so slow. A couple of issues, final
one is a bit of a show stopper (or at least a show slow-downer)

1. make file requires new versions of automake and autoconf. Not really a
problem but an issue worth noting. RedHat does not make appropriate rpms
available so needs to be build from source, easily done.

2. warning during config, "configure: WARNING: THREAD MUTEXES NOT AVAILABLE
FOR THIS COMPILER/ARCHITECTURE"

3. Finally, seems that htlib/Configuration.cc requires htconfig.h and cannot
find it. A quick look in changelog indicated it has recently been removed
(and replaced with config.h) so rather than getting my hands dirty, I
thought I'd check with the developers to see if it rings any bells. make
looks like this :

...
...
Making all in htlib
make[1]: Entering directory `/root/tars/htdig-3.2.0b4-20030727/htlib'
/bin/sh ../libtool --mode=compile
g++ -DHAVE_CONFIG_H -I. -I. -I../include -DDEFAULT_CONFIG_FILE=\"/usr/local/
apache/htdocs/htdig/conf/htdig.conf\" -I../include -I../htlib -I../htnet -I.
./htcommon -I../htword -I../db -I../db      -g -O2 -Wall -fno-rtti -fno-exce
ptions -c -o Configuration.lo `test -f 'Configuration.cc' || echo
'./'`Configuration.cc
mkdir .libs

g++ -DHAVE_CONFIG_H -I. -I. -I../include -DDEFAULT_CONFIG_FILE=\"/usr/local/
apache/htdocs/htdig/conf/htdig.conf\" -I../include -I../htlib -I../htnet -I.
./htcommon -I../htword -I../db -I../db -g -O2 -Wall -fno-rtti -fno-exception
s -c Configuration.cc  -fPIC -DPIC -o .libs/Configuration.o
Configuration.cc:20:22: htconfig.h: No such file or directory
make[1]: *** [Configuration.lo] Error 1
make[1]: Leaving directory `/root/tars/htdig-3.2.0b4-20030727/htlib'
make: *** [all-recursive] Error 1
[root@hp htdig-3.2.0b4-20030727]#


David



-------------------------------------------------------------
David Bannon                       D.B...@vp...
Phone 61 03 9925 4733              Fax 61 03 9925 4647
Mobile 0418 525687                 http://www.vpac.org
Systems Manager, Victorian Partnership for Advanced Computing
-------------------------------------------------------------
..... Humpty Dumpty was pushed !

[htdig-dev] Current Status as of snapshot 3.2.0b4-20030727

[Geoff H. <ghu...@us...>]

STATUS of ht://Dig branch 3-2-x

CHECKLIST FOR 3.2.0b5:
* Apply memory leak patches (Neal)
* Check bugs listed in bug-tracker...
* Polish release docs (Geoff)
* Must be able to
	(a) make check  and
	(b) index  www.htdig.org  using  "robotstxt_name: master-htdig"
  on all systems listed as "supported".
  Systems tested so far:
    - Mandrake 8.2, gcc 3.2 (lha, 21 May)
    - FreeBSD 4.6, gcc 2.95.3 (lha, 23 May)
    - Debian, Linux kernel 2.2.19, gcc 2.95.4 (lha, 23 May)
    - SunOS 5.8 = Solaris 2.8, gcc 3.1 (lha, 25 May)
    - SunOS 5.8 = Solaris 2.8, Sun cc with g++ 3.1 (lha, 29 May)
    - OS X (Jim, 30 May)
  Partly tested:
    - RedHat 8 (Jim, 1 June.  make check requires tweaking for apache)
    - SunOS 5.8 = Solaris 2.8, gcc 2.95.2
	    (lha. Makes check minus apache, Digs small htdig.org. 27 May)
    - SunOS 5.8 = Solaris 2.8, Sun cc with g++ 2.95.2
	    (lha. Makes check minus apache, Digs small htdig.org. 2 June)
    - RedHat 7.3
	    (lha. Makes check minus apache. Digs small htdig.org. 25 May)
    - Alpha Debian
    	    (lha. Makes check minus apache. Digs small htdig.org. 25 May)
  To be tested:
    - HP-UX 10.20, gcc 2.8.1 (Jesse)
    - RedHat, other versions anyone?
    - RedHat AdvanceServer Itanium II (David Bannon)
  Known to have problems:
    - SGI/Irix 6.5.3 using SGI compilers
      <http://www.geocrawler.com/mail/msg.php3?msg_id=8025827&list=8825>





RELEASES:
   3.2.0b5: Next release, July 2003
   3.2.0b4: "In progress" -- snapshots called "3.2.0b4" until prerelease.
   3.2.0b3: Released:  22 Feb 2001.
   3.2.0b2: Released:  11 Apr 2000.
   3.2.0b1: Released:   4 Feb 2000.

(Please note that everything added here should have a tracker PR# so
we can be sure they're fixed. Geoff is currently trying to add PR#s for
what's currently here.)

SHOWSTOPPERS:
* Mifluz database errors are a severe problem (PR#428295)
-- Does Neal's new zlib patch solve this for now?

KNOWN BUGS:
* Odd behavior with $(MODIFIED) and scores not working with
   wordlist_compress set but work fine without wordlist_compress.
   (the date is definitely stored correctly, even with compression on
    so this must be some sort of weird htsearch bug) PR#618737.
* META descriptions are somehow added to the database as FLAG_TITLE,
   not FLAG_DESCRIPTION. (PR#618738)
   Can anyone reproduce this?  I can't! -- Lachlan
   Me either. Let's remove the PR. -Geoff

PENDING PATCHES (available but need work):
* Additional support for Win32. (Neal)
* Memory improvements to htmerge. (Backed out b/c htword API changed.)
* Mifluz merge.

NEEDED FEATURES:
* Quim's new htsearch/qtest query parser framework.
* File/Database locking. PR#405764.

TESTING:
* httools programs: 
  (htload a test file, check a few characteristics, htdump and compare)
* Tests for new config file parser
* Duplicate document detection while indexing
* Major revisions to ExternalParser.cc, including fork/exec instead of popen,
  argument handling for parser/converter, allowing binary output from an
  external converter.
* ExternalTransport needs testing of changes similar to ExternalParser.

DOCUMENTATION:
* List of supported platforms/compilers is ancient. (PR#405279)
* Document all of htsearch's mappings of input parameters to config attributes
   to template variables. (Relates to PR#405278.)
  Should we make sure these config attributes are all documented in
  defaults.cc, even if they're only set by input parameters and never
  in the config file?
* Split attrs.html into categories for faster loading.
* Turn defaults.cc into an XML file for generating documentation and 
  defaults.cc.
* require.html is not updated to list new features and disk space
   requirements of 3.2.x (e.g. regex matching, database compression.)
   PRs# 405280 #405281.
* TODO.html has not been updated for current TODO list and
   completions.
   I've tried.  Someone "official" please check and remove this -- Lachlan
* Htfuzzy could use more documentation on what each fuzzy algorithm
   does. PR#405714.
* Document the list of all installed files and default
   locations. PR#405715.

OTHER ISSUES:
* Can htsearch actually search while an index is being created?
* The code needs a security audit, esp. htsearch. PR#405765.

Re: [htdig-dev] Recommended IDE (a little off topic)

[Gabriele B. <g.b...@co...>]

Hi Ted,

   I am glad you are getting an expert as far as Auto tools are
concerned. You're probably aware of the core changes I've committed this
week.

   Would you mind helping us testing and improving them?

Thanks
-Gabriele

Il gio, 2003-07-24 alle 16:39, Ted Stresen-Reuter ha scritto:
> Hi,
> 
> I've been watching the list regarding the recent patches and would 
> really, really like to contribute more, but not having a degree in 
> computer science nor having much C programming experience I'm finding 
> the learning curve to be rather steep.
> 
> That said, I should add that I've recently read much of the 
> documentation on the GNU Automake site and now have a much better 
> understanding of the Auto* tools and how they work together. 
> Additionally, I purchased a book called Mac OS X  for Unix Geeks (which 
> I'm not) because it has a couple of chapters that deal specifically 
> with building "legacy" (Unix) software on Mac OS X, including a chapter 
> on building dynamic shared libraries.
> 
> I'm writing to the list today to see if anyone can make a 
> recommendation on an IDE for Mac OS X (Jaguar). Apple's IDE, Project 
> Builder, is outstanding for native OS X (Cocoa, ObjectiveC, Java) 
> development, but isn't ideally suited for C development (or at least, 
> that's what their documentation would lead one to believe). I know 
> there are probably two or three really good open source IDEs out there 
> that integrate more closely with the Auto* suite of tools and before I 
> go mucking around trying to find one that works for me, I thought I'd 
> ask some of the developers on this list for recommendations.
> 
> I look forward to your replies and if you want to answer off-list, that 
> would be fine too.
> 
> Thanks for all of your hard work, all of you...
> 
> Ted Stresen-Reuter
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email sponsored by: Free pre-built ASP.NET sites including
> Data Reports, E-commerce, Portals, and Forums are available now.
> Download today and enter to win an XBOX or Visual Studio .NET.
> http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
> _______________________________________________
> ht://Dig Developer mailing list:
> htd...@li...
> List information (subscribe/unsubscribe, etc.)
> https://lists.sourceforge.net/lists/listinfo/htdig-dev
-- 
Gabriele Bartolini - Web Programmer
Comune di Prato - Prato - Tuscany - Italy
g.b...@co... | http://www.comune.prato.it
> find bin/laden -name osama -exec rm {} ;

[htdig-dev] Recommended IDE (a little off topic)

[Ted Stresen-R. <ted...@ma...>]

Hi,

I've been watching the list regarding the recent patches and would 
really, really like to contribute more, but not having a degree in 
computer science nor having much C programming experience I'm finding 
the learning curve to be rather steep.

That said, I should add that I've recently read much of the 
documentation on the GNU Automake site and now have a much better 
understanding of the Auto* tools and how they work together. 
Additionally, I purchased a book called Mac OS X  for Unix Geeks (which 
I'm not) because it has a couple of chapters that deal specifically 
with building "legacy" (Unix) software on Mac OS X, including a chapter 
on building dynamic shared libraries.

I'm writing to the list today to see if anyone can make a 
recommendation on an IDE for Mac OS X (Jaguar). Apple's IDE, Project 
Builder, is outstanding for native OS X (Cocoa, ObjectiveC, Java) 
development, but isn't ideally suited for C development (or at least, 
that's what their documentation would lead one to believe). I know 
there are probably two or three really good open source IDEs out there 
that integrate more closely with the Auto* suite of tools and before I 
go mucking around trying to find one that works for me, I thought I'd 
ask some of the developers on this list for recommendations.

I look forward to your replies and if you want to answer off-list, that 
would be fine too.

Thanks for all of your hard work, all of you...

Ted Stresen-Reuter

[htdig-dev] Current Status as of snapshot 3.2.0b4-20030720

[Geoff H. <ghu...@us...>]

STATUS of ht://Dig branch 3-2-x

CHECKLIST FOR 3.2.0b5:
* Apply memory leak patches (Neal)
* Check bugs listed in bug-tracker...
* Polish release docs (Geoff)
* Must be able to
	(a) make check  and
	(b) index  www.htdig.org  using  "robotstxt_name: master-htdig"
  on all systems listed as "supported".
  Systems tested so far:
    - Mandrake 8.2, gcc 3.2 (lha, 21 May)
    - FreeBSD 4.6, gcc 2.95.3 (lha, 23 May)
    - Debian, Linux kernel 2.2.19, gcc 2.95.4 (lha, 23 May)
    - SunOS 5.8 = Solaris 2.8, gcc 3.1 (lha, 25 May)
    - SunOS 5.8 = Solaris 2.8, Sun cc with g++ 3.1 (lha, 29 May)
    - OS X (Jim, 30 May)
  Partly tested:
    - RedHat 8 (Jim, 1 June.  make check requires tweaking for apache)
    - SunOS 5.8 = Solaris 2.8, gcc 2.95.2
	    (lha. Makes check minus apache, Digs small htdig.org. 27 May)
    - SunOS 5.8 = Solaris 2.8, Sun cc with g++ 2.95.2
	    (lha. Makes check minus apache, Digs small htdig.org. 2 June)
    - RedHat 7.3
	    (lha. Makes check minus apache. Digs small htdig.org. 25 May)
    - Alpha Debian
    	    (lha. Makes check minus apache. Digs small htdig.org. 25 May)
  To be tested:
    - HP-UX 10.20, gcc 2.8.1 (Jesse)
    - RedHat, other versions anyone?
    - RedHat AdvanceServer Itanium II (David Bannon)
  Known to have problems:
    - SGI/Irix 6.5.3 using SGI compilers
      <http://www.geocrawler.com/mail/msg.php3?msg_id=8025827&list=8825>





RELEASES:
   3.2.0b5: Next release, July 2003
   3.2.0b4: "In progress" -- snapshots called "3.2.0b4" until prerelease.
   3.2.0b3: Released:  22 Feb 2001.
   3.2.0b2: Released:  11 Apr 2000.
   3.2.0b1: Released:   4 Feb 2000.

(Please note that everything added here should have a tracker PR# so
we can be sure they're fixed. Geoff is currently trying to add PR#s for
what's currently here.)

SHOWSTOPPERS:
* Mifluz database errors are a severe problem (PR#428295)
-- Does Neal's new zlib patch solve this for now?

KNOWN BUGS:
* Odd behavior with $(MODIFIED) and scores not working with
   wordlist_compress set but work fine without wordlist_compress.
   (the date is definitely stored correctly, even with compression on
    so this must be some sort of weird htsearch bug) PR#618737.
* META descriptions are somehow added to the database as FLAG_TITLE,
   not FLAG_DESCRIPTION. (PR#618738)
   Can anyone reproduce this?  I can't! -- Lachlan
   Me either. Let's remove the PR. -Geoff

PENDING PATCHES (available but need work):
* Additional support for Win32. (Neal)
* Memory improvements to htmerge. (Backed out b/c htword API changed.)
* Mifluz merge.

NEEDED FEATURES:
* Quim's new htsearch/qtest query parser framework.
* File/Database locking. PR#405764.

TESTING:
* httools programs: 
  (htload a test file, check a few characteristics, htdump and compare)
* Tests for new config file parser
* Duplicate document detection while indexing
* Major revisions to ExternalParser.cc, including fork/exec instead of popen,
  argument handling for parser/converter, allowing binary output from an
  external converter.
* ExternalTransport needs testing of changes similar to ExternalParser.

DOCUMENTATION:
* List of supported platforms/compilers is ancient. (PR#405279)
* Document all of htsearch's mappings of input parameters to config attributes
   to template variables. (Relates to PR#405278.)
  Should we make sure these config attributes are all documented in
  defaults.cc, even if they're only set by input parameters and never
  in the config file?
* Split attrs.html into categories for faster loading.
* Turn defaults.cc into an XML file for generating documentation and 
  defaults.cc.
* require.html is not updated to list new features and disk space
   requirements of 3.2.x (e.g. regex matching, database compression.)
   PRs# 405280 #405281.
* TODO.html has not been updated for current TODO list and
   completions.
   I've tried.  Someone "official" please check and remove this -- Lachlan
* Htfuzzy could use more documentation on what each fuzzy algorithm
   does. PR#405714.
* Document the list of all installed files and default
   locations. PR#405715.

OTHER ISSUES:
* Can htsearch actually search while an index is being created?
* The code needs a security audit, esp. htsearch. PR#405765.

Re: [htdig-dev] New patch: ISO stdc++ library

[Lachlan A. <lh...@us...>]

Yes, it would be great to have his patches included.  I don't mind if=20
you ditch my patches.  Sorry I haven't been able to do much for the=20
past month, and won't be able to do anything for the next month or=20
more :(

Good luck getting 3.2.0b5 out!

Lachlan

On Fri, 18 Jul 2003 21:22, Gabriele Bartolini wrote:

>    I'd love (seriously) to see his patch become real part of the
> main CVS tree code and to constitute the base for further works,
> even though I am aware of some real hard work to make it go under
> MacOSX.

--=20
lh...@us...

Re: [htdig-dev] New patch: ISO stdc++ library

[Gabriele B. <g.b...@co...>]

Hi guys,

   I would like to thank Marco for his work. He was really happy to
contribute to our 'cause'; I am asking you a huge effort now.

   I believe that future (in portability) of ht://Dig really depends on
a solid configuration framework. Marco's job could be crucial in this
direction.

   I'd love (seriously) to see his patch become real part of the main
CVS tree code and to constitute the base for further works, even though
I am aware of some real hard work made by Lachlan, Jim and others to
make it go under MacOSX.

   I just wanna ask your opinion as far as this is concerned. I know
that part of your job could go wasted but I can really see the
importance of this patch.

   Also, the more we work independently without this patch, the more
changes we will need to do again in the future.

   I wished we could have already released 3.2.0b5, especially for
Lachlan and Neal's precious work (but mine too - I've been waiting the
'htnet' library to become part of a stable release for a long time
already, although I have the consolation of using it within the
ht://Check project).

   So please ... let me know:

- A) if you wanna include the patch in the code: this means we will have
to rediscuss some compatibility problems with MacOSX mainly (and then
find the solution again)
- B) let's wait and go on with this, and let it be just a patch which
could be reconsidered again in the future

   I know it is bad for me to say this but ... as we've been waiting for
so long for this release to come out, I can't see a better reason for us
to wait a bit more.

Well ... please let me know what you think about this. However, thank
you all for your attention and ... anyway .. have a great week-end!

Ciao
-Gabriele

Il lun, 2003-07-14 alle 17:20, Marco Nenciarini ha scritto:
> 
> This patch allow ht://Dig to use the ISO stdc++ library if it is avaliable,
> removing many warning issued (on gcc 3.2 og latest) by inclusion of
> deprecated headers like <iostream.h>
> 
> The patch is made against the latest snapshot and includes the G.Bartolini
> patch.
> 
> Please test it :)
> 
> Ciao
> 
> -- 
> ---------------------------------------------------------------------
> |    Marco Nenciarini    | Debian/GNU Linux Developer - Plug Member |
> | mn...@pr... | http://www.prato.linux.it/~mnencia       |
> ---------------------------------------------------------------------
> Key fingerprint = FED9 69C7 9E67 21F5 7D95  5270 6864 730D F095 E5E4
> 
-- 
Gabriele Bartolini - Web Programmer
Comune di Prato - Prato - Tuscany - Italy
g.b...@co... | http://www.comune.prato.it
> find bin/laden -name osama -exec rm {} ;

[htdig-dev] Re: Your changes to mp_alloc.c

[Lachlan A. <lh...@us...>]

Greetings Neal,

Yes, this is the "page leak" that was discussed late last month with=20
Krzysztof Gorgolewski.  My patch tickles a bug in the existing code=20
to write a page -- unless the page is flushed from the cache after=20
writing, any new chain which is introduced is lost.  Without my new=20
code, the only time the page is written is when it is about to be=20
flushed from the cache, so the bug doesn't manifest itself.  (I wrote=20
an email describing all of this, but I can't find it in my outbox, so=20
I may not have actually sent it ):

The best solution is to fix the original bug, but that involves yet=20
more changes to db.h etc to allocate room in each page header for the=20
list of overflow pages, or some similar. A simple way of avoiding the=20
problem is, of course, to remove all my new code to flush the cache=20
periodically, which is made redundant by using the "standalone"=20
database for the "weak compression" free list.  As a hack until I get=20
time to remove the flushing completely, CVS currently disables cache=20
flushing by setting  wordlist_cache_dirty_level  to a very large=20
value, so it *shouldn't* be a problem in the latest snapshot.  What=20
version are you using to generate the problem?

I'm off overseas for a month tomorrow, so I don't have time to work on=20
ht://Dig at all.  If you like, feel free to revert to the original=20
before I started messing things up...

On Tue, 15 Jul 2003 11:08, Neal Richter wrote:
> Lachlan,
>
> =09I'm seeing a startling difference in db.word.db sizes before and
> after your changes to mp_alloc.c and others.
>
>  77824 Jul 14 18:45 db.words.db
> 184320 Jul 14 18:24 db.words.db
>
> If I use htdump to dump these files, the dump is exactly the same.
>
> Ack!  Do you see this?
>
> Neal Richter
> Knowledgebase Developer
> RightNow Technologies, Inc.
> Customer Service for Every Web Site
> Office: 406-522-1485

--=20
lh...@us...

Re: [htdig-dev] Accessible version for ht:/dig: AAA- WAI

[Gilles D. <gr...@sc...>]

According to Gabriele Bartolini:
>    If you have some code to contribute, I'd love to hack the main CVS
> tree and make your changes available in the next release (hopefully
> soon!).
> 
>    Main changes I was thinking about regard mainly images and forms
> (with proper labels).
...
> Il mer, 2003-07-16 alle 20:15, Enrico Neri ha scritto:
> > dear all,
> > 2003 is the year of  Accessibility.
> > 
> > We have just developed the website in LAMP for S.Marc Venice Basilica.
> > 
> > website is @:  http://basilica.veniceplaza.biz
...
> > As you see it's just "formatting" work,  but it takes a lot of effort&patience.
> > 
> > If any of you need to do it, I'll forward you code. Just drop me a mail.
> > 
> > file to be  changed:
> > - htsearch.cc->make
> > - .conf
> > - header/footer.html

I'm especially curious about what had to be changed in htsearch.cc to
help with accessibility.  But yes, it would be great if the next release
of htsearch had accessibility built in by default.  Contributions like
this are always welcome.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] Accessible version for ht:/dig: AAA- WAI

[Gabriele B. <g.b...@co...>]

Ciao Enrico,

   I post you this message in english, but I'd like to get in touch with
you as I'm italian too. Anyway ...

   That would be great. Indeed, we've been talking so many times about
writing the code in HTML strict or XHTML and I promised myself to try
and make it accessible as well. But ... it's always the same old story:
I'll do it tomorrow ... :-)

   If you have some code to contribute, I'd love to hack the main CVS
tree and make your changes available in the next release (hopefully
soon!).

   Main changes I was thinking about regard mainly images and forms
(with proper labels).

   Let me know please.

Ciao and thank you
-Gabriele

Il mer, 2003-07-16 alle 20:15, Enrico Neri ha scritto:
> dear all,
> 2003 is the year of  Accessibility.
> 
> We have just developed the website in LAMP for S.Marc Venice Basilica.
> 
> website is @:  http://basilica.veniceplaza.biz
> 
> I have put htidig in action both for italian/english website AND italian/english
>  Accessible website.
> 
> URL:
> http://basilica.veniceplaza.biz/WAI/eng/search/search.bsm
> 
> you might see the for to access htsearch. (it's htbasilica)
> results have been formatted to fully accomplish WAI and Bobby AAA.
> 
> As you see it's just "formatting" work,  but it takes a lot of effort&patience.
> 
> If any of you need to do it, I'll forward you code. Just drop me a mail.
> 
> file to be  changed:
> - htsearch.cc->make
> - .conf
> - header/footer.html
> 
> PS:
> htdig has help dramatically the website as it has 300 webpages __needing__ a spider.
> 
> Have fun digging around!
> 
> Enrico
> CTO
> Veniceplaza
> -- 
> CMe@    http://213.214.82.247
> http    www.veniceplaza.net
> mobile  39 348 25 59 557 
> ph      39 049 410 44 15
> 
> -------------------------------------------------
> This mail sent through Veniceplaza Web Mail: http://www.veniceplaza.net/
> 
> 
> This email has been checked by Veniceplaza Real Time AntiVirus.
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: VM Ware
> With VMware you can run multiple operating systems on a single machine.
> WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines at the
> same time. Free trial click here: http://www.vmware.com/wl/offer/345/0
> _______________________________________________
> ht://Dig Developer mailing list:
> htd...@li...
> List information (subscribe/unsubscribe, etc.)
> https://lists.sourceforge.net/lists/listinfo/htdig-dev
-- 
Gabriele Bartolini - Web Programmer
Comune di Prato - Prato - Tuscany - Italy
g.b...@co... | http://www.comune.prato.it
> find bin/laden -name osama -exec rm {} ;

[htdig-dev] Accessible version for ht:/dig: AAA- WAI

[Enrico N. <en...@ve...>]

dear all,
2003 is the year of  Accessibility.

We have just developed the website in LAMP for S.Marc Venice Basilica.

website is @:  http://basilica.veniceplaza.biz

I have put htidig in action both for italian/english website AND italian/english
 Accessible website.

URL:
http://basilica.veniceplaza.biz/WAI/eng/search/search.bsm

you might see the for to access htsearch. (it's htbasilica)
results have been formatted to fully accomplish WAI and Bobby AAA.

As you see it's just "formatting" work,  but it takes a lot of effort&patience.

If any of you need to do it, I'll forward you code. Just drop me a mail.

file to be  changed:
- htsearch.cc->make
- .conf
- header/footer.html

PS:
htdig has help dramatically the website as it has 300 webpages __needing__ a spider.

Have fun digging around!

Enrico
CTO
Veniceplaza
-- 
CMe@    http://213.214.82.247
http    www.veniceplaza.net
mobile  39 348 25 59 557 
ph      39 049 410 44 15

-------------------------------------------------
This mail sent through Veniceplaza Web Mail: http://www.veniceplaza.net/


This email has been checked by Veniceplaza Real Time AntiVirus.

[htdig-dev] Changes to mp_alloc.c to prevent execesive recursion cause larger size

[Neal R. <ne...@ri...>]

Hey all,
	I'm seeing a bad effect of the recent changes to mp_alloc.c.  The
db.word.db is growing excessively.

	If I replace mp_alloc.c with a previous version (Revision 1.2 via
CVS), this issue goes away.  It is necessary to comment out a variable in
db.h, DB2_db.cc & WordDB.cc.

	Note that it does not cause DB corruption as htdump produces
EXACTLY equivalent text dumps of the db.word.db before and after the
change.

	If you Define DEBUG_CMPR in ml_cmpr.cc you notice during the run
that the memory pages seem to be flushed to disk a LOT more often.

	Could someone else test this and reply?

Example:
 77824 Jul 14 18:45 db.words.db   (old mp_alloc.c)
184320 Jul 14 18:24 db.words.db   (new mp_alloc.c)

And the size difference gets much worse the more pages you index.

Thanks!

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] FW: New configure patch

[Gabriele B. <bar...@in...>]

Hi Jim,

    I have great news about this. My friend Marco (he's great! he did this 
job for ht://Check and he's a Debian developer) today just worked on my 
patch and modified the code as well. He found many things that may be 
useful and may improve the code. He is going to post his patch (which is 
comprehensive of the code changes too) today and ... I am really curious to 
test it.

    Again ... we are kind of blind as far as OS X testing is concerned but 
I am sure that this patch is really straightforwarding. The only problem is 
related to Lachlan hand-changes to configure and I hope we could solve them 
this way (which is the clean way IMHO):

    So ... Marco, please show off yourself! :-)

Ciao
-Gabriele

At 11.57 11/07/2003 -0600, Jim Cole wrote:
>On Sunday, July 6, 2003, at 05:06 PM, Gabriele Bartolini wrote:
>
>>    I just send this patch towards new configure scripts; I converted
>>older
>>configure.in and related files to more standard procedures (even
>>though it is an
>>endless process and what I have done is just the beginning).
>...
>>     I should ask (especially OS X users) to kindly test this patch as
>>I did on my
>>Linux (and works).
>
>Hi - The configure script seemed to run without any obvious problems
>under OS X. However the build fails while trying to compile
>mhash_md5.c. The output is shown below. I haven't had time to look at
>anything in detail and don't know whether it is related or not, but
>figured I would pass it along in case anyone else had any immediate
>ideas.
>
>I applied your patch against a July 10 CVS checkout.
>
>Jim
>
>
>  gcc -DHAVE_CONFIG_H -I. -I. -I../include
>-DDEFAULT_CONFIG_FILE=\"/Users/greyleaf/local/htdig_cvs/conf/ htdig.conf\" 
>-I../include -I../htlib -I../htnet -I../htcommon
>-I../htword -I../db -I../db -g -O2 -Wall -c mhash_md5.c -o mhash_md5.o
> >/dev/null 2>&1
>make[1]: *** No rule to make target `error.c', needed by `error.lo'.
>Stop.
>make: *** [all-recursive] Error 1
>
>
>
>-------------------------------------------------------
>This SF.Net email sponsored by: Parasoft
>Error proof Web apps, automate testing & more.
>Download & eval WebKing and get a free book.
>www.parasoft.com/bulletproofapps1
>_______________________________________________
>ht://Dig Developer mailing list:
>htd...@li...
>List information (subscribe/unsubscribe, etc.)
>https://lists.sourceforge.net/lists/listinfo/htdig-dev

--
Gabriele Bartolini: Web Programmer, ht://Dig & IWA/HWG Member, ht://Check 
maintainer
Current Location: Prato, Tuscany, Italy
bar...@in... | http://www.prato.linux.it/~gbartolini | ICQ#129221447
 > "Leave every hope, ye who enter!", Dante Alighieri, Divine Comedy, The 
Inferno

[htdig-dev] New patch: ISO stdc++ library

[Marco N. <mn...@pr...>]

This patch allow ht://Dig to use the ISO stdc++ library if it is avaliable,
removing many warning issued (on gcc 3.2 og latest) by inclusion of
deprecated headers like <iostream.h>

The patch is made against the latest snapshot and includes the G.Bartolini
patch.

Please test it :)

Ciao

-- 
---------------------------------------------------------------------
|    Marco Nenciarini    | Debian/GNU Linux Developer - Plug Member |
| mn...@pr... | http://www.prato.linux.it/~mnencia       |
---------------------------------------------------------------------
Key fingerprint = FED9 69C7 9E67 21F5 7D95  5270 6864 730D F095 E5E4

[htdig-dev] Current Status as of snapshot 3.2.0b4-20030713

[Geoff H. <ghu...@us...>]

STATUS of ht://Dig branch 3-2-x

CHECKLIST FOR 3.2.0b5:
* Apply memory leak patches (Neal)
* Check bugs listed in bug-tracker...
* Polish release docs (Geoff)
* Must be able to
	(a) make check  and
	(b) index  www.htdig.org  using  "robotstxt_name: master-htdig"
  on all systems listed as "supported".
  Systems tested so far:
    - Mandrake 8.2, gcc 3.2 (lha, 21 May)
    - FreeBSD 4.6, gcc 2.95.3 (lha, 23 May)
    - Debian, Linux kernel 2.2.19, gcc 2.95.4 (lha, 23 May)
    - SunOS 5.8 = Solaris 2.8, gcc 3.1 (lha, 25 May)
    - SunOS 5.8 = Solaris 2.8, Sun cc with g++ 3.1 (lha, 29 May)
    - OS X (Jim, 30 May)
  Partly tested:
    - RedHat 8 (Jim, 1 June.  make check requires tweaking for apache)
    - SunOS 5.8 = Solaris 2.8, gcc 2.95.2
	    (lha. Makes check minus apache, Digs small htdig.org. 27 May)
    - SunOS 5.8 = Solaris 2.8, Sun cc with g++ 2.95.2
	    (lha. Makes check minus apache, Digs small htdig.org. 2 June)
    - RedHat 7.3
	    (lha. Makes check minus apache. Digs small htdig.org. 25 May)
    - Alpha Debian
    	    (lha. Makes check minus apache. Digs small htdig.org. 25 May)
  To be tested:
    - HP-UX 10.20, gcc 2.8.1 (Jesse)
    - RedHat, other versions anyone?
    - RedHat AdvanceServer Itanium II (David Bannon)
  Known to have problems:
    - SGI/Irix 6.5.3 using SGI compilers
      <http://www.geocrawler.com/mail/msg.php3?msg_id=8025827&list=8825>





RELEASES:
   3.2.0b5: Next release, July 2003
   3.2.0b4: "In progress" -- snapshots called "3.2.0b4" until prerelease.
   3.2.0b3: Released:  22 Feb 2001.
   3.2.0b2: Released:  11 Apr 2000.
   3.2.0b1: Released:   4 Feb 2000.

(Please note that everything added here should have a tracker PR# so
we can be sure they're fixed. Geoff is currently trying to add PR#s for
what's currently here.)

SHOWSTOPPERS:
* Mifluz database errors are a severe problem (PR#428295)
-- Does Neal's new zlib patch solve this for now?

KNOWN BUGS:
* Odd behavior with $(MODIFIED) and scores not working with
   wordlist_compress set but work fine without wordlist_compress.
   (the date is definitely stored correctly, even with compression on
    so this must be some sort of weird htsearch bug) PR#618737.
* META descriptions are somehow added to the database as FLAG_TITLE,
   not FLAG_DESCRIPTION. (PR#618738)
   Can anyone reproduce this?  I can't! -- Lachlan
   Me either. Let's remove the PR. -Geoff

PENDING PATCHES (available but need work):
* Additional support for Win32. (Neal)
* Memory improvements to htmerge. (Backed out b/c htword API changed.)
* Mifluz merge.

NEEDED FEATURES:
* Quim's new htsearch/qtest query parser framework.
* File/Database locking. PR#405764.

TESTING:
* httools programs: 
  (htload a test file, check a few characteristics, htdump and compare)
* Tests for new config file parser
* Duplicate document detection while indexing
* Major revisions to ExternalParser.cc, including fork/exec instead of popen,
  argument handling for parser/converter, allowing binary output from an
  external converter.
* ExternalTransport needs testing of changes similar to ExternalParser.

DOCUMENTATION:
* List of supported platforms/compilers is ancient. (PR#405279)
* Document all of htsearch's mappings of input parameters to config attributes
   to template variables. (Relates to PR#405278.)
  Should we make sure these config attributes are all documented in
  defaults.cc, even if they're only set by input parameters and never
  in the config file?
* Split attrs.html into categories for faster loading.
* Turn defaults.cc into an XML file for generating documentation and 
  defaults.cc.
* require.html is not updated to list new features and disk space
   requirements of 3.2.x (e.g. regex matching, database compression.)
   PRs# 405280 #405281.
* TODO.html has not been updated for current TODO list and
   completions.
   I've tried.  Someone "official" please check and remove this -- Lachlan
* Htfuzzy could use more documentation on what each fuzzy algorithm
   does. PR#405714.
* Document the list of all installed files and default
   locations. PR#405715.

OTHER ISSUES:
* Can htsearch actually search while an index is being created?
* The code needs a security audit, esp. htsearch. PR#405765.

Re: [htdig-dev] Massive Memory consumption of htnotify..

[Jim C. <li...@yg...>]

On Friday, July 11, 2003, at 07:26 AM, Robert Ribnitz wrote:

> I am just letting you guys know that htnotify might contain a memory  
> leak, leading to it using huge amounts of memory, and even aborting  
> due to lack of memory.
>
> At the moment I dont know where to look for the problem, so help would  
> be appreciated.

There is a know issue with htnotify involving the manner in which it  
opens some streams. Up to this point, I don't recall anyone reporting  
the resulting problem as affecting Linux, but it has caused problems  
under both Solaris and OS X. I think the fact that it has not thus far  
affected other platforms is more an accident than anything else, so  
maybe something has changed that now makes your platform susceptible.  
The problem is discussed in the following thread.

http://www.geocrawler.com/mail/thread.php3?subject=%5Bhtdig- 
dev%5D+OS+X+10.2&list=8825

You might try specifying the htnotify_prefix_file and  
htnotify_suffix_file and creating the corresponding text files. If that  
eliminates the problem you are encountering, then most likely the bug  
described above is at the root of the problem.

There is a patch that partially corrects the problem; however it  
neglects to treat one special case and ignores entirely another section  
of the code that is susceptible to the same sort of problem. When I get  
a chance, I will take a stab at putting together a more complete patch.

Jim

Re: [htdig-dev] FW: New configure patch

[Jim C. <li...@yg...>]

On Sunday, July 6, 2003, at 05:06 PM, Gabriele Bartolini wrote:

>    I just send this patch towards new configure scripts; I converted  
> older
> configure.in and related files to more standard procedures (even  
> though it is an
> endless process and what I have done is just the beginning).
...
>     I should ask (especially OS X users) to kindly test this patch as  
> I did on my
> Linux (and works).

Hi - The configure script seemed to run without any obvious problems  
under OS X. However the build fails while trying to compile  
mhash_md5.c. The output is shown below. I haven't had time to look at  
anything in detail and don't know whether it is related or not, but  
figured I would pass it along in case anyone else had any immediate  
ideas.

I applied your patch against a July 10 CVS checkout.

Jim


  gcc -DHAVE_CONFIG_H -I. -I. -I../include  
-DDEFAULT_CONFIG_FILE=\"/Users/greyleaf/local/htdig_cvs/conf/ 
htdig.conf\" -I../include -I../htlib -I../htnet -I../htcommon  
-I../htword -I../db -I../db -g -O2 -Wall -c mhash_md5.c -o mhash_md5.o  
 >/dev/null 2>&1
make[1]: *** No rule to make target `error.c', needed by `error.lo'.   
Stop.
make: *** [all-recursive] Error 1

Re: [htdig-dev] htdig stores program data in /etc istead of /var

[Geoff H. <ghu...@ws...>]

> another bug in htdig (3.1.6) might be worthy of your attention. It 
> looks like htdig creates a (temporary) file in /etc/htdig to store 
> some runtime info (and to signal a rebuild)

The ht://Dig source package does not create any files in "/etc/htdig" 
by default. Period. The configure script can set the various prefix 
directories, but a straight "./configure; make; make install" will put 
various files in /opt/www/htdig

The *Debian* package and/or affiliated shell/cron scripts may put files 
somewhere.

My reading of the Debian bug report is that the /etc/cron.weekly/htdig 
script installed as part of your package creates this file.

As I do not use Debian, I cannot give you any more information--only 
that it's not an upstream issue.

Cheers,
-Geoff

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

[htdig-dev] htdig stores program data in /etc istead of /var

[Robert R. <ri...@li...>]

Hello again,

another bug in htdig (3.1.6) might be worthy of your attention. It looks 
like htdig creates a (temporary) file in /etc/htdig to store some 
runtime info (and to signal a rebuild)

I think this info should go into /var rather than /etc

Full bugreport is at

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=199276

yours

Robert

[htdig-dev] Massive Memory consumption of htnotify..

[Robert R. <ri...@li...>]

Hello there,

I am the new maintainer of htdig in debian. The version that is in 
debian at the moment is 3.1.6.

I am just letting you guys know that htnotify might contain a memory 
leak, leading to it using huge amounts of memory, and even aborting due 
to lack of memory.

At the moment I dont know where to look for the problem, so help would 
be appreciated.

You'll find all relevant info at the url given below

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=200734

Any help would be appreciated

Robert