htdig-dev — Developer Discussion for the ht://Dig project

Re: [htdig-dev] 3.2.0b5 Testing

[Gilles D. <gr...@sc...>]

According to Joe R. Jah:
> On Thu, 13 Nov 2003, Gilles Detillieux wrote:
> > According to Joe R. Jah:
> > > htfuzzy metaphone dumps core, but it works fine with endings, etc.
> > 
> > Just a hunch, but I'd guess that you had a metaphone database left over
> > from 3.1.6, and that the newer DB code in 3.2.0b5 doesn't like it.  We
> > had put some tests for this in some of the other programs, but maybe not
> > this one.  Try deleting the databases and running htfuzzy metaphone again.
> > The same would probably go for soundex and accents databases, if you use
> > either of those.
> 
> Spot on!  I had an older db.metaphone.db left over; none of the other
> supported algorithms had any problem:
>         soundex
>         accents
>         endings
>         synonyms

endings and synonyms are generated a bit differently, and most importantly
are built in a temporary spot then moved into place, so they don't collide
with any existing databases until the new one is complete.  soundex and
accents use the same code as metaphone for writing the database from the
generated word list, so they'd all potentially have the same problem.
The scrambling of data may not have been bad enough to cause a segfault
(yet) but may have led to corrupt databases.  If you had existing
databases for accents and/or soundex in place, built by 3.1.x, before
regenerating them for 3.2.0b5, you should remove them and try again,
just as for metaphone.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

[htdig-dev] Adding allow_space_in_url to 3.2.0

[Gilles D. <gr...@sc...>]

According to Joe R. Jah, on htdig-general:
> On Thu, 13 Nov 2003, Gilles Detillieux wrote:
> > According to Joe R. Jah:
> > > I have been trying to port fileSpace.1 from 3.1.6 to 3.2.0b5.  At first it
> > > appeared to be relatively simple, attached, but:
...
> > This patch adds an allow_space_in_url attribute to htdig 3.2.0b5, so
> > that you can get htdig to handle URLs that contain embedded spaces.
> > Technically, this is a violation of RFC 2396, which says spaces should
> > be stripped out (as htdig does by default).  However, many web browsers
> > and HTML code generators violate this standard already, so enabling this
> > attribute allows htdig to handle these non-compliant URLs.  Even with
> > this attribute set, htdig still strips out all white space (leading,
> > trailing and embedded), except that space characters embedded within
> > the URL will be encoded as %20.
> ...
> 
> Thank you Gilles; it works like a charm; I have put it in the patch site:
> 
>   ftp://ftp.ccsf.org/htdig-patches/3.2.0b5/fileSpace.0
> 
> I vote +1 to avail this option for 3.2.0.

That raises a good question as to feature-freeze status.  Now that
3.2.0b5 is out, are we still in a feature freeze, or are new features
still allowed?  If the freeze is still on, then we need to vote in
any new features.

This particular one is a pretty simple addition which doesn't break
anything as far as I can see.  The description above could be the
basis of the defaults.cc entry, which the patch doesn't have yet.
I vote +1 as well.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] robots.txt not working

[Gilles D. <gr...@sc...>]

According to Andy Lewis:
> Look like the robots.txt file isn't being parsed properly.
>  
> I've used the
> <http://www.jumboclassifieds.com/~alewis/attrs.html#robotstxt_name>
> robotstxt_name tag and added the same name to my robots.txt file and I
> still see the
> default htdig name when indexing.
>  
> Any ideas? Running the lastest beta. Downloaded today.

It seems to me you're confusing the robotstxt_name attribute with
the user_agent attribute.  If by "I still see the default htdig name"
you mean that's what's showing up in the access_log, then you want to
change user_agent.

See http://www.htdig.org/dev/htdig-3.2/attrs.html#user_agent

There is a bug in 3.2.0b5 in that it doesn't correctly handle an empty
Disallow directive, but that doesn't seem to be the issue here.  The fix
for this latter bug is at

ftp://ftp.ccsf.org/htdig-patches/3.2.0b5/robots.0

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

[htdig-dev] Use mktemp in %post of .spec

[Karl E. <ke...@su...>]

Avoid predictable filenames in /tmp.

Fri Nov 14 15:46:33 2003  Karl Eichwalder  <ke...@su...>

	* contrib/htdig-3.2.0.spec (%post): Use 'mktemp'.

--- htdig-3.2.0b5/contrib/htdig-3.2.0.spec.~1~	2002-02-01 23:49:28.000000000 +0100
+++ htdig-3.2.0b5/contrib/htdig-3.2.0.spec	2003-11-14 15:46:06.000000000 +0100
@@ -54,10 +54,11 @@
 	SERVERNAME="`grep '^ServerName' /etc/httpd/conf/httpd.conf | awk 'NR == 1 {print $2}'`"
 	[ -z "$SERVERNAME" ] && SERVERNAME="`hostname -f`"
 	[ -z "$SERVERNAME" ] && SERVERNAME="localhost"
+        TMPFILE=$(mktemp /tmp/ht.XXXXXX) || exit 1
 	sed 's/^start_url:.*/#&\
-# (See end of file for this parameter.)/' /etc/htdig/htdig.conf > /tmp/ht.$$
-	cat /tmp/ht.$$ > /etc/htdig/htdig.conf
-	rm /tmp/ht.$$
+# (See end of file for this parameter.)/' /etc/htdig/htdig.conf > $TMPFILE
+	cat $TMPFILE > /etc/htdig/htdig.conf
+	rm $TMPFILE
 	cat >> /etc/htdig/htdig.conf <<!
 
 # Automatically set up by htdig RPM, from your current Apache httpd.conf...

Diff finished at Fri Nov 14 15:49:03

[htdig-dev] DESTDIR issue in installdir/Makefile.am

[Karl E. <ke...@su...>]

Fri Nov 14 11:50:42 2003  Karl Eichwalder  <ke...@su...>

	* installdir/Makefile.am (install-data-local): Don't add
	$(DESTDIR) part to HtFileType.


--- htdig-3.2.0b5/installdir/Makefile.am.~1~	2003-01-28 12:25:15.000000000 +0100
+++ htdig-3.2.0b5/installdir/Makefile.am	2003-11-13 17:59:23.000000000 +0100
@@ -45,6 +45,6 @@
 	@echo "Installing file-type identification script..."
 	$(mkinstalldirs) $(DESTDIR)$(bindir)
 	@if [ ! -f $(DESTDIR)$(bindir)/HtFileType ]; then \
-		sed -es%@\CONFIG_DIR@%$(DESTDIR)$(CONFIG_DIR)% $(top_srcdir)/installdir/HtFileType >$(DESTDIR)$(bindir)/HtFileType; \
+		sed -e s%@\CONFIG_DIR@%$(CONFIG_DIR)% $(top_srcdir)/installdir/HtFileType >$(DESTDIR)$(bindir)/HtFileType; \
 		chmod 755 $(DESTDIR)$(bindir)/HtFileType; \
 	fi

Diff finished at Thu Nov 13 17:59:30

Re: [htdig-dev] 3.2.0b5 Testing

[Joe R. J. <jj...@cl...>]

On Thu, 13 Nov 2003, Gilles Detillieux wrote:

> Date: Thu, 13 Nov 2003 15:56:36 -0600 (CST)
> From: Gilles Detillieux <gr...@sc...>
> To: Joe R. Jah <jj...@cl...>
> Cc: htd...@li...
> Subject: Re: [htdig-dev] 3.2.0b5 Testing
> 
> According to Joe R. Jah:
> > htfuzzy metaphone dumps core, but it works fine with endings, etc.
> 
> Just a hunch, but I'd guess that you had a metaphone database left over
> from 3.1.6, and that the newer DB code in 3.2.0b5 doesn't like it.  We
> had put some tests for this in some of the other programs, but maybe not
> this one.  Try deleting the databases and running htfuzzy metaphone again.
> The same would probably go for soundex and accents databases, if you use
> either of those.

Spot on!  I had an older db.metaphone.db left over; none of the other
supported algorithms had any problem:
        soundex
        accents
        endings
        synonyms

Regards,

Joe
-- 
     _/   _/_/_/       _/              ____________    __o
     _/   _/   _/      _/         ______________     _-\<,_
 _/  _/   _/_/_/   _/  _/                     ......(_)/ (_)
  _/_/ oe _/   _/.  _/_/ ah        jj...@cl...

[htdig-dev] purge and re-index failing

[Christopher M. <chr...@mc...>]

Greetings htdig folks,

 Recently I've been trying to have htDig purge and re-index items (via a
trigger in Postgres). The purge seems to work as I no longer see the
item in the search results, however, when I try to re-index, I cannot
bring the page back in unless I do a full index. I've just installed
3.2.0b5 hoping that this would help, but no luck. Here's some output
from my command line attempts to get it to work:


[root@lovelace bin]# ./htpurge -c /www/htdig/install/conf/ads.conf -u http://newfind.mcgill.ca/indexes/ads/?AdsID=10266860

[root@lovelace bin]# echo 'http://newfind.mcgill.ca/indexes/ads/?AdsID=1026860' | ./htdig - -s -v -m -c /www/htdig/install/conf/ads.conf

ht://dig Start Time: Thu Nov 13 16:36:02 2003

New server: newfind.mcgill.ca, 80
0:11472:0:http://newfind.mcgill.ca/indexes/ads/?AdsID=1026860:  (changed)  size = 660
htdig: Run complete
htdig: 1 server seen:
htdig:     newfind.mcgill.ca:80 1 document

HTTP statistics
===============
 Persistent connections    : Yes
 HEAD call before GET      : Yes
 Connections opened        : 2
 Connections closed        : 1
 Changes of server         : 0
 HTTP Requests             : 3
 HTTP KBytes requested     : 0.442383
 HTTP Average request time : 0 secs
 HTTP Average speed        : inf KBytes/secs

ht://dig End Time: Thu Nov 13 16:36:03 2003

So although this thing has been purged and re-entered, it no longer
shows up in the query results. Also, it seems that the dbs aren't being
updated after the htperge and htdig. Again more output from my konsole
(note the moddates and filesizes - also, the filesize of db.docdb
doesn't change between the purge and re-index):

[root@lovelace bin]# ls -ltr /www/htdig/install/var/ads
total 1584
-rw-r--r--    1 root     root        24576 Nov 13 13:35 db.excerpts.work
-rw-r--r--    1 root     root        24576 Nov 13 13:35 db.docs.index.work
-rw-r--r--    1 root     root        24576 Nov 13 13:35 db.docdb.work
-rw-r--r--    1 root     root        16384 Nov 13 16:14 db.words.db_weakcmpr
-rw-r--r--    1 root     root       619520 Nov 13 16:36 db.words.db
-rw-r--r--    1 root     root       655360 Nov 13 16:36 db.excerpts
-rw-r--r--    1 root     root       172032 Nov 13 16:36 db.docs.index
-rw-r--r--    1 root     root       344064 Nov 13 16:38 db.docdb

[root@lovelace bin]# ./htpurge -c /www/htdig/install/conf/ads.conf -u http://newfind.mcgill.ca/indexes/ads/?AdsID=1025825

[root@lovelace bin]# echo 'http://newfind.mcgill.ca/indexes/ads/?AdsID=1025825' | ./htdig - -s -v -m -c /www/htdig/install/conf/ads.conf

ht://dig Start Time: Thu Nov 13 17:05:14 2003

New server: newfind.mcgill.ca, 80
0:11475:0:http://newfind.mcgill.ca/indexes/ads/?AdsID=1025825:  (changed)  size = 336
htdig: Run complete
htdig: 1 server seen:
htdig:     newfind.mcgill.ca:80 1 document

HTTP statistics
===============
 Persistent connections    : Yes
 HEAD call before GET      : Yes
 Connections opened        : 2
 Connections closed        : 1
 Changes of server         : 0
 HTTP Requests             : 3
 HTTP KBytes requested     : 0.442383
 HTTP Average request time : 0 secs
 HTTP Average speed        : inf KBytes/secs

ht://dig End Time: Thu Nov 13 17:05:14 2003

[root@lovelace bin]# ls -ltr /www/htdig/install/var/ads
total 1584
-rw-r--r--    1 root     root        24576 Nov 13 13:35 db.excerpts.work
-rw-r--r--    1 root     root        24576 Nov 13 13:35 db.docs.index.work
-rw-r--r--    1 root     root        24576 Nov 13 13:35 db.docdb.work
-rw-r--r--    1 root     root        16384 Nov 13 16:14 db.words.db_weakcmpr
-rw-r--r--    1 root     root       619520 Nov 13 16:36 db.words.db
-rw-r--r--    1 root     root       655360 Nov 13 16:36 db.excerpts
-rw-r--r--    1 root     root       172032 Nov 13 16:36 db.docs.index
-rw-r--r--    1 root     root       344064 Nov 13 17:05 db.docdb


So, and info or help on this would be much appreciated.

Cheers,

Chris

-- 
Christopher Murtagh
Enterprise Systems Administrator
ISR / Web Communications Group 
McGill University
Montreal, Quebec
Canada

Tel.: (514) 398-3122
Fax:  (514) 398-2017

Re: [htdig-dev] 3.2.0b5 Testing

[Gilles D. <gr...@sc...>]

According to Joe R. Jah:
> htfuzzy metaphone dumps core, but it works fine with endings, etc.

Just a hunch, but I'd guess that you had a metaphone database left over
from 3.1.6, and that the newer DB code in 3.2.0b5 doesn't like it.  We
had put some tests for this in some of the other programs, but maybe not
this one.  Try deleting the databases and running htfuzzy metaphone again.
The same would probably go for soundex and accents databases, if you use
either of those.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] Almost there...

[Joe R. J. <jj...@cl...>]

On Thu, 13 Nov 2003, Gilles Detillieux wrote:

> Date: Thu, 13 Nov 2003 13:38:05 -0600 (CST)
> From: Gilles Detillieux <gr...@sc...>
> To: Joe R. Jah <jj...@cl...>
> Cc: htd...@li...
> Subject: Re: [htdig-dev] Almost there...
> 
> According to Joe R. Jah:
> > Job well done!  It configured/built/ran out of the box on my BSD/OS-4.3.1
> > with gcc 2.95.3 like a charm; It took only 96 minutes to index my site;)
> 
> How does this compare to earlier 3.2.0b4 snapshots, and to 3.1.6?
> Is 3.2.0b5 significantly slower than 3.1 releases, and is it better or
> worse than earlier 3.2 betas?

First of all I should correct the indexing time; that one was sent in
hurry to express my joy;) and didn't realize that it was indexing the site
twice; once for http and again for https;( I added a rewrite rule:

url_rewrite_rules:      https://(.*)                    http://\\1

And now 3.2.0b5 indexes my site, ~15,000 docs, in 54 minutes, even more
joyous;))  For comparison fully patched 3.1.6 indexes it in 12 minutes;
however, it indexes more pages because of the fileSpce.1 patch.  
Unfortunately in our site we have many file names that include space in
them.  Roughly about 5% more documents are indexed by my 3.1.6 than
3.2.0b5.  I'd say it takes five times longer for 3.2.0b5 to index the
site.

I can't directly compare the results of 3.2.0b5 with 3.2.0b4 because my
old statistics were taken on a slower machine.  Here is an old statistics
I have posted to the list:

Machine:        300 MHz PentiumII
RAM:            256 MB
SWAP:           768 MB
OS:             BSDI 4.01
Documents:      ~5,000

With different versions of htdig: 

3.1.5           11 Minutes
3.2.0b3         9 1/2 hours
3.2.0b4-031201  29 hours and 20 minutes
3.2.0b4-040801  > 12 days

You can see that 3.2.0b5's performance has greatly improved.

Regards,

Joe
-- 
     _/   _/_/_/       _/              ____________    __o
     _/   _/   _/      _/         ______________     _-\<,_
 _/  _/   _/_/_/   _/  _/                     ......(_)/ (_)
  _/_/ oe _/   _/.  _/_/ ah        jj...@cl...

Re: [htdig-dev] Next steps

[Gabriele B. <bar...@in...>]

At 13.16 13/11/2003 -0600, Gilles Detillieux wrote:
>Is someone going to post an announcement about this release on htdig-general
>and htdig-announce?  (Or has this been done and I missed it?)

No Gilles, I forgot. I will do it asap.

Ciao ciao,
-Gabriele
--
Gabriele Bartolini: Web Programmer, ht://Dig & IWA/HWG Member, ht://Check 
maintainer
Current Location: Melbourne, Victoria, Australia
bar...@in... | http://www.prato.linux.it/~gbartolini | ICQ#129221447
 > "Leave every hope, ye who enter!", Dante Alighieri, Divine Comedy, The 
Inferno

Re: [htdig-dev] Almost there...

[Gilles D. <gr...@sc...>]

According to Joe R. Jah:
> Job well done!  It configured/built/ran out of the box on my BSD/OS-4.3.1
> with gcc 2.95.3 like a charm; It took only 96 minutes to index my site;)

How does this compare to earlier 3.2.0b4 snapshots, and to 3.1.6?
Is 3.2.0b5 significantly slower than 3.1 releases, and is it better or
worse than earlier 3.2 betas?

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] Next steps

[Gilles D. <gr...@sc...>]

According to Gabriele Bartolini:
>     I am going over the release files right now. I will check that 
> everything is fine and then:
> 
> 1) update the CVS tag
> 2) rebuild the package
> 3) put it:
> 	- under the files section of the site
> 	- on sourceforge.net
> 4) publish it on:
> 	- sf.net	
> 	- freshmeat.net
> 	- and so on ...
> 
> Is that fine?

Is someone going to post an announcement about this release on htdig-general
and htdig-announce?  (Or has this been done and I missed it?)

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] 3.2.0b5 Testing

[Joe R. J. <jj...@cl...>]

Hi Folks,

htfuzzy metaphone dumps core, but it works fine with endings, etc.

gdb htfuzzy htfuzzy.core
GNU gdb 
Copyright 1998 Free Software Foundation, Inc.
This GDB was configured as "i386-unknown-bsdi4.3"...
Core was generated by `htfuzzy'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /usr/local/htdig/3.2/lib/htdig/libfuzzy-3.2.0.so...done.
Reading symbols from /usr/local/htdig/3.2/lib/htdig/libhtnet-3.2.0.so...done.
Reading symbols from /usr/local/htdig/3.2/lib/htdig/libcommon-3.2.0.so...done.
Reading symbols from /usr/local/htdig/3.2/lib/htdig/libhtword-3.2.0.so...done.
Reading symbols from /usr/local/htdig/3.2/lib/htdig_db/libhtdb-3.2.0.so...done.
Reading symbols from /usr/local/htdig/3.2/lib/htdig/libht-3.2.0.so...done.
Reading symbols from /usr/lib/libz.so...done.
Reading symbols from /usr/lib/libstdc++.so.1...done.
Reading symbols from /shlib/libm.so.0.0...done.
Reading symbols from /shlib/libgcc.so.1...done.
Reading symbols from /shlib/libc.so.2...done.
Reading symbols from /shlib/ld-bsdi.so...done.
#0  0x0 in ?? ()
(gdb) bt
#0  0x0 in ?? ()
#1  0x48132bd4 in CDB___ham_call_hash (hcp=0x9e07180, k=0x9e06220 "TLFNBR", len=6) at hash.c:1146
#2  0x48133604 in CDB___ham_lookup (dbc=0x9d74400, key=0x80478fc, sought=42, mode=DB_LOCK_WRITE) at hash.c:1641
#3  0x481326b3 in CDB___ham_c_put (dbc=0x9d74400, key=0x80478fc, data=0x80478e4, flags=15) at hash.c:953
#4  0x48122eac in CDB___db_put (dbp=0x9d6f800, txn=0x0, key=0x80478fc, data=0x80478e4, flags=0) at db_am.c:508
#5  0x4816627e in DB2_db::Put (this=0x9e07000, key=@0x8047964, data=@0x9c69570) at DB2_db.cc:266
#6  0x48059650 in Fuzzy::writeDB (this=0x804d0c0) at Fuzzy.cc:161
#7  0x804959e in main (ac=3, av=0x8047c58) at htfuzzy.cc:185
#8  0x8048f53 in __start ()
(gdb) q

Regards,

Joe
-- 
     _/   _/_/_/       _/              ____________    __o
     _/   _/   _/      _/         ______________     _-\<,_
 _/  _/   _/_/_/   _/  _/                     ......(_)/ (_)
  _/_/ oe _/   _/.  _/_/ ah        jj...@cl...

Re: [htdig-dev] 3.2.0b5 Testing

[Gilles D. <gr...@sc...>]

According to Jim Cole:
> On Nov 12, 2003, at 3:28 PM, Gilles Detillieux wrote:
> > According to Jim Cole:
> >> I did come across one minor bug in Display.cc involving start_ellipses
> >> and end_ellipses; but I think that it only manifests itself when
> >> max_excerpts is greater than one (a non-default setting). I will post 
> >> a
> >> bug report later this week.
> >
> > I think I see the cause of this bug.  In Display::buildExcerpts() it
> > mistakenly uses config->Value() instead of config->Find() to get the
> > string from the start_ellipses and end_ellipses attributes.  This code 
> > is
> > only used if max_excerpts is not 1 and the search word(s) is(are) 
> > found.
> > So did the ellipses simply not show up, or did a 0 appear in their 
> > place?
> 
> Your diagnosis is correct. I came to the same conclusion, and changing 
> from Value to Find did in fact solve the problem. Prior to the change a 
> 0 appeared in place of the ellipses. I was just holding off submitting 
> an official bug report until I had a chance to look at the 3.1.x code 
> and see what changed. I just took a quick look and it appears that at 
> some point there was a move away from using operator[] to access the 
> start_ and end_ellipses settings, at which time Value was substituted.

Yes, the operator [] works on a {Ht,}Configuration object, but not on a
pointer to that object.  In 3.2, we moved to using a pointer to the object
in most of the code, so config["foo"] had to become config->Find("foo").
I think the changes were made in one fell swoop, but as stuff got moved
back and forth between the 3.1 and 3.2 branches later, there was of
course the potential of getting it wrong when manually going from one
to the other.  I guess that's what the beta testing is for!

For completeness, here's the patch...

--- htsearch/Display.cc.orig	2003-10-25 07:40:23.000000000 -0500
+++ htsearch/Display.cc	2003-11-13 11:33:43.000000000 -0600
@@ -1776,7 +1776,7 @@
     }
     else
     {
-      text << config->Value("start_ellipses");
+      text << config->Find("start_ellipses");
 
       while ( *start && HtIsStrictWordChar( *start ) )
       {
@@ -1806,7 +1806,7 @@
       *end = '\0';
 
       text << hilight(match, start, urlanchor, fanchor);
-      text << config->Value("end_ellipses");
+      text << config->Find("end_ellipses");
 
       *end = endChar;
     }

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] 3.2.0b5 Testing

[Jim C. <li...@yg...>]

On Nov 12, 2003, at 3:28 PM, Gilles Detillieux wrote:

> According to Jim Cole:
>> I did come across one minor bug in Display.cc involving start_ellipses
>> and end_ellipses; but I think that it only manifests itself when
>> max_excerpts is greater than one (a non-default setting). I will post 
>> a
>> bug report later this week.
>
> I think I see the cause of this bug.  In Display::buildExcerpts() it
> mistakenly uses config->Value() instead of config->Find() to get the
> string from the start_ellipses and end_ellipses attributes.  This code 
> is
> only used if max_excerpts is not 1 and the search word(s) is(are) 
> found.
> So did the ellipses simply not show up, or did a 0 appear in their 
> place?

Your diagnosis is correct. I came to the same conclusion, and changing 
from Value to Find did in fact solve the problem. Prior to the change a 
0 appeared in place of the ellipses. I was just holding off submitting 
an official bug report until I had a chance to look at the 3.1.x code 
and see what changed. I just took a quick look and it appears that at 
some point there was a move away from using operator[] to access the 
start_ and end_ellipses settings, at which time Value was substituted.

Jim

[htdig-dev] robots.txt not working

[Andy L. <ju...@co...>]

All,
 
Look like the robots.txt file isn't being parsed properly.
 
I've used the
<http://www.jumboclassifieds.com/~alewis/attrs.html#robotstxt_name>
robotstxt_name tag and added the same name to my robots.txt file and I
still see the
default htdig name when indexing.
 
Any ideas? Running the lastest beta. Downloaded today.
 
Thanks in advance,
 
Andy

Re: [htdig-dev] 3.2.0b5 Testing

[Gilles D. <gr...@sc...>]

According to Jim Cole:
> I did come across one minor bug in Display.cc involving start_ellipses 
> and end_ellipses; but I think that it only manifests itself when 
> max_excerpts is greater than one (a non-default setting). I will post a 
> bug report later this week.

I think I see the cause of this bug.  In Display::buildExcerpts() it
mistakenly uses config->Value() instead of config->Find() to get the
string from the start_ellipses and end_ellipses attributes.  This code is
only used if max_excerpts is not 1 and the search word(s) is(are) found.
So did the ellipses simply not show up, or did a 0 appear in their place?

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

[htdig-dev] Htdig 3.1.6: The structure of the wordsDB

[<cm...@fr...>]

Hi!<br>
<br>
I am trying to write a htsearch applikation in JAVA. The problem is that i =
do not understand the structure of the wordsDB in htdig. I can connect to t=
he DB, but I receive data-values that make no sense to me.<br>
<br>
I am searching now for a documentation where the format of the wordsDB (db.=
words.db) is explained or a hint that shows me how to extract the key/data =
pairs that i get from the DB.<br>
<br>
Does somebody have a link or could explain the structure of this file?<br>
<br>
Cheers,<br>
Chris=0A=0A=0A<br>--<br><!-- AdSpace freenet EMO Webmail Banner --> =0A<a h=
ref=3D"http://adserver.freenet.de/click.ng/site=3Dfn&prod=3Dchetools&kat=3D=
rub&tbl=3Dwebmail&ppos=3D1&TransactionID=3D1068638242124424&rgtg=3D256" tar=
get=3D"_blank"><img src=3D"http://adserver.freenet.de/image.ng/site=3Dfn&pr=
od=3Dchetools&kat=3Drub&tbl=3Dwebmail&ppos=3D1&TransactionID=3D106863824212=
4424&rgtg=3D256" border=3D"0"></a><!-- / AdSpace -->=0A

Re: [htdig-dev] Next steps

[Gabriele B. <bar...@in...>]

>I CAN'T DO IT. We need Geoff because he's the owner of the project on 
>Freshmeat. :-(

Geoff kindly added me to freshmeat and I posted it there as well. Another 
little brick ... :-)

Ciao ciao
-Gabriele
--
Gabriele Bartolini: Web Programmer, ht://Dig & IWA/HWG Member, ht://Check 
maintainer
Current Location: Melbourne, Victoria, Australia
bar...@in... | http://www.prato.linux.it/~gbartolini | ICQ#129221447
 > "Leave every hope, ye who enter!", Dante Alighieri, Divine Comedy, The 
Inferno

[htdig-dev] 3.2.0b5 Testing

[Jim C. <li...@yg...>]

Hi - I built and tested the source tarball on OS X this morning. It 
passed all of the built in tests and seemed to work fine for a small 
site I was working with. I also did some more extensive testing with an 
old data set that used to kill htdig under OS X; no problems there. 
Sometime in the next week or two, I will try to install 3.2.0b5 on some 
high traffic Linux servers I maintain in order to beat on it a little 
more.

I did come across one minor bug in Display.cc involving start_ellipses 
and end_ellipses; but I think that it only manifests itself when 
max_excerpts is greater than one (a non-default setting). I will post a 
bug report later this week.

Thanks to everyone who contributed to finally getting a new release out 
the door! Sorry that I couldn't help more; it was sort of bad timing 
for me. I will hopefully have some free time later this month, so if 
there is an emerging todo list for rc1 (or whatever the next release 
is)...

Jim

Re: [htdig-dev] Almost there...

[Joe R. J. <jj...@cl...>]

On Mon, 10 Nov 2003, Neal Richter wrote:

> Date: Mon, 10 Nov 2003 14:54:37 -0700 (MST)
> From: Neal Richter <ne...@ri...>
> To: Lachlan Andrew <lh...@us...>
> Cc: Gilles Detillieux <gr...@sc...>, htd...@li...
> Subject: Re: [htdig-dev] Almost there...
> 
> 
> Nice!  Fantastic!

Job well done!  It configured/built/ran out of the box on my BSD/OS-4.3.1
with gcc 2.95.3 like a charm; It took only 96 minutes to index my site;)

> Joe R Jah?  Looks like he maintains a ftp patch download area @
> ftp://ftp.ccsf.org/htdig-patches/

Patches can be uploaded also.  Upload to ftp://ftp.ccsf.org/incoming/
I look forward to create 3.2.0b5 folder, and 3.2.0 soon there after;)

Regards,

Joe
-- 
     _/   _/_/_/       _/              ____________    __o
     _/   _/   _/      _/         ______________     _-\<,_
 _/  _/   _/_/_/   _/  _/                     ......(_)/ (_)
  _/_/ oe _/   _/.  _/_/ ah        jj...@cl...

Re: [htdig-dev] Next steps

[Gabriele B. <bar...@in...>]

>4) publish it on:
>         - sf.net

DONE! :-)

>         - freshmeat.net

I CAN'T DO IT. We need Geoff because he's the owner of the project on 
Freshmeat. :-(

Ciao ciao,
-Gabriele
--
Gabriele Bartolini: Web Programmer, ht://Dig & IWA/HWG Member, ht://Check 
maintainer
Current Location: Melbourne, Victoria, Australia
bar...@in... | http://www.prato.linux.it/~gbartolini | ICQ#129221447
 > "Leave every hope, ye who enter!", Dante Alighieri, Divine Comedy, The 
Inferno

[htdig-dev] Next steps

[Gabriele B. <bar...@in...>]

Hi guys,

    I am going over the release files right now. I will check that 
everything is fine and then:

1) update the CVS tag
2) rebuild the package
3) put it:
	- under the files section of the site
	- on sourceforge.net
4) publish it on:
	- sf.net	
	- freshmeat.net
	- and so on ...

Is that fine?

Also, let's not forget before 3.2.0RC1 goes out to update the Release Notes 
in order to include all the 'new' configuration attributes with - possibly 
- a link to them.

Ciao ciao,
-Gabriele
	
--
Gabriele Bartolini: Web Programmer, ht://Dig & IWA/HWG Member, ht://Check 
maintainer
Current Location: Melbourne, Victoria, Australia
bar...@in... | http://www.prato.linux.it/~gbartolini | ICQ#129221447
 > "Leave every hope, ye who enter!", Dante Alighieri, Divine Comedy, The 
Inferno

Re: [htdig-dev] Almost there...

[Gabriele B. <bar...@in...>]

At 14.54 10/11/2003 -0700, Neal Richter wrote:
>Nice thing is that I've been working on libhtdig again recently!

Great ... I think I need your 'help' for this for the next release, as I 
wanna put everything under autotools (so we can make 'make dist' work).

>Updated it in HEAD.

Yep ... I did it. I am gonna update tags right now according to Lachlan's 
changes.

Ciao ciao,
-Gabriele
--
Gabriele Bartolini: Web Programmer, ht://Dig & IWA/HWG Member, ht://Check 
maintainer
Current Location: Melbourne, Victoria, Australia
bar...@in... | http://www.prato.linux.it/~gbartolini | ICQ#129221447
 > "Leave every hope, ye who enter!", Dante Alighieri, Divine Comedy, The 
Inferno

Re: [htdig-dev] Release of 3.2.0b5 :)

[Gabriele B. <bar...@in...>]

At 23.05 09/11/2003 -0600, Gilles Detillieux wrote:
>What's missing in this script is Geoff's script to fix up all the file
>permissions.  I think this is a must for a release tarball.

OK. Got it. I will merge Geoff's and Lachlan's fixes and put it in the 
scripts dir.

-Gabriele
--
Gabriele Bartolini: Web Programmer, ht://Dig & IWA/HWG Member, ht://Check 
maintainer
Current Location: Melbourne, Victoria, Australia
bar...@in... | http://www.prato.linux.it/~gbartolini | ICQ#129221447
 > "Leave every hope, ye who enter!", Dante Alighieri, Divine Comedy, The 
Inferno