htdig-dev — Developer Discussion for the ht://Dig project

[htdig-dev] Re: [ htdig-Bugs-806419 ] Phrase beginning badwords

[Lachlan A. <lh...@us...>]

Scratch that...  Try the attached patch.

Regards,
Lachlan

On Sat, 1 Nov 2003 20:29, Lachlan Andrew wrote:
> The problem is more complicated than just
> beginning with a badword; a search for "des Ressources" succeeds.
>
> Could you please mail me the result of running

--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

[htdig-dev] Re: [ htdig-Bugs-806419 ] Phrase beginning badwords

[Lachlan A. <lh...@us...>]

Thanks for that, Franck.  The problem is more complicated than just=20
beginning with a badword; a search for "des Ressources" succeeds.

Could you please mail me the result of running
=09htsearch -vvvvv words=3D%22des+Ressources+Humaines%22
and
=09htsearch -vvvvv words=3D%22des+Ressources%22
?  The -v option sends output to standard error, which I can't view=20
remotely.

In the mean time, I'll see if I can reproduce the bug on my local=20
system.

Cheers,
Lachlan

On Fri, 31 Oct 2003 03:41, Franck Euzenot wrote:

> Here the URL    : http://www.dr20.cnrs.fr/cgi-bin/htsearch
> and the Keyword : "des Ressources Humaines Vous =EAtes ici"

--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

Re: [htdig-dev] Failed tests on OS X Panther with make check

[Lachlan A. <lh...@us...>]

That is *execute* permission not being set.  Just
    chmod 755 test/t_url
before running  make check.

Lachlan

On Thu, 30 Oct 2003 00:36, Ted Stresen-Reuter wrote:
> I did everything as sudo, first, because I know there are some
> permission issues in the version that comes out of CVS
--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

[htdig-dev] OT: Gabriele and Lachlan in Melbourne

[Gabriele B. <bar...@in...>]

Oooops ... the picture! :-)

-Gabriele

[htdig-dev] OT: Gabriele and Lachlan in Melbourne

[Gabriele B. <bar...@in...>]

Hi guys,

    just a quick off topic message. Yesterday I caught up with Lachlan here 
in Melbourne; it was nice to have a pizza together in the most famous 
italian street of Australia! :-)

    Anyway, we talked a bit about ht://Dig too, looking forward to seeing 
the new version out soon! I am sending a picture of the two us; I am the 
one on the left and Lachlan is of course on the right of the picture.

See ya,
-Gabriele
--
Gabriele Bartolini: Web Programmer, ht://Dig & IWA/HWG Member, ht://Check 
maintainer
Current Location: Melbourne, Victoria, Australia
bar...@in... | http://www.prato.linux.it/~gbartolini | ICQ#129221447
 > "Leave every hope, ye who enter!", Dante Alighieri, Divine Comedy, The 
Inferno

Re: [htdig-dev] Failed tests on OS X Panther with make check

[Ted Stresen-R. <ted...@ma...>]

I did everything as sudo, first, because I know there are some=20
permission issues in the version that comes out of CVS (I believe it is=20=

formally known as the Head branch). When those tests failed, I did=20
chown -R tedsr:admin * while inside the htdig source dir and tried=20
again with using sudo, and from this last run is the report I sent=20
along.

I went ahead and did make make install and everything seems to have=20
gone just fine. I haven't manually run the tests in make check to see=20
if they still fail, but the make and installation completed without=20
errors and simple searches show that the system is installed and=20
running.

As a side note, I'm having generally very good experiences with=20
Panther. There are a few oddities but the new features are compelling=20
enough to make up for them, and a whole lot more. I basically updated=20
for the new Finder and Expos=E9 (in addition to the improved Windows=20
compatibility since I work in a Windows environment). I'm a happy=20
consumer at this point...

Ted

On Oct 29, 2003, at 1:11 AM, Jim Cole wrote:

> You weren't by any chance logged in as root when you ran these tests?
>
> No Panther for me yet. I learned my lesson last time when I jumped the=20=

> gun with a same-day Panther install ;)
>
> Jim
>
> On Monday, October 27, 2003, at 08:25 PM, Ted Stresen-Reuter wrote:
>
>> hi,
>>
>> make check produces the following output on OS X Panther (I couldn't=20=

>> wait to install it...) I haven't tried making all and make install,=20=

>> yet, but I will. These failed test might just be problems with how=20
>> Apple sets up Apache.
>>
>> PASS: t_wordkey
>> PASS: t_wordlist
>> PASS: t_wordskip
>> PASS: t_wordbitstream
>> PASS: t_search
>> PASS: t_htdb
>> PASS: t_rdonly
>> PASS: t_trunc
>> PASS: t_url
>> Error:  Apache has not been designed to serve pages while
>>         running as root.  There are known race conditions that
>>         will allow any local user to read any file on the system.
>>         If you still desire to serve pages as root then
>>         add -DBIG_SECURITY_HOLE to the EXTRA_CFLAGS line in your
>>         src/Configuration file and rebuild the server.  It is
>>         strongly suggested that you instead modify the User
>>         directive in your httpd.conf file to list a non-root
>>         user.
>>
>> ****Could not start apache. This test may fail, but that is not=20
>> ht://Dig's fault
>>
>> htpurge: Database is empty!
>>
>> running htdig: expected
>> http://localhost:7400/set1/
>> http://localhost:7400/set1/bad_local.htm
>> http://localhost:7400/set1/script.html
>> http://localhost:7400/set1/site%201.html
>> http://localhost:7400/set1/site2.html
>> http://localhost:7400/set1/site3.html
>> http://localhost:7400/set1/site4.html
>> http://localhost:7400/set1/sub%2520dir/
>> http://localhost:7400/set1/sub%2520dir/empty%20file.html
>> http://localhost:7400/set1/title.html
>> but got
>>
>> FAIL: t_htdig
>> Error:  Apache has not been designed to serve pages while
>>         running as root.  There are known race conditions that
>>         will allow any local user to read any file on the system.
>>         If you still desire to serve pages as root then
>>         add -DBIG_SECURITY_HOLE to the EXTRA_CFLAGS line in your
>>         src/Configuration file and rebuild the server.  It is
>>         strongly suggested that you instead modify the User
>>         directive in your httpd.conf file to list a non-root
>>         user.
>>
>> ****Could not start apache. This test may fail, but that is not=20
>> ht://Dig's fault
>>
>> htpurge: Database is empty!
>>
>> running htdig: expected
>> http://localhost:7400/set1/
>> http://localhost:7400/set1/bad_local.htm
>> http://localhost:7400/set1/script.html
>> http://localhost:7400/set1/site%201.html
>> http://localhost:7400/set1/site2.html
>> http://localhost:7400/set1/site3.html
>> http://localhost:7400/set1/site4.html
>> http://localhost:7400/set1/sub%2520dir/
>> http://localhost:7400/set1/sub%2520dir/empty%20file.html
>> http://localhost:7400/set1/title.html
>> but got
>>
>> FAIL: t_htsearch
>> PASS: t_htmerge
>> Error:  Apache has not been designed to serve pages while
>>         running as root.  There are known race conditions that
>>         will allow any local user to read any file on the system.
>>         If you still desire to serve pages as root then
>>         add -DBIG_SECURITY_HOLE to the EXTRA_CFLAGS line in your
>>         src/Configuration file and rebuild the server.  It is
>>         strongly suggested that you instead modify the User
>>         directive in your httpd.conf file to list a non-root
>>         user.
>>
>> ****Could not start apache. This test may fail, but that is not=20
>> ht://Dig's fault
>>
>> Could not fetch URL
>> FAIL: t_htnet
>> Error:  Apache has not been designed to serve pages while
>>         running as root.  There are known race conditions that
>>         will allow any local user to read any file on the system.
>>         If you still desire to serve pages as root then
>>         add -DBIG_SECURITY_HOLE to the EXTRA_CFLAGS line in your
>>         src/Configuration file and rebuild the server.  It is
>>         strongly suggested that you instead modify the User
>>         directive in your httpd.conf file to list a non-root
>>         user.
>>
>> ****Could not start apache. This test may fail, but that is not=20
>> ht://Dig's fault
>>
>> running htdig: expected
>> http://localhost:7400/set1/
>> http://localhost:7400/set1/bad_local.htm
>> http://localhost:7400/set1/script.html
>> http://localhost:7400/set1/site%201.html
>> http://localhost:7400/set1/site2.html
>> http://localhost:7400/set1/site3.html
>> http://localhost:7400/set1/site4.html
>> http://localhost:7400/set1/sub%2520dir/
>> http://localhost:7400/set1/sub%2520dir/empty%20file.html
>> http://localhost:7400/set1/title.html
>> but got
>> http://localhost:7400/set1/
>> http://localhost:7400/set1/script.html
>> http://localhost:7400/set1/site%201.html
>> http://localhost:7400/set1/site2.html
>> http://localhost:7400/set1/site3.html
>> http://localhost:7400/set1/site4.html
>> http://localhost:7400/set1/title.html
>> FAIL: t_htdig_local
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> 4 of 14 tests failed
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> make[2]: *** [check-TESTS] Error 1
>> make[1]: *** [check-am] Error 2
>> make: *** [check-recursive] Error 1
>>
>>
>>
>> -------------------------------------------------------
>> This SF.net email is sponsored by: The SF.net Donation Program.
>> Do you like what SourceForge.net is doing for the Open
>> Source Community?  Make a contribution, and help us add new
>> features and functionality. Click here: =
http://sourceforge.net/donate/
>> _______________________________________________
>> ht://Dig Developer mailing list:
>> htd...@li...
>> List information (subscribe/unsubscribe, etc.)
>> https://lists.sourceforge.net/lists/listinfo/htdig-dev
>

Re: [htdig-dev] Failed tests on OS X Panther with make check

[Jim C. <li...@yg...>]

You weren't by any chance logged in as root when you ran these tests?

No Panther for me yet. I learned my lesson last time when I jumped the 
gun with a same-day Panther install ;)

Jim

On Monday, October 27, 2003, at 08:25 PM, Ted Stresen-Reuter wrote:

> hi,
>
> make check produces the following output on OS X Panther (I couldn't 
> wait to install it...) I haven't tried making all and make install, 
> yet, but I will. These failed test might just be problems with how 
> Apple sets up Apache.
>
> PASS: t_wordkey
> PASS: t_wordlist
> PASS: t_wordskip
> PASS: t_wordbitstream
> PASS: t_search
> PASS: t_htdb
> PASS: t_rdonly
> PASS: t_trunc
> PASS: t_url
> Error:  Apache has not been designed to serve pages while
>         running as root.  There are known race conditions that
>         will allow any local user to read any file on the system.
>         If you still desire to serve pages as root then
>         add -DBIG_SECURITY_HOLE to the EXTRA_CFLAGS line in your
>         src/Configuration file and rebuild the server.  It is
>         strongly suggested that you instead modify the User
>         directive in your httpd.conf file to list a non-root
>         user.
>
> ****Could not start apache. This test may fail, but that is not 
> ht://Dig's fault
>
> htpurge: Database is empty!
>
> running htdig: expected
> http://localhost:7400/set1/
> http://localhost:7400/set1/bad_local.htm
> http://localhost:7400/set1/script.html
> http://localhost:7400/set1/site%201.html
> http://localhost:7400/set1/site2.html
> http://localhost:7400/set1/site3.html
> http://localhost:7400/set1/site4.html
> http://localhost:7400/set1/sub%2520dir/
> http://localhost:7400/set1/sub%2520dir/empty%20file.html
> http://localhost:7400/set1/title.html
> but got
>
> FAIL: t_htdig
> Error:  Apache has not been designed to serve pages while
>         running as root.  There are known race conditions that
>         will allow any local user to read any file on the system.
>         If you still desire to serve pages as root then
>         add -DBIG_SECURITY_HOLE to the EXTRA_CFLAGS line in your
>         src/Configuration file and rebuild the server.  It is
>         strongly suggested that you instead modify the User
>         directive in your httpd.conf file to list a non-root
>         user.
>
> ****Could not start apache. This test may fail, but that is not 
> ht://Dig's fault
>
> htpurge: Database is empty!
>
> running htdig: expected
> http://localhost:7400/set1/
> http://localhost:7400/set1/bad_local.htm
> http://localhost:7400/set1/script.html
> http://localhost:7400/set1/site%201.html
> http://localhost:7400/set1/site2.html
> http://localhost:7400/set1/site3.html
> http://localhost:7400/set1/site4.html
> http://localhost:7400/set1/sub%2520dir/
> http://localhost:7400/set1/sub%2520dir/empty%20file.html
> http://localhost:7400/set1/title.html
> but got
>
> FAIL: t_htsearch
> PASS: t_htmerge
> Error:  Apache has not been designed to serve pages while
>         running as root.  There are known race conditions that
>         will allow any local user to read any file on the system.
>         If you still desire to serve pages as root then
>         add -DBIG_SECURITY_HOLE to the EXTRA_CFLAGS line in your
>         src/Configuration file and rebuild the server.  It is
>         strongly suggested that you instead modify the User
>         directive in your httpd.conf file to list a non-root
>         user.
>
> ****Could not start apache. This test may fail, but that is not 
> ht://Dig's fault
>
> Could not fetch URL
> FAIL: t_htnet
> Error:  Apache has not been designed to serve pages while
>         running as root.  There are known race conditions that
>         will allow any local user to read any file on the system.
>         If you still desire to serve pages as root then
>         add -DBIG_SECURITY_HOLE to the EXTRA_CFLAGS line in your
>         src/Configuration file and rebuild the server.  It is
>         strongly suggested that you instead modify the User
>         directive in your httpd.conf file to list a non-root
>         user.
>
> ****Could not start apache. This test may fail, but that is not 
> ht://Dig's fault
>
> running htdig: expected
> http://localhost:7400/set1/
> http://localhost:7400/set1/bad_local.htm
> http://localhost:7400/set1/script.html
> http://localhost:7400/set1/site%201.html
> http://localhost:7400/set1/site2.html
> http://localhost:7400/set1/site3.html
> http://localhost:7400/set1/site4.html
> http://localhost:7400/set1/sub%2520dir/
> http://localhost:7400/set1/sub%2520dir/empty%20file.html
> http://localhost:7400/set1/title.html
> but got
> http://localhost:7400/set1/
> http://localhost:7400/set1/script.html
> http://localhost:7400/set1/site%201.html
> http://localhost:7400/set1/site2.html
> http://localhost:7400/set1/site3.html
> http://localhost:7400/set1/site4.html
> http://localhost:7400/set1/title.html
> FAIL: t_htdig_local
> ====================
> 4 of 14 tests failed
> ====================
> make[2]: *** [check-TESTS] Error 1
> make[1]: *** [check-am] Error 2
> make: *** [check-recursive] Error 1
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: The SF.net Donation Program.
> Do you like what SourceForge.net is doing for the Open
> Source Community?  Make a contribution, and help us add new
> features and functionality. Click here: http://sourceforge.net/donate/
> _______________________________________________
> ht://Dig Developer mailing list:
> htd...@li...
> List information (subscribe/unsubscribe, etc.)
> https://lists.sourceforge.net/lists/listinfo/htdig-dev

Re: [htdig-dev] patch for bug #504087, duplicate URLs in htsearch collections

[Gilles D. <gr...@sc...>]

Yesterday, I wrote:
> I've pretty much run out of time to help out with this release, but before
> I leave you, I thought I'd submit the following patch for your testing and
> approval.  It should fix the duplicate URL problem in htsearch collections,
> in bug #504087.  I'm not sure what sort of performance impact it will have
> on large databases with lots of potential matches, though.  That may be
> something to consider/test for.
> 
> Cheers,
> Gilles
> 
> --- htsearch/Display.cc.orig	2003-10-25 07:40:23.000000000 -0500
> +++ htsearch/Display.cc	2003-10-27 17:55:52.000000000 -0600
> @@ -1895,6 +1895,27 @@ Display::sort(List *matches)
>      qsort((char *) array, numberOfMatches, sizeof(ResultMatch *),
>  	  array[0]->getSortFun());
>  
> +    // In case there are duplicate URLs across collections, keep "best" ones
> +    // after sorting them.
> +    Dictionary	goturl;
> +    String	url;
> +    int		j = 0;
> +    for (i = 0; i < numberOfMatches; i++)
> +    {
> +	Collection *collection = array[i]->getCollection();
> +	DocumentRef *ref = collection->getDocumentRef(array[i]->getID());
> +	url = ref->DocURL();
> +	HtURLRewriter::instance()->replace(url);
> +	if (goturl.Exists(url))
> +	    delete array[i];
> +	else
> +	{
> +	    array[j++] = array[i];
> +	    goturl.Add(url, 0);
> +	}
> +    }
> +    numberOfMatches = j;
> +
>      const String st = config->Find("sort");
>      if (!st.empty() && mystrncasecmp("rev", st, 3) == 0)
>      {

I've had just a few more quick thoughts about this patch which I thought
I should share with you folks.

1) I think there should probably be an explicit delete of "ref" and
"collection" at the bottom of the for loop, to prevent a memory leak.
I don't think C++ would do that automatically, would it?

2) This strikes me as a potentially ineffecient approach to this problem,
as it would require a second fetching of every DocumentRef for all
matching documents.  In the 3.1.6 code, the URL is available in the
ResultMatch class, so the collections.0 patch for 3.1.6 was able to do
this much less expensively.  It may be that a better way would be to
catch this in Display::buildMatchList(), as the list of results is being
first built, and to somehow merge the results from the collections such
that the best score for a given URL is retained.  That would be a bit
more complicated than my quick adaptation of the 3.1.6 patch above, and
I don't have the time now to do this properly.  Any other takers?

3) You may want to consider how serious bug #504087 is, and whether it
needs to be fixed before this release.  I don't doubt that this is a bug,
and not just a feature request, but it may not be serious enough to risk
introducing a potentially performance-impacting fix for it this close to
release.  On the other hand, if you can come up with a quick, reliable
and "inexpensive" fix for this, then please do.

Cheers,
Gilles

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

[htdig-dev] Failed tests on OS X Panther with make check

[Ted Stresen-R. <ted...@ma...>]

hi,

make check produces the following output on OS X Panther (I couldn't 
wait to install it...) I haven't tried making all and make install, 
yet, but I will. These failed test might just be problems with how 
Apple sets up Apache.

PASS: t_wordkey
PASS: t_wordlist
PASS: t_wordskip
PASS: t_wordbitstream
PASS: t_search
PASS: t_htdb
PASS: t_rdonly
PASS: t_trunc
PASS: t_url
Error:  Apache has not been designed to serve pages while
         running as root.  There are known race conditions that
         will allow any local user to read any file on the system.
         If you still desire to serve pages as root then
         add -DBIG_SECURITY_HOLE to the EXTRA_CFLAGS line in your
         src/Configuration file and rebuild the server.  It is
         strongly suggested that you instead modify the User
         directive in your httpd.conf file to list a non-root
         user.

****Could not start apache. This test may fail, but that is not 
ht://Dig's fault

htpurge: Database is empty!

running htdig: expected
http://localhost:7400/set1/
http://localhost:7400/set1/bad_local.htm
http://localhost:7400/set1/script.html
http://localhost:7400/set1/site%201.html
http://localhost:7400/set1/site2.html
http://localhost:7400/set1/site3.html
http://localhost:7400/set1/site4.html
http://localhost:7400/set1/sub%2520dir/
http://localhost:7400/set1/sub%2520dir/empty%20file.html
http://localhost:7400/set1/title.html
but got

FAIL: t_htdig
Error:  Apache has not been designed to serve pages while
         running as root.  There are known race conditions that
         will allow any local user to read any file on the system.
         If you still desire to serve pages as root then
         add -DBIG_SECURITY_HOLE to the EXTRA_CFLAGS line in your
         src/Configuration file and rebuild the server.  It is
         strongly suggested that you instead modify the User
         directive in your httpd.conf file to list a non-root
         user.

****Could not start apache. This test may fail, but that is not 
ht://Dig's fault

htpurge: Database is empty!

running htdig: expected
http://localhost:7400/set1/
http://localhost:7400/set1/bad_local.htm
http://localhost:7400/set1/script.html
http://localhost:7400/set1/site%201.html
http://localhost:7400/set1/site2.html
http://localhost:7400/set1/site3.html
http://localhost:7400/set1/site4.html
http://localhost:7400/set1/sub%2520dir/
http://localhost:7400/set1/sub%2520dir/empty%20file.html
http://localhost:7400/set1/title.html
but got

FAIL: t_htsearch
PASS: t_htmerge
Error:  Apache has not been designed to serve pages while
         running as root.  There are known race conditions that
         will allow any local user to read any file on the system.
         If you still desire to serve pages as root then
         add -DBIG_SECURITY_HOLE to the EXTRA_CFLAGS line in your
         src/Configuration file and rebuild the server.  It is
         strongly suggested that you instead modify the User
         directive in your httpd.conf file to list a non-root
         user.

****Could not start apache. This test may fail, but that is not 
ht://Dig's fault

Could not fetch URL
FAIL: t_htnet
Error:  Apache has not been designed to serve pages while
         running as root.  There are known race conditions that
         will allow any local user to read any file on the system.
         If you still desire to serve pages as root then
         add -DBIG_SECURITY_HOLE to the EXTRA_CFLAGS line in your
         src/Configuration file and rebuild the server.  It is
         strongly suggested that you instead modify the User
         directive in your httpd.conf file to list a non-root
         user.

****Could not start apache. This test may fail, but that is not 
ht://Dig's fault

running htdig: expected
http://localhost:7400/set1/
http://localhost:7400/set1/bad_local.htm
http://localhost:7400/set1/script.html
http://localhost:7400/set1/site%201.html
http://localhost:7400/set1/site2.html
http://localhost:7400/set1/site3.html
http://localhost:7400/set1/site4.html
http://localhost:7400/set1/sub%2520dir/
http://localhost:7400/set1/sub%2520dir/empty%20file.html
http://localhost:7400/set1/title.html
but got
http://localhost:7400/set1/
http://localhost:7400/set1/script.html
http://localhost:7400/set1/site%201.html
http://localhost:7400/set1/site2.html
http://localhost:7400/set1/site3.html
http://localhost:7400/set1/site4.html
http://localhost:7400/set1/title.html
FAIL: t_htdig_local
====================
4 of 14 tests failed
====================
make[2]: *** [check-TESTS] Error 1
make[1]: *** [check-am] Error 2
make: *** [check-recursive] Error 1

[htdig-dev] [Task #87782] WWW protocol related

[SourceForge.net <no...@so...>]

Task #87782 has been updated. 

Project: ht://Dig
Subproject: Testing 3.2
Summary: WWW protocol related
Complete: 35%
Status: Open
Authority  : nealr
Assigned to: nobody

Description: Use the documenation to test each one of these config verbs

# protocol

accept_language(1)

disable_cookies

cookies_input_file

head_before_get

http_proxy

http_proxy_authorization

http_proxy_exclude

ignore_dead_servers

max_retries

persistent_connections

robotstxt_name(2)

server_max_docs

server_wait_time

tcp_max_retries

tcp_wait_time

timeout

user_agent

authorization

max_connection_requests

nph





Follow-Ups:

-------------------------------------------------------
Date: 2003-10-28 02:31
By: angusgb

Comment:
I have no chance to test the HTTP proxy. So far I have tested successfully:



accept_language

disable_cookies

cookies_input_file

head_before_get

max_retries

persistent_connections

tcp_max_retries

tcp_wait_time

timeout

user_agent



I have experienced problems with this: please try 'ignore_dead_servers'. 

-------------------------------------------------------
For more info, visit:

http://sourceforge.net/pm/task.php?func=detailtask&project_task_id=87782&group_id=4593&group_project_id=32485

[htdig-dev] patch for bug #504087, duplicate URLs in htsearch collections

[Gilles D. <gr...@sc...>]

I've pretty much run out of time to help out with this release, but before
I leave you, I thought I'd submit the following patch for your testing and
approval.  It should fix the duplicate URL problem in htsearch collections,
in bug #504087.  I'm not sure what sort of performance impact it will have
on large databases with lots of potential matches, though.  That may be
something to consider/test for.

Cheers,
Gilles

--- htsearch/Display.cc.orig	2003-10-25 07:40:23.000000000 -0500
+++ htsearch/Display.cc	2003-10-27 17:55:52.000000000 -0600
@@ -1895,6 +1895,27 @@ Display::sort(List *matches)
     qsort((char *) array, numberOfMatches, sizeof(ResultMatch *),
 	  array[0]->getSortFun());
 
+    // In case there are duplicate URLs across collections, keep "best" ones
+    // after sorting them.
+    Dictionary	goturl;
+    String	url;
+    int		j = 0;
+    for (i = 0; i < numberOfMatches; i++)
+    {
+	Collection *collection = array[i]->getCollection();
+	DocumentRef *ref = collection->getDocumentRef(array[i]->getID());
+	url = ref->DocURL();
+	HtURLRewriter::instance()->replace(url);
+	if (goturl.Exists(url))
+	    delete array[i];
+	else
+	{
+	    array[j++] = array[i];
+	    goturl.Add(url, 0);
+	}
+    }
+    numberOfMatches = j;
+
     const String st = config->Find("sort");
     if (!st.empty() && mystrncasecmp("rev", st, 3) == 0)
     {

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] [Task #87784] Parsing

[Gilles D. <gr...@sc...>]

According to SourceForge.net:
> Task #87784 has been updated. 
> 
> Project: ht://Dig
> Subproject: Testing 3.2
> Summary: Parsing
> Complete: 0%
> Status: Open
> Authority  : nealr
> Assigned to: nobody
> 
> Description: Use the documenation to test each one of these config verbs
> 
> #parsing
> 
> description_meta_tag_names
> 
> date_format

Actually, date_format is not used for parsing, only for the format used in
search results.  While I haven't tested it exhaustively, i.e. using every
possible format strftime supports, I did try it and it works fine.

Sorry I don't have the time to contribute more to the exhaustive testing
of all attributes.  Good work on that, guys!

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

[htdig-dev] template variables vs. config attributes

[Gilles D. <gr...@sc...>]

According to Lachlan Andrew:
> Swapping the order of these solves the particular problem, but the 
> more fundamental problem is that any variable can (or should be able 
> to be) defined in terms of any other, and there is no ordering which 
> can allow that.  One possibility would be to go through and set each 
> variable *twice*, so that second time around, most variables should 
> have their right values.  Of course, chains of variables defined in 
> terms of each other would need more than two iterations...
> 
> Another option would be to merge  vars  with  config  so that the 
> literal string  ${PAGELIST}  would be added, and only evaluated when 
> the variable is finally expanded during output.

I think it's very important to keep a distinction between configuration
attributes and template variables.  They're two distinct things, and
for good reason.  I had thought a while ago, when adding to htsearch the
ability to reference any environment variable from a template, that it
might be a good idea also to be able to do so for any config attribute
as well.  Then it occurred to me that this could be a security risk,
as it would be too easy to reveal sensitive config information in a
result template, if you're not careful.

Right now, you can put an attribute name in allow_in_form to turn it into
a CGI parameter and related template variable.  It might make sense to
also add an allow_in_template attribute, to do just the config to template
relationship, without also the CGI parameter to config relationship,
so that you can securely expose some config attributes to the templates
without allowing a web user to override them.  However, I'd advise
against a wholesale merging of config attributes and template variables.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

[htdig-dev] [Task #87788] Searching UI

[SourceForge.net <no...@so...>]

Task #87788 has been updated. 

Project: ht://Dig
Subproject: Testing 3.2
Summary: Searching UI
Complete: 100%
Status: Closed
Authority  : nealr
Assigned to: lha

Description: Use the documenation to test each one of these config verbs

# Searching UI

allow_in_form	

build_select_lists

method_names

sort_names

restrict

exclude





Follow-Ups:

-------------------------------------------------------
Date: 2003-10-26 23:06
By: lha

Comment:
Done all but  sort_names  and  build_select_lists.

-------------------------------------------------------
For more info, visit:

http://sourceforge.net/pm/task.php?func=detailtask&project_task_id=87788&group_id=4593&group_project_id=32485

Re: [htdig-dev] Outstanding bug reports.

[Neal R. <ne...@ri...>]

> I've closed 578570, but can't test 829746.  Does anyone have root
> access to a Solaris box?  If not, someone will have to close this
> without testing.  Similarly, does anyone have access to a Solaris box
> running gcc 3.3 to close the -Wno-deprecated issue, 799938.

  I can look at the first one at work tommorow.. we have a solaris 8 box,
but I can't do the second as it doesn't have gcc 3.3 (and it's not my
machine so installing a new gcc would probably lead to bloodshed)

> Does anyone have access to Cygwin to check if 814268 is a common
> problem?  If ht://Dig doesn't work under Cygwin, that is probably a
> show-stopper (or can Neal's native Win32 port supplant it?).

  I'll check this out as well.

  I want to support BOTH cygwin & native ports.

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] Re: Testing Tasks

[Neal R. <ne...@ri...>]

Yea, ignore the BDB errors.

And most errors about unitialized memory read, and Conditional jump or
move depends on uninitialised value(s) are spurious.  They are formed by
the compilers code generation and there isn't to much that can be to
elimiate them at the C/C++-level.

You are kicking butt on the testing tasks..

On Fri, 24 Oct 2003, Lachlan Andrew wrote:

> Greetings all,
>
> Thanks, Neal, for your work setting up the tasks!
>
> When I run  valgrind,  I get
> ==4058== Conditional jump or move depends on uninitialised value(s)
> ==4058==    at 0x40300421: CDB___lock_put_nolock (lock.c:650)
> from various contexts.
>
> I don't fancy changing the BDB code (my last foray was rather
> forgettable :)  Does anyone think that this is an issue, or should we
> ignore it?
>
> Cheers,
> Lachlan
>
> On Fri, 24 Oct 2003 01:09, Neal Richter wrote:
>
> > I would also encourage each of you to use valgrind to check for
> > memory leaks while you are testing. Again, the depth you go looking
> > for them is up to you.
>
> --
> lh...@us...
> ht://Dig developer DownUnder  (http://www.htdig.org)
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: The SF.net Donation Program.
> Do you like what SourceForge.net is doing for the Open
> Source Community?  Make a contribution, and help us add new
> features and functionality. Click here: http://sourceforge.net/donate/
> _______________________________________________
> ht://Dig Developer mailing list:
> htd...@li...
> List information (subscribe/unsubscribe, etc.)
> https://lists.sourceforge.net/lists/listinfo/htdig-dev
>

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

[htdig-dev] [Task #87784] Parsing

[SourceForge.net <no...@so...>]

Task #87784 has been updated. 

Project: ht://Dig
Subproject: Testing 3.2
Summary: Parsing
Complete: 0%
Status: Open
Authority  : nealr
Assigned to: lha

Description: Use the documenation to test each one of these config verbs

#parsing

description_meta_tag_names

date_format

ignore_alt_text

max_doc_size

max_keywords

max_meta_description_length

max_description_length

max_descriptions

max_excerpts

max_head_length

noindex_end

noindex_start

mime_types

content_classifier

external_parsers

external_protocols

use_meta_description



-------------------------------------------------------
For more info, visit:

http://sourceforge.net/pm/task.php?func=detailtask&project_task_id=87784&group_id=4593&group_project_id=32485

[htdig-dev] [Task #87788] Searching UI

[SourceForge.net <no...@so...>]

Task #87788 has been updated. 

Project: ht://Dig
Subproject: Testing 3.2
Summary: Searching UI
Complete: 75%
Status: Open
Authority  : nealr
Assigned to: lha

Description: Use the documenation to test each one of these config verbs

# Searching UI

allow_in_form	

build_select_lists

method_names

sort_names

restrict

exclude





Follow-Ups:

-------------------------------------------------------
Date: 2003-10-26 23:06
By: lha

Comment:
Done all but  sort_names  and  build_select_lists.

-------------------------------------------------------
For more info, visit:

http://sourceforge.net/pm/task.php?func=detailtask&project_task_id=87788&group_id=4593&group_project_id=32485

[htdig-dev] [Task #87777] search display

[SourceForge.net <no...@so...>]

Task #87777 has been updated. 

Project: ht://Dig
Subproject: Testing 3.2
Summary: search display
Complete: 80%
Status: Open
Authority  : nealr
Assigned to: nealr

Description: Use the documenation to test each one of these config verbs



# display

matches_per_page

use_star_image

max_stars

next_page_text

no_excerpt_show_top

no_excerpt_text

no_next_page_text

no_page_list_header

no_page_number_text

no_prev_page_text

prev_page_text

page_list_header

maximum_page_buttons

maximum_pages

search_results_contenttype

search_results_footer

search_results_header

search_results_order

search_results_wrapper

search_rewrite_rules

nothing_found_file

syntax_error_file

add_anchors_to_excerpt

anchor_target

sort

excerpt_show_top



Follow-Ups:

-------------------------------------------------------
Date: 2003-10-26 23:04
By: lha

Comment:
Oops...  Sorry to double up, Neal.  I've tested all but search_results_header, search_reseults_footer, search_results_order and sort without noticing that you had taken this task... --Lachlan

-------------------------------------------------------
For more info, visit:

http://sourceforge.net/pm/task.php?func=detailtask&project_task_id=87777&group_id=4593&group_project_id=32485

[htdig-dev] [Task #87786] Query Limits

[SourceForge.net <no...@so...>]

Task #87786 has been updated. 

Project: ht://Dig
Subproject: Testing 3.2
Summary: Query Limits
Complete: 100%
Status: Closed
Authority  : nealr
Assigned to: nobody

Description: Use the documenation to test each one of these config verbs

#query limits

any_keywords

keywords

startday

startmonth

startyear

endday

endmonth

endyear





Follow-Ups:

-------------------------------------------------------
Date: 2003-10-26 21:20
By: lha

Comment:
I've checked any_keywords and keywords.  There is a small problem with anchors in the excerpts (bug report filed) but I don't think it should be addressed before 3.2.0.



Gilles has tested start/end dates

-------------------------------------------------------
For more info, visit:

http://sourceforge.net/pm/task.php?func=detailtask&project_task_id=87786&group_id=4593&group_project_id=32485

[htdig-dev] Outstanding bug reports.

[Lachlan A. <lh...@us...>]

Greetings all,

Thanks for the changes to Display.cc, and closing the bug report,=20
Gilles.

I've closed 578570, but can't test 829746.  Does anyone have root=20
access to a Solaris box?  If not, someone will have to close this=20
without testing.  Similarly, does anyone have access to a Solaris box=20
running gcc 3.3 to close the -Wno-deprecated issue, 799938.

Does anyone have access to Cygwin to check if 814268 is a common=20
problem?  If ht://Dig doesn't work under Cygwin, that is probably a=20
show-stopper (or can Neal's native Win32 port supplant it?).

I personally don't think that the memory usage of htmerge (823866)=20
should be addressed before release, since it sounds like it would=20
require a fairly significant rewrite (although I haven't checked the=20
code at all).

Could someone please check and close 829754 and 829761.  Also 823455=20
can be removed from  "Include in 3.2.0"  once the partial fix has=20
been checked.

On Sat, 25 Oct 2003 07:53, Gilles Detillieux wrote:
> I've got my changes into Display.cc as well, and it all
> seems to be working just as it should.  I've closed your bug
> report.  Could someone do likewise for bugs 578570 and 829746, as
> I've applied the bug fixes for these?

--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

[htdig-dev] [Task #87774] htpurge

[SourceForge.net <no...@so...>]

Task #87774 has been updated. 

Project: ht://Dig
Subproject: Testing 3.2
Summary: htpurge
Complete: 100%
Status: Closed
Authority  : nealr
Assigned to: lha

Description: # htpurge

remove_bad_urls

remove_unretrieved_urls



-------------------------------------------------------
For more info, visit:

http://sourceforge.net/pm/task.php?func=detailtask&project_task_id=87774&group_id=4593&group_project_id=32485

[htdig-dev] [Task #87776] Valid Words

[SourceForge.net <no...@so...>]

Task #87776 has been updated. 

Project: ht://Dig
Subproject: Testing 3.2
Summary: Valid Words
Complete: 95%
Status: Open
Authority  : nealr
Assigned to: nobody

Description: Use the documenation to test each one of these config verbs



# valid words

allow_numbers

extra_word_characters(1)

valid_punctuation

locale(1)

minimum_word_length

bad_word_list

maximum_word_length

translate_latin1(1)

Follow-Ups:

-------------------------------------------------------
Date: 2003-10-24 21:10
By: lha

Comment:
Could someone else please check  locale  and then close this task?  Thanks!

-------------------------------------------------------
For more info, visit:

http://sourceforge.net/pm/task.php?func=detailtask&project_task_id=87776&group_id=4593&group_project_id=32485

[htdig-dev] [Task #87778] Template Variables

[SourceForge.net <no...@so...>]

Task #87778 has been updated. 

Project: ht://Dig
Subproject: Testing 3.2
Summary: Template Variables
Complete: 95%
Status: Open
Authority  : nealr
Assigned to: nobody

Description: Use the documenation to test each one of these config verbs



# template variables

end_ellipses

start_ellipses

end_highlight

start_highlight

star_blank

star_image

star_patterns

image_url_prefix

plural_suffix

script_name

boolean_keywords(1)

boolean_syntax_errors(1)

page_number_separator

page_number_text

no_title_text



Follow-Ups:

-------------------------------------------------------
Date: 2003-10-26 20:36
By: lha

Comment:
I've tested all but  script_name.  At one earlier stage, I broke that, so could someone else please test it? --Lachlan

-------------------------------------------------------
For more info, visit:

http://sourceforge.net/pm/task.php?func=detailtask&project_task_id=87778&group_id=4593&group_project_id=32485

[htdig-dev] not all 3.1.6 bug fix patches in 3.2 yet

[Gilles D. <gr...@sc...>]

Hi, folks!  Sorry to throw another curve ball your way, but while
checking the 3.2 Display.cc code against my patched 3.1.6 code, I came
to the realisation that not all bug fixes for the 3.1.6 code, available
at ftp://ftp.ccsf.org/htdig-patches/3.1.6/, and which would still be
relevant/applicable to the 3.2 code, have actually been applied there.

Here's my summary of the patches that probably ought to go in 3.2:

timet_enddate.2		(bug #578570, fixes enddate handling bugs)
time_t.1		(fixes possible time_t overflow in Server.cc)
titleSpace.0		(decodes %20 space in file names in title field)
masking_noindex.0	(makes HTML.cc insert space after noindex stripping)
Date-viewing.0		(fixes typo in HTML.cc)
SolarisLogging.1	(fixes logging bug when words empty on Solaris)
documentation.1		(in part - fixes cf_types.html & htmerge.html)
documentation.2		(in part - fixes htdig.html, meta.html & requires.html)

With the exception of the first, and likely the documentation ones,
they should have a bug report opened (or re-opened if one was prematurely
closed as for #578570), and dealt with appropriately.

I'll handle timet_enddate.2, Date-viewing.0, SolarisLogging.1 and the two
documentation patches.  For the others, I thought I'd put them up here
for grabs by anyone who feels motivated to tackle them -- and also up for
discussion if any of you think some of these are of questionable value.
Please speak up in either case.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)