htdig-dev — Developer Discussion for the ht://Dig project

[htdig-dev] A flexible security solution for searching the desktop

[Wolfgang <mue...@fr...>]

Hi,

I am maintainer of the GNU Image Finding Tool and active in the 
Fer-de-Lance project that's been in (not very loud, but 
behind-the-scenes-active) exsistence since April last year. Within this 
project we work towards integration of searching services into the desktop.

I am mailing our list and a couple of other developer lists, because I think 
I have found an architecture that provides security while maintaining most of 
the advantages of demon-based search engine architectures. I think this 
architecture and associated tricks are flexible enough to encompass different 
search engines, so this mail is not about Medusa vs. htdig vs. GIFT, but 
rather how to work together to solve our common security problems for desktop 
integration of our engines. 

And, of course, I would be happy to get some suggestions for improvement 
and/or some developer time. I would be less happy if someone finds a 
fundamental flaw, but also this would be better than wasting my time trying 
to develop this stuff further. 

Now let's go into more detail.

GOAL:
The goal is to provide search services to the desktop user. These search 
services should not only encompass web-visible URLs, but rather all files 
accessible to the user as well as http/ftp/etc. accessible items.

ISSUE:
The first issue is -privacy-: the system should not tell us locations of 
files that we cannot read otherwise. For example: looking for some 
correspondence with the health insurance, we do not want to know that our 
colleague wrote last month three letters that match our search.

Second -memory consumption-: All indexes for similarity searching use memory 
which is either proportional to the size of each indexed file, or quite big 
to begin with. We do not want plenty of users that roll their own index, we 
want one index, otherwise we are likely to spend a multiple of our useful 
disk space on indexes.

SUGGESTION: Use a daemon and make sure that authentication is good. :-)

Too easy? Of course the problem lies in providing the authentification. 

What I suggest is to run a daemon which creates for each user U an unix 
domain socket which is readable *and* writable *only* by this one user U (and 
root, of course). All instructions to the indexing demon like e.g.

add item to index
delete item from index
move item within index (new URL for same file)
block item/subdirectory/pattern (don't index *.o files for example)
process query

would go through the socket. By knowing which socket received the request, we 
automatically know the user, and then we just have to compare for each result 
item, if it can be read by the user who issued the query. Of course we give 
back only the readable items.

We can create the sockets as user "nemo", and then chown them using a very 
small script running under root. So we would be root during a couple of 
seconds on startup, afterwards everything would happen as a user (nemo) who 
has write rights on one directory tree which is unreadable for all else. So 
there is not the issue of a big indexing program running under root for days 
and days in a row.

Adding an item is a (small) issue. We probably have to pipe the uuencoded (or 
something equivalent) binary through the socket in order to have it indexed 
on the other side of the socket. However, I guess the efficiency overhead is 
small compared to the indexing cost.

Things become a trifle more complex for adding items which are found on the 
web. Somebody indexing a web page should probably indicate who else (group, 
all) is allowed to know that somebody's indexed that page. If several users 
publish an URL the least restrictive rights are taken into account.

WHATS THERE? WHAT'S NEEDED?

Basically, I have tried out the socket stuff with a small test program. 
Works. Now I am starting to integrate that with the GIFT (which involves 
cleaning up some of my internet socket code).

What's still needed is the filter that stores which URLs are indexed under 
which owner, and with which rights. On each query GIFT can ask this filter, 
if a list of URLs can be given out as query result. Currently, I would like 
to base this filter on MySQL.

When that filter is in place, writing a medusa-plugin for the GIFT would be 
easy. I just finished a primitive htdig GIFT plugin which soon goes to CVS, 
so that one just needs some fleshing out.

CONCLUSION

I hope to have convinced you that we can get relatively easily a secure, yet 
memory efficient indexing solution for the desktop. If this is been already 
done, please tell me where. If my mail is a stupid suggestion, please tell me 
that, too. However, if you would like to participate in the coding and design 
effort or simply to share your opinion, please do not hesitate to subscribe 
to the fer-de-lance-development list.

Cheers,
Wolfgang

[htdig-dev] Incremenal Index Efficiency, Unicode, & 2GIG limit

[Neal R. <ne...@ri...>]

Hello again,

	I've got a couple more questions..

1.	Is there any need to rebuild the index from scratch
periodically?  Some commercial search engines use incremental indexing and
recommend that when the incremental portion of the index gets to be a
given size (say 20%) the entire index is rebuilt.

2.      Is it possible to turn stemming off for particular languages
during run time?  We have our own stemming tools.. (Porter Algorithm)

3.      (Unicode) Is the index (the core of the index code) capable of
doing multibyte searching?  For example if a fully escaped version of a
Japanese or other multibyte document was indexed.. and then searched with
a properly escaped query.. would valid matches occur? (exculde any UI or
upper level code in your thing here.)

4.      (2 Gig Limit)  Some of the archives will be at a million+ 
documents in size with an average length exceeding 2K.  Other than using
XFS or JFS, the solution in this case is to use multiple index files?

5.      Is there a way to add a 'field' to the index?  Ie.. multiple
documents share a source-id & a query is given to return the documents
with that source-id.  This could accomplished implicitly by modifying the
source-id to be some special alpha-numeric character (DJ23KJD823).. but
this has a small probability of giving false-positive search results.

Thanks for your help!


-- 
Neal Richter 
Knowledge base Developer
Right Now Technologies, Inc.
Customer Service for Every Web Site

[htdig-dev] searching multiple database from one form

[Greg L. <gr...@md...>]

	I've been experimenting with searching multiple databases from one search 
form via the pipe in 3.2 (11/23 build):
<input type="checkbox" name="config" 
value="aom1|aom2|aom2a|aom3|aom4|aom5|esr">
(htmerge is not working at the moment).  The searches appear to be working 
correctly.  One unusual result that I have noticed is that pages which are 
included in multiple databases will appear multiple times in the output.  i.e.

1850 Constitutional Convention - Archives of Maryland Online

             http://www.mdarchives.state.md.us/megafile/msa/speccol/sc2900/sc2908/html/convention1850.html 
01/02/02, 8161 bytes

  1850 Constitutional Convention - Archives of Maryland Online

             http://www.mdarchives.state.md.us/megafile/msa/speccol/sc2900/sc2908/html/convention1850.html 
01/02/02, 8161 bytes

Would it be possible to remove duplicate pages from the search results 
before they are output to the html page?  This is obviously something that 
htmerge would do is the databases were to be combined into one.

Gregory Lepore
Webmaster, State of Maryland
Supervisor, Archives of Maryland Online
410-260-6425

Re: [htdig-dev] 3.1.x - htfuzzy core dumps

[Geoff H. <ghu...@ws...>]

At 3:36 PM -0600 1/2/02, Gilles Detillieux wrote:
>I'd recommend you check the Red Hat errata for 7.1 and make sure you
>install all the recommended update RPMS for it - especially any compiler
>or library updates, as it would seem there's likely a problem there.

The default compiler for RH 7.1 should be OK and I don't remember any 
updates that could cause this problem. (Remember that I've been 
testing the code on RH 7.0, 7.1, 7.2... recently.)

Gabriele, if you'd like to access my workstation, which is currently 
running 7.2, we can see if we can confirm this on my machine with 
your files.

-Geoff

[htdig-dev] Mirror at www.jack-of-all-trades.net

[Bradley M. H. <bh...@ja...>]

The mirror at www.jack-of-all-trades.net will no longer be available.  This
is effective with the sending of this message.

Bradley M. Handy
Programmer/Analyst
Spring Arbor University
--www.arbor.edu
--mailto:bh...@ar...

Re: [htdig-dev] htdig as a shared library

[Geoff H. <ghu...@ws...>]

On Thu, 3 Jan 2002, Neal Richter wrote:

> 	Any tips/input from anyone on the best way to make a shared
> library containing htdig & libraries like (libfuzzy.a libcommon.a libht.a
> etc)?  [It looks straightforward..]

If you want a shared library, use 3.2--it builds shared libraries on many
platforms.

> 	We're looking at using htdig as an information archive search
> engine.  We use PHP for our web apps, and would want to call the functions
> of htsearch from a PHP page.

Have you taken a look at the various PHP wrappers for htsearch? There are
several in the contrib/ directory on the
website: <http://www.htdig.org/contrib/> and
<http://www.htdig.org/files/contrib/>

> 2.	What would be involved in getting 3.2.x further down the road of
> Unicode?  We are in the process of completing a move of a very large 500K+
> line project to be UTF-8.  Who is the point man for htdig/Unicode issues?
> 	We have the resources to assist htdig in moving to Unicode.

We don't have one at the moment. Those of us who are active developers
have our hands full with other projects and don't have the interest at the
moment to work on Unicode.

My feeling is that the first step towards Unicode would be to get the
htword/mifluz backend synched with the mifluz CVS tree, which should be
more useful for Unicode applications. (It's also less bug-prone from what
I can tell.) Since others seem interested in this possibility, perhaps I
should spin the 3-2-x branch back to the mainline and fork a branch for
synching with mifluz.

While I don't think any of us intended to see Unicode in 3.2, if you have
the resources to make this happen, it certainly won't be turned away.

Welcome!

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

[htdig-dev] htdig as a shared library

[Neal R. <ne...@ri...>]

Hello,
	I have two questions:

1.
	Any tips/input from anyone on the best way to make a shared
library containing htdig & libraries like (libfuzzy.a libcommon.a libht.a
etc)?  [It looks straightforward..]

	We're looking at using htdig as an information archive search
engine.  We use PHP for our web apps, and would want to call the functions
of htsearch from a PHP page.

	We're evaluating htdig 3.1.5 for this.  We may switch to 3.2
when it's no longer Beta.

2.	What would be involved in getting 3.2.x further down the road of
Unicode?  We are in the process of completing a move of a very large 500K+
line project to be UTF-8.  Who is the point man for htdig/Unicode issues?

	We are a commercial software company and rely heavily on open
source projects  Linux, MySQL, PHP, Apache to make the business go.

	We have the resources to assist htdig in moving to Unicode.  In
addition, we have a variety of software tools (Purify, Insure, etc).  We
use these tools as a part of our software cycle and would release ALL
fixes back to htdig as well as any improvements to the core of htdig.

	Thank you.

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site

Re: [htdig-dev] Please vote for 'accept_language' attribute

[Gilles D. <gr...@sc...>]

According to Gabriele Bartolini:
>     I have a reasonable request. First of all, it is not hard to do and I 
> think I could do it quickly. It is just that I don't wanna do it without 
> your opinion.
> 
>     I think it could be handy to configure the digger to accept a language 
> of a list of languages as the HTTP says for the 'Accept-Language' directive.
> 
>     What do you think of issuing a new configuration attribute called 
> 'accept_language'? By default it is empty, but it could be set for instance 
> to 'it' or 'en' or also 'it, en, en-us' and so on.
> 
> So:
> 
> Proposed attribute name: 'accept_language'
> Program: htdig
> Type: string
> Default: empty
> Example: en-us, en, it
> Description: - I will work it out with the best english I can produce! - :-)
> 
>     I wanna modify the HtHTTP code to handle this and put it into 
> ht://Check and, as long as this class is now synchronised with ht://Dig 
> one, I would like to do it for our dear search engine too. What do you think?
> 
>     I think this configuration attribute could have a URL and Server scope 
> as well.

Sure, if you're willing to implement this, I give it my

+1

However, shouldn't the attribute type be "string list"?  Also, for
consistency with other string list attributes, it might make sense if
the commas weren't needed, e.g. "accept_language: en-us en it"

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/~grdetil
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930

Re: [htdig-dev] The /dev directory on maindocs

[Gilles D. <gr...@sc...>]

According to Geoff Hutchison:
> On Fri, 21 Dec 2001, J. op den Brouw wrote:
> > wondering why the htdig web site links the developer link to
> > sourceforge and not to /dev anymore. And why /dev is still around
> > on the maindocs....
> 
> Ah, I forgot about all of this. When we moved to SF, we obviously needed
> to get rid of dev.htdig.org and move to /dev at the very least. But for
> the few documents, I didn't know if it was better just to host them in the
> "Documentation" section of the SF project page.
> 
> I don't think it matters much one way or the other, though granted having
> them in the Documentation section means they won't be mirrored.
> 
> So if people think we should move back towards having a full "developer
> site," we should do it. Whatever will be the most effective.

Well, given our experience over the past year, I'd think that the more
we rely on our own pages and the less we rely on SF pages, the more
effective our site will be.  I'd prefer we have our own developer site,
with links to any SF services on which we rely, but relying on SF no
more than necessary.  As long as they don't start to clamp down on our
web space, I think that's the better way to go.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/~grdetil
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930

Re: [htdig-dev] 3.1.x - htfuzzy core dumps

[Gilles D. <gr...@sc...>]

According to Gabriele Bartolini:
>     as you know I am back at my workplace. I am trying to test ht://Dig 
> 3.1.x (with the current CVS rep) with the italian dictionaries. When trying 
> to run 'htfuzzy endings' I got this error. I tried to track it down, but I 
> think it has something to do with the Berkeley DB code (put function).
> 
>     I have used the italian dictionaries for ISpell and set the endings 
> affix and dictionary configuration attributes.
> 
>     Anyway here is the backtrace. I hope you could understand more. I have 
> playing with the debug for hours, without coming to anything.

Well, the backtrace tells me where it's failing, but not why.
The createRoot function is really pretty simple code, so it's not clear
to me how it can do anything that would cause the db_put() function to
crash like this.  Very wierd.

Even wierder is the additional information you provided in your SourceForge
tracker bug report...

> However, it dumps a core even with the default files in english.
...
>    My system characteristics are: Linux Kernel 2.4.2-2,
> gcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-81).

Well, obviously we haven't been able to reproduce the problem on our
systems.  If it's failing with the standard english.0 and english.aff
files, there's definitely something very wrong with your system.

I'd recommend you check the Red Hat errata for 7.1 and make sure you
install all the recommended update RPMS for it - especially any compiler
or library updates, as it would seem there's likely a problem there.

Also note that in Red Hat 7.x, the KDE packages use htdig 3.2.0b3 for
indexing the KDE docs, so make sure you're not running into some sort of
conflict between 3.1 & 3.2 databases.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/~grdetil
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930

Re: [htdig-dev] Back again - A few changes

[Gilles D. <gr...@sc...>]

According to Gabriele Bartolini:
> Ciao guys,
> 
>     After ages, I can swear I was missing you! :-)
...
>     I hope everything is fine and I have not made any mistake. So guys, 
> pllease give me the *welcome back*, I need encouragements! :-)

Welcome back!!!  And a very happy new year to you.  It's great to have
you back.  3.2 development had almost reached a stand-still, so it's great
that you can now debug the HTTP and cookies support.  Once the finishing
touches are done on 3.1.6, maybe we can get 3.2 back on the road.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/~grdetil
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930

[htdig-dev] Please vote for 'accept_language' attribute

[Gabriele B. <an...@us...>]

Ciao guys,

    I have a reasonable request. First of all, it is not hard to do and I 
think I could do it quickly. It is just that I don't wanna do it without 
your opinion.

    I think it could be handy to configure the digger to accept a language 
of a list of languages as the HTTP says for the 'Accept-Language' directive.

    What do you think of issuing a new configuration attribute called 
'accept_language'? By default it is empty, but it could be set for instance 
to 'it' or 'en' or also 'it, en, en-us' and so on.

So:

Proposed attribute name: 'accept_language'
Program: htdig
Type: string
Default: empty
Example: en-us, en, it
Description: - I will work it out with the best english I can produce! - :-)

    I wanna modify the HtHTTP code to handle this and put it into 
ht://Check and, as long as this class is now synchronised with ht://Dig 
one, I would like to do it for our dear search engine too. What do you think?

    I think this configuration attribute could have a URL and Server scope 
as well.

Ciao and Happy New Year's! ;-)
-Gabriele
--
Gabriele Bartolini - Web Programmer
Current Location: Prato, Tuscany, Italy
an...@us... | http://www.prato.linux.it/~gbartolini | ICQ#129221447
 > find bin/laden -name osama -exec rm {} \;

Rif: Re: [htdig-dev] Cookies in 3.2beta

[Gabriele B. <an...@ti...>]

> Hi Gabriele,
> thanks for your fix! The crash is gone, but it doesn't seem to work yet. 

Yeah, you are right. The only problem with the Cookies code was that I had not yet written the management of the 'domain' field.

Now I added code for the control of the subdomains, as the specification says (I mean, at least 2 subdomains for 7 top level domains -com, org, etc.- and 3 for the others).

I had not enough time to test it, even though now it seems to work for the script you sent. I also put the same in ht://Check and it works fine as well.

I hope with some massive testing, the Cookie support may be considered to be *achieved*.

I had promised myself to do this before but ... you know ... :-)

Let me know if there is some problem and let's cross our fingers.

Ciao
-Gabriele

[htdig-dev] restrict_virtual_hosts patch

[Justin D. <ju...@ar...>]

I don't know whether anybody is interested, but I patched htsearch to 
override restrict to search only the virtual host.  It's pretty 
simplistic, but should work in most cases.  Using this patch you should 
be able to have several virtual hosts share a common htdig install/db 
while maintaining seperate search domains.

Feel free to contact me about anything relating to this patch.

--
Justin Day
ju...@ar...

Re: [htdig-dev] Cookies in 3.2beta

[Daniel N. <dan...@t-...>]

On Monday 17 December 2001 21:11, you wrote:=0A=0A> >It seems _cookie_j=
ar is never properly defined (it's only set to 0 at=0A> > the=0A> >=0A>=
  >top). What is the state of cookie support? I assume this crash is a=0A=
>  > known problem? Is this easy to fix? If so, I might try...=0A>=0A> =
Fixed. Thanx!!! :-)=0A=0AHi Gabriele,=0A=0Athanks for your fix! The cra=
sh is gone, but it doesn't seem to work yet. =0AUsing the attached test=
 case, htdig never reaches cookie3.pl. With a =0Acookie-enabled browser=
, cookie3.pl can be reached, a browser with cookies =0Adisbaled only re=
aches cookie2.pl (start surfing at start.pl).=0A=0AYou can test this by=
 copying the files to your cgi-bin and adapt at least =0Athe "domain" i=
n start.pl=0A=0ARegards=0A Daniel=0A=0A-- =0Ahttp://www.danielnaber.de=0A

Re: [htdig-dev] Seg faults in 12/9 build of 3.2

[Geoff H. <ghu...@ws...>]

On Wed, 26 Dec 2001, Greg Lepore wrote:

> The second error was on two very large collections that I was merging -
> Might this have something to do with the 2GB file size limit?

Probably. If you're running on a system with no such limit (Solaris pops
to mind), then there's something else amiss. The databases have been
tested well over the 2GB limit by Loic and there aren't direct problems in
the ht://Dig and mifluz code.

> Actually, I ran around 10 merges before I got the first error above, and 

Hmm. There are certainly bugs in the database code that ht://Dig 3.2 is
using. This doesn't seem like it's going to be resolved until I can get
some time to merge the ht://Dig and mifluz CVS trees again. Loic seems to
ignore messages from me.

> System is RedHat 7.2

On Intel, I assume.

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

[htdig-dev] Seg faults in 12/9 build of 3.2

[Greg L. <gr...@md...>]

	I have been using the 12/9/01 build of 3.2 since the 10th.  I have 
encountered three segmentation faults that have me stopped.  One occurred 
in htdig and the other two in htmerge.  The end of the htmerge errors are 
as follows:

htmerge: Duplicate, URL: 0x8712b18 ignoring merging copy
WordDB: CDB___memp_cmpr_read: expected DB_CMPR_FIRST flag set at pgno = 50435
WordDB: PANIC: Invalid argument
WordDBCursor::Get(17) failed DB_RUNRECOVERY: Fatal error, run database recovery
Segmentation fault
[root@search2 bin]#

and

WordDBCursor::Get(17) failed Cannot allocate memory
Segmentation fault

The second error was on two very large collections that I was merging -

-rw-r--r--    1 root     root         622M Dec 20 13:49 db.excerpts
-rw-r--r--    1 root     root         246M Dec 20 13:49 db.words.db
and
-rw-r--r--    1 root     root         1.5G Dec 20 14:07 db.excerpts
-rw-r--r--    1 root     root         381M Dec 20 11:03 db.words.db

Might this have something to do with the 2GB file size limit?

Actually, I ran around 10 merges before I got the first error above, and 
then all subsequent merges failed.  For the second error above, five 
previous merges were successful.

I ran gdb on htmerge but only got:
#0  0x00000000 in ?? ()

The htdig error only has:
1627:1005:3:http://www.mdarchives.state.md.us/megafile/msa/coagser/c2200/c2204/html/landrecordsrho123.html: 
+Segmentation fault

System is RedHat 7.2

Any ideas?
Gregory Lepore
Webmaster, State of Maryland
Supervisor, Archives of Maryland Online
410-260-6425

[htdig-dev] Re: [Fwd: Hello]

[Geoff H. <ghu...@ws...>]

Gee, you'd think they could manage to spell my username right considering
a) they just contacted me about something else
b) I have in a support request about this

And without the error messages from the Apache error log, I haven't a clue
what's broken. It works from the command-line just fine.

-Geoff

On Fri, 21 Dec 2001, Andrew Scherpbier wrote:

> -------- Original Message --------
> Subject: Hello
> Date: Thu, 20 Dec 2001 17:19:30 -0800
> From: Patrick McGovern <pmc...@va...>
> To: lo...@us..., gh...@us..., 
> sch...@us..., gr...@us...
> 
> 
> 
> Hi,
> 
> I wanted to give you guys a heads up that your search on http://www.htdig.org/
> is broken.
> 
> The error message that results points people to email st...@sf... (standard
> server error message).   We have been receiving about 8 emails a day for the
> past week.
> 
> Pat-
> 
> -- 
> Patrick McGovern                        Tel: 510-687-7080
> Director, SourceForge.net               pmc...@va... 
> 
> 
> 

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

Re: [htdig-dev] The /dev directory on maindocs

[Geoff H. <ghu...@ws...>]

On Fri, 21 Dec 2001, J. op den Brouw wrote:

> wondering why the htdig web site links the developer link to
> sourceforge and not to /dev anymore. And why /dev is still around
> on the maindocs....

Ah, I forgot about all of this. When we moved to SF, we obviously needed
to get rid of dev.htdig.org and move to /dev at the very least. But for
the few documents, I didn't know if it was better just to host them in the
"Documentation" section of the SF project page.

I don't think it matters much one way or the other, though granted having
them in the Documentation section means they won't be mirrored.

So if people think we should move back towards having a full "developer
site," we should do it. Whatever will be the most effective.

-Geoff

[htdig-dev] The /dev directory on maindocs

[J. op d. B. <MSQ...@st...>]

Hi,

wondering why the htdig web site links the developer link to
sourceforge and not to /dev anymore. And why /dev is still around
on the maindocs....

--jesse
--------------------------------------------------------------------
J. op den Brouw                           Johanna Westerdijkplein 75
Haagse Hogeschool                                  2521 EN  DEN HAAG
Faculty of Engeneering                                   Netherlands
Electrical Engeneering                                +31 70 4458936
-------------------- J.E...@st... --------------------

Linux - because reboots are for hardware changes

[htdig-dev] 3.1.x - htfuzzy core dumps

[Gabriele B. <an...@ti...>]

Ciao guys,

   as you know I am back at my workplace. I am trying to test ht://Dig 3.1.x (with the current CVS rep) with the italian dictionaries. When trying to run 'htfuzzy endings' I got this error. I tried to track it down, but I think it has something to do with the Berkeley DB code (put function).

   I have used the italian dictionaries for ISpell and set the endings affix and dictionary configuration attributes.

   Anyway here is the backtrace. I hope you could understand more. I have playing with the debug for hours, without coming to anything.

Thanks a lot!
Ciao
-Gabriele

(gdb) backtrace
#0  0x08073573 in __bam_cmp () at ../../htdig/htlib/StringMatch.cc:567
#1  0x0807a45c in __bam_search () at ../../htdig/htlib/StringMatch.cc:567
#2  0x080758a6 in __bam_c_search () at ../../htdig/htlib/StringMatch.cc:567
#3  0x08074b45 in __bam_c_put () at ../../htdig/htlib/StringMatch.cc:567
#4  0x0805b1b4 in __db_put () at ../../htdig/htlib/StringMatch.cc:567
#5  0x08050079 in DB2_db::Put (this=0x809d2c0, key=@0xbffff3f0,
    data=@0xbffff400) at ../../htdig/htlib/DB2_db.cc:337
#6  0x0804ac44 in Endings::createRoot (this=0x80916c8, rules=@0xbffff890,
    word2root=0x8096970 "/tmp/word2root.db",
    root2word=0x8096998 "/tmp/root2word.db",
    dictFile=0x8097f20 "/opt/htdig-3.1-dev/web/common/it/ispell/italian.words")
    at ../../htdig/htfuzzy/EndingsDB.cc:185
#7  0x0804a562 in Endings::createDB (this=0x80916c8, config=@0x80910e0)
    at ../../htdig/htlib/htString.h:44
#8  0x0804d588 in main (ac=2, av=0xbffffa1c)
    at ../../htdig/htfuzzy/htfuzzy.cc:230
#9  0x400b8177 in __libc_start_main (main=0x804cff8 <main>, argc=2,
    ubp_av=0xbffffa1c, init=0x8049820 <_init>, fini=0x80851d8 <_fini>,
    rtld_fini=0x4000e184 <_dl_fini>, stack_end=0xbffffa0c)
    at ../sysdeps/generic/libc-start.c:129

Ciao
-Gabriele

[htdig-dev] 3.1.x - htfuzzy core dumps

[Gabriele B. <g.b...@co...>]

Ciao guys,

    as you know I am back at my workplace. I am trying to test ht://Dig 
3.1.x (with the current CVS rep) with the italian dictionaries. When trying 
to run 'htfuzzy endings' I got this error. I tried to track it down, but I 
think it has something to do with the Berkeley DB code (put function).

    I have used the italian dictionaries for ISpell and set the endings 
affix and dictionary configuration attributes.

    Anyway here is the backtrace. I hope you could understand more. I have 
playing with the debug for hours, without coming to anything.

Thanks a lot!
Ciao
-Gabriele

(gdb) backtrace
#0  0x08073573 in __bam_cmp () at ../../htdig/htlib/StringMatch.cc:567
#1  0x0807a45c in __bam_search () at ../../htdig/htlib/StringMatch.cc:567
#2  0x080758a6 in __bam_c_search () at ../../htdig/htlib/StringMatch.cc:567
#3  0x08074b45 in __bam_c_put () at ../../htdig/htlib/StringMatch.cc:567
#4  0x0805b1b4 in __db_put () at ../../htdig/htlib/StringMatch.cc:567
#5  0x08050079 in DB2_db::Put (this=0x809d2c0, key=@0xbffff3f0,
     data=@0xbffff400) at ../../htdig/htlib/DB2_db.cc:337
#6  0x0804ac44 in Endings::createRoot (this=0x80916c8, rules=@0xbffff890,
     word2root=0x8096970 "/tmp/word2root.db",
     root2word=0x8096998 "/tmp/root2word.db",
     dictFile=0x8097f20 
"/opt/htdig-3.1-dev/web/common/it/ispell/italian.words")
     at ../../htdig/htfuzzy/EndingsDB.cc:185
#7  0x0804a562 in Endings::createDB (this=0x80916c8, config=@0x80910e0)
     at ../../htdig/htlib/htString.h:44
#8  0x0804d588 in main (ac=2, av=0xbffffa1c)
     at ../../htdig/htfuzzy/htfuzzy.cc:230
#9  0x400b8177 in __libc_start_main (main=0x804cff8 <main>, argc=2,
     ubp_av=0xbffffa1c, init=0x8049820 <_init>, fini=0x80851d8 <_fini>,
     rtld_fini=0x4000e184 <_dl_fini>, stack_end=0xbffffa0c)
     at ../sysdeps/generic/libc-start.c:129

Ciao
-Gabriele

--
Gabriele Bartolini - Computer Programmer
U.O. Rete Civica - Comune di Prato - Prato - Italia - Europa
g.b...@co... | http://www.po-net.prato.it/

  The nice thing about Windows is - It does not just crash,
  it displays a dialog box and lets you press 'OK' first.

Re: [htdig-dev] configure changes in 3.1.x

[Gilles D. <gr...@sc...>]

According to Gabriele Bartolini:
> Ciao Gilles,
> 
>     as you probably noticed, I changed the CONFIG.in and the configure.in 
> files for the with-'____' series of parameters regarding the some directory 
> and file customisation.
> 
>     I hope you don't mind if I did it without 'warning' you. I have to get 
> back on track!!! :-)
> 
> Ciao
> -Gabriele

Yes, I noticed the CVS update log messages.  I'll consider your e-mail
as the warning, even though it was too late to say anything about it.
I think your changes are good ones, which should resolve a few complaints
about 3.1.x's lack of configurability.  Unfortunately, they also mean I
need to fix the RPM .spec file in contrib accordingly, before 3.1.6 can
be released.  (I had been thinking of other changes to it anyway.)

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/~grdetil
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930

Re: [htdig-dev] Re: mktime fix?

[Gilles D. <gr...@sc...>]

According to Geoff Hutchison:
> Hi Gilles,
> 
> I seem to remember that you were planning on putting in your mktime() fix
> for the 3.1.6 release--this was the message you sent on Nov. 23. I don't
> see it in the current CVS version, but perhaps I somehow missed a
> revision?

Well, I meant to do it soon after that posting, but I never got around
to it then and it slipped my mind.  Thanks for the reminder.  I was also
looking at what it would take to completely get rid of all the strptime,
timegm and mktime usage, and it turns out that htdig/Retriever.cc still
uses strptime and timegm (or mytimegm) to parse DC date headers in
Retriever::got_time().  Also, htsearch uses mktime for the startyear
and endyear handling.  So, I may have addressed the most problematic
issue, but not everything.  I'm now wondering if I should tackle the
Retriever and Display code, or leave it at that.

> I'd like to clean up any remaining issues over the next few weeks so we
> can release 3.1.6 ASAP. (I have the time, so I'm taking care of the
> regex/rx issue, the release notes, the maindocs merges, the getpeername
> issue, and code into htmerge to catch empty databases to throw up more
> useful help text.)

OK, just give me a chance to get my last few tweaks and tests in before
final release.  I probably won't be around much between tomorrow and
Jan. 2, so maybe a mid-January release date would be ideal.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/~grdetil
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930

Re: [htdig-dev] htdig 3.2.0b4 and HP-UX 11.00

[Geoff H. <ghu...@ws...>]

On Thu, 20 Dec 2001 htd...@li... wrote:

>         NET_SIZE_T l;
> 
> NET_SIZE_T is effectively size_t for all SVR4 systems. It's hard-coded
> for other platforms in src/include/ap_config.h.

(In short, Apache decided this (void *) issue was strange too.)

I'd suggest changing the "can't determine" default to size_t in this
case. I can't think of another platform that does this sort of
strangeness.

So the test looks like this in configure.in: (I'm not going to commit to
CVS now, I'm
going to sleep.)

...
 extern "C" int getpeername(int, $sock_t *, $getpeername_length_t *);
 $sock_t s;  $getpeername_length_t l; ],
    [ getpeername(0, &s, &l); ],
    [ac_found=yes ; break 2],[ac_found=no])
  done
done

if test "$ac_found" = no
then
  AC_MSG_WARN([can't determine, using size_t])
  AC_DEFINE_UNQUOTED(GETPEERNAME_LENGTH_T, size_t)
else


--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/