htdig-dev — Developer Discussion for the ht://Dig project

[htdig-dev] New mirror

[Claus L. <nes...@ya...>]

After trying to make contact to you guys using all
email addresses I have found on the htdig website,
with no luck at all, not a single response from
anyone, I am now trying this list hoping for a
response :-)

I have setup I htdig mirror which is updated daily.
Info for the mirror page.

Organisation: Cofman.com http://www.cofman.com
Country: Denmark
Main Site: http://htdig.cofman.dk/
Developer Site: http://htdig.cofman.dk/dev
Files: http://htdig.cofman.dk/files
Patch Archive: none

Should this message actually land in a mailbox
somewhere, may I then suggest that you update the
mirror instruction page.

Thanks
Claus Larsen
Cofman.com


		
__________________________________
Do you Yahoo!?
Yahoo! Mail - 50x more storage than other providers!
http://promotions.yahoo.com/new_mail

[htdig-dev] Stemming algorithm

[Olivia J. <oli...@ya...>]

Can anyone give listing program of Dawson Stemmer or Dawson Stemmer's algorithm for me? If anyone have description about Dawson Stemmer, please send to me. Dawson Stemmer is stemming algorithm where developed by J.L. Dawson. 
 
Thanks,
 
Olivia

		
---------------------------------
Do you Yahoo!?
Yahoo! Mail Address AutoComplete - You start. We finish.

[htdig-dev] Re: htdig-dev digest, Vol 1 #937 - 1 msg

[Robert R. <ri...@li...>]

htd...@li... wrote:

>Send htdig-dev mailing list submissions to
>	htd...@li...
>
>To subscribe or unsubscribe via the World Wide Web, visit
>	https://lists.sourceforge.net/lists/listinfo/htdig-dev
>or, via email, send a message with subject or body 'help' to
>	htd...@li...
>
>You can reach the person managing the list at
>	htd...@li...
>
>When replying, please edit your Subject line so it is more specific
>than "Re: Contents of htdig-dev digest..."
>
>
>Today's Topics:
>
>   1. Is using htdig with Frontpage 2003 possible? (Julia Richter)
>
>--__--__--
>
>Message: 1
>Date: Thu, 29 Jul 2004 12:55:37 -0700 (PDT)
>From: Julia Richter <jri...@sb...>
>To: htd...@li...
>Subject: [htdig-dev] Is using htdig with Frontpage 2003 possible?
>
>Hi:
>
>I have been trying to find a search engine that will
>return items and not pages.  I built a website using
>Frontpage 2003 and is hosted on IIS. Do I need to get
>a host with a Linux server to use your product?
>
>The engine that came with Frontpage only returns
>pages.
>
>I have over 5000 books in  word tables on four pages.
>
>Will yur search engine work for me?
>
>Everyone I talk to has a different answer from make a
>database in Access to get Atom search for $10,000!
>
>I would so appreciate any help. 
>
>
>Regards,
>
>Julia
>  
>
Hello Julia,

Htdig is a 'solution' that has two components:

a) An indexing 'component' - Like a robot, it will follow documents 
(usually accessed through the hypertext transfer protocol), boild  them 
down to their 'keywords', and store these keywords in a (Berkely DB) 
Database for searching. Through the use of 'helper applicatiions' it 
becomes possible to index non-html content (among other things: word files).

b) A search component, usually including a search form (html) and a 
cgi-script returning (html-style) customizable result pages. This will 
only require a web server (eg. IIS), and return its result pages

On the other hand, I am not completely clear:

- You have '5000 books as word documents'  and want a web-based 
full-text search over those documents? - First of all the Word Document 
format is not ideal for searching (since its basically closed). Then 
you'll have a set  of 'results' (text passages), that you need to 
transform these results (from the lcosed proprietary word format into 
something a little more documented, like eg. PDF) and make them fit for 
web display (forget PDF, requires a plugin and breaks the meaning of the 
web).

Or do you have 4 pages with references (Books), written as a word 
table?- If thats the case I'd convert them into a proper database, 
allowing for a proper search -  A task for a database engineer, nothing 
to do with htdig. When you convert the database you should have an eye 
to use open formats so a bit company from redmond does not dictate your 
future choice of an operating system. - In that case you might want to 
look at open databases like MySQL(http://www.mysql.com), and building a 
search interface to the database eg. with php (http://www.php.net)
Or finally, do you have the full text of 5k books as word documents, and 
want to search through that - doable with htdig, but I'd personally 
prefer a more open format, like eg. DocBook...

So please elucidate us

Robert Ribnitz
Debian Maintainer of HT://Dig

Re: [htdig-dev] Is using htdig with Frontpage 2003 possible?

[Neal R. <ne...@ri...>]

Julia,

If you are really ambitious and want to help the win32 version of 
libhtdig along it will work.  You'll have to code a data importer in C/C++ that 
calls C functions to insert pseudo-documents.  At that point the standard 
search interface works.

Otherwise you REALLY might be better off starting with something like 
mnoGoSearch (http://mnogosearch.org/).... from their website's description 
they use a MYSQL database as the data storage basis of their engine.

This would allow you to take a look at their table schema and insert the 
appropriate rows in to appropriate tables and make it dance.

The other issue is the 'items' you will be returning.  Search results 
are a list of links.  You'll have to alter the search results to provide a 
some kind of action to take on the item since there is no webpage link to 
provide.

Have you tried looking on freshmeat.net for a product catalog with 
searching ability?

Thanks!

Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485


On Thu, 29 Jul 2004, Julia Richter wrote:

> Hi:
>
> I have been trying to find a search engine that will
> return items and not pages.  I built a website using
> Frontpage 2003 and is hosted on IIS. Do I need to get
> a host with a Linux server to use your product?
>
> The engine that came with Frontpage only returns
> pages.
>
> I have over 5000 books in  word tables on four pages.
>
> Will yur search engine work for me?
>
> Everyone I talk to has a different answer from make a
> database in Access to get Atom search for $10,000!
>
> I would so appreciate any help.
>
>
> Regards,
>
> Julia
>
> =====
>
> Education is not the filling of a pail, but the lighting of a fire.
>                                                                                ---W.B. Yeats
>
>
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by OSTG. Have you noticed the changes on
> Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
> one more big change to announce. We are now OSTG- Open Source Technology
> Group. Come see the changes on the new OSTG site. www.ostg.com
> _______________________________________________
> ht://Dig Developer mailing list:
> htd...@li...
> List information (subscribe/unsubscribe, etc.)
> https://lists.sourceforge.net/lists/listinfo/htdig-dev
>
>

[htdig-dev] Is using htdig with Frontpage 2003 possible?

[Julia R. <jri...@sb...>]

Hi:

I have been trying to find a search engine that will
return items and not pages.  I built a website using
Frontpage 2003 and is hosted on IIS. Do I need to get
a host with a Linux server to use your product?

The engine that came with Frontpage only returns
pages.

I have over 5000 books in  word tables on four pages.

Will yur search engine work for me?

Everyone I talk to has a different answer from make a
database in Access to get Atom search for $10,000!

I would so appreciate any help. 


Regards,

Julia

=====
 
Education is not the filling of a pail, but the lighting of a fire.
                                                                                ---W.B. Yeats

[htdig-dev] Re: Licenses, general state of packaging

[Robert R. <ri...@li...>]

Neal Richter wrote:

>Robert,
>
>  Thanks for your efforts here!  We have little contact from the other
>Linux distros.... which tells me that Debian is more on-the-ball on
>License audits of code that is included.
>  
>
Side note: it needs to be, since it is the only distribution i know of 
that rigurously distinguishes between 'free', 'free but depending on 
non-free stuff' (they call it contrib), and 'non-free'. This distinction 
is mainly a question of licensing.

>  
>
>>It is also my understanding that Sleepycat's license for DB is compatible
>>with GPL, but not LGPL.
>>    
>>
>
>  Yep.
>
>  
>
>>It's a whole other ball of wax for the 3.2 code base.  There, Neal
>>negotiated a special license with Sleepycat that allows us to distribute
>>our modified DB code with the LGPL'ed 3.2 ht://Dig code, provided the
>>bundled DB code is used only for ht://Dig or libhtdig.  If it's unbundled,
>>it reverts to Sleepycat's standard license.
>>
>>Neal and Geoff, please chime in if you have anything to add.
>>    
>>
>
>[This next part applies only to HtDig 3.2.x]
>
>  Note that for 3.2 the HtDig license is LGPL.
>
>  If a 'LGPLed 3.2 HtDig + custom DBD with custom Sleepycat License'
>causes you license indigestion for Debian..... feel free to make the whole
>thing GPL and remove the special Sleepycat/BDB License Exception in your
>source package.
>
>  If you do this, we would kindly ask that any patches you submit to us
>for HtDig 3.2.x be LGPLed by you so we can include them in the stock HtDig
>3.2.x tree.
>
>  I know this is a bit wierd, but it's legit since the HtDig group is
>allowed by copyright law (as copyright holders of our code) to interpret
>the LGPL as we see fit.
>
>  And Sleeycat did the same by making a HtDig specific exception to their
>license to allow us to move to the LGPL and still use BDB.
>
>  FYI:  It looks like for HtDig 4.0 we MAY be abandoning BDB, and be
>replacing it with an LGPL'd Lucene/CLucene.. so the wierd licensing issue
>goes away.
>
>Thanks again.
>
>Neal Richter
>Knowledgebase Developer
>  
>
Hello Neal,  Hello Developers' list,

At the moment I am still in the process of cleaning up the 3.1.6 
package, which will be released (and included in sarge) as 'htdig' 
(Version 1:3.1.6-8 or later). Htdig 3.2.0b6 (or later) will go into the 
tree as 'htdig-3.2' (and might not be included in sarge, who knows). At 
the moent the idea is to have the two packages conflict with each other, 
but thats really my problem. Alll this was done to remedy the speed 
issues 3.2.0b6 still has compared to 3.1.6.

To come back to the point: As a debian developer I am required to 
summarize the licensing situation in a file called 'copyright' in the 
debian-tree of the package.

All the following applied to 3.1.6 only:

- I was able to remove the 'db' folder (Berkely DB V.2.6.2) and replace 
it with the stock libdb2(-dev) package of Debian. In that process, I 
also got from a 4-clause BSD-Style license from Sleeepycat et al. to a 
3-Clause BSD-Style license. Since,however, this license is part of the 
package I link against, I no longer need to worry about it. You link a 
GPL package against libdb2, which seems to be fair enough (countless 
other debian packages also do it).

What I still need to do is to adapt the debian installation and 
maintenance scripts a little; so i guess my 'htdig' 3.1.6-8 package will 
enter the archive about the end of this week:

(Excerpt from the changelog)

>  * Patched htfuzzy so that it does not die on words of non-alpha 
> characters
>     only when searching for numbers is enabled and a soundex search is
>     performed. Thanks to Alex Kiesel for this patch. This is a patch 
> provided
>     by the Ht://Dig developer team.
>   * Patched htnotify race condition. Htnotify would sit there eating up
>     memory, appending newlines to an empty string. As Martin Kraemer
>     discovered, an additional check fixes the problem. This patch is also
>     provided by the Ht://Dig developer team.
>   * Added a patch that allows the correct skipping of JavaScript code. 
> Work
>     attributed to Gilles Detillieux. Patch provided by upstream.
>   * Have external parsers check for the maximal Document size 
> (max_doc_size).
>     Work attributed to Gilles Detillieux. Patch provided by upstream
>   * Fixed the rating of percent values. For some unknown reason, 
> including a
>     percent value would rate the document higher than it should. Patch
>     provided by upstream (and not attributable to a person).
>   * A set of patches brings the HTML documentation up to date, fixing 
> errors
>     and clearing ambinguities. Thanks to Gilles Detillieux for these 
> patches.
>     Also provided by the upstream developers.

So when htdig-3.1.6 is out (might be on of the last debian revisions to 
it), I'll focus all my attention on the 3.2.0 branch.

Robert

[htdig-dev] strptime.c

[Neal R. <ne...@ri...>]

Mr. Ruddy,
 	I'm a member of the HtDig (www.htdig.org) open source search
engine group.

	We are currently using your implementation of strptime in the
3.2.x (LGPL'd) version of HtDig.

     41 /*
     42 static char copyright[] =
     43 "@(#) Copyright (c) 1994 Powerdog Industries.  All rights reserved.";
     44 static char sccsid[] = "@(#)strptime.c  1.0 (Powerdog) 94/03/27";
     45 */

	The Debian people have raised the issue of license incompatibility
with the GPL/LGPL.  Your license is a classic 4-clause BSD-type with an
advertising clause.  HtDig has distributed both GPL and LGPL versions of
our software.

	Would it be possible for you to give us a BSD license for strptime
without the 'advertising clause'?

	Also some statement from you forgiving any past violation of the
advertising clause for the HtDig group and our users would be helpful too
;-).

	This may seem a bit silly, but it matters to the Debian people and
we want to do things properly.

	Thanks!

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] Licensing issues...

[Neal R. <ne...@ri...>]

On Mon, 26 Jul 2004, Glenn Maynard wrote:

> On Mon, Jul 26, 2004 at 03:11:52PM -0600, Neal Richter wrote:
> > > 2) Can I reasonably argue that htdig is gpl (or lgpl) if its linked
> > > against a 3 or a 4 cloause BSD license? - htdig .3.1.6 builds static
> > > libraries (.a) it links against.
> >
> >   Sure you can!
> >
> >   Note that although the Free Software Foundation may say that a 4-clause
> > BSD license is incompatible with the GPL (and they do)..... that opinion only
> > applies to code that the FSF holds the copyright for.
>
> Not quite.  I believe that opinion derives directly from the text of the
> license.  The GPL prohibits adding further restrictions (GPL#6), and the
> 4-clause BSD license does so.  Unless a copyright holder specifically says
> that he considers the 4-clause BSD license consistent with the GPL, it's
> not safe for Debian to assume otherwise.
>
> That is, the "default" interpretation, lacking a statement from (all of)
> the copyright holders of a work, should be the one that follows from the
> license text, and that's the FSF's interpretation, at least in this case.

  OK... This is good point.

> >   As a matter of copyright law, any copyright holder that licenses their
> > code under the GPL is free to interpret the GPL themselves.  And if that
> > person/group decides that a 4-clause BSD license is OK to include in THEIR
> > code... then the FSF's opinion on license compatibility is irrelevant
> > legally.
>
> Right.  However, if a license clarification doesn't follow from the license
> text, and/or differs from the common interpretation, then we must be careful.
> For example, if somebody licenses their work under the GPL, and says "by my
> interpretation of the GPL, you don't have to make source available to people
> you send binaries to", then their stated interpretation is clearly inconsistent
> with the actual text of the license.  In those cases, it's best to step
> carefully and conservatively.
>
> My recommendation to those authors would be to issue a specific exception,
> to the effect of "advertising clauses are permitted without being in
> violation of GPL#6", rather than using an unusual interpretation of the
> GPL itself.

  OK.  It's better to be diligent than cavalier on this issue.  If the
BSD advertising clause is even legal, then someone years from now could
pull a SCO and nail you on just the advertising clause alone.

  I looked into the state of advertising clauses in HtDig 3.1.6, they only
exist in the Berkeley DB directory (which Debian's build/package excludes
by linking against a libdb library) and all of them are "Regents of U of
C" which was stricken from the license recently.  So we're kosher ;-)

  In the current HtDig 3.2b6 (LGPL License) a problematic file is
htlib/strptime.cc:
     20  * 3. All advertising materials mentioning features or use of this
     21  *    software must display the following acknowledgement:
     22  *  This product includes software developed by Powerdog Industries.

This file was removed 2 years ago from the 3.1.x branch (strptime isn't
even called in 3.1.6).  I'm not sure why it wasn't removed from the 3.2 Branch.

It's only called once other than it's own internal recursive calls.

I'll either kill it, move to a different version or ask Powerdog to give us a 3-clause license.

If none of that works then we could amend our version of the LGPL License.
This may be advantageous anyway since it allows us to use NetBSD, OpenBSD,
FreeBSD code.... all organizations very unlikely to pull a SCO on anyone.
;-)

I'm taking it for granted that a 4-clause BSD is incompatible with the
LGPL as well since LGPL#10 is the same as GPL#6.  Note that we use the
1991 2.0 LGPL, not the 1999 2.1 'Lesser' version.

> >   Note that I intend no disrespect for any FSF people on this list!  This
> > is just basic copyright law.  A third party can't dictate how a copyright
> > holder interprets any license to it's own code.
>
> And I no disrespect for the authors of htdig, of course.  It's just best to
> be careful and explicit with issues like this.

  Good point.

  Just curious... are there any other examples other than OpenSSL where
Debian amended the GPL or LGPL?  Do you have a general policy discouraging
4-clause BSD Lics (effectively eliminating using *BSD code)?

  Thanks Glen!

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

[htdig-dev] Did anyone find my message about using wvHtml to parse word to HTML for htdig handy?

[<Gle...@co...>]

Hi,

I sent a message on the 7th of this month about using wvHtml to parse word
to HTML for indexing word documents.   Did anyone find this handy?

Cheers
Glen

Re: [htdig-dev] Licensing issues...

[Glenn M. <gl...@ze...>]

On Mon, Jul 26, 2004 at 03:11:52PM -0600, Neal Richter wrote:
> > 2) Can I reasonably argue that htdig is gpl (or lgpl) if its linked
> > against a 3 or a 4 cloause BSD license? - htdig .3.1.6 builds static
> > libraries (.a) it links against.
> 
>   Sure you can!
> 
>   Note that although the Free Software Foundation may say that a 4-clause
> BSD license is incompatible with the GPL (and they do)..... that opinion only
> applies to code that the FSF holds the copyright for.

Not quite.  I believe that opinion derives directly from the text of the
license.  The GPL prohibits adding further restrictions (GPL#6), and the
4-clause BSD license does so.  Unless a copyright holder specifically says
that he considers the 4-clause BSD license consistent with the GPL, it's
not safe for Debian to assume otherwise.

That is, the "default" interpretation, lacking a statement from (all of)
the copyright holders of a work, should be the one that follows from the
license text, and that's the FSF's interpretation, at least in this case.

>   As a matter of copyright law, any copyright holder that licenses their
> code under the GPL is free to interpret the GPL themselves.  And if that
> person/group decides that a 4-clause BSD license is OK to include in THEIR
> code... then the FSF's opinion on license compatibility is irrelevant
> legally.

Right.  However, if a license clarification doesn't follow from the license
text, and/or differs from the common interpretation, then we must be careful.
For example, if somebody licenses their work under the GPL, and says "by my
interpretation of the GPL, you don't have to make source available to people
you send binaries to", then their stated interpretation is clearly inconsistent
with the actual text of the license.  In those cases, it's best to step
carefully and conservatively.

My recommendation to those authors would be to issue a specific exception,
to the effect of "advertising clauses are permitted without being in
violation of GPL#6", rather than using an unusual interpretation of the
GPL itself.

I'm sure somebody else on d-legal can offer a better statement than I can,
though.

>   Note that I intend no disrespect for any FSF people on this list!  This
> is just basic copyright law.  A third party can't dictate how a copyright
> holder interprets any license to it's own code.

And I no disrespect for the authors of htdig, of course.  It's just best to
be careful and explicit with issues like this.

For example, all (known) GPL projects in Debian which link against OpenSSL
have such an exception, because it has an advertising clause, as well as
forced-renaming and a forced-acknowledgement clauses.

-- 
Glenn Maynard

Re: [htdig-dev] Licensing issues...

[Neal R. <ne...@ri...>]

> Here's my set of questions:
>
> 2) Can I reasonably argue that htdig is gpl (or lgpl) if its linked
> against a 3 or a 4 cloause BSD license? - htdig .3.1.6 builds static
> libraries (.a) it links against.

  Sure you can!

  Note that although the Free Software Foundation may say that a 4-clause
BSD license is incompatible with the GPL (and they do)..... that opinion only
applies to code that the FSF holds the copyright for.

  As a matter of copyright law, any copyright holder that licenses their
code under the GPL is free to interpret the GPL themselves.  And if that
person/group decides that a 4-clause BSD license is OK to include in THEIR
code... then the FSF's opinion on license compatibility is irrelevant
legally.

  Note that I intend no disrespect for any FSF people on this list!  This
is just basic copyright law.  A third party can't dictate how a copyright
holder interprets any license to it's own code.

  Here's where it gets interesting:  Debian could, by the rights given to
you by HtDig's use of the GPL (for 3.1.x and earlier), decide independently
that a 4-clause BDB license isn't OK.... but again.. that doesn't affect
us (HtDig) ... it just makes your head hurt!

Hey FSF people:  Why don't you upgrade your version of HtDig on
fsf.org!!??

Thanks!

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] Licenses..

[Neal R. <ne...@ri...>]

Robert,

  Thanks for your efforts here!  We have little contact from the other
Linux distros.... which tells me that Debian is more on-the-ball on
License audits of code that is included.

> It is also my understanding that Sleepycat's license for DB is compatible
> with GPL, but not LGPL.

  Yep.

> It's a whole other ball of wax for the 3.2 code base.  There, Neal
> negotiated a special license with Sleepycat that allows us to distribute
> our modified DB code with the LGPL'ed 3.2 ht://Dig code, provided the
> bundled DB code is used only for ht://Dig or libhtdig.  If it's unbundled,
> it reverts to Sleepycat's standard license.
>
> Neal and Geoff, please chime in if you have anything to add.

[This next part applies only to HtDig 3.2.x]

  Note that for 3.2 the HtDig license is LGPL.

  If a 'LGPLed 3.2 HtDig + custom DBD with custom Sleepycat License'
causes you license indigestion for Debian..... feel free to make the whole
thing GPL and remove the special Sleepycat/BDB License Exception in your
source package.

  If you do this, we would kindly ask that any patches you submit to us
for HtDig 3.2.x be LGPLed by you so we can include them in the stock HtDig
3.2.x tree.

  I know this is a bit wierd, but it's legit since the HtDig group is
allowed by copyright law (as copyright holders of our code) to interpret
the LGPL as we see fit.

  And Sleeycat did the same by making a HtDig specific exception to their
license to allow us to move to the LGPL and still use BDB.

  FYI:  It looks like for HtDig 4.0 we MAY be abandoning BDB, and be
replacing it with an LGPL'd Lucene/CLucene.. so the wierd licensing issue
goes away.

Thanks again.

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

[htdig-dev] Unbundling libdb2..

[Robert R. <ri...@li...>]

Hello,

htdig 3.16 comes with libdb2 (Version 2.6.4) it statically links 
against. I have managed to unbundle this Berkley DB libraries without 
much problem, I now (dynamically) link against the libdb2 package in 
debian. To build, libdb2-dev seems to be required. The whole db direcory 
can safely be deleted, as this is now provided by the external library.

The debian libdb2(-dev) packages (Version 2.7.7.0) come with a standard, 
3-clause BSD-style license.  Also, htdig will be able ot benefit from 
improvements in the libdb2 code. Unfortunately, it does not link against 
newer versions of the library (V3 or V4), the interface has changed there.

Robert

Re: [htdig-dev] Licenses..

[Geoff H. <ghu...@ws...>]

On Jul 22, 2004, at 12:24 PM, Gilles Detillieux wrote:

> My understanding, though I may be wrong (Geoff Hutchison could provide
> the definitive answer), is that the 3.1.x code base does not include
> any extensions or customisations to the Sleepycat Berkeley DB code,

This is correct. Over the 3.1.x series I think I made one update to the 
Sleepycat code when there were some significant bug fixes and it would 
not break database compatibility.

In the 3.1.x series, the db/ directory is a completely unmodified 
Berkeley DB distribution. The ChangeLog should give information on 
which version has been used, but I seem to remember it's something 
around version 3 of the Sleepycat releases.

> It is also my understanding that Sleepycat's license for DB is 
> compatible
> with GPL, but not LGPL.

That is correct.

> It's a whole other ball of wax for the 3.2 code base.  There, Neal
> negotiated a special license with Sleepycat that allows us to 
> distribute
> our modified DB code with the LGPL'ed 3.2 ht://Dig code, provided the
> bundled DB code is used only for ht://Dig or libhtdig.  If it's 
> unbundled,
> it reverts to Sleepycat's standard license.

Correct.

So if you're looking at the licenses for htdig-3.1.x, and wonder about 
the licenses in the db/ directory, you should contact Sleepycat with 
questions or check the Debian Berkeley DB packaging for that particular 
version. At one point, I believe the Debian htdig package simply 
required the appropriate BDB version and linked against the shared 
library rather than compile htdig with the db/ directory.

The main reasons we packaged the two together in the 3.1.x releases 
were that:
a) At the time, not as many people had the Berkeley DB code and it 
would be a bit of a pain to grab another package, install it, then 
install htdig. (This of course is much easier with packaging systems.)
b) It ensured that everyone running htdig was using the same version of 
the BDB and so we wouldn't have to do lots of extra testing for any 
bugs that cropped up with newer BDB versions and/or the interface code.

-Geoff

Re: [htdig-dev] rundig again..

[Gilles D. <gr...@sc...>]

According to Robert Ribnitz:
> 
> I know I am commenting on old code, but please have a look at:
> 
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=117887
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=139922
> 
> both revolve around the grep in rundig (for 3.1.6)
> 
> I have added the ^ in the debian package as the poster suggests

Another problem with this grep statement that was added to rundig (it's
not in the one bundled with 3.1.6 or 3.2.0b6), is that it sets DBDIR only
to the database_dir value specified in the default htdig.conf file.  If
an alternate config file is specified via -c, the script doesn't seem to
take that into account and grab the database_dir value from there.

A better approach is to do it as in the rundig script bundled in 3.2.0b6:
here it grabs the database_dir value after the command line arguments
are parsed.  I believe you could safely replace lines 13-24 of the 3.1.6
rundig script with lines 13-44 from 3.2.0b6's rundig to get the same
effect in 3.1.6.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] new rundig script (was: Various things..)

[Gilles D. <gr...@sc...>]

Yes, in both 3.1.6 and 3.2.0b6, and indeed as far back as I can
recall, the standard rundig script bundled with ht://Dig (installed
from installdir) has always passed -i to htdig by default.  I have no
problems with Robert's modified script, but be well aware that bundling
this one will mean that the default action of rundig will change from
what it was before.  That could lead to a lot of user confusion.

Mind you, there's already a lot of user confusion about rundig, so
the question is whether and by how much this change will aggravate
the confusion.  ;-P

On the plus side, Robert's script adds some flexibility to rundig, and
makes its usage more consistent with htdig's (i.e. update by default,
and use -i to reindex from scratch).

According to Lachlan Andrew:
> Greetings Robert,
> 
> In 3.2.0b6,  rundig  automatically passes  -i  to htdig.  If it didn't 
> in 3.1, that may account for some of the criticisms of its speed.  I 
> have been meaning to make it optional, but didn't want to break any 
> compatibility -- it looks like I don't have to worry about that.
> 
> Unfortunately, the role of  htmerge  has changed between 3.1 and 3.2 
> -- after  htdig,  the script should now call  htpurge  instead, so 
> I'm not sure if the 3.1.6 script will work as expected.
> 
> Cheers,
> Lachlan
> 
> On Wed, 21 Jul 2004 10:17 pm, Robert Ribnitz wrote:
> > I have a few observations:
> >
> > - it is possible to run htdig -i, meaning that the update is full
> > rather than incremental ('initial').
> > - if I use rundig -i, that option is passed to htdig (fine),  but
> > also to htpurge and  htnotify  which error out, since they don't
> > know any option -i
> >
> > I have therefore adapted the 'rundig' script of htdig-3.1.6, and
> > attach it here..
> >
> > Please test..
> >
> > Robert


-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] Licenses..

[Gilles D. <gr...@sc...>]

According to Robert Ribnitz:
> I am in the process of making my 'htdig' package (the one based on  3.1.6)
> lintian-clean (Lintian is a special program that checks for conformance 
> of the package to the debian packaging guidelines).
> 
> I found the following licenses (which I need to list in copyright):
> 
> Berkeley DB      Sleepycat license (4 clause BSD license listed in the 
> code, the current sleepycat license is 3 clause BSD)
> 
> your extensions
> to Berkeley DB   no idea (3 clause BSD?)
> 
> Config.guess/
> Config.sub          GPL Version 2                
> 
> htdig (3.1.6)       GPL Version 2

My understanding, though I may be wrong (Geoff Hutchison could provide
the definitive answer), is that the 3.1.x code base does not include
any extensions or customisations to the Sleepycat Berkeley DB code,
so it should be possible to update to a newer DB version (and license)
with minimal difficulty.  Exactly what licenses can now be applied to
the old version of DB bundled with 3.1.6 is probably more a question
for Sleepycat than for us, as I don't believe we've changed it.

It is also my understanding that Sleepycat's license for DB is compatible
with GPL, but not LGPL.

It's a whole other ball of wax for the 3.2 code base.  There, Neal
negotiated a special license with Sleepycat that allows us to distribute
our modified DB code with the LGPL'ed 3.2 ht://Dig code, provided the
bundled DB code is used only for ht://Dig or libhtdig.  If it's unbundled,
it reverts to Sleepycat's standard license.

Neal and Geoff, please chime in if you have anything to add.

> Config.guess/Config.sub are used to compile the Berkeley DB. Also, what 
> exactly are your extensions to Berkeley DB (that prevent replacing that 
> version with something more recent?).
> 
> I also replaced Config.guess/Config.sub with a more current version (2004).
> 
> At the moment,  the program still compiles with gcc/g++ 3.3,  with 
> deprecation warnings. I don't know whether it will still compile with 
> gcc-3.4

Our standard advice for compiling 3.1.6 under gcc/g++ 3.x is in
http://www.htdig.org/FAQ.html#q3.8

I don't know if replacing Config.guess/Config.sub makes those environment
variables unnecessary or not, but they are supposed to suppress the
deprecation warnings (unless there are new warnings in 3.3 or 3.4).
Keep us posted on how that all turns out.  Thanks.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] Licensing issues...

[Robert R. <ri...@li...>]

Gabriele Bartolini wrote:

>Hi Robert,
>
>   I am not the ideal person to give you an answer about licensing.
>However, I remember that for Berkeley DB ht://Dig has a special license
>(db/LICENSE), which allows version 3.0.55 (patched) to be distributed
>along with ht://Dig (and only with it).
>
>   It is an "ad-hoc" and special license and I personally would not
>mention it in the package description. I would therefore leave GPL
>(LGPL).
>  
>
Dear Gabriele, dear List,

unfortunately I have to.

In there I have:

1 x 3 Clause BSD License ('Sleepycat Software Inc")
2 x 4 Clause BSD  (" The President and Fellows of Harvard University.", 
" The Regents of the University of California")
  
According to http://www.opensource.org/licenses/bsd-license.php
/Note/: The advertising clause in the license appearing on BSD Unix 
files was officially rescinded 
<ftp://ftp.cs.berkeley.edu/pub/4bsd/README.Impt.License.Change> by the 
Director of the Office of Technology Licensing of the University of 
California on July 22 1999. He states that clause 3 is "hereby deleted 
in its entirety."

Do you think we can settle for : 3x 3 clause BSD license for this 
version of Berkeley DB with its modifications?

This would leave us witth the following:

Berkeley DB                                        3-clause BSD
configure.guess/confiuigure.sub              GPL v2
htdig itself                                              GPL v2

>Please guys say yours ... :-)
>
>Ciao and thank you Robert!
>-Gabriele
>
Robert

Re: [htdig-dev] htdig (3.1.6) search tips?

[Robert R. <ri...@li...>]

Lachlan Andrew wrote:

>Greetings Robert,
>
>Thanks for pointing that out to us.  (Out of interest, the best way to 
>"handball" issues to us through the sourceforge bug list 
><http://sourceforge.net/tracker/?group_id=4593&atid=104593>.)
>
>How does the attached documentation (for 3.2) rate?  Please comment on 
>it before I commit it.
>
>Thanks,
>Lachlan
>  
>
A document like this is ok, please commit (and provide a patch to 3.1.6, 
so I can add it as well.. Perhaps link to a shorter version of it from 
the default search template?


How much difference will there be between 3.1.6 and 3.2 beta ? (As you 
know, 'htdig' is going ot be the stable version; the beta will be 
packahed as htdig 3.2)

Robert

Re: [htdig-dev] htdig (3.1.6) search tips?

[Robert R. <ri...@li...>]

Lachlan Andrew wrote:

>Greetings Robert,
>
>Thanks for pointing that out to us.  (Out of interest, the best way to 
>"handball" issues to us through the sourceforge bug list 
><http://sourceforge.net/tracker/?group_id=4593&atid=104593>.)
>
>How does the attached documentation (for 3.2) rate?  Please comment on 
>it before I commit it.
>
>Thanks,
>Lachlan
>
>On Thu, 22 Jul 2004 12:25 am, Robert Ribnitz wrote:
>  
>
>>Ok, filling up the list.
>>
>>could you add the resolution of  bug 165420 to the todo list on the
>>webpage?
>>
>>http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=165420
>>
>>Its about search tips, or rather the syntax of expressions.
>>
>>Perhaps a FAQ entry on the syntax of search expressions?
>>    
>>
>
>  
>
>
> ------------------------------------------------------------------------
>
>
>   htsearch
>
What you prove here is certainly a good start; however the focus of it 
is clearly the developer of the search form.

What the authors of the bug had in mind was probably something more 
along the lines of:

http://www.google.ch/help/basics.html

or

http://www.google.ch/help/refinesearch.html

The focus of those documents is the end-user, not the developer of the 
search form.

I'll provide such a page shortly

Robert

Re: [htdig-dev] htdig (3.1.6) search tips?

[Lachlan A. <lh...@us...>]

Greetings Robert,

Thanks for pointing that out to us.  (Out of interest, the best way to 
"handball" issues to us through the sourceforge bug list 
<http://sourceforge.net/tracker/?group_id=4593&atid=104593>.)

How does the attached documentation (for 3.2) rate?  Please comment on 
it before I commit it.

Thanks,
Lachlan

On Thu, 22 Jul 2004 12:25 am, Robert Ribnitz wrote:
> Ok, filling up the list.
>
> could you add the resolution of  bug 165420 to the todo list on the
> webpage?
>
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=165420
>
> Its about search tips, or rather the syntax of expressions.
>
> Perhaps a FAQ entry on the syntax of search expressions?

-- 
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

Re: [htdig-dev] Licensing issues...

[Gabriele B. <g.b...@co...>]

Hi Robert,

   I am not the ideal person to give you an answer about licensing.
However, I remember that for Berkeley DB ht://Dig has a special license
(db/LICENSE), which allows version 3.0.55 (patched) to be distributed
along with ht://Dig (and only with it).

   It is an "ad-hoc" and special license and I personally would not
mention it in the package description. I would therefore leave GPL
(LGPL).

Please guys say yours ... :-)

Ciao and thank you Robert!
-Gabriele

Il gio, 2004-07-22 alle 11:06, Robert Ribnitz ha scritto:
> Hello List,
> 
> I am in the process of making my package ('htdig') lintian-clean. Part 
> of that cleanness (lintian does not complain about) is getting a summary 
> of all the licenses used into the copyright file.
> 
> My situation is as follows (for htdig 3.1.6):
> 
> htdig itself is GPL, more recent versions are LGPL
> config.guess/config.sub is GPL
> Berkley DB (v. 2.6.4) comes with a sleepycat license (4-clause BSD 
> style), the current sleepycat license is a 3-clause BSD-style license.
> They extended Berkeley DB in some way, I guess those extensions are 
> BSD-style, hopefully a 3 clause one.
> 
> Here's my set of questions:
> 
> 1) Can I assume the more recent 3-clause BSD to also apply to that 
> version of Berkeley DB, or am I bound by the 4 clause one?
> 2) Can I reasonably argue that htdig is gpl (or lgpl) if its linked 
> against a 3 or a 4 cloause BSD license? - htdig .3.1.6 builds static 
> libraries (.a) it links against.
> 
> In thope to not to be completely off-topic here
> 
> Robert Ribnitz
> Debian Maintainer  of htdig, NM in application.
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by BEA Weblogic Workshop
> FREE Java Enterprise J2EE developer tools!
> Get your free copy of BEA WebLogic Workshop 8.1 today.
> http://ads.osdn.com/?ad_id=4721&alloc_id=10040&op=click
> _______________________________________________
> ht://Dig Developer mailing list:
> htd...@li...
> List information (subscribe/unsubscribe, etc.)
> https://lists.sourceforge.net/lists/listinfo/htdig-dev

Re: [htdig-dev] source code

[Lachlan A. <lh...@us...>]

Greetings William,

On Tue, 20 Jul 2004 02:48 pm, xaxx wrote:
> Is there anyone there?

Yes, we're here :)

> All the site updates appear to end in 2002,
> and there does not appear to be a clear explanation of the search
> algorithm or the source code for the project anywhere on the site.

Things have been a bit slow of late, especially the documentation, but 
we're still working on it all.

> We have been using ht:/dig for our professional website and there
> are some very peculiar results which our webmaster cannot explain -
> largely because he does not understand the program.
>
> I have been asked to unravel the problem, and I can be patient as
> needed, but not if there is no one minding the store anymore.
>
> Please let me know how to get a copy of at least the htsearch
> module, and some decent documentation of the algorithm it
> implements.

Since you've been using it for some time, I assume you're using 
version 3.1.6.  The source is at 
<http://www.htdig.org/files/htdig-3.1.6.tar.gz>.  The search code is 
in the .../htsearch  subdirectory.  Search for "factor" in parser.cc 
and display.cc for the ranking calculations.

However, in 3.1.x, a lot of the ranking is done by  htdig  itself, so 
you'll probably need to look there if you are tracking down an 
anomoly.  Search for  factor  in  htcommon/DocumentRef.cc, 
htcommon/WordList.cc, htdig/Retriever.cc  to find some relevant code.

I am not aware of any documentation of the actual weighting algorithm.  
Perhaps once you have unravelled your mystery, you may be the best 
qualified person to write it!  I'll be happy to help out as I can, 
but I don't know the 3.1 code very well.

Cheers,
Lachlan

-- 
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

[htdig-dev] Licensing issues...

[Robert R. <ri...@li...>]

Hello List,

I am in the process of making my package ('htdig') lintian-clean. Part 
of that cleanness (lintian does not complain about) is getting a summary 
of all the licenses used into the copyright file.

My situation is as follows (for htdig 3.1.6):

htdig itself is GPL, more recent versions are LGPL
config.guess/config.sub is GPL
Berkley DB (v. 2.6.4) comes with a sleepycat license (4-clause BSD 
style), the current sleepycat license is a 3-clause BSD-style license.
They extended Berkeley DB in some way, I guess those extensions are 
BSD-style, hopefully a 3 clause one.

Here's my set of questions:

1) Can I assume the more recent 3-clause BSD to also apply to that 
version of Berkeley DB, or am I bound by the 4 clause one?
2) Can I reasonably argue that htdig is gpl (or lgpl) if its linked 
against a 3 or a 4 cloause BSD license? - htdig .3.1.6 builds static 
libraries (.a) it links against.

In thope to not to be completely off-topic here

Robert Ribnitz
Debian Maintainer  of htdig, NM in application.

[htdig-dev] Licenses..

[Robert R. <ri...@li...>]

Hello again,

I am in the process of making my 'htdig' package (the one based on  3.1.6)
lintian-clean (Lintian is a special program that checks for conformance 
of the package to the debian packaging guidelines).

I found the following licenses (which I need to list in copyright):

Berkeley DB      Sleepycat license (4 clause BSD license listed in the 
code, the current sleepycat license is 3 clause BSD)

your extensions
to Berkeley DB   no idea (3 clause BSD?)

Config.guess/
Config.sub          GPL Version 2                

htdig (3.1.6)       GPL Version 2

Config.guess/Config.sub are used to compile the Berkeley DB. Also, what 
exactly are your extensions to Berkeley DB (that prevent replacing that 
version with something more recent?).

I also replaced Config.guess/Config.sub with a more current version (2004).

At the moment,  the program still compiles with gcc/g++ 3.3,  with 
deprecation warnings. I don't know whether it will still compile with 
gcc-3.4

yours

Robert