ht://Dig

> This may sound like heresy, but should we back-port all of the useful
> features of ht://Dig 3.2 into 3.1 and give up on the 3.2 branch?  Joe
> showed a factor of 4 speed difference, and today I was getting a
> factor of over 10 difference (after much tweaking of the 3.2 code).

  Interesting... I guess it's time to do some heavy profiling.

  On the plus side of 3.2.0 is that we're tested it at RightNow very
well.. very few bugs so far.

> I'm not sure who wrote the new database code, but they seem to have
> left the team and no-one seems to be jumping up to pick up the
> pieces.  I certainly don't have the knowledge, or the time to learn.
> For a while, Neal seemed to have the interest and skill needed for
> rewriting the code, but he seems to have found greener pastures too.
> (Are you still with us, Neal?)

  Yes I am.... I'm working on HtDig code (or code that uses htdig)
frequently...  sorry I've been quiet... working on a PhD part-time and
working full-time...

> Does anyone know what new features require the new database structure?
> Phrase searching is the big one, and compression is the other one
> which will take a lot of work.  There's also the ability to set
> weights at query time rather than dig time, but that is presumably
> just a matter of replacing a "weight" field with a "flags" field and
> keeping the rest of the database structure unchanged.
>
> In addition, 3.1.6 builds much faster, and doesn't use automake (which
> I would like -- those tools seem a nightmare).
>
> My vote is that we release 3.2.0b6 basically as the code stands now,
> and then start 3.3.0a1 by back-porting features to 3.1.6.  For each,
> we'll measure the impact on performance, and decide which ones are
> worth it.

  I'd agree with the first part, but not the second until we try and
figure out why the code is slower.


Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Honestly after looking at lots of htdig 3.2.0 code to build and debug
libhtdig... there is a TON of redundant and confusing code in it.

  We've implemented way too many of our own data structures... we should
be using the STL... and the document parsing code is very wild to trace
through.

  And the searching code needs help, and we have a prototype from Loic for
new search code.

> My vote is that we release 3.2.0b6 basically as the code stands now,
> and then start 3.3.0a1 by back-porting features to 3.1.6.  For each,
> we'll measure the impact on performance, and decide which ones are
> worth it.

  There is another alternative to either flushing out the inefficient
cruft in 3.2.0 or backporting to 3.1.6

  We could look at integrating with Clucene, it's a C++ search engine.  It
is just the store-search portion.... no spidering engine and no UI.  It's
an API (as far as I can tell).

  It's a C++ rewrite of Java Lucene.

  It's worth considering... but would be a lot of work.  We would have to
carefully examine which htdig configs we could still support.

  The advantage is that CLucene is under active development by experienced
search-engine people, I believe one of the participants is an original
Altavista developer.   It's a fairly small code base, and it's LGPL.

  The disadvantages are that at the moment there is no DB compression,
it's not an enduser application (where HtDig is), and it will be a lot
of work.

Would we all be satisfied if we used a different project's 'guts'?  For
that matter we could look at moving our spidering code to use a different
library.

HtDig would concentrate on providing a flexible application
built upon a separate store/search engine and possibly other libraries for
things like spidering and user interface.

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

On Sat, 1 May 2004, Lachlan Andrew wrote:

> My vote is that we release 3.2.0b6 basically as the code stands now, 
> and then start 3.3.0a1 by back-porting features to 3.1.6.  For each, 
> we'll measure the impact on performance, and decide which ones are 
> worth it.
> 
> Thoughts?

If it is decided that development on the 3.2 branch is to be more or less
abandoned, I would suggest that we consider backporting only those pieces
that are most key to improving the usefulness of 3.1, which I believe to a
large extent consists of providing support for phrase searching and
perhaps addressing 32-bit limits imposed on database size. This of course
assumes that it is even feasible to do this without ripping the existing
code to shreds. If it is possible, I would suggest leaving the resulting
version on the 3.1 branch and then taking a step back and putting a lot of
thought into where to head with any 3.3 branch. Perhaps take a look at
other projects that could benefit future development (e.g. lucene and its
derivatives), other sorts of packaging (e.g. libraries, search engine
server), and what in general is happening these days with regard to search
products and technology.


Jim

Greetings Neal,

It's really good to hear from you!  Good luck with the PhD -- you are 
a brave man doing that part time!

Using Clucene is a very good idea.  (I'm Cc'ing this to the clucene 
developers' list, to hear their opinions.)  We had been talking about 
doing that eventually anyway.  We're already using another project's 
'guts' by using Mifluz/BDB, so I don't see any ideological problem 
there -- it's just a matter of scale.  In fact, I'd be perfectly 
happy to be subsumed by Clucene, as long as week keep backward 
compatibilty with ht://Dig.  As you say, the role of ht://Dig 
(spidering and user interface) is complementary to that of Clucene.  

The big problem is the amount of work, but all of the options are a 
lot of work.  I can really only afford to spend a couple of hours a 
week on ht://Dig.  To be viable, I think we need at least four times 
that (not including support for 3.1.x and 3.2, or developers adding 
new features).  The fact that only two people have so far responded 
to my mail reflects our dire straits...

Lachlan

On Sun, 2 May 2004 05:26 pm, Neal Richter wrote:
>   There is another alternative to either flushing out the
> inefficient cruft in 3.2.0 or backporting to 3.1.6
>
>   We could look at integrating with Clucene.
>
>   It's worth considering... but would be a lot of work.  We would
> have to carefully examine which htdig configs we could still
> support.
>
>   The advantage is that CLucene is under active development by
> experienced search-engine people, I believe one of the participants
> is an original Altavista developer.   It's a fairly small code
> base, and it's LGPL.
>
>   The disadvantages are that at the moment there is no DB
> compression, it's not an enduser application (where HtDig is), and
> it will be a lot of work.
>
> Would we all be satisfied if we used a different project's 'guts'? 
> For that matter we could look at moving our spidering code to use a
> different library.

-- 
lha@...
ht://Dig developer DownUnder  (http://www.htdig.org)

Greetings Jim,

Thanks for your reply.  Your comments about 3.3 are very sensible.  

The big problem with back-porting phrase searching is that this is 
exactly the one that requires the new database structure, which is 
what is giving us the headaches...

The reason I got involved with ht://Dig was so that KDE could search 
its docbook/xml help pages.  That needs the MIME handling code and/or 
external transports which don't have two slashes (like help:/foo).  
Porting these to 3.1 would not cause the headaches that changing the 
database structure entails.  I'm sure there are other examples of 
"light" features that could be back-ported without major dramas.

Cheers,
Lachlan

On Sun, 2 May 2004 06:55 pm, Jim wrote:
> On Sat, 1 May 2004, Lachlan Andrew wrote:
> > My vote is that we release 3.2.0b6 basically as the code stands
> > now, and then start 3.3.0a1 by back-porting features to 3.1.6. 
> > For each, we'll measure the impact on performance, and decide
> > which ones are worth it.
> >
> > Thoughts?
>
> If it is decided that development on the 3.2 branch is to be more
> or less abandoned, I would suggest that we consider backporting
> only those pieces that are most key to improving the usefulness of
> 3.1, which I believe to a large extent consists of providing
> support for phrase searching and perhaps addressing 32-bit limits
> imposed on database size.

-- 
lha@...
ht://Dig developer DownUnder  (http://www.htdig.org)

On Sat, 8 May 2004, Lachlan Andrew wrote:

> there -- it's just a matter of scale.  In fact, I'd be perfectly
> happy to be subsumed by Clucene, as long as week keep backward
> compatibilty with ht://Dig.  As you say, the role of ht://Dig
> (spidering and user interface) is complementary to that of Clucene.

  I believe the CLucene developers are more focused on providing a
indexing/searching library that other can build an application with.

  And given everyones time constraints, it makes more sense long-term to
throw in with them and participate.

>
> The big problem is the amount of work, but all of the options are a
> lot of work.  I can really only afford to spend a couple of hours a
> week on ht://Dig.  To be viable, I think we need at least four times

  It will be no trivial amount of work to convert htdig to use CLucene as
the index/search guts.... unless we make the decision that all
configuration options related to the current search/index code are
eligible for removal... we'll keep the ones we can, change the
definitions of some and others will die.

  If we try and be fully backwards compatible with the current subset of
search/index configs... we'll be asking for trouble.

  I'm subscribed to the CLucene list.. it's quite active and motivated...
and they are working on UTF-8 support... which is motivation enough to
consider the switch.

  I do think we should consider releasing 3.2.0 after a bit of
profiling...

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Hi Lachlan - Based on your response, I guess I am not clear on exactly
what you were suggesting originally. It sounded like you wanted feedback
on the idea of creating a 3.3 branch that was 3.1.6 + back-ports. Were you
excluding phrase search from the candidate back-ports? I would argue
that phrase search is probably the most important missing feature in the
3.1.6 branch and unless it is to be added I am not sure that I see the
value of a new branch or the investment of significant effort aimed at
enhancing 3.1.6.

If adding some level of proximity search to 3.1.6 is considered too
disruptive and 3.2 is leading down the wrong path, both of which seem at
least somewhat likely at this point, then my suggestion would be to pick
up some of the easy wins with a 3.1.7 and give thought to a 3.3 branch
that does not start out overly constrained by any existing code base.

Hopefully some of the people with better knowledge of the current code's
features and limitations can chime in at some point.

Jim


On Sat, 8 May 2004, Lachlan Andrew wrote:

> Greetings Jim,
> 
> Thanks for your reply.  Your comments about 3.3 are very sensible.  
> 
> The big problem with back-porting phrase searching is that this is 
> exactly the one that requires the new database structure, which is 
> what is giving us the headaches...
> 
> The reason I got involved with ht://Dig was so that KDE could search 
> its docbook/xml help pages.  That needs the MIME handling code and/or 
> external transports which don't have two slashes (like help:/foo).  
> Porting these to 3.1 would not cause the headaches that changing the 
> database structure entails.  I'm sure there are other examples of 
> "light" features that could be back-ported without major dramas.
> 
> Cheers,
> Lachlan
> 
> On Sun, 2 May 2004 06:55 pm, Jim wrote:
> > On Sat, 1 May 2004, Lachlan Andrew wrote:
> > > My vote is that we release 3.2.0b6 basically as the code stands
> > > now, and then start 3.3.0a1 by back-porting features to 3.1.6. 
> > > For each, we'll measure the impact on performance, and decide
> > > which ones are worth it.
> > >
> > > Thoughts?
> >
> > If it is decided that development on the 3.2 branch is to be more
> > or less abandoned, I would suggest that we consider backporting
> > only those pieces that are most key to improving the usefulness of
> > 3.1, which I believe to a large extent consists of providing
> > support for phrase searching and perhaps addressing 32-bit limits
> > imposed on database size.
> 
> -- 
> lha@...
> ht://Dig developer DownUnder  (http://www.htdig.org)
> 
> 
> 
>

On Sun, 9 May 2004 11:38 am, Jim wrote:
> Hi Lachlan - Based on your response, I guess I am not clear on
> exactly what you were suggesting originally. It sounded like you
> wanted feedback on the idea of creating a 3.3 branch that was 3.1.6
> + back-ports. Were you excluding phrase search from the candidate
> back-ports?

I'm not really clear on what I was suggesting either :)

I didn't mean to exclude phrase search entirely.  It probably isn't 
surprising that the most useful enhancement is also the most 
troublesome one.

> If adding some level of proximity search to 3.1.6 is considered too
> disruptive and 3.2 is leading down the wrong path, both of which
> seem at least somewhat likely at this point, then my suggestion
> would be to pick up some of the easy wins with a 3.1.7 and give
> thought to a 3.3 branch that does not start out overly constrained
> by any existing code base.
>
> Hopefully some of the people with better knowledge of the current
> code's features and limitations can chime in at some point.

My impression is that the ht://Dig project is basically dead :(  The 
existing code is of course still functional (and thanks Jim and 
Gilles for all the support you give to the users!), but I don't think 
there is enough enthusiasm to either release a new version, either 
3.2 or 3.3. If I get enthusiastic in the next couple of weeks, I 
might still try to put 3.2.0b6 together, but that is about as far as 
it will go...

It's been nice getting to know you all :)

Lachlan

-- 
lha@...
ht://Dig developer DownUnder  (http://www.htdig.org)

On Thu, 2004-05-13 at 08:58, Lachlan Andrew wrote:
> My impression is that the ht://Dig project is basically dead :(  The 
> existing code is of course still functional (and thanks Jim and 
> Gilles for all the support you give to the users!), but I don't think 
> there is enough enthusiasm to either release a new version, either 
> 3.2 or 3.3. If I get enthusiastic in the next couple of weeks, I 
> might still try to put 3.2.0b6 together, but that is about as far as 
> it will go...

  As a user, I'm very sorry to hear this -- I just deployed 3.2.0b5 on a
site I administrate, and I've been very pleased with it.  I've been
waiting until the 3.2.0 release cycle was over to start trying to
contribute some of my tweaks (I also need to talk to my employer about
the legalities first), but I guess if the project is stagnating I should
speak up (there's also the possibility that the imminent death of htdig
just makes this extra silly, of course... *shrug*).  In any case:
  So htdig does a bad job when multiple documents match a search in
similar ways; this shows up particularly when your search query matches
part of the header or footer of a section of your site, or when your
search results include threads from a mailing list archive (in which
case messages within a thread often show up consecutively in the
results, which adds a lot of noise).  I wrote some code (shoehorned in
as a ScoreMatch, more for easy control by the 'sort' parameter than for
any logical reason) which sorts the results once, then reduces the score
of any match which is similar to matches that are higher in the list,
then resorts the results; thus the high-ranked results that are returned
tend to be more unique than otherwise.  This is marginally helpful with
the header/footer problem (though the excerpts are still usually
identical in that case), and very helpful with the mailing-list-thread
problem. AFAICT it doesn't do too much harm to the results in the normal
case.
  We also found it beneficial to tweak results' scores by matching their
URLs against a handmade list of URL pieces and score-hacking factors
(mailing list archives are mediocre, IRC archives are usually unhelpful,
a particular section of documentation is generally very useful) -- I
know this is gross, but it did wonders for the effectiveness of our
search results, and a coworker of mine convinced me that it's not
totally against nature -- humans really do have special knowledge of
which sections of a site are generally "good," and with an hour or so of
tweaking we got things in a state where close results from a "bad"
section are presented above loose results from a "good" section when
appropriate (more or less).
  It seems to me that it would be useful to generalize these little
hacks into a search parameter listing which hacks should be applied; for
example, to select the two score hacks described in the above paragraphs
you could specify 'result_hacks=unique,urlmatch' in the search query or
htdig.conf.  htdig already has a couple of result hacks that could fit
into this scheme (backlink_factor and date_factor), and I can think of
one more at least that I'd like to add in my copious free time.  It
certainly would seem right to me to be able (a) to add stuff like the
above tweaks to the codebase without forcing everyone to care about it,
and (b) to test, tweak, and reorder the scoring hacks from a query
parameter while trying to get things configured to work well.
  As I said, I've got (wrongly-integrated) code for the two tweaks I
mentioned, which I can try to get into a presentable state, and I might
be able to find time semi-soon to do the work for the general
result_hacks parameter, if there are people that think either of those
would be worthwhile.  Are there such people?

On Thu, 13 May 2004, Lachlan Andrew wrote:

> Date: Thu, 13 May 2004 22:58:06 +1000
> From: Lachlan Andrew <lha@...>
> To: Jim <lists@...>
> Cc: htdig-dev@...
> Subject: Re: [htdig-dev] 3.2.0 - is it worth it??
> 
> On Sun, 9 May 2004 11:38 am, Jim wrote:
> > Hi Lachlan - Based on your response, I guess I am not clear on
> > exactly what you were suggesting originally. It sounded like you
> > wanted feedback on the idea of creating a 3.3 branch that was 3.1.6
> > + back-ports. Were you excluding phrase search from the candidate
> > back-ports?
> 
> I'm not really clear on what I was suggesting either :)
> 
> I didn't mean to exclude phrase search entirely.  It probably isn't 
> surprising that the most useful enhancement is also the most 
> troublesome one.
> 
> > If adding some level of proximity search to 3.1.6 is considered too
> > disruptive and 3.2 is leading down the wrong path, both of which
> > seem at least somewhat likely at this point, then my suggestion
> > would be to pick up some of the easy wins with a 3.1.7 and give
> > thought to a 3.3 branch that does not start out overly constrained
> > by any existing code base.
> >
> > Hopefully some of the people with better knowledge of the current
> > code's features and limitations can chime in at some point.
> 
> My impression is that the ht://Dig project is basically dead :(  The 
> existing code is of course still functional (and thanks Jim and 
> Gilles for all the support you give to the users!), but I don't think 
> there is enough enthusiasm to either release a new version, either 
> 3.2 or 3.3. If I get enthusiastic in the next couple of weeks, I 
> might still try to put 3.2.0b6 together, but that is about as far as 
> it will go...
> 
> It's been nice getting to know you all :)
> 
> Lachlan

You have been a great bulwark for htdig, specially recently.  You have
brought 3.2 to the verge of release.  How can you make such an ominous
pronouncement?  It can't be just the indexing speed; is it?

Regards,

Joe
-- 
     _/   _/_/_/       _/              ____________    __o
     _/   _/   _/      _/         ______________     _-\<,_
 _/  _/   _/_/_/   _/  _/                     ......(_)/ (_)
  _/_/ oe _/   _/.  _/_/ ah        jjah@...

On Thu, 13 May 2004, Andrew Moise wrote:

> Date: Thu, 13 May 2004 12:46:01 -0400
> From: Andrew Moise <chops@...>
> To: lha@...
> Cc: Jim <lists@...>, htdig-dev@...
> Subject: Re: [htdig-dev] 3.2.0 - is it worth it??
> 
> On Thu, 2004-05-13 at 08:58, Lachlan Andrew wrote:
> > My impression is that the ht://Dig project is basically dead :(  The 
> > existing code is of course still functional (and thanks Jim and 
> > Gilles for all the support you give to the users!), but I don't think 
> > there is enough enthusiasm to either release a new version, either 
> > 3.2 or 3.3. If I get enthusiastic in the next couple of weeks, I 
> > might still try to put 3.2.0b6 together, but that is about as far as 
> > it will go...
> 
>   As a user, I'm very sorry to hear this -- I just deployed 3.2.0b5 on a
> site I administrate, and I've been very pleased with it.  I've been
> waiting until the 3.2.0 release cycle was over to start trying to
> contribute some of my tweaks (I also need to talk to my employer about
> the legalities first), but I guess if the project is stagnating I should
> speak up (there's also the possibility that the imminent death of htdig
> just makes this extra silly, of course... *shrug*).  In any case:
>   So htdig does a bad job when multiple documents match a search in
> similar ways; this shows up particularly when your search query matches
> part of the header or footer of a section of your site, or when your
> search results include threads from a mailing list archive (in which
> case messages within a thread often show up consecutively in the
> results, which adds a lot of noise).  I wrote some code (shoehorned in
> as a ScoreMatch, more for easy control by the 'sort' parameter than for
> any logical reason) which sorts the results once, then reduces the score
> of any match which is similar to matches that are higher in the list,
> then resorts the results; thus the high-ranked results that are returned
> tend to be more unique than otherwise.  This is marginally helpful with
> the header/footer problem (though the excerpts are still usually
> identical in that case), and very helpful with the mailing-list-thread
> problem. AFAICT it doesn't do too much harm to the results in the normal
> case.
>   We also found it beneficial to tweak results' scores by matching their
> URLs against a handmade list of URL pieces and score-hacking factors
> (mailing list archives are mediocre, IRC archives are usually unhelpful,
> a particular section of documentation is generally very useful) -- I
> know this is gross, but it did wonders for the effectiveness of our
> search results, and a coworker of mine convinced me that it's not
> totally against nature -- humans really do have special knowledge of
> which sections of a site are generally "good," and with an hour or so of
> tweaking we got things in a state where close results from a "bad"
> section are presented above loose results from a "good" section when
> appropriate (more or less).
>   It seems to me that it would be useful to generalize these little
> hacks into a search parameter listing which hacks should be applied; for
> example, to select the two score hacks described in the above paragraphs
> you could specify 'result_hacks=unique,urlmatch' in the search query or
> htdig.conf.  htdig already has a couple of result hacks that could fit
> into this scheme (backlink_factor and date_factor), and I can think of
> one more at least that I'd like to add in my copious free time.  It
> certainly would seem right to me to be able (a) to add stuff like the
> above tweaks to the codebase without forcing everyone to care about it,
> and (b) to test, tweak, and reorder the scoring hacks from a query
> parameter while trying to get things configured to work well.
>   As I said, I've got (wrongly-integrated) code for the two tweaks I
> mentioned, which I can try to get into a presentable state, and I might
> be able to find time semi-soon to do the work for the general
> result_hacks parameter, if there are people that think either of those
> would be worthwhile.  Are there such people?

Count one;)  I like both.

Regards,

Joe
-- 
     _/   _/_/_/       _/              ____________    __o
     _/   _/   _/      _/         ______________     _-\<,_
 _/  _/   _/_/_/   _/  _/                     ......(_)/ (_)
  _/_/ oe _/   _/.  _/_/ ah        jjah@...

Greetings Joe,

Thanks for your reply, and don't let my mood swings concern you too 
much :)

No, it isn't just the indexing speed, although that is part of it.  
Mainly, there don't seem to be enough people working on the package 
(coding, documentation).  I'm not meaning to denigrate the great job 
that other people are doing in providing support (your patch 
repository, Jim's and others' answering of users' questions, Robert's 
packaging for Debian), but I feel overwhelmed by the amount of 
development needed to get 3.2.0 out.

Recently, I looked at the web sites listed in our "users" database, 
and found that about half had switched to google.  There are lots of 
other free search engines out there, and it's not clear to me that 
ht://Dig can (or should) compete.  One of the big problems with free 
software is that we spend all of our time duplicating effort. (If 
developers of OpenOffice.org, Koffice and the Gnome office suite 
combined their resources, they may be able to make a good competitor 
for MS Office, but divided they can't, IMHO.)

I'm perfectly happy to keep working on ht://Dig, but I am by nature a 
follower, and the project needs (among other things) a leader.  If 
someone comes along and brings some enthusiasm, that will be great.
That person need not be a programmer -- if someone wants to document 
all the new features that 3.2.0 has compared with 3.1.6, that would 
also be very inspiring.

$0.02

Lachlan

On Fri, 14 May 2004 07:25 am, Joe R. Jah wrote:
> On Thu, 13 May 2004, Lachlan Andrew wrote:
> >
> > My impression is that the ht://Dig project is basically dead :( 
> > The existing code is of course still functional
>
> You have been a great bulwark for htdig, specially recently.  You
> have brought 3.2 to the verge of release.  How can you make such an
> ominous pronouncement?  It can't be just the indexing speed; is it?

-- 
lha@...
ht://Dig developer DownUnder  (http://www.htdig.org)

On Sun, 16 May 2004, Lachlan Andrew wrote:

> Date: Sun, 16 May 2004 15:43:50 +1000
> From: Lachlan Andrew <lha@...>
> To: Joe R. Jah <jjah@...>
> Cc: htdig-dev@...
> Subject: Pessimism
> 
> Greetings Joe,
> 
> Thanks for your reply, and don't let my mood swings concern you too 
> much :)
> 
> No, it isn't just the indexing speed, although that is part of it.  
> Mainly, there don't seem to be enough people working on the package 
> (coding, documentation).  I'm not meaning to denigrate the great job 
> that other people are doing in providing support (your patch 
> repository, Jim's and others' answering of users' questions, Robert's 
> packaging for Debian), but I feel overwhelmed by the amount of 
> development needed to get 3.2.0 out.
> 
> Recently, I looked at the web sites listed in our "users" database, 
> and found that about half had switched to google.  There are lots of 
> other free search engines out there, and it's not clear to me that 
> ht://Dig can (or should) compete.  One of the big problems with free 
> software is that we spend all of our time duplicating effort. (If 
> developers of OpenOffice.org, Koffice and the Gnome office suite 
> combined their resources, they may be able to make a good competitor 
> for MS Office, but divided they can't, IMHO.)
> 
> I'm perfectly happy to keep working on ht://Dig, but I am by nature a 
> follower, and the project needs (among other things) a leader.  If 
> someone comes along and brings some enthusiasm, that will be great.
> That person need not be a programmer -- if someone wants to document 
> all the new features that 3.2.0 has compared with 3.1.6, that would 
> also be very inspiring.
> 
> $0.02
> 
> Lachlan

I believe we need a meeting of htdig-members.

Regards,

Joe
-- 
     _/   _/_/_/       _/              ____________    __o
     _/   _/   _/      _/         ______________     _-\<,_
 _/  _/   _/_/_/   _/  _/                     ......(_)/ (_)
  _/_/ oe _/   _/.  _/_/ ah        jjah@...

> > On Thu, 13 May 2004, Lachlan Andrew wrote:
> > >
> > > My impression is that the ht://Dig project is basically dead :( 
> > > The existing code is of course still functional
> >
> > You have been a great bulwark for htdig, specially recently.  You
> > have brought 3.2 to the verge of release.  How can you make such an
> > ominous pronouncement?  It can't be just the indexing speed; is it?

Il gio, 2004-05-13 alle 23:25, Joe R. Jah ha scritto:
> You have been a great bulwark for htdig, specially recently.  You have
> brought 3.2 to the verge of release.  How can you make such an ominous
> pronouncement?  It can't be just the indexing speed; is it?

Sorry guys, I had been away in the last couple weeks and I have been
busy at work.

I have been reading with sadness this thread, as I believe ht://Dig 3.2
is a good product and giving up now is like throwing away and wasting
the work of many people (including me).

I believe that 3.2 is worth it.

-Gabriele

Agreed..

And there are those of us that are quiet, but happily using the code in 
production...

peace
chris
On 31/05/2004, at 5:10 PM, Gabriele Bartolini wrote:

> Il gio, 2004-05-13 alle 23:25, Joe R. Jah ha scritto:
>> You have been a great bulwark for htdig, specially recently.  You have
>> brought 3.2 to the verge of release.  How can you make such an ominous
>> pronouncement?  It can't be just the indexing speed; is it?
>
> Sorry guys, I had been away in the last couple weeks and I have been
> busy at work.
>
> I have been reading with sadness this thread, as I believe ht://Dig 3.2
> is a good product and giving up now is like throwing away and wasting
> the work of many people (including me).
>
> I believe that 3.2 is worth it.
>
> -Gabriele
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: Oracle 10g
> Get certified on the hottest thing ever to hit the market... Oracle 
> 10g.
> Take an Oracle 10g class now, and we'll give you the exam FREE.
> http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
> _______________________________________________
> ht://Dig Developer mailing list:
> htdig-dev@...
> List information (subscribe/unsubscribe, etc.)
> https://lists.sourceforge.net/lists/listinfo/htdig-dev

I love htdig, especially the 3.2.0b5 release.

As far as I know, htdig is the only "industrial strength" indexing 
spider that supports a variety of file formats and that is open source, 
and that once it is set up and configured, runs fine for months on end. 
It has support for basic authentication (which is adequate when used in 
conjunction with Integrated Windows authentication). And AFAIK, it is 
also the only option for indexing an intranet...

I am on the verge of releasing a new version of our intranet. I've 
decided to go from 3.1.6 to 3.2 because of phrase searching (and 
because it seems like the first beta that is really ready for it), but 
it seems to me that 3.2.0b5 is actually faster at digging than 3.1.6, 
but it may be that the new version of the intranet simply has fewer 
redundant pages. Regardless, I am confident that 3.2.0b5 is ok for our 
production environment.

I am not a C programmer and thus am not able to contribute all that 
much. I can tell you, however, that there is a rather significant 
learning curve for using htdig (when compared to other information 
systems I've deployed) and this may contribute to any perceived drop in 
"market share".

Sometimes I dream about learning about the Mac OS X packager so that I 
can create an htdig package for it to aid in distribution, setup and 
maintenance, but even that is just beyond me (for now). Even rundig and 
rundig.sh, for as much as they offer, to me, fall short of being a 
complete solution (but rundig.sh sure goes a long way toward helping... 
don't get me wrong!).

Personally, I'm willing to live with occasional segmentation faults if 
I can easily recover from them. I recently had a situation in which a 
troublesome Word file appeared to be corrupting the databases (htdig 
would hang when trying to index the document via catdoc and subsequent 
searches against the databases would fail with segmentation faults). 
Recovering from such a situation required deleting the databases and 
recreating them from scratch: rundig.sh -i -s -c conf/myconf.conf... 
but I had to modify rundig to delete existing databases if -a was not 
used, otherwise, it didn't seem to be creating genuinely new databases 
(please correct me if I'm wrong).

What I'd really, really like to see is a rundig script that handles 
these sorts of situations a little more automatically. rundig.sh has 
been great to learn form (really, really great) but for people who 
don't have time or the inclination, there needs to be a more complete, 
double-clickable solution. But then again, maybe the target audience 
isn't people who aren't adept system administrators... I don't know... 
Just my thoughts (having been an htdig user since about 1998).

BTW, I've been scouring the documentation for the answer and have been 
unable to find it. Can someone tell me what role the different 
databases play and which, if any, are "temporary" (can be deleted after 
a dig, for example)?

Thanks to all the developers, Lachlan, Jim, Gabriele, Joe, Gilles, and 
Goeff and everyone else who has contributed and I look forward to 
testing out the most recent beta on my Mac OS X box.

Ted Stresen-Reuter

Greetings Gabriele,

I apologise for causing you sadness, and others concern.  You are 
right that 3.2 is a good product, and you have done a lot of valuable 
work on it, for which we are all grateful.  It was just that a slow 
point on the mailing lists corresponded to a time I was feeling 
generally pessimistic...

Cheers,
Lachlan

On Mon, 31 May 2004 05:10 pm, Gabriele Bartolini wrote:
>
> I have been reading with sadness this thread, as I believe ht://Dig
> 3.2 is a good product and giving up now is like throwing away and
> wasting the work of many people (including me).

-- 
lha@...
ht://Dig developer DownUnder  (http://www.htdig.org)

Greetings Ted,

On Mon, 31 May 2004 09:47 pm, Ted Stresen-Reuter wrote:
> I am not a C programmer and thus am not able to contribute all that
> much.

You have already been very helpful in testing on OS X.  There is lots 
that can be done that doesn't involve programming, if you have time.  
As well as testing, the documentation could be greatly improved.

Your suggestions on user-friendliness are also very constructive.  If 
you (or anyone else reading this!) would like to help without 
programming, you could design a complete GUI, specifying things like 
what the user does, what defaults to use, how to hide the complexity 
but still allow flexibility, and how it records/updates comments in 
the htdig.conf file.  Perhaps then someone who knows PHP could write 
a script to implement your spec.

Cheers,
Lachlan

-- 
lha@...
ht://Dig developer DownUnder  (http://www.htdig.org)

Greetings Ted,

As I understand it, -i *should* remove all files before it starts 
digging.  In particular, it removes  doc_db, doc_index, doc_excerpt, 
word_db (plus word_db+"_weakcmpr"), md5_db, url_log, doc_list and 
word_dump.  Do you know which files were not being deleted?

Cheers,
Lachlan

On Mon, 31 May 2004 09:47 pm, Ted Stresen-Reuter wrote:
> I had to modify rundig
> to delete existing databases if -a was not used, otherwise, it
> didn't seem to be creating genuinely new databases (please correct
> me if I'm wrong).

-- 
lha@...
ht://Dig developer DownUnder  (http://www.htdig.org)