htdig-dev — Developer Discussion for the ht://Dig project

Re: [htdig-dev] old PR#??? bug numbers (was: patch for argument handling in htsearch)

[Gilles D. <gr...@sc...>]

According to Lachlan Andrew:
> Finally, the "current status" emails refer to problems 
> numbers which don't match the SourceForge problem numbers.  
> Where can I find the original numbering?

Unless we can get the original bug reports from the old database, these
pieced together descriptions from old e-mail references will have to do...

PR#348 was fixed in 3.1.3.  Missing or invalid port number will get set
correctly.  I can't find an old e-mail giving more details, but I think
the bug is fixed in 3.2 as well, unless something similar was reopened
by all the changes to URL.cc.

PR#648 was fixed in 3.1.4.  Its most relevant e-mail is this...
> According to Geoff Hutchison:
> > >At 8:41 AM -0700 9/20/99, Ber...@wd... wrote:
> > >  >I hope it's not a big problem to include the keyword statement into
> > >  >the pagelist
> > >  >variable.
> > >
> > >No, but it's definitely a bug. It won't make 3.1.3, which I'm
> > >releasing tonight. Sorry.
> 
> There are a few other problems with inconsistent handling of input parameters,
> but this seems to be the most troublesome one.  It's an ugly hack, but until
> this problem is fixed, I think you can make sure the keywords parameter
> gets propagated in followups by putting this attribute in your config file:
> 
> allow_in_form: keywords
> 
> It's really meant for allowing input parameters to override config attributes,
> but a nice side effect of it is the listed input parameters get propagated.

I can't find any e-mail that includes the text of PR#650, but the STATUS
file has a synopsis: If exact isn't specified in the search_algorithms,
$(WORDS) is not set correctly.

Subject: Re: Meta Description incorrectly flagged (PR#859)
> At 1:51 PM -0400 6/11/00, rw...@in... wrote:
> >In the db.worddump file, (created with htdig -t) the value in the
> >"flags" column for words that appears in the meta description tags is the same
> >as that for those in title.
> >
> >The flags column is "2". Shouldn't this be "16" as defined in
> >HtWordReference.h?
> 
> Yes, this is correct. I'll have to hunt through and find out where it
> is set incorrectly. HtWordReference.h attempts to resolve the
> confusion in the code between META descriptions and ht://Dig's use of
> "description" to mean the link text.
> 
> -Geoff

Hope this helps.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] patch for argument handling in htsearch

[Geoff H. <ghu...@ws...>]

On Fri, 4 Oct 2002, Gilles Detillieux wrote:

> However, I think in practice a lot of these are actually supposed to
> be integer-only.  I think we'd need to check over how all attributes
> are used and label them consistently.

Good point.

> defaults.cc.  They're implemented in htsearch, but nothing in htdig tags
> words with their corresponding flag values yet.

Once upon a time, I put in the code for capitals--basically when
everything is "normalized" to lowercase, it checks to see if the strings
changed and tags the word accordingly.

(As I've done the mifluz code, I've found places like this where code
seems to be missing--perhaps "migrated" to htword/mifluz by Loic.)

> All this raises the question of whether we should be listing CGI input
> parameters in attrs.html (which is generated from defaults.cc).  To me,

Hmm. I thought one reason for doing this was to allow config-file
overrides for most, if not all CGI input. Yes, we should probably enhance
the hts_form.html file too, if not make it auto-generated.

> These PR# style bug numbers are from our old bug tracking database, prior
> to our move to SourceForge, and I don't think that database is accessible
> anywhere anymore.  At the time of the move, I think Geoff created new
> bug tracking entries for old bug reports that were still opened, so the
> STATUS file should be updated to reflect the new numbers.

I'll try to look those up now. Yes, the SF bug database "got" everything
from the old DB that wasn't closed.

> to avoid opening up big security holes (see myvictim.com URL above).
> It shouldn't be used for any attribute that defines part or all of a
> file name.  The config input parameter is checked for pathname components,
> but none of the other input parameters are.

Yes, I'm not even sure the current documentation for allow_in_form is good
enough for this yet. Perhaps it should give an explicit example of bad
behavior with a note explaining how you can shoot yourself in the foot.

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

Re: [htdig-dev] patch for argument handling in htsearch

[Gilles D. <gr...@sc...>]

According to Lachlan Andrew:
> I've almost finished some patches trying to address the
> "htsearch input parameters issue" (below).
>=20
> I've updated  defaults.cc  to list all variables used by=20
> any of the programs (according to "grep config"), and=20
> described them as best I can.  Where they are different,=20
> the input parameters to  htsearch  are also listed, and=20
> cross-referenced in both directions.  Since no information=20
> is better than mis-information, there are some '??'s and=20
> 'TO BE COMPLETED's.  Some entries which were out of=20
> alphabetical order have also been relocated. This patch is=20
> at
>     http://www.ee.mu.oz.au/staff/lha/pub/patch.defaults

I agree with most of the changes in this patch.  Good job!  To answer
some of your questions, here are a few points to clarify things.

The distinction between "number" and "integer" attribute types is
supposed to be that an attribute labeled "number" can be floating point.
However, I think in practice a lot of these are actually supposed to
be integer-only.  I think we'd need to check over how all attributes
are used and label them consistently.

The Block (Global, Server, URL) field indicates whether an
attribute can be set globally only, or if it can be overridden
with a different value in server blocks or URL blocks.  See
http://www.htdig.org/dev/htdig-3.2/cf_blocks.html

The code support for author_factor, caps_factor, and url_text_factor
is not complete, so I assume this is why the attributes weren't in
defaults.cc.  They're implemented in htsearch, but nothing in htdig tags
words with their corresponding flag values yet.

The remove_default_doc attribute should apply to https:// URLs as well
as http:// ones.  If it doesn't right now, I'd consider that a bug.

The keywords and endday, startday et al. are config attributes that
can be overridden by CGI input parameters, so they're not really
CGI input only.  All except keywords are documented for 3.1.6, in
http://www.htdig.org/attrs.html, if you want more complete descriptions.
(Support for negative numbers hasn't been added to 3.2 yet, but it will
be before 3.2.0b4 goes out.)

The format, matchesperpage, method and page CGI input parameters have
been around from the beginning, I think, but they are CGI input only,
not config attributes.

The config CGI input parameter is most definitely CGI only.  It wouldn't
make sense to specify the config file name in a config file, would it?

All this raises the question of whether we should be listing CGI input
parameters in attrs.html (which is generated from defaults.cc).  To me,
that would tend to blur the distinction between the two.  I know that in
many cases, a CGI input parameter and a config attribute of the same name
exist (and those config attributes should be documented), but I think
it would confuse the issue if we listed CGI-only parameter names here.
CGI input parameters are listed in http://www.htdig.org/hts_form.html

What to other developers think about this?  I'll hold off on committing
this patch until this question is resolved.  (Otherwise, you just know
that some Linux distribution will snatch up that snapshot and we'd be
hounded for a year with questions about why such and such an attribute
doesn't work in the config file. :-P )

> A second patch at
>     http://www.ee.mu.oz.au/staff/lha/pub/patch.inputs
> makes htsearch scan the existing  config  parameters, and=20
> overwrites them if they are given on the command line.  It=20
> also has the #ifdef option of checking that there are no=20
> invalid (hence ignored) command line arguments.  (This=20
> needs  cgi.h  to include "Dictionary.h", and I don't know=20
> how the  make  procedure handles dependencies, so it is=20
> disabled by default.)
>=20
> Before I start testing, could you please confirm that these=20
> are on the right track?

This second patch is a pretty dangerous one!  The whole reason for the
allow_in_form is to let you define, in a controlled manner, which
attributes can be overridden by CGI input parameters (beyond those
which htsearch already does by default).  If I read your patch correctly,
it will allow ANY config attribute to be overridden by a CGI input
parameter.  E.g.:

  http://my.victim.com/cgi-bin/htsearch?nothing_found_file=3D/etc/passwd

> Finally, the "current status" emails refer to problems=20
> numbers which don't match the SourceForge problem numbers. =20
> Where can I find the original numbering?

These PR# style bug numbers are from our old bug tracking database, prior
to our move to SourceForge, and I don't think that database is accessible
anywhere anymore.  At the time of the move, I think Geoff created new
bug tracking entries for old bug reports that were still opened, so the
STATUS file should be updated to reflect the new numbers.

> > * Not all htsearch input parameters are handled properly:=20
> > PR#648. Use a
> > =A0 =A0consistant mapping of input -> config -> template for=20
> > all inputs where
> > =A0 =A0it makes sense to do so (everything but "config" and =20
> > "words"?).
> >=20
> > * Document all of htsearch's mappings of input parameters=20
> > to config attributes
> > =A0 =A0to template variables. (Relates to PR#648.) Also make=20
> > sure these config
> > =A0 =A0attributes are all documented in defaults.cc, even if=20
> > they're only set by
> > =A0 =A0input parameters and never in the config file.

The original PR#648 referred to the keywords input parameter, which
couldn't be set to a default value by a config attribute prior to 3.1.4.
So, the original bug has been fixed (and likely closed), but in the
bug database comments I had suggested systematically going through
all CGI input parameters and making sure htsearch handles them all
consistently wherever appropriate.  I.e. unless there's a reason not to,
a pre-defined CGI input parameter should have a corresponding attribute
that it overrides, and this attribute value should make its way into a
template variable.  Also, any pre-defined CGI input parameter should be
processed by Display::createURL().  Likely the only ones that shouldn't
be done this way are page and config.  I think we're mostly there now, bu=
t
there may be a few stragglers left, both in the code and the documentatio=
n
(definitely in the latter).

Note that I stress "pre-defined" CGI input parameters.  You can't allow
a user to use any old attribute name as an input parameter, and have
that take precedence!  Even allow_in_form must be used very carefully
to avoid opening up big security holes (see myvictim.com URL above).
It shouldn't be used for any attribute that defines part or all of a
file name.  The config input parameter is checked for pathname components=
,
but none of the other input parameters are.

--=20
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] Text within an anchor tag/hyperlink being indexed -- why?

[Brian W. <bw...@st...>]

At 02:03 3/10/2002, Gilles Detillieux wrote:
>According to Jessica Biola:
> > Actually, to append to this problem I'm running into,
> > it doesn't appear to be the text WITHIN the anchor tag
> > -- rather, there is text between the beginning <A> and
> > the ending </A> of the anchor tag that contains the
> > word "large".
>
>Ah!  Well that certainly explains why I was unable to reproduce the
>problem earlier, based on your first message.  Yes, indexing text
>between the <A ...> and </A> tags as link description text for the
>referenced document is normal behaviour for htdig.


This is a common XML/SGML terminology problem - people get "tag" and 
"element" confused

  "<a href='blah'>" is an "A" start *tag*
  "</a>" is an "A" end *tag*

  "<a href='blah'>LARGE</a>" is an "A" *element*, which is made up of a
     start tag, content and end tag.

So: correct, the "A" tag does not contain "LARGE", but the "A" element does.

-------------------------
Brian White
Step Two Designs Pty Ltd
Knowledge Management Consultancy, SGML & XML
Phone: +612-93197901
Web:   http://www.steptwo.com.au/
Email: bw...@st...

Content Management Requirements Toolkit
112 CMS requirements, ready to cut-and-paste

Re: [htdig-dev] Text within an anchor tag/hyperlink being indexed -- why?

[Jessica B. <jes...@ya...>]

Thanks Gilles and Geoff for your responses.  Indeed,
description_factor is the setting that I overlooked
confusing that with the meta_description_factor.

--- Gilles Detillieux <gr...@sc...>
wrote:
> According to Jessica Biola:
> > Actually, to append to this problem I'm running
> into,
> > it doesn't appear to be the text WITHIN the anchor
> tag
> > -- rather, there is text between the beginning <A>
> and
> > the ending </A> of the anchor tag that contains
> the
> > word "large".
> 
> Ah!  Well that certainly explains why I was unable
> to reproduce the
> problem earlier, based on your first message.  Yes,
> indexing text
> between the <A ...> and </A> tags as link
> description text for the
> referenced document is normal behaviour for htdig.
> 
> > The question still is, though, how do I completely
> > wipe out any reference to text between these tags
> as
> > relevant to the linked document?
> 
> As Geoff pointed out, description_factor is the one
> that controls the
> score that will be applied to this text. 
> Unfortunately, the best you
> can do is drop the score of these documents so they
> appear at the end
> of the results instead of at the start.  In the
> future, there will be
> some sort of option for removing matches with a 0 or
> small score.
> 
> > > There's a word, "large" on fruit.html.  "large"
> does
> > > NOT appear anywhere within pineapple.html,
> however,
> > > when I htsearch on the index, both documents
> show as
> > > a
> > > match.  In fact, the base_score is very high for
> the
> > > query "large" on the document pineapple.html.
> > > 
> > > If I index pineapple.html alone, the query
> "large"
> > > yields no results.  So there is definitely some
> type
> > > of relationship between the two documents and
> the
> > > word
> > > "large".  htdump revealed the relationship by
> > > outputting this:
> ...
> > > What configuration attribute must set to zero in
> > > order
> > > for that extra anchor text being indexed and
> > > factored
> > > into pineapple.html's word list?  I've basically
> > > tried
> > > setting all "documented" factors to zero
> (including
> > > backlink).
> > > 
> > > I used the latest htdig-3.2.0b4-092902 as well
> as a
> > > January 2002 release -- both behave the same
> way.
> 
> By the January 2002 release, do you mean the 3.1.6
> release of Jan 31/02,
> or one of the January snapshots of 3.2.0b4?  Either
> way, the behaviour
> of description_factor will be similar, the main
> difference being that
> with 3.1.x, you need to reindex after changing this
> factor, whereas with
> 3.2.x you don't.
> 
> -- 
> Gilles R. Detillieux              E-mail:
> <gr...@sc...>
> Spinal Cord Research Centre       WWW:   
> http://www.scrc.umanitoba.ca/
> Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E
> 3J7  (Canada)
> 
> 
>
-------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> htdig-dev mailing list
> htd...@li...
>
https://lists.sourceforge.net/lists/listinfo/htdig-dev


__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com

Re: [htdig-dev] Text within an anchor tag/hyperlink being indexed -- why?

[Gilles D. <gr...@sc...>]

According to Jessica Biola:
> Actually, to append to this problem I'm running into,
> it doesn't appear to be the text WITHIN the anchor tag
> -- rather, there is text between the beginning <A> and
> the ending </A> of the anchor tag that contains the
> word "large".

Ah!  Well that certainly explains why I was unable to reproduce the
problem earlier, based on your first message.  Yes, indexing text
between the <A ...> and </A> tags as link description text for the
referenced document is normal behaviour for htdig.

> The question still is, though, how do I completely
> wipe out any reference to text between these tags as
> relevant to the linked document?

As Geoff pointed out, description_factor is the one that controls the
score that will be applied to this text.  Unfortunately, the best you
can do is drop the score of these documents so they appear at the end
of the results instead of at the start.  In the future, there will be
some sort of option for removing matches with a 0 or small score.

> > There's a word, "large" on fruit.html.  "large" does
> > NOT appear anywhere within pineapple.html, however,
> > when I htsearch on the index, both documents show as
> > a
> > match.  In fact, the base_score is very high for the
> > query "large" on the document pineapple.html.
> > 
> > If I index pineapple.html alone, the query "large"
> > yields no results.  So there is definitely some type
> > of relationship between the two documents and the
> > word
> > "large".  htdump revealed the relationship by
> > outputting this:
...
> > What configuration attribute must set to zero in
> > order
> > for that extra anchor text being indexed and
> > factored
> > into pineapple.html's word list?  I've basically
> > tried
> > setting all "documented" factors to zero (including
> > backlink).
> > 
> > I used the latest htdig-3.2.0b4-092902 as well as a
> > January 2002 release -- both behave the same way.

By the January 2002 release, do you mean the 3.1.6 release of Jan 31/02,
or one of the January snapshots of 3.2.0b4?  Either way, the behaviour
of description_factor will be similar, the main difference being that
with 3.1.x, you need to reindex after changing this factor, whereas with
3.2.x you don't.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] Text within an anchor tag/hyperlink being indexed -- why?

[Geoff H. <ghu...@ws...>]

> NOT appear anywhere within pineapple.html, however,
> when I htsearch on the index, both documents show as a
> match.  In fact, the base_score is very high for the
> query "large" on the document pineapple.html.
...
> The key part is "d:large".  According to the htdump
> doc page online, the "d" element is defined as:  "The
> text of links pointing to this document. (e.g. <a
> href="docURL">description</a>)"

Right. This is a "description" so you'll want to change the
description_factor attribute. If you're using 3.1.x, you'll need to
reindex as well. I'm guessing from your mention of htdump that you're
instead using 3.2.0b4 snapshots.

<http://www.htdig.org/attrs.html#description_factor>

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

[htdig-dev] punctuation in phrase searches

[Budd, S. <s....@ic...>]

Hello
  Appears to be a problem with htdig-3.2.0b4.0811 (gcc, solaris 5.7)

  a "all"   search for   e.coli    produces   42 hits
  a "all"   search for   ecoli     produces   42 hits as expected   ( . is
punctuation mark )
  a "All" phrase search  "ecoli"   produces   42 hits
  a "All" phrase search  "e.coli"  produces    0 hits    ( has . not been
removed ?)

is the phrase not having its  bad words and punctuation dealt with before
search?

Regards
Sinclair 

Re: [htdig-dev] Text within an anchor tag/hyperlink being indexed -- why?

[Jessica B. <jes...@ya...>]

Actually, to append to this problem I'm running into,
it doesn't appear to be the text WITHIN the anchor tag
-- rather, there is text between the beginning <A> and
the ending </A> of the anchor tag that contains the
word "large".

The question still is, though, how do I completely
wipe out any reference to text between these tags as
relevant to the linked document?

Thanks,
-Jes

--- Jessica Biola <jes...@ya...> wrote:
> I have two files on my test site being indexed.
> 
> fruit.html
> pineapple.html
> 
> There's a word, "large" on fruit.html.  "large" does
> NOT appear anywhere within pineapple.html, however,
> when I htsearch on the index, both documents show as
> a
> match.  In fact, the base_score is very high for the
> query "large" on the document pineapple.html.
> 
> If I index pineapple.html alone, the query "large"
> yields no results.  So there is definitely some type
> of relationship between the two documents and the
> word
> "large".  htdump revealed the relationship by
> outputting this:
> 
> 3  u:http://jtest/pineapple.htm  t:pineapples  a:0 
> m:1033563111  s:5824  H: Pineapple trees  h:
> l:1033563111    L:12    b:5     c:1     g:0     e:  
>  
>  n:      S:      d:large  A:Stump^ATrunk
> 
> The key part is "d:large".  According to the htdump
> doc page online, the "d" element is defined as: 
> "The
> text of links pointing to this document. (e.g. <a
> href="docURL">description</a>)"
> 
> fruit.html, the other HTML file indexed, contains a
> hyperlink to pineapple.html that looks like this:
> 
> <a href="pineapple.html"
>
onMouseOver="MM_showHideLayers('Pine_apple','','show','large','','hide','apple','','hide','tomato','')"><img
> border="0" src="images/pineapple.jpg" width="80"
> height="60"></a>
> 
> What configuration attribute must set to zero in
> order
> for that extra anchor text being indexed and
> factored
> into pineapple.html's word list?  I've basically
> tried
> setting all "documented" factors to zero (including
> backlink).
> 
> I used the latest htdig-3.2.0b4-092902 as well as a
> January 2002 release -- both behave the same way.

__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com

[htdig-dev] Text within an anchor tag/hyperlink being indexed -- why?

[Jessica B. <jes...@ya...>]

I have two files on my test site being indexed.

fruit.html
pineapple.html

There's a word, "large" on fruit.html.  "large" does
NOT appear anywhere within pineapple.html, however,
when I htsearch on the index, both documents show as a
match.  In fact, the base_score is very high for the
query "large" on the document pineapple.html.

If I index pineapple.html alone, the query "large"
yields no results.  So there is definitely some type
of relationship between the two documents and the word
"large".  htdump revealed the relationship by
outputting this:

3  u:http://jtest/pineapple.htm  t:pineapples  a:0 
m:1033563111  s:5824  H: Pineapple trees  h:
l:1033563111    L:12    b:5     c:1     g:0     e:    
 n:      S:      d:large  A:Stump^ATrunk

The key part is "d:large".  According to the htdump
doc page online, the "d" element is defined as:  "The
text of links pointing to this document. (e.g. <a
href="docURL">description</a>)"

fruit.html, the other HTML file indexed, contains a
hyperlink to pineapple.html that looks like this:

<a href="pineapple.html"
onMouseOver="MM_showHideLayers('Pine_apple','','show','large','','hide','apple','','hide','tomato','')"><img
border="0" src="images/pineapple.jpg" width="80"
height="60"></a>

What configuration attribute must set to zero in order
for that extra anchor text being indexed and factored
into pineapple.html's word list?  I've basically tried
setting all "documented" factors to zero (including
backlink).

I used the latest htdig-3.2.0b4-092902 as well as a
January 2002 release -- both behave the same way.

__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com

[htdig-dev] patch for argument handling in htsearch

[Lachlan A. <lh...@ee...>]

Greetings,

I've almost finished some patches trying to address the
"htsearch input parameters issue" (below).

I've updated  defaults.cc  to list all variables used by=20
any of the programs (according to "grep config"), and=20
described them as best I can.  Where they are different,=20
the input parameters to  htsearch  are also listed, and=20
cross-referenced in both directions.  Since no information=20
is better than mis-information, there are some '??'s and=20
'TO BE COMPLETED's.  Some entries which were out of=20
alphabetical order have also been relocated. This patch is=20
at
    http://www.ee.mu.oz.au/staff/lha/pub/patch.defaults

A second patch at
    http://www.ee.mu.oz.au/staff/lha/pub/patch.inputs
makes htsearch scan the existing  config  parameters, and=20
overwrites them if they are given on the command line.  It=20
also has the #ifdef option of checking that there are no=20
invalid (hence ignored) command line arguments.  (This=20
needs  cgi.h  to include "Dictionary.h", and I don't know=20
how the  make  procedure handles dependencies, so it is=20
disabled by default.)

Before I start testing, could you please confirm that these=20
are on the right track?

Finally, the "current status" emails refer to problems=20
numbers which don't match the SourceForge problem numbers. =20
Where can I find the original numbering?

Thanks :)
Lachlan


> * Not all htsearch input parameters are handled properly:=20
> PR#648. Use a
> =A0 =A0consistant mapping of input -> config -> template for=20
> all inputs where
> =A0 =A0it makes sense to do so (everything but "config" and =20
> "words"?).
>=20
> * Document all of htsearch's mappings of input parameters=20
> to config attributes
> =A0 =A0to template variables. (Relates to PR#648.) Also make=20
> sure these config
> =A0 =A0attributes are all documented in defaults.cc, even if=20
> they're only set by
> =A0 =A0input parameters and never in the config file.

--=20
Lachlan Andrew  Phone: +613 8344-3816 Fax: +613 8344-6678
Dept of Electrical and Electronic Engg		CRICOS Provider Code
University of Melbourne, Victoria, 3010  AUSTRALIA	00116K

[htdig-dev] Re: [ htdig-Bugs-614270 ] Scoring

[Gilles D. <gr...@sc...>]

A better place to post patches would be the htdig-general or htdig-dev
mailing list.  They'll have a wider audience that way, and are less
likely to get mangled by SourceForge's bug tracker.  Also, for purposes of
discussing changes to the code, htdig-dev is the best forum.  (That's why
I'm moving this discussion there.)

The reason no one has taken up the task yet is that it's a bit more
complicated than what your patch does.  First of all, the preferred way
to drop search results from the results list would be like how htsearch
handles matches outside of the date range.  I.e. you'd do something like:

	if (score <= 0.0)
	{
	    delete thisRef;
	    continue;
	}

Secondly, there's the issue of where in the code is the best place to
do this.  You placed the test after the final score has been calculated,
but at that point it's in logarithmic form.  It might be better to do it
before that.  It might even be better still to do it before date_factor
and backlink_factor are factored in, and before the other adjustments
are applied, so that if a document has a base score of 0 it's rejected
regardless of how these other factors might boost the score.  That would
mean the decision to select/reject is based solely on the word score
and not other scoring tweaks.

Finally, you could probably raise good arguments for doing it one way
or the other, or for having setable rejection thresholds as has been
suggested previously, or even for not rejecting them at all, so it may be
that this feature should be controlled by a new config attribute or two.

According to no...@so...:
> Bugs item #614270, was opened at 2002-09-25 01:58
> You can respond by visiting: 
> https://sourceforge.net/tracker/?func=detail&atid=104593&aid=614270&group_id=4593
> 
> Category: htsearch
> Group: feature-requests
> Status: Open
> Resolution: Later
> Priority: 5
> Submitted By: Nobody/Anonymous (nobody)
> Assigned to: Nobody/Anonymous (nobody)
> Summary: Scoring
> 
> Initial Comment:
> Is there a reason why an item appears in the result
> list that has a score or '0.0' ?  Is this intented or
> is it a bug ?
> 
> htdig-3.2.0b4-20020922
> 
> dr...@el...
> 
> ----------------------------------------------------------------------
> 
> Comment By: Nobody/Anonymous (nobody)
> Date: 2002-10-01 10:11
> 
> Message:
> Logged In: NO 
> 
> Please find below a patch for htdig-3.2.0b4-20020922 to not
> add '0.0' scored results to the results list ...
> 
> diff -uPr htdig-3.2.0b4-20020922.orig/htsearch/Display.cc
> htdig-3.2.0b4-20020922/htsearch/Display.cc
> --- htdig-3.2.0b4-20020922.orig/htsearch/Display.cc     Sat
> Jul 27 04:48:19 2002+++
> htdig-3.2.0b4-20020922/htsearch/Display.cc  Tue Oct  1
> 17:23:43 2002
> @@ -1459,7 +1459,7 @@
>         //
>         // Append this match to our list of matches.
>         //
> -       matches.Add(thisMatch, thisRef->DocURL());
> +       if (score > 0.0) matches.Add(thisMatch,
> thisRef->DocURL());
>  
>         // Get rid of it to free the memory!
>         delete thisRef;
> @@ -1470,11 +1470,11 @@
>           cerr << "   score " << score << "(" <<
> thisMatch->getScore() << "), maxScore " << maxScore <<",
> minScore " << minScore << endl;
>          }
>  
> -       if (maxScore < score)
> +       if (score > 0.0 && maxScore < score)
>           {if(debug) cerr << "Set maxScore = score" <<endl;
>             maxScore = score;
>            }
> -       if (minScore > score)
> +       if (score > 0.0 && minScore > score)
>           {if(debug) cerr << "Set minScore = score" <<endl;
>             minScore = score;
>            }
> 
> Hope this helps (and is in the right place) ...
> 
> dr...@el...
> 
> ----------------------------------------------------------------------
> 
> Comment By: Gilles Detillieux (grdetil)
> Date: 2002-09-25 06:57
> 
> Message:
> Logged In: YES 
> user_id=149687
> 
> It's not a bug, it's a lack of a feature. Doing just
> what you propose has been suggested before,
> and it will eventually find its way into the code,
> but no one has yet taken up the task.
> 
> ----------------------------------------------------------------------
> 
> You can respond by visiting: 
> https://sourceforge.net/tracker/?func=detail&atid=104593&aid=614270&group_id=4593
> 


-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] weakcmpr file

[Greg F. <gre...@ya...>]

--- Gilles Detillieux <gr...@sc...> wrote:
> 
> A simple enhancement to all the ht://Dig programs that would simplify
> scripts like this a bit would be to add a -D option, or something
> like
> it, to set config attributes from the command line.

Gilles,

Your insight is muchly appreciated.

Looking at where I am right now, I realize I was building a script with
two competing goals and I had boxed myself into a corner (but was still
swinging!)

I was originally trying to enhance rundig.3.2.sh so that it could
update a database in-place (which it currently does) as well as update
a database in a separate build directory.  So I was juggling the
"*.work" files and having databases in separate directories (thus the
htfuzzy problems).

I agree that adding a "-D" flag (or whatever) will be very useful.  For
now, my script will depend on two different config files (a build
config and a search config).

Thank you,
greg_fenton.

=====
Greg Fenton
gre...@ya...

__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com

[htdig-dev] Re: ExternalTransport patch, and URL format

[Lachlan A. <lh...@ee...>]

Greetings again Gilles,

Thanks for your feedback.  I've fixed up some of the 
sloppyness in the patch (patch.kurl, relative to 
htdig-3.2.0b4-20020922) and done some fairly thorough 
testing on it.  It now only removes leading slashes if 
there are at least the prescribed number.  It also stores 
the slash count as '0'+slashcount rather than 
'\1'+slashcount.  That retains the efficiency of using a 
single-character, but means that naive reading of the 
string gives the correct result (provided slashcount < 10).

I still haven't found out the "right" way to call test/url, 
but I have called it by hand, and the results match those 
of the original, with one exception.  It now normalises the 
host name (e.g. resolves aliases) for *all* protocols with 
an IP host, rather than just  http://.  It still only 
removes "index.html" from "http://".

I have also expanded the tests in  test/url.cc  to test the 
new functionality, and to test alias resolution.  The patch 
is in "patch.test".  It would be good to have a file of the 
"correct" output of this file so that changes to URL.cc 
could be tested simply with  diff,  but I didn't know how 
to put that in the patch.

In the process of testing, I noticed that double slashes 
('//') are normalised out *after* resolving '../'.  This 
means that '/foo//../' resolves to '/foo/', when UNIX would 
resolve it to '/'.  Is the current behaviour deliberate, or 
an oversight?

Finally, what is the schedule/criterion for the release of 
3.2.0b4?  The KDE team won't base their code on snapshots, 
and so I'd like to expedite this...  Which of the items in 
Geoff's regular "current status" posts have to be addressed 
before the release?

Thanks!
Lachlan

On Wed, 18 Sep 2002 08:58, Gilles Detillieux wrote:
> According to Lachlan Andrew:
> > Here is a patch (url-format.patch) to allow the format
> > of external_protocol URLs be <protocol>:<path>, rather
> > than <protcol>://<host>/<path>. It seems to work in the
> > cases I've tested, but I'm not sure how to try it on
> > the test suite, so I hope I haven't broken anything
> > else...
>
> For the 1st one, I don't have time to test it myself, but
> I'd like to wait for comments/testing from other
> developers if any is forthcoming.
>
> To answer some of the questions you ask in the code of
> your patch, the .get() after a sub seemed to be needed
> with some compilers, to avoid warnings.  For whatever
> reason, we couldn't just assign the result of a .sub() to
> another String, even though .sub() is supposed to return
> a String.  The .get() gives us a (char *) from that, and
> everything seems to work well that way.
>
> On line 324 of URL.cc, you say "(should also check the
> slashes are actually there...)".  I agree, the code
> should do this.
>
> The way the slash count is encoded in the Dictionary
> entries is kludgy, but I think there are similar kludges
> elsewhere in the code.  It's also self-contained in one
> method of the URL class, so I don't have a problem with
> it.

-- 
Lachlan Andrew  Phone: +613 8344-3816 Fax: +613 8344-6678
Dept of Electrical and Electronic Engg		CRICOS Provider Code
University of Melbourne, Victoria, 3010  AUSTRALIA	00116K

[htdig-dev] Creating locked files ( part of Adjustable Logging )

[Brian W. <bw...@st...>]

At the moment I am ( in spare moments ) putting together my
Adjustable Logging patch. At the moment I am putting
together some file locking, and I thought might give  a
progress report on a couple of aspects and get some feedback.


Ok. I have changed the way I want to do file locking. Given
there are two completely separate implementation underneath,
I thought I should define an interface class and a factory
method. The interface class is

   class LockedFile
   {
   private:
      //hide the delete operator
      void operator delete( void*) {}

   public:

      //Destructor
      virtual int Close() = 0;

      // Will return true if the object
      // is currently locked

      virtual int IsLocked() = 0;

      // Will attempt to lock the file
      // Will return true if the file is already
      // locked by this object or if it was able
      // to successfully lock it.
      //
      // Will use the wait mode specified in the
      // constructor

      virtual int Lock() = 0;

      // Will attempt to unlock the file
      // Will return true if the file is already
      // unlocked by this object or if it was able
      // to successfully unlock it.

      virtual int Unlock() = 0;

      // Will return the FILE* for use in other
      // application

      virtual FILE* GetFP() = 0;

      // Will detach the FILE* and then destroy
      // the rest of the object

      virtual FILE* DetachFPAndRemoveLock() = 0;
   };

And the factory method for creating one of these
beasts is:

   // Factory method for Creating a Locked File
   // If it can succeed it will return a pointer to
   // a locked file object. File is closed
   // by deleting the pointer (??)

   LockedFile*
   OpenLockedFile( const char* path, const char* mode, int waitForLock );

   LockedFile*
   OpenLockedFile( const char* path, const char* mode );


A typical usage would be

    LockedFile* lkdf = OpenLockedFile( "/path/to/file", "w" );

    if ( lkdf ) // The factory function will return 0 if it can't
                // create the locked file
    {
       // Use the GetFP method to access the FILE* for use in things
       // like fprintf
       fprintf( lkdf->GetFP(), "Hello World\n" );

       // This will remove the lock, close the file and destroy the object
       lkdf->Close();
    }


The aim is to provide a general lockfile class. If anyone can knows a
way to autocast a LockFile* to a FILE* it would be good. I *could* add
an operator like this:

    public:
          operator FILE*(){ return GetFP(); }

which would allow this to work:

    fprintf( *lkdf, "Hello World\n" );

but I thought that resolving a pointer to create a pointer is a trifle
counter-intuitive and could be a cause of major confusion.

The other thing is that I have been working my through and attmepting
to debug things very carefully. Part of doing this was that I defined
myself some TRACE macros:

   #ifndef USE_TRACE_MESSAGES

   #define TRACE(S)
   #define TRACEVAR(X)
   #define TRACEPVAR(P,X)

   #else

   #define TRACE(S) printf("%s:%d:%s\n", __FILE__, __LINE__, S );
   #define TRACEVAR(X) printf("%s:%d:%s=%s\n", __FILE__, __LINE__, \
                              #X, TraceVarStr(X) );
   #define TRACEPVAR(P,X) printf("%s:%d:%s=%s\n", __FILE__, __LINE__, \
                              #P "->" #X, \
                             ( P ? TraceVarStr(P->X) : "[" #P " is NULL]" ));

   ( version of TraceVarStr for every type you wish to trace )

   #endif


These can be left in the code, and will be preprocessed out of existence
for normal use.

My question here is : is there any existing feature or philospohy used
in the code that does this kind of thing? ( regex.cc and the db code seem
to have some kind of debugging code ) Is is worth having available as a
generally usable thing?

Regs

Brian White










-------------------------
Brian White
Step Two Designs Pty Ltd
Knowledge Management Consultancy, SGML & XML
Phone: +612-93197901
Web:   http://www.steptwo.com.au/
Email: bw...@st...

Content Management Requirements Toolkit
112 CMS requirements, ready to cut-and-paste

[htdig-dev] Current Status as of snapshot 3.2.0b4-20020929

[Geoff H. <ghu...@us...>]

STATUS of ht://Dig branch 3-2-x

RELEASES:
   3.2.0b4: In progress 
	(mifluz merge essentially finished, contact Geoff for patch to test)
   3.2.0b3: Released:  22 Feb 2001.
   3.2.0b2: Released:  11 Apr 2000.
   3.2.0b1: Released:   4 Feb 2000.

SHOWSTOPPERS:

KNOWN BUGS:
* Odd behavior with $(MODIFIED) and scores not working with
   wordlist_compress set but work fine without wordlist_compress.
   (the date is definitely stored correctly, even with compression on
    so this must be some sort of weird htsearch bug)
* Not all htsearch input parameters are handled properly: PR#648. Use a
   consistant mapping of input -> config -> template for all inputs where
   it makes sense to do so (everything but "config" and "words"?).
* If exact isn't specified in the search_algorithms, $(WORDS) is not set 
   correctly: PR#650. (The documentation for 3.2.0b1 is updated, but can
   we fix this?)
* META descriptions are somehow added to the database as FLAG_TITLE,
   not FLAG_DESCRIPTION. (PR#859)

PENDING PATCHES (available but need work):
* Additional support for Win32.
* Memory improvements to htmerge. (Backed out b/c htword API changed.)

NEEDED FEATURES:
* Field-restricted searching.
* Return all URLs.
* Handle noindex_start & noindex_end as string lists.

TESTING:
* httools programs: 
  (htload a test file, check a few characteristics, htdump and compare)
* Turn on URL parser test as part of test suite.
* htsearch phrase support tests
* Tests for new config file parser
* Duplicate document detection while indexing
* Major revisions to ExternalParser.cc, including fork/exec instead of popen,
  argument handling for parser/converter, allowing binary output from an
  external converter.
* ExternalTransport needs testing of changes similar to ExternalParser.

DOCUMENTATION:
* List of supported platforms/compilers is ancient.
* Add thorough documentation on htsearch restrict/exclude behavior
   (including '|' and regex).
* Document all of htsearch's mappings of input parameters to config attributes
   to template variables. (Relates to PR#648.) Also make sure these config
   attributes are all documented in defaults.cc, even if they're only set by
   input parameters and never in the config file.
* Split attrs.html into categories for faster loading.
* require.html is not updated to list new features and disk space
   requirements of 3.2.x (e.g. phrase searching, regex matching,
   external parsers and transport methods, database compression.)
* TODO.html has not been updated for current TODO list and completions.

OTHER ISSUES:
* Can htsearch actually search while an index is being created?
   (Does Loic's new database code make this work?)
* The code needs a security audit, esp. htsearch
* URL.cc tries to parse malformed URLs (which causes further problems)
   (It should probably just set everything to empty) This relates to 
   PR#348.

Re: [htdig-dev] weakcmpr file

[Gilles D. <gr...@sc...>]

According to Greg Fenton:
> --- Gilles Detillieux <gr...@sc...> wrote:
> > > It sure would help make some of the support scripts (rundig, etc.)
> > > easier to write if that file ended ".work".
> > 
> > True, although one extra move or copy in the script isn't that hard.
> 
> Actually, it is kind'a hard.
> 
> The problem (for my scenario) is that I need to update the DB without
> interfering with the "live" db.  To accomplish this, I run htfuzzy on
> the .work database(s).  But, htfuzzy doesn't support the "-a" flag, so
> I have to (somehow) move the words and words.db_weakcmpr to a
> non-".work" location, run htfuzzy on those, and move them back to
> ".work" so that the rest of my script can move the final results into
> place (thus minimizing downtime of the live search).
> 
> It's kind'a ugly, but I have it working by using two different config
> files, separate directories for build and search, etc.  It would just
> be "nicer" (and easier for people to follow) if htfuzzy supported "-a".

I know you said to disregard this message, but I did think of a simpler
workaround.  As long as you're dealing with separate directories and
config files, you can take it a step further and do away with the -a
option entirely.  Just do all the index updating (htdig, htpurge, htfuzzy)
with one config file and database_dir, and use another one for searching
the "live" database.  Then, when the update is complete, just move the
directories around so the updated one becomes the live one.  That would
keep your database downtime to a small fraction of a second between to
"mv" commands, which is even quicker that moving each database file into
place (and much quicker than copying).  Your script would be something
like this:

DIGD=/var/lib/htdig/digd
LIVE=/var/lib/htdig/live
mkdir -p $DIGD
cp -p $LIVE/* $DIGD/	# only if you're doing an update dig
htdig ...
htpurge ...
htfuzzy ...
htnotify ...
mv $LIVE $LIVE.old && mv $DIGD $LIVE	# quick swap of directories
rm -rf $LIVE.old			# slower removal of old DBs

There may be a bit of an inefficiency in having to copy ALL of the live
database files to the dig directory, but as the code stands right now in
3.2.0b4, it seems that all programs need all the database files.  Even if
you could move db.docs.index from $LIVE to $DIGD, instead of copying, it
wouldn't save you much as this is about the smallest file of the lot.

A simple enhancement to all the ht://Dig programs that would simplify
scripts like this a bit would be to add a -D option, or something like
it, to set config attributes from the command line.  This would let you
do away with extra config files when all you need to do is override one
or a few config attributes.  E.g.:

  htdig -c /etc/htdig/htdig.conf -Ddatabase_dir=$DIGD

I've often thought about adding something like this, but never got around
to it.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] weakcmpr file

[Gilles D. <gr...@sc...>]

According to Greg Fenton:
> Please ignore this latest long-winded email...it's been a long day and
> I had forgotten that I had already raised the "-a" issue in a previous
> post.

I was wondering at first why you seemed to be crossing two different
threads, but I see how they tie together, and what you're trying to
overcome.

> However, my offer to help still stands.  I'm eager to help out and a
> capable C/C++ programmer (though I've been away from it for a couple of
> years ... that whole Java thing...)

Well, looking at the -a support in htdig/htdig.cc and adding something like
it to htfuzzy/htfuzzy.cc might be a good warm-up exercise to get back into
C++.  :-)  Geoff or I can commit any patches you provide, if you don't have
CVS write access yet.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] weakcmpr file

[Greg F. <gre...@ya...>]

--- Gilles Detillieux <gr...@sc...> wrote:
>
> > It sure would help make some of the support scripts (rundig, etc.)
> > easier to write if that file ended ".work".
> 
> True, although one extra move or copy in the script isn't that hard.
> 

Actually, it is kind'a hard.

The problem (for my scenario) is that I need to update the DB without
interfering with the "live" db.  To accomplish this, I run htfuzzy on
the .work database(s).  But, htfuzzy doesn't support the "-a" flag, so
I have to (somehow) move the words and words.db_weakcmpr to a
non-".work" location, run htfuzzy on those, and move them back to
".work" so that the rest of my script can move the final results into
place (thus minimizing downtime of the live search).

It's kind'a ugly, but I have it working by using two different config
files, separate directories for build and search, etc.  It would just
be "nicer" (and easier for people to follow) if htfuzzy supported "-a".

This much work?  I'd be willing to take on the task myself if I can be
given some guidance on where to start in the code...but I don't want to
go chasing after something that is more work than its worth or
something that noone else cares about.

greg_fenton.

=====
Greg Fenton
gre...@ya...

__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com

Re: [htdig-dev] htfuzzy -a

[Gilles D. <gr...@sc...>]

According to Greg Fenton:
> htfuzzy does not have a "-a" flag.  This means that one cannot build
> fuzzy databases until after the words databases have been "made live".
> 
> So, for example, if I am trying to rebuild the databases offline (not
> usable by htsearch), then I have to do some scripting tricks to rebuild
> the fuzzy dbs or I have to build the dbs in one directory and move to
> another for searching (thus eliminating the benefit of "-a" on other
> programs).
> 
> Wouldn't it make sense to add a "-a" flag to htfuzzy, or am I just
> missing something?

It makes sense, for the so-called "word algorithms" that make use of
word_db (i.e. accents, soundex and metaphone).  However, even these
algorithms do all their work in memory, and only write out their databases
at the end, so doing this over a "live" database collection wouldn't put
these fuzzy algorithms out of commission for long.  These aren't needed
for endings and soundex, which use static dictionaries, nor the other
fuzzy algorithms which don't use databases.  It would complicate your
indexing scripts somewhat too, as you'd have up to 3 more .work files
to move into place.

Still, I think adding -a handling to htfuzzy is a good idea.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] weakcmpr file

[Gilles D. <gr...@sc...>]

According to Greg Fenton:
> Is there a reason that the .work file for the compressed words database
> is  "db.words.db.work_weakcmpr" and not "db.words.db_weakcmpr.work" ?

Yes, this is because the _weakcmpr handling is buried rather deeply in
the database code, and not integrated with the higher levels of the
ht://Dig code.  So, it just uses the existing word database name and
appends the _weakcmpr rather than using a different config attribute
name for this other file.

> It sure would help make some of the support scripts (rundig, etc.)
> easier to write if that file ended ".work".

True, although one extra move or copy in the script isn't that hard.

> Also, in my brief query of the code in CVS, would it not make sense to
> have WordListMulti.cc use the #define of DB_CMPR_SUFFIX in db/mp.h
> instead of hard-coded strings "_weakcmpr"?

Yes, it probably should be done that way.  However, the plan is for all
this _weakcmpr stuff to disappear when the upgrade to the latest mifluz
code is complete.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

[htdig-dev] htfuzzy -a

[Greg F. <gre...@ya...>]

htfuzzy does not have a "-a" flag.  This means that one cannot build
fuzzy databases until after the words databases have been "made live".

So, for example, if I am trying to rebuild the databases offline (not
usable by htsearch), then I have to do some scripting tricks to rebuild
the fuzzy dbs or I have to build the dbs in one directory and move to
another for searching (thus eliminating the benefit of "-a" on other
programs).

Wouldn't it make sense to add a "-a" flag to htfuzzy, or am I just
missing something?

greg_fenton.

=====
Greg Fenton
gre...@ya...

__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com

[htdig-dev] weakcmpr file

[Greg F. <gre...@ya...>]

Is there a reason that the .work file for the compressed words database
is  "db.words.db.work_weakcmpr" and not "db.words.db_weakcmpr.work" ?

It sure would help make some of the support scripts (rundig, etc.)
easier to write if that file ended ".work".

Also, in my brief query of the code in CVS, would it not make sense to
have WordListMulti.cc use the #define of DB_CMPR_SUFFIX in db/mp.h
instead of hard-coded strings "_weakcmpr"?

greg_fenton.

=====
Greg Fenton
gre...@ya...

__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com

Re: [htdig-dev] rundig.3.2.sh and htmerge

[Geoff H. <ghu...@ws...>]

On Thu, 26 Sep 2002, Greg Fenton wrote:

> However "rundig.3.2.sh" does search for "htmerge: " output as it
> summarizes the logged information from htdig, htpurge, htnotify and
> htstat.
> 
> Do one of these tools call htmerge?
> Or, is this fgrep just an oversite (left over from previous
> incarnations)?

Let's call it a "thinko." Someone asked for a 3.2 version of my rundig.sh
script and I just whipped it up--that's clearly a mistake.

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

[htdig-dev] rundig.3.2.sh and htmerge

[Greg F. <gre...@ya...>]

Reading the new "overview" document
(http://www.htdig.org/dev/htdig-3.2/all.html), it appears that
htmerge's role has changed somewhat drastically.  It is not called from
"rundig", nor from "rundig.3.2.sh".

However "rundig.3.2.sh" does search for "htmerge: " output as it
summarizes the logged information from htdig, htpurge, htnotify and
htstat.

Do one of these tools call htmerge?
Or, is this fgrep just an oversite (left over from previous
incarnations)?

Thanks in advance,
greg_fenton.

=====
Greg Fenton
gre...@ya...

__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com