[htdig-dev] Several questions...

[Lachlan A. <lh...@us...>]

Greetings all,

I'm checking through phrase searching, and have found several possible=20
bugs.  First, some questions...

1. Why do the documentation for  external_parser  and the comments=20
before  Retriever::got_word  both say that the word location must be=20
in the range 0-1000?   The HTML parser doesn't stick to that.  If=20
locations are just scaled down (rather than reduced modulo 1001),=20
that will break the phrase searches.  Is there a maximum in practice?

2. Every "meta" data entry (<title>, <meta ...> etc.) gets added as if=20
it starts at location 0.  This gives *heaps* of false-positives,=20
because the second word of *any* entry is deemed adjacent to the=20
first word of any *other* entry.  Could we add "meta" information at=20
successive locations starting from, say, location 10,000?

3. With phrase searching, do we still need  valid_punctuation?  For=20
example, "post-doctoral" currently gets entered as three words at the=20
*same* location:  "post", "doctoral" and "postdoctoral".  Would it be=20
better to convert queries for  post-doctoral  into the phrase "post=20
doctoral" in queries, and simply the words  post  and  doctoral  at=20
successive locations in the database?  As it stands, a search for=20
"the non-smoker" will match "the smoker", since all the words are=20
given the same position in the database, but a search for "the non=20
smoker" won't match "the non-smoker".  This also reduces the size of=20
the database (marginally in most cases, but significantly for=20
pathological documents).  Now that there is phrase searching, is=20
there any benefit of the current approach?

4. Does anybody know what the existing external parsers do about words=20
less than the minimum length?  Because they are passed the=20
configuration file, they *could* omit them.  Currently the HTML=20
parser omits them, but that introduces false-positives into phrase=20
queries, and I want to fix that.

Thanks!
Lachlan