htdig-dev — Developer Discussion for the ht://Dig project

Re: [htdig-dev] http_proxy_authorization

[Gabriele B. <g.b...@co...>]

Il mer, 2002-08-07 alle 17:35, Gilles Detillieux ha scritto:
> Your wording is just fine by me.  However, you need to change
> the "3.0" string in the 6th field of your defaults.cc entry for
> http_proxy_authorization to "3.2.0b4", because obviously this
> attribute hasn't been around since 3.0.

Ooooops. Sorry about that, and thanks Gilles. That's the cut and paste
inconvenient. I'll fix it immediately.

-Gabriele
--=20
Gabriele Bartolini - Web Programmer
Comune di Prato - Prato - Tuscany - Italy
g.b...@co... | http://www.comune.prato.it
> find bin/laden -name osama -exec rm {} ;

Re: [htdig-dev] status of synonyms?

[Gilles D. <gr...@sc...>]

According to Budd, Sinclair:
>   using htdig3.2.0b4-20020728  there are strange results for synonyms. 
>   (The capitalization below is for clarity, actual searches used lower
> case.)
>   In the synonym  file "abcense" is set as synonym with "absence".
> 
>  with htdig.conf set with 
>  search_algorithm:	exact:1  endings:0.1  
> 
>  searching for   abCense   produces  0  results
>                  abSence   produces  11 results
> 
> 
> 
>  with htdig.conf set with 
>  search_algorithm:	exact:1  endings:0.1   synonyms:0.9
> 
>  searching for   abCense   produces  783 results ( and these results contain
> "abSence" words ok)
>                  abSence   produces  11 results  ( same results as for no
> synonyms )
> 
> curious no?

Very.  I'm at a bit of a loss to explain this one.  If you do an htdump,
how many entries for "absence" and "abcense" do you find in the resulting
db.worddump file?

Does the word "abcense" appear anywhere else in your synonyms file, or
just on the one line with "absence"?

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] Searching within search results

[Gilles D. <gr...@sc...>]

According to Krause, Shannon:
> Does ht://Dig have the capability of searching ONLY within the results of a
> previous search? If it does, how would we configure it to do so.

Not exactly, but with the boolean search method you can construct
some pretty elaborate searches that would refine results from an
earlier query.  In the simplest case, using the "All" method (which
implies "and" between each word), you simply add more words to the
query to narrow down the results.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] http_proxy_authorization

[Gilles D. <gr...@sc...>]

According to Gabriele Bartolini:
>    I put the code for the Proxy authorization (Basic scheme) in the
> Document class. I've also added the configuration attribute into the
> defaults.cc file.
> 
>    Of course, as usual, it needs your fix, Geoff and Gilles. Indeed, I
> can't get rid of my 'italianish' language. But I like it, at least until
> girls like my *spaghetti* accent ... :-P   <just kidding>

Your wording is just fine by me.  However, you need to change
the "3.0" string in the 6th field of your defaults.cc entry for
http_proxy_authorization to "3.2.0b4", because obviously this
attribute hasn't been around since 3.0.

Ciao!

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] vi settings (was: htnet changes)

[Gilles D. <gr...@sc...>]

According to Gabriele Bartolini:
> P.S.: Do you have any suggestion for configuring vi in a good way, as
> far as indenting code is concerned? I realize that my code is sometimes
> messy.

The main thing is to set tabstop=8 because that's how tab characters are
viewed on most UNIX-like systems by default.  Beyond that is more a matter
of taste.  I tend to do my indenting manually, tabbing as many times as
needed and then adding 4 spaces for a half-indent when that's the level
of indentation called for.  Setting shiftwidth=4 might help if you use the
indent and unindent commands ">>" and "<<".  Also, if you like autoindent,
then you can set that to preserve indentation from line to line.

The thing to avoid is just holding down the space bar until the indentation
"feels about right", as that's not going to give you indentation that's
consistent with the style of the current code.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

[htdig-dev] Searching within search results

[Krause, S. <sk...@as...>]

Does ht://Dig have the capability of searching ONLY within the results of a
previous search? If it does, how would we configure it to do so.

Shannon Krause
Web Site Administrator
American Society for Quality
sk...@as...

RE: [htdig-dev] Cookie related...

[Gabriele B. <g.b...@co...>]

Ciao David,

   thanks for your contribution. I think your code gives more
flexibility to the HTTP library, therefore it's perfect.

   I just committed the changes, to both ht://Check and ht://Dig.

Ciao and thanks
-Gabriele

Il mar, 2002-08-06 alle 17:43, Reed, David ha scritto:
> Gabriele,
>=20
> Relative to a comment you made long ago (and I can't find anymore) about
> cookie support... I made a few changes to the cookie code.  I agree with
> using the RFC2109 spec.  However, I think we should only use that level o=
f
> the cookie spec if the cookie is received under that spec.  (Most
> Application Servers don't support that level of the spec and send netscap=
e
> style cookies.)
>=20
> Here are some changes that look to see what version of the cookie we
> received (defaulting to 0/netscape) and sending it back following the sam=
e
> protocol.  (Also is a change to enable cookies with SSL.)
>=20
> Let me know what you think.=20
>=20
> Thanks,
> -david

--=20
Gabriele Bartolini - Web Programmer
Comune di Prato - Prato - Tuscany - Italy
g.b...@co... | http://www.comune.prato.it
> find bin/laden -name osama -exec rm {} ;

[htdig-dev] http_proxy_authorization

[Gabriele B. <g.b...@co...>]

Ciao guys,

   I put the code for the Proxy authorization (Basic scheme) in the
Document class. I've also added the configuration attribute into the
defaults.cc file.

   Of course, as usual, it needs your fix, Geoff and Gilles. Indeed, I
can't get rid of my 'italianish' language. But I like it, at least until
girls like my *spaghetti* accent ... :-P   <just kidding>

Ciao and thanks
-Gabriele
--=20
Gabriele Bartolini - Web Programmer
Comune di Prato - Prato - Tuscany - Italy
g.b...@co... | http://www.comune.prato.it
> find bin/laden -name osama -exec rm {} ;

[htdig-dev] htnet changes

[Gabriele B. <g.b...@co...>]

Ciao guys,

   I just made a few changes on the htnet library. I made them a while
ago on ht://Check but wanted to wait some time and some tests on them
before moving to ht://Dig too.

   They regard essentially 2 features:
- IP address retrieval (I put it into the Connection class, which now
stores it as a string - it could be changed to a number if we think
reasonable);
- Proxy authorization feature (I was asked to enable it on ht://Check,
it is very similar to the credentials and Authentication header we
already had).

   Sorry if I put the first feature, which is unnecessary for now to
ht://Dig, but it is very useful for ht://Check; and as long as having
the code synchronised helps a lot, I just merged the lines in ht://Dig.

   The next step is to enable the 'http_proxy_authorization'
configuration attribute (it works already with ht://Check). I'll do it
in the near future.

Ciao and let me know if there's something wrong (for sure there will
be).

-Gabriele

P.S.: Do you have any suggestion for configuring vi in a good way, as
far as indenting code is concerned? I realize that my code is sometimes
messy.
--=20
Gabriele Bartolini - Web Programmer
Comune di Prato - Prato - Tuscany - Italy
g.b...@co... | http://www.comune.prato.it
> find bin/laden -name osama -exec rm {} ;

Re: [htdig-dev] Re: mifluz merge snapshot 2002-08-03

[Geoff H. <ghu...@ws...>]

> I did "make install"; didn't you expect it to install?

No, I'm trying to figure out whether there are problems in the build 
environment--I've been running from the build directories. With libtool, 
it's not quite the same as when you do a full install. It *should* be, 
but...

In any case, I'm working on your segfaults as well as some I seem to 
get. As I said, I'll be updating this as I can and I'll try to get a 
fairly stable version out in a day or two. Anyone who wants to help with 
the coding/debugging process should let me know.

-Geoff

Re: [htdig-dev] Re: mifluz merge snapshot 2002-08-03

[Joe R. J. <jj...@cl...>]

On Mon, 5 Aug 2002, Geoff Hutchison wrote:

> Date: Mon, 05 Aug 2002 17:29:26 -0400 (EDT)
> From: Geoff Hutchison <ghu...@ws...>
> To: Joe R. Jah <jj...@cl...>
> Cc: htdig3-dev <htd...@li...>
> Subject: Re: [htdig-dev] Re: mifluz merge snapshot 2002-08-03
> 
> On Mon, 5 Aug 2002, Joe R. Jah wrote:
> 
> > Configured with-rx; compiled OK, but htdig dumped core.  Here is a gdb
> > back trace:
> 
> > Reading symbols from /usr/local/htdig/3.2/lib/htdig/libhtnet-...so...done.
> > Reading symbols from /usr/local/htdig/3.2/lib/htdig/libcommon-...so...done.
> > Reading symbols from /usr/local/htdig/3.2/lib/htdig/libhtword-...so...done.
> > Reading symbols from /usr/local/htdig/3.2/lib/htdig/libht-...so...done.
> 
> Hmm. Did you actually do a "make install" for this version? Or is it
> reading symbols from a previous version?

I did "make install"; didn't you expect it to install?

$ ll /usr/local/htdig/3.2/lib/htdig
total 7509
-rwxr-xr-x  1 jjah  www   541850 Aug  4 23:55 libcommon-...so
-rw-r--r--  1 jjah  www   949958 Aug  4 23:55 libcommon.a
-rwxr-xr-x  1 jjah  www      782 Aug  4 23:55 libcommon.la
lrwxr-xr-x  1 jjah  www       15 Aug  4 23:55 libcommon.so -> libcommon-...so
-rwxr-xr-x  1 jjah  www   190438 Aug  4 23:55 libfuzzy-...so
-rw-r--r--  1 jjah  www  1002278 Aug  4 23:55 libfuzzy.a
-rwxr-xr-x  1 jjah  www      775 Aug  4 23:55 libfuzzy.la
lrwxr-xr-x  1 jjah  www       14 Aug  4 23:55 libfuzzy.so -> libfuzzy-...so
-rwxr-xr-x  1 jjah  www   424372 Aug  4 23:55 libht-...so
-rw-r--r--  1 jjah  www   897706 Aug  4 23:55 libht.a
-rwxr-xr-x  1 jjah  www      754 Aug  4 23:55 libht.la
lrwxr-xr-x  1 jjah  www       11 Aug  4 23:55 libht.so -> libht-...so
-rwxr-xr-x  1 jjah  www   271578 Aug  4 23:55 libhtnet-...so
-rw-r--r--  1 jjah  www   563546 Aug  4 23:55 libhtnet.a
-rwxr-xr-x  1 jjah  www      775 Aug  4 23:55 libhtnet.la
lrwxr-xr-x  1 jjah  www       14 Aug  4 23:55 libhtnet.so -> libhtnet-...so
-rwxr-xr-x  1 jjah  www   989397 Aug  4 23:55 libhtword-...so
-rw-r--r--  1 jjah  www  1734226 Aug  4 23:55 libhtword.a
-rwxr-xr-x  1 jjah  www      782 Aug  4 23:55 libhtword.la
lrwxr-xr-x  1 jjah  www       15 Aug  4 23:55 libhtword.so -> libhtword-...so

> Right now, I'm mostly interested to know if it builds and/or
> indexes. Searching may seem important, but we'll take this a step at a
> time. :-) (Alas, there seem to be enough bugs to keep me busy for a
> while.)
> 
> I have a version that's actually indexing right now, so there'll be
> another snapshot in a day or two.

OK.

Regards,

Joe
-- 
     _/   _/_/_/       _/              ____________    __o
     _/   _/   _/      _/         ______________     _-\<,_
 _/  _/   _/_/_/   _/  _/                     ......(_)/ (_)
  _/_/ oe _/   _/.  _/_/ ah        jj...@cl...

Re: [htdig-dev] Re: mifluz merge snapshot 2002-08-03

[Geoff H. <ghu...@ws...>]

On Mon, 5 Aug 2002, Joe R. Jah wrote:

> Configured with-rx; compiled OK, but htdig dumped core.  Here is a gdb
> back trace:

> Reading symbols from /usr/local/htdig/3.2/lib/htdig/libhtnet-...so...done.
> Reading symbols from /usr/local/htdig/3.2/lib/htdig/libcommon-...so...done.
> Reading symbols from /usr/local/htdig/3.2/lib/htdig/libhtword-...so...done.
> Reading symbols from /usr/local/htdig/3.2/lib/htdig/libht-...so...done.

Hmm. Did you actually do a "make install" for this version? Or is it
reading symbols from a previous version?

Right now, I'm mostly interested to know if it builds and/or
indexes. Searching may seem important, but we'll take this a step at a
time. :-) (Alas, there seem to be enough bugs to keep me busy for a
while.)

I have a version that's actually indexing right now, so there'll be
another snapshot in a day or two.

-Geoff

[htdig-dev] status of synonyms?

[Budd, S. <s....@ic...>]

Hello
  using htdig3.2.0b4-20020728  there are strange results for synonyms. 
  (The capitalization below is for clarity, actual searches used lower
case.)
  In the synonym  file "abcense" is set as synonym with "absence".

 with htdig.conf set with 
 search_algorithm:	exact:1  endings:0.1  

 searching for   abCense   produces  0  results
                 abSence   produces  11 results



 with htdig.conf set with 
 search_algorithm:	exact:1  endings:0.1   synonyms:0.9

 searching for   abCense   produces  783 results ( and these results contain
"abSence" words ok)
                 abSence   produces  11 results  ( same results as for no
synonyms )

curious no?

Re: [htdig-dev] Re: mifluz merge snapshot 2002-08-03

[Joe R. J. <jj...@cl...>]

On Sun, 4 Aug 2002, Geoff Hutchison wrote:

> Date: Sun, 4 Aug 2002 22:36:55 -0500
> From: Geoff Hutchison <ghu...@ws...>
> To: htdig3-dev <htd...@li...>
> Subject: [htdig-dev] Re: mifluz merge snapshot 2002-08-03

> * This version has received limited testing, so there are likely tons of 
> bugs.
>   Please report them to me or htd...@li... so they can 
> be squashed.

Configured with-rx; compiled OK, but htdig dumped core.  Here is a gdb
back trace:
-------------------------------8<------------------------------
$ gdb htdig htdig.core
This GDB was configured as "i386-unknown-bsdi4.3"...
Core was generated by `htdig'.
Program terminated with signal 6, Aborted.
Reading symbols from /usr/local/htdig/3.2/lib/htdig/libhtnet-...so...done.
Reading symbols from /usr/local/htdig/3.2/lib/htdig/libcommon-...so...done.
Reading symbols from /usr/local/htdig/3.2/lib/htdig/libhtword-...so...done.
Reading symbols from /usr/local/htdig/3.2/lib/htdig/libht-...so...done.
Reading symbols from /usr/lib/libz.so...done.
Reading symbols from /usr/local/lib/libiconv.so.2...done.
Reading symbols from /usr/lib/libstdc++.so.1...done.
Reading symbols from /shlib/libm.so.0.0...done.
Reading symbols from /shlib/libgcc.so.1...done.
Reading symbols from /shlib/libc.so.2...done.
Reading symbols from /shlib/ld-bsdi.so...done.
#0  0x482d948d in kill () from /shlib/libc.so.2
(gdb) bt
#0  0x482d948d in kill () from /shlib/libc.so.2
#1  0x483649b3 in abort () from /shlib/libc.so.2
#2  0x48137a0d in WordList::NotImplemented () at WordList.h:427
#3  0x8055868 in Retriever::Start (this=0x8047a70) at ../htword/WordList.h:207
#4  0x805d11e in main (ac=2, av=0x8047ce4) at htdig.cc:339
#5  0x804ea13 in __start ()
(gdb) q
-------------------------------8<------------------------------

htfuzzy, htnotify, and htpurge, all dumped core; here is their gdb back
trace;
-------------------------------8<------------------------------
...
(gdb) bt
#0  0x480de1af in yyparse (aConf=0x80ab000) at conf_parser.cxx:740
#1  0x480dc581 in HtConfiguration::Read (this=0x80ab000, filename=@0x8047c78) at HtConfiguration.cc:246
#2  0x804bb7f in main (ac=2, av=0x8047cdc) at htfuzzy.cc:131
#3  0x804b823 in __start ()

#0  0x480d51af in yyparse (aConf=0x80ad000) at conf_parser.cxx:740
740                    ((HtConfiguration *)aConf)->AddParsed(yyvsp[0].ConfLine->name,yyvsp[0].ConfLine->value);
...
(gdb) bt
#0  0x480d51af in yyparse (aConf=0x80ad000) at conf_parser.cxx:740
#1  0x480d3581 in HtConfiguration::Read (this=0x80ad000, filename=@0x8047c74) at HtConfiguration.cc:246
#2  0x804c297 in main (ac=1, av=0x8047ce8) at htnotify.cc:148
#3  0x804bfd3 in __start ()

#0  0x480d51af in yyparse (aConf=0x80ad000) at conf_parser.cxx:740
740                    ((HtConfiguration *)aConf)->AddParsed(yyvsp[0].ConfLine->name,yyvsp[0].ConfLine->value);
...
(gdb) bt 
#0  0x480d51af in yyparse (aConf=0x80ad000) at conf_parser.cxx:740
#1  0x480d3581 in HtConfiguration::Read (this=0x80ad000, filename=@0x8047c84) at HtConfiguration.cc:246
#2  0x804c178 in main (ac=1, av=0x8047ce8) at htpurge.cc:88
#3  0x804bf83 in __start ()
-------------------------------8<------------------------------

Regards,

Joe
-- 
     _/   _/_/_/       _/              ____________    __o
     _/   _/   _/      _/         ______________     _-\<,_
 _/  _/   _/_/_/   _/  _/                     ......(_)/ (_)
  _/_/ oe _/   _/.  _/_/ ah        jj...@cl...

[htdig-dev] Re: mifluz merge (linking help needed)

[Geoff H. <ghu...@ws...>]

On Mon, 5 Aug 2002 And...@wi... wrote:

> The link has changed, its at (no .diff):
> <http://www.htdig.org/files/snapshots/mifluz-merge-20020804.tar.gz>

Yes, I sent out a later message about this:
http://sourceforge.net/mailarchive/forum.php?thread_id=953142&forum_id=2688

At the least, the .tar.gz referenced above should build for everyone.

-Geoff

[htdig-dev] Re: mifluz merge (linking help needed)

[<And...@wi...>]

The link has changed, its at (no .diff):
<http://www.htdig.org/files/snapshots/mifluz-merge-20020804.tar.gz>


> Well, I've had the time to at least get a preliminary merge of the 
current
mifluz code -- it's taken quite a bit of time, not the least because it
needed updates to autoconf/automake/libtool and it took some time to beat
out bugs in that.

There's still one remaining problem before it can be tested--none of the
programs will link, due to:

./.libs/libfuzzy.so: undefined reference to `strncasecmp(char const *,
char const *, int)'

There are related linking problems with strcasecmp and the "my" versions
as well. I'm confused as I've added these to the main htconfig.h:

#define mystrncasecmp strncasecmp
#define mystrcasecmp strcasecmp

And yes, I do have working versions of strncasecmp and strcasecmp, which
is why autoconf/automake haven't compiled the replacement function.


I'm hoping a few people would be willing to grab the patch at:
<http://www.htdig.org/files/snapshots/mifluz-merge-20020803.diff.gz> build
it (relative to the latest source or snapshot) and see if they can also
help figure out what's going on with strcasecmp and company.

If there are other build errors, please let me know too--I'm still working
on finishing this up ASAP.

Andy Bach, Sys. Mangler
Internet: and...@wi... 
VOICE: (608) 261-5738  FAX 264-5030

"To understand recursion, we must first understand recursion."

[htdig-dev] Re: mifluz merge snapshot 2002-08-03

[Geoff H. <ghu...@ws...>]

This version of ht://Dig is an experimental snapshot of 3.2.0b4 including
a new version of the mifluz backend. With help testing this and beating 
out some bugs, I'll pour it into the main CVS repository. So if you're 
willing to try very bleeding edge code (i.e. may not compile or run 
correctly), some feedback would be very helpful.

The snapshots will be left in <http://www.htdig.org/files/snapshots/> 
but there's no script--I'm rolling them by hand as needed. I doubt I'll 
ever have time to roll more than one a day, so I'm not concerned by the 
date stamp. :-)

If you'd like to give it a spin, please see below, particularly about 
libiconv--which will eventually have to be bundled much like the 
Berkeley DB or how we bundled librx.

-Geoff

CAVEAT LECTOR:
* Some additional software may be required for compilation and/or 
running this
  version. Currently, known requirements are:
   - libiconv: <ftp://ftp.gnu.org/pub/gnu/libiconv/>

  Automake 1.6.3 and Autoconf 2.53 are required for updating/revising 
config and
  Makefiles. Older versions will cause problems in building.
* This version is extremely experimental. Don't blame me if it eats your 
files,
  disks, RAM, OS, etc. I doubt it, but I'm not making any promises.
  (NO WARRANTY EXPRESSED OR IMPLIED.)

* This version has received limited testing, so there are likely tons of 
bugs.
  Please report them to me or htd...@li... so they can 
be squashed.

* Not all features are implemented, e.g. revised documentation, faster 
searching
  lower memory requirements, etc. This will happen over time.

Re: [htdig-dev] Re: mifluz merge (linking help needed) (fwd)

[Geoff H. <ghu...@ws...>]

> I immediately get the following when I try to build.

I think it's going to be easier to just distribute a .tar.gz since the 
patch is so huge. It probably also didn't apply correctly since I'm 
working on the mainline and the recent snapshots have (unfortunately) 
been off of the htdig-3-2-x branch.

Sorry for the problems.

>> If there are other build errors, please let me know too--I'm still 
>> working
>> on finishing this up ASAP.

Well, I have those licked, but now I need to fix some segfaults. 
Hopefully I'll have something for people to test very shortly. I'm 
hoping to test the build on the SourceForge compile farm, which should 
hopefully cut down on some problems from the get-go.

-Geoff

[htdig-dev] Current Status as of snapshot 3.2.0b4-20020804

[Geoff H. <ghu...@us...>]

STATUS of ht://Dig branch 3-2-x

RELEASES:
   3.2.0b4: In progress 
	(mifluz merge essentially finished, contact Geoff for patch to test)
   3.2.0b3: Released:  22 Feb 2001.
   3.2.0b2: Released:  11 Apr 2000.
   3.2.0b1: Released:   4 Feb 2000.

SHOWSTOPPERS:

KNOWN BUGS:
* Odd behavior with $(MODIFIED) and scores not working with
   wordlist_compress set but work fine without wordlist_compress.
   (the date is definitely stored correctly, even with compression on
    so this must be some sort of weird htsearch bug)
* Not all htsearch input parameters are handled properly: PR#648. Use a
   consistant mapping of input -> config -> template for all inputs where
   it makes sense to do so (everything but "config" and "words"?).
* If exact isn't specified in the search_algorithms, $(WORDS) is not set 
   correctly: PR#650. (The documentation for 3.2.0b1 is updated, but can
   we fix this?)
* META descriptions are somehow added to the database as FLAG_TITLE,
   not FLAG_DESCRIPTION. (PR#859)

PENDING PATCHES (available but need work):
* Additional support for Win32.
* Memory improvements to htmerge. (Backed out b/c htword API changed.)

NEEDED FEATURES:
* Field-restricted searching.
* Return all URLs.
* Handle noindex_start & noindex_end as string lists.

TESTING:
* httools programs: 
  (htload a test file, check a few characteristics, htdump and compare)
* Turn on URL parser test as part of test suite.
* htsearch phrase support tests
* Tests for new config file parser
* Duplicate document detection while indexing
* Major revisions to ExternalParser.cc, including fork/exec instead of popen,
  argument handling for parser/converter, allowing binary output from an
  external converter.
* ExternalTransport needs testing of changes similar to ExternalParser.

DOCUMENTATION:
* List of supported platforms/compilers is ancient.
* Add thorough documentation on htsearch restrict/exclude behavior
   (including '|' and regex).
* Document all of htsearch's mappings of input parameters to config attributes
   to template variables. (Relates to PR#648.) Also make sure these config
   attributes are all documented in defaults.cc, even if they're only set by
   input parameters and never in the config file.
* Split attrs.html into categories for faster loading.
* require.html is not updated to list new features and disk space
   requirements of 3.2.x (e.g. phrase searching, regex matching,
   external parsers and transport methods, database compression.)
* TODO.html has not been updated for current TODO list and completions.

OTHER ISSUES:
* Can htsearch actually search while an index is being created?
   (Does Loic's new database code make this work?)
* The code needs a security audit, esp. htsearch
* URL.cc tries to parse malformed URLs (which causes further problems)
   (It should probably just set everything to empty) This relates to 
   PR#348.

Re: [htdig-dev] Re: mifluz merge (linking help needed) (fwd)

[Jim C. <gre...@yg...>]

Geoff Hutchison's bits of Sat, 3 Aug 2002 translated to:

>I'm hoping a few people would be willing to grab the patch at:
><http://www.htdig.org/files/snapshots/mifluz-merge-20020803.diff.gz> build
>it (relative to the latest source or snapshot) and see if they can also
>help figure out what's going on with strcasecmp and company.
>
>If there are other build errors, please let me know too--I'm still working
>on finishing this up ASAP.

I immediately get the following when I try to build.

Makefile:215: warning: overriding commands for target `_AM_File'
Makefile:205: warning: ignoring old commands for target `_AM_File'
Makefile:234: warning: overriding commands for target `[]'
Makefile:224: warning: ignoring old commands for target `[]'
make: *** No rule to make target `_AM_File,', needed by `all-recursive-am'.  Stop.

I am using htdig-3.2.0b4-20020728 with the above mentioned patch.
I get the same results under both OS X and Linux with GNU make
3.79.

Jim

[htdig-dev] Re: mifluz merge (linking help needed) (fwd)

[Geoff H. <ghu...@ws...>]

Well, I've had the time to at least get a preliminary merge of the current
mifluz code -- it's taken quite a bit of time, not the least because it
needed updates to autoconf/automake/libtool and it took some time to beat
out bugs in that.

There's still one remaining problem before it can be tested--none of the
programs will link, due to:

./.libs/libfuzzy.so: undefined reference to `strncasecmp(char const *,
char const *, int)'

There are related linking problems with strcasecmp and the "my" versions
as well. I'm confused as I've added these to the main htconfig.h:

#define mystrncasecmp strncasecmp
#define mystrcasecmp strcasecmp

And yes, I do have working versions of strncasecmp and strcasecmp, which
is why autoconf/automake haven't compiled the replacement function.


I'm hoping a few people would be willing to grab the patch at:
<http://www.htdig.org/files/snapshots/mifluz-merge-20020803.diff.gz> build
it (relative to the latest source or snapshot) and see if they can also
help figure out what's going on with strcasecmp and company.

If there are other build errors, please let me know too--I'm still working
on finishing this up ASAP.

-Geoff

Re: [htdig-dev] Error in zlib Compressor for WordDB

[Geoff H. <ghu...@ws...>]

On Fri, 2 Aug 2002, Neal Richter wrote:

> Ah.. might I suggest a changing of the description to make it clear that
> this config verb affects ONLY the excerpts DB and has nothing to do
> with the words DB?

Don't need to.

The link I gave was to the 3.1.x documentation--so there is no word DB
compression. The documentation for 3.2 says

        this attribute controls the amount of compression used in the
        doc_excerpt file.

I sent a link to the 3.1 documentation, largely pointing out that this has
been around a while. The appropriate 3.2 link is:

<http://www.htdig.org/dev/htdig-3.2/attrs.html#excerpt_length>

-Geoff

Re: [htdig-dev] Error in zlib Compressor for WordDB

[Neal R. <ne...@ri...>]

Ah.. might I suggest a changing of the description to make it clear that
this config verb affects ONLY the excerpts DB and has nothing to do
with the words DB?

Thanks!

On Fri, 2 Aug 2002, Geoff Hutchison wrote:

> On Fri, 2 Aug 2002, Neal Richter wrote:
> 
> > 	It looks like the excerpts DB is not compressed at all...
> 
> Not by default:
> <http://www.htdig.org/attrs.html#compression_level>
> 
> --
> -Geoff Hutchison
> Williams Students Online
> http://wso.williams.edu/
> 
> 
> 
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> htdig-dev mailing list
> htd...@li...
> https://lists.sourceforge.net/lists/listinfo/htdig-dev
> 

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] Error in zlib Compressor for WordDB

[Geoff H. <ghu...@ws...>]

On Fri, 2 Aug 2002, Neal Richter wrote:

> 	It looks like the excerpts DB is not compressed at all...

Not by default:
<http://www.htdig.org/attrs.html#compression_level>

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

Re: [htdig-dev] Error in zlib Compressor for WordDB

[Neal R. <ne...@ri...>]

	It looks like the excerpts DB is not compressed at all...

	This would be handy, since as the number of documents in an
archive increases the words DB growth flattens out (there are only so
many words in a language) where as the excerpts DB just keeps on growing.



On Tue, 30 Jul 2002, Geoff Hutchison wrote:

> 
> > 	Does anyone know exactly how to disable the BDB compression for
> > the word-database  db.words.db?
> 
> <http://www.htdig.org/dev/htdig-3.2/attrs.html#wordlist_compress>
> 
> -Geoff
> 
> 
> 
> -------------------------------------------------------
> This sf.net email is sponsored by: Dice - The leading online job board
> for high-tech professionals. Search and apply for tech jobs today!
> http://seeker.dice.com/seeker.epl?rel_code=31
> _______________________________________________
> htdig-dev mailing list
> htd...@li...
> https://lists.sourceforge.net/lists/listinfo/htdig-dev
> 

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485