htdig-dev — Developer Discussion for the ht://Dig project

[htdig-dev] German Umlaut in resulting search

[Bernier, M. <mel...@sd...>]

> Hi guys!
>=20
> I have installed htdig and I have a little problem with German =
Umlaut.  I
> can search for words with Umlaut without any problem.  When I search =
for
> say 'C34644' (a file containing Umlaut), the results from htdig comes =
back
> with strange characters instead of Umlaut (for example, I get a =
circle (=D8)
> instead of =FC, or I get a bit =C4 instead of a small =E4), and it =
seems to
> return that kind of results only for word documents.  What could be =
the
> problem?
>=20
> In order to use german dictionary, in my htdig.conf I have the =
following:
>=20
> ispell_dir=3D/usr/lib/ispell
>=20
> # to use german dictionary
> locale:                 de_DE
> lang_dir:               ${ispell_dir}
> endings_affix_file:     ${lang_dir}/german.aff
> endings_dictionary:     ${lang_dir}/german.0
>=20
In order to parse word documents, in my htdig.conf I have the =
following:

# to convert different binary documents (.doc, .ps, .pdf, .xls, .ppt)
#doc2html=3D/usr/local/bin/doc2html.pl
external_parsers:       application/pdf->text/html
/usr/local/bin/doc2html.pl \
                        application/postscript->text/html
/usr/local/bin/doc2htm
                        application/msword->text/html
/usr/local/bin/doc2html.pl
                        application/msexcel->text/html
/usr/local/bin/doc2html.p
                        application/vnd.ms-excel->text/html
/usr/local/bin/doc2h
                        application/vnd.ms-powerpoint->text/html
/usr/local/bin/

> By the way, I am using htdig on Suse 7.2 Linux distribution.
>=20
> Could you please reply by email since I didn't subscribe to the =
mailing
> list?
>=20
> Thanks in advance for any help.
>=20
> Melanie.

Re: [htdig-dev] Re: URL Rewrite patch for 3.1.6 snapshots

[Joe R. J. <jj...@cl...>]

On Wed, 17 Oct 2001, Gilles Detillieux wrote:

> Date: Wed, 17 Oct 2001 15:35:53 -0500 (CDT)
> From: Gilles Detillieux <gr...@sc...>
> To: Joe R. Jah <jj...@cl...>
> Cc: htd...@li...
> Subject: Re: [htdig-dev] Re: URL Rewrite patch for 3.1.6 snapshots
> 
> > I found 82 links from one document with META ROBOT: Noindex tag;)  I could
> > not find an efficient way of hunting down the other 138 links that were
> > unaccounted for in two 20 meg+ files; however, I must assume that they are
> > some sort of duplicates;-/
> 
> Hmm.  Too bad we couldn't get something more definitive.  I'm fairly
> confident that the changes to the HTML parser didn't break anything, but
> I'd feel much more comfortable if we could explain the missing files you
> discovered rather than just assuming it's OK.  If I recall, there were
> 88 URLs with doubled slashes that were eliminated in an earlier test,
> but that still leaves around 50 URLs unaccounted for.
> 
> If there's any way you can take a snapshot of your site, or a few major
> subdirectories, and duplicate them somewhere else where they won't get
> modified, it would be a big help in getting conclusive results.  If you
> index the exact same files with 3.1.5 and 3.1.6, you should be able to
> diff the output of htdig -vvv from both, and pinpoint exactly where the
> differences are happening.  I know this is asking a lot, but it would be
> a shame to release 3.1.6 after all the work that's gone into it, only to
> discover afterward that it introduced a serious bug.

Sorry it took such a long time to respond, but I have been very busy
lately.  It is not easy to prove a negative; however, I have tried a few
times to make 3.1.6 miss indexing files in stable snapshots of my site
without success;)

Here is a comparison of the latest 3.1.6 snapshot on a snapshot of my site
-- 163 HTML-only documents -- with 3.1.6-072901:

_______3.1.6-072901 + Armstrong patch + ssl.4_______
htdig:   Start digging: Sun Nov 11 18:15:43 PST 2001
htmerge: Start merging: Sun Nov 11 18:16:16 PST 2001  33 seconds
htmerge: Total word count: 13171
htmerge: Total documents: 163
htmerge: Total doc db size (in K): 1888
-------------------------8<-------------------------
__________3.1.6-111101 + ssl.5 + FAQ#5.14___________
htdig:   Start digging: Sun Nov 11 18:19:19 PST 2001
htmerge: Start merging: Sun Nov 11 18:20:58 PST 2001  99 seconds
htmerge: Total word count: 13171
htmerge: Total documents: 163
htmerge: Total doc db size (in K): 1888
-------------------------8<-------------------------
CPU:	350 MHz Pentium 
RAM:	384 Megs
OS:	BSDi-4.2

They both index the exact number of documents; this is as conclusive a
result as I can produce.  The only difference is the the time they take.

Incidentally, ssl.4 fails to apply to the latest snapshot because of the
recent changes to Connection.cc.  I have modified the patch to apply
cleanly to the latest snapshot of 3.1.6:

	ftp://ftp.ccsf.org/htdig-patches/3.1.6/ssl.5

Regards,

Joe
-- 
     _/   _/_/_/       _/              ____________    __o
     _/   _/   _/      _/         ______________     _-\<,_
 _/  _/   _/_/_/   _/  _/                     ......(_)/ (_)
  _/_/ oe _/   _/.  _/_/ ah        jj...@cl...

[htdig-dev] Current Status as of snapshot 3.2.0b4-111101

[Geoff H. <ghu...@us...>]

STATUS of ht://Dig branch 3-2-x

RELEASES:
   3.2.0b4: In progress
   3.2.0b3: Released:  22 Feb 2001.
   3.2.0b2: Released:  11 Apr 2000.
   3.2.0b1: Released:   4 Feb 2000.

SHOWSTOPPERS:

KNOWN BUGS:
* Odd behavior with $(MODIFIED) and scores not working with
   wordlist_compress set but work fine without wordlist_compress.
   (the date is definitely stored correctly, even with compression on
    so this must be some sort of weird htsearch bug)
* Not all htsearch input parameters are handled properly: PR#648. Use a
   consistant mapping of input -> config -> template for all inputs where
   it makes sense to do so (everything but "config" and "words"?).
* If exact isn't specified in the search_algorithms, $(WORDS) is not set 
   correctly: PR#650. (The documentation for 3.2.0b1 is updated, but can
   we fix this?)
* META descriptions are somehow added to the database as FLAG_TITLE,
   not FLAG_DESCRIPTION. (PR#859)

PENDING PATCHES (available but need work):
* Additional support for Win32.
* Memory improvements to htmerge. (Backed out b/c htword API changed.)
* MySQL patches to 3.1.x to be forward-ported and cleaned up.
  (Should really only attempt to use SQL for doc_db and related, not word_db)

NEEDED FEATURES:
* Field-restricted searching.
* Return all URLs.
* Handle noindex_start & noindex_end as string lists.
* Handle local_urls through file:// handler, for mime.types support.
* Handle directory redirects in RetrieveLocal.
* Merge with mifluz

TESTING:
* httools programs: 
  (htload a test file, check a few characteristics, htdump and compare)
* Turn on URL parser test as part of test suite.
* htsearch phrase support tests
* Tests for new config file parser
* Duplicate document detection while indexing
* Major revisions to ExternalParser.cc, including fork/exec instead of popen,
  argument handling for parser/converter, allowing binary output from an
  external converter.
* ExternalTransport needs testing of changes similar to ExternalParser.

DOCUMENTATION:
* List of supported platforms/compilers is ancient.
* Add thorough documentation on htsearch restrict/exclude behavior
   (including '|' and regex).
* Document all of htsearch's mappings of input parameters to config attributes
   to template variables. (Relates to PR#648.) Also make sure these config
   attributes are all documented in defaults.cc, even if they're only set by
   input parameters and never in the config file.
* Split attrs.html into categories for faster loading.
* require.html is not updated to list new features and disk space
   requirements of 3.2.x (e.g. phrase searching, regex matching,
   external parsers and transport methods, database compression.)
* TODO.html has not been updated for current TODO list and completions.

OTHER ISSUES:
* Can htsearch actually search while an index is being created?
   (Does Loic's new database code make this work?)
* The code needs a security audit, esp. htsearch
* URL.cc tries to parse malformed URLs (which causes further problems)
   (It should probably just set everything to empty) This relates to 
   PR#348.

Re: [htdig-dev] unlink searched word

[Gilles D. <gr...@sc...>]

According to Martin Resch:
> When Patterns were found and the results are shown in long-form, the
> search-words are linked with a 'a href' to the sites. 
> Can i stop this? Can I unlink the words which were found?

See http://www.htdig.org/attrs.html#add_anchors_to_excerpt

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/~grdetil
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930

Re: [htdig-dev] help with testing

[Gilles D. <gr...@sc...>]

According to tr...@ze...:
> on running rundig, after basic  (on OS X 10.1) installation, i got:
> 
> DB2 problem. . . : DB->get: method meaningless before open
> 
> about a thousand times and then, finally:
> 
> DB2 problem. . . : DB->get: method meaningless before open
> /bin/mv: rename /opt/www/var/htdig/root2word.db to 
> /opt/www/share/htdig/root2word.db: No such file or directory
> /bin/mv: rename /opt/www/var/htdig/word2root.db to 
> /opt/www/share/htdig/word2root.db: No such file or directory
> DB2 problem...: /opt/www/var/htdig/synonyms.db: Permission denied
> htfuzzy: Could not create database for algorithm: synonyms
> %
> 
> as i understand it, running 'rundig' after install is for indexing 
> the htdig site for test purposes.
> 
> i a newbie, so... is the above correct? and how do i proceed?

This message seems to have gotten lost in the torrent, so is it safe
to assume you've gotten past this problem somehow?  Your assessment of
rundig is correct, although if you configure your htdig.conf properly,
rundig will index your site and not htdig.org.  You can even use rundig
for daily database rebuilds if you have a small site (it takes about 4
min on my site right now).

I've never seen the "method meaningless before open" error before, but
something obviously messed up.  If it's still a problem, is there any
other error message before the first of those thousand or so messages?

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/~grdetil
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930

[htdig-dev] unlink searched word

[Martin R. <m....@te...>]

Hi everyone,

	i've got a question:
When Patterns were found and the results are shown in long-form, the
search-words are linked with a 'a href' to the sites. 
Can i stop this? Can I unlink the words which were found?

Thanks,

Martin

Re: [htdig-dev] nevermind! but read this one...

[<tr...@ma...>]

At 2:58 AM -0700 11/6/01, Jim Cole wrote:
>You might want to move to the general list. This is the
>development list, for which a lot of your questions are a bit
>off topic. I also expect that you will increase your chances
>for a prompt response to such questions by using the general
>list.

cool, thanks.
is there more traffic there?  ;)

i will.

Re: [htdig-dev] nevermind! but read this one...

[Jim C. <gre...@yg...>]

tr...@ze...'s bits of Tue, 6 Nov 2001 translated to:

>if you read my last posts -i got all that.
[snip]
><input type=hidden name=config value=htdig>
><input type=hidden name=restrict value="">
><input type=hidden name=exclude value="">
><br>
>
>what does the value... "htdig" do there?

Hi - It is the name of the configuration file, minus the '.conf'
extension. See http://www.htdig.org/hts_form.html

You might want to move to the general list. This is the
development list, for which a lot of your questions are a bit
off topic. I also expect that you will increase your chances
for a prompt response to such questions by using the general
list.


Jim

[htdig-dev] nevermind! but read this one...

[<tr...@ma...>]

if you read my last posts -i got all that.

now i'd like to know, there is one line in the sample search html that reads:

Sort by: <select name=sort>
<option value=score>Score
<option value=time>Time
<option value=title>Title
<option value=revscore>Reverse Score
<option value=revtime>Reverse Time
<option value=revtitle>Reverse Title
</select>
</font>
<input type=hidden name=config value=htdig>
<input type=hidden name=restrict value="">
<input type=hidden name=exclude value="">
<br>

what does the value... "htdig" do there?

thanks.

TedR

[htdig-dev] images -addendum

[<tr...@ma...>]

***add: as far as i can tell i have no "common", i have "share".***

i'm still just experimenting with testing htsearch by having had it 
rundig on htdig's website.

here are some excerpts from the instructions:
FAQ
***add: 4.16. How do I get htsearch to use the star image in a 
different directory than the default /htdig?

You must set either the image_url_prefix attribute, or both 
star_blank and star_image in your htdig.conf, to refer to the URL 
path for these files. You should also set this URL path similarly in 
in common/header.html and common/wrapper.html, as they also refer to 
the star.gif file. If you want to relocate other graphics, such as 
the buttons or the ht://Dig logo, you should change all references to 
these in htdig.conf and common/*.html.
***

1)
image_url_prefix

type:
string
used by:
htsearch
default:
IMAGE_URL_PREFIX
block:
Global
version:
all
description:
This specifies the directory portion of the URL used to display star 
images. This attribute isn't directly used by htsearch, but is used 
in the default URL forthe star_image and star_blank attributes, and 
other attributes may be defined in terms of this one.

The default value of this attribute is determined at compile time.

example:
image_url_prefix:	/images/htdig

2)
star_blank

type:
string
used by:
htsearch
default:
${image_url_prefix}/star_blank.gif
block:
Global
version:
all
description:
This specifies the URL to use to display a blank of the same size as 
the star defined in the star_image attribute or in the star_patterns 
attribute.
example:
star_blank:	http://www.somewhere.org/icons/noelephant.gif

3)
star_image

type:
string
used by:
htsearch
default:
${image_url_prefix}/star.gif
block:
Global
version:
all
description:
This specifies the URL to use to display a star. This allows you to 
use some other icon instead of a star. (We like the star...)
The display of stars can be turned on or off with the use_star_image 
attribute and the maximum number of stars that can be displayed is 
determined by the max_stars attribute.
Even though the image can be changed, the ALT value for the image 
will always be a '*'.
example:
star_image:	http://www.somewhere.org/icons/elephant.gif

4)
star_patterns

type:
string list
used by:
htsearch
default:
No default
block:
Global
version:
3.0 or later
description:
This attribute allows the star image to be changed depending on the 
URL or the match it is used for. This is mainly to make a visual 
distinction between matches on different web sites. The star image 
could be replaced with the logo of the company the match refers to.
It is advisable to keep all the images the same size in order to line 
things up properly in a short result listing.
The format is simple. It is a list of pairs. The first element of 
each pair is a pattern, the second element is a URL to the image for 
that pattern.
example:
star_patterns:	http://www.sdsu.edu /sdsu.gif \
http://www.ucsd.edu /ucsd.gif


what i am trying to do is to get the search results page to show these images.

the images i have in my install -htdig-3.2.0b4-102801 are:
star_blank.gif
star_blank.png
star.gif
star.png

i have been through the html files in 'share'....

besides wanting to get these displayed, i'd also so like to know 
where the parameters for the main results page reside, e.g., where 
would i edit the format of the results page, e.g., color, etc.?

thanks in advance.

Ted Rogers

[htdig-dev] ok, running nicely... question

[<tr...@ma...>]

about the default configuration:

i see the header, footer, long and short html files... but where the 
heck is the default file that uses all those htdig 'star' images?

i linked a few htdig images to the default (search htdig's www site) 
but i don't see the file to link the rest of the images, the "main" 
results file.

TIA

Ted Rogers

[htdig-dev] help with testing

[<tr...@ma...>]

on running rundig, after basic  (on OS X 10.1) installation, i got:

DB2 problem. . . : DB->get: method meaningless before open

about a thousand times and then, finally:

DB2 problem. . . : DB->get: method meaningless before open
/bin/mv: rename /opt/www/var/htdig/root2word.db to 
/opt/www/share/htdig/root2word.db: No such file or directory
/bin/mv: rename /opt/www/var/htdig/word2root.db to 
/opt/www/share/htdig/word2root.db: No such file or directory
DB2 problem...: /opt/www/var/htdig/synonyms.db: Permission denied
htfuzzy: Could not create database for algorithm: synonyms
%

as i understand it, running 'rundig' after install is for indexing 
the htdig site for test purposes.

i a newbie, so... is the above correct? and how do i proceed?

thank you.

Ted Rogers

[htdig-dev] Current Status as of snapshot 3.2.0b4-110401

[Geoff H. <ghu...@us...>]

STATUS of ht://Dig branch 3-2-x

RELEASES:
   3.2.0b4: In progress
   3.2.0b3: Released:  22 Feb 2001.
   3.2.0b2: Released:  11 Apr 2000.
   3.2.0b1: Released:   4 Feb 2000.

SHOWSTOPPERS:

KNOWN BUGS:
* Odd behavior with $(MODIFIED) and scores not working with
   wordlist_compress set but work fine without wordlist_compress.
   (the date is definitely stored correctly, even with compression on
    so this must be some sort of weird htsearch bug)
* Not all htsearch input parameters are handled properly: PR#648. Use a
   consistant mapping of input -> config -> template for all inputs where
   it makes sense to do so (everything but "config" and "words"?).
* If exact isn't specified in the search_algorithms, $(WORDS) is not set 
   correctly: PR#650. (The documentation for 3.2.0b1 is updated, but can
   we fix this?)
* META descriptions are somehow added to the database as FLAG_TITLE,
   not FLAG_DESCRIPTION. (PR#859)

PENDING PATCHES (available but need work):
* Additional support for Win32.
* Memory improvements to htmerge. (Backed out b/c htword API changed.)
* MySQL patches to 3.1.x to be forward-ported and cleaned up.
  (Should really only attempt to use SQL for doc_db and related, not word_db)

NEEDED FEATURES:
* Field-restricted searching.
* Return all URLs.
* Handle noindex_start & noindex_end as string lists.
* Handle local_urls through file:// handler, for mime.types support.
* Handle directory redirects in RetrieveLocal.
* Merge with mifluz

TESTING:
* httools programs: 
  (htload a test file, check a few characteristics, htdump and compare)
* Turn on URL parser test as part of test suite.
* htsearch phrase support tests
* Tests for new config file parser
* Duplicate document detection while indexing
* Major revisions to ExternalParser.cc, including fork/exec instead of popen,
  argument handling for parser/converter, allowing binary output from an
  external converter.
* ExternalTransport needs testing of changes similar to ExternalParser.

DOCUMENTATION:
* List of supported platforms/compilers is ancient.
* Add thorough documentation on htsearch restrict/exclude behavior
   (including '|' and regex).
* Document all of htsearch's mappings of input parameters to config attributes
   to template variables. (Relates to PR#648.) Also make sure these config
   attributes are all documented in defaults.cc, even if they're only set by
   input parameters and never in the config file.
* Split attrs.html into categories for faster loading.
* require.html is not updated to list new features and disk space
   requirements of 3.2.x (e.g. phrase searching, regex matching,
   external parsers and transport methods, database compression.)
* TODO.html has not been updated for current TODO list and completions.

OTHER ISSUES:
* Can htsearch actually search while an index is being created?
   (Does Loic's new database code make this work?)
* The code needs a security audit, esp. htsearch
* URL.cc tries to parse malformed URLs (which causes further problems)
   (It should probably just set everything to empty) This relates to 
   PR#348.

Re: [htdig-dev] help with 'make'

[<tr...@ma...>]

ok, i'm cool so far; i installed gnu make on the machine and it seems 
to be working from my dir after i copied it to htdig.

At 10:49 PM -0500 11/3/01, Geoff Hutchison wrote:
>On Sat, 3 Nov 2001 tr...@ma... wrote:
>
>>  the machine i'm installing is a Sun i believe, running FreeBSD.
>
>Run "gmake."
>
>--
>-Geoff Hutchison
>Williams Students Online
>http://wso.williams.edu/
>
>
>_______________________________________________
>htdig-dev mailing list
>htd...@li...
>https://lists.sourceforge.net/lists/listinfo/htdig-dev

Re: [htdig-dev] help with 'make'

[<tr...@ma...>]

hi,
thanks SO much for the reply.

i was sitting on the prompt hoping for some help.

so, without running configure again or anything -just going right 
from where i was (a prompt after that error)... on typing gmake
i got "gmake: Command not found".
?help?
Ted


At 10:49 PM -0500 11/3/01, Geoff Hutchison wrote:
>On Sat, 3 Nov 2001 tr...@ma... wrote:
>
>>  the machine i'm installing is a Sun i believe, running FreeBSD.
>
>Run "gmake."
>
>--
>-Geoff Hutchison
>Williams Students Online
>http://wso.williams.edu/
>
>
>_______________________________________________
>htdig-dev mailing list
>htd...@li...
>https://lists.sourceforge.net/lists/listinfo/htdig-dev

Re: [htdig-dev] help with 'make'

[Geoff H. <ghu...@ws...>]

On Sat, 3 Nov 2001 tr...@ma... wrote:

> the machine i'm installing is a Sun i believe, running FreeBSD.

Run "gmake."

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

[htdig-dev] help with 'make'

[<tr...@ma...>]

the machine i'm installing is a Sun i believe, running FreeBSD.
(G634-23 is me.)

what do you suppose i should do from here? (the 'configure' went very 
smoothly):
*************
Now you must run 'make' followed by 'make install'

%make
Making all in db
"Makefile", line 422: Need an operator
make: fatal errors encountered -- cannot continue
*** Error code 1

Stop in /usr2/websitedev/fall2001/G634-23/htdig-3.2.0b4-102801.
%

Re: [htdig-dev] PATCH: Content-type-aliases

[Geoff H. <ghu...@ws...>]

Hi Jamie,

You've sent a number of patches to the list--but so far my mailer has 
mangled most of them. I'd like to integrate them sometime in the near 
future--could you send them to me as attachments? (The 
content-type-alias.txt file was fine.)

Thanks,
-Geoff

Re: [htdig-dev] install placement for special case?

[Geoff H. <ghu...@ws...>]

At 4:45 PM -0500 11/3/01, tr...@ma... wrote:
>my question is, i have an account on the server just like each 
>student does (the same), can i install htdig into my account ? does 
>it have to be in some main server directory? or every students' 
>account?

You don't need to install it into every students account. However, 
you'll need to put the htsearch binary into a place that allows it to 
be a properly treated CGI. On some servers, this is a special 
directory. On other servers this is done by giving it a .cgi 
extension.

You may or may not want to have a separate config file for each 
student (e.g. to let them change the format of search results). But 
it's best to put all of these in the same place.

-- 
--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

[htdig-dev] install placement for special case?

[<tr...@ma...>]

Hi,

I'd like to install http://htDig or just htdig to act as a search 
engine for each separate student's website which they are building 
(i'll give them the form to slightly modify and insert into their 
html).

my question is, i have an account on the server just like each 
student does (the same), can i install htdig into my account ? does 
it have to be in some main server directory? or every students' 
account?

I'm a beginner so I hope this is understandable.

Ted Rogers

Re: [htdig-dev] looking for a bit of help

[Geoff H. <ghu...@ws...>]

At 4:22 AM -0700 11/2/01, Jim Cole wrote:
>I believe the patch the article refers to is unnecessary if you
>are working with the 3.1.6 snapshot (same goes for the 3.2.0b4
>snapshots if I recall correctly).

This is correct--I can confirm it builds and seems to run fine with 
3.1.6 or 3.2.0b4 snapshots under Mac OS X 10.1. (However, since I'm 
running over PPP, I can't do too much remote testing.)

-Geoff

Re: [htdig-dev] looking for a bit of help

[Jim C. <gre...@yg...>]

Hi - I would suggest that you try the 3.1.6 snapshot. It should
build/run on OS X right out of the box, though you may see a
number of warnings during the build. For obvious reasons, testing
under OS X (10.1) is quite limited.

You should also take a look at the archives. There were a couple
threads on OS X and htdig a few months ago. Finally, you might
want to check

  http://www.stepwise.com/Articles/Workbench/2000-02-26.01.html

I believe the patch the article refers to is unnecessary if you
are working with the 3.1.6 snapshot (same goes for the 3.2.0b4
snapshots if I recall correctly).


Jim


tr...@ze...'s bits of Fri, 2 Nov 2001 translated to:

>I kinda need to get htdig installed on a Mac OS X Server 10.1 pretty
>quickly and am hoping to find someone who wouldn't mind emailing with
>me  just a bit to get it up.
>
>I'm kind of between a rock and a hard place on this.
>
>I have already downloaded and am in the process of installing now.

[htdig-dev] looking for a bit of help

[<tr...@ma...>]

Hi,

I kinda need to get htdig installed on a Mac OS X Server 10.1 pretty 
quickly and am hoping to find someone who wouldn't mind emailing with 
me  just a bit to get it up.

I'm kind of between a rock and a hard place on this.

I have already downloaded and am in the process of installing now.

Thank you very much.

Ted Rogers
trogers at math dot fsu dot edu

Re: [htdig-dev] indentation?

[Geoff H. <ghu...@ws...>]

On Thu, 1 Nov 2001, Wolfgang Mueller wrote:

> The main problem I see is that I think a purely command line version of htdig 
> would be cool. Purely commandline in the sense that you can write some shell 
> that sais:
> 
> /path/to/htsearch --no-cgi --query-string="eggs and bacon" --sorted-by=score 
> >> results.xml

It's certainly not this "human readable," but you can certainly run
htsearch from the shell. For example many people have Perl or shell or
PHP "wrappers" around htsearch.

You can either set the QUERY_STRING environment variable, or supply the
query string on the command-line, e.g. (this may not quite be right--I
don't have a working example handy)

export REQUEST_METHOD=GET
export QUERY_STRING="config=..."
htsearch

or

htsearch "config=..."

> to call htsearch by a batch, and I would also like to get rid of the http 
> headers in the output(or did I overlook something??)

Gilles just commited a patch to 3.1.6 to do this--either to set the header
to another MIME type or to ignore it altogether.

In short, there's little reason your search-side situation isn't all that
different from another PHP/Perl wrapper.

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

Re: [htdig-dev] indentation?

[Wolfgang M. <Wol...@cu...>]

Hi, Gilles,
Thanks for your fast reply and for saving me from trouble.
> Before you go stripping out code from htsearch and end up having to
> maintain another search program in parallel to htsearch, be sure you
> fully investigate the template capabilities of htsearch.  There's almost

OK. In fact, I stumbled across these things *after* having had started to 
strip htdig.

> nothing in the output that you can change through some template file
> and/or config attribute.  In fact, I recently committed to the 3.1.6
> development code a number of new attributes that take care of the last
> few remaining areas that weren't configurable.  See the new attribute
> search_results_contenttype in this coming Sunday's 3.1.6 snapshot, as
> well as the existing add_anchors_to_excerpt attribute.

Ive got some CVS of htdig. So I'll update that.

> > What I would like to know: the summary strings you generate, are they
> > well-formed XML (in particular is there for each opening tag also a
> > closing tag? )
>
> You probably want to turn off anchors in excerpts (although they are
> well-formed HTML/XHTML), and all the rest is under control of the
> template files.  I think the distributed templates are all well-formed
> with closing tags.  Exceptions to this right now are the <br>, <hr>,
> <img>, <input> and <option> tags in the header, footer, nomatch, syntax
> and wrapper HTML files, but you're going to change them anyway.  Oh,
> there's also a non-self-terminating <br> tag in long.html and short.html
> that you may want to change to <br/> or remove.

Yes, I rolled myself some MRML.html, which generates query-result-element 
MRML tags.

The main problem I see is that I think a purely command line version of htdig 
would be cool. Purely commandline in the sense that you can write some shell 
that sais:

/path/to/htsearch --no-cgi --query-string="eggs and bacon" --sorted-by=score 
>> results.xml

(or some less fancy options style) and there is no interaction. Would you 
like me to start working on that some time? 

Currently, in fact, I have my ht_cli done (yes, a very ugly hack, and surely 
not commitable) and I am looking a bit more in the GIFT/popen side of things, 
which has me working on modifying Perl GIFT client for testing GIFT/HtDig.

To summarise: 
1) I will surely look deeper into templates (...and I will surely contribute 
the templates when they are done). 
2) I am waiting for your comments on adding some code that makes it possible 
to call htsearch by a batch, and I would also like to get rid of the http 
headers in the output(or did I overlook something??)
3) By simple modification of a couple of templates I seem to get the XML I 
want out of htsearch, at least for the case where there is no error (I tried 
that in the ten minutes between writing 2) and 3).

Cheers, and thanks,
Wolfgang

-- 
Dr. Wolfgang M&uuml;ller, assistant == teaching assistant
Personal page: http://cui.unige.ch/~vision/members/WolfgangMueller.html 
Maintainer, GNU Image Finding Tool (http://www.gnu.org/software/gift)