ht://Dig

Hi,

I am aware that htmerge needs temporary disk space for
sorting. 
Is there a default temporary directory used (i know
that we can change it by setting the TMPDIR
environment variable)? And, is there any way to
determine approximately how much of temporary disk
space is required for the sort (like is it close to
the database file size it creates)?

Thanks in advance,
Sunny

__________________________________________________
Do You Yahoo!?
Make a great connection at Yahoo! Personals.
http://personals.yahoo.com

Hi,

I'm using htdig 3-1-5 on solaris, with the french package=2E
Here's my problem :
I search a word with accent (such "d=E9c=E8s"), the query is "d=E9c=E8s or d=
eces", I
get 50 results=2E
I search the same word without accent (such "deces"), the query is "deces or
d=E9c=E8s", I
get 10 results=2E
Is there something to do to correct this ?

The search algorithmm is set to :       exact:1 synonyms:0=2E5 endings:0=2E1

Thanks,

	Laurent Pernot=2E

Hi there!

I suppose the bad words list contains the most often used words of the 
language. Is it imaginable that htdig indexes all files to be indexed 
and finds out the most often used words and prints them out, so I could 
decide which words I want to exclude from the index to speed up searching?

Would it help if I told you that the university of Leipzig has published 
word lists containing the 100, 1000 and 10000 most often used words of 
english, german, french and dutch at 
http://woclu2.informatik.uni-leipzig.de/html/wliste.html - no copyrights 
and no restrictions seem to be applied to the downloadable files?

Peter Asemann

Yes it should be optional.

When I have menu in the top of page and the alt's are:
'home', 'site one', 'site two'

htdig takes it like text and when I scan sites for 'site one' htdig shows 
me all sites where is top menu and shows me:
home site one site two.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

> I am rather surprised at the number of objections I've seen to this
> behaviour in htdig, added back in 3.1.4.  There must be a lot of sites
> that use the alt text attribute in incorrect or nonsensical ways.
>
> When used correctly, the text in the alt attribute of img tags should
> flow nicely with the text around it (that's how text-only browsers will
> use it after all), so I would have thought that the addition of this
> feature would be welcome.  I guess there are a lot of people out there
> who still use alt text for nonsense like "red ball" or "this is alt
> text".
>
> Should this feature be optional in future releases?

-- 
Emil W.
mailto:emilus@...

Hi All,

I run a mirror of a popular website. I also want to provide backend search
capabilities for all the other mirrors.  Basically I want to dig my local
mirror, but also allow other sites to query and return the links relative to
their mirror.  Is this possible w/ only 1 dig, and would it require a seperate
config file for each remote mirror?

My site:
http://flagspot.net/fotw/

All the others:
http://www.fotw.ca/flags/
http://www.netlinkit.dk/fotw/flags/
http://fotw.unis-ru.com/flags/
http://atlasgeo.span.ch/fotw/flags/
http://www.allstates-flag.com/fotw/flags/
http://www.crwflags.com/fotw/flags/
http://fotw.vexillum.com/flags/

TIA,

-Jim P.

__________________________________________________
Do You Yahoo!?
Make a great connection at Yahoo! Personals.
http://personals.yahoo.com

Hi there!

I'm gonna use htdig in a productive environment.
So what version should I install? Should I use multiple databases or just
one? If the search engine does not work I'll be in trouble, so I want the most
stable release and setup.
The 'environment' will be a rather big university, where sites of several
douzens of institutes will have to be indexed, with the possibility to restrict
the search to groups of or to single institutes.
I'd also like some hint what hardware I'll need. Fortunately "they" (my
employers) are willing to give me almost anything I need. It seems that even the
big number of institutes does not have more than a few gigabytes of data, so
only a part of that can be indexable files, and the database of htdig
probably does only take a part of that part.
So assuming a rather big index, what CPU/Ram would be needed to get results
really fast? I don't want google and alltheweb to be faster than a local
search.

Thanks in advance,

Peter Asemann

-- 
GMX - Die Kommunikationsplattform im Internet.
http://www.gmx.net

I'll just pass this on here.  I was creating an XML output interface 
(via templates) for htsearch output, and ran into what many others have 
run into - the problem of the content type.  So I made a small addition 
to Display that goes something like this...

    151         // cout << "Content-type: text/html\r\n\r\n";
    152
    153         // 
-------------------------------------------------------------------
    154         // Patch #1 - add support for alternate document type 
output (for XML)
    155         // 
-------------------------------------------------------------------
    156         String  content_type = config["search_results_contenttype"];
    157         if (content_type.length()) {
    158                cout << "Content-type: "; cout << content_type; 
cout << "\r\n\r\n";
    159         } else {
    160               cout << "Content-type: text/html\r\n\r\n";
    161         }
    162         // 
-------------------------------------------------------------------
    163         // End Patch #1
    164         // 
-------------------------------------------------------------------

So then in my config file, I just put...

    search_results_contenttype: text/xml

There are a few other places you have to patch the output - basically 
every place where a "Content-type: text/html" happens.  But you get the 
basic idea.

Tim Holt

Forwarded message from derekn@... (Derek Noonburg):

I've just released a new version of xpdf, my Portable Document Format
(PDF) viewer for X.

Xpdf runs under the X Window System on Unix, VMS, and OS/2.  The non-X
components of the package (pdftops, pdftotext, etc.) also run on Win32
systems.

Noticeable changes:

* Embed TrueType fonts in PostScript output.  (Added a "-noembtt" flag
  to pdftops.)
* Implement PDF 1.4 (128-bit) decryption.
* Bump supported PDF version number to 1.4.
* Text output for Simplified Chinese.  [Thanks to Cheung Siu Fai.]
* Read an app-defaults file for Xpdf.  Read a system-wide config file
  (<prefix>/etc/xpdfrc) if ~/.xpdfrc doesn't exist.
* Accept and verify owner password; if correct, allow all actions.
* Added a "-level2sep" option to pdftops to generate Level 2 separable
  PostScript.  The PostScript separation convention operators are used
  to handle custom (spot) colors.  [Thanks to Thomas Freitag for help
  on this.]
* Add support for FreeType 2 to the configure script.  Warning: this
  requires FT 2.0.5 or newer (see the INSTALL file for details).

See the `CHANGES' file for a complete list.

Source (C++ and C) is available, and it should be fairly easy to
compile for UNIX, VMS, OS/2, and Win32.

More information, source code, and precompiled binaries are on the
xpdf web page and ftp site:

    http://www.foolabs.com/xpdf/
    ftp://ftp.foolabs.com/pub/xpdf/

Source and Linux binaries are on sunsite.unc.edu, currently in
the incoming directory, but they will be moved to:

    ftp://ftp.ibiblio.org/pub/Linux/apps/graphics/viewers/X

On Fri, 26 Oct 2001, Sandeep Hulsandra wrote:

> I just read the BugTraq solution page to the "ht://dig Arbitrary File
> Inclusion Vulnerability".
> 
> So is the htdig-3.1.5 at - http://www.htdig.org/files/htdig-3.1.5.tar.gz

The BugTraq post and the resulting SecurityFocus database entry say
explicitly that 3.1.5 and before are vulnerable and 3.1.6 and newer are
not.

Since 3.1.6 has not been released yet, you can get a pre-release snapshot
at:
<http://www.htdig.org/files/snapshots/>

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

Hi,
From the source forge documentation,

"It may be possible for a remote attacker to cause a denial of service =
or under certain circumstances display arbitrary web-readable files. =
This is due to the fact that it is possible to use command line =
arguments from the web interface. In particular, the -c [filename] =
argument is normally used to specify an alternate configuration file. =
Using the web interface to request /dev/zero may cause a denial of =
service by exhausting resources on the host. A request for a =
web-readable file may cause it to be disclosed."

Just wanted to know how this its possible to use a command line argument =
from an http request?

Thanks,
Sandeep

A little background info:

I'm running version 3.2.0 B3 on RedHat Linux 7.1 and I'm not using any of
the default directories, therefore when I run htdig & htsearch I need to
specify the location of my configuration file with a "-c <config_dir>"


Here's the problem:

The only way of being able to call htsearch without having to put the
configuration file in the default directory, is to post the action as if I
was running it as a command line program.  Since I'm not using any of the
default directories, I need to specify to the htsearch where my
configuration file is located or else it always goes hunting at /etc which
is the default directory.

<form method="post"
action="/cgi-bin/htsearch?-c/files/htdigDB/myweb/conf/myweb.conf">

I also modified the templates (header/footer/wrapper/nomatch/syntax.html) to
make them fit to the action that I wanted them to take when a person
"refines" their search.  I thought I hacked it pretty well, but apparently
the buttons on the footer, the ones that create the previous page, page
list, and next page buttons/links they all have htsearch dynamically create
the hyperlink to go directly to the default directory to look for the
configuration file.

I read somewhere on the development page that it's a known bug (being able
to call a command line program from the form's action), but if it is a bug,
how else is it possible for a search form to call htsearch and be able to
use a specific configuration file that is not located in the default
directory?  Is there a way to modify the $(PREVPAGE), $(PAGELIST), and
$(NEXTPAGE) such that it'll point to a user defined directory?

thanks,
	natalie chin

A lot of my files being searched by htdig have space characters in their file names. In the search result pages these blanks are represented by "%20" in the hyperlinks pointing to the files. Since there are a lot of space characters it's very hard to read.
Is it possible to replace the %20's with a real blank or an underscore in the output?

Thanks,
Claus

At 9:22 AM +0200 10/25/01, Juergen Obermann wrote:
>It would be much better to have configuration directives
>'case_sensitive_urls' and 'case_insensitive_urls' in addition to the
>general boolean 'case_sensitive' to dig several servers in one run.

No, not really. At the moment, there are already *way* too many 
configuration attributes.

However, in version 3.2, you can specify certain attributes as having 
a per-server or per-URL context. At the moment, case_sensitive is not 
one of these attributes, but it will probably become a per-server 
attribute--the catch is fixing the places that deal with it.

If someone would like to look at adding this functionality to 3.2, I 
can point them in the right direction.

-- 
--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

On Thu, 25 Oct 2001, Ace wrote:

> So just because I am so curious: What is that mifluz, who makes it, what 
> does it do, and what does htdig do with it?

Mifluz is the standalone version of the ht://Dig 3.2 word database. It was
spun off as a separate project from ht://Dig by Loic Dacharay, who did a
lot of work on improving the base libraries and adding word database
compression to the package.

At the moment, Loic seems to have moved on to other things--I haven't
heard from him in around a year. And development on mifluz, unfortunately
seems to have stopped, though it's at a fairly mature level.

(Unfortunately, before Loic disappeared, he promised to help us resync the
parallel development on the htdig/mifluz code.)

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

I read the mailing list all the time for some weeks now and couldn't 
help to notice there's something in 3.2.0bx called mifluz that does 
something with some database and stuff...

So just because I am so curious: What is that mifluz, who makes it, what 
does it do, and what does htdig do with it?

Peter Asemann