htdig-dev — Developer Discussion for the ht://Dig project

[htdig-dev] PDF Parser

[Abbie G. <ag...@th...>]

Hi all, I'm up and running, well sort of.=20
=20
I did install HTDig, and pointed it towards a folder with just .html
files in it to test it...and voila got results.  I actually need this to
index pdf files though.
=20
I have so far done the following:
=20
Added to htdig.conf:
external_parsers <http://www.htdig.org/attrs.html#external_parsers> :
application/pdf->text/html /opt/www/htdig/bin/doc2html/doc2html.pl
=20
Installed the xpdf rpm
Installed the doc2html directory and scripts
=20
Set the paths for the pdftotext and pdfinfo, as well as setting the path
in doc2html.pl for the pdf2text.pl script
=20
I checked the largest file size of a pdf and increased the max file size
in htdig.conf as well.
=20
I run .rundig -v and it indexes one html document that I have at the top
level.  All permissions on files are fine I actually set them to 777 to
make sure it could get into the folders.  But it doesn't want to index
the pdfs...any ideas... =20
=20
I don't receive any error messages either.
=20
My file setup is /archives/folder/folder...etc
=20
I set htdig start_url at http://192.168.0.25/archives/
=20
I've tried moving a .pdf to the /archives file, but that doesn't work
either.
=20
Thanks!
=20
Abbie
=20
=20
=20
=20

[htdig-dev] Re: not allowing getopt::std

[Lachlan A. <lh...@us...>]

Greetings Andy,

Try the attached patch.  It is very rough-and-ready (since I'm in the=20
processes of also trying to add other functionality) but it should=20
get you going.

Let me know how you get on.

Cheers,
Lachlan

On Friday 31 January 2003 07:49, And...@wi... wrote:

> ...it would not find on double colons...

[htdig-dev] Re:Compiler Survey

[Lachlan A. <lh...@us...>]

On Wednesday 05 February 2003 10:45, Neal Richter wrote:
> What compiler is everyone using these days?

gcc version 3.2 (Mandrake Linux 9.0 3.2-1mdk)

Re: [htdig-dev] ./configure problem

[Gabriele B. <bar...@in...>]

At 14.21 04/02/2003 -0500, Abbie Greene wrote:

>System: RedHat Linux 8.0

You need these RPMs for backward compatibility (even though you'll keep 
getting some warnings - see FAQ http://www.htdig.org/FAQ.html#q3.8 )

compat-libstdc++-7.3-2.96.110.i386.rpm
compat-libstdc++-devel-7.3-2.96.110.i386.rpm

Hope this helps.

Ciao
-Gabriele
--
Gabriele Bartolini - Web Programmer - ht://Dig & IWA Member - ht://Check 
maintainer
Current Location: Prato, Tuscany, Italia
bar...@in... | http://www.prato.linux.it/~gbartolini | ICQ#129221447

[htdig-dev] Compiler Survey

[Neal R. <ne...@ri...>]

What compiler is everyone using these days?

Just curious.

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] Re: ./configure problem

[Neal R. <ne...@ri...>]

Abbie FYI:

There are two ways to avoid deprecated messages:

either use 'g++ -Wno-deprecated'

  option 1: CXXFLAGS="-Wno-deprecated" ./configure
  option 2: make clean; CXXFLAGS="-Wno-deprecated" make

or

install and use the older compiler suite on Redhat 8.0.  Get them on your
CDs or @ www.rpmfind.net

compat-libstdc++-7.3-2.96.110.i386.html
compat-libstdc++-devel-7.3- 2.96.110.i386.html
compat-gcc-objc-7.3-2.96.110.i386.html
compat-gcc-c++-7.3-2.96.110.i386.html
compat-gcc-7.3-2.96.110.i386.html

(this provides you with gcc296 & g++296 executables)

  option 1: CXX="g++296" ./configure
  option 2: make clean; CXX="g++296" make


On Wed, 5 Feb 2003, Lachlan Andrew wrote:

> Greetings Abbie,
>
> ht://Dig uses a style of C++ which gcc 3.2 considers "deprecated",
> which may be causing your problem.
>
> What does your  config.log  file say?
>
> Cheers,
> Lachlan
>
> On Wednesday 05 February 2003 06:21, Abbie Greene wrote:
>
> > Configure: error: To compile ht://Dig, you will need a C++ library.
> >  Try installing libstdc++.   Isn't gcc a c++ compiler?
> >
> > ...I checked and I already have libstdc++-3.2-7.i386.rpm installed
>
>
>
> -------------------------------------------------------
> This SF.NET email is sponsored by:
> SourceForge Enterprise Edition + IBM + LinuxWorld http://www.vasoftware.com
> _______________________________________________
> htdig-dev mailing list
> htd...@li...
> https://lists.sourceforge.net/lists/listinfo/htdig-dev
>

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

[htdig-dev] Re: ./configure problem

[Lachlan A. <lac...@ip...>]

Greetings Abbie,

ht://Dig uses a style of C++ which gcc 3.2 considers "deprecated",=20
which may be causing your problem.

What does your  config.log  file say?

Cheers,
Lachlan

On Wednesday 05 February 2003 06:21, Abbie Greene wrote:

> Configure: error: To compile ht://Dig, you will need a C++ library.
>  Try installing libstdc++.   Isn't gcc a c++ compiler?
>
> ...I checked and I already have libstdc++-3.2-7.i386.rpm installed

[htdig-dev] ./configure problem

[Abbie G. <ag...@th...>]

System: RedHat Linux 8.0
Installed gcc packages:
=20
I originally tried installing the htdig RPM, but my I honestly couldn't
find half the install files...so I decided it'd be better to install the
tar.=20
=20
I downloaded htdig-3.1.6 and began to follow the install instructions.
=20
Ran ./configure and when it finished I rec'd the following error:
=20
Configure: error: To compile ht://Dig, you will need a C++ library.  Try
installing libstdc++.   Isn't gcc a c++ compiler?
=20
...I checked and I already have libstdc++-3.2-7.i386.rpm installed=20
(Gcc-g77-3.2-7
Gcc-java-3.2-7
Gcc-c++-3.2-7
gcc-3.2-7
gcc-objc-3.2-7
gcc-gnat-3.2-7)
=20
I'm really a newbie at this, so any help is truly appreciated!  Oh, and
I tried a .make after just for the heck of it, and it returns that :
=20
Make: *** no targets specified and no makefile found. Stop.
=20
Thanks,
abbie

Re: [htdig-dev] Compile farm's Results and commit

[Gabriele B. <bar...@in...>]

>At the time, it didn't correctly compile C++ code. While it sounds like 
>that's fixed, I guess I'm also just trying to say that we have plenty of 
>OS X testers.

:-)   Got it  (sorry but my english is not that fine and I need further 
explanations! :-P  )

Ciao ciao
-Gabriele
--
Gabriele Bartolini - Web Programmer - ht://Dig & IWA Member - ht://Check 
maintainer
Current Location: Prato, Tuscany, Italia
an...@ti... | http://www.prato.linux.it/~gbartolini | ICQ#129221447

Re: [htdig-dev] Compile farm's Results and commit

[Geoff H. <ghu...@ws...>]

On Monday, February 3, 2003, at 12:48  PM, Gabriele Bartolini wrote:

> What kind of problem have you had specifically, Geoff?

At the time, it didn't correctly compile C++ code. While it sounds like 
that's fixed, I guess I'm also just trying to say that we have plenty 
of OS X testers.

-Geoff

Re: [htdig-dev] progress on splitting attrs...

[Gabriele B. <bar...@in...>]

Ciao Ted,

>I spent some time working on splitting the defaults.xml document into 
>individual documents and wanted to share my progress with the rest of you. 
>Unfortunately, it's a shorter path for me to learn XSLT than Perl so that 
>is currently how these files were produced.

Just one note. I think that the list on the left is too long and rather 
unaccessible (if I wanna go and reach the 'w' letter it is a problem); 
unfortunately, sorry, I can't find any other way of doing it except keeping 
a frameset. Other solutions require javascript and a self-expanding tree 
(personal opinion? I hate it!), or navigation bars and anchors.

Also, try and use the '/' for an empty tag with a space before, like:

<br />
instead of <br/> which is not always correctly recognised.

Another suggestion is not to use <b> and <i>, but replace them with 
'strong' and 'em' (which give more the idea of the structure instead of the 
presentation).

However I am sorry I can't help you right now, but I'll try to keep myself 
updated with your work.

Thanks a lot and Ciao!
-Gabriele
--
Gabriele Bartolini - Web Programmer - ht://Dig & IWA Member - ht://Check 
maintainer
Current Location: Prato, Tuscany, Italia
an...@ti... | http://www.prato.linux.it/~gbartolini | ICQ#129221447

[htdig-dev] htsearch raises internal server error when called by browser

[Robert K. <Rob...@so...>]

Hi there!


I have htdig 3.1.6. on apache.


Htsearch works fin, when called by commandline, but when called by
browser, the result is this:


Internal Server Error

The server encountered an internal error or misconfiguration and was
unable to complete your request.=20
Please contact the server administrator, web...@sw... and inform
them of the time the error occurred, and anything you might have done
that may have caused the error.=20
More information about this error may be available in the server error
log.=20
  _____ =20

Apache/1.3.26 Server at www.swlmx.de Port 80
=20
I have already seen this problem at the FAQ=92s, but none of the =
suggested
solution solved my problem.
Can you help?
=20
Regards,
_______________________________________________
Robert Kerschner
Softwarelandschaft Unternehmensberatung GmbH
Plenkerstra=DFe 17
A-3340 Waidhofen/Ybbs
Tel. +43 (7442) 54 124-38
Fax  +43 (7442) 54 124-10
E-Mail: rob...@so...
Internet: www.softwarelandschaft.at
<outbind://10-0000000044FFF1DDF84E6C4787D4A148F70C829144102000/www.softw
arelandschaft.at>=20
=20

[htdig-dev] progress on splitting attrs...

[Ted Stresen-R. <ted...@ma...>]

Hi,

I spent some time working on splitting the defaults.xml document into 
individual documents and wanted to share my progress with the rest of 
you. Unfortunately, it's a shorter path for me to learn XSLT than Perl 
so that is currently how these files were produced.

I had to make a very minor change to the defaults.dtd document Brian 
sent me (one of the attribute types was "numeric" and my xslt parser 
was complaining that that wasn't acceptable).

Here's where you can see the results from the first night of working 
with it. If feedback is positive, I'll keep sharing. If not, I'll just 
keep it to myself as a pet project ;-)

http://www.tedmasterweb.com/htdig/

Ted

Re: [htdig-dev] request for manage_attributes.pl (creates attributes based on defaults.xml)

[Brian W. <bw...@st...>]

Sorry - I have been a bit flat out and unable
to help.

I will make some time later today to dig out everything
I have and give a final opinion, but I will have to
back out after this. Sorry.

Regs

Brian


At 03:47 PM 03/02/2003, Ted Stresen-Reuter wrote:
>Brian,
>
>I'm still trying to get something productive done on splitting the 
>attributes into separate html files. Although the files you sent were 
>helpful, I think the one I'm really missing is the manage_attributes.pl 
>(or whichever one actually does the splitting using regular expressions). 
>If you have a moment, could you please send that one? It doesn't appear to 
>be a part of the most recent snapshots and would be very helpful.
>
>Also, I'm still new to XML so pardon what may be a stupid question, but, 
>rather than writing your own DTD, would it be possible for us to borrow 
>the DTD for XHTML and then modify that to meet our needs (adding our own 
>custom elements)? That way we could include all types of html in the 
>documentation (the examples, specifically).
>
>If the problem with including the HTML in the examples has to do with the 
>regular expressions that parse the defaults.xml file, I could take a look 
>at that... I'm pretty good with regex and might know a trick or two that 
>could fix that problem.
>
>Thanks and I look forward to hearing from you.
>
>Ted Stresen-Reuter
>
>
>
>-------------------------------------------------------
>This SF.NET email is sponsored by:
>SourceForge Enterprise Edition + IBM + LinuxWorld = Something 2 See!
>http://www.vasoftware.com
>_______________________________________________
>htdig-dev mailing list
>htd...@li...
>https://lists.sourceforge.net/lists/listinfo/htdig-dev

-------------------------
Brian White
Step Two Designs Pty Ltd
Knowledge Management Consultancy, SGML & XML
Phone: +612-93197901
Web:   http://www.steptwo.com.au/
Email: bw...@st...

Content Management Requirements Toolkit
112 CMS requirements, ready to cut-and-paste

Re: [htdig-dev] Re: Residual database problems

[Neal R. <ne...@ri...>]

On Tue, 4 Feb 2003, Lachlan Andrew wrote:

> On Saturday 01 February 2003 11:25, Neal Richter wrote:
>
> > If your error is repeatable, can you test it with
> > wordlist_compress_zlib & wordlist_compress dissabled and re-run
> > htpurge?  I'd like to see if the error still appears.
>
> The diagnostics only appear if compression is enabled.
>
> It takes over 24 hours (which becoems two days...) for the whole dig,
> and I'll try to find a smaller data set that causes the problem.  In
> the mean time, are there any other tests I can do to try to track it
> down?  For example, is it possible to decompress the file offline to
> compare it against the uncompressed one?

  Other than using htdump on both the zlib compressed WordDB and the
uncompressed WordDB I can't think of one.

  If there are differences in the dumps we could then see what effect it
has on searching.  If there are no differences I would lean towards a bug in
htpurge.

You may be able to hack htdump later to only uncompress the
BDB pages in question.

  Have you stepped through the htpurge code to see when/how this happens?

  I am skeptical that it's caused by the zlib compression code because:

1)  The changes to mp_cmpr are very minor to enable zlib compression and
involve no changes to the input data that would otherwise goto the mifluz
page compressor.

2)  If the compressed data was getting corrupted zlib should
report an error on the uncompress of that data-page.

http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/htdig/htdig/db/mp_cmpr.c.diff?r1=1.2&r2=1.3

Thanks.. I appreciate your effort on this.

One you get the htdumps tested for differences please report back!

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

[htdig-dev] Re: Residual database problems

[Lachlan A. <lh...@us...>]

On Saturday 01 February 2003 11:25, Neal Richter wrote:

> If your error is repeatable, can you test it with
> wordlist_compress_zlib & wordlist_compress dissabled and re-run
> htpurge?  I'd like to see if the error still appears.

The diagnostics only appear if compression is enabled.

It takes over 24 hours (which becoems two days...) for the whole dig,=20
and I'll try to find a smaller data set that causes the problem.  In=20
the mean time, are there any other tests I can do to try to track it=20
down?  For example, is it possible to decompress the file offline to=20
compare it against the uncompressed one?

Cheers,
Lachlan

Re: [htdig-dev] Compile farm's Results and commit

[Gabriele B. <bar...@in...>]

At 08.37 03/02/2003 -0600, Geoff Hutchison wrote:
>Just a note that I've had strange problems with the MacOS X 10.1 node on 
>the compile farm. Since there seem to be several of us testing on various 
>Macs, I think there would be plenty of noise if something broke on OS X.

What kind of problem have you had specifically, Geoff?

Ciao
-Gabriele


--
Gabriele Bartolini - Web Programmer - ht://Dig & IWA Member - ht://Check 
maintainer
Current Location: Prato, Tuscany, Italia
an...@ti... | http://www.prato.linux.it/~gbartolini | ICQ#129221447

Re: [htdig-dev] Release schedule for 3.2.0b5?

[Ted Stresen-R. <ted...@ma...>]

> defaults.xml either needs to shape up soon

Could you provide some details on what it needs, on what's missing?

Ted Stresen-Reuter

Re: [htdig-dev] Release schedule for 3.2.0b5?

[Geoff H. <ghu...@ws...>]

> Is there a list of tasks which *must* be completed before the release
> of 3.2.0b4/5?  If the "STATUS" file is that list, can I suggest that
> some things be classed as "not essential" (at least defaults.xml, and
> preferably most of it)?

The STATUS file is the list, though it's intended to be updated by 
other developers too! I've put everything in the STATUS list, though I 
agree that defaults.xml either needs to shape up soon or be left out of 
3.2.0b5.

I would definitely say that this zlib compression issue is a 
"showstopper" at the moment.

> Sorry for sounding impatient, and I know that everyone is busy, but it

No, good for you to bring this up and make some noise. :-)

Why don't you propose a list of what you think is essential for 3.2.0b5.

-Geoff

Re: [htdig-dev] Questions about Dictionary::hashCode

[Geoff H. <ghu...@ws...>]

>   I'm wondering if we couldn't add a String to the Dictionary class and
> use that instead of doing a malloc/strcpy everytime.. this function is
> called jillions of times.

That's probably a good idea.

>   I'm also curious as to why not use  Knuth's golden ratio hash 
> function,
> it's a well studied and known-good hash.
>
>   I can make the change and test it.. I'm just curious about the 
> rational
> for both the origional hash function and the change.

No one really cares *what* this hash function is. I certainly don't 
care if it's "well studied" if it works. The "original" hash you point 
to was one I derived in the 3.1 betas that seemed to show the fewest 
bad hashes on as many test sets I could find. One catch to this hash 
function is that it sees a lot of URLs, which means that "classic" hash 
functions don't always work well.

The change did seem to show performance improvements for me, though 
I'll admit I didn't count the number of bad hashes.

I've seen lots of "well it works" hash functions used in other 
situations, which is what I'd say about any replacements, Knuth or not.

-Geoff

Re: [htdig-dev] Compile farm's Results and commit

[Geoff H. <ghu...@ws...>]

>    Everything is ok on all Linux on all platforms (i686, Alpha, Sparc);
> MacOS x 10.1 still has that problem with shared libraries (as it was 
> before)
> whereas Solaris on a Sparc R220 doesn't go.

Just a note that I've had strange problems with the MacOS X 10.1 node 
on the compile farm. Since there seem to be several of us testing on 
various Macs, I think there would be plenty of noise if something broke 
on OS X.

-Geoff

Re: [htdig-dev] Re: not allowing getopt::std

[Geoff H. <ghu...@ws...>]

> Could someone who knows what  exact:  and  hidden:  mean please
> explain what they are for (and/or document them officially)?  I don't
> want to break anything while trying to fix the bug.

These are fuzzy algorithms essentially. You could have endings:blah. 
You're right that it's undocumented, and it should probably be taken 
out of the parser. (Nice idea to have per-word fuzzy possibilities, but 
maybe not the right way to do it.)

> On a related note, does anyone have any ideas for the syntax of "field
> restricted" searches?  I was thinking of something like "title:word"
> to search for "word" in the title field, or "heading:word" etc.  Was
> the plan to allow user-defined fields in meta-data to be searched?

Well, this is the "normal" syntax used by other sites. As for 
user-defined fields, it's certainly an ultimate goal but I think it's 
more important to:
a) get the new parser running.
b) get typical field-restricted searches like "title:word" going.

-Geoff

Re: [htdig-dev] request for manage_attributes.pl (creates attributes based on defaults.xml)

[Geoff H. <ghu...@ws...>]

> Also, I'm still new to XML so pardon what may be a stupid question, 
> but, rather than writing your own DTD, would it be possible for us to 
> borrow the DTD for XHTML and then modify that to meet our needs 
> (adding our own custom elements)? That way we could include all types 
> of html in the documentation (the examples, specifically).

But this is exactly "writing your own DTD." The point of an XML type is 
that it can be validated, so if you're using XHTML, it doesn't have 
"custom elements." What you're arguing about is the syntax of our 
custom DTD.

I'm not sure we really want all of XHTML in our DTD. It seems more like 
we want something pretty flexible, but maybe more like DocBook than 
XHTML.

-Geoff

[htdig-dev] request for manage_attributes.pl (creates attributes based on defaults.xml)

[Ted Stresen-R. <ted...@ma...>]

Brian,

I'm still trying to get something productive done on splitting the 
attributes into separate html files. Although the files you sent were 
helpful, I think the one I'm really missing is the manage_attributes.pl 
(or whichever one actually does the splitting using regular 
expressions). If you have a moment, could you please send that one? It 
doesn't appear to be a part of the most recent snapshots and would be 
very helpful.

Also, I'm still new to XML so pardon what may be a stupid question, 
but, rather than writing your own DTD, would it be possible for us to 
borrow the DTD for XHTML and then modify that to meet our needs (adding 
our own custom elements)? That way we could include all types of html 
in the documentation (the examples, specifically).

If the problem with including the HTML in the examples has to do with 
the regular expressions that parse the defaults.xml file, I could take 
a look at that... I'm pretty good with regex and might know a trick or 
two that could fix that problem.

Thanks and I look forward to hearing from you.

Ted Stresen-Reuter

[htdig-dev] Re: Efficiency Improvement for String.cc

[Neal R. <ne...@ri...>]

Hmm.  I've been using a nice stable 2.95.3 gcc (it's our production
compiler at the moment).

Interesting.  I'll have to test it on 3.1 ..I'd like to see what it does
with the wordkey WordDB inline functions.

FYI:

objdump --disassemble -S ./exe > exe.S

Will give you assembly with C source code lines interleaved (make sure to
use -g on compile)

> format will give more bang for your development buck.  It has the
> added advantages that it uses less disk space, and that it speeds up
> searching (assuming that is also disk bound).  However, I am really
> keen to get 3.2.0b5 out and I personally won't be working on
> optimisations until then.  Thoughts?

  Yep.. it's about time to start getting the WordDB more efficient.

  I'll post a few more patches (of small isolated changes) this week that
are well tested and we can decide whether to include them in 3.2.0b5.

  I've also got changes available to make it compile native win32..

Thanks.

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485