htdig-dev — Developer Discussion for the ht://Dig project

[htdig-dev] Initial results of swap to CLucene

[Neal R. <ne...@ri...>]

Hey all,
 	Here are the results from the first hacked version of adding 
CLucene.  What I did was comment out the 'put()' calls to the db.words.db 
& db.excerpts and add calls to CLucene at the Retreiver.cc level to insert 
the documents.

 	I'm still verifying that I'm passing in the same amount of 
information to both.

 	Searching TBA.

Thanks

------------------- compressed -----------

real    5m31.248s
user    1m14.760s
sys     0m6.860s

284k    webindex/db.docdb
92k     webindex/db.docs.index
5.3M    webindex/db.excerpts
60M     webindex/db.words.db
4.0k    webindex/luceneidx

Total Size: ~ 66MG

------------------- lucene -----------
real    2m31.635s
user    1m57.850s
sys     0m25.390s

284k    webindex/db.docdb
92k     webindex/db.docs.index
16k     webindex/db.excerpts
4.0k    webindex/db.words.db
22M     webindex/luceneidx

Total Size ~ 22 MG

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] htdig and libstdc++ requirement was: htdig on win32?)

[Jim <li...@yg...>]

On Wed, 2 Feb 2005, Gregory Hicks wrote:

> We have discussed the Win32 now, would you have any idea on where to
> look for the HTdig to be installed on my Solaris box?  I have the gcc
> compiler from SunFreeware.com that installs to /usr/local installed,
> but I also see the message "You might consider installing libstdc++..."
>
> gcc and libstdc++ are both reachable from $PATH and $LD_LIBRARY_PATH...
>
> Any thoughts?

Have you verified that a C++ compiler and library is really installed
and functional? Have you successfully compiled any other C++ code? If not,
you might want to try compiling a small test program. If you are not
familiar with C++, the following test might be helpful.

Create a file named cpptest.cpp with the following code.

#include <iostream>

int main(int, char* []) {
      std::cout << "Success" << std::endl;

      return 0;
}

Save the file and then execute the following command.

g++ -Wall cpptest.cpp -o cpptest

If you make it that far, execute the following.

./cpptest

You should see the word 'Success' output by the program.

If you can't make it this far, then something is missing or fundamentally
broken with respect to your environment or compiler install.


Jim

Re: [htdig-dev] HTML Parser and start of CLucene conversion

[Neal R. <ne...@ri...>]

>  As I see it, the primary reason that Nutch is somewhat unattractive to the 
> average HtDig user is that they must know how to configure Nutch to run as 
> Tomcat service, or know how to tweak the build system to build as a 
> standalone server.  Either is easy for a more novice user given their current 
> build system and 'How-To' docs.

   Ha!  The above 'Either' should be 'Neither'.  Those two options are 
definetly /not/ easy for the average user to make work.

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] HTML Parser and start of CLucene conversion

[Neal R. <ne...@ri...>]

> 4) According to the PyLucene author, converting Java Lucene to a
>   native library with gcj, then calling that library from a C
>   program is hopelessly hairy and not recommended. Too bad.
>
> 5) To my eye, Nutch does not look particularly rich in features or
>   configurability compared to HtDig.

   True, but it's known 200+ million document scalability can't be beat. 
And it's being supported by Yahoo Labs.

> 6) Word on the street is Xapian is the only competition to Lucene
>   in terms of scalability in terms of Free Software search
>   cores. Gmane uses Xapian against 20+ million documents.

   Xapian is GPL, Lucene/CLucene is LGPL.  Evidently the Xapian people 
didn't read the 4th paragraph of http://www.gnu.org/philosophy/why-not-lgpl.html

"Using the ordinary GPL is not advantageous for every library. There are
reasons that can make it better to use the Library GPL in certain cases. 
The most common case is when a free library's features are readily 
available for proprietary software through other alternative libraries. In that 
case, the library cannot give free software any particular advantage, so it is 
better to use the Library GPL for that library."

   Of course you (Jeff) and I dissagreed on this point a while back ;-)

   That said Xapian does look impressive.

> Anyway, I'm delighted to hear about this HtDig/Lucene experiment.
> Points #1, #2, and #3 suggest it may make sense to consider the idea
> of a pure Java HtDig which can be gcj compiled to native executables. From
> my perspective as a naive HtDig user I think that would rock, but
> there's probably lots of stuff I'm not thinking about. If anyone wants
> to try out the gcj/Lucene thing Doug Cutting's instructions [*] work
> fine provided you have gcj 3.4.x installed.

   If we really wanted a pure Java HtDig, I think we'd be better off 
throwing in with Nutch and adding the configurability of HtDig to it.

   As I see it, the primary reason that Nutch is somewhat unattractive to 
the average HtDig user is that they must know how to configure Nutch to run as 
Tomcat service, or know how to tweak the build system to build as a 
standalone server.  Either is easy for a more novice user given their 
current build system and 'How-To' docs.

   HtDig is still a forked CGI app, which means that our 
users don't have to worry about starting/monitoring a server daemon.  If 
we were to throw in with Nutch at some future date, it would be nice to 
make a simple option for Nutch to be built as a forked CGI app.

   I've looked at attempting to go the PyLucene route and compile Java with 
gjc and create the hairy wrapper libs for it.  It is ugly for many 
reasons.

   Going with CLucene at first has the advantage that we can get the 
code reorg done, and look at replacing the CLucene APIs with the 
equivalent Java-Lucene+Wrapper ones.. if it is even worth doing that.

   Thanks.

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] htdig and libstdc++ requirement was: htdig on win32?)

[Gregory H. <gh...@ca...>]

> Date: Tue, 1 Feb 2005 21:21:11 -0700 (MST)
> From: Neal Richter <ne...@ri...>
> To: Lar...@te...
> cc: Gregory Hicks <ghicks>, ghicks, htd...@li...
> Subject: Re: [htdig-dev] htdig and libstdc++ requirement was: htdig on win32?)
> 
> 
> You must start the setup.exe that comes with Cygwin.  That program will 
> download and install the proper version if gcc-g++.

Neal:

We have discussed the Win32 now, would you have any idea on where to
look for the HTdig to be installed on my Solaris box?  I have the gcc
compiler from SunFreeware.com that installs to /usr/local installed,
but I also see the message "You might consider installing libstdc++..."

gcc and libstdc++ are both reachable from $PATH and $LD_LIBRARY_PATH...

Any thoughts?

Regards,
Gregory Hicks

> 
> Cygwin is a bit difficult to configure sometimes if you aren't familiar 
> with it.
> 
> The package you downloaded may not be the appropriate version for the 
> version if cygwin you have installed.
> 
> Run setup.exe
> 
> Thanks
> 
> On Tue, 1 Feb 2005 Lar...@te... wrote:
> 
> > Hi Neal,
> >
> > My Root Directory is "C:\cygwin" and my Local Package Directory is
> > 
C:\downloads\http%3a%2f%2fwww.mirror.ac.uk%2fsites%2fsources.redhat.com%2fftp%2f
cygwin.
> >
> > I found the folder "gcc-g++-3.3.3-3.tar.bz2" in
> > 
C:\downloads\http%3a%2f%2fwww.mirror.ac.uk%2fsites%2fsources.redhat.com%2fftp%2f
cygwin\release\gcc
> > zipped.
> >
> > After unzip, I get the folder  "usr". Where is the right place to put
> > this folder  ?
> >
> > Thanks,
> > Larisa.
> >
> >
> >
> >
> >
> >
> > Neal Richter <ne...@ri...>
> > 01/31/2005 12:41 PM
> >
> >
> >        To:     Lar...@te...
> >        cc:     gh...@ma..., Gregory Hicks 
<gh...@ca...>,
> > htd...@li...
> >        Subject:        Re: [htdig-dev] htdig and libstdc++ requirement was: 
htdig on win32?)
> >
> >
> >
> > You need to install the gcc-g++ Cygwin package.
> >
> > Open the setup.exe again and select/install that package.
> >
> > Thanks.
> >
> > On Mon, 31 Jan 2005 Lar...@te... wrote:
> >
> >> Hi Neal,
> >>
> >> I am using version 3.1.6.
> >>
> >> I have followed "Idiot's Guide to installing ht://dig on Win32"  trying
> > to
> >> install htdig search engine on windows server 2003.
> >> During configuration step ( ./configure in  cygwin ) I've got an error:
> >>
> >> checking for ostream.h... no
> >> checking for iostream.h... no
> >> checking for fstream.h... no
> >> configure: error: To compile ht://Dig, you will need a C++ library. Try
> >> installing libstdc++.
> >>
> >> Please advise on how to fix this problem.
> >>
> >> Thanks,
> >> Larisa.
> >>
> >>
> >>
> >>
> >>
> >> Neal Richter <ne...@ri...>
> >> 01/30/2005 11:06 PM
> >>
> >>
> >>        To:     Gregory Hicks <gh...@ca...>
> >>        cc:     htd...@li...,
> > Lar...@te...,
> >> gh...@ma...
> >>        Subject:        Re: [htdig-dev] htdig and libstdc++ requirement
> > was: htdig on win32?)
> >>
> >>
> >>
> >> What version are you using?
> >>
> >> If you use the makefile.win32 and see the associated readme you can
> > build
> >> a native Win32 HtDig.
> >>
> >> Thanks
> >>
> >> On Sun, 30 Jan 2005, Gregory Hicks wrote:
> >>
> >>>
> >>>> From: Lar...@te...
> >>>> Date: Fri, 28 Jan 2005 19:15:57 -0500
> >>>>
> >>> [...snip...]
> >>>>
> >>>> I am getting the error saying that I need to install libstdc++.
> >> Currently,
> >>>> I don't any c++ compiler installed. There are lots of rpm packages and
> >> I
> >>>> don't seem to find a right one for cygwin.  Could someone please point
> >> ne
> >>>> to a right direction ?
> >>>
> >>> And I have Solaris 8/9 WITH gcc-3.4.1/3.4.2 installed WITH libstdc++
> >>> installed to /usr/local/lib which IS in $LD_LIBRARY_PATH and I get the
> >>> same error...
> >>>
> >>> I can see, in the configure script, where the error is generated but
> >>> cannot, for the life of me, figure out why it is occurring...
> >>>
> >>> Any thoughts?
> >>>
> >>>>
> >>>> Thanks in advance,
> >>>>
> >>>> Larisa
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> Gregory Hicks                           | Principal Systems Engineer
> >>> Cadence Design Systems                  | Direct:   408.576.3609
> >>> 555 River Oaks Pkwy M/S 6B1             | Fax:      408.894.3479
> >>> San Jose, CA 95134                      | Internet: gh...@ca...
> >>>
> >>> I am perfectly capable of learning from my mistakes.  I will surely
> >>> learn a great deal today.
> >>>
> >>> "A democracy is a sheep and two wolves deciding on what to have for
> >>> lunch.  Freedom is a well armed sheep contesting the results of the
> >>> decision." - Benjamin Franklin
> >>>
> >>> "The best we can hope for concerning the people at large is that they
> >>> be properly armed." --Alexander Hamilton
> >>>
> >>>
> >>>
> >>> -------------------------------------------------------
> >>> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> >>> Tool for open source databases. Create drag-&-drop reports. Save time
> >>> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> >>> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> >>> _______________________________________________
> >>> ht://Dig Developer mailing list:
> >>> htd...@li...
> >>> List information (subscribe/unsubscribe, etc.)
> >>> https://lists.sourceforge.net/lists/listinfo/htdig-dev
> >>>
> >>
> >>
> >
> >
> 
> -- 
> Neal Richter 
> Knowledgebase Developer
> RightNow Technologies, Inc.
> Customer Service for Every Web Site
> Office: 406-522-1485
> 

-------------------------------------------------------------------
Gregory Hicks                        | Principal Systems Engineer
Cadence Design Systems               | Direct:   408.576.3609
555 River Oaks Pkwy M/S 6B1          | Fax:      408.894.3400
San Jose, CA 95134                   | Internet: gh...@ca...

I am perfectly capable of learning from my mistakes.  I will surely
learn a great deal today.

"A democracy is a sheep and two wolves deciding on what to have for
lunch.  Freedom is a well armed sheep contesting the results of the
decision." - Benjamin Franklin

"The best we can hope for concerning the people at large is that they
be properly armed." --Alexander Hamilton

[htdig-dev] HTML Parser and start of CLucene conversion

[Jeff B. <je...@ja...>]

>        I'm starting a proof of concept conversion to use CLucene to
>replace the db.words.db ... it will still use BDB for the other db
>files.

Let me quickly report what I've learned as the Debian maintainer for
Java Lucene for the past 6 months.

0) Free software Java platforms like Kaffe can't build or run Java
   Lucene yet. So the toolchain is still linked to Sun's proprietary
   compiler/JVM. But it's fairly close and getting closer. Debian
   cares about this, may be less relevant for other platforms.

1) Java Lucene is a pretty quickly moving target. Point releases are
   every couple months, typically break a few things, and there is 
   a big revision in the works towards Lucene 2.0. Lucene just got
   approved as a top level Apache Foundation project and the momentum
   is huge.

2) Language ports like CLucene and C# Lucene seem to be lagging
   somewhat. The author of PyLucene told me his (quite convoluted)
   build process starts with Java Lucene instead of CLucene for
   exactly for this reason.

3) A Java application using Java Lucene can be compiled to native 
   code trivially using gcj, at least on Linux. Compilation is a 
   one liner and the resulting native binary is really fast. I'm
   thinking about shipping gcj compiled binaries for the Lucene
   demo programs with Debian.

4) According to the PyLucene author, converting Java Lucene to a 
   native library with gcj, then calling that library from a C
   program is hopelessly hairy and not recommended. Too bad.

5) To my eye, Nutch does not look particularly rich in features or 
   configurability compared to HtDig.

6) Word on the street is Xapian is the only competition to Lucene 
   in terms of scalability in terms of Free Software search 
   cores. Gmane uses Xapian against 20+ million documents.

Anyway, I'm delighted to hear about this HtDig/Lucene experiment.
Points #1, #2, and #3 suggest it may make sense to consider the idea
of a pure Java HtDig which can be gcj compiled to native executables. From
my perspective as a naive HtDig user I think that would rock, but
there's probably lots of stuff I'm not thinking about. If anyone wants
to try out the gcj/Lucene thing Doug Cutting's instructions [*] work
fine provided you have gcj 3.4.x installed.

[*] http://www.mail-archive.com/luc...@ja.../msg09089.html

Cheers,
Jeff

Re: [htdig-dev] htdig and libstdc++ requirement was: htdig on win32?)

[Neal R. <ne...@ri...>]

You must start the setup.exe that comes with Cygwin.  That program will 
download and install the proper version if gcc-g++.

Cygwin is a bit difficult to configure sometimes if you aren't familiar 
with it.

The package you downloaded may not be the appropriate version for the 
version if cygwin you have installed.

Run setup.exe

Thanks

On Tue, 1 Feb 2005 Lar...@te... wrote:

> Hi Neal,
>
> My Root Directory is "C:\cygwin" and my Local Package Directory is
> C:\downloads\http%3a%2f%2fwww.mirror.ac.uk%2fsites%2fsources.redhat.com%2fftp%2fcygwin.
>
> I found the folder "gcc-g++-3.3.3-3.tar.bz2" in
> C:\downloads\http%3a%2f%2fwww.mirror.ac.uk%2fsites%2fsources.redhat.com%2fftp%2fcygwin\release\gcc
> zipped.
>
> After unzip, I get the folder  "usr". Where is the right place to put
> this folder  ?
>
> Thanks,
> Larisa.
>
>
>
>
>
>
> Neal Richter <ne...@ri...>
> 01/31/2005 12:41 PM
>
>
>        To:     Lar...@te...
>        cc:     gh...@ma..., Gregory Hicks <gh...@ca...>,
> htd...@li...
>        Subject:        Re: [htdig-dev] htdig and libstdc++ requirement was: htdig on win32?)
>
>
>
> You need to install the gcc-g++ Cygwin package.
>
> Open the setup.exe again and select/install that package.
>
> Thanks.
>
> On Mon, 31 Jan 2005 Lar...@te... wrote:
>
>> Hi Neal,
>>
>> I am using version 3.1.6.
>>
>> I have followed "Idiot's Guide to installing ht://dig on Win32"  trying
> to
>> install htdig search engine on windows server 2003.
>> During configuration step ( ./configure in  cygwin ) I've got an error:
>>
>> checking for ostream.h... no
>> checking for iostream.h... no
>> checking for fstream.h... no
>> configure: error: To compile ht://Dig, you will need a C++ library. Try
>> installing libstdc++.
>>
>> Please advise on how to fix this problem.
>>
>> Thanks,
>> Larisa.
>>
>>
>>
>>
>>
>> Neal Richter <ne...@ri...>
>> 01/30/2005 11:06 PM
>>
>>
>>        To:     Gregory Hicks <gh...@ca...>
>>        cc:     htd...@li...,
> Lar...@te...,
>> gh...@ma...
>>        Subject:        Re: [htdig-dev] htdig and libstdc++ requirement
> was: htdig on win32?)
>>
>>
>>
>> What version are you using?
>>
>> If you use the makefile.win32 and see the associated readme you can
> build
>> a native Win32 HtDig.
>>
>> Thanks
>>
>> On Sun, 30 Jan 2005, Gregory Hicks wrote:
>>
>>>
>>>> From: Lar...@te...
>>>> Date: Fri, 28 Jan 2005 19:15:57 -0500
>>>>
>>> [...snip...]
>>>>
>>>> I am getting the error saying that I need to install libstdc++.
>> Currently,
>>>> I don't any c++ compiler installed. There are lots of rpm packages and
>> I
>>>> don't seem to find a right one for cygwin.  Could someone please point
>> ne
>>>> to a right direction ?
>>>
>>> And I have Solaris 8/9 WITH gcc-3.4.1/3.4.2 installed WITH libstdc++
>>> installed to /usr/local/lib which IS in $LD_LIBRARY_PATH and I get the
>>> same error...
>>>
>>> I can see, in the configure script, where the error is generated but
>>> cannot, for the life of me, figure out why it is occurring...
>>>
>>> Any thoughts?
>>>
>>>>
>>>> Thanks in advance,
>>>>
>>>> Larisa
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> Gregory Hicks                           | Principal Systems Engineer
>>> Cadence Design Systems                  | Direct:   408.576.3609
>>> 555 River Oaks Pkwy M/S 6B1             | Fax:      408.894.3479
>>> San Jose, CA 95134                      | Internet: gh...@ca...
>>>
>>> I am perfectly capable of learning from my mistakes.  I will surely
>>> learn a great deal today.
>>>
>>> "A democracy is a sheep and two wolves deciding on what to have for
>>> lunch.  Freedom is a well armed sheep contesting the results of the
>>> decision." - Benjamin Franklin
>>>
>>> "The best we can hope for concerning the people at large is that they
>>> be properly armed." --Alexander Hamilton
>>>
>>>
>>>
>>> -------------------------------------------------------
>>> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
>>> Tool for open source databases. Create drag-&-drop reports. Save time
>>> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
>>> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
>>> _______________________________________________
>>> ht://Dig Developer mailing list:
>>> htd...@li...
>>> List information (subscribe/unsubscribe, etc.)
>>> https://lists.sourceforge.net/lists/listinfo/htdig-dev
>>>
>>
>>
>
>

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

[htdig-dev] HTML Parser and start of CLucene conversion

[Neal R. <ne...@ri...>]

Hey all,

 	I'm starting a proof of concept conversion to use CLucene to 
replace the db.words.db	... it will still use BDB for the other db files.

 	The excerpts DB can likely be eliminated as well.

 	In my tests so far I can index 12,000 files in 4 minutes producing 
a 35MG Lucene Index.  The equivalent compressed standard index files total 
63MG and take 40 minutes to create.

 	Note that this is a loop inserting documents with a 
'simple_doc_insert' libhtdig api... not a spidering run.

 	In my code jockeying so far I notice that our HTML::parse() is 
fairly inefficient... it filters out the text between no_index_start & 
no_index_end tags, copies it then parses the copy.  Seems obvious 
that we can do this in the second while loop with state variables like 
we do for TITLE tags.  This would eliminate one linear scan through each 
document and needless memory usage.

 	Part of the reason for looking at how the HTML::parse is working 
is to make it unicode kosher by eliminating 'char' dependencies.

 	If anyone wants to help with the parse thing feel free.  All you 
need is to create a UTF8 HTML document with a multi-byte character and 
make sure the character is preserved in the debug output... we can worry 
about what happens to the word later.

 	For a refresher the reasons for strongly considering converting to 
CLucene are a follows:
 	* It's UTF8/Unicode capable
 	* Potentially much faster
 	* Supports field-based searching
 	* Allows us to ditch our nonstandard BDB 3.0.55
 	* Allows us to ditch thousands of lines of code in favor of an
 	  active project

 	If anyone seriously wants to help with this effort, I'd be willing 
to purchase a couple copies of Lucene in Action by Erik Hatcher and Otis 
Gospodnetic for people.  It covers the java version, however the C++ 
version attempts to be a strict translation to C++ in every way.  My copy 
is on the way.

 	At the moment I am concentrating on the spidering & indexing code 
to measure speed and size differences.  Searching will come after that.

 	Thanks.

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] htdig and libstdc++ requirement was: htdig on win32?)

[<Lar...@te...>]

Hi Neal,

My Root Directory is "C:\cygwin" and my Local Package Directory is 
C:\downloads\http%3a%2f%2fwww.mirror.ac.uk%2fsites%2fsources.redhat.com%2fftp%2fcygwin.

I found the folder "gcc-g++-3.3.3-3.tar.bz2" in 
C:\downloads\http%3a%2f%2fwww.mirror.ac.uk%2fsites%2fsources.redhat.com%2fftp%2fcygwin\release\gcc 
zipped.

 After unzip, I get the folder  "usr". Where is the right place to put 
this folder  ?

Thanks,
Larisa.






Neal Richter <ne...@ri...>
01/31/2005 12:41 PM

 
        To:     Lar...@te...
        cc:     gh...@ma..., Gregory Hicks <gh...@ca...>, 
htd...@li...
        Subject:        Re: [htdig-dev] htdig and libstdc++ requirement was: htdig on win32?)



You need to install the gcc-g++ Cygwin package.

Open the setup.exe again and select/install that package.

Thanks.

On Mon, 31 Jan 2005 Lar...@te... wrote:

> Hi Neal,
>
> I am using version 3.1.6.
>
> I have followed "Idiot's Guide to installing ht://dig on Win32"  trying 
to
> install htdig search engine on windows server 2003.
> During configuration step ( ./configure in  cygwin ) I've got an error:
>
> checking for ostream.h... no
> checking for iostream.h... no
> checking for fstream.h... no
> configure: error: To compile ht://Dig, you will need a C++ library. Try
> installing libstdc++.
>
> Please advise on how to fix this problem.
>
> Thanks,
> Larisa.
>
>
>
>
>
> Neal Richter <ne...@ri...>
> 01/30/2005 11:06 PM
>
>
>        To:     Gregory Hicks <gh...@ca...>
>        cc:     htd...@li..., 
Lar...@te...,
> gh...@ma...
>        Subject:        Re: [htdig-dev] htdig and libstdc++ requirement 
was: htdig on win32?)
>
>
>
> What version are you using?
>
> If you use the makefile.win32 and see the associated readme you can 
build
> a native Win32 HtDig.
>
> Thanks
>
> On Sun, 30 Jan 2005, Gregory Hicks wrote:
>
>>
>>> From: Lar...@te...
>>> Date: Fri, 28 Jan 2005 19:15:57 -0500
>>>
>> [...snip...]
>>>
>>> I am getting the error saying that I need to install libstdc++.
> Currently,
>>> I don't any c++ compiler installed. There are lots of rpm packages and
> I
>>> don't seem to find a right one for cygwin.  Could someone please point
> ne
>>> to a right direction ?
>>
>> And I have Solaris 8/9 WITH gcc-3.4.1/3.4.2 installed WITH libstdc++
>> installed to /usr/local/lib which IS in $LD_LIBRARY_PATH and I get the
>> same error...
>>
>> I can see, in the configure script, where the error is generated but
>> cannot, for the life of me, figure out why it is occurring...
>>
>> Any thoughts?
>>
>>>
>>> Thanks in advance,
>>>
>>> Larisa
>>>
>>>
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> Gregory Hicks                           | Principal Systems Engineer
>> Cadence Design Systems                  | Direct:   408.576.3609
>> 555 River Oaks Pkwy M/S 6B1             | Fax:      408.894.3479
>> San Jose, CA 95134                      | Internet: gh...@ca...
>>
>> I am perfectly capable of learning from my mistakes.  I will surely
>> learn a great deal today.
>>
>> "A democracy is a sheep and two wolves deciding on what to have for
>> lunch.  Freedom is a well armed sheep contesting the results of the
>> decision." - Benjamin Franklin
>>
>> "The best we can hope for concerning the people at large is that they
>> be properly armed." --Alexander Hamilton
>>
>>
>>
>> -------------------------------------------------------
>> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
>> Tool for open source databases. Create drag-&-drop reports. Save time
>> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
>> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
>> _______________________________________________
>> ht://Dig Developer mailing list:
>> htd...@li...
>> List information (subscribe/unsubscribe, etc.)
>> https://lists.sourceforge.net/lists/listinfo/htdig-dev
>>
>
>

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] htdig and libstdc++ requirement was: htdig on win32?)

[Neal R. <ne...@ri...>]

You need to install the gcc-g++ Cygwin package.

Open the setup.exe again and select/install that package.

Thanks.

On Mon, 31 Jan 2005 Lar...@te... wrote:

> Hi Neal,
>
> I am using version 3.1.6.
>
> I have followed "Idiot's Guide to installing ht://dig on Win32"  trying to
> install htdig search engine on windows server 2003.
> During configuration step ( ./configure in  cygwin ) I've got an error:
>
> checking for ostream.h... no
> checking for iostream.h... no
> checking for fstream.h... no
> configure: error: To compile ht://Dig, you will need a C++ library. Try
> installing libstdc++.
>
> Please advise on how to fix this problem.
>
> Thanks,
> Larisa.
>
>
>
>
>
> Neal Richter <ne...@ri...>
> 01/30/2005 11:06 PM
>
>
>        To:     Gregory Hicks <gh...@ca...>
>        cc:     htd...@li..., Lar...@te...,
> gh...@ma...
>        Subject:        Re: [htdig-dev] htdig and libstdc++ requirement was: htdig on win32?)
>
>
>
> What version are you using?
>
> If you use the makefile.win32 and see the associated readme you can build
> a native Win32 HtDig.
>
> Thanks
>
> On Sun, 30 Jan 2005, Gregory Hicks wrote:
>
>>
>>> From: Lar...@te...
>>> Date: Fri, 28 Jan 2005 19:15:57 -0500
>>>
>> [...snip...]
>>>
>>> I am getting the error saying that I need to install libstdc++.
> Currently,
>>> I don't any c++ compiler installed. There are lots of rpm packages and
> I
>>> don't seem to find a right one for cygwin.  Could someone please point
> ne
>>> to a right direction ?
>>
>> And I have Solaris 8/9 WITH gcc-3.4.1/3.4.2 installed WITH libstdc++
>> installed to /usr/local/lib which IS in $LD_LIBRARY_PATH and I get the
>> same error...
>>
>> I can see, in the configure script, where the error is generated but
>> cannot, for the life of me, figure out why it is occurring...
>>
>> Any thoughts?
>>
>>>
>>> Thanks in advance,
>>>
>>> Larisa
>>>
>>>
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> Gregory Hicks                           | Principal Systems Engineer
>> Cadence Design Systems                  | Direct:   408.576.3609
>> 555 River Oaks Pkwy M/S 6B1             | Fax:      408.894.3479
>> San Jose, CA 95134                      | Internet: gh...@ca...
>>
>> I am perfectly capable of learning from my mistakes.  I will surely
>> learn a great deal today.
>>
>> "A democracy is a sheep and two wolves deciding on what to have for
>> lunch.  Freedom is a well armed sheep contesting the results of the
>> decision." - Benjamin Franklin
>>
>> "The best we can hope for concerning the people at large is that they
>> be properly armed." --Alexander Hamilton
>>
>>
>>
>> -------------------------------------------------------
>> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
>> Tool for open source databases. Create drag-&-drop reports. Save time
>> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
>> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
>> _______________________________________________
>> ht://Dig Developer mailing list:
>> htd...@li...
>> List information (subscribe/unsubscribe, etc.)
>> https://lists.sourceforge.net/lists/listinfo/htdig-dev
>>
>
>

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] htdig and libstdc++ requirement was: htdig on win32?)

[<Lar...@te...>]

Hi Neal,

I am using version 3.1.6. 

I have followed "Idiot's Guide to installing ht://dig on Win32"  trying to 
install htdig search engine on windows server 2003.
During configuration step ( ./configure in  cygwin ) I've got an error:

checking for ostream.h... no
checking for iostream.h... no
checking for fstream.h... no
configure: error: To compile ht://Dig, you will need a C++ library. Try 
installing libstdc++.

Please advise on how to fix this problem.

Thanks,
Larisa. 





Neal Richter <ne...@ri...>
01/30/2005 11:06 PM

 
        To:     Gregory Hicks <gh...@ca...>
        cc:     htd...@li..., Lar...@te..., 
gh...@ma...
        Subject:        Re: [htdig-dev] htdig and libstdc++ requirement was: htdig on win32?)



What version are you using?

If you use the makefile.win32 and see the associated readme you can build 
a native Win32 HtDig.

Thanks

On Sun, 30 Jan 2005, Gregory Hicks wrote:

>
>> From: Lar...@te...
>> Date: Fri, 28 Jan 2005 19:15:57 -0500
>>
> [...snip...]
>>
>> I am getting the error saying that I need to install libstdc++. 
Currently,
>> I don't any c++ compiler installed. There are lots of rpm packages and 
I
>> don't seem to find a right one for cygwin.  Could someone please point 
ne
>> to a right direction ?
>
> And I have Solaris 8/9 WITH gcc-3.4.1/3.4.2 installed WITH libstdc++
> installed to /usr/local/lib which IS in $LD_LIBRARY_PATH and I get the
> same error...
>
> I can see, in the configure script, where the error is generated but
> cannot, for the life of me, figure out why it is occurring...
>
> Any thoughts?
>
>>
>> Thanks in advance,
>>
>> Larisa
>>
>>
>>
>>
>>
>
> ---------------------------------------------------------------------
> Gregory Hicks                           | Principal Systems Engineer
> Cadence Design Systems                  | Direct:   408.576.3609
> 555 River Oaks Pkwy M/S 6B1             | Fax:      408.894.3479
> San Jose, CA 95134                      | Internet: gh...@ca...
>
> I am perfectly capable of learning from my mistakes.  I will surely
> learn a great deal today.
>
> "A democracy is a sheep and two wolves deciding on what to have for
> lunch.  Freedom is a well armed sheep contesting the results of the
> decision." - Benjamin Franklin
>
> "The best we can hope for concerning the people at large is that they
> be properly armed." --Alexander Hamilton
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> ht://Dig Developer mailing list:
> htd...@li...
> List information (subscribe/unsubscribe, etc.)
> https://lists.sourceforge.net/lists/listinfo/htdig-dev
>

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] htdig and libstdc++ requirement was: htdig on win32?)

[Gregory H. <gh...@ca...>]

> From: Neal Richter <ne...@ri...>
> To: Gregory Hicks <ghicks>
> Cc: htd...@li..., Lar...@te..., 
ghicks
> Subject: Re: [htdig-dev] htdig and libstdc++ requirement was: htdig on 
win32?)
> Date: Sun, 30 Jan 2005 21:06:01 -0700 (MST)
> 
> 
> What version are you using?

htdig-3.1.6 and htdig-3.2.0b6 trying to configure on Solaris 8 and 
Solaris 9.  (Larisa is the one with the Win32 problem.)  I just have the 
same problem on Solaris.


> 
> If you use the makefile.win32 and see the associated readme you can 
build 
> a native Win32 HtDig.
> 
> Thanks
> 
> On Sun, 30 Jan 2005, Gregory Hicks wrote:
> 
> >
> >> From: Lar...@te...
> >> Date: Fri, 28 Jan 2005 19:15:57 -0500
> >>
> > [...snip...]
> >>
> >> I am getting the error saying that I need to install libstdc++. 
Currently,
> >> I don't any c++ compiler installed. There are lots of rpm packages 
and I
> >> don't seem to find a right one for cygwin.  Could someone please 
point ne
> >> to a right direction ?
> >
> > And I have Solaris 8/9 WITH gcc-3.4.1/3.4.2 installed WITH libstdc++
> > installed to /usr/local/lib which IS in $LD_LIBRARY_PATH and I get 
the
> > same error...
> >
> > I can see, in the configure script, where the error is generated but
> > cannot, for the life of me, figure out why it is occurring...
> >
> > Any thoughts?
> >
> >>
> >> Thanks in advance,
> >>
> >> Larisa
> >>
> >>
> >>
> >>
> >>
> >
> > 
---------------------------------------------------------------------
> > Gregory Hicks                           | Principal Systems Engineer
> > Cadence Design Systems                  | Direct:   408.576.3609
> > 555 River Oaks Pkwy M/S 6B1             | Fax:      408.894.3479
> > San Jose, CA 95134                      | Internet: 
gh...@ca...
> >
> > I am perfectly capable of learning from my mistakes.  I will surely
> > learn a great deal today.
> >
> > "A democracy is a sheep and two wolves deciding on what to have for
> > lunch.  Freedom is a well armed sheep contesting the results of the
> > decision." - Benjamin Franklin
> >
> > "The best we can hope for concerning the people at large is that 
they
> > be properly armed." --Alexander Hamilton
> >
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by: IntelliVIEW -- Interactive 
Reporting
> > Tool for open source databases. Create drag-&-drop reports. Save 
time
> > by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, 
etc.
> > Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> > _______________________________________________
> > ht://Dig Developer mailing list:
> > htd...@li...
> > List information (subscribe/unsubscribe, etc.)
> > https://lists.sourceforge.net/lists/listinfo/htdig-dev
> >
> 
> -- 
> Neal Richter 
> Knowledgebase Developer
> RightNow Technologies, Inc.
> Customer Service for Every Web Site
> Office: 406-522-1485
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive 
Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> ht://Dig Developer mailing list:
> htd...@li...
> List information (subscribe/unsubscribe, etc.)
> https://lists.sourceforge.net/lists/listinfo/htdig-dev

---------------------------------------------------------------------
Gregory Hicks                           | Principal Systems Engineer
Cadence Design Systems                  | Direct:   408.576.3609
555 River Oaks Pkwy M/S 6B1             | Fax:      408.894.3479
San Jose, CA 95134                      | Internet: gh...@ca...

I am perfectly capable of learning from my mistakes.  I will surely
learn a great deal today.

"A democracy is a sheep and two wolves deciding on what to have for
lunch.  Freedom is a well armed sheep contesting the results of the
decision." - Benjamin Franklin

"The best we can hope for concerning the people at large is that they
be properly armed." --Alexander Hamilton

Re: [htdig-dev] htdig and libstdc++ requirement was: htdig on win32?)

[Neal R. <ne...@ri...>]

What version are you using?

If you use the makefile.win32 and see the associated readme you can build 
a native Win32 HtDig.

Thanks

On Sun, 30 Jan 2005, Gregory Hicks wrote:

>
>> From: Lar...@te...
>> Date: Fri, 28 Jan 2005 19:15:57 -0500
>>
> [...snip...]
>>
>> I am getting the error saying that I need to install libstdc++. Currently,
>> I don't any c++ compiler installed. There are lots of rpm packages and I
>> don't seem to find a right one for cygwin.  Could someone please point ne
>> to a right direction ?
>
> And I have Solaris 8/9 WITH gcc-3.4.1/3.4.2 installed WITH libstdc++
> installed to /usr/local/lib which IS in $LD_LIBRARY_PATH and I get the
> same error...
>
> I can see, in the configure script, where the error is generated but
> cannot, for the life of me, figure out why it is occurring...
>
> Any thoughts?
>
>>
>> Thanks in advance,
>>
>> Larisa
>>
>>
>>
>>
>>
>
> ---------------------------------------------------------------------
> Gregory Hicks                           | Principal Systems Engineer
> Cadence Design Systems                  | Direct:   408.576.3609
> 555 River Oaks Pkwy M/S 6B1             | Fax:      408.894.3479
> San Jose, CA 95134                      | Internet: gh...@ca...
>
> I am perfectly capable of learning from my mistakes.  I will surely
> learn a great deal today.
>
> "A democracy is a sheep and two wolves deciding on what to have for
> lunch.  Freedom is a well armed sheep contesting the results of the
> decision." - Benjamin Franklin
>
> "The best we can hope for concerning the people at large is that they
> be properly armed." --Alexander Hamilton
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> ht://Dig Developer mailing list:
> htd...@li...
> List information (subscribe/unsubscribe, etc.)
> https://lists.sourceforge.net/lists/listinfo/htdig-dev
>

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] htdig and libstdc++ requirement was: htdig on win32?)

[Gregory H. <gh...@ca...>]

> From: Lar...@te...
> Date: Fri, 28 Jan 2005 19:15:57 -0500
> 
[...snip...]
> 
> I am getting the error saying that I need to install libstdc++. Currently, 
> I don't any c++ compiler installed. There are lots of rpm packages and I 
> don't seem to find a right one for cygwin.  Could someone please point ne 
> to a right direction ?

And I have Solaris 8/9 WITH gcc-3.4.1/3.4.2 installed WITH libstdc++
installed to /usr/local/lib which IS in $LD_LIBRARY_PATH and I get the
same error...

I can see, in the configure script, where the error is generated but
cannot, for the life of me, figure out why it is occurring...

Any thoughts?

> 
> Thanks in advance,
> 
> Larisa
> 
> 
> 
>  
> 

---------------------------------------------------------------------
Gregory Hicks                           | Principal Systems Engineer
Cadence Design Systems                  | Direct:   408.576.3609
555 River Oaks Pkwy M/S 6B1             | Fax:      408.894.3479
San Jose, CA 95134                      | Internet: gh...@ca...

I am perfectly capable of learning from my mistakes.  I will surely
learn a great deal today.

"A democracy is a sheep and two wolves deciding on what to have for
lunch.  Freedom is a well armed sheep contesting the results of the
decision." - Benjamin Franklin

"The best we can hope for concerning the people at large is that they
be properly armed." --Alexander Hamilton

[htdig-dev] Mirror

[Blair L. <ht...@ze...>]

Hi,

I was trying to set up a mirror for htdig, and the CVS times out at my
co-lo. If I do it from my home dsl account all is fine.

Is there some sort of unblocking thing I need to do with sourceforge for
it to work?

Our IP = 204.209.44.221

TTYL,
Blair.
-- 

[htdig-dev] htdig on win32 ?

[<Lar...@te...>]

Hello htdig Funs,

Oh - oh, I am trying to install htdig search engine on windows server 
2003. 
During configuration step ( I am following the guide donated by Jim 
Kerslake, please find at 
http://www.htdig.org/files/contrib/guides/Installing_on_Win32.html ),

I am getting the error saying that I need to install libstdc++. Currently, 
I don't any c++ compiler installed. There are lots of rpm packages and I 
don't seem to find a right one for cygwin.  Could someone please point ne 
to a right direction ?

Thanks in advance,

Larisa



 

[htdig-dev] ht dig users list

[Eric C. <ec...@ea...>]

Hey Y'all,

Not that it's all that critical, I know, but I was wondering if anyone 
were updating the list of websites using htdig on the htdig site any 
time soon. Not that I have it running quite yet (just working on getting 
a second site indexed off the same installation). I'd be really grateful 
if it would be possible to get a link. to Ananda Apothecary 
http://www.anandaapothecary.com

Could I do anything in return? A check, maybe?

Thanks,
Eric
ps. thanls for your work, the engine is great.

Re: [htdig-dev] htDig and Vignette??

[Neal R. <ne...@ri...>]

Looks like you are getting closer...

> ......This tells me that htdig is trying to retrieve the correct document
> (index.jsp)....I'm not sure what the deal is with the persistent
> connection error??.......
> ......Then you can see that there is a redirect to the LOGIN page......

   Try putting ' persistent_connections: false' in your config.

> Thereafter there are quite a few lines of similar content since each URL
> I'm trying to dig gets redirected in the same manner as above.   At the
> end, the login page is actually read and indexed....but that's the only
> page.

Thanks

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] htDig and Vignette??

[Bruce D. <bd...@qa...>]

I added the cookies_input_file and disable_cookies elements to the config 
file for this site as follows:

# enable cookies support
disable_cookies:        false
cookies_input_file:     /apps/www/htdig/conf/cookies.txt

I then added a single line to the cookies.txt file as follows:

testintranetportal      TRUE    /portal FALSE   0       epicentric 
d97692f64e7b540b0f504e238790307a

I got this info after clearing my cookies and then logging in to the site. 
   There is also a session id cookie (JSESSIONID) which gets set, but it 
is only persistent for the session.    The "epicentric" cookie seems to be 
the one which contains persistent login information and the value in the 
line above is what was stored in my browser for my login.    However, this 
still does not get me past the login redirect....    Here's some info from 
the htdig verbose output:

rundig: Start time: Wed Jan 26 14:13:43 EST 2005
ht://dig Start Time: Wed Jan 26 14:13:43 2005
Importing Cookies input file /apps/www/htdig/conf/cookies.txt

Cookies that have been correctly imported from: 
/apps/www/htdig/conf/cookies.txt
  1. epicentric: d97692f64e7b540b0f504e238790307a (Domain: 
testintranetportal)

......This tells me the htdig cookies file is being read correctly.......

Try to get through to host testintranetportal (port 80)
    2 - Open of the connection ok
        Assigning the server (testintranetportal) to the TCP connection
        Assigned the remote host testintranetportal
        Assigning the port (80) to the TCP connection
        Assigned the port 80
        Connecting via TCP to (testintranetportal:80)
New connection open successfully
Header line: HTTP/1.1 302 Found
Header line: Server: Microsoft-IIS/5.0
Header line: Date: Wed, 26 Jan 2005 19:09:14 GMT
Header line: X-Powered-By: ASP.NET
Discarded header line: X-Powered-By: ASP.NET
Header line: Connection: close
Header line: Server: WebSphere Application Server/5.0
Header line: Set-Cookie: JSESSIONID=00003WHZUFZ25GHUYKKT0E5Y5LI:-1;Path=/

........This tells me that a TCP connection can be made to the 
server...and in fact the server sets JSESSIONID.......

Retrieving document /portal/site/inside-test/index.jsp on host: 
testintranetportal:80
Http version      : HTTP/1.1
Server            : HTTP/1.1
Status Code       : 302
Reason            : Found
Access Time       : Wed, 26 Jan 2005 19:09:14 EST
Modification Time : Wed, 26 Jan 2005 19:13:44 EST
Content-type      : text/html; charset=UTF-8
Content-Language : en-US
Connection        : close
Persistent connection: not accepted
Body not retrieved
    2 - Connection closed (No persistent connection)
Request time: 0 secs
Contents:

Content Type: text/html; charset=UTF-8
Content Length: -1
Modification Time: 2005-01-26 19:13:44 EST
 redirect
redirect: 
http://testintranetportal/portal/site/inside-test/index.jsp?epi-content=LOGIN
resolving 
'http://testintranetportal/portal/site/inside-test/index.jsp?epi-content=LOGIN'
   pushing 
http://testintranetportal/portal/site/inside-test/index.jsp?epi-content=LOGIN

......This tells me that htdig is trying to retrieve the correct document 
(index.jsp)....I'm not sure what the deal is with the persistent 
connection error??.......
......Then you can see that there is a redirect to the LOGIN page......

Thereafter there are quite a few lines of similar content since each URL 
I'm trying to dig gets redirected in the same manner as above.   At the 
end, the login page is actually read and indexed....but that's the only 
page.

Perhaps this info provides some detail that might be helpful in further 
diagnosing the problem.    I still haven't heard back from my colleague 
who is supposed to be contacting Vignette.   Is there any way to tell 
where/how/if htdig is attempting to set the cookie or pass it to the 
host/server?   I didn't see anything in the log file about that.

Thanks for your help.

Bruce






Neal Richter <ne...@ri...> 
01/24/2005 08:51 PM

To
Bruce DeYoung <bd...@qa...>
cc
htd...@li...
Subject
Re: [htdig-dev] htDig and Vignette??






On Mon, 24 Jan 2005, Bruce DeYoung wrote:

> OK.   Here's the URL at the login page:
>
> 
http://testintranetportal/portal/site/insideQAD/index.jsp?epi-content=LOGIN

>
> Then, after logging in, here are a couple of URL's of content pages:
>
> 
http://testintranetportal/portal/site/insideQAD/index.jsp?front_door=true&epi_menuItemID=17b4d03e0ebb0d03c0bc8ed22890307a&epi_menuID=557c013f162725a5c2046e478790307a&epi_baseMenuID=557c013f162725a5c2046e478790307a

>
> and
>
> 
http://testintranetportal/portal/site/insideQAD/index.jsp?front_door=true&epi_menuItemID=8853e4e036d9d40ecfd048922890307a&epi_menuID=b65bac56c452abf6aeda32202890307a&epi_baseMenuID=557c013f162725a5c2046e478790307a


    ha ha.. this is almost as opaque as it gets.  I've solved your issue 
before via the cookies file and rewriting the URL.. but those are not very 

informative.

   For those URLs you need information on how they tell the CGI what to do 

and 'is there a sessionid buried in there'?

   Can you get this from Vignette or the people that connected Vignette 
to whatever CGI/ASP/JSP software that produces the website?

   The main question you want to answer is this:

Do I need to do anything to those URLs so that after a user clicks on a 
search result they are able to view that page without screwing up my 
reporting?

   A simple test would be to log into the site with one browser and 'cut' 
a 
link URL.  Then open up a second Browser (different one, not two IE 
windows) like Firefox (with the cookies all cleared) and paste the URL 
into it.  What happens?

   Will the search box be 'behind' the login screen?  ie the users will 
already be loged-in before they do their first search.

   Anyway, things to think about.

   Thanks

> Thanks again,
>
> Bruce
>
>
>
>
> Neal Richter <ne...@ri...>
> 01/24/2005 12:59 PM
>
> To
> Bruce DeYoung <bd...@qa...>
> cc
> htd...@li...
> Subject
> Re: [htdig-dev] htDig and Vignette??
>
>
>
>
>
>
> On Sun, 23 Jan 2005, Bruce DeYoung wrote:
>
>> Thanks Neal for the reply.   Unfortunately, I cannot provide a link to
> the
>> site since it is an intranet site only....at this time.
>
>   Post it anyway so I can take a look at it's structure.  Post the login
> URL then the first URL you see after a sucessful login.
>
>> My suspicion about this is that Vignette security is handled 
differently
>> than, say, standard Apache security.    Using the -u option with htdig
> and
>> supplying an authenticated user for our Apache-based sites works fine.
> I'm
>> not sure how Vignette authentication works, but I do know that when you
>> attempt to access the site, if your login cookie is not set, it will
>> rediret to a login page and request authentication information.
>
>   Open your cookies file in the browser and clear anything associated
> with
> this website, then relogin into the webiste and check the cookies.
>
>> I've asked our Vignette developer to request some assistance from
> Vignette
>> support as well.
>>
>> When you say "make sure cookie support is enabled", are you referring 
to
>> something in Vignette or in htDig?
>
>   I assume you are using HtDig 3.2B6
>
>   Look at the  cookies_input_file & disable_cookies settings in HtDig.
> The disable_cookies is 'true' be default.
>
>   My gut feeling is that it's setting a cookie.  You can take the
> contents of the cookie that the browser stores and load it in to the
> HtDig indexer via the cookies_input_file.
>
>   It may also be that the software checks the 'user_agent' string
> supplied
> by the browser/indexer and may disallow access if you aren't running a
> certain version of browser.
>
>   You can fake this buy setting the user_agent in HtDig to be the string
> supplied by IE.  Get it from your apache server weblogs.
>
>   I've seen both of these problems and worked around them this way.
>
>> And, I understand what you're saying about using the rewrite 
rules...and
> I
>> think you're right about that one.   So, once I'm able to dig the site,
> I
>> will look at the URL references and create a url_rewrite rule to remove
>> the session information.
>
>   Thanks.
>
>

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] Compression related performance

[David B. <dav...@bo...>]

>   I think you are right.. doing a dump & reinsert would be a nice way to 
> optimize for size.
>
>   However in my tests both of them had about the same fill factor.. so the 
> compression db files will still be smaller.

Well that means that it didn't optimize for size then ;)
Perhaps the theory about inserting in key order yielding 100% FF is 
incorrect.

>   The more odd result is that Jim has about a 5x speed improvement when he 
> disables compression.  I also see a speedup.

Obviously you'd expect that in the case that the task is CPU-bound.

> Joe gets a 14% slowdown.

If Joe's disks are very slow compared to yours, that might
explain the difference (because the compressed output file
is smaller and hence can be written to disk quicker). Another
possibility is differences in filesystem : if in one case the data
is not actually being flushed to disk during the measured elapsed test time,
and in the other case it is, that could make a big difference, and again the
difference would depend on the size of the file.

Just some random ideas...

I'd try running tools like top, vmstat and iostat during the
test execution and see if they reveal anything interesting.

Re: [htdig-dev] Compression related performance

[Neal R. <ne...@ri...>]

On Tue, 25 Jan 2005, David Boreham wrote:

> Observation: almost all of the difference in file size between the compressed 
> and non-compressed cases
> can be accounted for by the free bytes in leaf pages.
> Therefore, if the file were to be re-built inserting all the keys
> in key sort order, the resulting file (sans compression)
> should be quite close in size to the compressed one.
> I believe it should be possible to test this theory
> using db_dump and db_load back-to-back.


   I think you are right.. doing a dump & reinsert would be a nice way to 
optimize for size.

   However in my tests both of them had about the same fill factor.. so the 
compression db files will still be smaller.

   The more odd result is that Jim has about a 5x speed improvement when he 
disables compression.  I also see a speedup.

Joe gets a 14% slowdown.

   They both seem to have similar sytems and (Dual Xenons) a crawling local 
site (no network delay)

Jim:
These numbers are for a local dig on a dual 2.8 GHz Xeon box with RAID 5 
and a couple GB of RAM.

Joe:
My system is similar to yours, dual Xeon 2.4 GHz box RAID 5 10K RPM SCSI
drives, and 2.0 GB DDR RAM

FYI: David Boreham is a Berkeley DB Guru.

Thanks.

Neal Richter

Re: [htdig-dev] Compression related performance

[David B. <dav...@bo...>]

Observation: almost all of the difference in file size 
between the compressed and non-compressed cases
can be accounted for by the free bytes in leaf pages.
Therefore, if the file were to be re-built inserting all the keys
in key sort order, the resulting file (sans compression)
should be quite close in size to the compressed one.
I believe it should be possible to test this theory
using db_dump and db_load back-to-back.

>   436043776 Jan 15 00:00 db.words.db
...
>   113773568 Jan 14 09:36 db.words.db
...
>188M    Number of bytes free in tree leaf pages (57% ff).

Re: [htdig-dev] Compression related performance

[Jim <li...@yg...>]

On Tue, 18 Jan 2005, Neal Richter wrote:

>>> Thanks for the numbers.  Pretty different from Joe's.
>> 
>> Yes they are; can you explain it?
>
>  #1)  What is the size of  wordlist_cache_size?

I use the default.

>  #2)  wordlist_page_size?

Again the default for 3.2.0b6.

>  #3)  Are either of you using a journaling filesystem?

Yes. I use ext3.

>  #4)  Does one of you have somekind of high performance disk-drive?

Nothing extraordinary. The drive system consists of three 10k SCSI's with
hardware RAID 5.


Jim

Re: [htdig-dev] Compression related performance

[Jim <li...@yg...>]

On Sat, 15 Jan 2005, Neal Richter wrote:

> Jim,
> Thanks for the numbers.  Pretty different from Joe's.
>
> Question #1)
>
>   Could you download a fresh copy of BDB 3.0.55 from
> http://www.sleepycat.com/download/index.shtml
>
> It's easy to build, I am most interested in the results when you run db_stat 
> on the uncompressed files and report back the fill-factor.  This is a measure 
> of wasted space in the BDB pages.  This won't work on the compressed index.

#> db_stat -d db.words.db
53162   Btree magic number.
7       Btree version number.
Flags:
2       Minimum keys per-page.
8192    Underlying database page size.
4       Number of levels in the tree.
9069906 Number of keys in the tree.
282     Number of tree internal pages.
703316  Number of bytes free in tree internal pages (70% ff).
52945   Number of tree leaf pages.
188M    Number of bytes free in tree leaf pages (57% ff).
0       Number of tree duplicate pages.
0       Number of bytes free in tree duplicate pages (0% ff).
0       Number of tree overflow pages.
0       Number of bytes free in tree overflow pages (0% ff).
0       Number of pages on the free list.

> Question #2)
> What page_size are you using?

8192.

> Question #3)
> Do you ever see any errors when using the compressed index?

No. I have never seen it myself with 3.2.0b6. In addition to this
database, I maintain several others that are much larger and always built
with compression enabled.

Jim

[htdig-dev] htdig high speed US/California mirror at mirrormonster.com

[mirrors <mi...@mi...>]

We at mirrormonster.com would like to contribute from our bandwidth and 
resources to the opensource community, We are already mirroring large 
opensource projects such as Apache, Mozilla and plenty more are in the 
works.

The htdig mirror is available at http://www.mirrormonster.com/www.htdig.org/

  Mike