htdig-dev — Developer Discussion for the ht://Dig project

Re: [htdig-dev] htDig and Vignette??

[Neal R. <ne...@ri...>]

On Mon, 24 Jan 2005, Bruce DeYoung wrote:

> OK.   Here's the URL at the login page:
>
> http://testintranetportal/portal/site/insideQAD/index.jsp?epi-content=LOGIN
>
> Then, after logging in, here are a couple of URL's of content pages:
>
> http://testintranetportal/portal/site/insideQAD/index.jsp?front_door=true&epi_menuItemID=17b4d03e0ebb0d03c0bc8ed22890307a&epi_menuID=557c013f162725a5c2046e478790307a&epi_baseMenuID=557c013f162725a5c2046e478790307a
>
> and
>
> http://testintranetportal/portal/site/insideQAD/index.jsp?front_door=true&epi_menuItemID=8853e4e036d9d40ecfd048922890307a&epi_menuID=b65bac56c452abf6aeda32202890307a&epi_baseMenuID=557c013f162725a5c2046e478790307a

    ha ha.. this is almost as opaque as it gets.  I've solved your issue 
before via the cookies file and rewriting the URL.. but those are not very 
informative.

   For those URLs you need information on how they tell the CGI what to do 
and 'is there a sessionid buried in there'?

   Can you get this from Vignette or the people that connected Vignette 
to whatever CGI/ASP/JSP software that produces the website?

   The main question you want to answer is this:

Do I need to do anything to those URLs so that after a user clicks on a 
search result they are able to view that page without screwing up my 
reporting?

   A simple test would be to log into the site with one browser and 'cut' a 
link URL.  Then open up a second Browser (different one, not two IE 
windows) like Firefox (with the cookies all cleared) and paste the URL 
into it.  What happens?

   Will the search box be 'behind' the login screen?  ie the users will 
already be loged-in before they do their first search.

   Anyway, things to think about.

   Thanks

> Thanks again,
>
> Bruce
>
>
>
>
> Neal Richter <ne...@ri...>
> 01/24/2005 12:59 PM
>
> To
> Bruce DeYoung <bd...@qa...>
> cc
> htd...@li...
> Subject
> Re: [htdig-dev] htDig and Vignette??
>
>
>
>
>
>
> On Sun, 23 Jan 2005, Bruce DeYoung wrote:
>
>> Thanks Neal for the reply.   Unfortunately, I cannot provide a link to
> the
>> site since it is an intranet site only....at this time.
>
>   Post it anyway so I can take a look at it's structure.  Post the login
> URL then the first URL you see after a sucessful login.
>
>> My suspicion about this is that Vignette security is handled differently
>> than, say, standard Apache security.    Using the -u option with htdig
> and
>> supplying an authenticated user for our Apache-based sites works fine.
> I'm
>> not sure how Vignette authentication works, but I do know that when you
>> attempt to access the site, if your login cookie is not set, it will
>> rediret to a login page and request authentication information.
>
>   Open your cookies file in the browser and clear anything associated
> with
> this website, then relogin into the webiste and check the cookies.
>
>> I've asked our Vignette developer to request some assistance from
> Vignette
>> support as well.
>>
>> When you say "make sure cookie support is enabled", are you referring to
>> something in Vignette or in htDig?
>
>   I assume you are using HtDig 3.2B6
>
>   Look at the  cookies_input_file & disable_cookies settings in HtDig.
> The disable_cookies is 'true' be default.
>
>   My gut feeling is that it's setting a cookie.  You can take the
> contents of the cookie that the browser stores and load it in to the
> HtDig indexer via the cookies_input_file.
>
>   It may also be that the software checks the 'user_agent' string
> supplied
> by the browser/indexer and may disallow access if you aren't running a
> certain version of browser.
>
>   You can fake this buy setting the user_agent in HtDig to be the string
> supplied by IE.  Get it from your apache server weblogs.
>
>   I've seen both of these problems and worked around them this way.
>
>> And, I understand what you're saying about using the rewrite rules...and
> I
>> think you're right about that one.   So, once I'm able to dig the site,
> I
>> will look at the URL references and create a url_rewrite rule to remove
>> the session information.
>
>   Thanks.
>
>

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] htDig and Vignette??

[Bruce D. <bd...@qa...>]

OK.   Here's the URL at the login page:

http://testintranetportal/portal/site/insideQAD/index.jsp?epi-content=LOGIN

Then, after logging in, here are a couple of URL's of content pages:

http://testintranetportal/portal/site/insideQAD/index.jsp?front_door=true&epi_menuItemID=17b4d03e0ebb0d03c0bc8ed22890307a&epi_menuID=557c013f162725a5c2046e478790307a&epi_baseMenuID=557c013f162725a5c2046e478790307a

and

http://testintranetportal/portal/site/insideQAD/index.jsp?front_door=true&epi_menuItemID=8853e4e036d9d40ecfd048922890307a&epi_menuID=b65bac56c452abf6aeda32202890307a&epi_baseMenuID=557c013f162725a5c2046e478790307a

Thanks again,

Bruce




Neal Richter <ne...@ri...> 
01/24/2005 12:59 PM

To
Bruce DeYoung <bd...@qa...>
cc
htd...@li...
Subject
Re: [htdig-dev] htDig and Vignette??






On Sun, 23 Jan 2005, Bruce DeYoung wrote:

> Thanks Neal for the reply.   Unfortunately, I cannot provide a link to 
the
> site since it is an intranet site only....at this time.

   Post it anyway so I can take a look at it's structure.  Post the login 
URL then the first URL you see after a sucessful login.

> My suspicion about this is that Vignette security is handled differently
> than, say, standard Apache security.    Using the -u option with htdig 
and
> supplying an authenticated user for our Apache-based sites works fine. 
I'm
> not sure how Vignette authentication works, but I do know that when you
> attempt to access the site, if your login cookie is not set, it will
> rediret to a login page and request authentication information.

   Open your cookies file in the browser and clear anything associated 
with 
this website, then relogin into the webiste and check the cookies.

> I've asked our Vignette developer to request some assistance from 
Vignette
> support as well.
>
> When you say "make sure cookie support is enabled", are you referring to
> something in Vignette or in htDig?

   I assume you are using HtDig 3.2B6

   Look at the  cookies_input_file & disable_cookies settings in HtDig. 
The disable_cookies is 'true' be default.

   My gut feeling is that it's setting a cookie.  You can take the 
contents of the cookie that the browser stores and load it in to the 
HtDig indexer via the cookies_input_file.

   It may also be that the software checks the 'user_agent' string 
supplied 
by the browser/indexer and may disallow access if you aren't running a 
certain version of browser.

   You can fake this buy setting the user_agent in HtDig to be the string 
supplied by IE.  Get it from your apache server weblogs.

   I've seen both of these problems and worked around them this way.

> And, I understand what you're saying about using the rewrite rules...and 
I
> think you're right about that one.   So, once I'm able to dig the site, 
I
> will look at the URL references and create a url_rewrite rule to remove
> the session information.

   Thanks.

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] htDig and Vignette??

[Neal R. <ne...@ri...>]

On Sun, 23 Jan 2005, Bruce DeYoung wrote:

> Thanks Neal for the reply.   Unfortunately, I cannot provide a link to the
> site since it is an intranet site only....at this time.

   Post it anyway so I can take a look at it's structure.  Post the login 
URL then the first URL you see after a sucessful login.

> My suspicion about this is that Vignette security is handled differently
> than, say, standard Apache security.    Using the -u option with htdig and
> supplying an authenticated user for our Apache-based sites works fine. I'm
> not sure how Vignette authentication works, but I do know that when you
> attempt to access the site, if your login cookie is not set, it will
> rediret to a login page and request authentication information.

   Open your cookies file in the browser and clear anything associated with 
this website, then relogin into the webiste and check the cookies.

> I've asked our Vignette developer to request some assistance from Vignette
> support as well.
>
> When you say "make sure cookie support is enabled", are you referring to
> something in Vignette or in htDig?

   I assume you are using HtDig 3.2B6

   Look at the  cookies_input_file & disable_cookies settings in HtDig. 
The disable_cookies is 'true' be default.

   My gut feeling is that it's setting a cookie.  You can take the 
contents of the cookie that the browser stores and load it in to the 
HtDig indexer via the cookies_input_file.

   It may also be that the software checks the 'user_agent' string supplied 
by the browser/indexer and may disallow access if you aren't running a 
certain version of browser.

   You can fake this buy setting the user_agent in HtDig to be the string 
supplied by IE.  Get it from your apache server weblogs.

   I've seen both of these problems and worked around them this way.

> And, I understand what you're saying about using the rewrite rules...and I
> think you're right about that one.   So, once I'm able to dig the site, I
> will look at the URL references and create a url_rewrite rule to remove
> the session information.

   Thanks.

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] Compression related performance

[Neal R. <ne...@ri...>]

On Thu, 20 Jan 2005, Jim wrote:

> On Sat, 15 Jan 2005, Neal Richter wrote:
>
>> Question #1)
>> 
>>   Could you download a fresh copy of BDB 3.0.55 from
>> http://www.sleepycat.com/download/index.shtml
>> 
>> It's easy to build, I am most interested in the results when you run 
>> db_stat on the uncompressed files and report back the fill-factor.  This is 
>> a measure of wasted space in the BDB pages.  This won't work on the 
>> compressed index.
>
> How exactly do I obtain this information? In the entire package I see only
> one place that references fill-factor (diskspace.html) and all it says is
> that you can obtain it with db_stat. However I don't see anything called
> fill-factor in the db_stat output. What options are needed to extract this
> info? Or is it a value that needs to be computed from other output?

Jim:  The 'xx% ff' below is the 'fill factor'

db_stat -d db.words.db
53162   Btree magic number.
7       Btree version number.
Flags:
2       Minimum keys per-page.
1024    Underlying database page size.
4       Number of levels in the tree.
170348  Number of keys in the tree.
328     Number of tree internal pages.
109792  Number of bytes free in tree internal pages (67% ff).
7019    Number of tree leaf pages.
2541354 Number of bytes free in tree leaf pages (65% ff).
0       Number of tree duplicate pages.
0       Number of bytes free in tree duplicate pages (0% ff).
0       Number of tree overflow pages.
0       Number of bytes free in tree overflow pages (0% ff).
0       Number of pages on the free list.


Thanks

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

[htdig-dev] New proposed nofollow robot exclusion on links

[Neal R. <ne...@ri...>]

Read this!

http://www.searchenginejournal.com/index.php?p=1254

Basically they are proposing to support puting nofollow robot directives 
in individual links.  This is great.. I've been wishing for this for a 
while.  This makes it possible to more carefully direct the spider.. ie 
follow some links on a page, but not all without having to write huge 
list of exclusion patterns.

We should probably implement this since it looks like it could become a 
defacto standard.

Thanks.

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] htDig and Vignette??

[Neal R. <ne...@ri...>]

On Fri, 21 Jan 2005, Bruce DeYoung wrote:

> Hello htDig Developers!
>
> We have a development effort going on to manage and present content for
> our corporate intranet site via Vignette.  I have been asked to see if
> htDig can be used for searching this site.   We already use htDig for our
> external websites (standard content on Apache servers), so have some
> knowledge of how it works.   I've created a new config file for digging
> the vignette site and an account for searching has been created in
> Vignette's active directory.   However, htDig cannot seem to get past the
> LOGIN redirect.
>
> Has anyone used htDig with Vignette?   Anyone have any ideas about how to
> get Vignette to recognize the login/password and successfully get past the
> LOGIN redirect to the actual content?    (We are also looking into using
> the provided Autonomy license for keyword searching in Vignette....but
> this is a parallel effort.)

   Please post a link.. it's been a while since I looked at a site coupled 
with Vignette.  I think it will require two things:

#1) Make sure the cookie support is enabled

#2) Use the rewrite rules to strip Vignette sessionids off of the URLs, 
otherwise you'll be giving every user of search the same sessionid.

   For those of you who are not familiar with Vignette, they provide a 
software to serve up webpages with sessionids to enable tracking and 
reporting of site usage patterns.

   Thanks.

> Thanks,
>
> Bruce
>
>
> Bruce DeYoung
> Sr. Business Systems Architect
> Global Support Operations
>
> QAD Inc.
> Home Office
> Louisville CO  , USA
> www.qad.com
>
>
> bd...@qa...
> Tel:  720-890-4009
>
> This e-mail may contain QAD proprietary information and should be treated
> as confidential.
>

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

[htdig-dev] Re: configure: error: cannot compute sizeof (unsigned long long int), 77

[Gregory H. <gh...@ca...>]

> Date: Fri, 21 Jan 2005 23:07:51 +0100
> From: Martin Allert <al...@ar...>
> To: Gregory Hicks <ghicks>
> Subject: Re: [htdig] configure: error: cannot compute sizeof (unsigned long 
long int), 77
> 
> 
> 
> 			Hello Gregory,
> 
> 
> This is my configure string for htdig-3.1.6-ssl:
> 

This is what I did...

setenv LDFLAGS "-L/usr/local/
> LDFLAGS="-L/opt/openssl/lib -lssl -lcrypto" CPPFLAGS="-Wno-deprecated" 
CXXFLAGS="-Wno-deprecated" ./configure --prefix=/opt/htdig-3.1.6-ssl

metis% setenv LDFLAGS '-L/usr/local/ssl/lib -l ssl -l crypto'
metis% setenv CPPFLAGS "-Wno-deprecated"
metis% setenv CXXFLAGS "-Wno-deprecated"
metis% cd incoming/htdig/htdig-3.2.0b6
metis% ./configure --prefix=/usr/local/apache-1.3.9

 --with-config-dir=/opt/htdig-3.1.6-ssl/conf 
--with-database-dir=/opt/htdig-3.1.6-ssl/db/default 
--with-cgi-bin-dir=/opt/htdig-3.1.6-ssl/bin 
--with-image-dir=/opt/htdig-3.1.6-ssl/htdocs/images 
--with-search-dir=/opt/htdig-3.1.6-ssl/htdocs --with-image-url-prefix=/images 
--with-common-dir=/opt/htdig-3.1.6-ssl/common/default --with-rx
> 
> This is my configure string for htdig-3.2.0b5:
> 
> ./configure --prefix=/opt/htdig-3.2.0b5 --enable-bigfile --with-common-dir=/
> opt/htdig-3.2.0b5/common --with-database-dir=/opt/htdig-3.2.0b5/db 
--with-ssl=/o
> pt/openssl --with-zlib --without-apache 
--with-cgi-bin-dir=/opt/htdig-3.2.0b5/bi
> n --with-image-dir=/opt/htdig-3.2.0b5/htdocs/images
> 
> I use this gcc version (sunfreeware.com):
> 
> [root@blop:/data/BUILD/htdig2/htdig-3.2.0b5]> gcc -v
> Reading specs from /usr/local/lib/gcc-lib/sparc-sun-solaris2.7/3.3.2/specs
> Configured with: ../configure --with-as=/usr/ccs/bin/as 
--with-ld=/usr/ccs/bin/ld --disable-nls
> Thread model: posix
> gcc version 3.3.2

My gcc version (from sunfreeware.com):
metis% gcc -v
Reading specs from /usr/local/lib/gcc-lib/sparc-sun-solaris2.8/3.3.1/specs
Configured with: ../configure --with-as=/usr/ccs/bin/as 
--with-ld=/usr/ccs/bin/ld --disable-nls
Thread model: posix
gcc version 3.3.1

Still same error...


> 
> Did you also install the SunOS Header Files (SUNWhea)? Also the config.log
> output shows, that you are obviously missing a C++ Compiler (see the 
> "#ifdef __cplusplus" statement). So if you install the g++ compiler you will
> be fine. Additionally, if you set the CFLAGS variable before invoking 
configure,
> you need to set the CPPFLAGS and the CXXFLAGS variable also (see above).
> 
> Yours,
> 
> Martin
> 
> -- 
> 
> --------------------------------------------------------
>  arago AG, Institut fuer komplexes Datenmanagement
>  Am Niddatal 3, 60488 Frankfurt/Main, al...@ar...
>  Tel. 069/405680, Fax 069/40568111, http://www.arago.de
> --------------------------------------------------------
> 
> On Fri, Jan 21, 2005 at 01:48:15PM -0800, Gregory Hicks wrote:
> > Greetings:
> > 
> > I am trying to configure htdig-3.2.0b6 on a Solaris 8 box with gcc 3.3.1 and 
> > getting this error:
> > 
> > configure: error: cannot compute sizeof (unsigned long long int), 77
> > See `config.log' for more details.
> > 
> > (When I tried to configure htdig-3.1.6, the configure script could not find 
> > libstdc++ even though it exists:
> > 
> > /usr/local/lib:
> > libncurses++.a*      libstdc++.la*        libstdc++.so.5.0.5*
> > libstdc++.a          libstdc++.so@        libsupc++.a
> > libstdc++.a.2.10.0   libstdc++.so.5@      libsupc++.la*
> > 
> > and /usr/local/lib IS in LD_LIBRARY_PATH...)
> > 
> > Here are the last contents of config.log:
> > 
> > MAKEINFO='${SHELL} /home/ghicks/incoming/htdig/htdig-3.2.0b6/missing --run 
> > makeinfo'
> > MV='/bin/mv'
> > OBJEXT='o'
> > PACKAGE='htdig'
> > PACKAGE_BUGREPORT=''
> > PACKAGE_NAME=''
> > PACKAGE_STRING=''
> > PACKAGE_TARNAME=''
> > PACKAGE_VERSION=''
> > PATH_SEPARATOR=':'
> > PERL='/bin/perl'
> > RANLIB='ranlib'
> > RRDTOOL=''
> > SEARCH_DIR='${prefix}/htdocs/htdig'
> > SEARCH_FORM='search.html'
> > SED='/usr/xpg4/bin/sed'
> > SENDMAIL='/usr/lib/sendmail'
> > SET_MAKE=''
> > SHELL='/bin/bash'
> > STRIP='strip'
> > TAR='tar'
> > TESTS_FALSE=''
> > TESTS_TRUE='#'
> > TIME='/bin/time'
> > TIMEV='/bin/time'
> > USER='ghicks'
> > VERSION='3.2.0b6'
> > YACC='yacc'
> > ac_ct_AR='ar'
> > ac_ct_CC='gcc'
> > ac_ct_CXX='g++'
> > ac_ct_F77='g77'
> > ac_ct_RANLIB='ranlib'
> > ac_ct_STRIP='strip'
> > am__fastdepCC_FALSE=''
> > am__fastdepCC_TRUE='#'
> > am__fastdepCXX_FALSE=''
> > am__fastdepCXX_TRUE='#'
> > am__include='include'
> > am__leading_dot='.'
> > am__quote=''
> > bindir='${exec_prefix}/bin'
> > build='sparc-sun-solaris2.8'
> > build_alias=''
> > build_cpu='sparc'
> > build_os='solaris2.8'
> > build_vendor='sun'
> > datadir='${prefix}/share'
> > exec_prefix='NONE'
> > extra_ldflags=''
> > host='sparc-sun-solaris2.8'
> > host_alias=''
> > host_cpu='sparc'
> > host_os='solaris2.8'
> > host_vendor='sun'
> > includedir='${prefix}/include'
> > infodir='${prefix}/info'
> > install_sh='/home/ghicks/incoming/htdig/htdig-3.2.0b6/install-sh'
> > libdir='${exec_prefix}/lib'
> > libexecdir='${exec_prefix}/libexec'
> > localstatedir='${prefix}/var'
> > mandir='${prefix}/man'
> > oldincludedir='/usr/include'
> > prefix='/usr/local/apache-1.3.9'
> > program_transform_name='s,x,x,'
> > sbindir='${exec_prefix}/sbin'
> > sharedstatedir='${prefix}/com'
> > subdirs=''
> > sysconfdir='${prefix}/etc'
> > target_alias=''
> > 
> > ## ----------- ##
> > ## confdefs.h. ##
> > ## ----------- ##
> > 
> > #define HAVE_ALLOCA 1
> > #define HAVE_ALLOCA_H 1
> > #define HAVE_ALLOCA_H 1
> > #define HAVE_ARPA_INET_H 1
> > #define HAVE_DIRENT_H 1
> > #define HAVE_DLFCN_H 1
> > #define HAVE_FCNTL_H 1
> > #define HAVE_FLOAT_H 1
> > #define HAVE_INTTYPES_H 1
> > #define HAVE_LANGINFO_H 1
> > #define HAVE_LIBINTL_H 1
> > #define HAVE_LIBNSL 1
> > #define HAVE_LIBSOCKET 1
> > #define HAVE_LIBZ 1
> > #define HAVE_LIMITS_H 1
> > #define HAVE_LOCALE_H 1
> > #define HAVE_MALLOC_H 1
> > #define HAVE_MEMORY_H 1
> > #define HAVE_NETDB_H 1
> > #define HAVE_NETINET_IN_H 1
> > #define HAVE_STDBOOL_H 1
> > #define HAVE_STDBOOL_H 1
> > #define HAVE_STDDEF_H 1
> > #define HAVE_STDLIB_H 1
> > #define HAVE_STDLIB_H 1
> > #define HAVE_STRINGS_H 1
> > #define HAVE_STRINGS_H 1
> > #define HAVE_STRING_H 1
> > #define HAVE_STRING_H 1
> > #define HAVE_STRUCT_STAT_ST_BLKSIZE 1
> > #define HAVE_SYSLOG_H 1
> > #define HAVE_SYS_FILE_H 1
> > #define HAVE_SYS_IOCTL_H 1
> > #define HAVE_SYS_SOCKET_H 1
> > #define HAVE_SYS_STAT_H 1
> > #define HAVE_SYS_TIMEB_H 1
> > #define HAVE_SYS_TIME_H 1
> > #define HAVE_SYS_TYPES_H 1
> > #define HAVE_SYS_WAIT_H 1
> > #define HAVE_SYS_WAIT_H 1
> > #define HAVE_UNISTD_H 1
> > #define HAVE_UNISTD_H 1
> > #define HAVE_WAIT_H 1
> > #define HAVE_WCHAR_H 1
> > #define HAVE_WCTYPE_H 1
> > #define HAVE_ZLIB_H
> > #define HAVE__BOOL 1
> > #define HAVE__BOOL 1
> > #define PACKAGE "htdig"
> > #define PACKAGE_BUGREPORT ""
> > #define PACKAGE_NAME ""
> > #define PACKAGE_STRING ""
> > #define PACKAGE_TARNAME ""
> > #define PACKAGE_VERSION ""
> > #define STDC_HEADERS 1
> > #define STDC_HEADERS 1
> > #define TIME_WITH_SYS_TIME 1
> > #define TIME_WITH_SYS_TIME 1
> > #define VERSION "3.2.0b6"
> > #endif
> > #ifdef __cplusplus
> > extern "C" void std::exit (int) throw (); using std::exit;
> > 
> > configure: exit 1
> > 
> > And the console output:
> > 
> > checking for a BSD-compatible install... ./install-sh -c
> > checking whether build environment is sane... yes
> > checking for gawk... gawk
> > checking whether make sets $(MAKE)... yes
> > checking whether to enable maintainer-specific portions of Makefiles... no
> > checking for g++... g++
> > checking for C++ compiler default output file name... a.out
> > checking whether the C++ compiler works... yes
> > checking whether we are cross compiling... no
> > checking for suffix of executables...
> > checking for suffix of object files... o
> > checking whether we are using the GNU C++ compiler... yes
> > checking whether g++ accepts -g... yes
> > checking for style of include used by make... GNU
> > checking dependency style of g++... none
> > checking for gcc... gcc
> > checking whether we are using the GNU C compiler... yes
> > checking whether gcc accepts -g... yes
> > checking for gcc option to accept ANSI C... none needed
> > checking dependency style of gcc... none
> > checking how to run the C preprocessor... gcc -E
> > checking for egrep... egrep
> > checking for AIX... no
> > checking for flex... no
> > checking for lex... lex
> > checking for yywrap in -lfl... no
> > checking for yywrap in -ll... yes
> > checking lex output file root... lex.yy
> > checking whether yytext is a pointer... no
> > checking for bison... no
> > checking for byacc... no
> > checking for a BSD-compatible install... ./install-sh -c
> > checking whether ln -s works... yes
> > checking whether make sets $(MAKE)... (cached) yes
> > checking build system type... sparc-sun-solaris2.8
> > checking host system type... sparc-sun-solaris2.8
> > [...snip...]
> > checking whether stat file-mode macros are broken... no
> > checking for stdbool.h that conforms to C99... (cached) yes
> > checking for _Bool... (cached) yes
> > checking for an ANSI C-conforming const... yes
> > checking for inline... inline
> > checking for off_t... yes
> > checking for size_t... yes
> > checking for struct stat.st_blksize... yes
> > checking whether time.h and sys/time.h may both be included... (cached) yes
> > checking whether struct tm is in sys/time.h or time.h... time.h
> > checking for unsigned long long int... yes
> > checking size of unsigned long long int...
> > 
> > Any thoughts on what to modify?  I don't have a clue.
> > 
> > Regards,
> > Gregory Hicks
> > 
> > 
> > -------------------------------------------------------------------
> > Gregory Hicks                        | Principal Systems Engineer
> > Cadence Design Systems               | Direct:   408.576.3609
> > 555 River Oaks Pkwy M/S 6B1          | Fax:      408.894.3400
> > San Jose, CA 95134                   | Internet: gh...@ca...
> > 
> > I am perfectly capable of learning from my mistakes.  I will surely
> > learn a great deal today.
> > 
> > "A democracy is a sheep and two wolves deciding on what to have for
> > lunch.  Freedom is a well armed sheep contesting the results of the
> > decision." - Benjamin Franklin
> > 
> > "The best we can hope for concerning the people at large is that they
> > be properly armed." --Alexander Hamilton
> > 
> > 
> > 
> > 
> > -------------------------------------------------------
> > This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> > Tool for open source databases. Create drag-&-drop reports. Save time
> > by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> > Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> > _______________________________________________
> > ht://Dig general mailing list: <htd...@li...>
> > ht://Dig FAQ: http://htdig.sourceforge.net/FAQ.html
> > List information (subscribe/unsubscribe, etc.)
> > https://lists.sourceforge.net/lists/listinfo/htdig-general

-------------------------------------------------------------------
Gregory Hicks                        | Principal Systems Engineer
Cadence Design Systems               | Direct:   408.576.3609
555 River Oaks Pkwy M/S 6B1          | Fax:      408.894.3400
San Jose, CA 95134                   | Internet: gh...@ca...

I am perfectly capable of learning from my mistakes.  I will surely
learn a great deal today.

"A democracy is a sheep and two wolves deciding on what to have for
lunch.  Freedom is a well armed sheep contesting the results of the
decision." - Benjamin Franklin

"The best we can hope for concerning the people at large is that they
be properly armed." --Alexander Hamilton

[htdig-dev] Re: [htdig] configure: error: cannot compute sizeof (unsigned long long int), 77

[Gregory H. <gh...@ca...>]

> From: Gregory Hicks <ghicks>
> Subject: configure: error: cannot compute sizeof (unsigned long long int), 77
> Date: Fri, 21 Jan 2005 13:48:15 -0800 (PST)
> 
> Greetings:
> 
> I am trying to configure htdig-3.2.0b6 on a Solaris 8 box with gcc 3.3.1 and 
> getting this error:
> 
> configure: error: cannot compute sizeof (unsigned long long int), 77
> See `config.log' for more details.
> 
> (When I tried to configure htdig-3.1.6, the configure script could not find 
> libstdc++ even though it exists:
> 
> /usr/local/lib:
> libncurses++.a*      libstdc++.la*        libstdc++.so.5.0.5*
> libstdc++.a          libstdc++.so@        libsupc++.a
> libstdc++.a.2.10.0   libstdc++.so.5@      libsupc++.la*
> 
> and /usr/local/lib IS in LD_LIBRARY_PATH...)

Found some more info...

configure:23018: gcc -c -g -O2 -Wall  conftest.c >&5
conftest.c: In function `main':
conftest.c:72: warning: statement with no effect
configure:23024: $? = 0
configure:23027: test -z                         || test ! -s conftest.err
configure:23030: $? = 0
configure:23033: test -s conftest.o
configure:23036: $? = 0
configure:23047: result: time.h
configure:23057: checking for unsigned long long int
configure:23081: gcc -c -g -O2 -Wall  conftest.c >&5
configure:23087: $? = 0
configure:23090: test -z                         || test ! -s conftest.err
configure:23093: $? = 0
configure:23096: test -s conftest.o
configure:23099: $? = 0
configure:23110: result: yes
configure:23113: checking size of unsigned long long int
configure:23425: gcc -o conftest -g -O2 -Wall   conftest.c -lz -lnsl -lsocket  
>&5
configure:23428: $? = 0
configure:23430: ./conftest
ld.so.1: ./conftest: fatal: relocation error: file ./conftest: symbol 
__register_frame_info: referenced symbol not found
./configure: line 1: 27447 Killed                  ./conftest$ac_exeext
configure:23433: $? = 137
configure: program exited with status 137
configure: failed program was:
| /* confdefs.h.  */
|
| #define PACKAGE_NAME ""
| #define PACKAGE_TARNAME ""
| #define PACKAGE_VERSION ""
| #define PACKAGE_STRING ""
| #define PACKAGE_BUGREPORT ""
| #define PACKAGE "htdig"
| #define VERSION "3.2.0b6"
| #ifdef __cplusplus
| extern "C" void std::exit (int) throw (); using std::exit;
| #endif
| #define STDC_HEADERS 1
| #define HAVE_SYS_TYPES_H 1
| #define HAVE_SYS_STAT_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_STRING_H 1
| #define HAVE_MEMORY_H 1
| #define HAVE_STRINGS_H 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_UNISTD_H 1
| #define HAVE_DLFCN_H 1
| #define HAVE_LIBSOCKET 1
| #define HAVE_LIBNSL 1
| #define HAVE_ZLIB_H
| #define HAVE_LIBZ 1
| #define HAVE__BOOL 1
| #define HAVE_STDBOOL_H 1
| #define TIME_WITH_SYS_TIME 1
| #define HAVE_ALLOCA_H 1
| #define HAVE_ALLOCA 1
| #define HAVE_DIRENT_H 1
| #define STDC_HEADERS 1
| #define HAVE_SYS_WAIT_H 1
| #define HAVE_ARPA_INET_H 1
| #define HAVE_FCNTL_H 1
| #define HAVE_FLOAT_H 1
| #define HAVE_LANGINFO_H 1
| #define HAVE_LIBINTL_H 1
| #define HAVE_LIMITS_H 1
| #define HAVE_LOCALE_H 1
| #define HAVE_MALLOC_H 1
| #define HAVE_NETDB_H 1
| #define HAVE_NETINET_IN_H 1
| #define HAVE_STDDEF_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_STRING_H 1
| #define HAVE_STRINGS_H 1
| #define HAVE_SYS_FILE_H 1
| #define HAVE_SYS_IOCTL_H 1
| #define HAVE_SYS_SOCKET_H 1
| #define HAVE_SYS_TIME_H 1
| #define HAVE_SYS_TIMEB_H 1
| #define HAVE_SYSLOG_H 1
| #define HAVE_UNISTD_H 1
| #define HAVE_WCHAR_H 1
| #define HAVE_WCTYPE_H 1
| #define HAVE_SYS_WAIT_H 1
| #define HAVE_WAIT_H 1
| #define HAVE_ALLOCA_H 1
| #define HAVE__BOOL 1
| #define HAVE_STDBOOL_H 1
| #define HAVE_STRUCT_STAT_ST_BLKSIZE 1
| #define TIME_WITH_SYS_TIME 1
| /* end confdefs.h.  */
| #include <stdio.h>
| #if HAVE_SYS_TYPES_H
| # include <sys/types.h>
| #endif
| #if HAVE_SYS_STAT_H
| # include <sys/stat.h>
| #endif
| #if STDC_HEADERS
| # include <stdlib.h>
| # include <stddef.h>
| #else
| # if HAVE_STDLIB_H
| #  include <stdlib.h>
| # endif
| #endif
| #if HAVE_STRING_H
| # if !STDC_HEADERS && HAVE_MEMORY_H
| #  include <memory.h>
| # endif
| # include <string.h>
| #endif
| #if HAVE_STRINGS_H
| # include <strings.h>
| #endif
| #if HAVE_INTTYPES_H
| # include <inttypes.h>
| #else
| # if HAVE_STDINT_H
| #  include <stdint.h>
| # endif
| #endif
| #if HAVE_UNISTD_H
| # include <unistd.h>
| #endif
| long longval () { return (long) (sizeof (unsigned long long int)); }
| unsigned long ulongval () { return (long) (sizeof (unsigned long long int)); 
}| #include <stdio.h>
| #include <stdlib.h>
| int
| main ()
| {
|
|   FILE *f = fopen ("conftest.val", "w");
|   if (! f)
|     exit (1);
|   if (((long) (sizeof (unsigned long long int))) < 0)
|     {  
|       long i = longval ();
|       if (i != ((long) (sizeof (unsigned long long int))))
|       exit (1);
|       fprintf (f, "%ld\n", i);
|     }  
|   else
|     {  
|       unsigned long i = ulongval ();
|       if (i != ((long) (sizeof (unsigned long long int))))
|       exit (1);
|       fprintf (f, "%lu\n", i);
|     }  
|   exit (ferror (f) || fclose (f) != 0);
|
|   ;
|   return 0;
| }
configure:23443: error: cannot compute sizeof (unsigned long long int), 77
See `config.log' for more details.
## ---------------- ##
## Cache variables. ##
## ---------------- ##

ac_cv_build=sparc-sun-solaris2.8
ac_cv_build_alias=sparc-sun-solaris2.8
ac_cv_c_compiler_gnu=yes
ac_cv_c_const=yes
ac_cv_c_inline=inline
ac_cv_cxx_compiler_gnu=yes
ac_cv_env_CC_set=set
ac_cv_env_CC_value=gcc
ac_cv_env_CFLAGS_set=
ac_cv_env_CFLAGS_value=
ac_cv_env_CPPFLAGS_set=
ac_cv_env_CPPFLAGS_value=
ac_cv_env_CPP_set=
ac_cv_env_CPP_value=
ac_cv_env_CXXCPP_set=
ac_cv_env_CXXCPP_value=
ac_cv_env_CXXFLAGS_set=
ac_cv_env_CXXFLAGS_value=
ac_cv_env_CXX_set=
ac_cv_env_CXX_value=
ac_cv_env_F77_set=
ac_cv_env_F77_value=
ac_cv_env_FFLAGS_set=
ac_cv_env_FFLAGS_value=
ac_cv_env_LDFLAGS_set=
ac_cv_env_LDFLAGS_value=
ac_cv_env_build_alias_set=
ac_cv_env_build_alias_value=
ac_cv_env_host_alias_set=
ac_cv_env_host_alias_value=
ac_cv_env_target_alias_set=
ac_cv_env_target_alias_value=
ac_cv_exeext=
ac_cv_f77_compiler_gnu=yes
ac_cv_func_alloca_works=yes
ac_cv_header_alloca_h=yes
ac_cv_header_arpa_inet_h=yes
ac_cv_header_dirent_dirent_h=yes
ac_cv_header_dlfcn_h=yes
ac_cv_header_fcntl_h=yes
ac_cv_header_float_h=yes
ac_cv_header_inttypes_h=yes
ac_cv_header_langinfo_h=yes
ac_cv_header_libintl_h=yes
ac_cv_header_limits_h=yes
ac_cv_header_locale_h=yes
ac_cv_header_malloc_h=yes
ac_cv_header_memory_h=yes
ac_cv_header_netdb_h=yes
ac_cv_header_netinet_in_h=yes
ac_cv_header_stat_broken=no
ac_cv_header_stdbool_h=yes
ac_cv_header_stdc=yes
ac_cv_header_stddef_h=yes
ac_cv_header_stdint_h=no
ac_cv_header_stdlib_h=yes
ac_cv_header_string_h=yes
ac_cv_header_strings_h=yes
ac_cv_header_sys_file_h=yes
ac_cv_header_sys_ioctl_h=yes
ac_cv_header_sys_socket_h=yes
ac_cv_header_sys_stat_h=yes
ac_cv_header_sys_time_h=yes
ac_cv_header_sys_timeb_h=yes
ac_cv_header_sys_types_h=yes
ac_cv_header_sys_wait_h=yes
ac_cv_header_syslog_h=yes
ac_cv_header_time=yes
ac_cv_header_unistd_h=yes
ac_cv_header_wait_h=yes
ac_cv_header_wchar_h=yes
ac_cv_header_wctype_h=yes
ac_cv_header_zlib_h=yes
ac_cv_host=sparc-sun-solaris2.8
ac_cv_host_alias=sparc-sun-solaris2.8
ac_cv_lib_fl_yywrap=no
ac_cv_lib_l_yywrap=yes
ac_cv_lib_nsl_t_accept=yes
ac_cv_lib_socket_socket=yes
ac_cv_lib_z_inflateEnd=yes
ac_cv_member_struct_stat_st_blksize=yes
ac_cv_objext=o
ac_cv_path_AR=/usr/ccs/bin/ar
ac_cv_path_FIND=/bin/find
ac_cv_path_GUNZIP=/bin/gunzip
ac_cv_path_MV=/bin/mv
ac_cv_path_PERL=/bin/perl
ac_cv_path_SED=/usr/xpg4/bin/sed
ac_cv_path_SENDMAIL=/usr/lib/sendmail
ac_cv_path_SHELL=/bin/bash
ac_cv_path_TIME=/bin/time
ac_cv_prog_AWK=gawk
ac_cv_prog_CPP='gcc -E'
ac_cv_prog_CXXCPP='g++ -E'
ac_cv_prog_LEX=lex
ac_cv_prog_TAR=tar
ac_cv_prog_ac_ct_AR=ar
ac_cv_prog_ac_ct_CC=gcc
ac_cv_prog_ac_ct_CXX=g++
ac_cv_prog_ac_ct_F77=g77
ac_cv_prog_ac_ct_RANLIB=ranlib
ac_cv_prog_ac_ct_STRIP=strip
ac_cv_prog_cc_g=yes
ac_cv_prog_cc_stdc=
ac_cv_prog_cxx_g=yes
ac_cv_prog_egrep=egrep
ac_cv_prog_f77_g=yes
ac_cv_prog_lex_root=lex.yy
ac_cv_prog_lex_yytext_pointer=no
ac_cv_prog_make_make_set=yes
ac_cv_search_opendir='none required'
ac_cv_struct_tm=time.h
ac_cv_type__Bool=yes
ac_cv_type_off_t=yes
ac_cv_type_size_t=yes
ac_cv_type_unsigned_long_long_int=yes
ac_cv_working_alloca_h=yes
am_cv_CC_dependencies_compiler_type=none
am_cv_CXX_dependencies_compiler_type=none
htdig_cv_tests=no
lt_cv_deplibs_check_method=pass_all
lt_cv_file_magic_cmd='$MAGIC_CMD'
lt_cv_file_magic_test_file=
lt_cv_ld_reload_flag=-r
lt_cv_objdir=.libs
lt_cv_path_LD=/usr/ccs/bin/ld
lt_cv_path_LDCXX=/usr/ccs/bin/ld
lt_cv_path_NM='/usr/ccs/bin/nm -p'
lt_cv_path_SED=/usr/xpg4/bin/sed
lt_cv_prog_compiler_c_o=yes
lt_cv_prog_compiler_c_o_CXX=yes
lt_cv_prog_compiler_c_o_F77=yes
lt_cv_prog_compiler_rtti_exceptions=no
lt_cv_prog_gnu_ld=no
lt_cv_prog_gnu_ldcxx=no
lt_cv_sys_global_symbol_pipe='sed -n -e '\''s/^.*[      ]\([BDRT][BDRT]*\)[
][      ]*\(\)\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2\3 \3/p'\'''
lt_cv_sys_global_symbol_to_c_name_address='sed -n -e '\''s/^: \([^ ]*\) $/  
{\"\1\", (lt_ptr) 0},/p'\'' -e '\''s/^[BCDEGRST] \([^ ]*\) \([^ ]*\)$/  {"\2", 
(lt_ptr) \&\2},/p'\'''
lt_cv_sys_global_symbol_to_cdecl='sed -n -e '\''s/^. .* \(.*\)$/extern int 
\1;/p'\'''
lt_cv_sys_max_cmd_len=262144
lt_lt_cv_prog_compiler_c_o='"yes"'
lt_lt_cv_prog_compiler_c_o_CXX='"yes"'
lt_lt_cv_prog_compiler_c_o_F77='"yes"'
lt_lt_cv_sys_global_symbol_pipe='"sed -n -e '\''s/^.*[  ]\\([BDRT][BDRT]*\\)[
][      ]*\\(\\)\\([_A-Za-z][_A-Za-z0-9]*\\)\$/\\1 \\2\\3 \\3/p'\''"'
lt_lt_cv_sys_global_symbol_to_c_name_address='"sed -n -e '\''s/^: \\([^ ]*\\) 
\$/  {\\\"\\1\\\", (lt_ptr) 0},/p'\'' -e '\''s/^[BCDEGRST] \\([^ ]*\\) \\([^ 
]*\\)\$/  {\"\\2\", (lt_ptr) \\&\\2},/p'\''"'
lt_lt_cv_sys_global_symbol_to_cdecl='"sed -n -e '\''s/^. .* \\(.*\\)\$/extern 
int \\1;/p'\''"'
zlib_cv_zlib_h=yes

## ----------------- ##
## Output variables. ##
## ----------------- ##

ACLOCAL='${SHELL} /home/ghicks/incoming/htdig/htdig-3.2.0b6/missing --run 
aclocal-1.7'
ALLOCA=''
AMDEPBACKSLASH='\'
AMDEP_FALSE='#'
AMDEP_TRUE=''
AMTAR='${SHELL} /home/ghicks/incoming/htdig/htdig-3.2.0b6/missing --run tar'
APACHE=''
APACHE_MODULES=''
AR='/usr/ccs/bin/ar'
AUTOCONF='${SHELL} /home/ghicks/incoming/htdig/htdig-3.2.0b6/missing --run 
autoconf'
AUTOHEADER='${SHELL} /home/ghicks/incoming/htdig/htdig-3.2.0b6/missing --run 
autoheader'
AUTOMAKE='${SHELL} /home/ghicks/incoming/htdig/htdig-3.2.0b6/missing --run 
automake-1.7'
AWK='gawk'
CC='gcc'
CCDEPMODE='depmode=none'
CFLAGS='-g -O2 -Wall'
CGIBIN_DIR='${prefix}/cgi-bin'
COMMON_DIR='${datadir}/htdig'
CONFIG_DIR='${prefix}/conf'
CPP='gcc -E'
CPPFLAGS=''
CXX='g++'
CXXCPP='g++ -E'
CXXDEPMODE='depmode=none'
CXXFLAGS='-g -O2 -Wall -fno-rtti -fno-exceptions'
CYGPATH_W='echo'
DATABASE_DIR='${localstatedir}/htdig'
DEFAULT_CONFIG_FILE='${CONFIG_DIR}/htdig.conf'
DEFS=''
DEPDIR='.deps'
ECHO='echo'
ECHO_C=''
ECHO_N='-n'
ECHO_T=''
EGREP='egrep'
EXEEXT=''
F77='g77'
FFLAGS='-g -O2'
FIND='/bin/find'
GUNZIP='/bin/gunzip'
HAVE_SSL=''
HTDIG_MAJOR_VERSION='3'
HTDIG_MICRO_VERSION='0'
HTDIG_MINOR_VERSION='2'
IMAGE_DIR='${prefix}/htdocs/htdig'
IMAGE_URL_PREFIX='/htdig'
INSTALL_DATA='${INSTALL} -m 644'
INSTALL_PROGRAM='${INSTALL}'
INSTALL_SCRIPT='${INSTALL}'
INSTALL_STRIP_PROGRAM='${SHELL} $(install_sh) -c -s'
LDFLAGS=''
LEX='lex'
LEXLIB='-ll'
LEX_OUTPUT_ROOT='lex.yy'
LIBOBJS=''
LIBS='-lz -lnsl -lsocket '
LIBTOOL='$(SHELL) $(top_builddir)/libtool'
LN_S='ln -s'
LTLIBOBJS=''
MAINT='#'
MAINTAINER_MODE_FALSE=''
MAINTAINER_MODE_TRUE='#'
MAKEINFO='${SHELL} /home/ghicks/incoming/htdig/htdig-3.2.0b6/missing --run 
makeinfo'
MV='/bin/mv'
OBJEXT='o'
PACKAGE='htdig'
PACKAGE_BUGREPORT=''
PACKAGE_NAME=''
PACKAGE_STRING=''
PACKAGE_TARNAME=''
PACKAGE_VERSION=''
PATH_SEPARATOR=':'
PERL='/bin/perl'
RANLIB='ranlib'
RRDTOOL=''
SEARCH_DIR='${prefix}/htdocs/htdig'
SEARCH_FORM='search.html'
SED='/usr/xpg4/bin/sed'
SENDMAIL='/usr/lib/sendmail'
SET_MAKE=''
SHELL='/bin/bash'
STRIP='strip'
TAR='tar'
TESTS_FALSE=''
TESTS_TRUE='#'
TIME='/bin/time'
TIMEV='/bin/time'
USER='ghicks'
VERSION='3.2.0b6'
YACC='yacc'
ac_ct_AR='ar'
ac_ct_CC='gcc'
ac_ct_CXX='g++'
ac_ct_F77='g77'
ac_ct_RANLIB='ranlib'
ac_ct_STRIP='strip'
am__fastdepCC_FALSE=''
am__fastdepCC_TRUE='#'
am__fastdepCXX_FALSE=''
am__fastdepCXX_TRUE='#'
am__include='include'
am__leading_dot='.'
am__quote=''
bindir='${exec_prefix}/bin'
build='sparc-sun-solaris2.8'
build_alias=''
build_cpu='sparc'
build_os='solaris2.8'
build_vendor='sun'
datadir='${prefix}/share'
exec_prefix='NONE'
extra_ldflags=''
host='sparc-sun-solaris2.8'
host_alias=''
host_cpu='sparc'
host_os='solaris2.8'
host_vendor='sun'
includedir='${prefix}/include'
infodir='${prefix}/info'
install_sh='/home/ghicks/incoming/htdig/htdig-3.2.0b6/install-sh'
libdir='${exec_prefix}/lib'
libexecdir='${exec_prefix}/libexec'
localstatedir='${prefix}/var'
mandir='${prefix}/man'
oldincludedir='/usr/include'
prefix='/usr/local/apache-1.3.9'
program_transform_name='s,x,x,'
sbindir='${exec_prefix}/sbin'
sharedstatedir='${prefix}/com'
subdirs=''
sysconfdir='${prefix}/etc'
target_alias=''

## ----------- ##
## confdefs.h. ##
## ----------- ##

#define HAVE_ALLOCA 1
#define HAVE_ALLOCA_H 1
#define HAVE_ALLOCA_H 1
#define HAVE_ARPA_INET_H 1
#define HAVE_DIRENT_H 1
#define HAVE_DLFCN_H 1
#define HAVE_FCNTL_H 1
#define HAVE_FLOAT_H 1
#define HAVE_INTTYPES_H 1
#define HAVE_LANGINFO_H 1
#define HAVE_LIBINTL_H 1
#define HAVE_LIBNSL 1
#define HAVE_LIBSOCKET 1
#define HAVE_LIBZ 1
#define HAVE_LIMITS_H 1
#define HAVE_LOCALE_H 1
#define HAVE_MALLOC_H 1
#define HAVE_MEMORY_H 1
#define HAVE_NETDB_H 1
#define HAVE_NETINET_IN_H 1
#define HAVE_STDBOOL_H 1
#define HAVE_STDBOOL_H 1
#define HAVE_STDDEF_H 1
#define HAVE_STDLIB_H 1
#define HAVE_STDLIB_H 1
#define HAVE_STRINGS_H 1
#define HAVE_STRINGS_H 1
#define HAVE_STRING_H 1
#define HAVE_STRING_H 1
#define HAVE_STRUCT_STAT_ST_BLKSIZE 1
#define HAVE_SYSLOG_H 1
#define HAVE_SYS_FILE_H 1
#define HAVE_SYS_IOCTL_H 1
#define HAVE_SYS_SOCKET_H 1
#define HAVE_SYS_STAT_H 1
#define HAVE_SYS_TIMEB_H 1
#define HAVE_SYS_TIME_H 1
#define HAVE_SYS_TYPES_H 1
#define HAVE_SYS_WAIT_H 1
#define HAVE_SYS_WAIT_H 1
#define HAVE_UNISTD_H 1
#define HAVE_UNISTD_H 1
#define HAVE_WAIT_H 1
#define HAVE_WCHAR_H 1
#define HAVE_WCTYPE_H 1
#define HAVE_ZLIB_H
#define HAVE__BOOL 1
#define HAVE__BOOL 1
#define PACKAGE "htdig"
#define PACKAGE_BUGREPORT ""
#define PACKAGE_NAME ""
#define PACKAGE_STRING ""
#define PACKAGE_TARNAME ""
#define PACKAGE_VERSION ""
#define STDC_HEADERS 1
#define STDC_HEADERS 1
#define TIME_WITH_SYS_TIME 1
#define TIME_WITH_SYS_TIME 1
#define VERSION "3.2.0b6"
#endif
#ifdef __cplusplus
extern "C" void std::exit (int) throw (); using std::exit;

configure: exit 1

> 
> Here are the last contents of config.log:
> 
> MAKEINFO='${SHELL} /home/ghicks/incoming/htdig/htdig-3.2.0b6/missing --run 
> makeinfo'
> MV='/bin/mv'
> OBJEXT='o'
> PACKAGE='htdig'
> PACKAGE_BUGREPORT=''
> PACKAGE_NAME=''
> PACKAGE_STRING=''
> PACKAGE_TARNAME=''
> PACKAGE_VERSION=''
> PATH_SEPARATOR=':'
> PERL='/bin/perl'
> RANLIB='ranlib'
> RRDTOOL=''
> SEARCH_DIR='${prefix}/htdocs/htdig'
> SEARCH_FORM='search.html'
> SED='/usr/xpg4/bin/sed'
> SENDMAIL='/usr/lib/sendmail'
> SET_MAKE=''
> SHELL='/bin/bash'
> STRIP='strip'
> TAR='tar'
> TESTS_FALSE=''
> TESTS_TRUE='#'
> TIME='/bin/time'
> TIMEV='/bin/time'
> USER='ghicks'
> VERSION='3.2.0b6'
> YACC='yacc'
> ac_ct_AR='ar'
> ac_ct_CC='gcc'
> ac_ct_CXX='g++'
> ac_ct_F77='g77'
> ac_ct_RANLIB='ranlib'
> ac_ct_STRIP='strip'
> am__fastdepCC_FALSE=''
> am__fastdepCC_TRUE='#'
> am__fastdepCXX_FALSE=''
> am__fastdepCXX_TRUE='#'
> am__include='include'
> am__leading_dot='.'
> am__quote=''
> bindir='${exec_prefix}/bin'
> build='sparc-sun-solaris2.8'
> build_alias=''
> build_cpu='sparc'
> build_os='solaris2.8'
> build_vendor='sun'
> datadir='${prefix}/share'
> exec_prefix='NONE'
> extra_ldflags=''
> host='sparc-sun-solaris2.8'
> host_alias=''
> host_cpu='sparc'
> host_os='solaris2.8'
> host_vendor='sun'
> includedir='${prefix}/include'
> infodir='${prefix}/info'
> install_sh='/home/ghicks/incoming/htdig/htdig-3.2.0b6/install-sh'
> libdir='${exec_prefix}/lib'
> libexecdir='${exec_prefix}/libexec'
> localstatedir='${prefix}/var'
> mandir='${prefix}/man'
> oldincludedir='/usr/include'
> prefix='/usr/local/apache-1.3.9'
> program_transform_name='s,x,x,'
> sbindir='${exec_prefix}/sbin'
> sharedstatedir='${prefix}/com'
> subdirs=''
> sysconfdir='${prefix}/etc'
> target_alias=''
> 
> ## ----------- ##
> ## confdefs.h. ##
> ## ----------- ##
> 
> #define HAVE_ALLOCA 1
> #define HAVE_ALLOCA_H 1
> #define HAVE_ALLOCA_H 1
> #define HAVE_ARPA_INET_H 1
> #define HAVE_DIRENT_H 1
> #define HAVE_DLFCN_H 1
> #define HAVE_FCNTL_H 1
> #define HAVE_FLOAT_H 1
> #define HAVE_INTTYPES_H 1
> #define HAVE_LANGINFO_H 1
> #define HAVE_LIBINTL_H 1
> #define HAVE_LIBNSL 1
> #define HAVE_LIBSOCKET 1
> #define HAVE_LIBZ 1
> #define HAVE_LIMITS_H 1
> #define HAVE_LOCALE_H 1
> #define HAVE_MALLOC_H 1
> #define HAVE_MEMORY_H 1
> #define HAVE_NETDB_H 1
> #define HAVE_NETINET_IN_H 1
> #define HAVE_STDBOOL_H 1
> #define HAVE_STDBOOL_H 1
> #define HAVE_STDDEF_H 1
> #define HAVE_STDLIB_H 1
> #define HAVE_STDLIB_H 1
> #define HAVE_STRINGS_H 1
> #define HAVE_STRINGS_H 1
> #define HAVE_STRING_H 1
> #define HAVE_STRING_H 1
> #define HAVE_STRUCT_STAT_ST_BLKSIZE 1
> #define HAVE_SYSLOG_H 1
> #define HAVE_SYS_FILE_H 1
> #define HAVE_SYS_IOCTL_H 1
> #define HAVE_SYS_SOCKET_H 1
> #define HAVE_SYS_STAT_H 1
> #define HAVE_SYS_TIMEB_H 1
> #define HAVE_SYS_TIME_H 1
> #define HAVE_SYS_TYPES_H 1
> #define HAVE_SYS_WAIT_H 1
> #define HAVE_SYS_WAIT_H 1
> #define HAVE_UNISTD_H 1
> #define HAVE_UNISTD_H 1
> #define HAVE_WAIT_H 1
> #define HAVE_WCHAR_H 1
> #define HAVE_WCTYPE_H 1
> #define HAVE_ZLIB_H
> #define HAVE__BOOL 1
> #define HAVE__BOOL 1
> #define PACKAGE "htdig"
> #define PACKAGE_BUGREPORT ""
> #define PACKAGE_NAME ""
> #define PACKAGE_STRING ""
> #define PACKAGE_TARNAME ""
> #define PACKAGE_VERSION ""
> #define STDC_HEADERS 1
> #define STDC_HEADERS 1
> #define TIME_WITH_SYS_TIME 1
> #define TIME_WITH_SYS_TIME 1
> #define VERSION "3.2.0b6"
> #endif
> #ifdef __cplusplus
> extern "C" void std::exit (int) throw (); using std::exit;
> 
> configure: exit 1
> 
> And the console output:
> 
> checking for a BSD-compatible install... ./install-sh -c
> checking whether build environment is sane... yes
> checking for gawk... gawk
> checking whether make sets $(MAKE)... yes
> checking whether to enable maintainer-specific portions of Makefiles... no
> checking for g++... g++
> checking for C++ compiler default output file name... a.out
> checking whether the C++ compiler works... yes
> checking whether we are cross compiling... no
> checking for suffix of executables...
> checking for suffix of object files... o
> checking whether we are using the GNU C++ compiler... yes
> checking whether g++ accepts -g... yes
> checking for style of include used by make... GNU
> checking dependency style of g++... none
> checking for gcc... gcc
> checking whether we are using the GNU C compiler... yes
> checking whether gcc accepts -g... yes
> checking for gcc option to accept ANSI C... none needed
> checking dependency style of gcc... none
> checking how to run the C preprocessor... gcc -E
> checking for egrep... egrep
> checking for AIX... no
> checking for flex... no
> checking for lex... lex
> checking for yywrap in -lfl... no
> checking for yywrap in -ll... yes
> checking lex output file root... lex.yy
> checking whether yytext is a pointer... no
> checking for bison... no
> checking for byacc... no
> checking for a BSD-compatible install... ./install-sh -c
> checking whether ln -s works... yes
> checking whether make sets $(MAKE)... (cached) yes
> checking build system type... sparc-sun-solaris2.8
> checking host system type... sparc-sun-solaris2.8
> [...snip...]
> checking whether stat file-mode macros are broken... no
> checking for stdbool.h that conforms to C99... (cached) yes
> checking for _Bool... (cached) yes
> checking for an ANSI C-conforming const... yes
> checking for inline... inline
> checking for off_t... yes
> checking for size_t... yes
> checking for struct stat.st_blksize... yes
> checking whether time.h and sys/time.h may both be included... (cached) yes
> checking whether struct tm is in sys/time.h or time.h... time.h
> checking for unsigned long long int... yes
> checking size of unsigned long long int...
> 
> Any thoughts on what to modify?  I don't have a clue.
> 
> Regards,
> Gregory Hicks
> 
> 
> -------------------------------------------------------------------
> Gregory Hicks                        | Principal Systems Engineer
> Cadence Design Systems               | Direct:   408.576.3609
> 555 River Oaks Pkwy M/S 6B1          | Fax:      408.894.3400
> San Jose, CA 95134                   | Internet: gh...@ca...
> 
> I am perfectly capable of learning from my mistakes.  I will surely
> learn a great deal today.
> 
> "A democracy is a sheep and two wolves deciding on what to have for
> lunch.  Freedom is a well armed sheep contesting the results of the
> decision." - Benjamin Franklin
> 
> "The best we can hope for concerning the people at large is that they
> be properly armed." --Alexander Hamilton
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> ht://Dig general mailing list: <htd...@li...>
> ht://Dig FAQ: http://htdig.sourceforge.net/FAQ.html
> List information (subscribe/unsubscribe, etc.)
> https://lists.sourceforge.net/lists/listinfo/htdig-general

-------------------------------------------------------------------
Gregory Hicks                        | Principal Systems Engineer
Cadence Design Systems               | Direct:   408.576.3609
555 River Oaks Pkwy M/S 6B1          | Fax:      408.894.3400
San Jose, CA 95134                   | Internet: gh...@ca...

I am perfectly capable of learning from my mistakes.  I will surely
learn a great deal today.

"A democracy is a sheep and two wolves deciding on what to have for
lunch.  Freedom is a well armed sheep contesting the results of the
decision." - Benjamin Franklin

"The best we can hope for concerning the people at large is that they
be properly armed." --Alexander Hamilton

[htdig-dev] configure: error: cannot compute sizeof (unsigned long long int), 77

[Gregory H. <gh...@ca...>]

Greetings:

I am trying to configure htdig-3.2.0b6 on a Solaris 8 box with gcc 3.3.1 and 
getting this error:

configure: error: cannot compute sizeof (unsigned long long int), 77
See `config.log' for more details.

(When I tried to configure htdig-3.1.6, the configure script could not find 
libstdc++ even though it exists:

/usr/local/lib:
libncurses++.a*      libstdc++.la*        libstdc++.so.5.0.5*
libstdc++.a          libstdc++.so@        libsupc++.a
libstdc++.a.2.10.0   libstdc++.so.5@      libsupc++.la*

and /usr/local/lib IS in LD_LIBRARY_PATH...)

Here are the last contents of config.log:

MAKEINFO='${SHELL} /home/ghicks/incoming/htdig/htdig-3.2.0b6/missing --run 
makeinfo'
MV='/bin/mv'
OBJEXT='o'
PACKAGE='htdig'
PACKAGE_BUGREPORT=''
PACKAGE_NAME=''
PACKAGE_STRING=''
PACKAGE_TARNAME=''
PACKAGE_VERSION=''
PATH_SEPARATOR=':'
PERL='/bin/perl'
RANLIB='ranlib'
RRDTOOL=''
SEARCH_DIR='${prefix}/htdocs/htdig'
SEARCH_FORM='search.html'
SED='/usr/xpg4/bin/sed'
SENDMAIL='/usr/lib/sendmail'
SET_MAKE=''
SHELL='/bin/bash'
STRIP='strip'
TAR='tar'
TESTS_FALSE=''
TESTS_TRUE='#'
TIME='/bin/time'
TIMEV='/bin/time'
USER='ghicks'
VERSION='3.2.0b6'
YACC='yacc'
ac_ct_AR='ar'
ac_ct_CC='gcc'
ac_ct_CXX='g++'
ac_ct_F77='g77'
ac_ct_RANLIB='ranlib'
ac_ct_STRIP='strip'
am__fastdepCC_FALSE=''
am__fastdepCC_TRUE='#'
am__fastdepCXX_FALSE=''
am__fastdepCXX_TRUE='#'
am__include='include'
am__leading_dot='.'
am__quote=''
bindir='${exec_prefix}/bin'
build='sparc-sun-solaris2.8'
build_alias=''
build_cpu='sparc'
build_os='solaris2.8'
build_vendor='sun'
datadir='${prefix}/share'
exec_prefix='NONE'
extra_ldflags=''
host='sparc-sun-solaris2.8'
host_alias=''
host_cpu='sparc'
host_os='solaris2.8'
host_vendor='sun'
includedir='${prefix}/include'
infodir='${prefix}/info'
install_sh='/home/ghicks/incoming/htdig/htdig-3.2.0b6/install-sh'
libdir='${exec_prefix}/lib'
libexecdir='${exec_prefix}/libexec'
localstatedir='${prefix}/var'
mandir='${prefix}/man'
oldincludedir='/usr/include'
prefix='/usr/local/apache-1.3.9'
program_transform_name='s,x,x,'
sbindir='${exec_prefix}/sbin'
sharedstatedir='${prefix}/com'
subdirs=''
sysconfdir='${prefix}/etc'
target_alias=''

## ----------- ##
## confdefs.h. ##
## ----------- ##

#define HAVE_ALLOCA 1
#define HAVE_ALLOCA_H 1
#define HAVE_ALLOCA_H 1
#define HAVE_ARPA_INET_H 1
#define HAVE_DIRENT_H 1
#define HAVE_DLFCN_H 1
#define HAVE_FCNTL_H 1
#define HAVE_FLOAT_H 1
#define HAVE_INTTYPES_H 1
#define HAVE_LANGINFO_H 1
#define HAVE_LIBINTL_H 1
#define HAVE_LIBNSL 1
#define HAVE_LIBSOCKET 1
#define HAVE_LIBZ 1
#define HAVE_LIMITS_H 1
#define HAVE_LOCALE_H 1
#define HAVE_MALLOC_H 1
#define HAVE_MEMORY_H 1
#define HAVE_NETDB_H 1
#define HAVE_NETINET_IN_H 1
#define HAVE_STDBOOL_H 1
#define HAVE_STDBOOL_H 1
#define HAVE_STDDEF_H 1
#define HAVE_STDLIB_H 1
#define HAVE_STDLIB_H 1
#define HAVE_STRINGS_H 1
#define HAVE_STRINGS_H 1
#define HAVE_STRING_H 1
#define HAVE_STRING_H 1
#define HAVE_STRUCT_STAT_ST_BLKSIZE 1
#define HAVE_SYSLOG_H 1
#define HAVE_SYS_FILE_H 1
#define HAVE_SYS_IOCTL_H 1
#define HAVE_SYS_SOCKET_H 1
#define HAVE_SYS_STAT_H 1
#define HAVE_SYS_TIMEB_H 1
#define HAVE_SYS_TIME_H 1
#define HAVE_SYS_TYPES_H 1
#define HAVE_SYS_WAIT_H 1
#define HAVE_SYS_WAIT_H 1
#define HAVE_UNISTD_H 1
#define HAVE_UNISTD_H 1
#define HAVE_WAIT_H 1
#define HAVE_WCHAR_H 1
#define HAVE_WCTYPE_H 1
#define HAVE_ZLIB_H
#define HAVE__BOOL 1
#define HAVE__BOOL 1
#define PACKAGE "htdig"
#define PACKAGE_BUGREPORT ""
#define PACKAGE_NAME ""
#define PACKAGE_STRING ""
#define PACKAGE_TARNAME ""
#define PACKAGE_VERSION ""
#define STDC_HEADERS 1
#define STDC_HEADERS 1
#define TIME_WITH_SYS_TIME 1
#define TIME_WITH_SYS_TIME 1
#define VERSION "3.2.0b6"
#endif
#ifdef __cplusplus
extern "C" void std::exit (int) throw (); using std::exit;

configure: exit 1

And the console output:

checking for a BSD-compatible install... ./install-sh -c
checking whether build environment is sane... yes
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether to enable maintainer-specific portions of Makefiles... no
checking for g++... g++
checking for C++ compiler default output file name... a.out
checking whether the C++ compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking for style of include used by make... GNU
checking dependency style of g++... none
checking for gcc... gcc
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ANSI C... none needed
checking dependency style of gcc... none
checking how to run the C preprocessor... gcc -E
checking for egrep... egrep
checking for AIX... no
checking for flex... no
checking for lex... lex
checking for yywrap in -lfl... no
checking for yywrap in -ll... yes
checking lex output file root... lex.yy
checking whether yytext is a pointer... no
checking for bison... no
checking for byacc... no
checking for a BSD-compatible install... ./install-sh -c
checking whether ln -s works... yes
checking whether make sets $(MAKE)... (cached) yes
checking build system type... sparc-sun-solaris2.8
checking host system type... sparc-sun-solaris2.8
[...snip...]
checking whether stat file-mode macros are broken... no
checking for stdbool.h that conforms to C99... (cached) yes
checking for _Bool... (cached) yes
checking for an ANSI C-conforming const... yes
checking for inline... inline
checking for off_t... yes
checking for size_t... yes
checking for struct stat.st_blksize... yes
checking whether time.h and sys/time.h may both be included... (cached) yes
checking whether struct tm is in sys/time.h or time.h... time.h
checking for unsigned long long int... yes
checking size of unsigned long long int...

Any thoughts on what to modify?  I don't have a clue.

Regards,
Gregory Hicks


-------------------------------------------------------------------
Gregory Hicks                        | Principal Systems Engineer
Cadence Design Systems               | Direct:   408.576.3609
555 River Oaks Pkwy M/S 6B1          | Fax:      408.894.3400
San Jose, CA 95134                   | Internet: gh...@ca...

I am perfectly capable of learning from my mistakes.  I will surely
learn a great deal today.

"A democracy is a sheep and two wolves deciding on what to have for
lunch.  Freedom is a well armed sheep contesting the results of the
decision." - Benjamin Franklin

"The best we can hope for concerning the people at large is that they
be properly armed." --Alexander Hamilton

Re: [htdig-dev] Compression related performance

[Jim <li...@yg...>]

On Sat, 15 Jan 2005, Neal Richter wrote:

> Question #1)
>
>   Could you download a fresh copy of BDB 3.0.55 from
> http://www.sleepycat.com/download/index.shtml
>
> It's easy to build, I am most interested in the results when you run db_stat 
> on the uncompressed files and report back the fill-factor.  This is a measure 
> of wasted space in the BDB pages.  This won't work on the compressed index.

How exactly do I obtain this information? In the entire package I see only
one place that references fill-factor (diskspace.html) and all it says is
that you can obtain it with db_stat. However I don't see anything called
fill-factor in the db_stat output. What options are needed to extract this
info? Or is it a value that needs to be computed from other output?

Jim

Re: [htdig-dev] Compression related performance

[Joe R. J. <jj...@cl...>]

On Tue, 18 Jan 2005, Neal Richter wrote:

> Date: Tue, 18 Jan 2005 11:47:40 -0700 (MST)
> From: Neal Richter <ne...@ri...>
> To: Joe R. Jah <jj...@cl...>
> Cc: Jim <li...@yg...>, htd...@li...
> Subject: Re: [htdig-dev] Compression related performance
> 
> 
> >> Jim,
> >>  	Thanks for the numbers.  Pretty different from Joe's.
> >
> > Yes they are; can you explain it?
> 
>    #1)  What is the size of  wordlist_cache_size?

I haven't specified it in my htdig.conf file, so it must be the default
10000000.

>    #2)  wordlist_page_size?

I did vary that from 0, 1024, 2048, and 4096.  The optimum number happens
to be 2048 in my system; however, when it comes to compression and no
compression my system performance has been consistently better with
compression, regardless of this factor.

>    #3)  Are either of you using a journaling filesystem?

I do not.

>    #4)  Does one of you have somekind of high performance disk-drive?

Here is my disk-drive system:

RAID 5 10K RPM, 1" Ultra3 (Ultra 160) SCSI Hard Drives.

> I'm pretty well stumped on this one if none of these 4 is different 
> between you... we know the process is very I/O bound, so if one is much 
> faster and all other things are equal...
> 
> -- 
> Neal Richter 
> Knowledgebase Developer
> RightNow Technologies, Inc.
> Customer Service for Every Web Site
> Office: 406-522-1485

Regards,

Joe
-- 
     _/   _/_/_/       _/              ____________    __o
     _/   _/   _/      _/         ______________     _-\<,_
 _/  _/   _/_/_/   _/  _/                     ......(_)/ (_)
  _/_/ oe _/   _/.  _/_/ ah        jj...@cl...

Re: [htdig-dev] Compression related performance

[Neal R. <ne...@ri...>]

>> Jim,
>>  	Thanks for the numbers.  Pretty different from Joe's.
>
> Yes they are; can you explain it?

   #1)  What is the size of  wordlist_cache_size?

   #2)  wordlist_page_size?

   #3)  Are either of you using a journaling filesystem?

   #4)  Does one of you have somekind of high performance disk-drive?

I'm pretty well stumped on this one if none of these 4 is different 
between you... we know the process is very I/O bound, so if one is much 
faster and all other things are equal...

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] Compression related performance

[Joe R. J. <jj...@cl...>]

On Sat, 15 Jan 2005, Neal Richter wrote:

> Date: Sat, 15 Jan 2005 16:06:09 -0700 (MST)
> From: Neal Richter <ne...@ri...>
> To: Jim <li...@yg...>
> Cc: htd...@li...
> Subject: Re: [htdig-dev] Compression related performance
> 
> Jim,
>  	Thanks for the numbers.  Pretty different from Joe's.

Yes they are; can you explain it?

> Question #1)
> 
>     Could you download a fresh copy of BDB 3.0.55 from
> http://www.sleepycat.com/download/index.shtml
> 
> It's easy to build, I am most interested in the results when you run 
> db_stat on the uncompressed files and report back the fill-factor.  This 
> is a measure of wasted space in the BDB pages.  This won't work on the 
> compressed index.
> 
> Question #2)
>    What page_size are you using?
> 
> Question #3)
>    Do you ever see any errors when using the compressed index?
> 
>    Joe:  Do you?

No I don't.

Regards,

Joe
-- 
     _/   _/_/_/       _/              ____________    __o
     _/   _/   _/      _/         ______________     _-\<,_
 _/  _/   _/_/_/   _/  _/                     ......(_)/ (_)
  _/_/ oe _/   _/.  _/_/ ah        jj...@cl...

> Thanks
> 
> Neal
> 
> On Fri, 14 Jan 2005, Jim wrote:
> 
> > Here is another performance comparison collected when the server was under 
> > less load. The data is essentially the same as that used for the last set of 
> > numbers.
> >
> > Uncompressed
> > ------------
> > 416.28 user
> > 97.72 system
> > 1:30:23 elapsed
> >
> >  10510336 Jan 15 00:00 db.docdb
> >  10444800 Jan 15 00:00 db.docs.index
> >  90488832 Jan 15 00:00 db.excerpts
> >  436043776 Jan 15 00:00 db.words.db
> >
> > Compressed
> > ----------
> > 2008.63 user
> > 133.35 system
> > 2:40:16 elapsed
> >
> >  10526720 Jan 14 09:35 db.docdb
> >  10444800 Jan 14 09:35 db.docs.index
> >  90611712 Jan 14 09:36 db.excerpts
> >  113773568 Jan 14 09:36 db.words.db
> >     16384 Jan 14 09:36 db.words.db_weakcmpr
> >
> >
> > I do have a large data set that I could test with if it would be of real
> > value. However I would need to try rebuilding ht://Dig with large file
> > support before I could consider dropping compression for that set.
> >
> >
> > Jim
> >
> >
> > -------------------------------------------------------
> > The SF.Net email is sponsored by: Beat the post-holiday blues
> > Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
> > It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
> > _______________________________________________
> > ht://Dig Developer mailing list:
> > htd...@li...
> > List information (subscribe/unsubscribe, etc.)
> > https://lists.sourceforge.net/lists/listinfo/htdig-dev
> >
> 
> -- 
> Neal Richter 
> Knowledgebase Developer
> RightNow Technologies, Inc.
> Customer Service for Every Web Site
> Office: 406-522-1485

Re: [htdig-dev] Compression related performance

[Neal R. <ne...@ri...>]

Jim,
 	Thanks for the numbers.  Pretty different from Joe's.

Question #1)

    Could you download a fresh copy of BDB 3.0.55 from
http://www.sleepycat.com/download/index.shtml

It's easy to build, I am most interested in the results when you run 
db_stat on the uncompressed files and report back the fill-factor.  This 
is a measure of wasted space in the BDB pages.  This won't work on the 
compressed index.

Question #2)
   What page_size are you using?

Question #3)
   Do you ever see any errors when using the compressed index?

   Joe:  Do you?

Thanks

Neal

On Fri, 14 Jan 2005, Jim wrote:

> Here is another performance comparison collected when the server was under 
> less load. The data is essentially the same as that used for the last set of 
> numbers.
>
> Uncompressed
> ------------
> 416.28 user
> 97.72 system
> 1:30:23 elapsed
>
>  10510336 Jan 15 00:00 db.docdb
>  10444800 Jan 15 00:00 db.docs.index
>  90488832 Jan 15 00:00 db.excerpts
>  436043776 Jan 15 00:00 db.words.db
>
> Compressed
> ----------
> 2008.63 user
> 133.35 system
> 2:40:16 elapsed
>
>  10526720 Jan 14 09:35 db.docdb
>  10444800 Jan 14 09:35 db.docs.index
>  90611712 Jan 14 09:36 db.excerpts
>  113773568 Jan 14 09:36 db.words.db
>     16384 Jan 14 09:36 db.words.db_weakcmpr
>
>
> I do have a large data set that I could test with if it would be of real
> value. However I would need to try rebuilding ht://Dig with large file
> support before I could consider dropping compression for that set.
>
>
> Jim
>
>
> -------------------------------------------------------
> The SF.Net email is sponsored by: Beat the post-holiday blues
> Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
> It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
> _______________________________________________
> ht://Dig Developer mailing list:
> htd...@li...
> List information (subscribe/unsubscribe, etc.)
> https://lists.sourceforge.net/lists/listinfo/htdig-dev
>

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

[htdig-dev] Compression related performance

[Jim <li...@yg...>]

Here is another performance comparison collected when the server was under 
less load. The data is essentially the same as that used for the last set 
of numbers.

Uncompressed
------------
416.28 user
97.72 system
1:30:23 elapsed

   10510336 Jan 15 00:00 db.docdb
   10444800 Jan 15 00:00 db.docs.index
   90488832 Jan 15 00:00 db.excerpts
   436043776 Jan 15 00:00 db.words.db

Compressed
----------
2008.63 user
133.35 system
2:40:16 elapsed

   10526720 Jan 14 09:35 db.docdb
   10444800 Jan 14 09:35 db.docs.index
   90611712 Jan 14 09:36 db.excerpts
   113773568 Jan 14 09:36 db.words.db
      16384 Jan 14 09:36 db.words.db_weakcmpr


I do have a large data set that I could test with if it would be of real
value. However I would need to try rebuilding ht://Dig with large file
support before I could consider dropping compression for that set.


Jim

Re: [htdig-dev] Re: [htdig-members] Results of Possible Efficiency improvment for 3.2.0

[Jim <li...@yg...>]

On Thu, 13 Jan 2005, Joe R. Jah wrote:

>>    10592256 Jan 10 09:46 db.docdb
>>    10461184 Jan 10 09:44 db.docs.index
>>    90120192 Jan 10 09:44 db.excerpts
>>    113465344 Jan 10 09:44 db.words.db
>>       16384 Jan 10 09:44 db.words.db_weakcmpr
>
> Not that it makes much of a difference, but it looks like you do not
> compress the excerpts, compression_level: 0; right?

Nope. I am using the default, which I believe is 6. However I am using a
non-default max_head_length (15000). Perhaps that accounts for the
difference in what you expected.

> Hmmm, no swapping!  That may be the deciding factor why the change in your
> elapsed time is so different than mine; or may be your CPU speed;-/
>
> My system is similar to yours, dual Xeon 2.4 GHz box RAID 5 10K RPM SCSI
> drives, and 2.0 GB DDR RAM; What's your CPU speed?

Dual Xeon 2.8 GHz. In terms of hardware, everything else sounds the same.
I did run the original tests under moderate load. I will try them again at
quieter times and see if that makes any difference.

Jim

Re: [htdig-dev] Re: [htdig-members] Results of Possible Efficiency improvment for 3.2.0

[Jim <li...@yg...>]

On Thu, 13 Jan 2005, Neal Richter wrote:

>>> 	As far as speed testing I would love it if a couple of you could 
>>> index a decent sized site (pefferable local to eliminate network 
>>> latency) with and without the index compression enabled and report back 
>>> your speed & size results.
>>> 
>>> wordlist_compress:         false
>>> wordlist_compress_zlib:    false
>> 
>> Here is what I got.
>> 
>> With compression:
>> 2296.03 user
>> 224.52 system
>> 8:10:56 elapsed
>> 
>> 10592256 Jan 10 09:46 db.docdb
>> 10461184 Jan 10 09:44 db.docs.index
>> 90120192 Jan 10 09:44 db.excerpts
>> 113465344 Jan 10 09:44 db.words.db
>> 16384 Jan 10 09:44 db.words.db_weakcmpr
>> 
>> Without compression:
>> 450.46 user
>> 123.33 system
>> 4:56:01 elapsed
>
>  Wow.. quite a difference.
>
>
>> Misplaced the file sizes for this run. I can rerun it some night if it
>> is important.
>
>   If you can some time..

   10526720 Jan 14 00:13 db.docdb
   10428416 Jan 14 00:13 db.docs.index
   90021888 Jan 14 00:13 db.excerpts
   435798016 Jan 14 00:12 db.words.db


Jim

Re: [htdig-dev] Re: [htdig-members] Results of Possible Efficiency improvment for 3.2.0

[Joe R. J. <jj...@cl...>]

On Wed, 12 Jan 2005, Jim wrote:

> Date: Wed, 12 Jan 2005 23:02:12 -0700 (MST)
> From: Jim <li...@yg...>
> To: Neal Richter <ne...@ri...>
> Cc: htd...@li...
> Subject: [htdig-dev] Re: [htdig-members] Results of Possible Efficiency improvment for 3.2.0
> 
> Moved this to developers list.
> 
> On Wed, 5 Jan 2005, Neal Richter wrote:
> 
> > 	As far as speed testing I would love it if a couple of you could 
> > index a decent sized site (pefferable local to eliminate network latency) 
> > with and without the index compression enabled and report back your speed & 
> > size results.
> >
> > wordlist_compress:         false
> > wordlist_compress_zlib:    false
> 
> Here is what I got.
> 
> With compression:
>    2296.03 user
>    224.52 system
>    8:10:56 elapsed
> 
>    10592256 Jan 10 09:46 db.docdb
>    10461184 Jan 10 09:44 db.docs.index
>    90120192 Jan 10 09:44 db.excerpts
>    113465344 Jan 10 09:44 db.words.db
>       16384 Jan 10 09:44 db.words.db_weakcmpr

Not that it makes much of a difference, but it looks like you do not
compress the excerpts, compression_level: 0; right?

> Without compression:
>    450.46 user
>    123.33 system
>    4:56:01 elapsed
> 
>    Misplaced the file sizes for this run. I can rerun it some night if it
>    is important.

Add a `/bin/ls -l $DBDIR` line at the end of your rundig-test script, and
never misplace them;)

> These numbers are for a local dig on a dual Xeon box with RAID 5 and a
> couple GB of RAM. No swapping occurred.

Hmmm, no swapping!  That may be the deciding factor why the change in your
elapsed time is so different than mine; or may be your CPU speed;-/

My system is similar to yours, dual Xeon 2.4 GHz box RAID 5 10K RPM SCSI
drives, and 2.0 GB DDR RAM; What's your CPU speed?

Regards,

Joe
-- 
     _/   _/_/_/       _/              ____________    __o
     _/   _/   _/      _/         ______________     _-\<,_
 _/  _/   _/_/_/   _/  _/                     ......(_)/ (_)
  _/_/ oe _/   _/.  _/_/ ah        jj...@cl...

[htdig-dev] Re: [htdig-members] Results of Possible Efficiency improvment for 3.2.0

[Neal R. <ne...@ri...>]

On Wed, 12 Jan 2005, Jim wrote:

> Moved this to developers list.
>
> On Wed, 5 Jan 2005, Neal Richter wrote:
>
>> 	As far as speed testing I would love it if a couple of you could 
>> index a decent sized site (pefferable local to eliminate network latency) 
>> with and without the index compression enabled and report back your speed & 
>> size results.
>> 
>> wordlist_compress:         false
>> wordlist_compress_zlib:    false
>
> Here is what I got.
>
> With compression:
>  2296.03 user
>  224.52 system
>  8:10:56 elapsed
>
>  10592256 Jan 10 09:46 db.docdb
>  10461184 Jan 10 09:44 db.docs.index
>  90120192 Jan 10 09:44 db.excerpts
>  113465344 Jan 10 09:44 db.words.db
>     16384 Jan 10 09:44 db.words.db_weakcmpr
>
> Without compression:
>  450.46 user
>  123.33 system
>  4:56:01 elapsed

   Wow.. quite a difference.


>  Misplaced the file sizes for this run. I can rerun it some night if it
>  is important.

    If you can some time..

   Thanks.

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

[htdig-dev] Re: [htdig-members] Results of Possible Efficiency improvment for 3.2.0

[Jim <li...@yg...>]

Moved this to developers list.

On Wed, 5 Jan 2005, Neal Richter wrote:

> 	As far as speed testing I would love it if a couple of you could 
> index a decent sized site (pefferable local to eliminate network latency) 
> with and without the index compression enabled and report back your speed & 
> size results.
>
> wordlist_compress:         false
> wordlist_compress_zlib:    false

Here is what I got.

With compression:
   2296.03 user
   224.52 system
   8:10:56 elapsed

   10592256 Jan 10 09:46 db.docdb
   10461184 Jan 10 09:44 db.docs.index
   90120192 Jan 10 09:44 db.excerpts
   113465344 Jan 10 09:44 db.words.db
      16384 Jan 10 09:44 db.words.db_weakcmpr

Without compression:
   450.46 user
   123.33 system
   4:56:01 elapsed

   Misplaced the file sizes for this run. I can rerun it some night if it
   is important.

These numbers are for a local dig on a dual Xeon box with RAID 5 and a
couple GB of RAM. No swapping occurred.

Jim

[htdig-dev] Setup Mirror

[Marcus T. M. <mo...@yo...>]

Hi Guys,

I have setup a mirror for ht://Dig as I like the product, and figure that
a mirror is always helping.

I am mirroring the maindocs and the cvs stuff for you.  You can find it
located at

http://htdig.youknowtherest.net/ - This is the maindocs website
http://htdig.youknowtherest.net/htdig-3-1-x/ - this is one of the cvs'
http://htdig.youknowtherest.net/htdig-3-1-x/ - this is the other cvs

If there is something wrong with the way I have done it, or if you would
like anything changed, please let me know.

Kind regards

Marcus McGough

[htdig-dev] Italian mirror is now fully functional

[Marco N. <mn...@pr...>]

The italian mirror (www.it.htdig.org) now support full ftp and http
mirror (even patch and files) and is updated daily.

The new mirror script use puf to get files over http from mainsite,
and use the index.html files to remove old files.

The URIS are: 
http://www.it.htdig.org/files/
http://www.it.htdig.org/htdig-patches/
ftp://ftp.it.htdig.org/pub/htdig/files
ftp://ftp.it.htdig.org/pub/htdig/htdig-patches

The mirror script is attached.

-- 
---------------------------------------------------------------------
|    Marco Nenciarini    | Debian/GNU Linux Developer - Plug Member |
| mn...@pr... | http://www.prato.linux.it/~mnencia       |
---------------------------------------------------------------------
Key fingerprint = FED9 69C7 9E67 21F5 7D95  5270 6864 730D F095 E5E4

Re: [htdig-dev] divide and conquer

[Neal R. <ne...@ri...>]

Klaus,

Thanks for the info!

If we can get enough developers interested I've got 
an idea to dumb the current db code and move to using Lucene/Clucene. 
This should scale much better and remove a large chunk of code from active 
maintenance.

On Sun, 12 Dec 2004, minimoa wrote:

> I have been using htdig for years to index my local pc. htdig is part of the
> Suse linux distribution I use.
> On a 2.4GHz cpu indexing the entire pc took 2 hours with htdig 3.1.6, but with
> version 3.2.0b6 it takes much longer. The words database becomes htdig's
> bottleneck when it exceeds 200MB on my pc. So I divided the search domain
> according to local directory structure and got smaller words databases and
> faster digs. The disadvantage is a lot of config files.
>
> Searching itself is faster: 3000 hits for FAQ are found in 30 secs by 3.2.0b6
> while 3.1.6 needed 60 secs. And phrases are a quantum leap.
>
> Currently merging of databases is not any faster than indexing directly.
> So if database update cannot be accelerated due to limitations of db software,
> techniques for divide and conquer could be improved and should be made more
> common in documentation and configuration examples of htdig.
>
> Thanks for a great software
> Klaus

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

[htdig-dev] divide and conquer

[minimoa <mi...@gm...>]

I have been using htdig for years to index my local pc. htdig is part of the 
Suse linux distribution I use.
On a 2.4GHz cpu indexing the entire pc took 2 hours with htdig 3.1.6, but with 
version 3.2.0b6 it takes much longer. The words database becomes htdig's 
bottleneck when it exceeds 200MB on my pc. So I divided the search domain 
according to local directory structure and got smaller words databases and 
faster digs. The disadvantage is a lot of config files.

Searching itself is faster: 3000 hits for FAQ are found in 30 secs by 3.2.0b6 
while 3.1.6 needed 60 secs. And phrases are a quantum leap.

Currently merging of databases is not any faster than indexing directly.
So if database update cannot be accelerated due to limitations of db software, 
techniques for divide and conquer could be improved and should be made more 
common in documentation and configuration examples of htdig.

Thanks for a great software
Klaus

[htdig-dev] Installation request

[Constantinos L. <la...@gm...>]

Hi, 
I seek for a well known ht://dig user/developer to help me doing a
fresh installation of ht://dig on my linux (redhat 9) server.
ht://dig will replace my site http://www.allthecoins.com search
engines back-end and algos.

Please advice. (With a price quote)

Constantinos Laitsas