htdig-dev

[htdig-dev] Re: Cygwin words.db Compression

[Steve E. <Ste...@ch...>]

Hi Lachlan,

Thanks for the suggestion :-)  I tried it in all the environments I listed earlier, but it didn't appear to work using the default settings for the compression flags. Here's what htdb_dump reports for all attempts:

================
C:\htdig\bin>htdb_dump -Wz -p c:/htdig/var/htdig/db.words.db
htdb_dump: open: c:/htdig/var/htdig/db.words.db: No such file or directory

C:\htdig\bin>htdb_dump -W -p c:/htdig/var/htdig/db.words.db
htdb_dump: c:/htdig/var/htdig/db.words.db: file size not a multiple of the pages
ize
htdb_dump: open: c:/htdig/var/htdig/db.words.db: Invalid argument

C:\htdig\bin>
================

The db.words.db.work_weakcmpr file gets created now, and words.db has size to it, but it still seems like it's corrupt or something since I can't dump it. Perhaps I've used the wrong command? htsearch doesn't seem to like words.db either:

================
C:\htdig\bin>htsearch
Enter value for words: patients
WordDB: DB->cursor: method meaningless before open
Content-type: text/html
================

I'm working on the Win32 native build as Neal suggested. 

Thanx

>>> Lachlan Andrew <lh...@us...> 10/03/03 08:18AM >>>
Greetings Steve,

Thanks for the very clear bug report.  Someone else has the same 
problem.  It's bug #814268...

This may be my fault.  What happens if you replace the NULL in line 
806 of db/mp_cmpr.c  by  dbenv ?  That is, make it
     if(CDB_db_create(&dbp, dbenv, 0) != 0

That was changed to avoid the possibility of infinite loops, but is a 
bit of a kludge.  If making the change described above works, then 
I'll try to fix it properly.

Cheers,
Lachlan

On Fri, 3 Oct 2003 02:51, Steve Eidemiller wrote:
> I'm compiling htdig-3.2.0b4-20090928 under Cygwin 1.5.5 using gcc
> 3.3.1, on both Windows XP Pro SP1 and Windows 2000 Server SP4.
> db.words.db is always a zero length file

-- 
lh...@us... 
ht://Dig developer DownUnder  (http://www.htdig.org)
__________________________________

Confidentiality Statement:
This email/fax, including attachments, may include confidential and/or proprietary information and may be used only by the person or entity to which it is addressed. If the reader of this email/fax is not the intended recipient or his or her agent, the reader is hereby notified that any dissemination, distribution or copying of this email/fax is prohibited. If you have received this email/fax in error, please notify the sender by replying to this message and deleting this email or destroying this facsimile immediately.

Re: [htdig-dev] Cygwin words.db Compression

[Steve E. <Ste...@ch...>]

Neal,

I ran into a little trouble with the build using the htdig-3.2.0b4-20090928 full snapshot. Here were the steps I used:

1. Installed MinGW 3.1.0-1 (newbie alert!!)
2. Added mingw-zlib 1.1.4-1 to my Cygwin setup (per configure's suggestion)
3. Launched Cygwin
4. export CC='gcc -mno-cygwin'
5. export CXX='gcc -mno-cygwin'
6. ./configure --host=mingw32 --build=mingw32 --target=mingw32 --prefix=c:/htdig
7. make
8. Went to lunch :-)
9. Observed  the following output leading to the error(s):

================
 gcc -mno-cygwin -DHAVE_CONFIG_H -I. -I. -I. -I./../htlib -g -O2 -c os_oflags.c
 -DDLL_EXPORT -DPIC -o .libs/os_oflags.o
os_oflags.c: In function `CDB___db_omode':
os_oflags.c:69: error: `S_IRGRP' undeclared (first use in this function)
os_oflags.c:69: error: (Each undeclared identifier is reported only once
os_oflags.c:69: error: for each function it appears in.)
os_oflags.c:71: error: `S_IWGRP' undeclared (first use in this function)
os_oflags.c:73: error: `S_IROTH' undeclared (first use in this function)
os_oflags.c:75: error: `S_IWOTH' undeclared (first use in this function)
make[2]: *** [os_oflags.lo] Error 1
make[2]: Leaving directory `/home/htdig320b4/htdig-3.2.0b4-20030928-Win32/db'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/home/htdig320b4/htdig-3.2.0b4-20030928-Win32/db'
make: *** [all-recursive] Error 1
================

Looks like the #define's at the end of db_int.h aren't happening? They appear dependent on some Win32 flags. Do I need another export or configure directive for Win32? Thanks for your patience, I'm a bit of a newbie here :)

Thanx
-Steve

>>> Neal Richter <ne...@ri...> 10/02/03 04:41PM >>>

Hey,
	I have produced a set of makefiles for a native windows binaries.
You do need cygwin to run 'make' (the makefiles are for GNU make).  The
makefiles use the Microsoft compiler.

	Could you get a copy of the latest snapshot and try and do the
build?  I'll work with you to get it fixed if it's still broken.

	We've tested older snapshots of HtDig compiled Win32 native and
run nearly a million documents through it....

	If this doesn't satisfy your needs, I'd be willing to put in some
time looking at the cygwin build.

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485


__________________________________

Confidentiality Statement:
This email/fax, including attachments, may include confidential and/or proprietary information and may be used only by the person or entity to which it is addressed. If the reader of this email/fax is not the intended recipient or his or her agent, the reader is hereby notified that any dissemination, distribution or copying of this email/fax is prohibited. If you have received this email/fax in error, please notify the sender by replying to this message and deleting this email or destroying this facsimile immediately.

Re: [htdig-dev] Cygwin words.db Compression

[Neal R. <ne...@ri...>]

I've seen this.  The short answer is don't use MinGW.  MinGW is missing
many of the things we take for granted in a 'normal' unix and that cygwin
supplies.  If you do some Googling you'll find some information to read
about what MinGW lacks.

Basically MinGW attempts to use the incomplete Posix.1 subsystem in WinNT
do provide a unix like environment.  Cygwin does a much more complete job
by finishing off the holes in WinNT's Posix.1 subsystem, fixing what is
buggy, and replacing what sucked.

Please try this in a cygwin shell:

cp ./db/db.h.win32 ./db/db.h
cp ./db/db_config.h.win32 ./db/db_config.h
cp ./include/htconfig.h.win32 ./include/htconfig.h

make -f Makefile.win32

That should fire off a build using Microsoft's compilers...

Thanks!  Neal.

On Fri, 3 Oct 2003, Steve Eidemiller wrote:

> Neal,
>
> I ran into a little trouble with the build using the htdig-3.2.0b4-20090928 full snapshot. Here were the steps I used:
>
> 1. Installed MinGW 3.1.0-1 (newbie alert!!)
> 2. Added mingw-zlib 1.1.4-1 to my Cygwin setup (per configure's suggestion)
> 3. Launched Cygwin
> 4. export CC='gcc -mno-cygwin'
> 5. export CXX='gcc -mno-cygwin'
> 6. ./configure --host=mingw32 --build=mingw32 --target=mingw32 --prefix=c:/htdig
> 7. make
> 8. Went to lunch :-)
> 9. Observed  the following output leading to the error(s):
>
> ================
>  gcc -mno-cygwin -DHAVE_CONFIG_H -I. -I. -I. -I./../htlib -g -O2 -c os_oflags.c
>  -DDLL_EXPORT -DPIC -o .libs/os_oflags.o
> os_oflags.c: In function `CDB___db_omode':
> os_oflags.c:69: error: `S_IRGRP' undeclared (first use in this function)
> os_oflags.c:69: error: (Each undeclared identifier is reported only once
> os_oflags.c:69: error: for each function it appears in.)
> os_oflags.c:71: error: `S_IWGRP' undeclared (first use in this function)
> os_oflags.c:73: error: `S_IROTH' undeclared (first use in this function)
> os_oflags.c:75: error: `S_IWOTH' undeclared (first use in this function)
> make[2]: *** [os_oflags.lo] Error 1
> make[2]: Leaving directory `/home/htdig320b4/htdig-3.2.0b4-20030928-Win32/db'
> make[1]: *** [all] Error 2
> make[1]: Leaving directory `/home/htdig320b4/htdig-3.2.0b4-20030928-Win32/db'
> make: *** [all-recursive] Error 1
> ================
>
> Looks like the #define's at the end of db_int.h aren't happening? They appear dependent on some Win32 flags. Do I need another export or configure directive for Win32? Thanks for your patience, I'm a bit of a newbie here :)
>
> Thanx
> -Steve
>
> >>> Neal Richter <ne...@ri...> 10/02/03 04:41PM >>>
>
> Hey,
> 	I have produced a set of makefiles for a native windows binaries.
> You do need cygwin to run 'make' (the makefiles are for GNU make).  The
> makefiles use the Microsoft compiler.
>
> 	Could you get a copy of the latest snapshot and try and do the
> build?  I'll work with you to get it fixed if it's still broken.
>
> 	We've tested older snapshots of HtDig compiled Win32 native and
> run nearly a million documents through it....
>
> 	If this doesn't satisfy your needs, I'd be willing to put in some
> time looking at the cygwin build.
>
> Neal Richter
> Knowledgebase Developer
> RightNow Technologies, Inc.
> Customer Service for Every Web Site
> Office: 406-522-1485
>
>
> __________________________________
>
> Confidentiality Statement:
> This email/fax, including attachments, may include confidential and/or proprietary information and may be used only by the person or entity to which it is addressed. If the reader of this email/fax is not the intended recipient or his or her agent, the reader is hereby notified that any dissemination, distribution or copying of this email/fax is prohibited. If you have received this email/fax in error, please notify the sender by replying to this message and deleting this email or destroying this facsimile immediately.
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> ht://Dig Developer mailing list:
> htd...@li...
> List information (subscribe/unsubscribe, etc.)
> https://lists.sourceforge.net/lists/listinfo/htdig-dev
>

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

[htdig-dev] Release plans?

[Lachlan A. <lh...@us...>]

Greetings all,

Neal recently suggested releasing a new interrim release in September. =20
Since yet another deadline has passed, could I ask that those who=20
hold the "keys" to  www.htdig.org  set some guidelines for when we=20
can release the next beta?  It was suggested that we, the developers,=20
do that.  The problem is that we can't enforce the timelines...

I vote that, once the Windows  db.words.db  bug is ironed out, we=20
release 3.2.0b5 / 3.2.0rc1.  If anyone has a reason *not* to, could=20
they please mail it to the group?

Thanks in advance to all involved,
Lachlan

(Do I sound like a broken record? :)

--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

Re: [htdig-dev] Release plans?

[Jesse op d. B. <ht...@op...>]

Hi all,

if the mainstream of servers and/or operating systems
work correct with the latest snapshot and there are
only obscure bugs, then my personal view is that
we should release a new beta, maybe a RC.
There are still people that download, install and
use 3.2b3..........

----- Original Message ----- 
From: "Lachlan Andrew" <lh...@us...>
To: <htd...@li...>
Sent: Sunday, October 05, 2003 1:04 PM
Subject: [htdig-dev] Release plans?


> Greetings all,
> 
> Neal recently suggested releasing a new interrim release in September.  
> Since yet another deadline has passed, could I ask that those who 
> hold the "keys" to  www.htdig.org  set some guidelines for when we 
> can release the next beta?  It was suggested that we, the developers, 
> do that.  The problem is that we can't enforce the timelines...
> 
> I vote that, once the Windows  db.words.db  bug is ironed out, we 
> release 3.2.0b5 / 3.2.0rc1.  If anyone has a reason *not* to, could 
> they please mail it to the group?
> 
> Thanks in advance to all involved,
> Lachlan
> 
> (Do I sound like a broken record? :)

--Jesse

Re: [htdig-dev] Release plans?

[Gabriele B. <bar...@in...>]

>I vote that, once the Windows  db.words.db  bug is ironed out, we
>release 3.2.0b5 / 3.2.0rc1.  If anyone has a reason *not* to, could
>they please mail it to the group?

I vote +1.

I guess the actual snapshot is the best 3.2 version so far. So ... better a 
3.2.0b5 than a buggy 3.2.0b3 still around ...

Ciao
-Gabriele
--
Gabriele Bartolini: Web Programmer, ht://Dig & IWA/HWG Member, ht://Check 
maintainer
Current Location: Melbourne, Victoria, Australia
bar...@in... | http://www.prato.linux.it/~gbartolini | ICQ#129221447
 > "Leave every hope, ye who enter!", Dante Alighieri, Divine Comedy, The 
Inferno

Re: [htdig-dev] Release plans?

[Gilles D. <gr...@sc...>]

According to Gabriele Bartolini:
> >I vote that, once the Windows  db.words.db  bug is ironed out, we
> >release 3.2.0b5 / 3.2.0rc1.  If anyone has a reason *not* to, could
> >they please mail it to the group?
> 
> I vote +1.
> 
> I guess the actual snapshot is the best 3.2 version so far. So ... better a 
> 3.2.0b5 than a buggy 3.2.0b3 still around ...

I'd agree!  Lachlan & Neal, you'd probably know better than anyone whether
the code is ready for a beta release, as you've done most of the work on it
in the past several months.  If you say it's ready, then I say go for it.

+1

As for who holds the keys, it's really up to the developers as a group to
decide on these matters.  Beyond that, the only holding of the keys there's
been is that Geoff has personally put the finishing touches on each release
since 3.1.0b1, so he knows best the ins and outs of how it's done.  If any
of you want to take that over, just say the word, and I'm sure Geoff would
share with you his procedures, scripts, checklists, etc.  (It would be good
to have all that on the developer web site for posterity in any case, don't
you think?)

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] Release plans?

[Joe R. J. <jj...@cl...>]

On Mon, 6 Oct 2003, Gilles Detillieux wrote:

> Date: Mon, 6 Oct 2003 15:29:20 -0500 (CDT)
> From: Gilles Detillieux <gr...@sc...>
> To: Gabriele Bartolini <bar...@in...>
> Cc: lh...@us..., htd...@li...
> Subject: Re: [htdig-dev] Release plans?
> 
> According to Gabriele Bartolini:
> > >I vote that, once the Windows  db.words.db  bug is ironed out, we
> > >release 3.2.0b5 / 3.2.0rc1.  If anyone has a reason *not* to, could
> > >they please mail it to the group?
> > 
> > I vote +1.
> > 
> > I guess the actual snapshot is the best 3.2 version so far. So ... better a 
> > 3.2.0b5 than a buggy 3.2.0b3 still around ...
> 
> I'd agree!  Lachlan & Neal, you'd probably know better than anyone whether
> the code is ready for a beta release, as you've done most of the work on it
> in the past several months.  If you say it's ready, then I say go for it.
> 
> +1

+1

Regards,

Joe
-- 
     _/   _/_/_/       _/              ____________    __o
     _/   _/   _/      _/         ______________     _-\<,_
 _/  _/   _/_/_/   _/  _/                     ......(_)/ (_)
  _/_/ oe _/   _/.  _/_/ ah        jj...@cl...

Re: [htdig-dev] Release plans?

[Neal R. <ne...@ri...>]

On Sun, 5 Oct 2003, Lachlan Andrew wrote:

> Greetings all,
>
> Neal recently suggested releasing a new interrim release in September.
> Since yet another deadline has passed, could I ask that those who
> hold the "keys" to  www.htdig.org  set some guidelines for when we
> can release the next beta?  It was suggested that we, the developers,
> do that.  The problem is that we can't enforce the timelines...

  Mutiny?  Ok, Pirate Lachlan.  I think the issue is that we don't have a
plan to get it to the next step...

> I vote that, once the Windows  db.words.db  bug is ironed out, we
> release 3.2.0b5 / 3.2.0rc1.  If anyone has a reason *not* to, could
> they please mail it to the group?

  I second this idea and suggest that on next Sunday Oct 12 we
institute a 'code-freeze' and spend the remainder of Oct testing the code
and call it 3.2RC1.  3.2RC1 goes out Sun Nov 2.

Proposed plan:

  Testing would consist of us figuring out some way to divide up
test points.  The basic idea is to go through the process of changing the
configuration file in reasonable ways and indexing/searching to
formally  exercise   those verbs.  Ideally we'd like to test each of the
approx 200 verbs.  A similar method for each executables command-line
switches in needed.

  We could probably vote on proposed changes to default config values.

  Of course we need to test index some websites, but doing that on a large
scale is difficult... I propose we set some reasonable goal here and
handle indexing/parsing bugs as we see them after release.

  The goals for 3.2RC2 are to fix any bugs that come up.  The only
allowable commits in this time-frame are those that fix bugs or cleanup
memory leaks.  When its ready we institute a 'code-freeze' and formally
test the parts affected by bug fixes.. At the end of this we can vote
for calling it 3.2 final.

  The next step would be devoting ourselves to quickly handling bugs that
come in from users from some specified period and release '3.2.1'.  I put
this in so we don't leave all this work up to Gilles, our devoted
maintainer of 3.1.X.

  Formal testing is yucky and delays us stamping it 3.2-final, but I'd
rather spend a month doing this than see silly errors found by a
score of endusers.

  The next step would comming up with a test plan, basically a list of
test points so we can divvy up the work.  This needs to be complete by
next weekend.

Thanks.

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

[htdig-dev] Re: Release plans?

[Lachlan A. <lh...@us...>]

On Mon, 6 Oct 2003 05:08, Neal Richter wrote:

>   Mutiny?  Ok, Pirate Lachlan.

*grin*  I don't mean to step on the toes of those who have worked very=20
hard for many years.  Apologies to them if I was curt.

> I think the issue is that we don't
> have a plan to get it to the next step...

Good point.

> I suggest that on next Sunday Oct 12 we
> institute a 'code-freeze' and spend the remainder of Oct testing
> the code and call it 3.2RC1.  3.2RC1 goes out Sun Nov 2.

Good idea.  Do you think the  db.words.db  bug will be sorted out by=20
then?  Does anyone have access to Cygwin?

> Proposed plan:
>
>   Testing would consist of us figuring out some way to divide up
> test points.  The basic idea is to go through the process of
> changing the configuration file in reasonable ways and
> indexing/searching to formally  exercise   those verbs.  Ideally
> we'd like to test each of the approx 200 verbs.  A similar method
> for each executables command-line switches in needed.

Thanks for taking the lead on this, Neal.  The testing sounds like a=20
big job!

I'll start by tentatively taking attributes starting with A-D, and=20
then post a list of those which I *don't* think I'll be able to test.

Would it be worth writing extra scripts in .../test/ so that we can=20
re-test easily after bug fixes and before future releases?  Or would=20
that be more effort than it is worth?

>   The next step would be devoting ourselves to quickly handling
> bugs that come in from users from some specified period and release
> '3.2.1'.  I put this in so we don't leave all this work up to
> Gilles, our devoted maintainer of 3.1.X.

Hear, hear!!

>   Formal testing is yucky and delays us stamping it 3.2-final, but
> I'd rather spend a month doing this than see silly errors found by
> a score of endusers.

Absolutely.

>   The next step would comming up with a test plan, basically a list
> of test points so we can divvy up the work.  This needs to be
> complete by next weekend.

Aye, Aye, Cap'n!

Lachlan

--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

[htdig-dev] Re: Re: Release plans?

[Lachlan A. <lh...@us...>]

On Tue, 7 Oct 2003 21:55, Lachlan Andrew wrote:

> I'll start by tentatively taking attributes starting with A-D, and
> then post a list of those which I *don't* think I'll be able to
> test.


Scratch that...

Attached is a partial grouping of attributes which may be able to be=20
tested together.  Feel free to re-group them.

--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

Re: [htdig-dev] Re: Re: Release plans?

[Neal R. <ne...@ri...>]

Thanks!

This is a lot of work.

Everyone:  Please let me know what kind of time you'd be willing to put in
to get this stuff tested??!!

When we do the code-freeze on Monday, other than fixing the
Jim/Gilles/robot.txt issue we should be testing and fixing till we
covered the test cases.

Let's get this done so we can talk about 4.0 and restructuring.

Neal.

On Wed, 8 Oct 2003, Lachlan Andrew wrote:

> On Tue, 7 Oct 2003 21:55, Lachlan Andrew wrote:
>
> > I'll start by tentatively taking attributes starting with A-D, and
> > then post a list of those which I *don't* think I'll be able to
> > test.
>
>
> Scratch that...
>
> Attached is a partial grouping of attributes which may be able to be
> tested together.  Feel free to re-group them.
>
> --
> lh...@us...
> ht://Dig developer DownUnder  (http://www.htdig.org)

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] Re: Re: Release plans?

[Lachlan A. <lh...@us...>]

Greetings all,

I have a question about the interpretation of  allow_numbers.
If  allow_numbers  is false, should digits be considered separators? =20
Looking at the code, it seems someone wanted to say that  "3G", "Y2K"=20
and "X11" would be words, even if  allow_numbers  is false, because=20
they contain at least one letter:

  int alpha =3D 0;
  for(const unsigned char *p =3D
        (const unsigned char*)(const char*)(char *)word; *p; p++) {
    if(IsStrictChar(*p) || (allow_numbers && IsDigit(*p))) {
      alpha =3D 1;
    } else if(IsControl(*p)) {
      return status | WORD_NORMALIZE_CONTROL;
    }
  }

  //
  // Reject if contains no alpha characters
  //
  if(!alpha) return status | WORD_NORMALIZE_NOALPHA;



Current behaviour is to *ignore*  allow_numbers  and to default to=20
treating digits as letters [since WORD_TYPE_DIGIT is included in =20
IsChar()  and  IsStrictChar()].

I propose the following behaviour:

1. If  allow_numbers  is true then digits are treated the same as=20
extra_word_characters.
2. If  allow_numbers  is false, then digits are treated as ("invalid")=20
punctuation.
3. The default be changed to  allow_numbers=3Dtrue  (which is=20
compatibile with the current buggy default behaviour).

Any objections?

Lachlan

On Sat, 11 Oct 2003 05:56, Neal Richter wrote:

> Everyone:  Please let me know what kind of time you'd be willing to
> put in to get this stuff tested??!!

--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

Re: [htdig-dev] Re: Re: Release plans?

[Neal R. <ne...@ri...>]

Sounds good to me.

Please make the fix but hold off committing it... see
previous message, I need to do some busy work on the sourceforge site
first.... and we need to wait to hear from anyone with objections ;-)

Thanks.

On Mon, 13 Oct 2003, Lachlan Andrew wrote:

> Greetings all,
>
> I have a question about the interpretation of  allow_numbers.
> If  allow_numbers  is false, should digits be considered separators?
> Looking at the code, it seems someone wanted to say that  "3G", "Y2K"
> and "X11" would be words, even if  allow_numbers  is false, because
> they contain at least one letter:
>
>   int alpha = 0;
>   for(const unsigned char *p =
>         (const unsigned char*)(const char*)(char *)word; *p; p++) {
>     if(IsStrictChar(*p) || (allow_numbers && IsDigit(*p))) {
>       alpha = 1;
>     } else if(IsControl(*p)) {
>       return status | WORD_NORMALIZE_CONTROL;
>     }
>   }
>
>   //
>   // Reject if contains no alpha characters
>   //
>   if(!alpha) return status | WORD_NORMALIZE_NOALPHA;
>
>
>
> Current behaviour is to *ignore*  allow_numbers  and to default to
> treating digits as letters [since WORD_TYPE_DIGIT is included in
> IsChar()  and  IsStrictChar()].
>
> I propose the following behaviour:
>
> 1. If  allow_numbers  is true then digits are treated the same as
> extra_word_characters.
> 2. If  allow_numbers  is false, then digits are treated as ("invalid")
> punctuation.
> 3. The default be changed to  allow_numbers=true  (which is
> compatibile with the current buggy default behaviour).
>
> Any objections?
>
> Lachlan
>
> On Sat, 11 Oct 2003 05:56, Neal Richter wrote:
>
> > Everyone:  Please let me know what kind of time you'd be willing to
> > put in to get this stuff tested??!!
>
> --
> lh...@us...
> ht://Dig developer DownUnder  (http://www.htdig.org)
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: SF.net Giveback Program.
> SourceForge.net hosts over 70,000 Open Source Projects.
> See the people who have HELPED US provide better services:
> Click here: http://sourceforge.net/supporters.php
> _______________________________________________
> ht://Dig Developer mailing list:
> htd...@li...
> List information (subscribe/unsubscribe, etc.)
> https://lists.sourceforge.net/lists/listinfo/htdig-dev
>

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

[htdig-dev] allow_numbers

[Lachlan A. <lh...@us...>]

Greetings all,

On reflection, I think the behaviour that seems to have been intended=20
is better.  I've filed a bug report (with patch) to implement:

1. If allow_numbers is false, words must contain at least one=20
non-digit (2001 not a word, X11 is).
2. If allow_numbers is true, digits are equivalent to letters.

Comments/testing welcome.

Cheers,
Lachlan

On Tue, 14 Oct 2003 07:59, Neal Richter wrote:
> Sounds good to me.
>
> On Mon, 13 Oct 2003, Lachlan Andrew wrote:
> > 1. If  allow_numbers  is true then digits are treated the same as
> > extra_word_characters.
> > 2. If  allow_numbers  is false, then digits are treated as
> > ("invalid") punctuation.
> > 3. The default be changed to  allow_numbers=3Dtrue  (which is
> > compatibile with the current buggy default behaviour).

--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

Re: [htdig-dev] allow_numbers

[Ted Stresen-R. <ted...@ma...>]

I'm happy to do testing on Mac OS X. I have a Linux box at my disposal 
but it is a production machine so it is hard to test on. When 
requesting help testing, please provide a test case (the steps one must 
take to complete the test) and the intended behavior (so the testers 
know what to look for and what shouldn't be appearing).

I try to follow all the threads on this list but lately I seem to be 
missing every other message! Bear with me...

Thanks,

Ted Stresen-Reuter

On Monday, October 13, 2003, at 05:37  PM, Lachlan Andrew wrote:

> Greetings all,
>
> On reflection, I think the behaviour that seems to have been intended
> is better.  I've filed a bug report (with patch) to implement:
>
> 1. If allow_numbers is false, words must contain at least one
> non-digit (2001 not a word, X11 is).
> 2. If allow_numbers is true, digits are equivalent to letters.
>
> Comments/testing welcome.
>
> Cheers,
> Lachlan
>
> On Tue, 14 Oct 2003 07:59, Neal Richter wrote:
>> Sounds good to me.
>>
>> On Mon, 13 Oct 2003, Lachlan Andrew wrote:
>>> 1. If  allow_numbers  is true then digits are treated the same as
>>> extra_word_characters.
>>> 2. If  allow_numbers  is false, then digits are treated as
>>> ("invalid") punctuation.
>>> 3. The default be changed to  allow_numbers=true  (which is
>>> compatibile with the current buggy default behaviour).
>
> -- 
> lh...@us...
> ht://Dig developer DownUnder  (http://www.htdig.org)
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: SF.net Giveback Program.
> SourceForge.net hosts over 70,000 Open Source Projects.
> See the people who have HELPED US provide better services:
> Click here: http://sourceforge.net/supporters.php
> _______________________________________________
> ht://Dig Developer mailing list:
> htd...@li...
> List information (subscribe/unsubscribe, etc.)
> https://lists.sourceforge.net/lists/listinfo/htdig-dev
>

[htdig-dev] Re: testing

[Lachlan A. <lh...@us...>]

Thanks for the offer of testing, Ted.

Regarding test cases, I think that the main part of testing at the=20
moment is actually generating the test cases.  Essentially, we have=20
to use each of the features of ht://Dig and make sure that it works=20
as documented.  Neal has suggested testing each of the configuration=20
attributes and command line arguments.  If we're keen, we should also=20
test each template variable.

=46rom the attached list of attributes, select a group of attributes. =20
Write a config file which sets each of them to some value.  One by=20
one, change the attribute in a way which should produce an observable=20
change, and make sure you observe that change.  For example, if you=20
were testing the "meta" group, you would check that, with=20
create_url_list=3Dtrue, it correctly creates a list of URLs retrieved,=20
and that with create_url_list=3Dfalse, it doesn't create such a list.

This testing may be very simplistic, but it does reveal bugs.

Thanks again,
Lachlan

On Tue, 14 Oct 2003 23:23, Ted Stresen-Reuter wrote:
> I'm happy to do testing on Mac OS X.
> When requesting help testing, please provide a test case (the steps
> one must take to complete the test) and the intended behavior (so
> the testers know what to look for and what shouldn't be appearing).

--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

Re: [htdig-dev] allow_numbers

[Gilles D. <gr...@sc...>]

According to Lachlan Andrew:
> On reflection, I think the behaviour that seems to have been intended 
> is better.  I've filed a bug report (with patch) to implement:
> 
> 1. If allow_numbers is false, words must contain at least one 
> non-digit (2001 not a word, X11 is).
> 2. If allow_numbers is true, digits are equivalent to letters.
> 
> Comments/testing welcome.

I would agree that this is the desirable behaviour, and it is what 3.1.x
implements.  Somewhere in the creation of the WordType class in 3.2, a few
errors were made in porting over the logic of the WordList class in 3.1.
The logical error was in assuming that IsStrictChar() returned false
for digits, when it in fact returns true.  Without actually testing your
patch beyond a visual "walk-through", the new logic appears to be correct.
That the WordType class read allow_numbers as Value rather than Boolean
was just bizarre, but I guess an understandable oversight.

I got thouroughly confused in reading your patch, though, because it is
reversed, with the new code appearing in the first file and the old code
in the second, rather than the other way around.  Taking that into account,
though, the patch seems right to me.  I think it should be committed ASAP.

Speaking of committing to CVS, Neal, what's up with your request to hold off?

According to Neal Richter:
>> Please make the fix but hold off committing it... see
>> previous message, I need to do some busy work on the sourceforge site
>> first.... and we need to wait to hear from anyone with objections ;-)

What was the busy work other than adding bug tracker categories, which
you mentioned in your earlier message?  Are you done and is it OK to
commit now?  I noticed some people have been doing so.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] allow_numbers

[Neal R. <ne...@ri...>]

On Tue, 14 Oct 2003, Gilles Detillieux wrote:

> Speaking of committing to CVS, Neal, what's up with your request to hold off?

[Snip]

> What was the busy work other than adding bug tracker categories, which
> you mentioned in your earlier message?  Are you done and is it OK to
> commit now?  I noticed some people have been doing so.

  Nothing really.. the idea was at this point we should have
a 'Include_in_3.2' bug associated with each 'commit' until we release.
This is purely for the purpose of 'tracking' the bugs & fixes.

  I was wanting to get the 'Include_in_3.2' group added first... and clean
up the bug list.  That's done.

  No offense intended there... I was just trying to get some measure of
organization to the 'Feature-Freeze' state we are in now.

  ie.. during 'feature-freeze' there should be a bug created for each
issue, and any commits should list the bug number in the commit message.

  This isn't an attempt to exert control.. just to help organize the
process.

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] allow_numbers

[Gilles D. <gr...@sc...>]

According to Neal Richter:
> On Tue, 14 Oct 2003, Gilles Detillieux wrote:
> > Speaking of committing to CVS, Neal, what's up with your request to hold off?
...
> > What was the busy work other than adding bug tracker categories, which
> > you mentioned in your earlier message?  Are you done and is it OK to
> > commit now?  I noticed some people have been doing so.
> 
>   Nothing really.. the idea was at this point we should have
> a 'Include_in_3.2' bug associated with each 'commit' until we release.
> This is purely for the purpose of 'tracking' the bugs & fixes.
> 
>   I was wanting to get the 'Include_in_3.2' group added first... and clean
> up the bug list.  That's done.
> 
>   No offense intended there... I was just trying to get some measure of
> organization to the 'Feature-Freeze' state we are in now.
> 
>   ie.. during 'feature-freeze' there should be a bug created for each
> issue, and any commits should list the bug number in the commit message.
> 
>   This isn't an attempt to exert control.. just to help organize the
> process.

No offense taken at all.  I was just a bit puzzled as to the reason
and the status of your request to hold off.  This helps clarify things
a lot.  I'm all for organizing the process better, but I think what you
just said needed to be stated explicitly so we all understand what the
process ought to be.  Thanks.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)

Re: [htdig-dev] allow_numbers

[Lachlan A. <lh...@us...>]

On Wed, 15 Oct 2003 04:01, Gilles Detillieux wrote:

> I got thouroughly confused in reading your patch, though, because
> it is reversed, with the new code appearing in the first file and
> the old code in the second, rather than the other way around.

Oops... :)
=20
> Taking that into account, though, the patch seems right to me.  I
> think it should be committed ASAP.

Done.  Could you (or someone) please confirm it and close the bug=20
report?

Thanks,
Lachlan

--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

[htdig-dev] Re: Cygwin words.db Compression

[Lachlan A. <lh...@us...>]

Phew...  then it might not be my fault :)  Sorry, I have no more=20
ideas, so I'll leave you in Neal's capable hands...

Cheers,
Lachlan

On Sat, 4 Oct 2003 01:36, Steve Eidemiller wrote:
> it didn't appear to work

--=20
lh...@us...
ht://Dig developer DownUnder  (http://www.htdig.org)

Re: [htdig-dev] Cygwin words.db Compression

[<st...@ei...>]

Thanx for the suggestion. Here's what I get:

================
cl -nologo -W3 -DZLIB_DLL -MD -I../include -DDBUG_OFF -D_WIN32 -DWIN32 -
D__WIN32
__ -IL:/win32/include/zlib -DDEFAULT_CONFIG_FILE=\"c:\htdig\demo.conf\" -
DCOMMON
_DIR=\"c:\htdig\demo.db\templates\" -DBIN_DIR=\"c:\htdig\demo.db\bin\" -
DCONFIG_
DIR=\"c:\htdig\" -DIMAGE_URL_PREFIX=\"/rnt/rnm/img\" -
DDATABASE_DIR=\"c:\htdig\d
emo.db\" -Fowin32/dirent_local.obj -c dirent_local.c
dirent_local.c
dirent_local.c(26) : fatal error C1083: Cannot open include file: 'iostream.h':
No such file or directory
make[1]: *** [win32/dirent_local.obj] Error 2
make[1]: Leaving directory `/home/htdig320b4/htdig-3.2.0b4-20031005/db'
make: *** [db.build] Error 2
================

I'm using the 2003 .NET compiler (Vc7/cl.exe), and it doesn't have iostream.h 
or ostream.h headers (only the C++ versions iostream and ostream). Commenting 
out the '#include <iostream.h>' line in dirent_local.c (probably a bad idea) 
appears to let compilation continue through the BDB code without errors. The 
next error is as follows:

================
cl -DHAVE_CONFIG_H -I../db -I. -I../htword -I../htcommon -nologo -W3 -DZLIB_DLL
-MD -I../include -DDBUG_OFF -D_WIN32 -DWIN32 -D__WIN32__ -
IL:/win32/include/zlib
 -DDEFAULT_CONFIG_FILE=\"c:\htdig\demo.conf\" -
DCOMMON_DIR=\"c:\htdig\demo.db\te
mplates\" -DBIN_DIR=\"c:\htdig\demo.db\bin\" -DCONFIG_DIR=\"c:\htdig\" -
DIMAGE_U
RL_PREFIX=\"/rnt/rnm/img\" -DDATABASE_DIR=\"c:\htdig\demo.db\" -GX -
Fowin32/Conf
iguration.obj -c /Tp Configuration.cc
Configuration.cc
c:\cygwin\home\htdig320b4\htdig-3.2.0b4-20031005-B\htlib\htString.h(28) : fatal
error C1083: Cannot open include file: 'iostream.h': No such file or directory
make[1]: *** [win32/Configuration.obj] Error 2
make[1]: Leaving directory `/home/htdig320b4/htdig-3.2.0b4-20031005-B/htlib'
make: *** [htlib.build] Error 2
================

I attempted to make use of several suggested fixes for htString.h regarding 
ostream and the std namespace from the mailing lists, but none of them seemed 
to work. The issue seems to be that HAVE_STD and HAVE_NAMESPACES checks at the 
top of the file aren't working correctly. I was sure to copy over 
your .h.win32 headers as specified.

With a little tweaking around, it looks like the compile will continue *IF* I 
can get the std namespace working for the appropriate references, and *IF* I 
can get the files to reference iostream and ostream instead of the .h 
versions. Does this sound like the right path? Lots of other files have this 
same conditional block. Any ideas?

Cheers!!

>
> Please try this in a cygwin shell:
> 
> cp ./db/db.h.win32 ./db/db.h
> cp ./db/db_config.h.win32 ./db/db_config.h
> cp ./include/htconfig.h.win32 ./include/htconfig.h
> 
> make -f Makefile.win32
> 
> That should fire off a build using Microsoft's compilers...
> 
> Thanks!  Neal.
> 
> Neal Richter
> Knowledgebase Developer
> RightNow Technologies, Inc.
> Customer Service for Every Web Site
> Office: 406-522-1485
> 

Re: [htdig-dev] Cygwin words.db Compression

[Neal R. <ne...@ri...>]

Keep pounding.  I didn't use .NET compilers to make these files so I'm not
suprised you see these errors.

Before you do lots of changes to the code... isn't there a cl switch to
accept the older C++ no-namespace idioms?

On Fri, 10 Oct 2003 st...@ei... wrote:

> Thanx for the suggestion. Here's what I get:
>
> ================
> cl -nologo -W3 -DZLIB_DLL -MD -I../include -DDBUG_OFF -D_WIN32 -DWIN32 -
> D__WIN32
> __ -IL:/win32/include/zlib -DDEFAULT_CONFIG_FILE=\"c:\htdig\demo.conf\" -
> DCOMMON
> _DIR=\"c:\htdig\demo.db\templates\" -DBIN_DIR=\"c:\htdig\demo.db\bin\" -
> DCONFIG_
> DIR=\"c:\htdig\" -DIMAGE_URL_PREFIX=\"/rnt/rnm/img\" -
> DDATABASE_DIR=\"c:\htdig\d
> emo.db\" -Fowin32/dirent_local.obj -c dirent_local.c
> dirent_local.c
> dirent_local.c(26) : fatal error C1083: Cannot open include file: 'iostream.h':
> No such file or directory
> make[1]: *** [win32/dirent_local.obj] Error 2
> make[1]: Leaving directory `/home/htdig320b4/htdig-3.2.0b4-20031005/db'
> make: *** [db.build] Error 2
> ================
>
> I'm using the 2003 .NET compiler (Vc7/cl.exe), and it doesn't have iostream.h
> or ostream.h headers (only the C++ versions iostream and ostream). Commenting
> out the '#include <iostream.h>' line in dirent_local.c (probably a bad idea)
> appears to let compilation continue through the BDB code without errors. The
> next error is as follows:
>
> ================
> cl -DHAVE_CONFIG_H -I../db -I. -I../htword -I../htcommon -nologo -W3 -DZLIB_DLL
> -MD -I../include -DDBUG_OFF -D_WIN32 -DWIN32 -D__WIN32__ -
> IL:/win32/include/zlib
>  -DDEFAULT_CONFIG_FILE=\"c:\htdig\demo.conf\" -
> DCOMMON_DIR=\"c:\htdig\demo.db\te
> mplates\" -DBIN_DIR=\"c:\htdig\demo.db\bin\" -DCONFIG_DIR=\"c:\htdig\" -
> DIMAGE_U
> RL_PREFIX=\"/rnt/rnm/img\" -DDATABASE_DIR=\"c:\htdig\demo.db\" -GX -
> Fowin32/Conf
> iguration.obj -c /Tp Configuration.cc
> Configuration.cc
> c:\cygwin\home\htdig320b4\htdig-3.2.0b4-20031005-B\htlib\htString.h(28) : fatal
> error C1083: Cannot open include file: 'iostream.h': No such file or directory
> make[1]: *** [win32/Configuration.obj] Error 2
> make[1]: Leaving directory `/home/htdig320b4/htdig-3.2.0b4-20031005-B/htlib'
> make: *** [htlib.build] Error 2
> ================
>
> I attempted to make use of several suggested fixes for htString.h regarding
> ostream and the std namespace from the mailing lists, but none of them seemed
> to work. The issue seems to be that HAVE_STD and HAVE_NAMESPACES checks at the
> top of the file aren't working correctly. I was sure to copy over
> your .h.win32 headers as specified.
>
> With a little tweaking around, it looks like the compile will continue *IF* I
> can get the std namespace working for the appropriate references, and *IF* I
> can get the files to reference iostream and ostream instead of the .h
> versions. Does this sound like the right path? Lots of other files have this
> same conditional block. Any ideas?
>
> Cheers!!
>
> >
> > Please try this in a cygwin shell:
> >
> > cp ./db/db.h.win32 ./db/db.h
> > cp ./db/db_config.h.win32 ./db/db_config.h
> > cp ./include/htconfig.h.win32 ./include/htconfig.h
> >
> > make -f Makefile.win32
> >
> > That should fire off a build using Microsoft's compilers...
> >
> > Thanks!  Neal.
> >
> > Neal Richter
> > Knowledgebase Developer
> > RightNow Technologies, Inc.
> > Customer Service for Every Web Site
> > Office: 406-522-1485
> >
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: SF.net Giveback Program.
> SourceForge.net hosts over 70,000 Open Source Projects.
> See the people who have HELPED US provide better services:
> Click here: http://sourceforge.net/supporters.php
> _______________________________________________
> ht://Dig Developer mailing list:
> htd...@li...
> List information (subscribe/unsubscribe, etc.)
> https://lists.sourceforge.net/lists/listinfo/htdig-dev
>

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485

Re: [htdig-dev] Re: Cygwin words.db Compression

[Neal R. <ne...@ri...>]

On Fri, 3 Oct 2003, Steve Eidemiller wrote:

> Hi Lachlan,
>
> Thanks for the suggestion :-)  I tried it in all the environments I listed earlier, but it didn't appear to work using the default settings for the compression flags. Here's what htdb_dump reports for all attempts:
>
> ================
> C:\htdig\bin>htdb_dump -Wz -p c:/htdig/var/htdig/db.words.db
> htdb_dump: open: c:/htdig/var/htdig/db.words.db: No such file or directory
>
> C:\htdig\bin>htdb_dump -W -p c:/htdig/var/htdig/db.words.db
> htdb_dump: c:/htdig/var/htdig/db.words.db: file size not a multiple of the pages
> ize

  This is strange given the above error.  IF this error is accurate it's a
harbinger of bad values in the DB.  I have fixed this and thought I
checked it in!  Basically I tracked the state of the file pointer and at
somepoint the system twesks it to 'text' mode, and this hoses the DB.

I'll check CVS to see if I got that fix in.  If the fix is in CVS, then
its a new bug!



>
> On Fri, 3 Oct 2003 02:51, Steve Eidemiller wrote:
> > I'm compiling htdig-3.2.0b4-20090928 under Cygwin 1.5.5 using gcc
> > 3.3.1, on both Windows XP Pro SP1 and Windows 2000 Server SP4.
> > db.words.db is always a zero length file
>
> --
> lh...@us...
> ht://Dig developer DownUnder  (http://www.htdig.org)
> __________________________________
>
> Confidentiality Statement:
> This email/fax, including attachments, may include confidential and/or proprietary information and may be used only by the person or entity to which it is addressed. If the reader of this email/fax is not the intended recipient or his or her agent, the reader is hereby notified that any dissemination, distribution or copying of this email/fax is prohibited. If you have received this email/fax in error, please notify the sender by replying to this message and deleting this email or destroying this facsimile immediately.
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> ht://Dig Developer mailing list:
> htd...@li...
> List information (subscribe/unsubscribe, etc.)
> https://lists.sourceforge.net/lists/listinfo/htdig-dev
>

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485