htdig-dev — Developer Discussion for the ht://Dig project

[htdig-dev] configure changes in 3.1.x

[Gabriele B. <g.b...@co...>]

Ciao Gilles,

    as you probably noticed, I changed the CONFIG.in and the configure.in 
files for the with-'____' series of parameters regarding the some directory 
and file customisation.

    I hope you don't mind if I did it without 'warning' you. I have to get 
back on track!!! :-)

Ciao
-Gabriele

--
Gabriele Bartolini - Computer Programmer
U.O. Rete Civica - Comune di Prato - Prato - Italia - Europa
g.b...@co... | http://www.po-net.prato.it/

  The nice thing about Windows is - It does not just crash,
  it displays a dialog box and lets you press 'OK' first.

Re: [htdig-dev] htdig 3.2.0b4 and HP-UX 11.00

[<htd...@li...>]

On Thu, Dec 20, 2001 at 01:53:23AM -0500, Geoff Hutchison wrote:
> On Thu, 20 Dec 2001 htd...@li... wrote:
> 
> >         l = sizeof(sa_client);
> >         if ((getpeername(sock_in, &sa_client, &l)) < 0) {
> 
> That's fine, but what's the declaration for l?

        NET_SIZE_T l;

NET_SIZE_T is effectively size_t for all SVR4 systems. It's hard-coded
for other platforms in src/include/ap_config.h.

-- 
albert chin (ch...@th...)

Re: [htdig-dev] htdig 3.2.0b4 and HP-UX 11.00

[Geoff H. <ghu...@ws...>]

On Thu, 20 Dec 2001 htd...@li... wrote:

>         l = sizeof(sa_client);
>         if ((getpeername(sock_in, &sa_client, &l)) < 0) {

That's fine, but what's the declaration for l?

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

Re: [htdig-dev] htdig 3.2.0b4 and HP-UX 11.00

[<htd...@li...>]

On Thu, Dec 20, 2001 at 01:37:22AM -0500, Geoff Hutchison wrote:
> On Wed, 19 Dec 2001, Geoff Hutchison wrote:
> 
> > I'll add a test for "Psocklen_t" to the list to loop through, joining
> > 'size_t,' 'int,' 'unsigned int,' 'long unsigned int,' and 'socklen_t.' I'm
> 
> Bah. I discovered that there are actually a few Solaris 8 machines that I
> can use. So I can confirm that I can code a nicely-working configure test.
> 
> The catch is to get working code. 
> 
> Basically I'm stuck because it seems like Sun is doing something
> nonsensical. The third (namelen) parameter is supposed to be initialized
> to the size. But you can't assign to a void! So I haven't the faintest
> what Sun wants you to do. The man page for this claims that it's always a
> 'socklen_t*' but that's not what's actually in the header file.
> 
> At the moment, I don't have a good answer. If someone can find me examples
> of calls to getpeername() that work on Solaris 8 in other code (Apache?),
> please send them to the list and I'll get things working.

BTW, HP-UX 11.00 has the same problem (when using the C++ compiler).

From apache-1.3.22/src/main/http_main.c
    int sock_in;

    ...

    struct sockaddr sa_server, sa_client;

    ...

    sock_in = fileno(stdin);
    sock_out = fileno(stdout);

    ...

        l = sizeof(sa_client);
        if ((getpeername(sock_in, &sa_client, &l)) < 0) {
/* get peername will fail if the input isn't a socket */
            perror("getpeername");
            memset(&sa_client, '\0', sizeof(sa_client));
        }

-- 
albert chin (ch...@th...)

Re: [htdig-dev] htdig 3.2.0b4 and HP-UX 11.00

[Geoff H. <ghu...@ws...>]

On Wed, 19 Dec 2001, Geoff Hutchison wrote:

> I'll add a test for "Psocklen_t" to the list to loop through, joining
> 'size_t,' 'int,' 'unsigned int,' 'long unsigned int,' and 'socklen_t.' I'm

Bah. I discovered that there are actually a few Solaris 8 machines that I
can use. So I can confirm that I can code a nicely-working configure test.

The catch is to get working code. 

Basically I'm stuck because it seems like Sun is doing something
nonsensical. The third (namelen) parameter is supposed to be initialized
to the size. But you can't assign to a void! So I haven't the faintest
what Sun wants you to do. The man page for this claims that it's always a
'socklen_t*' but that's not what's actually in the header file.

At the moment, I don't have a good answer. If someone can find me examples
of calls to getpeername() that work on Solaris 8 in other code (Apache?),
please send them to the list and I'll get things working.

Otherwise, this will have to wait until I return from break to look for
such examples.

-Geoff

[htdig-dev] Re: mktime fix?

[Geoff H. <ghu...@ws...>]

Hi Gilles,

I seem to remember that you were planning on putting in your mktime() fix
for the 3.1.6 release--this was the message you sent on Nov. 23. I don't
see it in the current CVS version, but perhaps I somehow missed a
revision?

I'd like to clean up any remaining issues over the next few weeks so we
can release 3.1.6 ASAP. (I have the time, so I'm taking care of the
regex/rx issue, the release notes, the maindocs merges, the getpeername
issue, and code into htmerge to catch empty databases to throw up more
useful help text.)

-Geoff

[htdig-dev] Re: [htdig] compilation of htdig 3.1.6 on Mac OS X client v. 10.1.1

[Geoff H. <ghu...@ws...>]

On Tue, 18 Dec 2001, Gilles Detillieux wrote:

> >  #include <ostream.h>
> 
> OK, so should we just make this same change in the source distribution,
> or is there a possibility that this will cause problems on other systems?
> Do we need configure tests for these header files?  I don't think 3.2
> does any, but then 3.2 doesn't have a phenomenal record for portability.

I don't think ensuring full portability in some areas has been at the top
of my list for the 3.2 development. That may seem short-sighted, but I'd
rather see certain gaping holes fixed (e.g. query parser, mifluz) before
going full-bore at the need for additional configure tests.

That's beside the point.

Unfortunately, the whole *stream.h header issue seems to be a real
quagmire on the systems I've seen. IIRC, ostream.h doesn't exist on some
platforms, iostream.h doesn't exist on others.... So clearly a simple
check for the appropriate header file is needed. I'll add that to the
3.1.6 configure script in a moment.

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

Re: [htdig-dev] htdig 3.2.0b4 and HP-UX 11.00

[Geoff H. <ghu...@ws...>]

On Wed, 19 Dec 2001 htd...@li... wrote:

> The prototype for getpeername() on Solaris 8/SPARC is:
>    typedef void *Psocklen_t; 
>    int getpeername (int, struct sockaddr *, Psocklen_t);

Thanks for reminding me of this--the configure script loops through
several common types for the latter two parameters and I suppose it was
inevitable that someone would decide to make this call even more
complicated.

I'll add a test for "Psocklen_t" to the list to loop through, joining
'size_t,' 'int,' 'unsigned int,' 'long unsigned int,' and 'socklen_t.' I'm
assuming that Solaris 8 has that 'typedef' statement in either
<sys/types.h> or <sys/socket.h> or will another include need to be added
to the test program?

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

[htdig-dev] htdig 3.2.0b4 and HP-UX 11.00

[<htd...@li...>]

The prototype for getpeername() on Solaris 8/SPARC is:
   typedef void *Psocklen_t; 
   int getpeername (int, struct sockaddr *, Psocklen_t);

The logic in configure.in to detect this is not going to work because
the test program would be:
  #include <sys/types.h>
  #include <sy/socket.h>

  extern "C" int getpeername (int, struct sockaddr *, void *);
  struct sockaddr s;
  void l;

  ...

Clearly the 'void l' is not going to work. What is the solution?

-- 
albert chin (ch...@th...)

Re: [htdig-dev] Scandinavian characters do not work in latest htdig.

[Gilles D. <gr...@sc...>]

According to dA K6:
> I recently took the latest snapshot (1216) of htdig and compiled it.
> It runs ok, but...

Which version?  There should be snapshots for 3.1.6 and 3.2.0b4.  If you'=
ve
set things up correctly (see below) and are still having problems, it wou=
ld
be useful to know which version you're running, not just the snapshot dat=
e.

> ... when doing a search it seems that the engine does not recognize
> Scandinavian characters like =E5=C5 =E4=C4 =F6=D6. I tried to search wi=
th ä
> values instead of the usual scandichar, but that didn't work at all.
> I even tried url encoded scandicars.

You don't mention whether you've set your locale.  htdig won't recognise
accented characters unless you do.  See http://www.htdig.org/FAQ.html#q5.=
8
Also, you must ensure that locale support is installed and working on you=
r
system, and pick a locale name that's defined on your system.

--=20
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/~g=
rdetil
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930

[htdig-dev] Scandinavian characters do not work in latest htdig.

[dA K. <da...@ho...>]

Hi.

I recently took the latest snapshot (1216) of htdig and compiled it.
It runs ok, but...

... when doing a search it seems that the engine does not recognize
Scandinavian characters like åÅ äÄ öÖ. I tried to search with ä
values instead of the usual scandichar, but that didn't work at all.
I even tried url encoded scandicars.

Is there any scandichar support available in htdig? If not, when will
it be available? If there will not be any support for scandichars,
I will be forced to use mnogo again :(

So please htdig developers, please do include scandichar support soon,
'cause htdig is so much easier to use and configure than mnogo :)

I'd really apreciate any kind of feedback regarding this issue.


Thanx,
/dA_K6

Re: [htdig-dev] Problems with whatsnew.pl

[Gilles D. <gr...@sc...>]

According to Jim Cole:
> Hi - I think this was just joke gone wrong ;) There was a typo in
> the original message that mentioned the year 2999, where I
> presume 1999 was intended. That is the reason for the reference
> to "a thousand years from now". Least that is my guess.

Yes, last week my sausage fingers typed:
> whatsnew.pl hasn't even kept up with early developments in
> the 3.1.x series in 2999, ...

That's what Alex was jokingly poking at, I'm sure.
Of course I meant `888.  ;-)

I guess the slow progress on ht://Dig development this past year or so
is a bit of a touchy subject, though.  We'll get there eventually.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/~grdetil
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930

Re: [htdig-dev] Cookies in 3.2beta

[Gabriele B. <an...@us...>]

>Program received signal SIGSEGV, Segmentation fault
>0x40023822 in HtHTTP::SetRequestCommand (this=0x80e8e08, cmd=@0xbfffe96c)
>at HtHTTP.cc:61
>616           _cookie_jar->SetHTTPRequest_CookiesString(_url, cmd
>
>It seems _cookie_jar is never properly defined (it's only set to 0 at the
>top). What is the state of cookie support? I assume this crash is a known
>problem? Is this easy to fix? If so, I might try...

Fixed. Thanx!!! :-)

-Gabriele
--
Gabriele Bartolini - Web Programmer
Current Location: Prato, Tuscany, Italy
an...@us... | http://www.prato.linux.it/~gbartolini | ICQ#129221447
 > find bin/laden -name osama -exec rm {} \;

Re: [htdig-dev] Concerning FTP access / mirroring

[J. op d. B. <MSQ...@st...>]

On Sun, 16 Dec 2001, Geoff Hutchison wrote:

> Of course if someone knows how you could do some sort of 
> timeout/alarm in the update script, that would solve the problem. I 
> could even stick the mirror scripts as examples on the website.


Something like:

<command> &
sleep <seconds>
if <command> still on ps list, then kill it.


Found this on a list:

----------------------------
  command &
  sleep 20
  if [ $(ps -p $! | wc -l ) -eq 2 ]; then echo "ERROR"; else echo "OK"; fi
---------------------------




--jesse
--------------------------------------------------------------------
J. op den Brouw                           Johanna Westerdijkplein 75
Haagse Hogeschool                                  2521 EN  DEN HAAG
Faculty of Engeneering                                   Netherlands
Electrical Engeneering                                +31 70 4458936
-------------------- J.E...@st... --------------------

Linux - because reboots are for hardware changes

Rif: Re: [htdig-dev] Problems with whatsnew.pl

[Gabriele B. <an...@ti...>]

> Hi - I think this was just joke gone wrong ;) There was a typo in
> the original message that mentioned the year 2999, where I
> presume 1999 was intended. That is the reason for the reference
> to "a thousand years from now". Least that is my guess.

I hope so. :)

-Gabriele

Re: [htdig-dev] Problems with whatsnew.pl

[Jim C. <gre...@yg...>]

Hi - I think this was just joke gone wrong ;) There was a typo in
the original message that mentioned the year 2999, where I
presume 1999 was intended. That is the reason for the reference
to "a thousand years from now". Least that is my guess.

Jim

Geoff Hutchison's bits of Sun, 16 Dec 2001 translated to:

>At 11:57 AM +1100 12/17/01, Alexander Cohen wrote:
>>What? They are still working on 3.1.x almost a thousand years from now?
>
>OK, let's get some things straight. The point of a production branch
>is to fix bugs as they come up, so that people have a solid, stable,

Re: [htdig-dev] Problems with whatsnew.pl

[Gabriele B. <an...@us...>]

Dear Geoff,

    I agree with *everything* you said, every *single* word. You have all 
my *moral* support and I hope from now the real one again.

    Thanks for the great work you and Gilles are doing in your spare time.

Ciao
-Gabriele

At 21.52 16/12/2001 -0600, Geoff Hutchison wrote:
>At 11:57 AM +1100 12/17/01, Alexander Cohen wrote:
>>What? They are still working on 3.1.x almost a thousand years from now?
>
>OK, let's get some things straight. The point of a production branch is to 
>fix bugs as they come up, so that people have a solid, stable, secure 
>release. People still use the 2.0.x and 2.2.x series of the Linux kernel, 
>even though there's a "stable" 2.4.x series.
>
>I suspect even when 3.2.0 or 3.2.1 or 3.2.2 versions of ht://Dig are 
>released, there will be people using 3.1.6 or whatever the current 3.1.x 
>release is.
>
>>Talk about a slow development cycle, will the universal increase of
>>entropy then be an important factor in the development cycle of the 3.2
>
>One of the side points of free software is that you have the source. If 
>you think the development is moving slowly or not going in a direction 
>you'd like, you should contribute or find people to contribute in the 
>direction you want. It's all there in the copyright. That's your freedom.
>
>A popular saying in the US is that "with freedom comes responsibility."
>
>At the moment, we have two people actively contributing and both of us are 
>quite busy. Gilles and I also handle quite a bit of support on the mailing 
>list and this obviously cuts from our time to code.
>
>So I'll leave it to you. If you think the development cycle of 3.2 is 
>moving too slowly, contribute yourself in any of a number of ways, find 
>other contributors in your stead, or don't complain.
>
>Cheers,
>-Geoff
>
>_______________________________________________
>htdig-dev mailing list
>htd...@li...
>https://lists.sourceforge.net/lists/listinfo/htdig-dev

--
Gabriele Bartolini - Web Programmer
Current Location: Prato, Tuscany, Italy
an...@us... | http://www.prato.linux.it/~gbartolini | ICQ#129221447
 > find bin/laden -name osama -exec rm {} \;

Re: [htdig-dev] Problems with whatsnew.pl

[Gabriele B. <an...@us...>]

G'day Mate.

>What? They are still working on 3.1.x almost a thousand years from now?

This is all done for what we usually call *stability*.

>Talk about a slow development cycle, will the universal increase of
>entropy then be an important factor in the development cycle of the 3.2
>series?

OK. You write from Victoria, Australia, don't you? Probably Melbourne, ok? 
I have been there for 5 months, you could have come up before and maybe you 
could have started working on the project with me. As long as I could not 
find any Linux machine that could give me free access to the Internet! I 
only had my computer which crashed. Because with people that write code for 
free, that's what happens. Nobody pays for your work or your hardware. It 
is just *yours*.

I think that what you said could be *real* but it is really unfair: I feel 
sorry because for a long time I could not help Geoff and Gilles and the 
Group providing my code. I feel bad when I see other people like them 
stealing time from their sparetime in order to work on ht://Dig. But one 
thing can be said, and I am proud of that because it makes us feel like 
*brotherhood*: I love computers and they are my passion. And I am sure that 
can be said for Geoff and Gilles as well, and many other people in all over 
the world that believe in opensource!

So please don't say that, in that way! Nobody's *slave* of somebody and we 
all do this for free. Even now, when I spending money for the Internet 
connection at home at 5 AM (still jetleg) in order to reply to you.

That's all. Ciao and say 'hi' to my friends in Melbourne!

-Gabriele
--
Gabriele Bartolini - Web Programmer
Current Location: Prato, Tuscany, Italy
an...@us... | http://www.prato.linux.it/~gbartolini | ICQ#129221447
 > find bin/laden -name osama -exec rm {} \;

Re: [htdig-dev] Back again - A few changes

[Geoff H. <ghu...@ws...>]

At 8:35 PM +0100 12/16/01, Gabriele Bartolini wrote:
>    I also tried to clean a bit the constructors (paying attention to 
>the initialization list).
>
>    As far as the Connection class is concerned, I decided to remove 
>one constructor, which -I think- can give us only maintanance 
>troubles: I am talking about the default one, which now is:

Yes, these changes look fine to me, though we should obviously test 
thoroughly. :-)

>    I hope everything is fine and I have not made any mistake. So 
>guys, pllease give me the *welcome back*, I need encouragements! :-)

Wecome back! I hope you enjoyed Australia. We can certainly use some help!

Ciao,
-Geoff

Re: [htdig-dev] Problems with whatsnew.pl

[Geoff H. <ghu...@ws...>]

At 11:57 AM +1100 12/17/01, Alexander Cohen wrote:
>What? They are still working on 3.1.x almost a thousand years from now?

OK, let's get some things straight. The point of a production branch 
is to fix bugs as they come up, so that people have a solid, stable, 
secure release. People still use the 2.0.x and 2.2.x series of the 
Linux kernel, even though there's a "stable" 2.4.x series.

I suspect even when 3.2.0 or 3.2.1 or 3.2.2 versions of ht://Dig are 
released, there will be people using 3.1.6 or whatever the current 
3.1.x release is.

>Talk about a slow development cycle, will the universal increase of
>entropy then be an important factor in the development cycle of the 3.2

One of the side points of free software is that you have the source. 
If you think the development is moving slowly or not going in a 
direction you'd like, you should contribute or find people to 
contribute in the direction you want. It's all there in the 
copyright. That's your freedom.

A popular saying in the US is that "with freedom comes responsibility."

At the moment, we have two people actively contributing and both of 
us are quite busy. Gilles and I also handle quite a bit of support on 
the mailing list and this obviously cuts from our time to code.

So I'll leave it to you. If you think the development cycle of 3.2 is 
moving too slowly, contribute yourself in any of a number of ways, 
find other contributors in your stead, or don't complain.

Cheers,
-Geoff

Re: [htdig-dev] Concerning FTP access / mirroring

[Geoff H. <ghu...@ws...>]

At 1:25 PM +0100 12/16/01, J. op den Brouw wrote:
>  > Given the quirks with CVS on SF, I'd be a bit leary about using it, even
>>  for a trial period, because it's sort of an all or nothing proposition.
>>  Mind you, the maindocs updates have been going pretty well now, haven't
>>  they?
>
>Ahhh, mirorring maindocs can go haywire if 'files' are added to the
>repository.
>Hmm, if that's so, let's not do it.

I'm just worried about complaints I've gotten from the SF sysadmins. 
The scripts I use to update the website itself seem to hang 
frequently--the CVS request never quits. This is obviously something 
I'd like to work out before we put new things in the repository!

Of course if someone knows how you could do some sort of 
timeout/alarm in the update script, that would solve the problem. I 
could even stick the mirror scripts as examples on the website.

-Geoff

Re: [htdig-dev] Problems with whatsnew.pl

[Alexander C. <A....@la...>]

What? They are still working on 3.1.x almost a thousand years from now?
Talk about a slow development cycle, will the universal increase of
entropy then be an important factor in the development cycle of the 3.2
series?

		--- Alex

> The modified date for whatsnew.pl should be an immediate clue.  The
> 3.2 version of htdig didn't start to be developed until the last couple
> years, and whatsnew.pl hasn't even kept up with early developments in
> the 3.1.x series in 2999, so it's pretty hopeless to try to get it to
> work with 3.2.  There is a different whatsnew script on ftp.ccsf.org
> that can handle 3.1.x databases, but it's a bit tricky to get it to
> work because it requires special Perl libraries.  There's nothing
> equivalent for 3.2.

-- 
-----------------------------------
Alexander Cohen
Web Developer
La Trobe University - ITS
A....@la...
M: 0419-595-817
W: (03) 9479-3444
-----------------------------------

[htdig-dev] Back again - A few changes

[Gabriele B. <an...@us...>]

Ciao guys,

    After ages, I can swear I was missing you! :-)

    Anyway, this week-end I could finally work again on both ht://Check and 
ht://Dig (a bit). I was concerned about looking at the changes done on the 
htnet library on ht://Dig and port them on ht://Check. I found a few 
changes were done so I looked at them and compared with ht://Check code, in 
order to make it the same.

    I came up with a few ideas regarding especially the management of this 
*holy* connection dynamic object. After this class on C++ I attended in 
Melbourne, I feel much stronger on the subject and also on OO. Anyway, I 
noticed that the connection object is not always needed (because for 
instance not all of the Transport derived object use it - i.e.HtFile) and 
the pointer that manages it is hold by the Transport class. Also now we 
have 2 different types of connections, the 'basic' and the 'SSL' one.

    I reasonably think that, in order to avoid double deletes, we *must* 
give responsability of this to one particular class: the one I find the 
most adapt, is the Transport one. Let me clearify that this can also *not 
happen* (HtFile again).

    As far as the Transport class is concerned, I decided to change the 
default constructor:

Transport::Transport()  becomes now Transport::Transport(Connection* 
connection = 0).

    This *trick* I think is very useful later on. Let me explain. HTTP 
needs a connection, right? so I forced the constructor to be:
HtHTTP::HtHTTP(Connection&) and the HtHTTP::HtHTTP() to be private and 
empty, which avoids any possible default creation by a compiler. As long as 
I wrote the destructor to be virtual and pure, HtHTTP is now an abstract 
class (which is what we want). The derived classes are HtHTTPBasic 
and  HtHTTPSecure, which *must* create a Connection object (SSLConnection 
in the latter one, but it is derived from Connection - so no problem).

    The default constructor of HtHTTPBasic is now very short, because it 
has to call the HtHTTP constructor by passing a new allocated Connection 
object:
HtHTTPBasic::HtHTTPBasic() : HtHTTP(*(new Connection))  [...]
  and the Secure version is very similar.

    I also tried to clean a bit the constructors (paying attention to the 
initialization list).

    As far as the Connection class is concerned, I decided to remove one 
constructor, which -I think- can give us only maintanance troubles: I am 
talking about the default one, which now is:

Connection::Connection(int socket = -1)

    I hope everything is fine and I have not made any mistake. So guys, 
pllease give me the *welcome back*, I need encouragements! :-)

Ciao ciao
-Gabriele
--
Gabriele Bartolini - Web Programmer
Current Location: Prato, Tuscany, Italy
an...@us... | http://www.prato.linux.it/~gbartolini | ICQ#129221447
 > find bin/laden -name osama -exec rm {} \;

Re: [htdig-dev] Concerning FTP access / mirroring

[J. op d. B. <ms...@st...>]

Gilles Detillieux wrote:

> The drawback to using CVS is that everyone has to use it.  You can't
> really have an alternate method of updating the files, otherwise the
> repository doesn't get updated.

Everyone already uses it. It's the best way to create a mirror for
the maindocs. Mostly, if not all, mirrors are ran on 24/7 machines
with a big internet pipe, and are managed on a routinely basis by
profesionals, and they know CVS for sure and are using it.
I don't think that there are 'personal' mirrors. 

Better still, is to use one program only for updating mirrors. That's
why it would be nice if the patch site ran CVS (but it only has
advantage
if the 'files' can be mirrored with CVS).

> Another option open to htdig developers is ssh access and the scp command.
> Personally I like being able to use scp to copy in any contributed works.
> I find this more convenient than the old method of using CVS.  Is there
> some way of setting up automatic updating/mirroring using ssh access?

SSH/SCP can be used to copy, but I don't know if it can be used to
update. I've used rsync over SSH for updating directories, and that
works
very well, so you'll need a third program to generate a list of files to
be tranferred for updating the mirror.
 
> Given the quirks with CVS on SF, I'd be a bit leary about using it, even
> for a trial period, because it's sort of an all or nothing proposition.
> Mind you, the maindocs updates have been going pretty well now, haven't
> they?

Ahhh, mirorring maindocs can go haywire if 'files' are added to the
repository.
Hmm, if that's so, let's not do it.

--Jesse op den Brouw

[htdig-dev] Current Status as of snapshot 3.2.0b4-121601

[Geoff H. <ghu...@us...>]

STATUS of ht://Dig branch 3-2-x

RELEASES:
   3.2.0b4: In progress
   3.2.0b3: Released:  22 Feb 2001.
   3.2.0b2: Released:  11 Apr 2000.
   3.2.0b1: Released:   4 Feb 2000.

SHOWSTOPPERS:

KNOWN BUGS:
* Odd behavior with $(MODIFIED) and scores not working with
   wordlist_compress set but work fine without wordlist_compress.
   (the date is definitely stored correctly, even with compression on
    so this must be some sort of weird htsearch bug)
* Not all htsearch input parameters are handled properly: PR#648. Use a
   consistant mapping of input -> config -> template for all inputs where
   it makes sense to do so (everything but "config" and "words"?).
* If exact isn't specified in the search_algorithms, $(WORDS) is not set 
   correctly: PR#650. (The documentation for 3.2.0b1 is updated, but can
   we fix this?)
* META descriptions are somehow added to the database as FLAG_TITLE,
   not FLAG_DESCRIPTION. (PR#859)

PENDING PATCHES (available but need work):
* Additional support for Win32.
* Memory improvements to htmerge. (Backed out b/c htword API changed.)
* MySQL patches to 3.1.x to be forward-ported and cleaned up.
  (Should really only attempt to use SQL for doc_db and related, not word_db)

NEEDED FEATURES:
* Field-restricted searching.
* Return all URLs.
* Handle noindex_start & noindex_end as string lists.
* Handle local_urls through file:// handler, for mime.types support.
* Handle directory redirects in RetrieveLocal.
* Merge with mifluz

TESTING:
* httools programs: 
  (htload a test file, check a few characteristics, htdump and compare)
* Turn on URL parser test as part of test suite.
* htsearch phrase support tests
* Tests for new config file parser
* Duplicate document detection while indexing
* Major revisions to ExternalParser.cc, including fork/exec instead of popen,
  argument handling for parser/converter, allowing binary output from an
  external converter.
* ExternalTransport needs testing of changes similar to ExternalParser.

DOCUMENTATION:
* List of supported platforms/compilers is ancient.
* Add thorough documentation on htsearch restrict/exclude behavior
   (including '|' and regex).
* Document all of htsearch's mappings of input parameters to config attributes
   to template variables. (Relates to PR#648.) Also make sure these config
   attributes are all documented in defaults.cc, even if they're only set by
   input parameters and never in the config file.
* Split attrs.html into categories for faster loading.
* require.html is not updated to list new features and disk space
   requirements of 3.2.x (e.g. phrase searching, regex matching,
   external parsers and transport methods, database compression.)
* TODO.html has not been updated for current TODO list and completions.

OTHER ISSUES:
* Can htsearch actually search while an index is being created?
   (Does Loic's new database code make this work?)
* The code needs a security audit, esp. htsearch
* URL.cc tries to parse malformed URLs (which causes further problems)
   (It should probably just set everything to empty) This relates to 
   PR#348.