Re: [htdig-dev] head_before_get attribute (was: 3.2RC1 Feature Freeze)

[Gabriele B. <bar...@in...>]

At 16.01 21/10/2003 -0500, Gilles Detillieux wrote:
> > 2) the server does not support HEAD (I have seen cases like this 
> unfortunately)
>OK, that sounds pretty important.  I hadn't heard that one before.

I meant that some server administrators may turn off the HEAD method (in 
Apache you can use the Limit directive).

>but don't support the HEAD request.  Wouldn't this be an argument against
>overriding head_before_get during an incremental dig?

I guess it is a matter of choosing the less painful solution. In the normal 
case (p/c on and hbg on) overriding is not done; however, in the 
incremental dig, one more request is made (HEAD) without success and 
hopefully - after that - the document GETs retrieved. There is a bit of 
overhead for sure but the question is: is it better to have a bit of 
overhead in some cases (minority) or to prevent users from getting the 
benefit from using always a workin HEAD call when updating the database?

The other way is to remove the override and leave everything in the hands 
of the user (I would not mind this - of course providing a better 
documentation).

With the changes done yesterday we have moved towards a clearer situation 
anyway, because:
- head before get is now true by default
- head before get has been detached by persistent connections and has 
become independent

> > 3) cases where the persistent communication between htdig and the server
> > does not work at 100%: there can be some problems with persistent
> > connections and HEAD calls (I experience this kind of problems sometimes
> > with ht://Check and some NT servers)
>
>Again, is this going to be a problem if we don't allow turning off
>head_before_get during an update dig?

I guess this could be fixable, because the problem comes up with persistent 
connections - which may be still disabled.

>with these questionably compliant servers, then wouldn't they need a way
>of turning off head_before_get unconditionally, whether it's an update
>dig or an initial one?

Yes, that'd be great.

Again, I guess we have to balance what we can do in order to make things 
easier to the user but, at the same time, leave the users enough freedom in 
order to configure their systems the way they want. Also, with 3.2, the 
server and URL blocks have added more dimensions to the space of 
configurability available to users and ... more "clear" attributes are 
available and more the toy gets perfect.

>This is what I was getting at before about this option never being
>explained adequately.

You're right.

>   On the surface, it seemed to be rather useless,
>but with these new revelations that have come out of your testing, it
>seems there may indeed be a need for turning this off in some cases.
>That's the sort of thing that should be documented so others (developers
>and end-users) know what you'd use this for.

So ... we have 2 possibilities now:

1) leave the code as is
2) remove the overriding of the head before get in the incremental dig

In both cases we need to write down a better documentation for this 
attribute (especially in the option 2 where we should talk about the 
benefits of a HEAD call in the incremental dig).

I must confess. I would prefer option 2, as I think users' must have full 
control of the tool and IMHO by adding a default behaviour of HEAD before 
GET to the system we've done our part.

So tell me what you think, especially you Gilles and Neal that have 
followed this thread. I am more than happy to (in case) rechange the code 
today.

Ciao ciao
-Gabriele
--
Gabriele Bartolini: Web Programmer, ht://Dig & IWA/HWG Member, ht://Check 
maintainer
Current Location: Melbourne, Victoria, Australia
bar...@in... | http://www.prato.linux.it/~gbartolini | ICQ#129221447
 > "Leave every hope, ye who enter!", Dante Alighieri, Divine Comedy, The 
Inferno