naviserver-devel

[naviserver-devel] Simple http proxy handler

[Stephen D. <sd...@gm...>]

tcl/http.tcl:

if {[ns_config -bool ns/server/[ns_info server] enablehttpproxy off]} {
    ns_register_proxy GET  http ns_proxy_handler_http
    ns_register_proxy POST http ns_proxy_handler_http
    nsv_set ns:proxy allow [ns_config ns/server/[ns_info server] allowhttpproxy]
}


This is going to break on the first HEAD request...

This really should be a module of it's own, in the external modules
directory.  So far the functionality is pretty limited and arbitrary,
but obviously really useful.  But there's a lot that needs to be added
to be a fully useful proxy implementation, and probably a lot of
people who would be interested.  This really would benefit from being
an external module, with it's own release cycle etc.

Some thought needs to go into the API for registering proxy handlers.

At the moment the handler is triggered exclusively by the client
sending the full protocol and host  in the request line, i.e.:

    GET http://foo.com/blah.html HTTP/1.0

The problem is that the spec says HTTP 1.1 clients are supposed to
send the full protocol and host on each request.  If we move it full
HTTP 1.1 compliance, this is going tp break.

As it stands, the proxy code is only triggered if the client is
explicitly configured to treat the server as it's proxy, but it's also
useful to have a transparent proxy, e.g. to send some section of the
URL hierarchy to a legacy Apache server :-)

So, removing the requirement that the full protocol and host needed to
trigger the proxy code isn't such a problem, it would enable
transparent proxies.

Registering proxy handlers to a specific method / protocol combination
seems wrong to me.  Even if you added HEAD to the list of methods in
the configuration, what happens when a client send something else?
Shouldn't it be passed through to the back end server, which perhaps
does know how to handle the method?


We needs some ideas for what the API for registering proxy handlers
should look like.  Do we even need one?  Is the existing
ns_register_proc not sufficient?  Perhaps if we added the ability to
register a proc for a 'default' method, *, that would do?

Re: [naviserver-devel] Simple http proxy handler

[Vlad S. <vl...@cr...>]

It started as small proxy to handle special cases when we need access to 
private LAN inside our LAN to provision some servers via WEB. I never 
intended to make it full HTTP proxy but even in current state it does 
work and we access our appliance boxes though this module.

To make it more usefull, it needs to be in separate module, i agree.
Currently it uses internal proxy which is in the core, that's why 
specific register by protocol method, this is how it defined and used in 
the op.c and used in queue.c.

Not sure about transparent proxy on Tcl level, this needs a lot of code 
  it is more like proxy driver as i see it.

Stephen Deasey wrote:
> tcl/http.tcl:
> 
> if {[ns_config -bool ns/server/[ns_info server] enablehttpproxy off]} {
>     ns_register_proxy GET  http ns_proxy_handler_http
>     ns_register_proxy POST http ns_proxy_handler_http
>     nsv_set ns:proxy allow [ns_config ns/server/[ns_info server] allowhttpproxy]
> }
> 
> 
> This is going to break on the first HEAD request...
> 
> This really should be a module of it's own, in the external modules
> directory.  So far the functionality is pretty limited and arbitrary,
> but obviously really useful.  But there's a lot that needs to be added
> to be a fully useful proxy implementation, and probably a lot of
> people who would be interested.  This really would benefit from being
> an external module, with it's own release cycle etc.
> 
> Some thought needs to go into the API for registering proxy handlers.
> 
> At the moment the handler is triggered exclusively by the client
> sending the full protocol and host  in the request line, i.e.:
> 
>     GET http://foo.com/blah.html HTTP/1.0
> 
> The problem is that the spec says HTTP 1.1 clients are supposed to
> send the full protocol and host on each request.  If we move it full
> HTTP 1.1 compliance, this is going tp break.
> 
> As it stands, the proxy code is only triggered if the client is
> explicitly configured to treat the server as it's proxy, but it's also
> useful to have a transparent proxy, e.g. to send some section of the
> URL hierarchy to a legacy Apache server :-)
> 
> So, removing the requirement that the full protocol and host needed to
> trigger the proxy code isn't such a problem, it would enable
> transparent proxies.
> 
> Registering proxy handlers to a specific method / protocol combination
> seems wrong to me.  Even if you added HEAD to the list of methods in
> the configuration, what happens when a client send something else?
> Shouldn't it be passed through to the back end server, which perhaps
> does know how to handle the method?
> 
> 
> We needs some ideas for what the API for registering proxy handlers
> should look like.  Do we even need one?  Is the existing
> ns_register_proc not sufficient?  Perhaps if we added the ability to
> register a proc for a 'default' method, *, that would do?
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Simple http proxy handler

[Stephen D. <sd...@gm...>]

On 9/18/06, Vlad Seryakov <vl...@cr...> wrote:
> It started as small proxy to handle special cases when we need access to
> private LAN inside our LAN to provision some servers via WEB. I never
> intended to make it full HTTP proxy but even in current state it does
> work and we access our appliance boxes though this module.
>
> To make it more usefull, it needs to be in separate module, i agree.
> Currently it uses internal proxy which is in the core, that's why
> specific register by protocol method, this is how it defined and used in
> the op.c and used in queue.c.
>
> Not sure about transparent proxy on Tcl level, this needs a lot of code
>   it is more like proxy driver as i see it.


Well, no.  You could transparent proxy right now by using
ns_register_proc and ns_http to forward the request.  It's *less*
code.

There's two separate issues: where the HTTP proxy code lives; and how
it is activated.  The only think stopping you moving the code to the
nsproxy module -- which is the right thing to do even if you don't
plan to immediately add more to the simple implementation you have now
-- is the naming clash with Zoran's bundled nsproxy module.  The
activation stuff I guess needs some thought.

Exposing this as Tcl commands is definitely a good move. But with the
activation bug and naming confusion it's sort of going in the wrong
direction.  It would be great to set things off on the right foot so
that people could more easily contribute.

Re: [naviserver-devel] Simple http proxy handler

[Vlad S. <vl...@cr...>]

More code i meant to implement the whole full-blown proxy, was not 
clear. As for activation, this is how the core works, everything is a 
callback using url-specific hash entry, so current proxy implementation 
works the same way, just using protocol/method instead of method/url for 
regular HTTP. So i am not sure if we need to re-write this.

Name clash exists, but the module can be named as nshttpproxy

Stephen Deasey wrote:
> 
> Well, no.  You could transparent proxy right now by using
> ns_register_proc and ns_http to forward the request.  It's *less*
> code.
> 
> There's two separate issues: where the HTTP proxy code lives; and how
> it is activated.  The only think stopping you moving the code to the
> nsproxy module -- which is the right thing to do even if you don't
> plan to immediately add more to the simple implementation you have now
> -- is the naming clash with Zoran's bundled nsproxy module.  The
> activation stuff I guess needs some thought.
> 
> Exposing this as Tcl commands is definitely a good move. But with the
> activation bug and naming confusion it's sort of going in the wrong
> direction.  It would be great to set things off on the right foot so
> that people could more easily contribute.
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Simple http proxy handler

[Stephen D. <sd...@gm...>]

On 9/18/06, Vlad Seryakov <vl...@cr...> wrote:
> More code i meant to implement the whole full-blown proxy, was not
> clear. As for activation, this is how the core works, everything is a
> callback using url-specific hash entry

Yeah, and I'm saying it's broken because the HTTP 1.1 spec says that
normal clients, *not* expecting to be talking to a proxy, are supposed
to send the full protocol and host.  In our code, that will activate a
*proxy* callback, but it should just be a a normal HTTP response for
whatever the server has for that URL

At least, that's how I read the RFC.

That's why I say that although adding Tcl access to this stuff is
good, it's going in the wrong direction by exposing a buggy
implementation.

The concept of the current (before you) code is also weak, due to the
need to register a callback for each and every method to be proxied.

Because this has never been exposed to Tcl before, and because seems
to be (to me) a genuine bug which could bight us as we add more HTTP
1.1 support, it seems like now would be the time to take a step back
and look at what we really want in this API.

It's only one call to register, one to unregister, and some Tcl
wrappers.  Do you think it's not worth it?



> so current proxy implementation
> works the same way, just using protocol/method instead of method/url for
> regular HTTP. So i am not sure if we need to re-write this.
>

Re: [naviserver-devel] Simple http proxy handler

[Vlad S. <vl...@cr...>]

> 
> Because this has never been exposed to Tcl before, and because seems
> to be (to me) a genuine bug which could bight us as we add more HTTP
> 1.1 support, it seems like now would be the time to take a step back
> and look at what we really want in this API.
> 
> It's only one call to register, one to unregister, and some Tcl
> wrappers.  Do you think it's not worth it?
> 

Ok, what is the new API and logic you want to add? Or i missed it?

Re: [naviserver-devel] Simple http proxy handler

[Stephen D. <sd...@gm...>]

On 9/18/06, Vlad Seryakov <vl...@cr...> wrote:
> >
> > Because this has never been exposed to Tcl before, and because seems
> > to be (to me) a genuine bug which could bight us as we add more HTTP
> > 1.1 support, it seems like now would be the time to take a step back
> > and look at what we really want in this API.
> >
> > It's only one call to register, one to unregister, and some Tcl
> > wrappers.  Do you think it's not worth it?
> >
>
> Ok, what is the new API and logic you want to add? Or i missed it?
>


I don't know, I'm asking...

I was wondering why registering a proxy handler is different than a
regular registered proc.  Do they have to be different?

The main difference seems to be that you can register a proxy for a
protocol, i.e. http or https, but you don't have that option with
regular registered procs.

Is this an important distinction to make though?  Registered procs can
still check the protocol, manually, if they care.

Re having to register for each method, that also is sometimes a
problem for registered procs. Maybe allowing something like:

  ns_register_proc * /* callback

Where the only supported symbol is '*', meaning 'default, if no more
specific method is registered'.  Perhaps that's too confusing --
people will expect full glob support.  I don't think glob support is
possible, or even desirable here.  Url handlers are supposed to be
deterministic, only one will match some URL (unlike filters).

That's what occurs to me off the top of my head...

Re: [naviserver-devel] Simple http proxy handler

[Vlad S. <vl...@cr...>]

> 
> I don't know, I'm asking...
> 
> I was wondering why registering a proxy handler is different than a
> regular registered proc.  Do they have to be different?
> 
> The main difference seems to be that you can register a proxy for a
> protocol, i.e. http or https, but you don't have that option with
> regular registered procs.

correct

> Is this an important distinction to make though?  Registered procs can
> still check the protocol, manually, if they care.

right

> Re having to register for each method, that also is sometimes a
> problem for registered procs. Maybe allowing something like:
> 
>   ns_register_proc * /* callback
> 
> Where the only supported symbol is '*', meaning 'default, if no more
> specific method is registered'.  Perhaps that's too confusing --
> people will expect full glob support.  I don't think glob support is
> possible, or even desirable here.  Url handlers are supposed to be
> deterministic, only one will match some URL (unlike filters).

Agree, i will take a look. but glob does not work with url-specific 
hashes and we want to keep them, they are very fast


-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Simple http proxy handler

[Stephen D. <sd...@gm...>]

On 9/18/06, Vlad Seryakov <vl...@cr...> wrote:
> >
> > I don't know, I'm asking...
> >
> > I was wondering why registering a proxy handler is different than a
> > regular registered proc.  Do they have to be different?
> >
> > The main difference seems to be that you can register a proxy for a
> > protocol, i.e. http or https, but you don't have that option with
> > regular registered procs.
>
> correct
>
> > Is this an important distinction to make though?  Registered procs can
> > still check the protocol, manually, if they care.
>
> right
>
> > Re having to register for each method, that also is sometimes a
> > problem for registered procs. Maybe allowing something like:
> >
> >   ns_register_proc * /* callback
> >
> > Where the only supported symbol is '*', meaning 'default, if no more
> > specific method is registered'.  Perhaps that's too confusing --
> > people will expect full glob support.  I don't think glob support is
> > possible, or even desirable here.  Url handlers are supposed to be
> > deterministic, only one will match some URL (unlike filters).
>
> Agree, i will take a look. but glob does not work with url-specific
> hashes and we want to keep them, they are very fast
>

Agree. Maybe it was confusing to suggest using '*' syntax to mean
'default' and not 'globbing pattern'.

What do you think to allowing defaults for methods?

i.e. If a GET request comes in, and a GET handler is registered, run
it. This will be just as quick as now.

If a PROPFIND request comes in, there is no PROPFIND handler, but
there is a default handler, run the default handler.

I'm not sure if this *is* feasable.  But sometimes it seems like this
would be useful.

Re: [naviserver-devel] Simple http proxy handler

[Vlad S. <vl...@cr...>]

It will require 2 hash calls but currently it results in error, so 2 
calls will not be that bad if default handler will be able to process 
request.

Stephen Deasey wrote:
> On 9/18/06, Vlad Seryakov <vl...@cr...> wrote:
>>> I don't know, I'm asking...
>>>
>>> I was wondering why registering a proxy handler is different than a
>>> regular registered proc.  Do they have to be different?
>>>
>>> The main difference seems to be that you can register a proxy for a
>>> protocol, i.e. http or https, but you don't have that option with
>>> regular registered procs.
>> correct
>>
>>> Is this an important distinction to make though?  Registered procs can
>>> still check the protocol, manually, if they care.
>> right
>>
>>> Re having to register for each method, that also is sometimes a
>>> problem for registered procs. Maybe allowing something like:
>>>
>>>   ns_register_proc * /* callback
>>>
>>> Where the only supported symbol is '*', meaning 'default, if no more
>>> specific method is registered'.  Perhaps that's too confusing --
>>> people will expect full glob support.  I don't think glob support is
>>> possible, or even desirable here.  Url handlers are supposed to be
>>> deterministic, only one will match some URL (unlike filters).
>> Agree, i will take a look. but glob does not work with url-specific
>> hashes and we want to keep them, they are very fast
>>
> 
> Agree. Maybe it was confusing to suggest using '*' syntax to mean
> 'default' and not 'globbing pattern'.
> 
> What do you think to allowing defaults for methods?
> 
> i.e. If a GET request comes in, and a GET handler is registered, run
> it. This will be just as quick as now.
> 
> If a PROPFIND request comes in, there is no PROPFIND handler, but
> there is a default handler, run the default handler.
> 
> I'm not sure if this *is* feasable.  But sometimes it seems like this
> would be useful.
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Simple http proxy handler

[Michael A. C. <cle...@gm...>]

On 9/18/06, Stephen Deasey <sd...@gm...> wrote:
> I was wondering why registering a proxy handler is different than a
> regular registered proc.  Do they have to be different?

As I've been reading the NaviServer sources to acquaint myself I was
wondering the same thing.

> The main difference seems to be that you can register a proxy for a
> protocol, i.e. http or https, but you don't have that option with
> regular registered procs.
>
> Is this an important distinction to make though?  Registered procs can
> still check the protocol, manually, if they care.

I don't think it is an important decision to make because the
registered proc can already check in the cases it matters.  Back in my
ACS 4.2 days I'd regularly introspect whether the conn was ssl or not
in a registered proc.

> Re having to register for each method, that also is sometimes a
> problem for registered procs. Maybe allowing something like:
>
>   ns_register_proc * /* callback
>
> Where the only supported symbol is '*', meaning 'default, if no more
> specific method is registered'.  Perhaps that's too confusing --
> people will expect full glob support.  I don't think glob support is
> possible, or even desirable here.  Url handlers are supposed to be
> deterministic, only one will match some URL (unlike filters).

I think the "*" would confuse peope (re: globbing).  What about the
empty string? After all, a regular expression of "" will match any
input, so there is a conceptual linkage available.

> That's what occurs to me off the top of my head...

Ditto.

Michael