naviserver-devel

[naviserver-devel] ns_conn channel: kills kittens?

[Stephen D. <sd...@gm...>]

Here's some problems with ns_conn channel:

- ssl: if you use it (or any other comm module) you'll be surprised
when 'puts' gets written in clear text.

- If you 'puts' rather than ns_write, you may be surprised when
automatic chunking, encoding, compression doesn't work.

- maxconnections: The server thinks it knows how many clients it's
serving, but by stashing away the socket and returning the conn, it's
under counting.

- Accounting for number-of-bytes-written is weak (as it is with the ns
background writer threads).


We could fix this by creating a custom channel type to wrap the conn
socket rather than use the standard TcpChannel.

Unfortunately, we don't really have the infrastructure to support this:

The current writer thread stuff separates the Sock from the Conn and
writes directly via the NsDriver* calls, so it passes the SSL test,
but doesn't handle the other stuff. For that, you really need a Conn
available.

So, we might have to move to a model where we can separate a Conn from
it's conn-thread.

Secondly, none of our Ns_ConnWrite* calls support an asynchronous mode.


Or..?

Re: [naviserver-devel] ns_conn channel: kills kittens?

[Vlad S. <vl...@cr...>]

i personally never used ns_conn but use writers all the time.


Stephen Deasey wrote:
> Here's some problems with ns_conn channel:
> 
> - ssl: if you use it (or any other comm module) you'll be surprised
> when 'puts' gets written in clear text.
> 
> - If you 'puts' rather than ns_write, you may be surprised when
> automatic chunking, encoding, compression doesn't work.
> 
> - maxconnections: The server thinks it knows how many clients it's
> serving, but by stashing away the socket and returning the conn, it's
> under counting.
> 
> - Accounting for number-of-bytes-written is weak (as it is with the ns
> background writer threads).
> 
> 
> We could fix this by creating a custom channel type to wrap the conn
> socket rather than use the standard TcpChannel.
> 
> Unfortunately, we don't really have the infrastructure to support this:
> 
> The current writer thread stuff separates the Sock from the Conn and
> writes directly via the NsDriver* calls, so it passes the SSL test,
> but doesn't handle the other stuff. For that, you really need a Conn
> available.
> 
> So, we might have to move to a model where we can separate a Conn from
> it's conn-thread.
> 
> Secondly, none of our Ns_ConnWrite* calls support an asynchronous mode.
> 
> 
> Or..?
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

Re: [naviserver-devel] ns_conn channel: kills kittens?

[Gustaf N. <ne...@wu...>]

Stephen Deasey schrieb:
> Here's some problems with ns_conn channel:
>
> - ssl: if you use it (or any other comm module) you'll be surprised
> when 'puts' gets written in clear text.
>
> - If you 'puts' rather than ns_write, you may be surprised when
> automatic chunking, encoding, compression doesn't work.
>
> - maxconnections: The server thinks it knows how many clients it's
> serving, but by stashing away the socket and returning the conn, it's
> under counting.
>
> - Accounting for number-of-bytes-written is weak (as it is with the ns
> background writer threads).
>   
all true, but effects mostly the writer commands (puts/fcopy/etc). Maybe 
one should
add handling of persistent connections to that list. The problem is not
so much that raw puts/fcopy don't handle these cases (how should it),
but that there is currently no nice way to deal with such issues.

> We could fix this by creating a custom channel type to wrap the conn
> socket rather than use the standard TcpChannel.
>
> ...
> Secondly, none of our Ns_ConnWrite* calls support an asynchronous mode.
>   
It should be possible to do async i/o on the raw channel
(maybe rename "ns_conn channel" to "ns_conn rawchannel"), call
ns_write from the readable/writable callbacks and provide
an argument to ns_write for the channel. Based on that it should be
possible to provide a fcopy based on ns_write with little effort.

This does not address the problem of needing the Conn structure. An option
might be to provide essentially a deep copy of conn and pass this in
the custom channel.

We discussed something very similar some time ago in context of the writer
threads, where a more flexible model on passing control between spooling
and worker threads sounds interesting (e.g. checking on file uploads
the quota via database, do spooling, do postprocessing in workter thread).

-gustaf

Re: [naviserver-devel] ns_conn channel: kills kittens?

[Vasiljevic Z. <zv...@ar...>]

On 03.10.2007, at 09:59, Gustaf Neumann wrote:

> It should be possible to do async i/o on the raw channel
> (maybe rename "ns_conn channel" to "ns_conn rawchannel"), call
> ns_write from the readable/writable callbacks and provide
> an argument to ns_write for the channel. Based on that it should be
> possible to provide a fcopy based on ns_write with little effort.

Possible, but for what purpose? Why not just call ns_write
in the first place?

If we really want to expose the complete protocol support
(chunking, encryption, compression, whatever) that server
is providing, to a Tcl channel, then you need the.....

>
> This does not address the problem of needing the Conn structure.

Exactly. You need the Conn structure....


> An option
> might be to provide essentially a deep copy of conn and pass this in
> the custom channel.

which is (may be?) a lot of plumbing...

>
> We discussed something very similar some time ago in context of the  
> writer
> threads, where a more flexible model on passing control between  
> spooling
> and worker threads sounds interesting (e.g. checking on file uploads
> the quota via database, do spooling, do postprocessing in workter  
> thread).

Indeed we did. I was not that much underway in this area of code
so I cannot comment if this is possible or not and if yes what
level of effort is needed to make this happen. To me, the most important
thing is: usage. We cook what we need to eat, at least we are doing that
(we're commercial company living from what we sell). So far, my main
concern was how to get the server doing something *else* beside serving
HTTP reqests (which it does excellent anyway) so I made that
[ns_conn chan] short-circuit in no-time getting much for no-work.
Of course a nice Tcl-channel wrapper over the Conn, utilizing
all that's below is a nice-to-have feature, but at the moment
we (I speak about myself, selfishly) do not need such thing.
If somebody does, I may help a little with the implementation
but I cannot lead it (constant lack of resources!).

Cheers,
Zoran

Re: [naviserver-devel] ns_conn channel: kills kittens?

[Vasiljevic Z. <zv...@ar...>]

On 02.10.2007, at 23:50, Stephen Deasey wrote:

>
> - ssl: if you use it (or any other comm module) you'll be surprised
> when 'puts' gets written in clear text.
>

I won=92t

> - If you 'puts' rather than ns_write, you may be surprised when
> automatic chunking, encoding, compression doesn't work.

I won't (be surprised)

>
> - maxconnections: The server thinks it knows how many clients it's
> serving, but by stashing away the socket and returning the conn, it's
> under counting.

Probably.

>
> - Accounting for number-of-bytes-written is weak (as it is with the ns
> background writer threads).

Do not understand?

>

And now the explanation: the idea of getting the raw connection
socket was to *avoid* all that HTTP stuff that server is imposing
and to use propriatery protocol after the initial HTTP comm
handshake. So, a modified HTTP client would establish the communication
using HTTP headers (and all that jazz) and then pass the client
(and on the server to the custom protocol code) the RAW socket so
both can communicate as they wish, w/o server interfering into the
communication. Clear? I already explained that in numerous emails
since about 2 years ago.
We already utilize that approach in our product where clients use
HTTP protocol to obtain the bi-directional comm channel between the
peers. After that we pump our own data over that pipe. In this =20
particular
example, we are a backup application where clients connect to a public
HTTP server and get a private pipe where the actual data to be backed
up is transferred from client to the server using our own propriatery
wire protocol. So you use all that HTTP infrastructure to establish
the "wire" only. Afterwards we are playing our own music.


>
> We could fix this by creating a custom channel type to wrap the conn
> socket rather than use the standard TcpChannel.
>
> Unfortunately, we don't really have the infrastructure to support =20
> this:
>
> The current writer thread stuff separates the Sock from the Conn and
> writes directly via the NsDriver* calls, so it passes the SSL test,
> but doesn't handle the other stuff. For that, you really need a Conn
> available.
>
> So, we might have to move to a model where we can separate a Conn from
> it's conn-thread.
>
> Secondly, none of our Ns_ConnWrite* calls support an asynchronous =20
> mode.
>

I do not think this is needed. There is already the ns_write that does
all that things. If one really wants to handle that from within the
Tcl channel, then yes, you need a special Tcl channel driver that would
do what ns_write is doing. But then again, you might need all that
async stuff (fconfigure $chan -blocking false) which isn't in place.
I see that very pragmatically: it is not needed by anybody I know so =20
far.
What server already offers is sufficient for most. The [ns_conn chan]
satisfies "cornerstone" requirements that some weirdos (like myself)
may have. If needed, we may plug that out and solve in some other =20
fashion,
I do not care. It seemed just a very easy and convenient place
(the ns_conn command) to add this small change. To develop a full-blown
Tcl channel driver may require too much work. I currently do not see
justification for all that (additional) work. We may be better off
concentrating on other, more important areas, that affect everybody.

Unless you have some time to burn... I have fairly long experience
about writing a Tcl channel driver (hey, this is no rocket science)
so I may be of help if you/somebody wants to fill-in this "hole".

Cheers,
Zoran