naviserver-devel

[naviserver-devel] Socket buffer sizes

[Vasiljevic Z. <zv...@ar...>]

Hi!

I am in the process of evaluating the possibility to
"correct" the current behaviour WRT manipulating
socket buffer sizes...

So far I could understand from reading tons of docs
found all over the internet, the socket buffer sizes are
crucial for optimizing the network peformance related to
fast, high-latency links. We have customers that would
benefit from this so I need to find the way to make this
happen in the Naviserver.

I did find out that different OS'es handle such cases
differently. Most of them require admins to tune either
system or per-app buffer sizes to achieve good BDP
(Bandwidh-Delay Product). Some (newer Linux'es) do this
automatically.

Currently we do already manipulate the socket buffer sizes
in the main driver code (only).
Unfortunately this manipulation happens (at least for the
read-buffer-size) too late! As I understand, the socket
buffers for reading must be setup BEFORE listen() call.
Whereas we manipulate the sizes AFTER the connection is
established.
Furthermore, the Tcl API commands related to sockets
(like ns_socklisten et.al.) also need to be expanded to
allow for explicit buffer size manipulation.

This all would definitely require some internal changes
perhaps even adding new Ns API calls.

I wonder if anybody already thought about this topic and
what was the outcome?

Cheers
Zoran

Re: [naviserver-devel] Socket buffer sizes

[Stephen D. <sd...@gm...>]

On Mon, May 10, 2010 at 10:39 AM, Vasiljevic Zoran <zv...@ar...> wrote:
> Hi!
>
> I am in the process of evaluating the possibility to
> "correct" the current behaviour WRT manipulating
> socket buffer sizes...
>
> So far I could understand from reading tons of docs
> found all over the internet, the socket buffer sizes are
> crucial for optimizing the network peformance related to
> fast, high-latency links. We have customers that would
> benefit from this so I need to find the way to make this
> happen in the Naviserver.


We talked about this in summer 2008, I think. It will be in your mail
or mailing list archives somewhere...


> I did find out that different OS'es handle such cases
> differently. Most of them require admins to tune either
> system or per-app buffer sizes to achieve good BDP
> (Bandwidh-Delay Product). Some (newer Linux'es) do this
> automatically.


Newer is relative. I think by now this means any linux, bsd, solaris
less than a decade old...


> Currently we do already manipulate the socket buffer sizes
> in the main driver code (only).
> Unfortunately this manipulation happens (at least for the
> read-buffer-size) too late! As I understand, the socket
> buffers for reading must be setup BEFORE listen() call.
> Whereas we manipulate the sizes AFTER the connection is
> established.


Which code are you looking at? IIRC, you removed this code years ago:

    http://bitbucket.org/naviserver/naviserver/changeset/a6e8a6348da0

Re: [naviserver-devel] Socket buffer sizes

[Vasiljevic Z. <zv...@ar...>]

On 10.05.2010, at 12:55, Stephen Deasey wrote:
>
> Which code are you looking at? IIRC, you removed this code years ago:
>
>    http://bitbucket.org/naviserver/naviserver/changeset/a6e8a6348da0
>

We do still have people using old'er OS'es and by scrapping this out
they were forced to change buffer sizes on the OS level (system-wide)
which created side-effects we are now fighting with. And it is very
"expensive" to do this on the customer site :-(

Hence I thought to re-vive the socket buffers manipulation and
extend it to all API (including Tcl). But this time I wanted to make
it "correct".

Mostly people using HTTP services only will not be really caring too
much about this. But we use HTTP connections to shuffle TB of data
accross networks and we are very interested in doing all sorts of
optimisations that we can.

What I have in mind is to:

a. make necessary changes to the driver code so that socket options
are set before we actually call listen().

b. add ?-socketbuffersize size? option to all Tcl API commands that
work with sockets.

The a. is really the most work as it will require either chaning
existing (sub-optimal) and/or adding new (more optimal) C-APIs.

The b. is relatively harmless and should be straight-forward once
the a. is done.

The default should be "no changes" as mostly the OS'es would do
the right thing.

BTW... I have really found out that only newer Linuxes do this
automatically (re-size buffers to accomondate to high-latency links).
So far neither Solaris nor Mac OSX nor Windows do this.
I may be wrong but I was not able to find anything related on the  
internet.

Cheers
Zoran

Re: [naviserver-devel] Socket buffer sizes

[Stephen D. <sd...@gm...>]

On Mon, May 10, 2010 at 12:39 PM, Vasiljevic Zoran <zv...@ar...> wrote:
>
> What I have in mind is to:
>
> a. make necessary changes to the driver code so that socket options
> are set before we actually call listen().
>
> The a. is really the most work as it will require either chaning
> existing (sub-optimal) and/or adding new (more optimal) C-APIs.


Can you just take this from the config file? Make the default 0 and if
0, don't config the bufsize.


> b. add ?-socketbuffersize size? option to all Tcl API commands that
> work with sockets.
>
> The b. is relatively harmless and should be straight-forward once
> the a. is done.


Should there be some way to 'ignore' this if the OS can handle it
automatically?  Maybe you just pass [ns_config socket bufsize] to the
option and if it's 0, ignore it...?

Re: [naviserver-devel] Socket buffer sizes

[Vasiljevic Z. <zv...@ar...>]

On 10.05.2010, at 14:38, Stephen Deasey wrote:

>
> Should there be some way to 'ignore' this if the OS can handle it
> automatically?  Maybe you just pass [ns_config socket bufsize] to the
> option and if it's 0, ignore it...?

Sure. This is what I wanted to do. On zero, ignore (== don't set).
As of driver code... the same would apply. Default is zero so no
changes. One can influence this from the outside if needed.

Cheers
Zoran

Re: [naviserver-devel] Socket buffer sizes

[Andrew P. <at...@pi...>]

On Mon, May 10, 2010 at 11:39:26AM +0200, Vasiljevic Zoran wrote:

> So far I could understand from reading tons of docs
> found all over the internet, the socket buffer sizes are
> crucial for optimizing the network peformance related to
> fast, high-latency links.

Sounds like what the hpn-ssh patches do for ssh:

http://www.psc.edu/networking/projects/hpn-ssh/

-- 
Andrew Piskorski <at...@pi...>
http://www.piskorski.com/

Re: [naviserver-devel] Socket buffer sizes

[Stephen D. <sd...@gm...>]

On Mon, May 10, 2010 at 7:22 PM, Andrew Piskorski <at...@pi...> wrote:
> On Mon, May 10, 2010 at 11:39:26AM +0200, Vasiljevic Zoran wrote:
>
>> So far I could understand from reading tons of docs
>> found all over the internet, the socket buffer sizes are
>> crucial for optimizing the network peformance related to
>> fast, high-latency links.
>
> Sounds like what the hpn-ssh patches do for ssh:
>
> http://www.psc.edu/networking/projects/hpn-ssh/


Zoran:

Looks like it would be a mistake to use setsockopt() on Linux >=
2.6.17 (released June 2006) ie. RHEL 5+ (ignoring any patches Redhat
may have backported):

    http://www.psc.edu/networking/projects/tcptune/#detailed
    http://kbase.redhat.com/faq/docs/DOC-3079

Re: [naviserver-devel] Socket buffer sizes

[Vasiljevic Z. <zv...@ar...>]

On 10.05.2010, at 21:13, Stephen Deasey wrote:

>
> Looks like it would be a mistake to use setsockopt() on Linux >=

I know that. Therefore I said that some OS'es do this automatically.
For good or for worse I will (re)add this knobs and leave defaults to
zero (== no changes) in both driver code and Tcl API calls. Who needs
or wants can play with them. Who doesn't will just use what OS is  
giving.
Reason: we have tons of customers running old OS versions and we need
to support them for some years to come.

Cheers
Zoran