naviserver-devel — Developer discussion

Re: [naviserver-devel] compilation error

[Kulcsár F. <cru...@ne...>]

Dear Gustaf,

many thanks for your help. The problem is solved. I have used tcl8.6-trunk and it has 
"-fvisibility=hidden" in its configure script. But later I realized that with tcl8.6 naviserver has error messages at startup. With tcl8.5.11 the server is working well.

Regards,

feri

Re: [naviserver-devel] compilation error

[Gustaf N. <ne...@wu...>]

Dear Ferenc,

The problem occurs during linking of nsthreadtest, which is a utility 
for testing and not needed for the operations of naviserver. 
Nevertheless, it should certainly work.... and it does work for me.

when i go to the directory naviserver/nsthread and issue there

    rm *.o
    rm nsthreadtest
    make

i see
    /bin/rm -Rf libnsthread.so
    gcc -shared -O2 -fomit-frame-pointer -Wall -Wno-implicit-int -fPIC  
-pipe -m64  -I../include -I"/usr/local/ns/include"  -DHAVE_CONFIG_H  
-L../nsthread -L../nsd -L../nsdb -o libnsthread.so error.o master.o 
memory.o mutex.o cslock.o rwlock.o reentrant.o sema.o thread.o tls.o 
time.o pthread.o fork.o signal.o winthread.o   -L/usr/local/ns/lib 
-ltcl8.5 -ldl -lgcc_s  -lieee -lm  -Wl,--export-dynamic  
-L/usr/local/ns/lib -Wl,-rpath,:/usr/local/ns/lib
gcc -O2 -fomit-frame-pointer -Wall -Wno-implicit-int -fPIC  -pipe -m64  
-I../include -I"/usr/local/ns/include"  -DHAVE_CONFIG_H   -c -o 
nsthreadtest.o nsthreadtest.c
    /bin/rm -Rf nsthreadtest
    gcc -L../nsthread -L../nsd -L../nsdb -o nsthreadtest nsthreadtest.o 
-lpthread  libnsthread.so -L/usr/local/ns/lib -ltcl8.5 -ldl -lgcc_s  
-lieee -lm  -Wl,--export-dynamic  -L/usr/local/ns/lib 
-Wl,-rpath,:/usr/local/ns/lib

i.e. the program "nsthreadtest" links correctly.

Notice, that the default complile steps on my machine don't include as 
your case the option "-fvisibility=hidden".

My Machine data:
    gcc version 4.6.2 20111027 (Red Hat 4.6.2-1) (GCC)
    Linux alice.wu-wien.ac.at 3.2.2-1.fc16.x86_64 #1 SMP Thu Jan 26 
03:21:58 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
    ./configure --prefix=/usr/local/ns --enable-threads 
--with-tcl=/usr/local/ns/lib --enable-64bit


Notice the following difference:
library libnsthread.so (and all *.o components) built with 
-fvisibility=hidden
   $ nm libnsthread.so | fgrep Ns_Tls
      0000000000002b20 t Ns_TlsAlloc
      0000000000002c00 t Ns_TlsGet
      0000000000002ba0 t Ns_TlsSet

ibrary libnsthread.so (and all *.o components) built without 
-fvisibility=hidden
   $ nm libnsthread.so | fgrep Ns_Tls
      0000000000004260 T Ns_TlsAlloc
      0000000000004360 T Ns_TlsGet
      00000000000042f0 T Ns_TlsSet

You see, the external symbols are only included in the .so-file without 
the hidden visibility.
The question is, where is the "-fvisibility=hidden" coming from. Have 
you added it manually to the Makefile?
If so, why?

-gustaf neumann


Am 14.02.12 08:24, schrieb Kulcsár Ferenc:
> Hello!
>
> When I compile naviserver-4.99.4 the following error occures:
>
> gcc -shared -O2 -fomit-frame-pointer -Wall -Wno-implicit-int -fPIC  -pipe -fvisibility=hidden -m64  -I../include -I"/home/feri/tanul/include"  -DHAVE_CONFIG_H  -L../nsthread -L../nsd -L../nsdb -o libnsthread.so error.o master.o memory.o mutex.o cslock.o rwlock.o reentrant.o sema.o thread.o tls.o time.o pthread.o fork.o signal.o winthread.o   -L/home/feri/tanul/lib -ltcl8.6 -ldl -lgcc_s  -lieee -lm  -Wl,--export-dynamic  -L/home/feri/tanul/lib -Wl,-rpath,:/home/feri/tanul/lib
> gcc -O2 -fomit-frame-pointer -Wall -Wno-implicit-int -fPIC  -pipe -fvisibility=hidden -m64  -I../include -I"/home/feri/tanul/include"  -DHAVE_CONFIG_H   -c -o nsthreadtest.o nsthreadtest.c
> /bin/rm -Rf nsthreadtest
> gcc -L../nsthread -L../nsd -L../nsdb -o nsthreadtest nsthreadtest.o -lpthread  libnsthread.so -L/home/feri/tanul/lib -ltcl8.6 -ldl -lgcc_s  -lieee -lm  -Wl,--export-dynamic  -L/home/feri/tanul/lib -Wl,-rpath,:/home/feri/tanul/lib
> nsthreadtest.o: In function `Pthread':
> nsthreadtest.c:(.text+0x19): undefined reference to `Ns_TlsSet'
> nsthreadtest.c:(.text+0x25): undefined reference to `Ns_MutexLock'
> nsthreadtest.c:(.text+0x47): undefined reference to `Ns_CondWait'
> nsthreadtest.c:(.text+0x5d): undefined reference to `Ns_MutexUnlock'
> nsthreadtest.c:(.text+0x67): undefined reference to `Ns_MasterLock'
> ......
>
> The compilation environment is:
>
> $ uname -a
> Linux archlap.aneder.bel 3.2.5-1-ARCH #1 SMP PREEMPT Tue Feb 7 08:34:36 CET 2012 x86_64 Intel(R) Pentium(R) Dual CPU T2390 @ 1.86GHz GenuineIntel GNU/Linux
> Archlinux
>
> $ ./configure --prefix=/home/feri/tanul --enable-threads --enable-shared --with-tcl=/home/feri/tanul/lib --enable-dependency-tracking
>
> $ make -v
> GNU Make 3.82
> Built for x86_64-unknown-linux-gnu
>
> $ gcc -vhi
> Using built-in specs.
> COLLECT_GCC=gcc
> COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-unknown-linux-gnu/4.6.2/lto-wrapper
> Target: x86_64-unknown-linux-gnu
> Configured with: /build/src/gcc-4.6-20120120/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.archlinux.org/ --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++ --enable-shared --enable-threads=posix --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-clocale=gnu --disable-libstdcxx-pch --enable-libstdcxx-time --enable-gnu-unique-object --enable-linker-build-id --with-ppl --enable-cloog-backend=isl --enable-lto --enable-gold --enable-ld=default --enable-plugin --with-plugin-ld=ld.gold --disable-multilib --disable-libssp --enable-checking=release
> Thread model: posix
> gcc version 4.6.2 20120120 (prerelease) (GCC)
>
> glibc-2.15
>
> Any help is appreciated.
>
> TIA
>
> feri
>
> ------------------------------------------------------------------------------
> Keep Your Developer Skills Current with LearnDevNow!
> The most comprehensive online learning library for Microsoft developers
> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
> Metro Style Apps, more. Free future releases when you subscribe now!
> http://p.sf.net/sfu/learndevnow-d2d
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] compilation error

[Kulcsár F. <cru...@ne...>]

Hello!

When I compile naviserver-4.99.4 the following error occures:

gcc -shared -O2 -fomit-frame-pointer -Wall -Wno-implicit-int -fPIC  -pipe -fvisibility=hidden -m64  -I../include -I"/home/feri/tanul/include"  -DHAVE_CONFIG_H  -L../nsthread -L../nsd -L../nsdb -o libnsthread.so error.o master.o memory.o mutex.o cslock.o rwlock.o reentrant.o sema.o thread.o tls.o time.o pthread.o fork.o signal.o winthread.o   -L/home/feri/tanul/lib -ltcl8.6 -ldl -lgcc_s  -lieee -lm  -Wl,--export-dynamic  -L/home/feri/tanul/lib -Wl,-rpath,:/home/feri/tanul/lib
gcc -O2 -fomit-frame-pointer -Wall -Wno-implicit-int -fPIC  -pipe -fvisibility=hidden -m64  -I../include -I"/home/feri/tanul/include"  -DHAVE_CONFIG_H   -c -o nsthreadtest.o nsthreadtest.c
/bin/rm -Rf nsthreadtest
gcc -L../nsthread -L../nsd -L../nsdb -o nsthreadtest nsthreadtest.o -lpthread  libnsthread.so -L/home/feri/tanul/lib -ltcl8.6 -ldl -lgcc_s  -lieee -lm  -Wl,--export-dynamic  -L/home/feri/tanul/lib -Wl,-rpath,:/home/feri/tanul/lib
nsthreadtest.o: In function `Pthread':
nsthreadtest.c:(.text+0x19): undefined reference to `Ns_TlsSet'
nsthreadtest.c:(.text+0x25): undefined reference to `Ns_MutexLock'
nsthreadtest.c:(.text+0x47): undefined reference to `Ns_CondWait'
nsthreadtest.c:(.text+0x5d): undefined reference to `Ns_MutexUnlock'
nsthreadtest.c:(.text+0x67): undefined reference to `Ns_MasterLock'
......

The compilation environment is:

$ uname -a
Linux archlap.aneder.bel 3.2.5-1-ARCH #1 SMP PREEMPT Tue Feb 7 08:34:36 CET 2012 x86_64 Intel(R) Pentium(R) Dual CPU T2390 @ 1.86GHz GenuineIntel GNU/Linux
Archlinux

$ ./configure --prefix=/home/feri/tanul --enable-threads --enable-shared --with-tcl=/home/feri/tanul/lib --enable-dependency-tracking

$ make -v
GNU Make 3.82
Built for x86_64-unknown-linux-gnu

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-unknown-linux-gnu/4.6.2/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: /build/src/gcc-4.6-20120120/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.archlinux.org/ --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++ --enable-shared --enable-threads=posix --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-clocale=gnu --disable-libstdcxx-pch --enable-libstdcxx-time --enable-gnu-unique-object --enable-linker-build-id --with-ppl --enable-cloog-backend=isl --enable-lto --enable-gold --enable-ld=default --enable-plugin --with-plugin-ld=ld.gold --disable-multilib --disable-libssp --enable-checking=release
Thread model: posix
gcc version 4.6.2 20120120 (prerelease) (GCC) 

glibc-2.15

Any help is appreciated.

TIA

feri

Re: [naviserver-devel] nsproxy module: change comm to sockets instead of pipes?

[Gustaf N. <ne...@wu...>]

Am 28.01.12 22:22, schrieb Ibrahim Tannir:
>
> On 28-Jan-12 22:09, Stephen Deasey wrote:
>> On Sat, Jan 28, 2012 at 8:19 PM, Ibrahim Tannir<it...@ar...>   wrote:
>>> However, the entire asynchronous IO in Windows is really messy
>>> business, since the entire mechanism of processing IO
>>> notifications is tied to a window.
>> I see what y'all mean. That's a lot of non-shared code.
>>
>> I suppose you can always use a unix domain socket on the unixy side
>> while using a socket on the loopback interface for Windows and still
>> use the same poll() code on both.
> Right. That's the general idea.
Sounds very reasonable. i doubt that the performance impact will be 
significant.

-gustaf

Re: [naviserver-devel] nsproxy module: change comm to sockets instead of pipes?

[Ibrahim T. <it...@ar...>]

On 28-Jan-12 22:09, Stephen Deasey wrote:
> On Sat, Jan 28, 2012 at 8:19 PM, Ibrahim Tannir<it...@ar...>  wrote:
>>
>> However, the entire asynchronous IO in Windows is really messy
>> business, since the entire mechanism of processing IO
>> notifications is tied to a window.
>
> I see what y'all mean. That's a lot of non-shared code.
>
> I suppose you can always use a unix domain socket on the unixy side
> while using a socket on the loopback interface for Windows and still
> use the same poll() code on both.

Right. That's the general idea.

Re: [naviserver-devel] nsproxy module: change comm to sockets instead of pipes?

[Stephen D. <sd...@gm...>]

On Sat, Jan 28, 2012 at 8:19 PM, Ibrahim Tannir <it...@ar...> wrote:
>
> However, the entire asynchronous IO in Windows is really messy
> business, since the entire mechanism of processing IO
> notifications is tied to a window.

I see what y'all mean. That's a lot of non-shared code.

I suppose you can always use a unix domain socket on the unixy side
while using a socket on the loopback interface for Windows and still
use the same poll() code on both.

Re: [naviserver-devel] nsproxy module: change comm to sockets instead of pipes?

[Ibrahim T. <it...@ar...>]

Hello Everybody,

According to Microsoft's documentation:

http://msdn.microsoft.com/en-us/library/windows/desktop/aa365141%28v=vs.85%29.aspx

> Asynchronous (overlapped) read and write operations are not supported by anonymous pipes. This means that you cannot use the ReadFileEx and WriteFileEx functions with anonymous pipes. In addition, the lpOverlapped parameter of ReadFile and WriteFile is ignored when these functions are used with anonymous pipes.

That's why Zoran and I ruled out using unnamed pipes. Named pipes 
don't seem a good alternative either for several reasons, the 
least of them being the need to create a name on the fly. 
However, the entire asynchronous IO in Windows is really messy 
business, since the entire mechanism of processing IO 
notifications is tied to a window.

-- Ibrahim

On 28-Jan-12 19:03, Stephen Deasey wrote:
> On Sat, Jan 28, 2012 at 1:26 PM, Zoran Vasiljevic<zv...@ar...>  wrote:
>>
>> On Windows there are some limitations as how you can
>> do non-blocking operations... In particular, there
>> seems to be a problem with non-blocking reads/writes
>> on unnamed pipes... They simply do not work on windows,
>> at least as to our knowledge.
>
> I don't know anything about Windows, but i t looks like PIPE_NOWAIT
> switches to non-blocking mode but is a legacy of ye olde lan manager,
> and instead you should use overlapped IO with the FILE_FLAG_OVERLAPPED
> flag:
>
>    http://msdn.microsoft.com/en-us/library/windows/desktop/aa365788%28v=vs.85%29.aspx
>
> This is all with named pipes. But it says the followinf about anon pipes:
>
>    Anonymous pipes are implemented using a named pipe with
>    a unique name. Therefore, you can often pass a handle to an
>    anonymous pipe to a function that requires a handle to a named pipe.
>
> So maybe you can substitute a named pipe with the appropriate flags
> for the anon pipe?

Re: [naviserver-devel] nsproxy module: change comm to sockets instead of pipes?

[Stephen D. <sd...@gm...>]

On Sat, Jan 28, 2012 at 1:26 PM, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On Windows there are some limitations as how you can
> do non-blocking operations... In particular, there
> seems to be a problem with non-blocking reads/writes
> on unnamed pipes... They simply do not work on windows,
> at least as to our knowledge.

I don't know anything about Windows, but i t looks like PIPE_NOWAIT
switches to non-blocking mode but is a legacy of ye olde lan manager,
and instead you should use overlapped IO with the FILE_FLAG_OVERLAPPED
flag:

  http://msdn.microsoft.com/en-us/library/windows/desktop/aa365788%28v=vs.85%29.aspx

This is all with named pipes. But it says the followinf about anon pipes:

  Anonymous pipes are implemented using a named pipe with
  a unique name. Therefore, you can often pass a handle to an
  anonymous pipe to a function that requires a handle to a named pipe.

So maybe you can substitute a named pipe with the appropriate flags
for the anon pipe?

Re: [naviserver-devel] nsproxy module: change comm to sockets instead of pipes?

[Zoran V. <zv...@ar...>]

Ups...

On 28.01.2012, at 14:26, Zoran Vasiljevic wrote:

> Does anybody have any reasons why we should NOT go after b.?

That is... a. (use sockets for both Unix and Win).

[naviserver-devel] nsproxy module: change comm to sockets instead of pipes?

[Zoran V. <zv...@ar...>]

Hi!

Long time no hear...

We are trying to merge some of our local changes to the
current repository... There are some obstacles though...

We use the Win platform among others and must make sure
the code performs on Win's equally well as on Unices. 
We are still not there but we are approaching...

In general:
Most of the windows changes are now cluttered over the 
code in form of ifdef's which is ugly and unsupportable.
Over the time we will try to avoid this by moving all 
the platform-different stuff out and wrapping it in one 
special directory or file (today's nswin32.c). 
This will be a lenghtly process, as there are LOTS of 
places in the code we need to re-visit, but we should do 
that in order to keep the code maintainable.

In partiucular:
One thing is pretty complicated though.... the nsproxy module. 
It spawns external program (proxy) and maintains the 
communication to that program over standard unnamed
(anonymus) pipes. It also utilizes the non-blocking 
operations to the proxy so we can eventually recover 
from runaway proxies. This runs on Unix very good.

On Windows there are some limitations as how you can
do non-blocking operations... In particular, there 
seems to be a problem with non-blocking reads/writes 
on unnamed pipes... They simply do not work on windows,
at least as to our knowledge.

So there we have a problem... If we want to make nsproxy
work on Unix and windows equally well, we must either
  a. use sockets for both platforms
or
  b. use pipes on Unix (as now) and sockets on windows

The b. is what we've implemented so far. It is conceptually 
working well but it is UGLY code cluttered with define's 
all arround. It is also difficult to expand on and to maintain.

The a. is what I am really after. It *may* add certain 
performance penalty (which at this monent I cannot quantify)
but would lead to cleaner and simpler code.
Apropos performance... I have a feeling that we will not have
much troubles in this area. After all, proxy operations are
per-se slow, so adding some more ticks won't change much IMO.

Does anybody have any reasons why we should NOT go after b.?
Or... is there any other option that is worth considering?

Cheers
Zoran

 

[naviserver-devel] Naviserver/AOLserver on same box

[Ian H. <har...@gm...>]

I installed AOLserver and it works fine.  I tried to install
Naviserver and I have this little problem:

ldd /usr/local/ns/bin/nscp.so
	linux-vdso.so.1 =>  (0x00007fff47541000)
	libnsthread.so => /usr/local/lib/libnsthread.so (0x00007fab03c08000)
	libnsd.so => /usr/local/lib/libnsd.so (0x00007fab03994000)
	libtcl8.4.so.0 => /usr/lib/libtcl8.4.so.0 (0x00007fab036c3000)
	libdl.so.2 => /lib/libdl.so.2 (0x00007fab034bf000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007fab032a9000)
	libm.so.6 => /lib/libm.so.6 (0x00007fab03026000)
	libc.so.6 => /lib/libc.so.6 (0x00007fab02cc5000)
	libpthread.so.0 => /lib/libpthread.so.0 (0x00007fab02aa9000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fab0402b000)

which is a problem because the Naviserver libraries are in /usr/local/ns/lib.

How can I have these two apps play nice?

Ian

[naviserver-devel] cancel option for ns_ssl command (nsssl module)

[Victor G. <vg...@wu...>]

Hello there list,

I was giving a try to the nsssl module and found a
small bug when using the ns_ssl command, the cancel option specifically.
According to the code, the syntax should be 'ns_ssl
cancel id' , but even though I am using the right syntax the call
returns an error complaining about wrong syntax. I just went to the
code and saw that when using the cancel option the code checks or
expects to have 1 argument; when it should expect 2 (cancel option
itself plus the request id).

So here you have the small patch.

diff -r a2d7f6bf1f1f nsssl.c
--- a/nsssl.c   Fri Nov 07 20:42:14 2008 +0000
+++ b/nsssl.c   Mon Sep 13 11:50:53 2010 +0200
@@ -844,7 +844,7 @@ SSLObjCmd(ClientData arg, Tcl_Interp *in
     }

     case HCancelIdx:
-        if (objc != 2) {
+        if (objc != 3) {
             Tcl_WrongNumArgs(interp, 2, objv, "id");
             return TCL_ERROR;
         }


Maybe someone could please checkt it.
Thanks in advance.

Best,
-- 
-vg

Re: [naviserver-devel] Socket buffer sizes

[Vasiljevic Z. <zv...@ar...>]

On 10.05.2010, at 21:13, Stephen Deasey wrote:

>
> Looks like it would be a mistake to use setsockopt() on Linux >=

I know that. Therefore I said that some OS'es do this automatically.
For good or for worse I will (re)add this knobs and leave defaults to
zero (== no changes) in both driver code and Tcl API calls. Who needs
or wants can play with them. Who doesn't will just use what OS is  
giving.
Reason: we have tons of customers running old OS versions and we need
to support them for some years to come.

Cheers
Zoran 

Re: [naviserver-devel] Socket buffer sizes

[Stephen D. <sd...@gm...>]

On Mon, May 10, 2010 at 7:22 PM, Andrew Piskorski <at...@pi...> wrote:
> On Mon, May 10, 2010 at 11:39:26AM +0200, Vasiljevic Zoran wrote:
>
>> So far I could understand from reading tons of docs
>> found all over the internet, the socket buffer sizes are
>> crucial for optimizing the network peformance related to
>> fast, high-latency links.
>
> Sounds like what the hpn-ssh patches do for ssh:
>
> http://www.psc.edu/networking/projects/hpn-ssh/


Zoran:

Looks like it would be a mistake to use setsockopt() on Linux >=
2.6.17 (released June 2006) ie. RHEL 5+ (ignoring any patches Redhat
may have backported):

    http://www.psc.edu/networking/projects/tcptune/#detailed
    http://kbase.redhat.com/faq/docs/DOC-3079

Re: [naviserver-devel] Socket buffer sizes

[Andrew P. <at...@pi...>]

On Mon, May 10, 2010 at 11:39:26AM +0200, Vasiljevic Zoran wrote:

> So far I could understand from reading tons of docs
> found all over the internet, the socket buffer sizes are
> crucial for optimizing the network peformance related to
> fast, high-latency links.

Sounds like what the hpn-ssh patches do for ssh:

http://www.psc.edu/networking/projects/hpn-ssh/

-- 
Andrew Piskorski <at...@pi...>
http://www.piskorski.com/

Re: [naviserver-devel] Socket buffer sizes

[Vasiljevic Z. <zv...@ar...>]

On 10.05.2010, at 14:38, Stephen Deasey wrote:

>
> Should there be some way to 'ignore' this if the OS can handle it
> automatically?  Maybe you just pass [ns_config socket bufsize] to the
> option and if it's 0, ignore it...?

Sure. This is what I wanted to do. On zero, ignore (== don't set).
As of driver code... the same would apply. Default is zero so no
changes. One can influence this from the outside if needed.

Cheers
Zoran

Re: [naviserver-devel] Socket buffer sizes

[Stephen D. <sd...@gm...>]

On Mon, May 10, 2010 at 12:39 PM, Vasiljevic Zoran <zv...@ar...> wrote:
>
> What I have in mind is to:
>
> a. make necessary changes to the driver code so that socket options
> are set before we actually call listen().
>
> The a. is really the most work as it will require either chaning
> existing (sub-optimal) and/or adding new (more optimal) C-APIs.


Can you just take this from the config file? Make the default 0 and if
0, don't config the bufsize.


> b. add ?-socketbuffersize size? option to all Tcl API commands that
> work with sockets.
>
> The b. is relatively harmless and should be straight-forward once
> the a. is done.


Should there be some way to 'ignore' this if the OS can handle it
automatically?  Maybe you just pass [ns_config socket bufsize] to the
option and if it's 0, ignore it...?

Re: [naviserver-devel] Socket buffer sizes

[Vasiljevic Z. <zv...@ar...>]

On 10.05.2010, at 12:55, Stephen Deasey wrote:
>
> Which code are you looking at? IIRC, you removed this code years ago:
>
>    http://bitbucket.org/naviserver/naviserver/changeset/a6e8a6348da0
>

We do still have people using old'er OS'es and by scrapping this out
they were forced to change buffer sizes on the OS level (system-wide)
which created side-effects we are now fighting with. And it is very
"expensive" to do this on the customer site :-(

Hence I thought to re-vive the socket buffers manipulation and
extend it to all API (including Tcl). But this time I wanted to make
it "correct".

Mostly people using HTTP services only will not be really caring too
much about this. But we use HTTP connections to shuffle TB of data
accross networks and we are very interested in doing all sorts of
optimisations that we can.

What I have in mind is to:

a. make necessary changes to the driver code so that socket options
are set before we actually call listen().

b. add ?-socketbuffersize size? option to all Tcl API commands that
work with sockets.

The a. is really the most work as it will require either chaning
existing (sub-optimal) and/or adding new (more optimal) C-APIs.

The b. is relatively harmless and should be straight-forward once
the a. is done.

The default should be "no changes" as mostly the OS'es would do
the right thing.

BTW... I have really found out that only newer Linuxes do this
automatically (re-size buffers to accomondate to high-latency links).
So far neither Solaris nor Mac OSX nor Windows do this.
I may be wrong but I was not able to find anything related on the  
internet.

Cheers
Zoran

Re: [naviserver-devel] Socket buffer sizes

[Stephen D. <sd...@gm...>]

On Mon, May 10, 2010 at 10:39 AM, Vasiljevic Zoran <zv...@ar...> wrote:
> Hi!
>
> I am in the process of evaluating the possibility to
> "correct" the current behaviour WRT manipulating
> socket buffer sizes...
>
> So far I could understand from reading tons of docs
> found all over the internet, the socket buffer sizes are
> crucial for optimizing the network peformance related to
> fast, high-latency links. We have customers that would
> benefit from this so I need to find the way to make this
> happen in the Naviserver.


We talked about this in summer 2008, I think. It will be in your mail
or mailing list archives somewhere...


> I did find out that different OS'es handle such cases
> differently. Most of them require admins to tune either
> system or per-app buffer sizes to achieve good BDP
> (Bandwidh-Delay Product). Some (newer Linux'es) do this
> automatically.


Newer is relative. I think by now this means any linux, bsd, solaris
less than a decade old...


> Currently we do already manipulate the socket buffer sizes
> in the main driver code (only).
> Unfortunately this manipulation happens (at least for the
> read-buffer-size) too late! As I understand, the socket
> buffers for reading must be setup BEFORE listen() call.
> Whereas we manipulate the sizes AFTER the connection is
> established.


Which code are you looking at? IIRC, you removed this code years ago:

    http://bitbucket.org/naviserver/naviserver/changeset/a6e8a6348da0

[naviserver-devel] Socket buffer sizes

[Vasiljevic Z. <zv...@ar...>]

Hi!

I am in the process of evaluating the possibility to
"correct" the current behaviour WRT manipulating
socket buffer sizes...

So far I could understand from reading tons of docs
found all over the internet, the socket buffer sizes are
crucial for optimizing the network peformance related to
fast, high-latency links. We have customers that would
benefit from this so I need to find the way to make this
happen in the Naviserver.

I did find out that different OS'es handle such cases
differently. Most of them require admins to tune either
system or per-app buffer sizes to achieve good BDP
(Bandwidh-Delay Product). Some (newer Linux'es) do this
automatically.

Currently we do already manipulate the socket buffer sizes
in the main driver code (only).
Unfortunately this manipulation happens (at least for the
read-buffer-size) too late! As I understand, the socket
buffers for reading must be setup BEFORE listen() call.
Whereas we manipulate the sizes AFTER the connection is
established.
Furthermore, the Tcl API commands related to sockets
(like ns_socklisten et.al.) also need to be expanded to
allow for explicit buffer size manipulation.

This all would definitely require some internal changes
perhaps even adding new Ns API calls.

I wonder if anybody already thought about this topic and
what was the outcome?

Cheers
Zoran

Re: [naviserver-devel] Ns_ConnWriteData needs Ns_ConnFlushData ?

[Vasiljevic Z. <zv...@ar...>]

On 05.05.2010, at 16:54, Stephen Deasey wrote:

> In general:
>
>    result = Ns_ConnReturnData(conn, status, data, data_length,  
> mime_type);
>
> For responses without a body:
>
>    result = Ns_ConnReturnStatus(conn, status);
>
> For 304's in particular:
>
>    result = Ns_ConnReturnNotModified(conn);
>
> ...and related in nsd/returnresp.c.  If there's a useful HTTP response
> that's missing, you should probably just add it.

Perfect. Even better than I thought :-)

>
>
> This example is kid of buggy:

It is. Just made quick one to illustrate what's up.
But a good explanation! Thanks. Have learned something new.

Cheers
Zoran

Re: [naviserver-devel] Ns_ConnWriteData needs Ns_ConnFlushData ?

[Stephen D. <sd...@gm...>]

On Wed, May 5, 2010 at 8:46 AM, Vasiljevic Zoran <zv...@ar...> wrote:
>
> On 04.05.2010, at 18:58, Stephen Deasey wrote:
>
>>
>> What is being buffered that you're trying to flush?
>
> The headers! It is just a kind-of 304-type of response or
> a response containing just the headers, no data.
>
> In the "previous life" I used SetRequiredHeaders
> and then FlushHeaders. Now I need to set each thing
> per line and then call Ns_ConnWrite with NULL buffer:
>
> was:
>
>          Ns_ConnSetRequiredHeaders(conn, mime, size);
>          result = Ns_ConnFlushHeaders(conn, 200);
>
> is:
>
>          Ns_ConnSetTypeHeader(conn, mime);
>          Ns_ConnSetLengthHeader(conn, size);
>          Ns_ConnSetResponseStatus(conn, 200);
>          if (Ns_ConnWriteData(conn, NULL, 0, 0) != NS_OK) {
>              result = 0;
>          } else {
>              result = size;
>          }
>
> So the 2 liner becomes 8 liner. It is not that I really
> care about that, but what I find obscure is the ConnWriteData.
> Something like this would be more "readable"
>
>          Ns_ConnSetTypeHeader(conn, mime);
>          Ns_ConnSetLengthHeader(conn, size);
>          Ns_ConnSetResponseStatus(conn, 200);
>         result = Ns_ConnFlush(conn);
>
> So the hypothetical Ns_ConnFlush would write all that it can
> and return the number of bytes written. Or something like that.
> Is there some other means of achieving the above that I am not
> aware of?


In general:

    result = Ns_ConnReturnData(conn, status, data, data_length, mime_type);

For responses without a body:

    result = Ns_ConnReturnStatus(conn, status);

For 304's in particular:

    result = Ns_ConnReturnNotModified(conn);

...and related in nsd/returnresp.c.  If there's a useful HTTP response
that's missing, you should probably just add it.


This example is kid of buggy:

>          Ns_ConnSetTypeHeader(conn, mime);
>          Ns_ConnSetLengthHeader(conn, size);
>          Ns_ConnSetResponseStatus(conn, 200);
>          if (Ns_ConnWriteData(conn, NULL, 0, 0) != NS_OK) {
>              result = 0;
>          } else {
>              result = size;
>          }

You've converted the NS_OK/NS_ERROR response from Ns_ConnWriteData to
a byte count, but that doesn't really tell you anything useful. In the
NS_ERROR case, some bytes might actually have been written, but you've
bashed it to zero. Setting result to size (the length header) is
misleading because the number of bytes written includes the bytes of
the headers, which the length header does not include. The byte count
can also be inflated because of character encoding and
chunked-encoding framing. It might actually be decreased because
compression.

What you can't do is compare bytes written to your original buffer
size and determine whether your write was successful. Which is one of
the reasons why FlushHeaders is depreciated.

Re: [naviserver-devel] Ns_ConnFlushHeaders() deprecated: why?

[Vasiljevic Z. <zv...@ar...>]

On 04.05.2010, at 18:48, Stephen Deasey wrote:

> Flushing headers is discouraged because it isn't needed for HTTP, it's
> slow, and often extra headers need to be added depending on which
> Write* calls you use.

See my other mail...
It is that my response is just headers, nothinh else. So I need
the means of "closing" the request and sending just the headers
w/o response body.

>
> I guess you could give access to nContentSent in a new public API.
> What are you using it for? How are you writing the data?

Well no data, just headers! I set some headers and want to send
them to the remote w/o supplying any data. I gave some example
in the other email to illustrate.

Cheers
Zoran

Re: [naviserver-devel] Ns_ConnWriteData needs Ns_ConnFlushData ?

[Vasiljevic Z. <zv...@ar...>]

On 04.05.2010, at 18:58, Stephen Deasey wrote:

>
> What is being buffered that you're trying to flush?

The headers! It is just a kind-of 304-type of response or
a response containing just the headers, no data.

In the "previous life" I used SetRequiredHeaders
and then FlushHeaders. Now I need to set each thing
per line and then call Ns_ConnWrite with NULL buffer:

was:

          Ns_ConnSetRequiredHeaders(conn, mime, size);
          result = Ns_ConnFlushHeaders(conn, 200);

is:

          Ns_ConnSetTypeHeader(conn, mime);
          Ns_ConnSetLengthHeader(conn, size);
          Ns_ConnSetResponseStatus(conn, 200);
          if (Ns_ConnWriteData(conn, NULL, 0, 0) != NS_OK) {
              result = 0;
          } else {
              result = size;
          }

So the 2 liner becomes 8 liner. It is not that I really
care about that, but what I find obscure is the ConnWriteData.
Something like this would be more "readable"

          Ns_ConnSetTypeHeader(conn, mime);
          Ns_ConnSetLengthHeader(conn, size);
          Ns_ConnSetResponseStatus(conn, 200);
	 result = Ns_ConnFlush(conn);

So the hypothetical Ns_ConnFlush would write all that it can
and return the number of bytes written. Or something like that.
Is there some other means of achieving the above that I am not
aware of?

Cheers
Zoran

Re: [naviserver-devel] Ns_ConnWriteData needs Ns_ConnFlushData ?

[Stephen D. <sd...@gm...>]

On Tue, May 4, 2010 at 4:58 PM, Vasiljevic Zoran <zv...@ar...> wrote:
> Hi!
>
> Quite often I see:
>
>    Ns_ConnWriteData(conn, NULL, 0, 0);
>
> This is OK but not very readable. Can we add something like
>
>    Ns_ConnFlushData(conn, flags)
>
> convenience wrapper that would supply the NULL buffer and
> zero bytes to the Ns_ConnWriteData() ?
>
> This would add zero functionality but would make the
> programmers intention more clear.


Hmm... well it's not *too* often that you see it. Some of the places
where it's currently done is probably a mistake, and sometimes it's
because the api doesn't quite let you do what you need.

For example, return.c:ReturnRange() flushes the headers by calling
Ns_ConnWriteData with a null buffer before calling
Ns_ConnSendFileVec() because the latter is low level enough, for
flexibility, that it completely ignores headers, for alternate
protocols etc. But the vector sending code can actually handle memory
buffers as well as files, so ideally you'd want to dump the headers to
a string buffer then pass them on to SendVec. Or something...

What is being buffered that you're trying to flush?