naviserver-devel — Developer discussion

Re: [naviserver-devel] [ns_conn peeraddr] always = 0.0.0.0 on OpenBSD 4.0/sparc64?

[Stephen D. <sd...@gm...>]

On 2/7/07, Vlad Seryakov <vl...@cr...> wrote:
> I updated configure.in in the CVS, can you run autogen.sh on CVS version
> and see if it will detect it right.
>

Actually I think you want AC_CHECK_FUNCS (with an 'S'), which
automatically defines the correct variables.

Just add it to the existing checks:

  AC_CHECK_FUNCS([timegm fork1 drand48 random _NSGetEnviron unsetenv inet_ntop])

Re: [naviserver-devel] [ns_conn peeraddr] always = 0.0.0.0 on OpenBSD 4.0/sparc64?

[Vlad S. <vl...@cr...>]

I updated configure.in in the CVS, can you run autogen.sh on CVS version 
and see if it will detect it right.

Michael A. Cleverly wrote:
> On 2/7/07, Vlad Seryakov <vl...@cr...> wrote:
>> But when we check for inet_ntop there is no u_int8_t or other type
>> references, just inet_ntop(0, (char *)0, (char *)0, 0);
>>
>> Can you check config.log for inet_ntop checking
> 
> It's needed by sys/socket.h which is included.
> 
> Here is that part of the config.log output:
> 
> configure:13970: checking for inet_ntop
> configure:13989: gcc -pipe -c -g -O2  conftest.c >&5
> In file included from conftest.c:57:
> /usr/include/sys/socket.h:145: error: syntax error before "u_int8_t"
> /usr/include/sys/socket.h:163: error: syntax error before "u_int8_t"
> /usr/include/sys/socket.h:166: error: syntax error before "u_int64_t"
> /usr/include/sys/socket.h:230: error: syntax error before "uid_t"
> /usr/include/sys/socket.h:235: error: syntax error before "gid_t"
> /usr/include/sys/socket.h:348: error: syntax error before "socklen_t"
> /usr/include/sys/socket.h:352: error: syntax error before "socklen_t"
> /usr/include/sys/socket.h:374: error: syntax error before "socklen_t"
> /usr/include/sys/socket.h:429: error: syntax error before "caddr_t"
> /usr/include/sys/socket.h:433: error: syntax error before "caddr_t"
> In file included from conftest.c:57:
> /usr/include/sys/socket.h:444: error: syntax error before "socklen_t"
> /usr/include/sys/socket.h:445: error: syntax error before "socklen_t"
> /usr/include/sys/socket.h:446: error: syntax error before "socklen_t"
> /usr/include/sys/socket.h:447: error: syntax error before "uid_t"
> /usr/include/sys/socket.h:448: error: syntax error before "socklen_t"
> /usr/include/sys/socket.h:449: error: syntax error before "socklen_t"
> /usr/include/sys/socket.h:450: error: syntax error before "socklen_t"
> /usr/include/sys/socket.h:452: error: syntax error before "recv"
> /usr/include/sys/socket.h:452: error: syntax error before "size_t"
> /usr/include/sys/socket.h:453: error: syntax error before "recvfrom"
> /usr/include/sys/socket.h:453: error: syntax error before "size_t"
> /usr/include/sys/socket.h:454: error: syntax error before "recvmsg"
> /usr/include/sys/socket.h:455: error: syntax error before "send"
> /usr/include/sys/socket.h:455: error: syntax error before "size_t"
> /usr/include/sys/socket.h:456: error: syntax error before "sendto"
> /usr/include/sys/socket.h:457: error: syntax error before "size_t"
> /usr/include/sys/socket.h:458: error: syntax error before "sendmsg"
> /usr/include/sys/socket.h:459: error: syntax error before "socklen_t"
> configure:13995: $? = 1
> configure: failed program was:
> | /* confdefs.h.  */
> |
> | #define PACKAGE_NAME "NaviServer"
> | #define PACKAGE_TARNAME "naviserver"
> | #define PACKAGE_VERSION "4.99.1"
> | #define PACKAGE_STRING "NaviServer 4.99.1"
> | #define PACKAGE_BUGREPORT "nav...@li..."
> | #define PACKAGE "naviserver"
> | #define VERSION "4.99.1"
> | #define NS_MAJOR_VERSION 4
> | #define NS_MINOR_VERSION 99
> | #define NS_RELEASE_SERIAL 1
> | #define NS_VERSION "4.99"
> | #define NS_PATCH_LEVEL "4.99.1"
> | #define NS_RELEASE_LEVEL NS_ALPHA_RELEASE
> | #define USE_THREAD_ALLOC 1
> | #define _REENTRANT 1
> | #define _THREAD_SAFE 1
> | #define TCL_THREADS 1
> | #define STDC_HEADERS 1
> | #define HAVE_SYS_TYPES_H 1
> | #define HAVE_SYS_STAT_H 1
> | #define HAVE_STDLIB_H 1
> | #define HAVE_STRING_H 1
> | #define HAVE_MEMORY_H 1
> | #define HAVE_STRINGS_H 1
> | #define HAVE_INTTYPES_H 1
> | #define HAVE_STDINT_H 1
> | #define HAVE_UNISTD_H 1
> | #define WORDS_BIGENDIAN 1
> | #define NO_VALUES_H 1
> | #define HAVE_LIMITS_H 1
> | #define HAVE_SYS_PARAM_H 1
> | #define TCL_WIDE_INT_IS_LONG 1
> | #define HAVE_SYS_TIME_H 1
> | #define TIME_WITH_SYS_TIME 1
> | #define HAVE_STRUCT_TM_TM_ZONE 1
> | #define HAVE_TM_ZONE 1
> | #define HAVE_GMTIME_R 1
> | #define HAVE_LOCALTIME_R 1
> | #define HAVE_TM_GMTOFF 1
> | #define HAVE_INTTYPES_H 1
> | #define HAVE_U_INT32_T 1
> | #define HAVE_U_INT8_T 1
> | #define HAVE_TIMEGM 1
> | #define HAVE_POLL 1
> | #define HAVE_DRAND48 1
> | #define HAVE_RANDOM 1
> | #define HAVE_GETADDRINFO 1
> | #define HAVE_GETNAMEINFO 1
> | #define HAVE_MTSAFE_DNS 1
> | #define HAVE_ZLIB_H 1
> | #define HAVE_LIBZ 1
> | #define HAVE_STRUCT_SOCKADDR_IN_SIN_LEN 1
> | #define HAVE_CMMSG 1
> | /* end confdefs.h.  */
> | #include <sys/socket.h>
> | #include <arpa/inet.h>
> | int
> | main ()
> | {
> | inet_ntop(0, (char *)0, (char *)0, 0);
> |   ;
> |   return 0;
> | }
> configure:14027: result: no
> 
> Michael
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier.
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] [ns_conn peeraddr] always = 0.0.0.0 on OpenBSD 4.0/sparc64?

[Michael A. C. <cle...@gm...>]

On 2/7/07, Vlad Seryakov <vl...@cr...> wrote:
> But when we check for inet_ntop there is no u_int8_t or other type
> references, just inet_ntop(0, (char *)0, (char *)0, 0);
>
> Can you check config.log for inet_ntop checking

It's needed by sys/socket.h which is included.

Here is that part of the config.log output:

configure:13970: checking for inet_ntop
configure:13989: gcc -pipe -c -g -O2  conftest.c >&5
In file included from conftest.c:57:
/usr/include/sys/socket.h:145: error: syntax error before "u_int8_t"
/usr/include/sys/socket.h:163: error: syntax error before "u_int8_t"
/usr/include/sys/socket.h:166: error: syntax error before "u_int64_t"
/usr/include/sys/socket.h:230: error: syntax error before "uid_t"
/usr/include/sys/socket.h:235: error: syntax error before "gid_t"
/usr/include/sys/socket.h:348: error: syntax error before "socklen_t"
/usr/include/sys/socket.h:352: error: syntax error before "socklen_t"
/usr/include/sys/socket.h:374: error: syntax error before "socklen_t"
/usr/include/sys/socket.h:429: error: syntax error before "caddr_t"
/usr/include/sys/socket.h:433: error: syntax error before "caddr_t"
In file included from conftest.c:57:
/usr/include/sys/socket.h:444: error: syntax error before "socklen_t"
/usr/include/sys/socket.h:445: error: syntax error before "socklen_t"
/usr/include/sys/socket.h:446: error: syntax error before "socklen_t"
/usr/include/sys/socket.h:447: error: syntax error before "uid_t"
/usr/include/sys/socket.h:448: error: syntax error before "socklen_t"
/usr/include/sys/socket.h:449: error: syntax error before "socklen_t"
/usr/include/sys/socket.h:450: error: syntax error before "socklen_t"
/usr/include/sys/socket.h:452: error: syntax error before "recv"
/usr/include/sys/socket.h:452: error: syntax error before "size_t"
/usr/include/sys/socket.h:453: error: syntax error before "recvfrom"
/usr/include/sys/socket.h:453: error: syntax error before "size_t"
/usr/include/sys/socket.h:454: error: syntax error before "recvmsg"
/usr/include/sys/socket.h:455: error: syntax error before "send"
/usr/include/sys/socket.h:455: error: syntax error before "size_t"
/usr/include/sys/socket.h:456: error: syntax error before "sendto"
/usr/include/sys/socket.h:457: error: syntax error before "size_t"
/usr/include/sys/socket.h:458: error: syntax error before "sendmsg"
/usr/include/sys/socket.h:459: error: syntax error before "socklen_t"
configure:13995: $? = 1
configure: failed program was:
| /* confdefs.h.  */
|
| #define PACKAGE_NAME "NaviServer"
| #define PACKAGE_TARNAME "naviserver"
| #define PACKAGE_VERSION "4.99.1"
| #define PACKAGE_STRING "NaviServer 4.99.1"
| #define PACKAGE_BUGREPORT "nav...@li..."
| #define PACKAGE "naviserver"
| #define VERSION "4.99.1"
| #define NS_MAJOR_VERSION 4
| #define NS_MINOR_VERSION 99
| #define NS_RELEASE_SERIAL 1
| #define NS_VERSION "4.99"
| #define NS_PATCH_LEVEL "4.99.1"
| #define NS_RELEASE_LEVEL NS_ALPHA_RELEASE
| #define USE_THREAD_ALLOC 1
| #define _REENTRANT 1
| #define _THREAD_SAFE 1
| #define TCL_THREADS 1
| #define STDC_HEADERS 1
| #define HAVE_SYS_TYPES_H 1
| #define HAVE_SYS_STAT_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_STRING_H 1
| #define HAVE_MEMORY_H 1
| #define HAVE_STRINGS_H 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_STDINT_H 1
| #define HAVE_UNISTD_H 1
| #define WORDS_BIGENDIAN 1
| #define NO_VALUES_H 1
| #define HAVE_LIMITS_H 1
| #define HAVE_SYS_PARAM_H 1
| #define TCL_WIDE_INT_IS_LONG 1
| #define HAVE_SYS_TIME_H 1
| #define TIME_WITH_SYS_TIME 1
| #define HAVE_STRUCT_TM_TM_ZONE 1
| #define HAVE_TM_ZONE 1
| #define HAVE_GMTIME_R 1
| #define HAVE_LOCALTIME_R 1
| #define HAVE_TM_GMTOFF 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_U_INT32_T 1
| #define HAVE_U_INT8_T 1
| #define HAVE_TIMEGM 1
| #define HAVE_POLL 1
| #define HAVE_DRAND48 1
| #define HAVE_RANDOM 1
| #define HAVE_GETADDRINFO 1
| #define HAVE_GETNAMEINFO 1
| #define HAVE_MTSAFE_DNS 1
| #define HAVE_ZLIB_H 1
| #define HAVE_LIBZ 1
| #define HAVE_STRUCT_SOCKADDR_IN_SIN_LEN 1
| #define HAVE_CMMSG 1
| /* end confdefs.h.  */
| #include <sys/socket.h>
| #include <arpa/inet.h>
| int
| main ()
| {
| inet_ntop(0, (char *)0, (char *)0, 0);
|   ;
|   return 0;
| }
configure:14027: result: no

Michael

Re: [naviserver-devel] [ns_conn peeraddr] always = 0.0.0.0 on OpenBSD 4.0/sparc64?

[Stephen D. <sd...@gm...>]

On 2/7/07, Vlad Seryakov <vl...@cr...> wrote:
> But when we check for inet_ntop there is no u_int8_t or other type
> references, just inet_ntop(0, (char *)0, (char *)0, 0);
>
> Can you check config.log for inet_ntop checking
>

The custom check is unnecessary. It should just be:

  AC_CHECK_FUNCS([inet_ntop])

which merely checks that the function is linkable in libc, which is enough.

Re: [naviserver-devel] [ns_conn peeraddr] always = 0.0.0.0 on OpenBSD 4.0/sparc64?

[Vlad S. <vl...@cr...>]

But when we check for inet_ntop there is no u_int8_t or other type 
references, just inet_ntop(0, (char *)0, (char *)0, 0);

Can you check config.log for inet_ntop checking

Michael A. Cleverly wrote:
> On 2/6/07, Stephen Deasey <sd...@gm...> wrote:
>> On 2/6/07, Vlad Seryakov <vl...@cr...> wrote:
>>> On Linux HAVE_INET_NTOP is defined, i assume on OpenBSD not and
>>> the part with the union is not working
>>
>> It should be defined, because it seems to have it:
>>
>>   http://www.openbsd.org/cgi-bin/man.cgi?query=inet_ntop
>>
>>
>> Anyway, our custom test for this seems a bit over the top. Maybe it's
>> not correct?  Other people seem to be just using AC_CHECK_FUNCS():
>>
>>   http://www.google.com/codesearch?q=file%3Aconfigure%5C.ac+inet_ntop
> 
> When I change include/Makefile.global from:
> 
>     DEFS             = -DHAVE_CONFIG_H
> 
> to:
> 
>     DEFS             = -DHAVE_CONFIG_H -DHAVE_INET_NTOP
> 
> and recompile I get correct IP addresses reported by [ns_conn
> peeraddr].  Also Vlad's a.c program reports 127.0.0.1 instead of
> 0.0.0.0 too.
> 
> So the check for inet_ntop() doesn't work right on OpenBSD/sparc64.
> The union alternative in reentrant.c works on 32-bit & 64-bit little
> endian OpenBSD it appears.
> 
> The check fails because u_int8_t, and others, are undefined.  On
> OpenBSD they are defined in sys/types.h but the check only includes
> sys/socket.h and arpa/inet.h.
> 
> Thanks for all the help!
> 
> Michael
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier.
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] [ns_conn peeraddr] always = 0.0.0.0 on OpenBSD 4.0/sparc64?

[Michael A. C. <cle...@gm...>]

On 2/6/07, Stephen Deasey <sd...@gm...> wrote:
> On 2/6/07, Vlad Seryakov <vl...@cr...> wrote:
> > On Linux HAVE_INET_NTOP is defined, i assume on OpenBSD not and
> > the part with the union is not working
>
>
> It should be defined, because it seems to have it:
>
>   http://www.openbsd.org/cgi-bin/man.cgi?query=inet_ntop
>
>
> Anyway, our custom test for this seems a bit over the top. Maybe it's
> not correct?  Other people seem to be just using AC_CHECK_FUNCS():
>
>   http://www.google.com/codesearch?q=file%3Aconfigure%5C.ac+inet_ntop

When I change include/Makefile.global from:

    DEFS             = -DHAVE_CONFIG_H

to:

    DEFS             = -DHAVE_CONFIG_H -DHAVE_INET_NTOP

and recompile I get correct IP addresses reported by [ns_conn
peeraddr].  Also Vlad's a.c program reports 127.0.0.1 instead of
0.0.0.0 too.

So the check for inet_ntop() doesn't work right on OpenBSD/sparc64.
The union alternative in reentrant.c works on 32-bit & 64-bit little
endian OpenBSD it appears.

The check fails because u_int8_t, and others, are undefined.  On
OpenBSD they are defined in sys/types.h but the check only includes
sys/socket.h and arpa/inet.h.

Thanks for all the help!

Michael

Re: [naviserver-devel] [ns_conn peeraddr] always = 0.0.0.0 on OpenBSD 4.0/sparc64?

[Vlad S. <vl...@cr...>]

Looks like broken inet_ntop on OpenBSD/sparc

Michael A. Cleverly wrote:
> On 2/6/07, Vlad Seryakov <vl...@cr...> wrote:
>> Try this one:
>>
>> #include <ns.h>
>>
>> main()
>> {
>>      unsigned char b[4];
>>      struct in_addr addr;
>>
>>      addr.s_addr = inet_addr("127.0.0.1");
>>      memcpy(b, &addr.s_addr, 4);
>>      printf("%u.%u.%u.%u\n", b[0], b[1], b[2], b[3]);
>> }
> 
> That produces 127.0.0.1 on OpenBSD 4.0/sparc64 :-)
> 
> Michael
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier.
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] [ns_conn peeraddr] always = 0.0.0.0 on OpenBSD 4.0/sparc64?

[Michael A. C. <cle...@gm...>]

On 2/6/07, Vlad Seryakov <vl...@cr...> wrote:
> Try this one:
>
> #include <ns.h>
>
> main()
> {
>      unsigned char b[4];
>      struct in_addr addr;
>
>      addr.s_addr = inet_addr("127.0.0.1");
>      memcpy(b, &addr.s_addr, 4);
>      printf("%u.%u.%u.%u\n", b[0], b[1], b[2], b[3]);
> }

That produces 127.0.0.1 on OpenBSD 4.0/sparc64 :-)

Michael

Re: [naviserver-devel] [ns_conn peeraddr] always = 0.0.0.0 on OpenBSD 4.0/sparc64?

[Michael A. C. <cle...@gm...>]

On 2/6/07, Michael A. Cleverly <cle...@gm...> wrote:
> On 2/6/07, Vlad Seryakov <vl...@cr...> wrote:
> > Try to compile this and see if you get 127.0.0.1 printed
> >
> > #include <ns.h>
> >
> > main()
> > {
> >     struct in_addr addr;
> >     addr.s_addr = inet_addr("127.0.0.1");
> >     printf("%s\n", ns_inet_ntoa(addr));
> > }
> >
> > gcc -I /usr/local/ns/include -o a a.c /usr/local/ns/lib/libnsthread.so
>
> michael@joe:~$ ./a
> 0.0.0.0

I get 127.0.0.1 when I compile Tcl 8.4.14, Naviserver & the above
program on OpenBSD 3.9/amd64 and OpenBSD 3.8/macppc.

So I wonder if it has to do with being on sparc64 being big endian
while amd64 is little endian?

Michael

Re: [naviserver-devel] [ns_conn peeraddr] always = 0.0.0.0 on OpenBSD 4.0/sparc64?

[Vlad S. <vl...@cr...>]

Try this one:

#include <ns.h>

main()
{
     unsigned char b[4];
     struct in_addr addr;

     addr.s_addr = inet_addr("127.0.0.1");
     memcpy(b, &addr.s_addr, 4);
     printf("%u.%u.%u.%u\n", b[0], b[1], b[2], b[3]);
}

Stephen Deasey wrote:
> On 2/6/07, Vlad Seryakov <vl...@cr...> wrote:
>> On Linux HAVE_INET_NTOP is defined, i assume on OpenBSD not and
>> the part with the union is not working
> 
> 
> It should be defined, because it seems to have it:
> 
>   http://www.openbsd.org/cgi-bin/man.cgi?query=inet_ntop
> 
> 
> Anyway, our custom test for this seems a bit over the top. Maybe it's
> not correct?  Other people seem to be just using AC_CHECK_FUNCS():
> 
>   http://www.google.com/codesearch?q=file%3Aconfigure%5C.ac+inet_ntop
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier.
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] [ns_conn peeraddr] always = 0.0.0.0 on OpenBSD 4.0/sparc64?

[Stephen D. <sd...@gm...>]

On 2/6/07, Vlad Seryakov <vl...@cr...> wrote:
> On Linux HAVE_INET_NTOP is defined, i assume on OpenBSD not and
> the part with the union is not working


It should be defined, because it seems to have it:

  http://www.openbsd.org/cgi-bin/man.cgi?query=inet_ntop


Anyway, our custom test for this seems a bit over the top. Maybe it's
not correct?  Other people seem to be just using AC_CHECK_FUNCS():

  http://www.google.com/codesearch?q=file%3Aconfigure%5C.ac+inet_ntop

Re: [naviserver-devel] [ns_conn peeraddr] always = 0.0.0.0 on OpenBSD 4.0/sparc64?

[Vlad S. <vl...@cr...>]

On Linux HAVE_INET_NTOP is defined, i assume on OpenBSD not and
the part with the union is not working

char *
ns_inet_ntoa(struct in_addr addr)
{
     Tls *tlsPtr = GetTls();
#if defined(HAVE_INET_NTOP)
     inet_ntop(AF_INET, &addr, tlsPtr->nabuf, sizeof(tlsPtr->nabuf));
#else
     union {
         unsigned long l;
         unsigned char b[4];
     } u;

     u.l = (unsigned long) addr.s_addr;
     sprintf(tlsPtr->nabuf, "%u.%u.%u.%u", u.b[0], u.b[1], u.b[2], u.b[3]);
#endif
     return tlsPtr->nabuf;
}


Michael A. Cleverly wrote:
> On 2/6/07, Vlad Seryakov <vl...@cr...> wrote:
>> Try to compile this and see if you get 127.0.0.1 printed
>>
>> #include <ns.h>
>>
>> main()
>> {
>>     struct in_addr addr;
>>     addr.s_addr = inet_addr("127.0.0.1");
>>     printf("%s\n", ns_inet_ntoa(addr));
>> }
>>
>> gcc -I /usr/local/ns/include -o a a.c /usr/local/ns/lib/libnsthread.so
> 
> michael@joe:~$ ./a
> 0.0.0.0
> 
> :-/
> 
> Michael
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier.
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] [ns_conn peeraddr] always = 0.0.0.0 on OpenBSD 4.0/sparc64?

[Michael A. C. <cle...@gm...>]

On 2/6/07, Vlad Seryakov <vl...@cr...> wrote:
> Try to compile this and see if you get 127.0.0.1 printed
>
> #include <ns.h>
>
> main()
> {
>     struct in_addr addr;
>     addr.s_addr = inet_addr("127.0.0.1");
>     printf("%s\n", ns_inet_ntoa(addr));
> }
>
> gcc -I /usr/local/ns/include -o a a.c /usr/local/ns/lib/libnsthread.so

michael@joe:~$ ./a
0.0.0.0

:-/

Michael

Re: [naviserver-devel] [ns_conn peeraddr] always = 0.0.0.0 on OpenBSD 4.0/sparc64?

[Vlad S. <vl...@cr...>]

Try to compile this and see if you get 127.0.0.1 printed


#include <ns.h>

main()
{
    struct in_addr addr;
    addr.s_addr = inet_addr("127.0.0.1");
    printf("%s\n", ns_inet_ntoa(addr));
}

gcc -I /usr/local/ns/include -o a a.c /usr/local/ns/lib/libnsthread.so


Michael A. Cleverly wrote:
> I've compiled naviserver on OpenBSD 4.0/sparc64.  What baffles me is
> that [ns_conn peeraddr] always returns 0.0.0.0 for the client IP.  The
> access.log also shows 0.0.0.0 as the source IP address.
> 
> For example:
> 
> 0.0.0.0 - - [05/Feb/2007:19:08:16 -0700] "GET / HTTP/1.1" 200 865 ""
> "Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.0.7)
> Gecko/20060911 Camino/1.0.3" 3.562137
> 0.0.0.0 - - [05/Feb/2007:20:45:37 -0700] "GET / HTTP/1.0" 200 865 ""
> "Lynx/2.8.5rel.4 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.7j" 4.294168
> 
> The first is from my powerbook; the second is from lynx (local to the box).
> 
> Just to rule out some completely screwed up tcp configuration I wrote
> a small tclsh script.  When I point a browser to it tclsh knows the
> real peer's address.
> 
>     proc accept {sock peer port} {after 1000 [list reply $sock]}
> 
>     proc reply {sock} {
>         puts $sock "HTTP/1.0 200 OK"
>         puts $sock "MIME-Version: 1.0"
>         puts $sock "Content-Type: text/plain"
>         puts $sock ""
>         puts $sock [fconfigure $sock -peername]
>         close $sock
>     }
> 
>     socket -server accept -myaddr 0.0.0.0 80
>     vwait forever
> 
> I get:
> 
>     67.172.241.159 c-67-172-241-159.hsd1.ut.comcast.net 64323
>     127.0.0.1 localhost 4660
> 
> Any thoughts on why nsd doesn't see the real peer address?
> 
> Thanks,
> 
> Michael
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier.
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

[naviserver-devel] [ns_conn peeraddr] always = 0.0.0.0 on OpenBSD 4.0/sparc64?

[Michael A. C. <cle...@gm...>]

I've compiled naviserver on OpenBSD 4.0/sparc64.  What baffles me is
that [ns_conn peeraddr] always returns 0.0.0.0 for the client IP.  The
access.log also shows 0.0.0.0 as the source IP address.

For example:

0.0.0.0 - - [05/Feb/2007:19:08:16 -0700] "GET / HTTP/1.1" 200 865 ""
"Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.0.7)
Gecko/20060911 Camino/1.0.3" 3.562137
0.0.0.0 - - [05/Feb/2007:20:45:37 -0700] "GET / HTTP/1.0" 200 865 ""
"Lynx/2.8.5rel.4 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.7j" 4.294168

The first is from my powerbook; the second is from lynx (local to the box).

Just to rule out some completely screwed up tcp configuration I wrote
a small tclsh script.  When I point a browser to it tclsh knows the
real peer's address.

    proc accept {sock peer port} {after 1000 [list reply $sock]}

    proc reply {sock} {
        puts $sock "HTTP/1.0 200 OK"
        puts $sock "MIME-Version: 1.0"
        puts $sock "Content-Type: text/plain"
        puts $sock ""
        puts $sock [fconfigure $sock -peername]
        close $sock
    }

    socket -server accept -myaddr 0.0.0.0 80
    vwait forever

I get:

    67.172.241.159 c-67-172-241-159.hsd1.ut.comcast.net 64323
    127.0.0.1 localhost 4660

Any thoughts on why nsd doesn't see the real peer address?

Thanks,

Michael

Re: [naviserver-devel] Driver with epoll

[Vlad S. <vl...@cr...>]

We do not need it, using sendfile as etc has its own limitations, only 
for static files, no templates and other dynamic stuff. For only static 
files lighttpd is fine and it is more effective to use it instead of 
using naviserver

Stephen Deasey wrote:
> On 1/23/07, Zoran Vasiljevic <zv...@ar...> wrote:
>> Am 23.01.2007 um 17:14 schrieb Vlad Seryakov:
>>
>>> and lighttpd also uses sendfile which is faster than user-spave
>>> read-send operation fastpath uses
>> There you go. They do it all in kernel. I guess this can't be beat.
>> Anyways, it is good to know where we stand.
>>
> 
> I think this may only apply to larger files, and some old benchmarks I
> saw showed it was actualy slower for smaller files.  Not that this
> isn't important, but it's not the second coming...  :-)
> 
> For sendfile() to work we need to figure out the new driver callback
> interface so we can add a Sendfile() callback.  Otherwise, large file
> downloads will break with SSL and other socket drivers.
> 
> There's stuff about the driver interface in the archives...
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Driver with epoll

[Vlad S. <vl...@cr...>]

> 
> How are you using ab?  What exactly are you testing?

ab -n 1000 -n 150000 http://localhost/1

> And remember why poll() is inefficient: it's the scanning of the array
> of active file descriptors. The whole array needs to be scanned even
> if only one socket has a new event.  Therefore you'd expect
> performance problems to show up with extremely large numbers of active
> connections. If you're not testing with concurrency of 1000 ~ 4000
> connections, I wouldn't expect to see much of a difference, and that
> will be tough with ab which uses one thread per connection, IIRC.

Still slower

> It's difficult to compare your epoll() implementation to the original
> poll() as it's spread throughout the driver.  Please consider using
> something like NsPoll(), which is a very thin wrapper.

What i 've sent is a piece of code that should replace DriverThread 
function in driver.c, after that it compiles and runs. No spreading

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Driver with epoll

[Stephen D. <sd...@gm...>]

On 1/23/07, Zoran Vasiljevic <zv...@ar...> wrote:
>
> Am 23.01.2007 um 17:14 schrieb Vlad Seryakov:
>
> > and lighttpd also uses sendfile which is faster than user-spave
> > read-send operation fastpath uses
>
> There you go. They do it all in kernel. I guess this can't be beat.
> Anyways, it is good to know where we stand.
>

I think this may only apply to larger files, and some old benchmarks I
saw showed it was actualy slower for smaller files.  Not that this
isn't important, but it's not the second coming...  :-)

For sendfile() to work we need to figure out the new driver callback
interface so we can add a Sendfile() callback.  Otherwise, large file
downloads will break with SSL and other socket drivers.

There's stuff about the driver interface in the archives...

Re: [naviserver-devel] Driver with epoll

[Stephen D. <sd...@gm...>]

On 1/23/07, Vlad Seryakov <vl...@cr...> wrote:
> Hi,
>
> I was playing with epoll and changed driver to see if performance will
> be better with it. Using ab utility i actually got worse performance
> with epoll, i suspect may be i implemented it not very effectively.


How are you using ab?  What exactly are you testing?

Remember: epoll() will increase the efficiency of the driver loop, but
you've already optimised that by applying the patch which accepts
multiple new connections on each iteration, so there will be
significantly less iterations to optimise.

And remember why poll() is inefficient: it's the scanning of the array
of active file descriptors. The whole array needs to be scanned even
if only one socket has a new event.  Therefore you'd expect
performance problems to show up with extremely large numbers of active
connections. If you're not testing with concurrency of 1000 ~ 4000
connections, I wouldn't expect to see much of a difference, and that
will be tough with ab which uses one thread per connection, IIRC.


It's difficult to compare your epoll() implementation to the original
poll() as it's spread throughout the driver.  Please consider using
something like NsPoll(), which is a very thin wrapper.

Re: [naviserver-devel] Driver with epoll

[Zoran V. <zv...@ar...>]

Am 23.01.2007 um 17:14 schrieb Vlad Seryakov:

> and lighttpd also uses sendfile which is faster than user-spave
> read-send operation fastpath uses

There you go. They do it all in kernel. I guess this can't be beat.
Anyways, it is good to know where we stand.

Re: [naviserver-devel] Driver with epoll

[Vlad S. <vl...@cr...>]

I know, i just read over lighttpd and other server who uses epoll/kqueue 
and all emphasize usage of advanced polling systems. So my goal was to 
check if epoll alone adds any performance, i tested server hitting wrong 
url, so server responded with 404 in both cases.

Last time i tested lighttpd and naviserver with returning static file 
and lighttpd also uses sendfile which is faster than user-spave 
read-send operation fastpath uses. Also, naviserver uses generic 
filters/urlspace mechanism which is slower than direct file return in 
lighttpd so i am aware that we are not designed to be extremely fast in 
returning static files.

Zoran Vasiljevic wrote:
> Am 23.01.2007 um 16:54 schrieb Vlad Seryakov:
> 
>> I was playing with epoll and changed driver to see if performance will
>> be better with it. Using ab utility i actually got worse performance
>> with epoll, i suspect may be i implemented it not very effectively.
> 
> 
> Perhaps we should split the processing in chunks and
> measure how much % of time each chunk requires.
> Then we'd know better what knobs to adjust?
> 
> I recall you telling something about 8000 vs. 2000
> req/sec in favour of lighthttpd? Well, thats 4 TIMES.
> Obviously, there is something much deeper that is different
> in the design. I could not imagine epoll() usage is the
> key.  It must be something else!
> 
> 
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Driver with epoll

[Zoran V. <zv...@ar...>]

Am 23.01.2007 um 16:54 schrieb Vlad Seryakov:

> I was playing with epoll and changed driver to see if performance will
> be better with it. Using ab utility i actually got worse performance
> with epoll, i suspect may be i implemented it not very effectively.


Perhaps we should split the processing in chunks and
measure how much % of time each chunk requires.
Then we'd know better what knobs to adjust?

I recall you telling something about 8000 vs. 2000
req/sec in favour of lighthttpd? Well, thats 4 TIMES.
Obviously, there is something much deeper that is different
in the design. I could not imagine epoll() usage is the
key.  It must be something else!

[naviserver-devel] Driver with epoll

[Vlad S. <vl...@cr...>]

Hi,

I was playing with epoll and changed driver to see if performance will 
be better with it. Using ab utility i actually got worse performance 
with epoll, i suspect may be i implemented it not very effectively.

----------------------------------------------------------------------------------------

#include <sys/epoll.h>

#define MAX_POLL                  4096

#define POLL_TRIGGER              1
#define POLL_DRIVER               2
#define POLL_SOCK                 3
#define POLL_CLOSE                4

#define EPOLL_ADD(sock, dtype, dptr)     ev.events = EPOLLIN | EPOLLERR 
| EPOLLHUP;\
                                          ev.data.ptr = &pollType[sock]; \
                                          pollType[sock].fd = sock; \
                                          pollType[sock].type = dtype; \
                                          pollType[sock].ptr = dptr; \
                                          rc = epoll_ctl(epoll_fd, 
EPOLL_CTL_ADD, sock, &ev); \
                                          //Ns_Log(Notice, "epoll_add: 
%d %d %p = %d %d", sock, dtype, dptr, rc, errno);

#define EPOLL_DEL(sock, msg)             pollType[sock].type = 0; \
                                          rc = epoll_ctl(epoll_fd, 
EPOLL_CTL_DEL, sock, NULL); \
                                          //Ns_Log(Notice, "epoll_del: 
%s %d = %d %d", msg, sock, rc, errno);
typedef struct {
     int fd;
     int type;
     void *ptr;
} PollType;

static void
DriverThread(void *ignored)
{
     PollType       pollType[MAX_POLL], *poll;
     struct         epoll_event ev, pollEvent[MAX_POLL];
     char          *errstr, c, drain[1024];
     int            rc, n, i, nfds, stopping, accepted, epoll_fd;
     Sock          *sockPtr, *closePtr, *nextPtr, *waitPtr, *readPtr;
     Ns_Time        now, diff;
     Driver        *drvPtr;

     Ns_ThreadSetName("-driver-");

     /*
      * Loop forever until signalled to shutdown and all
      * connections are complete and gracefully closed.
      */

     Ns_Log(Notice, "driver: accepting connections");
     closePtr = waitPtr = readPtr = NULL;
     Ns_GetTime(&now);
     stopping = 0;

     epoll_fd = epoll_create(MAX_POLL);
     EPOLL_ADD(drvPipe[0], POLL_TRIGGER, NULL);

     drvPtr = firstDrvPtr;
     while (drvPtr != NULL) {
         if (drvPtr->sock != INVALID_SOCKET) {
             EPOLL_ADD(drvPtr->sock, POLL_DRIVER, drvPtr);
             drvPtr = drvPtr->nextPtr;
         }
     }

     while (!stopping) {

         /*
          * Set the bits for all active drivers if a connection
          * isn't already pending.
          */

         nfds = epoll_wait(epoll_fd, pollEvent, MAX_POLL, 1000);

         Ns_GetTime(&now);

         for (i = 0; i < nfds; i++) {

             poll = (PollType*)pollEvent[i].data.ptr;

             switch (poll->type) {
              case POLL_TRIGGER:

                 if ((pollEvent[i].events & EPOLLIN)
                      && recv(drvPipe[0], &c, 1, 0) != 1) {
                      errstr = ns_sockstrerror(ns_sockerrno);
                      Ns_Fatal("driver: trigger recv() failed: %s", errstr);
                 }
                 break;

              case POLL_CLOSE:

                 /*
                  * Update the current time and drain and/or release any
                  * closing sockets.
                  */

                 sockPtr = (Sock*)poll->ptr;

                 if (pollEvent[i].events & EPOLLIN) {
                     n = recv(sockPtr->sock, drain, sizeof(drain), 0);
                     if (n <= 0) {
                         sockPtr->timeout = now;
                     }
                 }
                 if (Ns_DiffTime(&sockPtr->timeout, &now, &diff) <= 0) {
                     EPOLL_DEL(sockPtr->sock, "close: timeout");
                     SockRelease(sockPtr, SOCK_CLOSETIMEOUT, 0);
                 }
                 break;

              case POLL_SOCK:

                 /*
                  * Attempt read-ahead of any new connections.
                  */

                 sockPtr = (Sock*)poll->ptr;

                 if (!(pollEvent[i].events & EPOLLIN)) {
                     if (Ns_DiffTime(&sockPtr->timeout, &now, &diff) <= 0) {
                         EPOLL_DEL(sockPtr->sock, "sock: timeout");
                         SockRelease(sockPtr, SOCK_READTIMEOUT, 0);
                     }

                 } else {

                     /*
                      * If enabled, perform read-ahead now.
                      */

                     sockPtr->keep = 0;
                     if (sockPtr->drvPtr->opts & NS_DRIVER_ASYNC) {
                         n = SockRead(sockPtr, 0);
                     } else {
                         n = SOCK_READY;
                     }

                     /*
                      * Queue for connection processing if ready.
                      */

                     switch (n) {
                     case SOCK_MORE:
                         SockTimeout(sockPtr, &now, 
sockPtr->drvPtr->recvwait);
                         break;

                     case SOCK_SPOOL:
                         EPOLL_DEL(sockPtr->sock, "sock: spool");
                         SockSpoolerQueue(sockPtr->drvPtr, sockPtr);
                         break;

                     case SOCK_READY:
                         EPOLL_DEL(sockPtr->sock, "sock: ready");
                         if (SockQueue(sockPtr, &now) == NS_TIMEOUT) {
                             Push(sockPtr, waitPtr);
                         }
                         break;

                     default:
                         EPOLL_DEL(sockPtr->sock, "sock: unknown");
                         SockRelease(sockPtr, n, errno);
                         break;
                     }
                 }
                 break;

              case POLL_DRIVER:

                 /*
                  * If configured, try to accept more than one request, 
under heavy load
                  * this helps to process more requests
                  */

                 accepted = 0;
                 drvPtr = (Driver*)poll->ptr;

                 while (accepted < drvPtr->acceptsize &&
                        drvPtr->queuesize < drvPtr->maxqueuesize
                        && (pollEvent[i].events & EPOLLIN)
                        && (sockPtr = SockAccept(drvPtr)) != NULL) {

                     /*
                      * Queue the socket immediately if request is provided
                      */

                     if (sockPtr->drvPtr->opts & NS_DRIVER_QUEUE_ONACCEPT) {

                         if (SockQueue(sockPtr, &now) == NS_TIMEOUT) {
                             Push(sockPtr, waitPtr);
                         }

                     } else {

                        /*
                         * Put the socket on the read-ahead list.
                         */

                         EPOLL_ADD(sockPtr->sock, POLL_SOCK, sockPtr);
                         SockTimeout(sockPtr, &now, 
sockPtr->drvPtr->recvwait);
                     }
                     accepted++;
                 }
                 break;

               default:

                 /*
                  * Unknown socket ended up in the queue
                  */

                 EPOLL_DEL(poll->fd, "unknown");
                 Ns_Log(Notice, "driver: %d: wrong type for %d", n, 
poll->fd);
             }
         }

         /*
          * Attempt to queue any pending connection
          * after reversing the list to ensure oldest
          * connections are tried first.
          */

         if (waitPtr != NULL) {
             sockPtr = NULL;
             while ((nextPtr = waitPtr) != NULL) {
                 waitPtr = nextPtr->nextPtr;
                 Push(nextPtr, sockPtr);
             }

             while (sockPtr != NULL) {
                 nextPtr = sockPtr->nextPtr;
                 if (SockQueue(sockPtr, &now) == NS_TIMEOUT) {
                     Push(sockPtr, waitPtr);
                 }
                 sockPtr = nextPtr;
             }
         }

         /*
          * Check for shutdown and get the list of any closing
          * or keepalive sockets.
          */

         Ns_MutexLock(&drvLock);
         sockPtr       = firstClosePtr;
         firstClosePtr = NULL;
         stopping      = driverShutdown;
         Ns_MutexUnlock(&drvLock);

         /*
          * Update the timeout for each closing socket and add to the
          * close list if some data has been read from the socket
          * (i.e., it's not a closing keep-alive connection).
          */

         while (sockPtr != NULL) {
             nextPtr = sockPtr->nextPtr;
             if (sockPtr->keep) {
                 EPOLL_ADD(sockPtr->sock, POLL_SOCK, sockPtr);
                 SockTimeout(sockPtr, &now, sockPtr->drvPtr->keepwait);
             } else {
                 if (shutdown(sockPtr->sock, 1) != 0) {
                     SockRelease(sockPtr, SOCK_SHUTERROR, errno);
                 } else {
                     EPOLL_ADD(sockPtr->sock, POLL_CLOSE, sockPtr);
                     SockTimeout(sockPtr, &now, sockPtr->drvPtr->closewait);
                 }
             }
             sockPtr = nextPtr;
         }

         /*
          * Close the active drivers if shutdown is pending.
          */

         if (stopping) {
             drvPtr = firstDrvPtr;
             while (drvPtr != NULL) {
                 if (drvPtr->sock != INVALID_SOCKET) {
                     ns_sockclose(drvPtr->sock);
                     drvPtr->sock = INVALID_SOCKET;
                 }
                 drvPtr = drvPtr->nextPtr;
             }
         }
     }

     Ns_Log(Notice, "exiting");
     Ns_MutexLock(&drvLock);
     drvStopped = 1;
     Ns_CondBroadcast(&drvCond);
     Ns_MutexUnlock(&drvLock);
}


-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] [naviserver-commits] vtmalloc ChangeLog, 1.4, 1.5 tclThreadAlloc.c, 1.2, 1.3

[Zoran V. <zv...@ar...>]

Am 23.01.2007 um 16:24 schrieb Stephen Deasey:

> I thought the goal was less memory fragmentation -- not necessarily
> speed which was pretty good with zippy -- and fragmentation was solved
> by using mmap for memory which could be returned to the system.
>

Well, we wouldn't like to be slower than zippy. At least not
more than 25%. Returning the memory to the system is now
relatively suboptimal as it happens only on thread exit
which is problematic for threads that never exit (driver.c).

> So fragmentation is only a problem for small allocations, in the  
> real world?
>

Mostly yes.

> Also, why did you choose this strategy, using the system malloc for >
> 8000b, rather than modify the way the per thread mmap pools are
> released to the OS.  This used to be on thread exit, right?  If this
> was causing bloating in long-lived threads, you could flush the pool
> when some low water mark is reached.
>

One after another... this is still on the todo list of mine.
It requires quite a bit of testing and needs time. Coding is
simple, testing is complex. The limitation to 8K was simple
way to get less bloating with large concurrent number of threads.
But we'd still have to return memory to the system or for
reusal to other threads more agressively than we do now. The
tricky part is to find the optimum between the speed and the
memory consumption, having in mind the typical usage patterns
that we encounter in naviserver and applications built on
top of it.

Re: [naviserver-devel] [naviserver-commits] vtmalloc ChangeLog, 1.4, 1.5 tclThreadAlloc.c, 1.2, 1.3

[Stephen D. <sd...@gm...>]

On 1/23/07, Zoran Vasiljevic <zv...@ar...> wrote:
>
> Am 23.01.2007 um 15:42 schrieb Zoran Vasiljevic:
>
> > This way the system allocator
> > is handling those and it can toss free'd blocks
> > to other threads reducing peak memory usage.
> > Since allocations of that sizes (and larger) are
> > pretty rare, we need not worry about locking
> > contention as it is the case with smaller allocations.
>
> More info...
>
> Actually that lowers the amount of memory released
> to the system (as system allocs mostly never do that)
> but it exremely reduces the peak memory usage.
> In our app, when we start a job, the VM goes to about 300MB
> with initial VTmalloc, to 235MB with modified VTmalloc
> (8K allocs) and to about 225MB with system alloc.
> At the same time, VTmalloc is up to 20 times faster than
> system alloc (thats on Mac). Thats a pretty good bargain
> for us: invest about 10MB more but get far better performance
> is absolutely acceptable. I haven't tested Linux or Solaris
> yet, but I guess I will have the same result (apart from the
> difference between the OS alloc and us).
>
> What I would like to do is to lower that usage even more
> on the cost of more locking contention, as we can tolerate
> that. Just to see how low and fast we can go.
>


I thought the goal was less memory fragmentation -- not necessarily
speed which was pretty good with zippy -- and fragmentation was solved
by using mmap for memory which could be returned to the system.

So fragmentation is only a problem for small allocations, in the real world?

Also, why did you choose this strategy, using the system malloc for >
8000b, rather than modify the way the per thread mmap pools are
released to the OS.  This used to be on thread exit, right?  If this
was causing bloating in long-lived threads, you could flush the pool
when some low water mark is reached.