naviserver-devel — Developer discussion

Re: [naviserver-devel] lurking bugs: conn threads

[Jeff R. <dv...@di...>]

Stephen Deasey wrote:
> Interesting, but I wonder if we're not thinking this through
> correctly. My suggestion, and your here, and Gustaf's recet work are
> all aimed at refining the model as it currently is, but I wonder if
> we're even attempting to do the right thing?

Do we even know what the right thing is?  It could be any of
- maximize performance at any cost
- minimize resource usage
- adapt to dynamically changing workload
- minimize admin workload

And so forth.  I think there is no one-size-fits-all answer, but it 
should be possible, and hopefully easy, to get something that fits 
closely enough.

>> So I'm assuming that the available processing power - the number of
>> threads - should correlate to how busy the server is.  A server that is
>> 50% busy should have 50% of its full capacity working.
>
> But what is busy, CPU?  There needs to be an appropriate max number of
> threads to handle the max expected load, considering the capabilities
> of the machine. Too many and the machine will run slower. But why kill
> them when we're no longer busy?

I don't know how to precisely define, let alone measure busy, which is 
why I'm picking something that is readily measurable.

A thread is busy in the sense that it is unavailable to process new 
requests, but there are different reasons why it might be unavailable - 
either it's cpu bound, or it's blocking on something like a database 
call or i/o.  Different reasons for being busy might suggest different 
responses.

> - naviserver conn threads use a relatively large amount of memory
> because there tends to be one or more tcl interps associated with each
> one

Different requests could have different memory/resource requirements, 
which is a really nice thing about pools.  I'm hypothesizing that there 
are several different categories that requests fall into, based on the 
server resources needed to serve them and the time needed to complete them.

Server resources (memory) is either 'small' for requests that do not 
need a tcl interp (although tcl filters could tend to make this a 
nonexistent set), or 'big' for those that do.  Time is either slow or 
fast, by some arbitrary measure.

So a small/fast pool could be set up to serve static resources, a 
big/fast pool for non-database scripts, and a big/slow pool for database 
stuff.  I'm not sure what could be small/slow, maybe a c-coded proxy 
server or very large static files being delivered over slow connections.

The small/fast pool would only need a small number of threads with a 
high maxconnsperthread, while the large/slow pool might have many 
threads as most of those will be blocking on database access at any 
given time.

The important question in all of this is if a complex segmented setup 
like this works better in practice than a single large-enough pool of 
equal threads.  To which I don't have a good answer.

> - killing threads kills interps which frees memory
>
> But this is only useful if you can use the memory more profitably some
> where else, and I'm not sure you can.

Not only that, but memory isn't always released back to the system when 
free()d.  (vtamlloc is supposed to be able to, but I haven't had too 
much success with it so far.)  So freeing memory by shutting down 
threads won't necessarily make that available to your database.

However, memory that is not used by a process could be swapped out, 
making more physical ram available for other processes.  Having 20 
threads all used a little bit could keep them all in memory while having 
just 1 used a lot would keep a smaller working set.

This is a much bigger concern for low-resource systems.  Big systems 
nowadays have more physical memory than you can shake a stick at and 
swapping seems almost quaint.

> I think it might be better to drop min/max conn threads and just have
> n conn threads, always:

I've heard this recommendation before, in the context of tuning apache 
for high workloads - set maxservers=minservers=startservers.

I think it would make tuning easier for a lot of people if these was a 
basic "systemsize" parameter that is small/medium/large that set various 
other parameters to preset values.  As to what those values should be, 
that would take some thinking and experimentation.

> Thread pools are used throughout the server: multiple pools of conn
> threads, driver spool threads, scheduled proc threads, job threads,
> etc. so one clean way to tackle this might be to create a new
> nsd/pools.c which implements a very simple generic thread pool which
> has n threads, fifo ordering for requests, a tcl interface for
> dynamically setting the number of threads, and thread recycling after
> n requests. Then try to implement conn threads in terms of it.

I was thinking the exact same thing.

Sorry for the rambling/scattered thoughts, having a long commute does 
that :/

-J

Re: [naviserver-devel] lurking bugs: conn threads

[Andrew P. <at...@pi...>]

On Fri, Oct 26, 2012 at 08:30:26PM +0100, Stephen Deasey wrote:

> I was thinking it could work something like this:
> 
> - driver acquires lock, takes first conn thread off queue, releases lock

What if there are no conn threads waiting in the queue?

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] lurking bugs: conn threads

[Stephen D. <sd...@gm...>]

Interesting, but I wonder if we're not thinking this through
correctly. My suggestion, and your here, and Gustaf's recet work are
all aimed at refining the model as it currently is, but I wonder if
we're even attempting to do the right thing?

> So I'm assuming that the available processing power - the number of
> threads - should correlate to how busy the server is.  A server that is
> 50% busy should have 50% of its full capacity working.

But what is busy, CPU?  There needs to be an appropriate max number of
threads to handle the max expected load, considering the capabilities
of the machine. Too many and the machine will run slower. But why kill
them when we're no longer busy?

- naviserver conn threads use a relatively large amount of memory
because there tends to be one or more tcl interps associated with each
one

- killing threads kills interps which frees memory

But this is only useful if you can use the memory more profitably some
where else, and I'm not sure you can.

It is incoming load which drives conn thread creation, and therefore
memory usage, not availability of memory. So if you kill of some conn
threads when they're not needed, freeing up some memory for some other
system, how do you get the memory back when you create conn threads
again? There needs to be some higher mechanism which has a global view
of, say your database and web server requirements, and can balance the
memory needs between them.

I think it might be better to drop min/max conn threads and just have
n conn threads, always:

- simpler code

- predictable memory footprint

- bursty loads aren't delayed waiting for conn threads/interps to be created

- interps can be fully pre-warmed without delaying requests

- could back-port aolserver's ns_pools command to dynamically set the
nconnthreads setting

With ns_pools you could do something like use a scheduled proc to set
the nconnthreads down to 10 from 20 between 3-5am when your database
is taking a hefty dump.


Thread pools are used throughout the server: multiple pools of conn
threads, driver spool threads, scheduled proc threads, job threads,
etc. so one clean way to tackle this might be to create a new
nsd/pools.c which implements a very simple generic thread pool which
has n threads, fifo ordering for requests, a tcl interface for
dynamically setting the number of threads, and thread recycling after
n requests. Then try to implement conn threads in terms of it.


btw. an idea for pre-warming conn thread interps: generate a synthetic
request to /_ns/pool/foo/warmup (or whatever) when the thread is
created, before it is added to the queue. This would cause the tcl
source code to be byte compiled, and this could be controlled
precisely be registering a proc for that path.

Re: [naviserver-devel] code cleanup

[Maurizio M. <Mau...@sp...>]

Dear all,
	I do apologize for my inactivity but some health problem has
"distracted" me from the computer. Hopefully I should be able to come back
fully operative pretty soon.

First of all I would like to thank Gustav for his work.

Secondly, it seems to me there are two major activities going on at the
moment, I do not know how to call them:

1. call cleanup / consolidation  - I would call this "baseline"
2. discussion/work on new features    - I would call this "new developments"

Perhaps we should extend the baseline activity to all modules that could be
possibly needed by an OpenACS application, e.g. on top the standard ones
also modules like nsoracle, nsopenssl, nsldap and so on.... what do you
think? 

Wouldn't also make sense to work first on this consolidated and extended
baseline, before actually concentrating/working on new developments?

Thanks a lot,
Maurizio



-----Original Message-----
From: Gustaf Neumann [mailto:ne...@wu...] 
Sent: 26 October 2012 21:12
To: Navidevel
Subject: [naviserver-devel] code cleanup

Dear friends,

Over the last days i did some cleanup that i have just committed with one
changeset to bitbucket. With these changes, naviserver still compiles with
"clang -Wall -pedantic", and passes all regression tests. Furthermore, it
compiles but as well with Microsoft Visual Studio 11 without warning on most
files (still things to do, such as bind for udp sockets, pread emulation,
read-operations in the spooler, ...; just did nsd 
+ nsthread). Most likely the visual studio port needs more
still more work, i did not manage to link with whatever
tcl8.5.12 produces (it looks for a tcl84t.lib, which is not produced per
default from tcl, most likely, one needs more setup-magic from Maurizio). I
have used Maurizio's project files, and added essentially the flags
_CRT_SECURE_NO_WARNINGS and _CRT_NONSTDC_NO_DEPRECATE.

The huge number of casts from Maurizio's first attempt are not needed (e.g.
assigning void* to a typed pointer, such as the results of the *alloc*
functions. However, it complained about many of the size_t assignments,
which convinced me to proceed this way.

The changes are:

- use size_t, ssize_t, off_t wherever appropriate (sometimes dangerous, when
negative values are used for sizes, or  when e.g. Tcl_GetWideIntFromObj() is
used; i tried to be conservative in such cases)

- push casts to "int" as far as possible down the code (e.g. 
down to Tcl_* functions accepting sizes, but defined with "int", such as
Tcl_WriteChars, Tcl_NewStringObj,
Tcl_NewByteArrayObj)

- be more careful on redefining macros (MSC seems a moving
target)

- provide base definition for inttypes for MSC WIN32 and WIN64

- use NS_SOCKET instead of SOCKET (which is defined under Windows). We
should stick with macros in our own namespace whenever possible.

- added the project files and the solution file for visual studio (without
the SQL Server file containing the configuration state, which is 35MB)

As far i can tell, everything works fine under Linux and Mac OS X, but i
certainly can't exclude some sign mismatches on seldomly used paths.

-gustaf neumann


----------------------------------------------------------------------------
--
The Windows 8 Center
In partnership with Sourceforge
Your idea - your app - 30 days. Get started!
http://windows8center.sourceforge.net/
what-html-developers-need-to-know-about-coding-windows-8-metro-style-apps/
_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] lurking bugs: conn threads

[Stephen D. <sd...@gm...>]

On Thu, Oct 25, 2012 at 10:31 AM, Gustaf Neumann <ne...@wu...> wrote:
> I don't think, that a major problem comes from the "racy"
> notification of queuing  events to the connection threads.
> This has advantages (make os responsible, which does this
> very efficiently, less mutex requirements) and disadvantages
> (little control).


I think the current code works something like this:

- driver thread acquires lock, puts connection on queue, broadcasts to
conn threads, releases lock

- every conn thread waiting on the condition is woken up in some
arbitrary but approximately round-robin order, and maybe some of the
conn threads which aren't currently waiting pick up that message when
they do wait, because for performance these things aren't strictly
guaranteed (I may be remembering this wrong)

- n conn threads race to acquire the lock

- the one which gets the lock first take the conn from the queue and
release the lock

- it runs the connection, acquires the lock again, puts the conn back
on the queue, and release the lock

- meanwhile the other woken conn threads acquire then release the lock
with possibly nothing to do.


So for each request there can be up to 6 lock/unlock sequences by the
driver and active conn thread, plus a lock/unlock by n other conn
threads, all on one contended lock, plus the context switching
overhead, and this all happens in an undesirable order.


> By having a conn-thread-queue, the
> threads have to update this queue with their status
> information (being created, warming up, free, busy,
> will-die) which requires some overhead and more mutex locks
> on the driver.


I was thinking it could work something like this:


- driver acquires lock, takes first conn thread off queue, releases lock

- driver thread puts new socket in conn structure and the signals on
cond to that one thread (no locking, I don't think)

- that conn thread wakes up, takes no locks, runs connection

- conn thread acquires driver lock, puts conn back on front of queue,
releases lock


Four lock/unlock, lock/unlock sequences, two threads.

[naviserver-devel] code cleanup

[Gustaf N. <ne...@wu...>]

Dear friends,

Over the last days i did some cleanup that i have just 
committed with one changeset to bitbucket. With these 
changes, naviserver
still compiles with "clang -Wall -pedantic", and passes all 
regression tests. Furthermore, it compiles but as well with
Microsoft Visual Studio 11 without warning on most files 
(still things to do, such as bind for udp sockets, pread 
emulation, read-operations in the spooler, ...; just did nsd 
+ nsthread). Most likely the visual studio port needs more 
still more work, i did not manage to link with whatever 
tcl8.5.12 produces (it looks for a tcl84t.lib, which is not 
produced per default from tcl, most likely, one needs more 
setup-magic from Maurizio). I have used Maurizio's project 
files, and added essentially the flags 
_CRT_SECURE_NO_WARNINGS and _CRT_NONSTDC_NO_DEPRECATE.

The huge number of casts from Maurizio's first attempt are 
not needed (e.g. assigning void* to a typed pointer, such as 
the results of the *alloc* functions. However, it complained 
about many of the size_t assignments, which convinced me to 
proceed this way.

The changes are:

- use size_t, ssize_t, off_t wherever appropriate (sometimes 
dangerous, when negative values are used for sizes, or  when 
e.g. Tcl_GetWideIntFromObj() is used; i tried to be 
conservative in such cases)

- push casts to "int" as far as possible down the code (e.g. 
down to Tcl_* functions accepting sizes, but defined with 
"int", such as Tcl_WriteChars, Tcl_NewStringObj, 
Tcl_NewByteArrayObj)

- be more careful on redefining macros (MSC seems a moving 
target)

- provide base definition for inttypes for MSC WIN32 and WIN64

- use NS_SOCKET instead of SOCKET (which is defined under 
Windows). We should stick with macros in our own namespace 
whenever possible.

- added the project files and the solution file for visual 
studio (without the SQL Server file containing the 
configuration state, which is 35MB)

As far i can tell, everything works fine under Linux and Mac 
OS X, but i certainly can't exclude some sign mismatches on 
seldomly used paths.

-gustaf neumann

Re: [naviserver-devel] Hello

[Stephen D. <sd...@gm...>]

On Wed, Oct 24, 2012 at 5:01 PM, Agustin Lopez <Agu...@uv...> wrote:
>
> When I run the server I get error with Ns_ConfigSection from my compiled
> nsldap.
> I suppose that it is a known problem. Any pointer to work it?

What is the exact error message?

Re: [naviserver-devel] compression

[Stephen D. <sd...@gm...>]

On Thu, Oct 25, 2012 at 9:20 PM, Jeff Rogers <dv...@di...> wrote:
> It looks like we're enabling compression for all http/1.1 requests
> regardless of whether it was specified in the request header, or even
> specifically disallowed.  This seems incorrect, but the code has been in
> place for several years (connio.c:CheckCompress) ;  is there a reason
> not to change that?

I think the spec says that for HTTP/1.1, if the client doesn't
explicitly say to NOT send the body gzipped, say by using a q value of
0 (which we don't actually check for, woops...), then the server can
choose the encoding, although it should prefer the identity, unless
that's unavailable. So we're being very pushy...

There's a bunch of tests for this in tests/ns_adp_compress.test,
including with and without the Accept-Encoding header and q values,
but many of them are disabled with -constraints knownBug. I guess it's
something someone thought about but no one got round to fixing it.

This page describes how to run these particular tests:

  http://wiki.tcl.tk/21659

> Also on compression, a separate compression stream is pre-allocated and
> initialized for each pre-allocated Conn, whether or not compression is
> even enabled for the server.  The causes a fairly large initial memory
> footprint.  I think this pre-initialization could be made optional, and
> bypassed entirely when compression isn't enabled.  Any thoughts?

Seems reasonable.

[naviserver-devel] compression

[Jeff R. <dv...@di...>]

It looks like we're enabling compression for all http/1.1 requests 
regardless of whether it was specified in the request header, or even 
specifically disallowed.  This seems incorrect, but the code has been in 
place for several years (connio.c:CheckCompress) ;  is there a reason 
not to change that?

Also on compression, a separate compression stream is pre-allocated and 
initialized for each pre-allocated Conn, whether or not compression is 
even enabled for the server.  The causes a fairly large initial memory 
footprint.  I think this pre-initialization could be made optional, and 
bypassed entirely when compression isn't enabled.  Any thoughts?

-J

Re: [naviserver-devel] lurking bugs: conn threads

[Jeff R. <dv...@di...>]

I started working on some related ideas on a fork I created 
(naviserver-queues).  The main thing I'm trying to improve is how the 
state-managing code is spread out across several functions and different 
threads.

My approach is to have a separate monitor thread for each server that 
checks all the threads in each pool to see that there are enough threads 
running, that threads die when they have been around too long (services 
too many requests), and that threads die when no longer needed.  The 
driver thread still just queues the requests, but now there's an extra 
layer of control, and yes, overhead.

The question of how many threads are needed is an interesting one: is it 
better to create threads quickly in response to traffic, or to create 
them more slowly in case the traffic is very bursty?  The answer is of 
course, it depends.

So I'm assuming that the available processing power - the number of 
threads - should correlate to how busy the server is.  A server that is 
50% busy should have 50% of its full capacity working.  I'm using the 
wait queue length as a proxy for business:  if the wait queue is 50% 
full, then there should be 50% of maxthreads running.  But this seems 
like it unnecessarily underpowers the server, I added in an adjustable 
correction factor to scale this, so that you could tune for eager thread 
creation so you will be close to maxthreads when the server is 20% busy, 
or you could wait until the server is 80% busy before spawning more than 
a few threads.  On the idle end it works similarly;  if you tuned for 
eager thread creation then the threads wait longer to idle out.

I expect that this approach would lead to a sort of self-balancing: if 
the queue gets bigger then it starts getting serviced faster and stops 
growing, while if it shrinks then it gets serviced slower and stops 
shrinking.  There's room for experimentation here on exactly how to tune 
it; in my revision this logic is all in one place so it's simple.

My initial testing shows that the server handles the same throughput 
(total req/s about the same) and is a bit more equitable (smaller 
difference between slowest and fastest request) but slightly less 
responsive (which is expected, since the requests inherently spend 
longer in the wait queue.)

I'm still cleaning it up and it's definitely not ready for prime-time, 
but I'd be interested to hear what others think.

-J

Gustaf Neumann wrote:
> I don't think, that a major problem comes from the "racy"
> notification of queuing  events to the connection threads.
> This has advantages (make os responsible, which does this
> very efficiently, less mutex requirements) and disadvantages
> (little control).
>
> While the current architecture with the cond-broadcast is
> certainly responsible for the problem of simultaneous dieing
> threads (the OS determines, which thread receives sees the
> condition first, therefore round robin), a list of the
> linked connection threads does not help to determine on how
> many threads are actually needed, how bursty thread creation
> should be, how to handle short resource quenches (e.g.
> caused by locks, etc.). By having a conn-thread-queue, the
> threads have to update this queue with their status
> information (being created, warming up, free, busy,
> will-die) which requires some overhead and more mutex locks
> on the driver. The thread-status-handling is done currently
> automatically, a "busy" request ignores currently the
> condition, etc.
>
> On the good side, we would have more control over the
> threads. When a dieing thread notifies the
> conn-thread-queue, one can control thread-creation via this
> hook the same way as on situations, where requests are
> queued. Another good aspect is, that the thread-idle-timeout
> starts to makes sense again on busy sites. Currently, the
> thread-reduction works via counter, since unneeded threads
> die and won't be recreated unless the traffic requires it
> (which works in practice quite well). For busy sites, the
> thread-idle timeout is not needed this way.
>
> currently we have a one-way communication from the driver to
> the conn-threads. with the conn-thread-list (or array), one
> has a two way communication, ... at least, how i understand
> this for now.

>> I think this is racy because all conn threads block on a single
>> condition variable. The driver thread and conn threads must cooperate
>> to manage the whole life cycle and the code to manage the state is
>> spread around.
>>
>> If instead all conn thread were in a queue, each with it's own
>> condition variable, the driver thread could have sole responsibility
>> for choosing which conn thread to run by signalling it directly,
>> probably in LIFO order rather than the current semi-round-robin order
>> which tends to cause all conn threads to expire at once. Conn threads
>> would return to the front of the queue, unless wishing to expire in
>> which case they'd go on the back of the queue, and the driver would
>> signal when it was convenient to do so. Something like that...
>>

[naviserver-devel] Module nsldap

[Agustin L. <Agu...@uv...>]

Hi!

Anybody has one module for ldap acess?

Thanks,
Agustin

Re: [naviserver-devel] lurking bugs: conn threads

[Gustaf N. <ne...@wu...>]

I don't think, that a major problem comes from the "racy" 
notification of queuing  events to the connection threads. 
This has advantages (make os responsible, which does this 
very efficiently, less mutex requirements) and disadvantages 
(little control).

While the current architecture with the cond-broadcast is 
certainly responsible for the problem of simultaneous dieing 
threads (the OS determines, which thread receives sees the 
condition first, therefore round robin), a list of the 
linked connection threads does not help to determine on how 
many threads are actually needed, how bursty thread creation 
should be, how to handle short resource quenches (e.g. 
caused by locks, etc.). By having a conn-thread-queue, the 
threads have to update this queue with their status 
information (being created, warming up, free, busy, 
will-die) which requires some overhead and more mutex locks 
on the driver. The thread-status-handling is done currently 
automatically, a "busy" request ignores currently the 
condition, etc.

On the good side, we would have more control over the 
threads. When a dieing thread notifies the 
conn-thread-queue, one can control thread-creation via this 
hook the same way as on situations, where requests are 
queued. Another good aspect is, that the thread-idle-timeout 
starts to makes sense again on busy sites. Currently, the 
thread-reduction works via counter, since unneeded threads 
die and won't be recreated unless the traffic requires it 
(which works in practice quite well). For busy sites, the 
thread-idle timeout is not needed this way.

currently we have a one-way communication from the driver to 
the conn-threads. with the conn-thread-list (or array), one 
has a two way communication, ... at least, how i understand 
this for now.

-gustaf neumann

On 11.10.12 14:02, Stephen Deasey wrote:
> On Wed, Oct 10, 2012 at 9:44 PM, Jeff Rogers <dv...@di...> wrote:
>> It is possible to get into a situation where there are connections
>> queued but no conn threads running to handle them, meaning nothing
>> happens until a new connection comes in.  When this happens the server
>> will also not shut down cleanly.  As far as I can figure, this can only
>> happen if the connection queue is larger than connsperthread and the
>> load is very bursty (i.e., a load test);  all the existing conn threads
>> can hit their cpt and exit, but a new conn thread only starts when a new
>> connection is queued.  I think the solution here is to limit
>> maxconnections to no more than connsperthread.  Doing so exposes a less
>> severe problem where connections waiting in the driver thread don't get
>> queued for some time; it's less of a problem because there is a timeout
>> and the dirver thread will typically wake up on a closing socket fairly
>> soon, but it can still result in a simple request taking ~3s to
>> complete.  I don't know how to fix this latter problem.
> I think this is racy because all conn threads block on a single
> condition variable. The driver thread and conn threads must cooperate
> to manage the whole life cycle and the code to manage the state is
> spread around.
>
> If instead all conn thread were in a queue, each with it's own
> condition variable, the driver thread could have sole responsibility
> for choosing which conn thread to run by signalling it directly,
> probably in LIFO order rather than the current semi-round-robin order
> which tends to cause all conn threads to expire at once. Conn threads
> would return to the front of the queue, unless wishing to expire in
> which case they'd go on the back of the queue, and the driver would
> signal when it was convenient to do so. Something like that...
>
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] Hello

[Agustin L. <Agu...@uv...>]

Hello all !

This is my presentation to the list.
I am Agustin Lopez, system analyst of University of Valencia, Spain.

We are running (from seven years) a big OpenACS installation (dotLrn) 
with aolserver.
We are listen to Gustaf Neumann talk very well from naviserver :-)

Our aolserver works Ok, with some problems like big memory usage, ...
and periodical restarts.

I will like to test Naviserver. I am compiled it and it is running ok, but
we need some modules like nsldap (perhaps other).

When I run the server I get error with Ns_ConfigSection from my compiled 
nsldap.
I suppose that it is a known problem. Any pointer to work it?

Best regards,
Agustin

Re: [naviserver-devel] [AOLSERVER] Naviserver Win-64 Sources

[Gustaf N. <ne...@wu...>]

On 17.10.12 14:05, Gustaf Neumann wrote:
>>     #ifdef NS_NOCOMPAT
>> -#  error "No compatibility macros at present"
>> +// #  error "No compatibility macros at present"
>>     #endif
>>
>> This is my error, sorry, should be under _WIN64
>>
>> ...
> Is anybody aware about NS_NOCOMPAT ?
> This seem to stem from the win32 project files.
> Can this be removed?
since this is apparently of no use anymore (maurizio could 
compile
the code), i've removed this macro.

-gustaf neumann

Re: [naviserver-devel] var name changes

[Gustaf N. <ne...@wu...>]

On 17.10.12 13:21, Gustaf Neumann wrote:
> With this committed, there are still the typcast changes to
> address,
....

dear all, i have done one more cleanup round and removed all
warnings when naviserver is compiled with "clang -Wall 
-pedantic".
These changes were mostly mixtures between function pointers and
data pointers (it warns about passing a function pointer via 
"void*",
i have used the rather generic type Ns_Callback instead)
and arithmetics on "void*" (which is a gnu extension).
I have as well removed a bunch of unneeded assignments.

What are the opinions about changes from maurizios diff like

-    argPtr = ns_malloc(sizeof(Arg));
+    argPtr = (Arg *) ns_malloc(sizeof(Arg));

or

  NsTclServerObjCmd(ClientData arg, Tcl_Interp *interp, int 
objc, Tcl_Obj **objv)
  {
      int          opt;
-    NsInterp    *itPtr = arg;
+    NsInterp    *itPtr = (NsInterp *) arg;

do we really need these? changes in the first category could 
be done e.g. with a
macro...

-gustaf neumann

Re: [naviserver-devel] [AOLSERVER] Naviserver Win-64 Sources

[Maurizio M. <Mau...@sp...>]

No, sorry, wrong and quick answer. Ibrahim observation stays valid.

Apologies,
Maurizio


-----Original Message-----
From: Gustaf Neumann [mailto:ne...@wu...] 
Sent: 17 October 2012 15:03
To: nav...@li...
Subject: Re: [naviserver-devel] [AOLSERVER] Naviserver Win-64 Sources

On 17.10.12 14:49, Maurizio Martignano wrote:
> 1.Macros are defined.
> 2. What is not accepted is the starting with multiple format strings, e.g.
> ""%d %d %d %" <- 1st string PRId64 <- 2nd string " %" <--- and so on
can you rephrase this. What do you mean with "starting with multiple format
strings"?

With the macros defined, the string

"%d %d %d %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64

is equivalent with

"%d %d %d %" "I64d" " %" "I64d" " %" "I64d" " %" "I64d"

is equivalent with

"%d %d %d %I64d %I64d %I64d %I64d"


-gn
>
> -----Original Message-----
> From: Gustaf Neumann [mailto:ne...@wu...]
> Sent: 17 October 2012 14:06
> To: nav...@li...
> Subject: Re: [naviserver-devel] [AOLSERVER] Naviserver Win-64 Sources
>
> On 17.10.12 12:35, Maurizio Martignano wrote:
>> +#ifdef _WIN64
>> +        Ns_DStringPrintf(dsPtr, "%d %d %d %l64d %l64d %l64d
>> %l64d",
>> +#else
>>             Ns_DStringPrintf(dsPtr, "%d %d %d %" PRId64 " %"
>> PRId64 " %" PRId64 " %" PRId64,
>> +#endif
>>
>>
>> question arise: is this just _WIN64 or as well _WIN32?
>> Why not define PRId64 correctly for _WIN64? (and _WIN32?)
>>
>> [MM] Because the PRId64 thing works with gcc and not with Visual 
>> Studio (or to be correct I did not manage to make it work). I believe 
>> it works if you use MinGW, so this is why I used only _WIN64
> "PRId64" has nothing to do with gcc, it is supposed to be defined by 
> inttypes.h. From googling around it seems that these macros are often 
> a problem with visual c, not only a problem for _WIN64.
>
>   From your changes, i deduce, that PRId64 and PRIuMAX are not defined  
> for your platform, but PRIu64 seems to be there (at least, you did not 
> change this).  Please check, if the following change fixes the issues 
> with the PRI* macros under your platform
> https://bitbucket.org/naviserver/naviserver/changeset/92877ff5b9625aac
> 023c91
> 1dcd862894cfd441ce
>
>> Or, why have you commented out certain code (a) for win64 and (b) for 
>> all platforms which is certain useful:
>>
>>     #ifdef NS_NOCOMPAT
>> -#  error "No compatibility macros at present"
>> +// #  error "No compatibility macros at present"
>>     #endif
>>
>> This is my error, sorry, should be under _WIN64
>>
>> ...
> Is anybody aware about NS_NOCOMPAT ?
> This seem to stem from the win32 project files.
> Can this be removed?
>
>>     #ifdef HAVE_TCL_GETMEMORYINFO
>> -        Tcl_GetMemoryInfo(&ds);
>> -        Tcl_DStringResult(interp, &ds);
>> +        // Tcl_GetMemoryInfo(&ds);
>> +        // Tcl_DStringResult(interp, &ds);
>>     #endif
>>             return TCL_OK;
>>
>> same as above under _WIN64
> Why is this a problem with _WIN64?
>
> -gustaf neumann
>
> ----------------------------------------------------------------------
> ------
> --
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics Download AppDynamics Lite 
> for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>
> ----------------------------------------------------------------------
> -------- Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics Download AppDynamics Lite 
> for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel


----------------------------------------------------------------------------
--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics Download AppDynamics Lite for
free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] var name changes

[Maurizio M. <Mau...@sp...>]

Splendid!

I believe in the end all the codebase and community will benefit from this
activity.


Thank you,
Maurizio


-----Original Message-----
From: Gustaf Neumann [mailto:ne...@wu...] 
Sent: 17 October 2012 13:21
To: Navidevel
Subject: [naviserver-devel] var name changes

Dear all,

i have just commited a change addression all var name changes that maurizio
brought up.
The most common use-case of the variable "new" is of the form

    Tcl_CreateHashEntry(&table, value, &new);

The tcl source code uses new typically "isNew" instead of new, so we loose
nothing on readability etc. by making the same change. i would have objected
to change the variable name to e.g. "mm_new" just to silence a certain
compiler, but in terms of readability, "isNew" looks actually better.
Similarly, tcl avoids the variable name "bool",

    Tcl_GetBooleanFromObj(interp, objv[3], &boolValue)

etc. So, i have put together a single changeset to address these issues.
I hope, i found all cases. If anyone objects, feel free to undo this change.

With this committed, there are still the typcast changes to address, and to
look closer of the few other changes of maurizio (so far, i have not come
across the other K&R style files in the naviserver sources).

-gustaf neumann


----------------------------------------------------------------------------
--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics Download AppDynamics Lite for
free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] [AOLSERVER] Naviserver Win-64 Sources

[Gustaf N. <ne...@wu...>]

On 17.10.12 14:49, Maurizio Martignano wrote:
> 1.Macros are defined.
> 2. What is not accepted is the starting with multiple format strings, e.g.
> ""%d %d %d %" <- 1st string PRId64 <- 2nd string " %" <--- and so on
can you rephrase this. What do you mean with "starting with 
multiple format strings"?

With the macros defined, the string

"%d %d %d %" PRId64 " %" PRId64 " %" PRId64 " %" PRId64

is equivalent with

"%d %d %d %" "I64d" " %" "I64d" " %" "I64d" " %" "I64d"

is equivalent with

"%d %d %d %I64d %I64d %I64d %I64d"


-gn
>
> -----Original Message-----
> From: Gustaf Neumann [mailto:ne...@wu...]
> Sent: 17 October 2012 14:06
> To: nav...@li...
> Subject: Re: [naviserver-devel] [AOLSERVER] Naviserver Win-64 Sources
>
> On 17.10.12 12:35, Maurizio Martignano wrote:
>> +#ifdef _WIN64
>> +        Ns_DStringPrintf(dsPtr, "%d %d %d %l64d %l64d %l64d
>> %l64d",
>> +#else
>>             Ns_DStringPrintf(dsPtr, "%d %d %d %" PRId64 " %"
>> PRId64 " %" PRId64 " %" PRId64,
>> +#endif
>>
>>
>> question arise: is this just _WIN64 or as well _WIN32?
>> Why not define PRId64 correctly for _WIN64? (and _WIN32?)
>>
>> [MM] Because the PRId64 thing works with gcc and not with Visual
>> Studio (or to be correct I did not manage to make it work). I believe
>> it works if you use MinGW, so this is why I used only _WIN64
> "PRId64" has nothing to do with gcc, it is supposed to be defined by
> inttypes.h. From googling around it seems that these macros are often a
> problem with visual c, not only a problem for _WIN64.
>
>   From your changes, i deduce, that PRId64 and PRIuMAX are not defined  for
> your platform, but PRIu64 seems to be there (at least, you did not change
> this).  Please check, if the following change fixes the issues with the PRI*
> macros under your platform
> https://bitbucket.org/naviserver/naviserver/changeset/92877ff5b9625aac023c91
> 1dcd862894cfd441ce
>
>> Or, why have you commented out certain code (a) for win64 and (b) for
>> all platforms which is certain useful:
>>
>>     #ifdef NS_NOCOMPAT
>> -#  error "No compatibility macros at present"
>> +// #  error "No compatibility macros at present"
>>     #endif
>>
>> This is my error, sorry, should be under _WIN64
>>
>> ...
> Is anybody aware about NS_NOCOMPAT ?
> This seem to stem from the win32 project files.
> Can this be removed?
>
>>     #ifdef HAVE_TCL_GETMEMORYINFO
>> -        Tcl_GetMemoryInfo(&ds);
>> -        Tcl_DStringResult(interp, &ds);
>> +        // Tcl_GetMemoryInfo(&ds);
>> +        // Tcl_DStringResult(interp, &ds);
>>     #endif
>>             return TCL_OK;
>>
>> same as above under _WIN64
> Why is this a problem with _WIN64?
>
> -gustaf neumann
>
> ----------------------------------------------------------------------------
> --
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics Download AppDynamics Lite for
> free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] [AOLSERVER] Naviserver Win-64 Sources

[Gustaf N. <ne...@wu...>]

Is this maybe version dependent? to be on the safe side, i 
added PRIu64 in case it is not defined
-g
On 17.10.12 14:25, Ibrahim Tannir wrote:
> Hi Gustaf,
>
> Quickly grepped all include files. No PRI* defined anywhere.
> Maurizio must have missed this one.
>
> Cheers,
> Ibrahim
>
> On 17-Oct-12 14:05, Gustaf Neumann wrote:
>> On 17.10.12 12:35, Maurizio Martignano wrote:
>>    From your changes, i deduce, that PRId64 and PRIuMAX are
>> not defined  for your platform, but PRIu64 seems to be there
>> (at least, you did not change this).  Please check, if the
>> following
>> change fixes the issues with the PRI* macros under your platform
>> https://bitbucket.org/naviserver/naviserver/changeset/92877ff5b9625aac023c911dcd862894cfd441ce
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] [AOLSERVER] Naviserver Win-64 Sources

[Maurizio M. <Mau...@sp...>]

1.Macros are defined.
2. What is not accepted is the starting with multiple format strings, e.g.
""%d %d %d %" <- 1st string PRId64 <- 2nd string " %" <--- and so on




-----Original Message-----
From: Gustaf Neumann [mailto:ne...@wu...] 
Sent: 17 October 2012 14:06
To: nav...@li...
Subject: Re: [naviserver-devel] [AOLSERVER] Naviserver Win-64 Sources

On 17.10.12 12:35, Maurizio Martignano wrote:
>
> +#ifdef _WIN64
> +        Ns_DStringPrintf(dsPtr, "%d %d %d %l64d %l64d %l64d
> %l64d",
> +#else
>            Ns_DStringPrintf(dsPtr, "%d %d %d %" PRId64 " %"
> PRId64 " %" PRId64 " %" PRId64,
> +#endif
>
>
> question arise: is this just _WIN64 or as well _WIN32?
> Why not define PRId64 correctly for _WIN64? (and _WIN32?)
>
> [MM] Because the PRId64 thing works with gcc and not with Visual 
> Studio (or to be correct I did not manage to make it work). I believe 
> it works if you use MinGW, so this is why I used only _WIN64
"PRId64" has nothing to do with gcc, it is supposed to be defined by
inttypes.h. From googling around it seems that these macros are often a
problem with visual c, not only a problem for _WIN64.

 From your changes, i deduce, that PRId64 and PRIuMAX are not defined  for
your platform, but PRIu64 seems to be there (at least, you did not change
this).  Please check, if the following change fixes the issues with the PRI*
macros under your platform
https://bitbucket.org/naviserver/naviserver/changeset/92877ff5b9625aac023c91
1dcd862894cfd441ce

> Or, why have you commented out certain code (a) for win64 and (b) for 
> all platforms which is certain useful:
>
>    #ifdef NS_NOCOMPAT
> -#  error "No compatibility macros at present"
> +// #  error "No compatibility macros at present"
>    #endif
>
> This is my error, sorry, should be under _WIN64
>
> ...

Is anybody aware about NS_NOCOMPAT ?
This seem to stem from the win32 project files.
Can this be removed?

>
>    #ifdef HAVE_TCL_GETMEMORYINFO
> -        Tcl_GetMemoryInfo(&ds);
> -        Tcl_DStringResult(interp, &ds);
> +        // Tcl_GetMemoryInfo(&ds);
> +        // Tcl_DStringResult(interp, &ds);
>    #endif
>            return TCL_OK;
>
> same as above under _WIN64
Why is this a problem with _WIN64?

-gustaf neumann

----------------------------------------------------------------------------
--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics Download AppDynamics Lite for
free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] [AOLSERVER] Naviserver Win-64 Sources

[Ibrahim T. <it...@ar...>]

Hi Gustaf,

Quickly grepped all include files. No PRI* defined anywhere. 
Maurizio must have missed this one.

Cheers,
Ibrahim

On 17-Oct-12 14:05, Gustaf Neumann wrote:
> On 17.10.12 12:35, Maurizio Martignano wrote:
>   From your changes, i deduce, that PRId64 and PRIuMAX are
> not defined  for your platform, but PRIu64 seems to be there
> (at least, you did not change this).  Please check, if the
> following
> change fixes the issues with the PRI* macros under your platform
> https://bitbucket.org/naviserver/naviserver/changeset/92877ff5b9625aac023c911dcd862894cfd441ce

Re: [naviserver-devel] [AOLSERVER] Naviserver Win-64 Sources

[Gustaf N. <ne...@wu...>]

On 17.10.12 12:35, Maurizio Martignano wrote:
>
> +#ifdef _WIN64
> +        Ns_DStringPrintf(dsPtr, "%d %d %d %l64d %l64d %l64d
> %l64d",
> +#else
>            Ns_DStringPrintf(dsPtr, "%d %d %d %" PRId64 " %"
> PRId64 " %" PRId64 " %" PRId64,
> +#endif
>
>
> question arise: is this just _WIN64 or as well _WIN32?
> Why not define PRId64 correctly for _WIN64? (and _WIN32?)
>
> [MM] Because the PRId64 thing works with gcc and not with Visual Studio (or
> to be correct I did not manage to make it work). I believe it works if you
> use MinGW, so this is why I used only _WIN64
"PRId64" has nothing to do with gcc, it is supposed to be 
defined
by inttypes.h. From googling around it seems that these macros
are often a problem with visual c, not only a problem for 
_WIN64.

 From your changes, i deduce, that PRId64 and PRIuMAX are
not defined  for your platform, but PRIu64 seems to be there
(at least, you did not change this).  Please check, if the 
following
change fixes the issues with the PRI* macros under your platform
https://bitbucket.org/naviserver/naviserver/changeset/92877ff5b9625aac023c911dcd862894cfd441ce

> Or, why have you commented out certain code (a) for win64 and (b) for all
> platforms which is certain useful:
>
>    #ifdef NS_NOCOMPAT
> -#  error "No compatibility macros at present"
> +// #  error "No compatibility macros at present"
>    #endif
>
> This is my error, sorry, should be under _WIN64
>
> ...

Is anybody aware about NS_NOCOMPAT ?
This seem to stem from the win32 project files.
Can this be removed?

>
>    #ifdef HAVE_TCL_GETMEMORYINFO
> -        Tcl_GetMemoryInfo(&ds);
> -        Tcl_DStringResult(interp, &ds);
> +        // Tcl_GetMemoryInfo(&ds);
> +        // Tcl_DStringResult(interp, &ds);
>    #endif
>            return TCL_OK;
>
> same as above under _WIN64
Why is this a problem with _WIN64?

-gustaf neumann

[naviserver-devel] var name changes

[Gustaf N. <ne...@wu...>]

Dear all,

i have just commited a change addression all var name 
changes that maurizio brought up.
The most common use-case of the variable "new" is of the form

    Tcl_CreateHashEntry(&table, value, &new);

The tcl source code uses new typically "isNew" instead of 
new, so we loose nothing
on readability etc. by making the same change. i would have 
objected to change
the variable name to e.g. "mm_new" just to silence a certain 
compiler, but in terms
of readability, "isNew" looks actually better.
Similarly, tcl avoids the variable name "bool",

    Tcl_GetBooleanFromObj(interp, objv[3], &boolValue)

etc. So, i have put together a single changeset to address 
these issues.
I hope, i found all cases. If anyone objects, feel free to 
undo this change.

With this committed, there are still the typcast changes to 
address,
and to look closer of the few other changes of maurizio (so 
far, i have
not come across the other K&R style files in the naviserver 
sources).

-gustaf neumann

Re: [naviserver-devel] [AOLSERVER] Naviserver Win-64 Sources

[Stephen D. <sd...@gm...>]

On Wed, Oct 17, 2012 at 4:55 AM, Maurizio Martignano
<Mau...@sp...> wrote:
> OK
>
> Simple reason: Visual Studio complains about that stuff and it is annoying.

Looks like Visual Studio 2012 comes with a C compiler:

  http://msdn.microsoft.com/en-us/library/bb384838.aspx

"Visual Studio includes a C compiler that you can use to create
everything from basic C programs to Windows API applications."

"By default, the Visual C++ compiler treats all files that end in .c
as C source code,"

Maybe there's a bug in your build scripts that are forcing C code to
be treated as C++?

Re: [naviserver-devel] [AOLSERVER] Naviserver Win-64 Sources

[Maurizio M. <Mau...@sp...>]

Dear Gustav,
	Thank you for your mail message.

My answer here below.

Maurizio


-----Original Message-----
From: Gustaf Neumann [mailto:ne...@wu...] 
Sent: 17 October 2012 11:47
To: nav...@li...
Subject: Re: [naviserver-devel] [AOLSERVER] Naviserver Win-64 Sources

Maurizio,

the changes you have done are textually quite large, the patch is 6000 lines
+.
Please, don't expect, we can apply this and forget about it, we need some
time to digest this. Please, don't be impatient, we all have limited time.

For some problems, it seems for me to be better to follow the tcl
conventions, since it is not unlikely, that who ever works on the details of
naviserver will have to deal with the tcl sources as well.

What it makes this also more time consuming is that some of the changes
should be probably done differently, like the cases described below that i
have picked randomly. This list is most like not complete.

all the best
-gustaf neumann

+#ifdef _WIN64
+        Ns_DStringPrintf(dsPtr, "%d %d %d %l64d %l64d %l64d
%l64d",
+#else
          Ns_DStringPrintf(dsPtr, "%d %d %d %" PRId64 " %" 
PRId64 " %" PRId64 " %" PRId64,
+#endif


question arise: is this just _WIN64 or as well _WIN32?
Why not define PRId64 correctly for _WIN64? (and _WIN32?)

[MM] Because the PRId64 thing works with gcc and not with Visual Studio (or
to be correct I did not manage to make it work). I believe it works if you
use MinGW, so this is why I used only _WIN64


Or, why have you commented out certain code (a) for win64 and (b) for all
platforms which is certain useful:

  #ifdef NS_NOCOMPAT
-#  error "No compatibility macros at present"
+// #  error "No compatibility macros at present"
  #endif

This is my error, sorry, should be under _WIN64

...

  #ifdef HAVE_TCL_GETMEMORYINFO
-        Tcl_GetMemoryInfo(&ds);
-        Tcl_DStringResult(interp, &ds);
+        // Tcl_GetMemoryInfo(&ds);
+        // Tcl_DStringResult(interp, &ds);
  #endif
          return TCL_OK;

same as above under _WIN64

another problem is that there are some fixes included, which have nothing to
do with
_WIN64 and which are fixed already in the tip version on bitbucket.

--- /usr/local/src/naviserver-4.99.4/nsd/compress.c    
2010-03-07 17:08:38.000000000 +0100
+++ ./nsd/compress.c    2012-10-13 10:03:54.000000000 +0200
@@ -299,7 +299,11 @@
      return;
  }

+#ifdef _WIN64
+int
+#else
  char *
+#endif
  Ns_CompressBufsGzip(Ns_CompressStream *stream, struct iovec *bufs, int
nbufs, int flags,
                      Ns_DString *dsPtr)

[MM]
That looked to me as an error, but did not dare to correct it. Just correct
the fix and take away the _WIN64 stuff if you believe it is right

----------------------------------------------------------------------------
--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics Download AppDynamics Lite for
free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel