Re: [naviserver-devel] lurking bugs: conn threads

[Stephen D. <sd...@gm...>]

On Fri, Oct 26, 2012 at 11:41 PM, Jeff Rogers <dv...@di...> wrote:
> Stephen Deasey wrote:
>> .., but I wonder if
>> we're even attempting to do the right thing?
>
> Do we even know what the right thing is?  It could be any of
> - ...
> - minimize resource usage
> - adapt to dynamically changing workload
> - ...

Right, but you can't actually use the memory that temporarily running
fewer threads frees up because the threads may be restarted at any
moment, so the goal of adapting and minimizing is not being met.
Memory is being wasted, not recycled. (maybe, that's what I'm
suggesting...)

> Not only that, but memory isn't always released back to the system when
> free()d.  (vtamlloc is supposed to be able to, but I haven't had too
> much success with it so far.)  So freeing memory by shutting down
> threads won't necessarily make that available to your database.

I think glibc these days uses POSIX_MADV_DONTNEED on ranges within
mmap'd areas it uses for malloc. It doesn't reduce the address usage,
so this only shows up as lower RSS under top.


> Server resources (memory) is either 'small' for requests that do not
> need a tcl interp (although tcl filters could tend to make this a
> nonexistent set), or 'big' for those that do.  Time is either slow or
> fast, by some arbitrary measure.
>
> So a small/fast pool could be set up to serve static resources, a
> big/fast pool for non-database scripts, and a big/slow pool for database
> stuff.

Naviserver has some stuff which AOLserver doesn't to help with this
partitioning. Gustaf gave an example the other day where a server gets
a small burst of traffic, a couple of page requests, but the browser
simultaneously requests all the css and javascript etc., which causes
a bunch of new threads to be created and a stall as they all have
their interps allocated.

You can create a non-tcl pool as well as the default pool and then use
some tricks to force certain types of requests not to use Tcl.

- ACS registers a default handler for /*, but naviserver exposes
ns_register_fastpath with which you can re-register the pure C
fastpath handler for /*.css etc.

- ACS has an elaborate search path for files so the above is not
enough. But naviserver provides the url2file callback interface. A
pure C version of the algorithm which maps a url path to a file path
code be coded and then the fastpath code would correctly find package
specific static assets.

- There's auth filters and so on registered for /* but again they
aren't needed for /*.css. You can use ns_shortcut_filter to push a
null filter to the front of the filter queue which simply returns OK
and prevents the rest from running.


> The small/fast pool would only need a small number of threads with a
> high maxconnsperthread, while the large/slow pool might have many
> threads as most of those will be blocking on database access at any
> given time.

This is sort of a recreation of the memory situation I was suggesting
may bot be working. Balancing memory between naviserver and postgress
is like balancing the threads (memory) in two pools. The max number of
threads across all pools can't be so high that it overwhelms the
server. Within that constraint you have to balance the threads between
the pools. If you have two pools each with 10 threads, and one pool is
busy but the other is idle, then the server is not running to
capacity, but you may have to reject requests. If you increase the
threads in the busy pool, the other pool may also become busy and now
the server is overwhelmed.

Tcl-using conn threads are often so memory intensive it seems like it
would always be a win to have two conn thread pools for Tcl and
non-Tcl threads. To partition further there's ns_limits but that's not
hooked up and needs more work.