naviserver-devel — Developer discussion

Re: [naviserver-devel] Clock.tcl repeat initialisation after ns_eval

[Jeff R. <dv...@di...>]

Gustaf Neumann wrote:

> also the definition of proc "clock", which redefines itself in terms of
> an ensemble. This makes it sensitive to the definition order.


>> I notice the same test case doesn't raise an error in Aolserver. Seems
>> after a ns_eval the ::tcl::clock:: vars left not populated so
>> ::tcl::clock::Initialize runs cleanly. Couldn't ascertain from my
>> limited understanding of the internals why it should be different.
> aolserver does essentially the same thing as naviserver. my guess is
> that this is sheer luck in the order of definitions. Furthermore, a
> single call to lock "at the right time" can heal everything or lead to
> the error, since clock is kind of a "self modifying" code...

Sheer luck is definitely a possibility, but also naviserver is missing 
the ensemble serialization that aolserver has, so things defined (or 
redefined) as ensembles would get lost.

I've copied this code over from aolserver.  I don't know if it will help 
the clock problem, although it is possible clock was the motivating 
issue for that code in the first place.

In aolserver the code is wrapped to prevent it from executing on pre-8.5 
tcl.  Is that still needed, or is 8.4 dead enough to have 8.5 as a 
default requirement?

-J

Re: [naviserver-devel] Clock.tcl repeat initialisation after ns_eval

[Gustaf N. <ne...@wu...>]

Am 08.07.13 15:21, schrieb David Osborne:
> That certainly does the trick. Thanks.
>
> What's your opinion. Could this be considered a bug in Tcl?
> Would seem a lot cleaner if ::clock::Initialize just ran 
> ::tcl::clock::ClearCaches before initialising the TZData array which 
> would allow ::tcl::clock::Initalize to run multiple times.
The problem is not only the caches (otherwise it would have been 
sufficient to exclude the ::tcl::* namespaces from the blueprint), but 
also the definition of proc "clock", which redefines itself in terms of 
an ensemble. This makes it sensitive to the definition order.

Btw, my impression is that when ":localtime" is used instead of 
":Tcl/Localtime" ....

proc clock_test {} {
     ns_log Notice [clock format [clock scan now] -format "%Y-%m-%d %H:%M:%S" -timezone :localtime]
     return true
}

... then everything is fine even when we do not exclude "proc clock" 
from the blueprint. Can you check this?

Why are you using ":Tcl/Localtime" at the first place? Is this usage 
documented?

> I notice the same test case doesn't raise an error in Aolserver. Seems 
> after a ns_eval the ::tcl::clock:: vars left not populated so 
> ::tcl::clock::Initialize runs cleanly. Couldn't ascertain from my 
> limited understanding of the internals why it should be different.
aolserver does essentially the same thing as naviserver. my guess is 
that this is sheer luck in the order of definitions. Furthermore, a 
single call to lock "at the right time" can heal everything or lead to 
the error, since clock is kind of a "self modifying" code...

I still think, that leaving out the tcl namespaces "::tcl::* is the way 
to go; but i would prefer, if we don't have to muck around with ::clock.

-gustaf neumann

Re: [naviserver-devel] Clock.tcl repeat initialisation after ns_eval

[David O. <da...@qc...>]

That certainly does the trick. Thanks.

What's your opinion. Could this be considered a bug in Tcl?
Would seem a lot cleaner if ::clock::Initialize just ran
::tcl::clock::ClearCaches before initialising the TZData array which would
allow ::tcl::clock::Initalize to run multiple times.

I notice the same test case doesn't raise an error in Aolserver. Seems
after a ns_eval the ::tcl::clock:: vars left not populated so
::tcl::clock::Initialize runs cleanly. Couldn't ascertain from my limited
understanding of the internals why it should be different.

-- 
David Osborne

On 5 July 2013 19:19, Gustaf Neumann <ne...@wu...> wrote:

>  Dear David,
>
> Now we exclude proc ::clock from the blueprint and this should solve this
> issue.
> By doing so, we are loosing some flexibility (namely to overload ::clock
> with a tailored version), but i think we can live with this.
>
> Please try again
> -gustaf neumann
>
> PS: Not sure, why ::clock is defined so wierd by Tcl.
>

Re: [naviserver-devel] Clock.tcl repeat initialisation after ns_eval

[Gustaf N. <ne...@wu...>]

Dear David,

Now we exclude proc ::clock from the blueprint and this should solve 
this issue.
By doing so, we are loosing some flexibility (namely to overload ::clock
with a tailored version), but i think we can live with this.

Please try again
-gustaf neumann

PS: Not sure, why ::clock is defined so wierd by Tcl.

Am 05.07.13 16:04, schrieb David Osborne:
> Thanks again Gustaf,
>
> Have done what you suggested (a little downlevel):
>
> % info patchlevel
> 8.5.8
> % ns_info patchlevel
> 4.99.5
>
> And, yes, running on the main thread there is no problem
>
> % clock_test
> [05/Jul/2013:14:45:38][12827.7f1ea56ba700][-command-] Notice: 
> 2013-07-05 14:45:38
> true
> % ns_eval expr {1+1}
> 2
> % [05/Jul/2013:14:45:48][12827.7f1ea1a82700][-ns_job_0-] Notice: 
> Starting thread: -ns_job_0-
>
> % clock_test
> [05/Jul/2013:14:45:56][12827.7f1ea56ba700][-command-] Notice: 
> 2013-07-05 14:45:56
> true
>
>
> But if I try to run the proc on a different thread, the issue is present:
>
> % ns_schedule_proc -once 0 clock_test
> 1
> % [05/Jul/2013:14:46:23][12827.7f1ea2418700][-sched-] Notice: 
> 2013-07-05 14:46:23
>
> % ns_eval expr {1+1}
> 2
> % ns_schedule_proc -once 0 clock_test
> 2
> % [05/Jul/2013:14:46:36][12827.7f1ea2418700][-sched-] Error: time zone 
> ":Tcl/Localtime" not found
> time zone ":Tcl/Localtime" not found
>     while executing
> "return -options $opts $retval"
>     (procedure "::tcl::clock::format" line 18)
>     invoked from within
> "clock format [clock scan now] -format "%Y-%m-%d %H:%M:%S" -timezone 
> :Tcl/Localtime"
>     (procedure "clock_test" line 5)
>     invoked from within
> "clock_test"
>     while executing callback
> ns:tclschedproc clock_test
>
>
> -- 
> David Osborne
>
>
> On 5 July 2013 13:52, Gustaf Neumann <ne...@wu... 
> <mailto:ne...@wu...>> wrote:
>
>     Hi David,
>
>     strange, the indicated modification did make the change for me
>     (first broken as described in your email, and now fine).
>     Can it be, that your installation is  "manually" sourcing
>     some tcl init files (such as clock.tcl)?
>
>     Please try the following:
>     - use a default configuration of naviserver, e.g. in /usr/local/ns
>     - copy osborne-procs.tcl to
>     /usr/local/ns/modules/tcl/osborne-procs.tcl
>       where osborne-procs.tcl contains just the proc clock_test
>       from your last email.
>     - use the sample config as distributed with naviserver
>       /usr/local/ns/bin/nsd -c -u nsadmin -t
>     /usr/local/ns/conf/nsd-config.tcl
>
>
>     $ /usr/local/ns/bin/nsd -c -u nsadmin -t /usr/local/ns/conf/nsd-config.tcl
>     ...
>     %
>     % clock_test
>     [05/Jul/2013:14:35:11][75910.101873000][-command-] Notice: 2013-07-05 14:35:11
>     true
>     % ns_eval expr {2+1}
>     3
>     % [05/Jul/2013:14:35:34][75910.101924000][-ns_job_0-] Notice: Starting thread: -ns_job_0-
>     % clock_test
>     [05/Jul/2013:14:35:42][75910.101873000][-command-] Notice: 2013-07-05 14:35:42
>     true
>     %  ns_eval expr {2+1}
>     3
>     % clock_test
>     [05/Jul/2013:14:36:27][75910.101873000][-command-] Notice: 2013-07-05 14:36:27
>     true
>
>     % set tcl_patchLevel
>     8.5.13
>     % ns_info patchlevel
>     4.99.6
>
>     We can certainly define a blacklist to avoid including e.g. "proc
>     clock"
>     in the blueprint, but first i want to understand that this is
>     really necessary.
>
>     as you can see, i am using on my notebook slightly newer versions,
>     but i doubt this makes a difference
>
>     -gustaf
>
>     Am 05.07.13 12:51, schrieb David Osborne:
>>     Thanks Gustaf.
>>
>>     Makes sense. I can see how that change keeps the explicit calls
>>     to namespace eval ::tcl* out of the statescript.
>>
>>     But in this case it doesn't resolve the problem for us. The
>>     clock::Initalization scripts still ends up being called after an
>>     ns_eval, and ::tcl::clock::CachedSystemTimeZone stays set - hence
>>     containing an undefined timezone.
>>
>>     Looking at the statescript that is being saved during ns_eval,
>>     could it be something to do with the way the clock ensemble is
>>     being declared?
>>
>>     proc clock args {
>>         namespace eval ::tcl::clock [list namespace ensemble create
>>     -command [uplevel 1 [list namespace origin [lindex [info level 0]
>>     0]]] -subcommands {
>>                 add clicks format microseconds milliseconds scan seconds
>>             }]
>>     ...snip...
>>
>>
>>
>>     -- 
>>     David Osborne
>>
>>     On 4 July 2013 21:12, Gustaf Neumann <ne...@wu...
>>     <mailto:ne...@wu...>> wrote:
>>
>>         Hi David,
>>
>>         tricky thing: naviserver collects in its blueprint all
>>         defined namepaces.
>>         With Tcl 8.5, several built-in commands went from C to
>>         scripted Tcl,
>>         such as the implementation of clock. Tcl uses the ::tcl namespace
>>         for that. It seems, as if the blueprint of Tcl (including
>>         ::tcl::* variables
>>         and commands) messes up initialization of the ::clock command.
>>
>>         The easiest fix is not to include the stuff from the ::tcl
>>         namespace
>>         in the blueprint, since Tcl takes care about this during
>>         initialization.
>>
>>         The fix is quite simple:
>>         https://bitbucket.org/naviserver/naviserver/commits/a91fe6050ddba5ae5b42222f6c0dbc36acc004b2
>>         you should be able to simply add that line to your 4.99.5
>>         installation.
>>
>>         All the best
>>         -gustaf neumann
>>
>>

Re: [naviserver-devel] Clock.tcl repeat initialisation after ns_eval

[David O. <da...@qc...>]

Thanks again Gustaf,

Have done what you suggested (a little downlevel):

% info patchlevel
8.5.8
% ns_info patchlevel
4.99.5

And, yes, running on the main thread there is no problem

% clock_test
[05/Jul/2013:14:45:38][12827.7f1ea56ba700][-command-] Notice: 2013-07-05
14:45:38
true
% ns_eval expr {1+1}
2
% [05/Jul/2013:14:45:48][12827.7f1ea1a82700][-ns_job_0-] Notice: Starting
thread: -ns_job_0-

% clock_test
[05/Jul/2013:14:45:56][12827.7f1ea56ba700][-command-] Notice: 2013-07-05
14:45:56
true


But if I try to run the proc on a different thread, the issue is present:

% ns_schedule_proc -once 0 clock_test
1
% [05/Jul/2013:14:46:23][12827.7f1ea2418700][-sched-] Notice: 2013-07-05
14:46:23

% ns_eval expr {1+1}
2
% ns_schedule_proc -once 0 clock_test
2
% [05/Jul/2013:14:46:36][12827.7f1ea2418700][-sched-] Error: time zone
":Tcl/Localtime" not found
time zone ":Tcl/Localtime" not found
    while executing
"return -options $opts $retval"
    (procedure "::tcl::clock::format" line 18)
    invoked from within
"clock format [clock scan now] -format "%Y-%m-%d %H:%M:%S" -timezone
:Tcl/Localtime"
    (procedure "clock_test" line 5)
    invoked from within
"clock_test"
    while executing callback
ns:tclschedproc clock_test


-- 
David Osborne


On 5 July 2013 13:52, Gustaf Neumann <ne...@wu...> wrote:

>  Hi David,
>
> strange, the indicated modification did make the change for me
> (first broken as described in your email, and now fine).
> Can it be, that your installation is  "manually" sourcing
> some tcl init files (such as clock.tcl)?
>
> Please try the following:
> - use a default configuration of naviserver, e.g. in /usr/local/ns
> - copy osborne-procs.tcl to /usr/local/ns/modules/tcl/osborne-procs.tcl
>   where osborne-procs.tcl contains just the proc clock_test
>   from your last email.
> - use the sample config as distributed with naviserver
>   /usr/local/ns/bin/nsd -c -u nsadmin -t /usr/local/ns/conf/nsd-config.tcl
>
>
> $ /usr/local/ns/bin/nsd -c -u nsadmin -t /usr/local/ns/conf/nsd-config.tcl
> ...
> %
> % clock_test
> [05/Jul/2013:14:35:11][75910.101873000][-command-] Notice: 2013-07-05 14:35:11
> true
> % ns_eval expr {2+1}
> 3
> % [05/Jul/2013:14:35:34][75910.101924000][-ns_job_0-] Notice: Starting thread: -ns_job_0-
> % clock_test
> [05/Jul/2013:14:35:42][75910.101873000][-command-] Notice: 2013-07-05 14:35:42
> true
> %  ns_eval expr {2+1}
> 3
> % clock_test
> [05/Jul/2013:14:36:27][75910.101873000][-command-] Notice: 2013-07-05 14:36:27
> true
>
> % set tcl_patchLevel
> 8.5.13
> % ns_info patchlevel
> 4.99.6
>
>
> We can certainly define a blacklist to avoid including e.g. "proc clock"
> in the blueprint, but first i want to understand that this is really
> necessary.
>
> as you can see, i am using on my notebook slightly newer versions,
> but i doubt this makes a difference
>
> -gustaf
>
> Am 05.07.13 12:51, schrieb David Osborne:
>
>   Thanks Gustaf.
>
>  Makes sense. I can see how that change keeps the explicit calls to namespace
> eval ::tcl* out of the statescript.
>
> But in this case it doesn't resolve the problem for us. The
> clock::Initalization scripts still ends up being called after an ns_eval,
> and ::tcl::clock::CachedSystemTimeZone stays set - hence containing an
> undefined timezone.
>
>  Looking at the statescript that is being saved during ns_eval, could it
> be something to do with the way the clock ensemble is being declared?
>
> proc clock args {
>     namespace eval ::tcl::clock [list namespace ensemble create -command
> [uplevel 1 [list namespace origin [lindex [info level 0] 0]]]  -subcommands
> {
>             add clicks format microseconds milliseconds scan seconds
>         }]
>  ...snip...
>
>
>
> --
> David Osborne
>
> On 4 July 2013 21:12, Gustaf Neumann <ne...@wu...> wrote:
>
>>  Hi David,
>>
>> tricky thing: naviserver collects in its blueprint all defined namepaces.
>> With Tcl 8.5, several built-in commands went from C to scripted Tcl,
>> such as the implementation of clock. Tcl uses the ::tcl namespace
>> for that. It seems, as if the blueprint of Tcl (including ::tcl::*
>> variables
>> and commands) messes up initialization of the ::clock command.
>>
>> The easiest fix is not to include the stuff from the ::tcl namespace
>> in the blueprint, since Tcl takes care about this during initialization.
>>
>> The fix is quite simple:
>>
>> https://bitbucket.org/naviserver/naviserver/commits/a91fe6050ddba5ae5b42222f6c0dbc36acc004b2
>> you should be able to simply add that line to your 4.99.5 installation.
>>
>> All the best
>> -gustaf neumann
>>
>
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>

Re: [naviserver-devel] Clock.tcl repeat initialisation after ns_eval

[Gustaf N. <ne...@wu...>]

Hi David,

strange, the indicated modification did make the change for me
(first broken as described in your email, and now fine).
Can it be, that your installation is  "manually" sourcing
some tcl init files (such as clock.tcl)?

Please try the following:
- use a default configuration of naviserver, e.g. in /usr/local/ns
- copy osborne-procs.tcl to /usr/local/ns/modules/tcl/osborne-procs.tcl
   where osborne-procs.tcl contains just the proc clock_test
   from your last email.
- use the sample config as distributed with naviserver
   /usr/local/ns/bin/nsd -c -u nsadmin -t /usr/local/ns/conf/nsd-config.tcl


$ /usr/local/ns/bin/nsd -c -u nsadmin -t /usr/local/ns/conf/nsd-config.tcl
...
%
% clock_test
[05/Jul/2013:14:35:11][75910.101873000][-command-] Notice: 2013-07-05 14:35:11
true
% ns_eval expr {2+1}
3
% [05/Jul/2013:14:35:34][75910.101924000][-ns_job_0-] Notice: Starting thread: -ns_job_0-
% clock_test
[05/Jul/2013:14:35:42][75910.101873000][-command-] Notice: 2013-07-05 14:35:42
true
%  ns_eval expr {2+1}
3
% clock_test
[05/Jul/2013:14:36:27][75910.101873000][-command-] Notice: 2013-07-05 14:36:27
true

% set tcl_patchLevel
8.5.13
% ns_info patchlevel
4.99.6

We can certainly define a blacklist to avoid including e.g. "proc clock"
in the blueprint, but first i want to understand that this is really 
necessary.

as you can see, i am using on my notebook slightly newer versions,
but i doubt this makes a difference

-gustaf

Am 05.07.13 12:51, schrieb David Osborne:
> Thanks Gustaf.
>
> Makes sense. I can see how that change keeps the explicit calls to 
> namespace eval ::tcl* out of the statescript.
>
> But in this case it doesn't resolve the problem for us. The 
> clock::Initalization scripts still ends up being called after an 
> ns_eval, and ::tcl::clock::CachedSystemTimeZone stays set - hence 
> containing an undefined timezone.
>
> Looking at the statescript that is being saved during ns_eval, could 
> it be something to do with the way the clock ensemble is being declared?
>
> proc clock args {
>     namespace eval ::tcl::clock [list namespace ensemble create 
> -command  [uplevel 1 [list namespace origin [lindex [info level 0] 
> 0]]]  -subcommands {
>             add clicks format microseconds milliseconds scan seconds
>         }]
> ...snip...
>
>
>
> -- 
> David Osborne
>
> On 4 July 2013 21:12, Gustaf Neumann <ne...@wu... 
> <mailto:ne...@wu...>> wrote:
>
>     Hi David,
>
>     tricky thing: naviserver collects in its blueprint all defined
>     namepaces.
>     With Tcl 8.5, several built-in commands went from C to scripted Tcl,
>     such as the implementation of clock. Tcl uses the ::tcl namespace
>     for that. It seems, as if the blueprint of Tcl (including ::tcl::*
>     variables
>     and commands) messes up initialization of the ::clock command.
>
>     The easiest fix is not to include the stuff from the ::tcl namespace
>     in the blueprint, since Tcl takes care about this during
>     initialization.
>
>     The fix is quite simple:
>     https://bitbucket.org/naviserver/naviserver/commits/a91fe6050ddba5ae5b42222f6c0dbc36acc004b2
>     you should be able to simply add that line to your 4.99.5
>     installation.
>
>     All the best
>     -gustaf neumann
>
>

Re: [naviserver-devel] Clock.tcl repeat initialisation after ns_eval

[David O. <da...@qc...>]

Thanks Gustaf.

Makes sense. I can see how that change keeps the explicit calls to namespace
eval ::tcl* out of the statescript.

But in this case it doesn't resolve the problem for us. The
clock::Initalization scripts still ends up being called after an ns_eval,
and ::tcl::clock::CachedSystemTimeZone stays set - hence containing an
undefined timezone.

Looking at the statescript that is being saved during ns_eval, could it be
something to do with the way the clock ensemble is being declared?

proc clock args {
    namespace eval ::tcl::clock [list namespace ensemble create -command
[uplevel 1 [list namespace origin [lindex [info level 0] 0]]]  -subcommands
{
            add clicks format microseconds milliseconds scan seconds
        }]
...snip...



-- 
David Osborne

On 4 July 2013 21:12, Gustaf Neumann <ne...@wu...> wrote:

>  Hi David,
>
> tricky thing: naviserver collects in its blueprint all defined namepaces.
> With Tcl 8.5, several built-in commands went from C to scripted Tcl,
> such as the implementation of clock. Tcl uses the ::tcl namespace
> for that. It seems, as if the blueprint of Tcl (including ::tcl::*
> variables
> and commands) messes up initialization of the ::clock command.
>
> The easiest fix is not to include the stuff from the ::tcl namespace
> in the blueprint, since Tcl takes care about this during initialization.
>
> The fix is quite simple:
>
> https://bitbucket.org/naviserver/naviserver/commits/a91fe6050ddba5ae5b42222f6c0dbc36acc004b2
> you should be able to simply add that line to your 4.99.5 installation.
>
> All the best
> -gustaf neumann
>

Re: [naviserver-devel] Clock.tcl repeat initialisation after ns_eval

[Gustaf N. <ne...@wu...>]

Hi David,

tricky thing: naviserver collects in its blueprint all defined namepaces.
With Tcl 8.5, several built-in commands went from C to scripted Tcl,
such as the implementation of clock. Tcl uses the ::tcl namespace
for that. It seems, as if the blueprint of Tcl (including ::tcl::* variables
and commands) messes up initialization of the ::clock command.

The easiest fix is not to include the stuff from the ::tcl namespace
in the blueprint, since Tcl takes care about this during initialization.

The fix is quite simple:
https://bitbucket.org/naviserver/naviserver/commits/a91fe6050ddba5ae5b42222f6c0dbc36acc004b2
you should be able to simply add that line to your 4.99.5 installation.

All the best
-gustaf neumann

Am 04.07.13 18:06, schrieb David Osborne:
> Hi,
>
> Wondering if you can help us understand what's happening here. 
> (Currently testing under naviserver 4.99.5 with Tcl 8.5.11)
>
> Our test case is the following proc:
>
> proc clock_test {} {
>     log Notice [clock format [clock scan now] -format "%Y-%m-%d 
> %H:%M:%S" -timezone :Tcl/Localtime]
>     return true
> }
>
> If we open a naviserver control port and run the command it works:
>
> > ns_schedule_proc -once  0 clock_test
> [04/Jul/2013:16:17:49][2278.7f08c0bf5700][-sched-] Notice: 2013-07-04 
> 16:37:49
>
> However, if we now run any ns_eval and repeat, the clock command will 
> fail:
>
> > ns_eval expr {2+1}
> 3
> > ns_schedule_proc -once  0 clock_test
> [04/Jul/2013:16:25:17][2278.7f08c0b73700][-sched-] Error: time zone 
> ":Tcl/Localtime" not found
>
> We can see why this is failing. It's because, after the ns_eval, 
> namespace eval clock is called from Tcl's init script. This means the 
> next call to the clock ensemble will run clock::Initialize again. This 
> resets the TZData array (removing :Tcl:Localtime) but :Tcl/Localtime 
> is still cached in the ::tcl::clock::CachedSystemTimeZone variable... 
> hence the error.
>
> If we now run the proc in its own thread, it will work again until 
> another ns_eval is performed.
>
> > ns_schedule_proc -once -thread 0 clock_test
> [04/Jul/2013:16:46:06][18122.7f2c93f53700][-sched:7-] Notice: 
> 2013-07-04 16:46:06
> > ns_eval expr {2+2}
> > ns_schedule_proc -once -thread 0 clock_test
> [04/Jul/2013:16:46:15][18122.7f2c93f53700][-sched:8-] Error: time zone 
> ":Tcl/Localtime" not found
>
> It's a bit of a contrived test case but believe it or not, we are 
> encountering basically the same situation in production.
>
> If ::tcl::clock::ClearCaches were to be called before the second 
> ::tcl::clock::Initialize I think everything would be fine. But would 
> be interested to hear opinions on why this is manifesting itself in 
> these situations.
>
> Regards,
> -- 
> David

[naviserver-devel] Clock.tcl repeat initialisation after ns_eval

[David O. <da...@qc...>]

Hi,

Wondering if you can help us understand what's happening here. (Currently
testing under naviserver 4.99.5 with Tcl 8.5.11)

Our test case is the following proc:

proc clock_test {} {
    log Notice [clock format [clock scan now] -format "%Y-%m-%d %H:%M:%S"
-timezone :Tcl/Localtime]
    return true
}

If we open a naviserver control port and run the command it works:

> ns_schedule_proc -once  0 clock_test
[04/Jul/2013:16:17:49][2278.7f08c0bf5700][-sched-] Notice: 2013-07-04
16:37:49

However, if we now run any ns_eval and repeat, the clock command will fail:

> ns_eval expr {2+1}
3
> ns_schedule_proc -once  0 clock_test
[04/Jul/2013:16:25:17][2278.7f08c0b73700][-sched-] Error: time zone
":Tcl/Localtime" not found

We can see why this is failing. It's because, after the ns_eval, namespace
eval clock is called from Tcl's init script. This means the next call to
the clock ensemble will run clock::Initialize again. This resets the TZData
array (removing :Tcl:Localtime) but :Tcl/Localtime is still cached in the
::tcl::clock::CachedSystemTimeZone variable... hence the error.

If we now run the proc in its own thread, it will work again until another
ns_eval is performed.

> ns_schedule_proc -once -thread 0 clock_test
[04/Jul/2013:16:46:06][18122.7f2c93f53700][-sched:7-] Notice: 2013-07-04
16:46:06
> ns_eval expr {2+2}
> ns_schedule_proc -once -thread 0 clock_test
[04/Jul/2013:16:46:15][18122.7f2c93f53700][-sched:8-] Error: time zone
":Tcl/Localtime" not found

It's a bit of a contrived test case but believe it or not, we are
encountering basically the same situation in production.

If ::tcl::clock::ClearCaches were to be called before the second
::tcl::clock::Initialize
I think everything would be fine. But would be interested to hear opinions
on why this is manifesting itself in these situations.

Regards,
-- 
David

Re: [naviserver-devel] gzipped content delivery for static files

[Gustaf N. <ne...@wu...>]

Hi Jeff,

many thanks for the feedback.

The logic is as follows:
- when gzip_static is activated, then fastpath checks for the
   existence of a .gz file and uses it if not outdated.
- if gzip_static is activated and there exists a gzip file
   and it is outdated, then the gzipcmd comes into play
   to update it, but only, when it is configured
   (by default it is not configured).

So an attack vector is more complex than you sketched,
but i guess possible, when gzipcmd was activated.

In general, you are right: using a tcl proc is better
since it makes it easy to change the behavior without
touching C. I have added a boolean parameter "gzip_refresh"
to make the intention of the site-admin for refreshing
gzip files on the fly clear. A new tcl command "ns_gzipfile"
is used now for gzipping (this command requires tcl 8.5 to work)

https://bitbucket.org/naviserver/naviserver/commits/f7dca733625553ab802c93d35d471524e5a13e3e

all the best
-gustaf neumann

Am 04.07.13 04:38, schrieb Jeff Rogers:
> Gustaf Neumann wrote:
>
>> When the gzip_cmd is configured, NaviServer keeps track of
>> updating the .gz file for the cases the source file is updated.
>> There is no burden for the admin. The gzip command is
>> called via Tcl "exec", which can in turn be executed via
>> nsproxy (see e.g. OpenACS). I took this approach
>> over an in-memory variant to avoid memory bloats in
>> case huge gzip files should be compressed. Note that
>> the compression overhead is typically just a one-time
>> operation. The website maintained defines, what
>> files should be sent gzipped by compressing these
>> once.
> I think there's a potential security hole here.  I didn't come up with a
> proper exploit, but if a user can get control of a filename (e.g., if
> there is an ability to upload files), then an arbitrary string could get
> passed to the exec command, including but not limited to [] (which would
> let tcl do commmand expansion) or spaces (which could cause the filename
> to be interpreted as multiple words and hijack the exec behavior).
>
> Using Tcl_DStringAppendElement instead of Tcl_DStringAppend should
> prevent this, as it will force the filename to be a proper list element.
>
> Alternately, it would be more flexible to change the definition of the
> zipCmd to be a tcl command that is passed the filename and zipfile name,
> e.g.,  "gzip_cmd file file.gz", with the tcl definition of gzip_cmd
> choosing how to handle it, whether by exec or compressing in-process
> (e.g., with 'zlib compress'), or choosing based on the file size.
>
> -J
>

Re: [naviserver-devel] gzipped content delivery for static files

[Jeff R. <dv...@di...>]

Gustaf Neumann wrote:

> When the gzip_cmd is configured, NaviServer keeps track of
> updating the .gz file for the cases the source file is updated.
> There is no burden for the admin. The gzip command is
> called via Tcl "exec", which can in turn be executed via
> nsproxy (see e.g. OpenACS). I took this approach
> over an in-memory variant to avoid memory bloats in
> case huge gzip files should be compressed. Note that
> the compression overhead is typically just a one-time
> operation. The website maintained defines, what
> files should be sent gzipped by compressing these
> once.

I think there's a potential security hole here.  I didn't come up with a 
proper exploit, but if a user can get control of a filename (e.g., if 
there is an ability to upload files), then an arbitrary string could get 
passed to the exec command, including but not limited to [] (which would 
let tcl do commmand expansion) or spaces (which could cause the filename 
to be interpreted as multiple words and hijack the exec behavior).

Using Tcl_DStringAppendElement instead of Tcl_DStringAppend should 
prevent this, as it will force the filename to be a proper list element.

Alternately, it would be more flexible to change the definition of the 
zipCmd to be a tcl command that is passed the filename and zipfile name, 
e.g.,  "gzip_cmd file file.gz", with the tcl definition of gzip_cmd 
choosing how to handle it, whether by exec or compressing in-process 
(e.g., with 'zlib compress'), or choosing based on the file size.

-J

[naviserver-devel] gzipped content delivery for static files

[Gustaf N. <ne...@wu...>]

Am 04.12.12 23:55, schrieb Gustaf Neumann:
> Am 04.12.12 20:06, schrieb Stephen Deasey:
>> - we should actually ship some code which searches for *.gz versions 
>> of static files
> this would mean to keep a .gz version and a non-.gz version in the 
> file system for the cases, where gzip is not an accepted encoding. Not 
> sure, i would like to manage these files and to keep it in sync.... 
> the fast-path cache could keep gzipped copies, invalidation is already 
> there.

Dear all,

The updated version of naviserver supports now the
delivery of gzipped content via fastpath:

- added option "gzip_static" for "ns/fastpath" (default false)

Send the gzipped version of the file if available and the client
accepts gzipped content. When a file path/foo.ext is requested, and
there exists a file path/foo.ext.gz, and the timestamp of
the gzipped file is equal or newer than the source file, use the
gzipped file for delivery.

- added option "gzip_cmd" for "ns/fastpath" (default "")

Command for zipping files in case the (static) gzipped version of
the file is older than the source. The command is just used for
re-gzipping outdated files, it does not actively compress files,
which were previously not compressed (this would be wasteful for
e.g. large tmp files, there is not cleanup, etc.). If this parameter
is not defined, outdated gzipped files are ignored, and a warning is
written to the error.log. Example setting: "/usr/bin/gzip -9".

- added section for fastpath configuration to ns_return describing the
new and old configuration options (description of fastpath
parameters was completely missing)

When the gzip_cmd is configured, NaviServer keeps track of
updating the .gz file for the cases the source file is updated.
There is no burden for the admin. The gzip command is
called via Tcl "exec", which can in turn be executed via
nsproxy (see e.g. OpenACS). I took this approach
over an in-memory variant to avoid memory bloats in
case huge gzip files should be compressed. Note that
the compression overhead is typically just a one-time
operation. The website maintained defines, what
files should be sent gzipped by compressing these
once.

Gzipped content delivery happens now for
- ns_returnfile
- ns_respond -file
- static files from pagedir
when the corresponding .gz files are available.

all the best
-gustaf neumann

Re: [naviserver-devel] improved ns_http and ns_ssl

[Gustaf N. <ne...@wu...>]

Dear all,
my sample code had a small mistake

ns_register_filter preauth POST * proxy::upstream -target https://localhost:8443 {-timeout 5:0}

should of course read as

ns_register_filter preauth POST * proxy::upstream -target https://localhost:8443 -timeout 5:0

Sorry for the confusion
-gn

Am 18.06.13 12:28, schrieb Gustaf Neumann:
> ...
> tcl/proxy.tcl
> ===================================================================================
> # Tiny Reverse Proxy
> #
> # - deliver from upstream http and https server (spooling of large content)
> # - delivery of local content (optionally compressed)
> # - obtain statistics via nsstats from the proxy server
> #
> #                                               -Gustaf Neumann (June 2013)
> #
> #
> # To get feedback from the proxy, we want to execute e.g. nsstats.tcl,
> # which is provided in the proxy-admin directory
> #
> ns_register_tcl GET /proxy-admin/*.tcl
>
> #
> # Deliver images from the local files
> #
> ns_register_filter preauth GET  /img/* proxy::local
>
> ns_register_filter preauth GET  * proxy::upstream -target https://localhost:8443
> ns_register_filter preauth POST * proxy::upstream -target https://localhost:8443 {-timeout 5:0}
>
> ...

[naviserver-devel] improved ns_http and ns_ssl

[Gustaf N. <ne...@wu...>]

Dear friends of NaviServer,

The updates version of NaviServer contains now improved
support for handling large files. One can now specify to
"ns_http" the parameters "-spoolize" and "-file". When the
retrieved content is larger than spoolsize, then it is
spooled to a temp file and the file name is reported back
to the variable denoted by file (same pattern is used for
other return values as well). This addition makes it
practical to retrieve substantially large files with ns_http
(and ns_ssl) as well without bloating the memory. The
retrieved files can be conveniently be further delivered
via the writer threads. I've made the same changes to
ns_ssl and took the opportunity to made ns_ssl working
with asynchronous I/O, to refactor the code, etc.

With these changes, one can use NaviServer with
reverse-proxy functionality for http and https backends.
The sample script below supports

  - delivery from upstream http and https server
    (spooling of large content)
  - delivery of local content (optionally compressed)
  - obtain statistics via nsstats from the proxy server

The current implementation supports upstream only
HTTP/1.0 (due to the different socket management in Ns_Task)
and has no support for delivering upstream packages
directly to the client. Since upstreams connections are
quite fast and due to spooling this is many applications already
useful. On my local machine, it uses with 50 threads
just 150 MB after startup (due to the tiny blue-print).

The sample script proxy.tcl uses "nsf::proc"
(from http://next-scripting.org) to allow for nice
configuration per filter rule. If there is no nsf install,
one can replace it by a simple "proc" and do the
configuration differently...

Best regards
-gustaf neumann

PS: the script requires NaviServer 4.99.6

/usr/local/ns/bin/nsd -f -t  /usr/local/proxy-https/etc/proxy-conf-https.tcl

/usr/local/proxy-https/etc/proxy-conf-https.tcl
/usr/local/proxy-https/pages/img/DotLrnLogo.gif
/usr/local/proxy-https/pages/proxy-admin/nsstats.tcl
/usr/local/proxy-https/tcl/proxy.tcl


tcl/proxy.tcl
===================================================================================
# Tiny Reverse Proxy
#
# - deliver from upstream http and https server (spooling of large content)
# - delivery of local content (optionally compressed)
# - obtain statistics via nsstats from the proxy server
#
#                                               -Gustaf Neumann (June 2013)
#
#
# To get feedback from the proxy, we want to execute e.g. nsstats.tcl,
# which is provided in the proxy-admin directory
#
ns_register_tcl GET /proxy-admin/*.tcl

#
# Deliver images from the local files
#
ns_register_filter preauth GET  /img/* proxy::local

ns_register_filter preauth GET  * proxy::upstream -target https://localhost:8443
ns_register_filter preauth POST * proxy::upstream -target https://localhost:8443 {-timeout 5:0}

namespace eval ::proxy {

     #
     # Local handler (deliver from disk)
     #
     # Serve the requested content from the server page directory. If
     # the client accepts compressed content, the file is served from a
     # compressed file. If the compressed file does not exist, or is
     # stale, the content is compressed on the fly.
     #
     nsf::proc local { what } {

	set fn [ns_server pagedir]/[ns_conn url]
	if {[file readable $fn]} {
	    set mime [ns_guesstype $fn]

	    if {[ns_conn zipaccepted] && [ns_set iget [ns_conn headers] Range] eq ""} {
		if {![file readable $fn.gz] || [file mtime $fn] > [file mtime $fn.gz]} {
		    # compress on the fly (once)
		    exec gzip -9 < $fn > $fn.gz
		}
		if {[file readable $fn.gz]} {
		    set fn $fn.gz
		    ns_set put [ns_conn outputheaders] Vary Accept-Encoding
		    ns_set put [ns_conn outputheaders] Content-Encoding gzip
		}
	    }
	    ns_respond -file $fn
	} else {
	    ns_returnnotfound
	}
	return filter_break
     }

     #
     # Upstream handler (deliver from a different server)
     #
     # Serve the requested file from an upstream server, which might be
     # an http or https server. NaviServer acts as a reverse proxy
     # server.  Files larger than a certain size (spoolsize) are
     # spooled to a temp file and are delivered asynchronously via the
     # writer threads if configured. Note that we can specify for every
     # filter rule different parameters (e.g. different timeouts).
     #
     nsf::proc upstream { what -target {-timeout 10:0} {-spoolsize 10000}} {

	if {[string match /proxy-admin/*tcl [ns_conn url]]} {
	    return filter_ok
	}
	
	#
	# Assemble URL
	#
	set url $target[ns_conn url]
	set query [ns_conn query]
	if {$query ne ""} {append url ?$query}
	set content [ns_conn content]

	#
	# Update Headers
	#
	set queryHeaders [ns_conn headers]
	ns_set update $queryHeaders X-Forwarded-For [ns_conn peeraddr]

	set http_cmd [expr {[string match https://* $target] ? "ns_ssl" : "ns_http"}]

	#
	# Build query for the upstream server
	#
	set cmd $http_cmd
	lappend cmd queue \
	    -method [ns_conn method] \
	    -headers $queryHeaders \
	    -timeout $timeout
	if {$content ne ""} {lappend cmd -body $content}
	lappend cmd $url

	set handle [{*}$cmd]
	
	#
	# Build reply headers
	#
	set replyHeaders [ns_set create]
	ns_set update $replyHeaders X-Processed Naviserver

	#
	# Wait for results
	#
	$http_cmd wait -result R -headers $replyHeaders -status S -spoolsize $spoolsize -file F $handle

	if {[info exists F]} {
	    ns_log notice "Spooled [file size $F] bytes to $F"
	    ns_respond -status $S -headers $replyHeaders -file $F
	    file delete $F
	} else {
	    ns_respond -status $S -headers $replyHeaders -binary $R
	}
	return filter_break
     }
}
===================================================================================


etc/proxy-conf-https.tcl
===================================================================================
#
# Configuration for https reverse proxy
# (upstream https)
#
ns_section ns/server/default
#ns_param	minthreads	30
#ns_param	maxthreads	30

ns_section	"ns/server/default/modules"
ns_param	nssock		nssock.so
ns_param	nssssl		nsssl.so

ns_section	"ns/server/default/module/nssock"
ns_param        port           	9002
ns_param        writerthreads	10

ns_section	"ns/server/default/module/nssssl"
ns_param	port          	9443
ns_param	certificate	/usr/local/oacs-head/openacs-4/etc/server.pem
ns_param      	ciphers    	"ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP"

#
# Place for local file deliveries
#
ns_section ns/server/default/fastpath
ns_param	serverdir 	/usr/local/proxy-https/

#
# Place for Tcl libraries to load for proxy
#
ns_section ns/server/default/tcl
ns_param	library		/usr/local/proxy-https/tcl
===================================================================================

Re: [naviserver-devel] Nsssl - small logging glitch

[David O. <da...@qc...>]

Gustaf,

That change did indeed clean up the log nicely. Looks fine now.

Thanks for that,

--
David


On 4 June 2013 14:11, Gustaf Neumann <ne...@wu...> wrote:

>  Hi David,
>
> no, this is not a configuration problem, but a problem with the shutdown
> order.
>
> This issue should be fixed by the following change:
>
> https://bitbucket.org/naviserver/naviserver/commits/d812197ba4853bd50d003f14555cf538e69da33b?at=default
>
> Naviserver has still a small problem, that a shutdown of the server might
> crash during the shutdown of the threads (it looks like a problem with
> Tcl's thread local storage being freed to early), but this seems to depend
> on the timings and is not easy to debug, since tcl shutdown contains still
> some moving magic. However, this happens so late that it is in practice not
> a problem.
>
> Please try the change...
>
> -gustaf neumann
>
> Am 04.06.13 13:09, schrieb David Osborne:
>
>   Hi,
>
> Thanks for 4.99.5. It's been working well for us so far.
>
>  I've noticed a small glitch in the logging of nsssl shutdowns (not sure
> if it's new to 4.99.5 or not)
>
>  When NsWaitDriversShutdown logs the "stopped" message for nsssl the
> driver name seems to gain an extra unprintable character.
> [9715.7f556c6f2700][-main-] Notice: [driver:nsssl#]: stopped
>  where # is an unprintable character (possibly 007F)
>
> I think it could be an issue relating to how NsWaitDriversShutdown in
> particular accesses the driver name, rather than a more global encoding
> problem, since NsStopDrivers logs the "stopping" message for nsssl without
> any similar problems on the same installation.
>
>  Could this be a local configuration problem?
>  This is using the encoding defaults from conf/sample-config.tcl
>
>  Regards,
> David Osborne
>
>
> ------------------------------------------------------------------------------
> How ServiceNow helps IT people transform IT departments:
> 1. A cloud service to automate IT design, transition and operations
> 2. Dashboards that offer high-level views of enterprise services
> 3. A single system of record for all IT processeshttp://p.sf.net/sfu/servicenow-d2d-j
>
>
>
> _______________________________________________
> naviserver-devel mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>
>
> --
> Univ.Prof. Dr. Gustaf Neumann
> Institute of Information Systems and New Media
> WU Vienna
> Augasse 2-6, A-1090 Vienna, AUSTRIA
>
>
>
> ------------------------------------------------------------------------------
> How ServiceNow helps IT people transform IT departments:
> 1. A cloud service to automate IT design, transition and operations
> 2. Dashboards that offer high-level views of enterprise services
> 3. A single system of record for all IT processes
> http://p.sf.net/sfu/servicenow-d2d-j
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>

Re: [naviserver-devel] Nsssl - small logging glitch

[Gustaf N. <ne...@wu...>]

Hi David,

no, this is not a configuration problem, but a problem with the shutdown 
order.

This issue should be fixed by the following change:
https://bitbucket.org/naviserver/naviserver/commits/d812197ba4853bd50d003f14555cf538e69da33b?at=default

Naviserver has still a small problem, that a shutdown of the server 
might crash during the shutdown of the threads (it looks like a problem 
with Tcl's thread local storage being freed to early), but this seems to 
depend on the timings and is not easy to debug, since tcl shutdown 
contains still some moving magic. However, this happens so late that it 
is in practice not a problem.

Please try the change...

-gustaf neumann

Am 04.06.13 13:09, schrieb David Osborne:
> Hi,
>
> Thanks for 4.99.5. It's been working well for us so far.
>
> I've noticed a small glitch in the logging of nsssl shutdowns (not 
> sure if it's new to 4.99.5 or not)
>
> When NsWaitDriversShutdown logs the "stopped" message for nsssl the 
> driver name seems to gain an extra unprintable character.
> [9715.7f556c6f2700][-main-] Notice: [driver:nsssl#]: stopped
> where # is an unprintable character (possibly 007F)
>
> I think it could be an issue relating to how NsWaitDriversShutdown in 
> particular accesses the driver name, rather than a more global 
> encoding problem, since NsStopDrivers logs the "stopping" message for 
> nsssl without any similar problems on the same installation.
>
> Could this be a local configuration problem?
> This is using the encoding defaults from conf/sample-config.tcl
>
> Regards,
> David Osborne
>
>
> ------------------------------------------------------------------------------
> How ServiceNow helps IT people transform IT departments:
> 1. A cloud service to automate IT design, transition and operations
> 2. Dashboards that offer high-level views of enterprise services
> 3. A single system of record for all IT processes
> http://p.sf.net/sfu/servicenow-d2d-j
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel


-- 
Univ.Prof. Dr. Gustaf Neumann
Institute of Information Systems and New Media
WU Vienna
Augasse 2-6, A-1090 Vienna, AUSTRIA

[naviserver-devel] Nsssl - small logging glitch

[David O. <da...@qc...>]

Hi,

Thanks for 4.99.5. It's been working well for us so far.

I've noticed a small glitch in the logging of nsssl shutdowns (not sure if
it's new to 4.99.5 or not)

When NsWaitDriversShutdown logs the "stopped" message for nsssl the driver
name seems to gain an extra unprintable character.
[9715.7f556c6f2700][-main-] Notice: [driver:nsssl#]: stopped
where # is an unprintable character (possibly 007F)

I think it could be an issue relating to how NsWaitDriversShutdown in
particular accesses the driver name, rather than a more global encoding
problem, since NsStopDrivers logs the "stopping" message for nsssl without
any similar problems on the same installation.

Could this be a local configuration problem?
This is using the encoding defaults from conf/sample-config.tcl

Regards,
David Osborne

Re: [naviserver-devel] status update

[Gustaf N. <ne...@wu...>]

Dear all,

after doing some more polishing (static analyzers, more valgrind test) 
and updating NEWS,
i've today tagged NaviServer and the modules mentioned below with 
naviserver-4.99.5 on bitbucket
and uploaded new tar files to sourceforge.

all the best
-gustaf neumann

Am 06.05.13 00:12, schrieb Gustaf Neumann:
> Am 03.05.13 13:52, schrieb Bernhard van Woerden:
>> My feeble attempt at this is a pull request on bitbucket.
>>
>> Hope this is of some use.
> Bernhard, many thanks, this was a great help. I've made yet
> another iteration over the NEWS file and tried to group changes
> in the top section.
>
> How should we do packaging?
> We have at least the following components with changes
>   - naviserver
>   - nsssl
>   - nsstats
>   - nsdbpg
>   - nssmtpd
>   - nsauthpam
> where the first four have larger changes. Any other
> good options other than
>   - naviserver
>   - naviserver-nsssl
>   ...?
>
> Should we mention changes from the drivers/packages
> as well in the NEWS file?
>
> Any objections to use the same tag on bitbucket for these packages
> to denote compatibility?
>
> -gustaf
>
>
>> Gustaf has been very industrious.
>>
>> -Bernhard
>>
>> On 29 April 2013 18:33, Bernhard van Woerden <ber...@qc... 
>> <mailto:ber...@qc...>> wrote:
>>
>>     ok, I'll have a go. I really need to read the diff for each
>>     commit so it will take me a day or 2 to put together a draft
>>     version.
>>
>>     - Bernhard
>>
>>
>>     On 29 April 2013 10:41, Gustaf Neumann <ne...@wu...
>>     <mailto:ne...@wu...>> wrote:
>>
>>         Dear Bernhard,
>>
>>         Would be great, if you could help here. Releasing a 4.99.5
>>         version soon and leaving further cleanup and
>>         documentation improvements for a 5.0 release sounds
>>         good to me. The biggest task for 4.99.5 is to provide a
>>         "cooked version" of the changlog for the NEWS file.
>>         Since the tagging of 4.99.4 more than 150 commits
>>         went in.... time flies like an arrow. I hope to be able to
>>         do this until the end of the week.
>>
>>         -gustaf neumann
>>
>>
>>         On 25.04.13 10:38, Bernhard van Woerden wrote:
>>>         Thanks Gustaf,
>>>
>>>         Do you plan to tag a new release ?
>>>         We want to put together Debian packages and it would be nice
>>>         to reference a release with changes since 4.99.1
>>>
>>>         Regards
>>>         Bernhard
>>>
>>>         On 25 April 2013 09:02, Gustaf Neumann <ne...@wu...
>>>         <mailto:ne...@wu...>> wrote:
>>>
>>>             Dear naviservlers,
>>>
>>>             a short update on the changes in the tip version of
>>>             Naviserver
>>>             and some modules on bitbucket:
>>>
>>>             - i've not made the big-cleanup of unused functions, but
>>>               marked several as deprecated. The server reports
>>>               now in the error.log when deprecated (C- or
>>>               Tcl-implemented) functions are used.
>>>
>>>             - the introspection interface for per-server and per-pool
>>>               reporting is implemented and works nicely (used e.g. in
>>>               the updated nsstats.tcl or in my set of munin plugins).
>>>               The old interface (which was not virtual-server and
>>>               pool-aware) is marked as deprecated.
>>>
>>>             - i've moved several locks from global mutexes on all
>>>               pools to muxtes per pool where appropriate to further
>>>               improve scalability (most people won't notice)
>>>
>>>             - nsdbpg: i've updated the interface of nsdbpg from
>>>               the old string based interface to Tcl_Objs and
>>>             implemented
>>>               for the provided SQL statements a new Tcl_Obj type named
>>>               " parsedSQL", which can reduce string copying and the
>>>               parsing of SQL statements for the bind-var emulation
>>>               significantly. When the Tcl_Obj is preserved, the
>>>               SQL statement is processed once (at the first call)
>>>               and the resulting C structures are preserved in the
>>>               internal representation of the Tcl_Obj. In later calls,
>>>               the internal representation of the parsed SQL statement
>>>               can be reused without overhead.
>>>
>>>             - I am updating the documentation whenever i step over
>>>               some documentation problems, so it is improving, but
>>>               there is still a log way to go.
>>>
>>>             When downloading the tip version of naviserver, please
>>>             get also nsdbpg/nsssl/nsstats when you use these modules.
>>>
>>>             We are using the tip version of naviserver and the modules
>>>             in production since a while. I think, we saw the last crash
>>>             last year in october or so.
>>>
>>>             best regards
>>>             -gustaf neumann
>>>
>>
>>

Re: [naviserver-devel] status update

[Gustaf N. <ne...@wu...>]

Am 03.05.13 13:52, schrieb Bernhard van Woerden:
> My feeble attempt at this is a pull request on bitbucket.
>
> Hope this is of some use.
Bernhard, many thanks, this was a great help. I've made yet
another iteration over the NEWS file and tried to group changes
in the top section.

How should we do packaging?
We have at least the following components with changes
   - naviserver
   - nsssl
   - nsstats
   - nsdbpg
   - nssmtpd
   - nsauthpam
where the first four have larger changes. Any other
good options other than
   - naviserver
   - naviserver-nsssl
   ...?

Should we mention changes from the drivers/packages
as well in the NEWS file?

Any objections to use the same tag on bitbucket for these packages
to denote compatibility?

-gustaf


> Gustaf has been very industrious.
>
> -Bernhard
>
> On 29 April 2013 18:33, Bernhard van Woerden <ber...@qc... 
> <mailto:ber...@qc...>> wrote:
>
>     ok, I'll have a go. I really need to read the diff for each commit
>     so it will take me a day or 2 to put together a draft version.
>
>     - Bernhard
>
>
>     On 29 April 2013 10:41, Gustaf Neumann <ne...@wu...
>     <mailto:ne...@wu...>> wrote:
>
>         Dear Bernhard,
>
>         Would be great, if you could help here. Releasing a 4.99.5
>         version soon and leaving further cleanup and
>         documentation improvements for a 5.0 release sounds
>         good to me. The biggest task for 4.99.5 is to provide a
>         "cooked version" of the changlog for the NEWS file.
>         Since the tagging of 4.99.4 more than 150 commits
>         went in.... time flies like an arrow. I hope to be able to
>         do this until the end of the week.
>
>         -gustaf neumann
>
>
>         On 25.04.13 10:38, Bernhard van Woerden wrote:
>>         Thanks Gustaf,
>>
>>         Do you plan to tag a new release ?
>>         We want to put together Debian packages and it would be nice
>>         to reference a release with changes since 4.99.1
>>
>>         Regards
>>         Bernhard
>>
>>         On 25 April 2013 09:02, Gustaf Neumann <ne...@wu...
>>         <mailto:ne...@wu...>> wrote:
>>
>>             Dear naviservlers,
>>
>>             a short update on the changes in the tip version of
>>             Naviserver
>>             and some modules on bitbucket:
>>
>>             - i've not made the big-cleanup of unused functions, but
>>               marked several as deprecated. The server reports
>>               now in the error.log when deprecated (C- or
>>               Tcl-implemented) functions are used.
>>
>>             - the introspection interface for per-server and per-pool
>>               reporting is implemented and works nicely (used e.g. in
>>               the updated nsstats.tcl or in my set of munin plugins).
>>               The old interface (which was not virtual-server and
>>               pool-aware) is marked as deprecated.
>>
>>             - i've moved several locks from global mutexes on all
>>               pools to muxtes per pool where appropriate to further
>>               improve scalability (most people won't notice)
>>
>>             - nsdbpg: i've updated the interface of nsdbpg from
>>               the old string based interface to Tcl_Objs and implemented
>>               for the provided SQL statements a new Tcl_Obj type named
>>               " parsedSQL", which can reduce string copying and the
>>               parsing of SQL statements for the bind-var emulation
>>               significantly. When the Tcl_Obj is preserved, the
>>               SQL statement is processed once (at the first call)
>>               and the resulting C structures are preserved in the
>>               internal representation of the Tcl_Obj. In later calls,
>>               the internal representation of the parsed SQL statement
>>               can be reused without overhead.
>>
>>             - I am updating the documentation whenever i step over
>>               some documentation problems, so it is improving, but
>>               there is still a log way to go.
>>
>>             When downloading the tip version of naviserver, please
>>             get also nsdbpg/nsssl/nsstats when you use these modules.
>>
>>             We are using the tip version of naviserver and the modules
>>             in production since a while. I think, we saw the last crash
>>             last year in october or so.
>>
>>             best regards
>>             -gustaf neumann
>>
>
>
>         ------------------------------------------------------------------------------
>         Try New Relic Now & We'll Send You this Cool Shirt
>         New Relic is the only SaaS-based application performance
>         monitoring service
>         that delivers powerful full stack analytics. Optimize and
>         monitor your
>         browser, app, & servers with just a few lines of code. Try New
>         Relic
>         and get this awesome Nerd Life shirt!
>         http://p.sf.net/sfu/newrelic_d2d_apr
>         _______________________________________________
>         naviserver-devel mailing list
>         nav...@li...
>         <mailto:nav...@li...>
>         https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>
>
>
>     -- 
>     Bernhard van Woerden
>     Qcode Software Limited
>     www.qcode.co.uk <http://www.qcode.co.uk>
>     T 01463 227488 <tel:01463%20227488>
>
>
>
>
> -- 
> Bernhard van Woerden
> Qcode Software Limited
> www.qcode.co.uk <http://www.qcode.co.uk>
> T 01463 227488
>
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite
> It's a free troubleshooting tool designed for production
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://p.sf.net/sfu/appdyn_d2d_ap2
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel


-- 
Univ.Prof. Dr. Gustaf Neumann
Institute of Information Systems and New Media
WU Vienna
Augasse 2-6, A-1090 Vienna, AUSTRIA

Re: [naviserver-devel] status update

[Bernhard v. W. <ber...@qc...>]

My feeble attempt at this is a pull request on bitbucket.

Hope this is of some use.
Gustaf has been very industrious.

-Bernhard

On 29 April 2013 18:33, Bernhard van Woerden <ber...@qc...> wrote:

> ok, I'll have a go. I really need to read the diff for each commit so it
> will take me a day or 2 to put together a draft version.
>
> - Bernhard
>
>
> On 29 April 2013 10:41, Gustaf Neumann <ne...@wu...> wrote:
>
>>  Dear Bernhard,
>>
>> Would be great, if you could help here. Releasing a 4.99.5
>> version soon and leaving further cleanup and
>> documentation improvements for a 5.0 release sounds
>> good to me. The biggest task for 4.99.5 is to provide a
>> "cooked version" of the changlog for the NEWS file.
>> Since the tagging of 4.99.4 more than 150 commits
>> went in.... time flies like an arrow. I hope to be able to
>> do this until the end of the week.
>>
>> -gustaf neumann
>>
>>
>> On 25.04.13 10:38, Bernhard van Woerden wrote:
>>
>> Thanks Gustaf,
>>
>>  Do you plan to tag a new release ?
>> We want to put together Debian packages and it would be nice
>> to reference a release with changes since 4.99.1
>>
>>  Regards
>> Bernhard
>>
>> On 25 April 2013 09:02, Gustaf Neumann <ne...@wu...> wrote:
>>
>>>  Dear naviservlers,
>>>
>>> a short update on the changes in the tip version of Naviserver
>>> and some modules on bitbucket:
>>>
>>> - i've not made the big-cleanup of unused functions, but
>>>   marked several as deprecated. The server reports
>>>   now in the error.log when deprecated (C- or
>>>   Tcl-implemented) functions are used.
>>>
>>> - the introspection interface for per-server and per-pool
>>>   reporting is implemented and works nicely (used e.g. in
>>>   the updated nsstats.tcl or in my set of munin plugins).
>>>   The old interface (which was not virtual-server and
>>>   pool-aware) is marked as deprecated.
>>>
>>> - i've moved several locks from global mutexes on all
>>>   pools to muxtes per pool where appropriate to further
>>>   improve scalability (most people won't notice)
>>>
>>> - nsdbpg: i've updated the interface of nsdbpg from
>>>   the old string based interface to Tcl_Objs and implemented
>>>   for the provided SQL statements a new Tcl_Obj type named
>>>   " parsedSQL", which can reduce string copying and the
>>>   parsing of SQL statements for the bind-var emulation
>>>   significantly. When the Tcl_Obj is preserved, the
>>>   SQL statement is processed once (at the first call)
>>>   and the resulting C structures are preserved in the
>>>   internal representation of the Tcl_Obj. In later calls,
>>>   the internal representation of the parsed SQL statement
>>>   can be reused without overhead.
>>>
>>> - I am updating the documentation whenever i step over
>>>   some documentation problems, so it is improving, but
>>>   there is still a log way to go.
>>>
>>> When downloading the tip version of naviserver, please
>>> get also nsdbpg/nsssl/nsstats when you use these modules.
>>>
>>> We are using the tip version of naviserver and the modules
>>> in production since a while. I think, we saw the last crash
>>> last year in october or so.
>>>
>>> best regards
>>> -gustaf neumann
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Try New Relic Now & We'll Send You this Cool Shirt
>> New Relic is the only SaaS-based application performance monitoring
>> service
>> that delivers powerful full stack analytics. Optimize and monitor your
>> browser, app, & servers with just a few lines of code. Try New Relic
>> and get this awesome Nerd Life shirt!
>> http://p.sf.net/sfu/newrelic_d2d_apr
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>>
>>
>
>
> --
> Bernhard van Woerden
> Qcode Software Limited
> www.qcode.co.uk
> T 01463 227488
>



-- 
Bernhard van Woerden
Qcode Software Limited
www.qcode.co.uk
T 01463 227488

[naviserver-devel] startup delays

[Jeff R. <dv...@di...>]

Hi all,

This message:
http://openacs.org/forums/message-view?message_id=4027670
tapped into something I've been thinking about for a while.

I have a server that takes around 5 minutes to start or restart.  The 
major reason for this is that it's a slow $2/mo openvz VPS and you get 
what you pay for, but it seems that other people have similar issues 
although not as bad.

In any case, the underlying problem is that the server can't start 
serving requests until after startup is complete and since startup can 
do arbitrary things, it can take arbitrarily long.

My ideas here aren't completely baked, so I'd like to bounce the 
concepts off people before I attempt to implement them.

One thought I had is to have a sort of "pre-startup" server running that 
can answer http requests and just respond "server starting, please 
wait".  That would be an improvement over the current situation where 
you get a connection refused. (or it may stay in a "connecting" state 
until the server actually starts if the listen socket is prebound) 
However, restarts would still lead to a period of time where no content 
is being served.

Another thought is to enhance the pre-bind mechanism to tell the server 
to not just bind particular sockets before dropping privileges, but to 
pass in already-opened descriptors and tell the server what they are. 
Then when restarting, the currently running server would pass its own 
open listen socket fds to the new server to prebind and then keep 
running until the new server signals that it is ready to start (e.g., by 
a signal).  So the timeline looks like;

parent server:                     child server:
start
serve requests ...
ns_server_restart
   fork()
   exec("nsd -b 0.0.0.0:80:@5")     starts, notes that fd 5 is its listen
signal(SIGUSR2, cleanup and exit)   socket
keeps running                      runs startup scripts
serve requests ...                 ...
...                                kill(getppid(), SIGUSR2)
cleanup and exit                   serve requests


There's various ways this could be made to work, but the core idea is to 
keep the old server running until the new one is ready.  This approach 
wouldn't work so well with daemontools where the server is typically 
restarted by just having the current one exit, but it would be a 
possible alternative.

Does anyone else think this is worth pursuing?

-J

Re: [naviserver-devel] status update

[Bernhard v. W. <ber...@qc...>]

ok, I'll have a go. I really need to read the diff for each commit so it
will take me a day or 2 to put together a draft version.

- Bernhard


On 29 April 2013 10:41, Gustaf Neumann <ne...@wu...> wrote:

>  Dear Bernhard,
>
> Would be great, if you could help here. Releasing a 4.99.5
> version soon and leaving further cleanup and
> documentation improvements for a 5.0 release sounds
> good to me. The biggest task for 4.99.5 is to provide a
> "cooked version" of the changlog for the NEWS file.
> Since the tagging of 4.99.4 more than 150 commits
> went in.... time flies like an arrow. I hope to be able to
> do this until the end of the week.
>
> -gustaf neumann
>
>
> On 25.04.13 10:38, Bernhard van Woerden wrote:
>
> Thanks Gustaf,
>
>  Do you plan to tag a new release ?
> We want to put together Debian packages and it would be nice
> to reference a release with changes since 4.99.1
>
>  Regards
> Bernhard
>
> On 25 April 2013 09:02, Gustaf Neumann <ne...@wu...> wrote:
>
>>  Dear naviservlers,
>>
>> a short update on the changes in the tip version of Naviserver
>> and some modules on bitbucket:
>>
>> - i've not made the big-cleanup of unused functions, but
>>   marked several as deprecated. The server reports
>>   now in the error.log when deprecated (C- or
>>   Tcl-implemented) functions are used.
>>
>> - the introspection interface for per-server and per-pool
>>   reporting is implemented and works nicely (used e.g. in
>>   the updated nsstats.tcl or in my set of munin plugins).
>>   The old interface (which was not virtual-server and
>>   pool-aware) is marked as deprecated.
>>
>> - i've moved several locks from global mutexes on all
>>   pools to muxtes per pool where appropriate to further
>>   improve scalability (most people won't notice)
>>
>> - nsdbpg: i've updated the interface of nsdbpg from
>>   the old string based interface to Tcl_Objs and implemented
>>   for the provided SQL statements a new Tcl_Obj type named
>>   " parsedSQL", which can reduce string copying and the
>>   parsing of SQL statements for the bind-var emulation
>>   significantly. When the Tcl_Obj is preserved, the
>>   SQL statement is processed once (at the first call)
>>   and the resulting C structures are preserved in the
>>   internal representation of the Tcl_Obj. In later calls,
>>   the internal representation of the parsed SQL statement
>>   can be reused without overhead.
>>
>> - I am updating the documentation whenever i step over
>>   some documentation problems, so it is improving, but
>>   there is still a log way to go.
>>
>> When downloading the tip version of naviserver, please
>> get also nsdbpg/nsssl/nsstats when you use these modules.
>>
>> We are using the tip version of naviserver and the modules
>> in production since a while. I think, we saw the last crash
>> last year in october or so.
>>
>> best regards
>> -gustaf neumann
>>
>>
>
>
> ------------------------------------------------------------------------------
> Try New Relic Now & We'll Send You this Cool Shirt
> New Relic is the only SaaS-based application performance monitoring service
> that delivers powerful full stack analytics. Optimize and monitor your
> browser, app, & servers with just a few lines of code. Try New Relic
> and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>


-- 
Bernhard van Woerden
Qcode Software Limited
www.qcode.co.uk
T 01463 227488

Re: [naviserver-devel] status update

[Gustaf N. <ne...@wu...>]

Dear Bernhard,

Would be great, if you could help here. Releasing a 4.99.5
version soon and leaving further cleanup and
documentation improvements for a 5.0 release sounds
good to me. The biggest task for 4.99.5 is to provide a
"cooked version" of the changlog for the NEWS file.
Since the tagging of 4.99.4 more than 150 commits
went in.... time flies like an arrow. I hope to be able to
do this until the end of the week.

-gustaf neumann

On 25.04.13 10:38, Bernhard van Woerden wrote:
> Thanks Gustaf,
>
> Do you plan to tag a new release ?
> We want to put together Debian packages and it would be 
> nice to reference a release with changes since 4.99.1
>
> Regards
> Bernhard
>
> On 25 April 2013 09:02, Gustaf Neumann <ne...@wu... 
> <mailto:ne...@wu...>> wrote:
>
>     Dear naviservlers,
>
>     a short update on the changes in the tip version of
>     Naviserver
>     and some modules on bitbucket:
>
>     - i've not made the big-cleanup of unused functions, but
>       marked several as deprecated. The server reports
>       now in the error.log when deprecated (C- or
>       Tcl-implemented) functions are used.
>
>     - the introspection interface for per-server and per-pool
>       reporting is implemented and works nicely (used e.g. in
>       the updated nsstats.tcl or in my set of munin plugins).
>       The old interface (which was not virtual-server and
>       pool-aware) is marked as deprecated.
>
>     - i've moved several locks from global mutexes on all
>       pools to muxtes per pool where appropriate to further
>       improve scalability (most people won't notice)
>
>     - nsdbpg: i've updated the interface of nsdbpg from
>       the old string based interface to Tcl_Objs and
>     implemented
>       for the provided SQL statements a new Tcl_Obj type
>     named
>       " parsedSQL", which can reduce string copying and the
>       parsing of SQL statements for the bind-var emulation
>       significantly. When the Tcl_Obj is preserved, the
>       SQL statement is processed once (at the first call)
>       and the resulting C structures are preserved in the
>       internal representation of the Tcl_Obj. In later calls,
>       the internal representation of the parsed SQL statement
>       can be reused without overhead.
>
>     - I am updating the documentation whenever i step over
>       some documentation problems, so it is improving, but
>       there is still a log way to go.
>
>     When downloading the tip version of naviserver, please
>     get also nsdbpg/nsssl/nsstats when you use these modules.
>
>     We are using the tip version of naviserver and the modules
>     in production since a while. I think, we saw the last
>     crash
>     last year in october or so.
>
>     best regards
>     -gustaf neumann
>

Re: [naviserver-devel] status update

[Bernhard v. W. <ber...@qc...>]

Thanks Gustaf,

Do you plan to tag a new release ?
We want to put together Debian packages and it would be nice to reference a
release with changes since 4.99.1

Regards
Bernhard

On 25 April 2013 09:02, Gustaf Neumann <ne...@wu...> wrote:

>  Dear naviservlers,
>
> a short update on the changes in the tip version of Naviserver
> and some modules on bitbucket:
>
> - i've not made the big-cleanup of unused functions, but
>   marked several as deprecated. The server reports
>   now in the error.log when deprecated (C- or
>   Tcl-implemented) functions are used.
>
> - the introspection interface for per-server and per-pool
>   reporting is implemented and works nicely (used e.g. in
>   the updated nsstats.tcl or in my set of munin plugins).
>   The old interface (which was not virtual-server and
>   pool-aware) is marked as deprecated.
>
> - i've moved several locks from global mutexes on all
>   pools to muxtes per pool where appropriate to further
>   improve scalability (most people won't notice)
>
> - nsdbpg: i've updated the interface of nsdbpg from
>   the old string based interface to Tcl_Objs and implemented
>   for the provided SQL statements a new Tcl_Obj type named
>   " parsedSQL", which can reduce string copying and the
>   parsing of SQL statements for the bind-var emulation
>   significantly. When the Tcl_Obj is preserved, the
>   SQL statement is processed once (at the first call)
>   and the resulting C structures are preserved in the
>   internal representation of the Tcl_Obj. In later calls,
>   the internal representation of the parsed SQL statement
>   can be reused without overhead.
>
> - I am updating the documentation whenever i step over
>   some documentation problems, so it is improving, but
>   there is still a log way to go.
>
> When downloading the tip version of naviserver, please
> get also nsdbpg/nsssl/nsstats when you use these modules.
>
> We are using the tip version of naviserver and the modules
> in production since a while. I think, we saw the last crash
> last year in october or so.
>
> best regards
> -gustaf neumann
>
>
> ------------------------------------------------------------------------------
> Try New Relic Now & We'll Send You this Cool Shirt
> New Relic is the only SaaS-based application performance monitoring service
> that delivers powerful full stack analytics. Optimize and monitor your
> browser, app, & servers with just a few lines of code. Try New Relic
> and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>

[naviserver-devel] status update

[Gustaf N. <ne...@wu...>]

Dear naviservlers,

a short update on the changes in the tip version of Naviserver
and some modules on bitbucket:

- i've not made the big-cleanup of unused functions, but
   marked several as deprecated. The server reports
   now in the error.log when deprecated (C- or
   Tcl-implemented) functions are used.

- the introspection interface for per-server and per-pool
   reporting is implemented and works nicely (used e.g. in
   the updated nsstats.tcl or in my set of munin plugins).
   The old interface (which was not virtual-server and
   pool-aware) is marked as deprecated.

- i've moved several locks from global mutexes on all
   pools to muxtes per pool where appropriate to further
   improve scalability (most people won't notice)

- nsdbpg: i've updated the interface of nsdbpg from
   the old string based interface to Tcl_Objs and implemented
   for the provided SQL statements a new Tcl_Obj type named
   " parsedSQL", which can reduce string copying and the
   parsing of SQL statements for the bind-var emulation
   significantly. When the Tcl_Obj is preserved, the
   SQL statement is processed once (at the first call)
   and the resulting C structures are preserved in the
   internal representation of the Tcl_Obj. In later calls,
   the internal representation of the parsed SQL statement
   can be reused without overhead.

- I am updating the documentation whenever i step over
   some documentation problems, so it is improving, but
   there is still a log way to go.

When downloading the tip version of naviserver, please
get also nsdbpg/nsssl/nsstats when you use these modules.

We are using the tip version of naviserver and the modules
in production since a while. I think, we saw the last crash
last year in october or so.

best regards
-gustaf neumann