naviserver-devel — Developer discussion

Re: [naviserver-devel] ns_urlencode on Windows Naviserver

[Brian F. <Bri...@qu...>]

Hi Maurizio

I really don't know. I don't have access to a Linux installation to check. Are you suggesting that the API may have changed since AOLserver?

Brian


From: Maurizio Martignano [mailto:Mau...@sp...]
Sent: 12 May 2017 12:31
To: nav...@li...
Subject: Re: [naviserver-devel] ns_urlencode on Windows Naviserver

Dear Brian,
               Is this really a "Windows Naviserver" issue or is it a matter of a new implementation of the function, different from what was previously done in Aolserver?

The two implementations seem to me quite different.

Thanks a lot,
Maurizio


From: Brian Fenton [mailto:Bri...@qu...]
Sent: 12 May 2017 11:15
To: 'nav...@li...' <nav...@li...<mailto:nav...@li...>>
Subject: [naviserver-devel] ns_urlencode on Windows Naviserver

I think there might be a problem with ns_urlencode on Windows Naviserver.

I just ran this example (taken directly from the documentation):
ns_urlencode http://www.aolserver.com/redirect.adp?url=http://www.aol.com&t=1,2,3

And the result was unexpected: http://www.aolserver.com/redirect.adp?url%3dhttp://www.aol.com%26t%3d1%2c2%2c

The correct result is http%3a%2f%2fwww%2eaolserver%2ecom%2fredirect%2eadp%3furl%3dhttp%3a%2f%2fwww%2eaol%2ecom%26t%3d1%2c2%2c3

This is breaking some of our existing AOLserver code during migration testing.

Any suggestions?

Brian Fenton
Senior Developer

Phone: +353 1 679 9933
grantmanagementsoftware.com<http://www.grantmanagementsoftware.com>

[AIMS Grant and Programme Management]<http://www.grantmanagementsoftware.com/>




[Twitter]<https://twitter.com/AIMS_qcl>

[LinkedIn]<https://www.linkedin.com/grp/home?gid=2978658>

Quest Computing Ltd. Ushers Court | 31-33 Ushers Quay | Dublin 8, Ireland






This e-mail transmission may contain confidential information that is intended for the individual or entity named on the e-mail address. If you are not the intended recipient, please reply to the sender so that Quest Computing Ltd can arrange for the proper delivery and then please delete the message from your inbox.

If you have received this e-mail in error, you are hereby notified that any disclosure, copying, distribution, or reliance upon the contents of this e-mail is strictly prohibited. Quest Computing Ltd., Ushers Court, 31-33 Ushers Quay, Dublin 8, Ireland. Quest Computing Ltd. is registered in Ireland No. 146435.

Re: [naviserver-devel] ns_urlencode on Windows Naviserver

[Maurizio M. <Mau...@sp...>]

Dear Brian,

               Is this really a "Windows Naviserver" issue or is it a matter
of a new implementation of the function, different from what was previously
done in Aolserver? 

 

The two implementations seem to me quite different.

 

Thanks a lot,

Maurizio

 

 

From: Brian Fenton [mailto:Bri...@qu...] 
Sent: 12 May 2017 11:15
To: 'nav...@li...'
<nav...@li...>
Subject: [naviserver-devel] ns_urlencode on Windows Naviserver

 

I think there might be a problem with ns_urlencode on Windows Naviserver.

 

I just ran this example (taken directly from the documentation):

ns_urlencode http://www.aolserver.com/redirect.adp?url=http://www.aol.com
<http://www.aolserver.com/redirect.adp?url=http://www.aol.com&t=1,2,3>
&t=1,2,3

 

And the result was unexpected:
http://www.aolserver.com/redirect.adp?url%3dhttp://www.aol.com%26t%3d1%2c2%2
c

 

The correct result is
http%3a%2f%2fwww%2eaolserver%2ecom%2fredirect%2eadp%3furl%3dhttp%3a%2f%2fwww
%2eaol%2ecom%26t%3d1%2c2%2c3

 

This is breaking some of our existing AOLserver code during migration
testing.

 

Any suggestions?

 



Brian Fenton
Senior Developer

Phone: +353 1 679 9933
 <http://www.grantmanagementsoftware.com> grantmanagementsoftware.com

 <http://www.grantmanagementsoftware.com/> 


 



 <https://twitter.com/AIMS_qcl> 

 <https://www.linkedin.com/grp/home?gid=2978658> 

Quest Computing Ltd. Ushers Court | 31-33 Ushers Quay | Dublin 8, Ireland


 


 


This e-mail transmission may contain confidential information that is
intended for the individual or entity named on the e-mail address. If you
are not the intended recipient, please reply to the sender so that Quest
Computing Ltd can arrange for the proper delivery and then please delete the
message from your inbox.

If you have received this e-mail in error, you are hereby notified that any
disclosure, copying, distribution, or reliance upon the contents of this
e-mail is strictly prohibited. Quest Computing Ltd., Ushers Court, 31-33
Ushers Quay, Dublin 8, Ireland. Quest Computing Ltd. is registered in
Ireland No. 146435.

 

 

[naviserver-devel] ns_urlencode on Windows Naviserver

[Brian F. <Bri...@qu...>]

I think there might be a problem with ns_urlencode on Windows Naviserver.

I just ran this example (taken directly from the documentation):
ns_urlencode http://www.aolserver.com/redirect.adp?url=http://www.aol.com&t=1,2,3

And the result was unexpected: http://www.aolserver.com/redirect.adp?url%3dhttp://www.aol.com%26t%3d1%2c2%2c

The correct result is http%3a%2f%2fwww%2eaolserver%2ecom%2fredirect%2eadp%3furl%3dhttp%3a%2f%2fwww%2eaol%2ecom%26t%3d1%2c2%2c3

This is breaking some of our existing AOLserver code during migration testing.

Any suggestions?

Brian Fenton
Senior Developer

Phone: +353 1 679 9933
grantmanagementsoftware.com<http://www.grantmanagementsoftware.com>

[AIMS Grant and Programme Management]<http://www.grantmanagementsoftware.com/>




[Twitter]<https://twitter.com/AIMS_qcl>

[LinkedIn]<https://www.linkedin.com/grp/home?gid=2978658>

Quest Computing Ltd. Ushers Court | 31-33 Ushers Quay | Dublin 8, Ireland






This e-mail transmission may contain confidential information that is intended for the individual or entity named on the e-mail address. If you are not the intended recipient, please reply to the sender so that Quest Computing Ltd can arrange for the proper delivery and then please delete the message from your inbox.

If you have received this e-mail in error, you are hereby notified that any disclosure, copying, distribution, or reliance upon the contents of this e-mail is strictly prohibited. Quest Computing Ltd., Ushers Court, 31-33 Ushers Quay, Dublin 8, Ireland. Quest Computing Ltd. is registered in Ireland No. 146435.

Re: [naviserver-devel] ns_proxy hang

[Gustaf N. <ne...@wu...>]

I'll look at it at the weekend - unless someone else can fix this before 
this.

-g
Am 10.05.17 um 16:00 schrieb David Osborne:
>  Increasing waittimeout doesn't seem to have any effect on this problem.
>
> I have backtraces of all threads at the point of the hang here:
> https://gist.github.com/davidqc/ebee38528b0a40a0b8d028981ad933e6
>
> Thread 19 I think is the culprit:
>
> Thread 19 (Thread 0x7fffaaffd700 (LWP 17652)):
> #0  0x00007ffff6322b89 in __libc_waitpid (pid=pid@entry=17651, stat_loc=stat_loc@entry=0x7fffaaffcde4, options=options@entry=0)
>      at ../sysdeps/unix/sysv/linux/waitpid.c:40
> #1  0x00007ffff7b5aa4c in Ns_WaitForProcess (pid=17651, exitcodePtr=0x0) at exec.c:178
> #2  0x00007ffff1b68615 in ReaperThread (UNUSED_arg=0x44f3) at nsproxylib.c:2935
> #3  0x00007ffff74b886d in NsThreadMain (arg=<optimized out>) at thread.c:232
> #4  0x00007ffff74b98a9 in ThreadMain (arg=<optimized out>) at pthread.c:830
> #5  0x00007ffff5e500a4 in start_thread (arg=0x7fffaaffd700) at pthread_create.c:309
> #6  0x00007ffff635162d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
>
> On 10 May 2017 at 13:04, Gustaf Neumann <ne...@wu... 
> <mailto:ne...@wu...>> wrote:
>
>     We don't see such hangs either. Does increasing the waittimeout
>     solve this issue?
>     ... not as a fix, but to narrow the problem down.
>     -gn
>
>     Am 10.05.17 um 13:38 schrieb David Osborne:
>>     The manifestation for us in production is quite insidious and
>>     difficult to spot the root cause of. Certainly not easy to see
>>     via the logs.
>>
>>     One place we are experiencing it is in code which generates
>>     outgoing emails. Sometimes, when the outgoing email is
>>     particularly large, nsproxy times out while using a external
>>     utility to convert (the large) html to text. This never seems
>>     like a big issue at the time.
>>
>>     But it's the NEXT task which calls nsproxy which will hang
>>     forever without error. The then system just slowly grinds to a
>>     halt until naviserver is restarted.
>>
>>     nscp 2> ns_proxy configure exec
>>     -env {} -exec /usr/lib/naviserver/bin/nsproxy -init {} -reinit {}
>>     -maxslaves 8 -maxruns 0 -gettimeout 0 -evaltimeout 0 -sendtimeout
>>     5000 -recvtimeout 5000 -waittimeout 1000 -idletimeout 300000
>>
>>     As I mentioned, when I watched the hang in gdb, it seemed to be
>>     waitpid() (via NsWaitProcess) where the hung threads are waiting.
>
>
>     ------------------------------------------------------------------------------
>     Check out the vibrant tech community on one of the world's most
>     engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>     _______________________________________________
>     naviserver-devel mailing list
>     nav...@li...
>     <mailto:nav...@li...>
>     https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>     <https://lists.sourceforge.net/lists/listinfo/naviserver-devel>
>
>
>
>
> -- 
> David Osborne
> Qcode Software Limited
> http://www.qcode.co.uk
> T: +44 (0)1463 896484
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel


-- 
Univ.Prof. Dr. Gustaf Neumann
WU Vienna
Institute of Information Systems and New Media
Welthandelsplatz 1, A-1020 Vienna, Austria

Re: [naviserver-devel] ns_proxy hang

[David O. <da...@qc...>]

 Increasing waittimeout doesn't seem to have any effect on this problem.

I have backtraces of all threads at the point of the hang here:
https://gist.github.com/davidqc/ebee38528b0a40a0b8d028981ad933e6

Thread 19 I think is the culprit:

Thread 19 (Thread 0x7fffaaffd700 (LWP 17652)):
#0  0x00007ffff6322b89 in __libc_waitpid (pid=pid@entry=17651,
stat_loc=stat_loc@entry=0x7fffaaffcde4, options=options@entry=0)
    at ../sysdeps/unix/sysv/linux/waitpid.c:40
#1  0x00007ffff7b5aa4c in Ns_WaitForProcess (pid=17651,
exitcodePtr=0x0) at exec.c:178
#2  0x00007ffff1b68615 in ReaperThread (UNUSED_arg=0x44f3) at nsproxylib.c:2935
#3  0x00007ffff74b886d in NsThreadMain (arg=<optimized out>) at thread.c:232
#4  0x00007ffff74b98a9 in ThreadMain (arg=<optimized out>) at pthread.c:830
#5  0x00007ffff5e500a4 in start_thread (arg=0x7fffaaffd700) at
pthread_create.c:309
#6  0x00007ffff635162d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111


On 10 May 2017 at 13:04, Gustaf Neumann <ne...@wu...> wrote:

> We don't see such hangs either. Does increasing the waittimeout solve this
> issue?
> ... not as a fix, but to narrow the problem down.
> -gn
>
> Am 10.05.17 um 13:38 schrieb David Osborne:
>
> The manifestation for us in production is quite insidious and difficult to
> spot the root cause of. Certainly not easy to see via the logs.
>
> One place we are experiencing it is in code which generates outgoing
> emails. Sometimes, when the outgoing email is particularly large, nsproxy
> times out while using a external utility to convert (the large) html to
> text. This never seems like a big issue at the time.
>
> But it's the NEXT task which calls nsproxy which will hang forever without
> error. The then system just slowly grinds to a halt until naviserver is
> restarted.
>
> nscp 2> ns_proxy configure exec
> -env {} -exec /usr/lib/naviserver/bin/nsproxy -init {} -reinit {}
> -maxslaves 8 -maxruns 0 -gettimeout 0 -evaltimeout 0 -sendtimeout 5000
> -recvtimeout 5000 -waittimeout 1000 -idletimeout 300000
>
> As I mentioned, when I watched the hang in gdb, it seemed to be waitpid()
> (via NsWaitProcess) where the hung threads are waiting.
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>


-- 
David Osborne
Qcode Software Limited
http://www.qcode.co.uk
T: +44 (0)1463 896484

Re: [naviserver-devel] ns_proxy hang

[Gustaf N. <ne...@wu...>]

We don't see such hangs either. Does increasing the waittimeout solve 
this issue?
... not as a fix, but to narrow the problem down.
-gn

Am 10.05.17 um 13:38 schrieb David Osborne:
> The manifestation for us in production is quite insidious and 
> difficult to spot the root cause of. Certainly not easy to see via the 
> logs.
>
> One place we are experiencing it is in code which generates outgoing 
> emails. Sometimes, when the outgoing email is particularly large, 
> nsproxy times out while using a external utility to convert (the 
> large) html to text. This never seems like a big issue at the time.
>
> But it's the NEXT task which calls nsproxy which will hang forever 
> without error. The then system just slowly grinds to a halt until 
> naviserver is restarted.
>
> nscp 2> ns_proxy configure exec
> -env {} -exec /usr/lib/naviserver/bin/nsproxy -init {} -reinit {} 
> -maxslaves 8 -maxruns 0 -gettimeout 0 -evaltimeout 0 -sendtimeout 5000 
> -recvtimeout 5000 -waittimeout 1000 -idletimeout 300000
>
> As I mentioned, when I watched the hang in gdb, it seemed to be 
> waitpid() (via NsWaitProcess) where the hung threads are waiting.

Re: [naviserver-devel] ns_proxy hang

[David O. <da...@qc...>]

The manifestation for us in production is quite insidious and difficult to
spot the root cause of. Certainly not easy to see via the logs.

One place we are experiencing it is in code which generates outgoing
emails. Sometimes, when the outgoing email is particularly large, nsproxy
times out while using a external utility to convert (the large) html to
text. This never seems like a big issue at the time.

But it's the NEXT task which calls nsproxy which will hang forever without
error. The then system just slowly grinds to a halt until naviserver is
restarted.

nscp 2> ns_proxy configure exec
-env {} -exec /usr/lib/naviserver/bin/nsproxy -init {} -reinit {}
-maxslaves 8 -maxruns 0 -gettimeout 0 -evaltimeout 0 -sendtimeout 5000
-recvtimeout 5000 -waittimeout 1000 -idletimeout 300000

As I mentioned, when I watched the hang in gdb, it seemed to be waitpid()
(via NsWaitProcess) where the hung threads are waiting.


On 10 May 2017 at 10:15, Gustaf Neumann <ne...@wu...> wrote:

> Dear David,
>
> i have not been able to look into this yet. However, i've checked our logs
> on several servers using nsproxy intensively, but did not see any problems.
> How is the proxy setup (config section and "ns_proxy configure" command) in
> your environment?
>
> -g
>
>

Re: [naviserver-devel] ns_proxy hang

[Gustaf N. <ne...@wu...>]

Dear David,

i have not been able to look into this yet. However, i've checked our 
logs on several servers using nsproxy intensively, but did not see any 
problems. How is the proxy setup (config section and "ns_proxy 
configure" command) in your environment?

-g
Am 10.05.17 um 10:50 schrieb David Osborne:
> Has anyone got any ideas on debugging this issue?
> Turns out it's causing us many more operational issues that we first 
> thought...
>
> Regards,
> -- 
> David
>
> On 18 April 2017 at 15:44, David Osborne <da...@qc... 
> <mailto:da...@qc...>> wrote:
>
>     Hi,
>
>     We're encountering an occasional problem which means calls to
>     ns_proxy hang forever.
>     It seems to be when an ns_proxy eval command timeouts during
>     evaluation & it produces a large amount of output.
>     Under those circumstances ns_proxy becomes inoperable.
>
>     Following is a contrived test case which I think demonstrates what
>     we are experiencing.
>     (The ns_proxy timeout here is set unrealistically small to force
>     the timeout - in production the timeout is 1 second)
>
>
>     Normal test case using command:
>
>     |ns_proxy eval $handle [list exec dd if=/dev/urandom bs=1K count=23
>     | base64] 1|
>
>
>     % set handle [ns_proxy get exec]
>     [07/Apr/2017:13:06:57][28009.7fa7b760b700][-command-]
>     Debug(nsproxy): set expire in 300000 ms for pool exec slave 28197
>     [07/Apr/2017:13:06:57][28009.7fa7b760b700][-command-]
>     Debug(nsproxy): nsproxy: slave 28197 started
>     [07/Apr/2017:13:06:57][28197.7f3f5dcc3700][-main-] Notice: OpenSSL
>     1.0.1t  3 May 2016 initialized
>     [07/Apr/2017:13:06:57][28009.7fa78bfff700][-nsproxy:reap-] Notice:
>     starting
>     [07/Apr/2017:13:06:57][28009.7fa78bfff700][-nsproxy:reap-]
>     Debug(nsproxy): reaper run
>     [07/Apr/2017:13:06:57][28009.7fa78bfff700][-nsproxy:reap-]
>     Debug(nsproxy): reaper checks pool exec
>     [07/Apr/2017:13:06:57][28009.7fa78bfff700][-nsproxy:reap-]
>     Debug(nsproxy): reaper sets timeout based on idle diff
>     1491567117.835940 of pool exec
>     [07/Apr/2017:13:06:57][28009.7fa78bfff700][-nsproxy:reap-]
>     Debug(nsproxy): reaper waits for cond with timeout 1491567117.835940
>     exec-7
>
>     % set result [ns_proxy eval $handle [list exec dd if=/dev/urandom
>     bs=1K count=23 | base64] 1]
>     [07/Apr/2017:13:07:09][28009.7fa7b760b700][-command-]
>     Debug(nsproxy): proxy send pool exec slave 28197: exec dd
>     if=/dev/urandom bs=1K count=23 | base64
>     could not wait for proxy "exec-7": timeout waiting for evaluation
>
>     % ns_proxy active exec
>     {handle exec-7 slave 28197 start 1491566829:518164 script {exec dd
>     if=/dev/urandom bs=1K count=23 | base64}}
>
>     % ns_proxy cleanup
>     [07/Apr/2017:13:07:26][28009.7fa7b760b700][-command-]
>     Debug(nsproxy): nsproxy [exec]: close slave 28197 (expire 1000 ms)
>     [07/Apr/2017:13:07:26][28009.7fa7b760b700][-command-]
>     Debug(nsproxy): set expire in 1000 ms for pool exec slave 28197
>     [07/Apr/2017:13:07:26][28009.7fa7b760b700][-command-]
>     Debug(nsproxy): nsproxy [exec]: slave 28197 closed
>     [07/Apr/2017:13:07:26][28009.7fa78bfff700][-nsproxy:reap-]
>     Debug(nsproxy): reaper run
>     [07/Apr/2017:13:07:26][28009.7fa78bfff700][-nsproxy:reap-]
>     Debug(nsproxy): reaper checks pool exec
>     [07/Apr/2017:13:07:26][28009.7fa78bfff700][-nsproxy:reap-]
>     Debug(nsproxy): reaper sets timeout based on idle diff
>     1491567146.613112 of pool exec
>     [07/Apr/2017:13:07:26][28009.7fa78bfff700][-nsproxy:reap-]
>     Debug(nsproxy): reaper waits for cond with timeout 1491567146.613112
>
>
>     Hanging test case using command (note: same command except with
>     slightly larger output):
>
>     |ns_proxy eval $handle [list exec dd if=/dev/urandom bs=1K count=24
>     | base64] 1|
>
>
>     % set handle [ns_proxy get exec]
>     [07/Apr/2017:13:08:13][28009.7fa7b760b700][-command-]
>     Debug(nsproxy): set expire in 300000 ms for pool exec slave 28968
>     [07/Apr/2017:13:08:13][28009.7fa7b760b700][-command-]
>     Debug(nsproxy): nsproxy: slave 28968 started
>     [07/Apr/2017:13:08:13][28968.7f0059a9e700][-main-] Notice: OpenSSL
>     1.0.1t  3 May 2016 initialized
>     [07/Apr/2017:13:08:13][28009.7fa78bfff700][-nsproxy:reap-]
>     Debug(nsproxy): reaper run
>     [07/Apr/2017:13:08:13][28009.7fa78bfff700][-nsproxy:reap-]
>     Debug(nsproxy): reaper checks pool exec
>     [07/Apr/2017:13:08:13][28009.7fa78bfff700][-nsproxy:reap-]
>     Debug(nsproxy): reaper sets timeout based on idle diff
>     1491567193.223279 of pool exec
>     [07/Apr/2017:13:08:13][28009.7fa78bfff700][-nsproxy:reap-]
>     Debug(nsproxy): reaper waits for cond with timeout 1491567193.223279
>     exec-7
>
>     % set result [ns_proxy eval $handle [list exec dd if=/dev/urandom
>     bs=1K count=24 | base64] 1]
>     [07/Apr/2017:13:08:21][28009.7fa7b760b700][-command-]
>     Debug(nsproxy): proxy send pool exec slave 28968: exec dd
>     if=/dev/urandom bs=1K count=24 | base64
>     could not wait for proxy "exec-7": timeout waiting for evaluation
>
>     % ns_proxy active exec
>     {handle exec-7 slave 28968 start 1491566901:234806 script {exec dd
>     if=/dev/urandom bs=1K count=24 | base64}}
>
>     % ns_proxy cleanup
>     [07/Apr/2017:13:08:40][28009.7fa7b760b700][-command-]
>     Debug(nsproxy): nsproxy [exec]: close slave 28968 (expire 1000 ms)
>     [07/Apr/2017:13:08:40][28009.7fa7b760b700][-command-]
>     Debug(nsproxy): set expire in 1000 ms for pool exec slave 28968
>     [07/Apr/2017:13:08:40][28009.7fa7b760b700][-command-]
>     Debug(nsproxy): nsproxy [exec]: slave 28968 closed
>     [07/Apr/2017:13:08:40][28009.7fa78bfff700][-nsproxy:reap-]
>     Debug(nsproxy): reaper run
>     [07/Apr/2017:13:08:40][28009.7fa78bfff700][-nsproxy:reap-]
>     Debug(nsproxy): reaper checks pool exec
>     [07/Apr/2017:13:08:40][28009.7fa78bfff700][-nsproxy:reap-]
>     Debug(nsproxy): reaper sets timeout based on idle diff
>     1491567220.342132 of pool exec
>
>     ns_proxy cleanup never returns.
>
>
>     The location of the hang is during waitpid() called via
>     Ns_WaitProcess
>     <https://bitbucket.org/naviserver/naviserver/src/5b852119ea5f3e3e796eb9385c22162a070a6b24/nsproxy/nsproxylib.c?at=default&fileviewer=file-view-default#nsproxylib.c-2935>
>
>     The system seems to think that the process hasn't exited.
>     Any ideas what is going on here?
>
>
>     Regards,
>     -- 
>     David
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel


-- 
Univ.Prof. Dr. Gustaf Neumann
WU Vienna
Institute of Information Systems and New Media
Welthandelsplatz 1, A-1020 Vienna, Austria

Re: [naviserver-devel] ns_proxy hang

[David O. <da...@qc...>]

Has anyone got any ideas on debugging this issue?
Turns out it's causing us many more operational issues that we first
thought...

Regards,
-- 
David

On 18 April 2017 at 15:44, David Osborne <da...@qc...> wrote:

> Hi,
>
> We're encountering an occasional problem which means calls to ns_proxy
> hang forever.
> It seems to be when an ns_proxy eval command timeouts during evaluation &
> it produces a large amount of output.
> Under those circumstances ns_proxy becomes inoperable.
>
> Following is a contrived test case which I think demonstrates what we are
> experiencing.
> (The ns_proxy timeout here is set unrealistically small to force the
> timeout - in production the timeout is 1 second)
>
>
> Normal test case using command:
>
> ns_proxy eval $handle [list exec dd if=/dev/urandom bs=1K count=23 | base64] 1
>
>
> % set handle [ns_proxy get exec]
> [07/Apr/2017:13:06:57][28009.7fa7b760b700][-command-] Debug(nsproxy): set
> expire in 300000 ms for pool exec slave 28197
> [07/Apr/2017:13:06:57][28009.7fa7b760b700][-command-] Debug(nsproxy):
> nsproxy: slave 28197 started
> [07/Apr/2017:13:06:57][28197.7f3f5dcc3700][-main-] Notice: OpenSSL 1.0.1t
>  3 May 2016 initialized
> [07/Apr/2017:13:06:57][28009.7fa78bfff700][-nsproxy:reap-] Notice:
> starting
> [07/Apr/2017:13:06:57][28009.7fa78bfff700][-nsproxy:reap-]
> Debug(nsproxy): reaper run
> [07/Apr/2017:13:06:57][28009.7fa78bfff700][-nsproxy:reap-]
> Debug(nsproxy): reaper checks pool exec
> [07/Apr/2017:13:06:57][28009.7fa78bfff700][-nsproxy:reap-]
> Debug(nsproxy): reaper sets timeout based on idle diff 1491567117.835940 of
> pool exec
> [07/Apr/2017:13:06:57][28009.7fa78bfff700][-nsproxy:reap-]
> Debug(nsproxy): reaper waits for cond with timeout 1491567117.835940
> exec-7
>
> % set result [ns_proxy eval $handle [list exec dd if=/dev/urandom bs=1K
> count=23 | base64] 1]
> [07/Apr/2017:13:07:09][28009.7fa7b760b700][-command-] Debug(nsproxy):
> proxy send pool exec slave 28197: exec dd if=/dev/urandom bs=1K count=23 |
> base64
> could not wait for proxy "exec-7": timeout waiting for evaluation
>
> % ns_proxy active exec
> {handle exec-7 slave 28197 start 1491566829:518164 script {exec dd
> if=/dev/urandom bs=1K count=23 | base64}}
>
> % ns_proxy cleanup
> [07/Apr/2017:13:07:26][28009.7fa7b760b700][-command-] Debug(nsproxy):
> nsproxy [exec]: close slave 28197 (expire 1000 ms)
> [07/Apr/2017:13:07:26][28009.7fa7b760b700][-command-] Debug(nsproxy): set
> expire in 1000 ms for pool exec slave 28197
> [07/Apr/2017:13:07:26][28009.7fa7b760b700][-command-] Debug(nsproxy):
> nsproxy [exec]: slave 28197 closed
> [07/Apr/2017:13:07:26][28009.7fa78bfff700][-nsproxy:reap-]
> Debug(nsproxy): reaper run
> [07/Apr/2017:13:07:26][28009.7fa78bfff700][-nsproxy:reap-]
> Debug(nsproxy): reaper checks pool exec
> [07/Apr/2017:13:07:26][28009.7fa78bfff700][-nsproxy:reap-]
> Debug(nsproxy): reaper sets timeout based on idle diff 1491567146.613112 of
> pool exec
> [07/Apr/2017:13:07:26][28009.7fa78bfff700][-nsproxy:reap-]
> Debug(nsproxy): reaper waits for cond with timeout 1491567146.613112
>
>
> Hanging test case using command (note: same command except with slightly
> larger output):
>
> ns_proxy eval $handle [list exec dd if=/dev/urandom bs=1K count=24 | base64] 1
>
>
> % set handle [ns_proxy get exec]
> [07/Apr/2017:13:08:13][28009.7fa7b760b700][-command-] Debug(nsproxy): set
> expire in 300000 ms for pool exec slave 28968
> [07/Apr/2017:13:08:13][28009.7fa7b760b700][-command-] Debug(nsproxy):
> nsproxy: slave 28968 started
> [07/Apr/2017:13:08:13][28968.7f0059a9e700][-main-] Notice: OpenSSL 1.0.1t
>  3 May 2016 initialized
> [07/Apr/2017:13:08:13][28009.7fa78bfff700][-nsproxy:reap-]
> Debug(nsproxy): reaper run
> [07/Apr/2017:13:08:13][28009.7fa78bfff700][-nsproxy:reap-]
> Debug(nsproxy): reaper checks pool exec
> [07/Apr/2017:13:08:13][28009.7fa78bfff700][-nsproxy:reap-]
> Debug(nsproxy): reaper sets timeout based on idle diff 1491567193.223279 of
> pool exec
> [07/Apr/2017:13:08:13][28009.7fa78bfff700][-nsproxy:reap-]
> Debug(nsproxy): reaper waits for cond with timeout 1491567193.223279
> exec-7
>
> % set result [ns_proxy eval $handle [list exec dd if=/dev/urandom bs=1K
> count=24 | base64] 1]
> [07/Apr/2017:13:08:21][28009.7fa7b760b700][-command-] Debug(nsproxy):
> proxy send pool exec slave 28968: exec dd if=/dev/urandom bs=1K count=24 |
> base64
> could not wait for proxy "exec-7": timeout waiting for evaluation
>
> % ns_proxy active exec
> {handle exec-7 slave 28968 start 1491566901:234806 script {exec dd
> if=/dev/urandom bs=1K count=24 | base64}}
>
> % ns_proxy cleanup
> [07/Apr/2017:13:08:40][28009.7fa7b760b700][-command-] Debug(nsproxy):
> nsproxy [exec]: close slave 28968 (expire 1000 ms)
> [07/Apr/2017:13:08:40][28009.7fa7b760b700][-command-] Debug(nsproxy): set
> expire in 1000 ms for pool exec slave 28968
> [07/Apr/2017:13:08:40][28009.7fa7b760b700][-command-] Debug(nsproxy):
> nsproxy [exec]: slave 28968 closed
> [07/Apr/2017:13:08:40][28009.7fa78bfff700][-nsproxy:reap-]
> Debug(nsproxy): reaper run
> [07/Apr/2017:13:08:40][28009.7fa78bfff700][-nsproxy:reap-]
> Debug(nsproxy): reaper checks pool exec
> [07/Apr/2017:13:08:40][28009.7fa78bfff700][-nsproxy:reap-]
> Debug(nsproxy): reaper sets timeout based on idle diff 1491567220.342132 of
> pool exec
>
> ns_proxy cleanup never returns.
>
>
> The location of the hang is during waitpid() called via Ns_WaitProcess
> <https://bitbucket.org/naviserver/naviserver/src/5b852119ea5f3e3e796eb9385c22162a070a6b24/nsproxy/nsproxylib.c?at=default&fileviewer=file-view-default#nsproxylib.c-2935>
>
> The system seems to think that the process hasn't exited.
> Any ideas what is going on here?
>
>
> Regards,
> --
> David
>
>

[naviserver-devel] ns_proxy hang

[David O. <da...@qc...>]

Hi,

We're encountering an occasional problem which means calls to ns_proxy hang
forever.
It seems to be when an ns_proxy eval command timeouts during evaluation &
it produces a large amount of output.
Under those circumstances ns_proxy becomes inoperable.

Following is a contrived test case which I think demonstrates what we are
experiencing.
(The ns_proxy timeout here is set unrealistically small to force the
timeout - in production the timeout is 1 second)


Normal test case using command:

ns_proxy eval $handle [list exec dd if=/dev/urandom bs=1K count=23 | base64] 1


% set handle [ns_proxy get exec]
[07/Apr/2017:13:06:57][28009.7fa7b760b700][-command-] Debug(nsproxy): set
expire in 300000 ms for pool exec slave 28197
[07/Apr/2017:13:06:57][28009.7fa7b760b700][-command-] Debug(nsproxy):
nsproxy: slave 28197 started
[07/Apr/2017:13:06:57][28197.7f3f5dcc3700][-main-] Notice: OpenSSL 1.0.1t
 3 May 2016 initialized
[07/Apr/2017:13:06:57][28009.7fa78bfff700][-nsproxy:reap-] Notice: starting
[07/Apr/2017:13:06:57][28009.7fa78bfff700][-nsproxy:reap-] Debug(nsproxy):
reaper run
[07/Apr/2017:13:06:57][28009.7fa78bfff700][-nsproxy:reap-] Debug(nsproxy):
reaper checks pool exec
[07/Apr/2017:13:06:57][28009.7fa78bfff700][-nsproxy:reap-] Debug(nsproxy):
reaper sets timeout based on idle diff 1491567117.835940 of pool exec
[07/Apr/2017:13:06:57][28009.7fa78bfff700][-nsproxy:reap-] Debug(nsproxy):
reaper waits for cond with timeout 1491567117.835940
exec-7

% set result [ns_proxy eval $handle [list exec dd if=/dev/urandom bs=1K
count=23 | base64] 1]
[07/Apr/2017:13:07:09][28009.7fa7b760b700][-command-] Debug(nsproxy): proxy
send pool exec slave 28197: exec dd if=/dev/urandom bs=1K count=23 | base64
could not wait for proxy "exec-7": timeout waiting for evaluation

% ns_proxy active exec
{handle exec-7 slave 28197 start 1491566829:518164 script {exec dd
if=/dev/urandom bs=1K count=23 | base64}}

% ns_proxy cleanup
[07/Apr/2017:13:07:26][28009.7fa7b760b700][-command-] Debug(nsproxy):
nsproxy [exec]: close slave 28197 (expire 1000 ms)
[07/Apr/2017:13:07:26][28009.7fa7b760b700][-command-] Debug(nsproxy): set
expire in 1000 ms for pool exec slave 28197
[07/Apr/2017:13:07:26][28009.7fa7b760b700][-command-] Debug(nsproxy):
nsproxy [exec]: slave 28197 closed
[07/Apr/2017:13:07:26][28009.7fa78bfff700][-nsproxy:reap-] Debug(nsproxy):
reaper run
[07/Apr/2017:13:07:26][28009.7fa78bfff700][-nsproxy:reap-] Debug(nsproxy):
reaper checks pool exec
[07/Apr/2017:13:07:26][28009.7fa78bfff700][-nsproxy:reap-] Debug(nsproxy):
reaper sets timeout based on idle diff 1491567146.613112 of pool exec
[07/Apr/2017:13:07:26][28009.7fa78bfff700][-nsproxy:reap-] Debug(nsproxy):
reaper waits for cond with timeout 1491567146.613112


Hanging test case using command (note: same command except with slightly
larger output):

ns_proxy eval $handle [list exec dd if=/dev/urandom bs=1K count=24 | base64] 1


% set handle [ns_proxy get exec]
[07/Apr/2017:13:08:13][28009.7fa7b760b700][-command-] Debug(nsproxy): set
expire in 300000 ms for pool exec slave 28968
[07/Apr/2017:13:08:13][28009.7fa7b760b700][-command-] Debug(nsproxy):
nsproxy: slave 28968 started
[07/Apr/2017:13:08:13][28968.7f0059a9e700][-main-] Notice: OpenSSL 1.0.1t
 3 May 2016 initialized
[07/Apr/2017:13:08:13][28009.7fa78bfff700][-nsproxy:reap-] Debug(nsproxy):
reaper run
[07/Apr/2017:13:08:13][28009.7fa78bfff700][-nsproxy:reap-] Debug(nsproxy):
reaper checks pool exec
[07/Apr/2017:13:08:13][28009.7fa78bfff700][-nsproxy:reap-] Debug(nsproxy):
reaper sets timeout based on idle diff 1491567193.223279 of pool exec
[07/Apr/2017:13:08:13][28009.7fa78bfff700][-nsproxy:reap-] Debug(nsproxy):
reaper waits for cond with timeout 1491567193.223279
exec-7

% set result [ns_proxy eval $handle [list exec dd if=/dev/urandom bs=1K
count=24 | base64] 1]
[07/Apr/2017:13:08:21][28009.7fa7b760b700][-command-] Debug(nsproxy): proxy
send pool exec slave 28968: exec dd if=/dev/urandom bs=1K count=24 | base64
could not wait for proxy "exec-7": timeout waiting for evaluation

% ns_proxy active exec
{handle exec-7 slave 28968 start 1491566901:234806 script {exec dd
if=/dev/urandom bs=1K count=24 | base64}}

% ns_proxy cleanup
[07/Apr/2017:13:08:40][28009.7fa7b760b700][-command-] Debug(nsproxy):
nsproxy [exec]: close slave 28968 (expire 1000 ms)
[07/Apr/2017:13:08:40][28009.7fa7b760b700][-command-] Debug(nsproxy): set
expire in 1000 ms for pool exec slave 28968
[07/Apr/2017:13:08:40][28009.7fa7b760b700][-command-] Debug(nsproxy):
nsproxy [exec]: slave 28968 closed
[07/Apr/2017:13:08:40][28009.7fa78bfff700][-nsproxy:reap-] Debug(nsproxy):
reaper run
[07/Apr/2017:13:08:40][28009.7fa78bfff700][-nsproxy:reap-] Debug(nsproxy):
reaper checks pool exec
[07/Apr/2017:13:08:40][28009.7fa78bfff700][-nsproxy:reap-] Debug(nsproxy):
reaper sets timeout based on idle diff 1491567220.342132 of pool exec

ns_proxy cleanup never returns.


The location of the hang is during waitpid() called via Ns_WaitProcess
<https://bitbucket.org/naviserver/naviserver/src/5b852119ea5f3e3e796eb9385c22162a070a6b24/nsproxy/nsproxylib.c?at=default&fileviewer=file-view-default#nsproxylib.c-2935>

The system seems to think that the process hasn't exited.
Any ideas what is going on here?


Regards,
-- 
David

Re: [naviserver-devel] ns_chan and ns_connchan

[Gustaf N. <ne...@wu...>]

> Am 07.04.17 um 09:43 schrieb Wolfgang Winkler:
>> We have some long running scripts, e.g. shrinking of large PDF files,
>> and want to prevent reverse proxy and browser timeouts.
>> To achieve this, we are trying to periodically send small packages
>> from the server to the browser while these scripts are running.
Dear Wolfgang,

maybe you find the following snippet useful, which obtains a pdf file 
from an upstream server
via HTTPS with timeouts in the background. It uses the actual version of 
the
revproxy module and NaviServer, but does does not handle redirects etc. 
which
could be added to a specialized version of that backendReply handler.

-g

===================================================================
set pdfUrl https://www.w3.org/Protocols/HTTP/1.1/rfc2616.pdf
set timeout 10.0

if {[catch {
     set queryHeaders [ns_conn headers]
     ns_set update $queryHeaders X-Forwarded-For [ns_conn peeraddr]
     ns_set update $queryHeaders Connection close
     
     set pdfChan [ns_connchan open \
		     -method GET \
		     -headers $queryHeaders \
		     -timeout $timeout \
		     -version 1.0 \
		     $pdfUrl]
} errorMsg]} {
     ns_return 408 text/plain "error while trying to obtain $pdfUrl\n($errorMsg)"
} else {
     set frontendChan [ns_connchan detach]
     ns_connchan callback -timeout $timeout $pdfChan  \
	[list ::revproxy::backendReply $pdfChan $frontendChan $pdfUrl $timeout 0] rex
}
===================================================================

Re: [naviserver-devel] revproxy callback setup

[Gustaf N. <ne...@wu...>]

Am 10.04.17 um 17:58 schrieb David Osborne:
> The order change is changing which callback is timing out first and 
> backendReply was triggered. It doesn't deal with the timeout properly, 
> it just immediately starts trying to read from the backend chan.
Thanks for the changes. The exception handling was just partially 
implemented. I've generalized the code based on your comments, and 
pushed the changes to bitbucket (but could do just marginal testing). -g

Re: [naviserver-devel] revproxy callback setup

[David O. <da...@qc...>]

This had a knock-on effect meaning the timeout stopped functioning for me.

The order change is changing which callback is timing out first and
backendReply
was triggered. It doesn't deal with the timeout properly, it just
immediately starts trying to read from the backend chan.

So adding timeout handling to backendReply like the following worked:
https://bitbucket.org/davidqc/revproxy/branches/compare/davidqc/revproxy:tip%0Dnaviserver/revproxy:tip#diff

Another option which worked was removing the timeout from the
backendReply callback
completely and deal with it all from within spool.

On 10 April 2017 at 14:29, David Osborne <da...@qc...> wrote:

> Hi,
>
> While testing the revproxy code I coming across the occasional error like
> the following:
>
> connchan "conn237" does not exist
>     while executing
> "ns_connchan  callback  -timeout $timeout  $frontendChan  [list  spool  $frontendChan  $backendChan  client  $timeout  0  ]  rex"
>
> Seems like sometimes the callback on the $backendChan can be created,
> triggered, and autoclose the $frontendChan before the callback creation
> on the $frontendChan is even created.
>
> This is not an error that affects the user since the request has been
> dealt with fully but the proxy code throws errors.
>
> I found swapping the two ns_connchan callback statements solves this eg.
> https://bitbucket.org/davidqc/revproxy/commits/
> 93180d1a66c82f3e87a39b69e4cbd25c9926c1f3
>
> --
> David
>

[naviserver-devel] revproxy callback setup

[David O. <da...@qc...>]

Hi,

While testing the revproxy code I coming across the occasional error like
the following:

connchan "conn237" does not exist
    while executing
"ns_connchan  callback  -timeout $timeout  $frontendChan  [list  spool
 $frontendChan  $backendChan  client  $timeout  0  ]  rex"

Seems like sometimes the callback on the $backendChan can be created,
triggered, and autoclose the $frontendChan before the callback creation on
the $frontendChan is even created.

This is not an error that affects the user since the request has been dealt
with fully but the proxy code throws errors.

I found swapping the two ns_connchan callback statements solves this eg.
https://bitbucket.org/davidqc/revproxy/commits/93180d1a66c82f3e87a39b69e4cbd25c9926c1f3

-- 
David

Re: [naviserver-devel] ns_chan and ns_connchan

[Gustaf N. <ne...@wu...>]

Am 07.04.17 um 09:43 schrieb Wolfgang Winkler:
> We have some long running scripts, e.g. shrinking of large PDF files, 
> and want to prevent reverse proxy and browser timeouts.
> To achieve this, we are trying to periodically send small packages 
> from the server to the browser while these scripts are running.
Normally, the easiest approch is to "stream" the results back to the 
client, is to start the result page with "ns_headers" and send partial 
results via "ns_write", i.e. something in the lines of:

    ns_headers -binary 200 application/pdf
    ...
    ns_write ...
    ...

But maybe you are doing such an approach already, but you want to spool 
in the background using "ns_conn detach", but you have not used the 
"-binary" flag for ns_headers?

-gn

Re: [naviserver-devel] ns_chan and ns_connchan

[Gustaf N. <ne...@wu...>]

Am 07.04.17 um 10:57 schrieb David Osborne:
> Re the first issue, I only have experience of ns_connchan but I 
> believe the situation is the same.
> You can specify r,w,e, or x for the *argument* ?when? upon creation of 
> the callback, but the callback will be triggered upon timeout with a 
>  *value* of $when = "t" regardless of what you specify as the argument 
> when setting up the callback.
>
> "The optional argument /when/ can consist of one or more characters of 
> r, w, e, or x, specifying, when the callback should fire. "

I've updated the man-page to make the distinction between the input 
parameter "when" and the reported "condition" more clear.
https://bitbucket.org/naviserver/naviserver/commits/c29128a5a4d3bfdadfd11de28d276144bcaea727

-g

Re: [naviserver-devel] revproxy timeouts

[David O. <da...@qc...>]

Spot on - that works great - thanks!

On 7 April 2017 at 16:01, Gustaf Neumann <ne...@wu...> wrote:

> Hi David,
>
> thanks for testing!
> I think, i have fixed the problem with:
>
>     https://bitbucket.org/naviserver/naviserver/commits/
> 221d0c8f629bb29235e156a781ca5599284c3e9d
>
> The bug was a problem in nsssl triggered by "ns_connchan close".
> The same verison of revproxy should work.
>
> all the best
> -g
>
>
> Am 06.04.17 um 12:21 schrieb David Osborne:
>
> Hi Gustaf,
>
> I can confirm that if I switch to using http (rather than https) then your
> timeout modification works perfectly.
> So looking like it's (at least) ssl specific,
>
> On 5 April 2017 at 18:03, Gustaf Neumann <ne...@wu...> wrote:
>
>> Am 05.04.17 um 18:45 schrieb David Osborne:
>> > So although the timeout has occurred in the proxy code, the connection
>> > appears to still have to wait for the backend to close the connection
>> > before continuing.
>> on my test setup, everything is closed correctly. this is either
>> ssl-specific, or linux-specific. will check.
>> -gn
>>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>


-- 
David Osborne
Qcode Software Limited
http://www.qcode.co.uk
T: +44 (0)1463 896484

Re: [naviserver-devel] revproxy timeouts

[Gustaf N. <ne...@wu...>]

Hi David,

thanks for testing!
I think, i have fixed the problem with:

https://bitbucket.org/naviserver/naviserver/commits/221d0c8f629bb29235e156a781ca5599284c3e9d

The bug was a problem in nsssl triggered by "ns_connchan close".
The same verison of revproxy should work.

all the best
-g


Am 06.04.17 um 12:21 schrieb David Osborne:
> Hi Gustaf,
>
> I can confirm that if I switch to using http (rather than https) then 
> your timeout modification works perfectly.
> So looking like it's (at least) ssl specific,
>
> On 5 April 2017 at 18:03, Gustaf Neumann <ne...@wu... 
> <mailto:ne...@wu...>> wrote:
>
>     Am 05.04.17 um 18:45 schrieb David Osborne:
>     > So although the timeout has occurred in the proxy code, the connection
>     > appears to still have to wait for the backend to close the
>     connection
>     > before continuing.
>     on my test setup, everything is closed correctly. this is either
>     ssl-specific, or linux-specific. will check.
>     -gn
>

Re: [naviserver-devel] ns_chan and ns_connchan

[David O. <da...@qc...>]

We seem to be able to write happily to a connchan channel directly using a
string representation as follows eg.

 ns_connchan write $to "HTTP/1.0 504 Gateway Timout\r\n\r\n"

This is a guess, but could you be forcing an implicit type conversion of
your string by manipulating it with non-bytearray aware commands?

eg.

if { $content ne "" } {
will force an internal string representation of $content

but
if { [string length $content] > 0 } {
will not


On 7 April 2017 at 10:06, Wolfgang Winkler <wolfgang.winkler@digital-conc
epts.com> wrote:

> I'm trying to write out html code or javascript code fragments to show the
> progress of the operations.
>
> Am 2017-04-07 um 10:57 schrieb David Osborne:
>
> Re the first issue, I only have experience of ns_connchan but I believe
> the situation is the same.
> You can specify r,w,e, or x for the *argument* ?when? upon creation of the
> callback, but the callback will be triggered upon timeout with a  *value*
> of $when = "t" regardless of what you specify as the argument when setting
> up the callback.
>
> "The optional argument *when* can consist of one or more characters of r,
> w, e, or x, specifying, when the callback should fire. "
>
> Re the binary problem, What is it you are writing to the channel?
>
> On 7 April 2017 at 08:43, Wolfgang Winkler <wolfgang.winkler@digital-conc
> epts.com> wrote:
>
>> Hi!
>>
>> We have some long running scripts, e.g. shrinking of large PDF files, and
>> want to prevent reverse proxy and browser timeouts.
>>
>> To achieve this, we are trying to periodically send small packages from
>> the server to the browser while these scripts are running.
>>
>> First we tried with ns_conn and ns_sockcallback
>>
>> set mychan [ns_conn channel]
>>
>> ns_sockcallback $mychan noop t 1
>>
>> proc noop {handle when} {
>>   # do something
>> }
>>
>> ns_chan close $mychan
>>
>> When we call this, we get the following error:
>>
>> error invalid when specification "t": should be one/more of r, w, e, or x
>>
>> which contradicts the documentation of the command here:
>>
>> https://naviserver.sourceforge.io/n/naviserver/files/ns_sockcallback.html
>>
>> Then we tried something similiar with ns_connchan
>>
>> set mychan [ns_connchan detach]
>>
>> ns_connchan callback -timeout 1 $mychan noop "r"
>>
>> but a ns_connchan write $mychan throws the following error:
>>
>> ns_connchan: only binary channels are currently supported. Channel conn0
>> is not binary
>>
>> Is there a solution to this problem?
>>
>> regards,
>>
>> Wolfgang
>>
>> --
>>
>> *Wolfgang Winkler*
>> Geschäftsführung
>> wol...@di...
>> mobil +43.699.19971172 <+43%20699%2019971172>
>>
>> dc:*büro*
>> digital concepts Novak Winkler OG
>> Software & Design
>> Landstraße 68, 5. Stock, 4020 Linz
>> www.digital-concepts.com
>> tel +43.732.997117.72
>> tel +43.699.1997117.2
>>
>> Firmenbuchnummer: 192003h
>> Firmenbuchgericht: Landesgericht Linz
>>
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>>
>>
>
>
> --
> David Osborne
> Qcode Software Limited
> http://www.qcode.co.uk
> T: +44 (0)1463 896484 <01463%20896484>
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> _______________________________________________
> naviserver-devel mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>
> --
>
> *Wolfgang Winkler*
> Geschäftsführung
> wol...@di...
> mobil +43.699.19971172 <+43%20699%2019971172>
>
> dc:*büro*
> digital concepts Novak Winkler OG
> Software & Design
> Landstraße 68, 5. Stock, 4020 Linz
> www.digital-concepts.com
> tel +43.732.997117.72
> tel +43.699.1997117.2
>
> Firmenbuchnummer: 192003h
> Firmenbuchgericht: Landesgericht Linz
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>


-- 
David Osborne
Qcode Software Limited
http://www.qcode.co.uk
T: +44 (0)1463 896484 <01463%20896484>

Re: [naviserver-devel] ns_chan and ns_connchan

[Wolfgang W. <wol...@di...>]

I'm trying to write out html code or javascript code fragments to show 
the progress of the operations.


Am 2017-04-07 um 10:57 schrieb David Osborne:
> Re the first issue, I only have experience of ns_connchan but I 
> believe the situation is the same.
> You can specify r,w,e, or x for the *argument* ?when? upon creation of 
> the callback, but the callback will be triggered upon timeout with a 
>  *value* of $when = "t" regardless of what you specify as the argument 
> when setting up the callback.
>
> "The optional argument /when/ can consist of one or more characters of 
> r, w, e, or x, specifying, when the callback should fire. "
>
> Re the binary problem, What is it you are writing to the channel?
>
> On 7 April 2017 at 08:43, Wolfgang Winkler 
> <wol...@di... 
> <mailto:wol...@di...>> wrote:
>
>     Hi!
>
>     We have some long running scripts, e.g. shrinking of large PDF
>     files, and want to prevent reverse proxy and browser timeouts.
>
>     To achieve this, we are trying to periodically send small packages
>     from the server to the browser while these scripts are running.
>
>     First we tried with ns_conn and ns_sockcallback
>
>         set mychan [ns_conn channel]
>
>         ns_sockcallback $mychan noop t 1
>
>         proc noop {handle when} {
>           # do something
>         }
>
>         ns_chan close $mychan
>
>     When we call this, we get the following error:
>
>         error invalid when specification "t": should be one/more of r,
>         w, e, or x
>
>     which contradicts the documentation of the command here:
>
>     https://naviserver.sourceforge.io/n/naviserver/files/ns_sockcallback.html
>     <https://naviserver.sourceforge.io/n/naviserver/files/ns_sockcallback.html>
>
>     Then we tried something similiar with ns_connchan
>
>     set mychan [ns_connchan detach]
>
>         ns_connchan callback -timeout 1 $mychan noop "r"
>
>     but a ns_connchan write $mychan throws the following error:
>
>         ns_connchan: only binary channels are currently supported.
>         Channel conn0 is not binary
>
>     Is there a solution to this problem?
>
>     regards,
>
>     Wolfgang
>
>
>     -- 
>
>     *Wolfgang Winkler*
>     Geschäftsführung
>     wol...@di...
>     <mailto:wol...@di...>
>     mobil +43.699.19971172 <tel:+43%20699%2019971172>
>
>     dc:*büro*
>     digital concepts Novak Winkler OG
>     Software & Design
>     Landstraße 68, 5. Stock, 4020 Linz
>     www.digital-concepts.com <http://www.digital-concepts.com>
>     tel +43.732.997117.72
>     tel +43.699.1997117.2
>
>     Firmenbuchnummer: 192003h
>     Firmenbuchgericht: Landesgericht Linz
>
>
>
>     ------------------------------------------------------------------------------
>     Check out the vibrant tech community on one of the world's most
>     engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>     _______________________________________________
>     naviserver-devel mailing list
>     nav...@li...
>     <mailto:nav...@li...>
>     https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>     <https://lists.sourceforge.net/lists/listinfo/naviserver-devel>
>
>
>
>
> -- 
> David Osborne
> Qcode Software Limited
> http://www.qcode.co.uk
> T: +44 (0)1463 896484
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] ns_chan and ns_connchan

[David O. <da...@qc...>]

Re the first issue, I only have experience of ns_connchan but I believe the
situation is the same.
You can specify r,w,e, or x for the *argument* ?when? upon creation of the
callback, but the callback will be triggered upon timeout with a  *value*
of $when = "t" regardless of what you specify as the argument when setting
up the callback.

"The optional argument *when* can consist of one or more characters of r,
w, e, or x, specifying, when the callback should fire. "

Re the binary problem, What is it you are writing to the channel?

On 7 April 2017 at 08:43, Wolfgang Winkler <
wol...@di...> wrote:

> Hi!
>
> We have some long running scripts, e.g. shrinking of large PDF files, and
> want to prevent reverse proxy and browser timeouts.
>
> To achieve this, we are trying to periodically send small packages from
> the server to the browser while these scripts are running.
>
> First we tried with ns_conn and ns_sockcallback
>
> set mychan [ns_conn channel]
>
> ns_sockcallback $mychan noop t 1
>
> proc noop {handle when} {
>   # do something
> }
>
> ns_chan close $mychan
>
> When we call this, we get the following error:
>
> error invalid when specification "t": should be one/more of r, w, e, or x
>
> which contradicts the documentation of the command here:
>
> https://naviserver.sourceforge.io/n/naviserver/files/ns_sockcallback.html
>
> Then we tried something similiar with ns_connchan
>
> set mychan [ns_connchan detach]
>
> ns_connchan callback -timeout 1 $mychan noop "r"
>
> but a ns_connchan write $mychan throws the following error:
>
> ns_connchan: only binary channels are currently supported. Channel conn0
> is not binary
>
> Is there a solution to this problem?
>
> regards,
>
> Wolfgang
>
> --
>
> *Wolfgang Winkler*
> Geschäftsführung
> wol...@di...
> mobil +43.699.19971172 <+43%20699%2019971172>
>
> dc:*büro*
> digital concepts Novak Winkler OG
> Software & Design
> Landstraße 68, 5. Stock, 4020 Linz
> www.digital-concepts.com
> tel +43.732.997117.72
> tel +43.699.1997117.2
>
> Firmenbuchnummer: 192003h
> Firmenbuchgericht: Landesgericht Linz
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>


-- 
David Osborne
Qcode Software Limited
http://www.qcode.co.uk
T: +44 (0)1463 896484

[naviserver-devel] ns_chan and ns_connchan

[Wolfgang W. <wol...@di...>]

Hi!

We have some long running scripts, e.g. shrinking of large PDF files, 
and want to prevent reverse proxy and browser timeouts.

To achieve this, we are trying to periodically send small packages from 
the server to the browser while these scripts are running.

First we tried with ns_conn and ns_sockcallback

    set mychan [ns_conn channel]

    ns_sockcallback $mychan noop t 1

    proc noop {handle when} {
       # do something
    }

    ns_chan close $mychan

When we call this, we get the following error:

    error invalid when specification "t": should be one/more of r, w, e,
    or x

which contradicts the documentation of the command here:

https://naviserver.sourceforge.io/n/naviserver/files/ns_sockcallback.html

Then we tried something similiar with ns_connchan

set mychan [ns_connchan detach]

    ns_connchan callback -timeout 1 $mychan noop "r"

but a ns_connchan write $mychan throws the following error:

    ns_connchan: only binary channels are currently supported. Channel
    conn0 is not binary

Is there a solution to this problem?

regards,

Wolfgang


-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] revproxy timeouts

[David O. <da...@qc...>]

Hi Gustaf,

I can confirm that if I switch to using http (rather than https) then your
timeout modification works perfectly.
So looking like it's (at least) ssl specific,

On 5 April 2017 at 18:03, Gustaf Neumann <ne...@wu...> wrote:

> Am 05.04.17 um 18:45 schrieb David Osborne:
> > So although the timeout has occurred in the proxy code, the connection
> > appears to still have to wait for the backend to close the connection
> > before continuing.
> on my test setup, everything is closed correctly. this is either
> ssl-specific, or linux-specific. will check.
> -gn
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] revproxy timeouts

[Gustaf N. <ne...@wu...>]

Am 05.04.17 um 18:45 schrieb David Osborne:
> So although the timeout has occurred in the proxy code, the connection 
> appears to still have to wait for the backend to close the connection 
> before continuing.
on my test setup, everything is closed correctly. this is either 
ssl-specific, or linux-specific. will check.
-gn

Re: [naviserver-devel] revproxy timeouts

[David O. <da...@qc...>]

Thanks very much.

That's similar to what I was trying (although I was setting the timeout for
the backendReply callback only since it should be the first thing to be
called on reply if I understand rightly).

I get the same behaviour with your code.
Say I request a backend URL that takes 10 seconds to reply, and the proxy
timeout is 5 seconds...

I see the following:

1. URL requested in browser.
2. Browser starts to spin waiting for reply
<after 5 seconds>
3. I see the timeout being logged in the proxy and the 504 write is
performed.
4. No change in browser continues to spin waiting for a reply
<after another 5 seconds>
5. The browser receives a 504 and completes the request.

So although the timeout has occurred in the proxy code, the connection
appears to still have to wait for the backend to close the connection
before continuing.

-- 
David