naviserver-devel — Developer discussion

Re: [naviserver-devel] ns_proxy revisited

[Zoran V. <zv...@ar...>]

On 14.09.2006, at 22:10, Stephen Deasey wrote:

>
>
> But now the initiating thread is tied up with an idle wait -- i.e. you
> can't intiate the job and have it run in the background.  This
> issolates the code externaly, but it requires resources (a thread) for
> each job you start.
>
> The ns_job command can handle detached jobs.

True. There you go... signals? Or when we make the Win port
some other magic Windows stuff?
Well, I'm open to suggestions...
Syntax?

>
> Per-pool, I guess, at least for hard limits. Otherwise each proxy
> could only handle one job.  You could get fancy and re-set soft limits
> for each job, for some proxy pool for general jobs, but that sounds
> like version 2...

Yes! Version 2... The most important thing for us at this moment
is to get a stable code.
I will put this all into an RFE so we don't forget it over time.

Re: [naviserver-devel] ns_proxy revisited

[Vlad S. <vl...@cr...>]

New command will be better and more clear

Zoran Vasiljevic wrote:
> On 14.09.2006, at 20:12, Zoran Vasiljevic wrote:
> 
>> A list of poolId/script. If we would expand the
>> [ns_proxy clear ?pool?]
>> to be
>> [ns_proxy clear ?pool ?handle??]
>> then you'd be able to:
>>
>>     ns_proxy clear pool pool-7
> 
> 
> AHHH! Help!
> 
> The [ns_proxy clear] actually clears proxies which
> are left in the pool and not ones currently running.
> What you'd need would be to clear the process that
> is running!
> Now, we either need new command OR we must use some
> ugly switches:
> 
>     ns_proxy clear ?-running? ?pool? ?handle?
> 
> or
> 
>     ns_proxy stop ?pool? ?handle?
> 
> Someting else?
> 
> Now what to do?
> 
> Zoran
> 
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] ns_proxy revisited

[Stephen D. <sd...@gm...>]

On 9/14/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 14.09.2006, at 21:52, Stephen Deasey wrote:
>
> >
> >
> > This code looks wrong:
> >
> >   /*
> >    * It is pain in the neck to get a satisfactory definition of
> >    * u_int_XX_t or uintXX_t as different OS'es do that in different
> >    * header files and sometimes even do not define such types at all.
> >    * We choose to define them ourselves here and stop the blues.
> >    */
> >
> >   typedef unsigned int   uint32;
> >   typedef unsigned short uint16;
> >
> >
> > What if int's are 64 bit?  There is this code in the Tcl example
> > extension:
> >
> >
> >   #ifdef HAVE_INTTYPES_H
> >   #   include <inttypes.h>
> >   typedef uint32_t sha_uint32_t;
> >   #else
> >   #   if ((1<<31)<0)
> >   typedef unsigned long sha_uint32_t;
> >   #   else
> >   typedef unsigned int sha_uint32_t;
> >   #   endif
> >   #endif
>
> I spend 3 days peeking all arround at three different platforms
> (Solaris, Linux, Macosx) and was not able to find a common denominator.
>
> The int's are 32 bits AFAIK. The long ints (or ptrs) could be 64 bit on
> 64 box.
>

Oh, OK.  So on an AMD64 box, pointers are 64bit, but int's are 32?
You need long int to get 64bit?

Re: [naviserver-devel] ns_proxy revisited

[Stephen D. <sd...@gm...>]

On 9/14/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 14.09.2006, at 21:46, Stephen Deasey wrote:
>
> > What are your rules for checking for a long running process?
> >
> > If the ns_proxy command is for isolating badly behaved code, it seems
> > to me there could usefully be a few more built in checks for bad
> > behaviour, rather than having to implement these checks manually.
>
> True.
>
> >
> > For example, how about a switch which defines the maximum running time
> > for the command? You can handle this now as Zoran points out by
> > waiting for the result with a timeout.  But this is awkward to handle
> > if you have a 'detached' proxy job.
>
> This can be done "indirectly":
>
>    set milliseconds 2000
>    set p [ns_proxy send $p some_broken_hanging_script]
>    if {[catch {ns_proxy wait $p $milliseconds}]} {
>       ns_proxy put $p
>    }
>
> Put it in a procedure and there you go.


But now the initiating thread is tied up with an idle wait -- i.e. you
can't intiate the job and have it run in the background.  This
issolates the code externaly, but it requires resources (a thread) for
each job you start.

The ns_job command can handle detached jobs.


> >
> > Explicitly passing the runtime-timeout would also eliminate some race
> > conditions. Let's say you start a job, then in some other thread get a
> > list of jobs including pid, figure out a job has taken too long, then
> > kill the pid.  The job could exit between the time you get the pid and
> > the time your decide to kill it. Pid's recycle (eventualy), and you
> > kill some completely unrelated proces.
> Heh... bad cruel world... but it isnt really that black really.
> Normally the pid wrapup should take some time...
>
> >
> > Runtime limits could be implemented with a timer and signal handler in
> > the proxy process.
>
> Or as above.
>
> >
> > One common bad behaviour other than lack of thread safety is memory
> > leaks.  How about some switches which set rlimits for the process,
> > maximum heap memory and such?  This would also be a nice safety
> > feature -- say you're converting images or movies uploaded by some
> > untrusted source and you want to ensure a flaw in your converter
> > library doesn't compromise your server.
> >
> > Maybe a chroot switch would be useful?
> >
>
> Yes! All this could be really useful. The only trouble is
> the syntax. How to pass this to the new process?
> Over the cmd line? Env array? Store this per pool (most probably)?
>

Per-pool, I guess, at least for hard limits. Otherwise each proxy
could only handle one job.  You could get fancy and re-set soft limits
for each job, for some proxy pool for general jobs, but that sounds
like version 2...

Re: [naviserver-devel] ns_proxy revisited

[Zoran V. <zv...@ar...>]

On 14.09.2006, at 20:12, Zoran Vasiljevic wrote:

>
> A list of poolId/script. If we would expand the
> [ns_proxy clear ?pool?]
> to be
> [ns_proxy clear ?pool ?handle??]
> then you'd be able to:
>
>     ns_proxy clear pool pool-7


AHHH! Help!

The [ns_proxy clear] actually clears proxies which
are left in the pool and not ones currently running.
What you'd need would be to clear the process that
is running!
Now, we either need new command OR we must use some
ugly switches:

    ns_proxy clear ?-running? ?pool? ?handle?

or

    ns_proxy stop ?pool? ?handle?

Someting else?

Now what to do?

Zoran

Re: [naviserver-devel] ns_proxy crash

[Zoran V. <zv...@ar...>]

On 14.09.2006, at 21:59, Stephen Deasey wrote:

> AtShutdown callback

Yes. That's it.
Ah. with that waiting mess I just wanted to make sure
the server will eventually exit even if I mess something
up in the reaper thread and this stucks somewhere.

Re: [naviserver-devel] ns_proxy revisited

[Zoran V. <zv...@ar...>]

On 14.09.2006, at 21:52, Stephen Deasey wrote:

>
>
> This code looks wrong:
>
>   /*
>    * It is pain in the neck to get a satisfactory definition of
>    * u_int_XX_t or uintXX_t as different OS'es do that in different
>    * header files and sometimes even do not define such types at all.
>    * We choose to define them ourselves here and stop the blues.
>    */
>
>   typedef unsigned int   uint32;
>   typedef unsigned short uint16;
>
>
> What if int's are 64 bit?  There is this code in the Tcl example  
> extension:
>
>
>   #ifdef HAVE_INTTYPES_H
>   #   include <inttypes.h>
>   typedef uint32_t sha_uint32_t;
>   #else
>   #   if ((1<<31)<0)
>   typedef unsigned long sha_uint32_t;
>   #   else
>   typedef unsigned int sha_uint32_t;
>   #   endif
>   #endif

I spend 3 days peeking all arround at three different platforms
(Solaris, Linux, Macosx) and was not able to find a common denominator.

The int's are 32 bits AFAIK. The long ints (or ptrs) could be 64 bit on
64 box.

My assumption seems to be mostly OK but, allright...
And what about unisgned short? Could this be 32bit
on 64 bit box? I doubt, but I better ask...


>
> ---------------------------------------------------------------------- 
> ---
> Using Tomcat but need to do more? Need to support web services,  
> security?
> Get stuff done quickly with pre-integrated technology to make your  
> job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache  
> Geronimo
> http://sel.as-us.falkag.net/sel? 
> cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] ns_proxy crash

[Stephen D. <sd...@gm...>]

On 9/13/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 13.09.2006, at 04:55, Vlad Seryakov wrote:
>
> > Had ns_proxy running, typed in nscp ns_shutdown and nsd crahsed
>
> Eh... this is true. In the attempt to cleanup as much as possible
> I prematurely cleaned a pool having references to it.
> Actually, pools still having references to them should not be
> touched and should rather leak.
> OTOH, at the shutdown, there should be no references as all
> threads should have already been exited. I guess at this point
> we have still some trouble...
>
> The immediate and not clean solution is just to skip such pools
> and let them leak. I'm emitting the log message in such case as
> the reminder:
>
> [13/Sep/2006:10:22:37][1132.2684415384][-main-] Warning: nsproxy:
> pool "test" not destroyed, has 1 active slaves
>
> I do not think it is possible to avoid this, after thinking about
> this for a while. But still, lets leave this here as we might be
> more clever in the future.
>

Currently you're cleaning up with a AtExit handler and manually
calculating a timeout to wait for proxies to exit.   I think you
really want an AtShutdown callback.  These already handle a
notification and then timeout sequence.

I think at the moment a clean exit will take shutdown-timeout +
proxy-shutdown-timeout, which might be surprising to people.

Re: [naviserver-devel] ns_proxy revisited

[Zoran V. <zv...@ar...>]

On 14.09.2006, at 21:46, Stephen Deasey wrote:

> What are your rules for checking for a long running process?
>
> If the ns_proxy command is for isolating badly behaved code, it seems
> to me there could usefully be a few more built in checks for bad
> behaviour, rather than having to implement these checks manually.

True.

>
> For example, how about a switch which defines the maximum running time
> for the command? You can handle this now as Zoran points out by
> waiting for the result with a timeout.  But this is awkward to handle
> if you have a 'detached' proxy job.

This can be done "indirectly":

   set milliseconds 2000
   set p [ns_proxy send $p some_broken_hanging_script]
   if {[catch {ns_proxy wait $p $milliseconds}]} {
      ns_proxy put $p
   }

Put it in a procedure and there you go.


>
> Explicitly passing the runtime-timeout would also eliminate some race
> conditions. Let's say you start a job, then in some other thread get a
> list of jobs including pid, figure out a job has taken too long, then
> kill the pid.  The job could exit between the time you get the pid and
> the time your decide to kill it. Pid's recycle (eventualy), and you
> kill some completely unrelated proces.
Heh... bad cruel world... but it isnt really that black really.
Normally the pid wrapup should take some time...

>
> Runtime limits could be implemented with a timer and signal handler in
> the proxy process.

Or as above.

>
> One common bad behaviour other than lack of thread safety is memory
> leaks.  How about some switches which set rlimits for the process,
> maximum heap memory and such?  This would also be a nice safety
> feature -- say you're converting images or movies uploaded by some
> untrusted source and you want to ensure a flaw in your converter
> library doesn't compromise your server.
>
> Maybe a chroot switch would be useful?
>

Yes! All this could be really useful. The only trouble is
the syntax. How to pass this to the new process?
Over the cmd line? Env array? Store this per pool (most probably)?

Re: [naviserver-devel] Relesase Build Problem

[Stephen D. <sd...@gm...>]

The tarball is for naviserver-4.99.1.

naviserver-4.99.2 hasn't been released yet and contains a lot of
changes, including the contrib directory.


On 9/13/06, Ian Harding <iha...@de...> wrote:
> I downloaded the tarballs from sourceforge, and there seem to be a
> couple things missing.
>
> The contrib directory is not present, and sample-config.tcl does not
> exist.  Other than that, things seem to go OK...
>
> -----------
>
> for i in contrib/nsd.tcl contrib/mimetypes.tcl sample-config.tcl; do \
>         /usr/local/naviserver-4.99.1/install-sh -c -m 644 $i
> /usr/local/ns/conf/                    ; \
> done
> cp: cannot stat `contrib/nsd.tcl': No such file or directory
> chmod: cannot access `/usr/local/ns/conf/#inst.29201#': No such file
> or director                    y
> mv: cannot stat `/usr/local/ns/conf/#inst.29201#': No such file or directory
> cp: cannot stat `contrib/mimetypes.tcl': No such file or directory
> chmod: cannot access `/usr/local/ns/conf/#inst.29208#': No such file
> or director                    y
> mv: cannot stat `/usr/local/ns/conf/#inst.29208#': No such file or directory
> cp: cannot stat `sample-config.tcl': No such file or directory
> chmod: cannot access `/usr/local/ns/conf/#inst.29215#': No such file
> or director                    y
> mv: cannot stat `/usr/local/ns/conf/#inst.29215#': No such file or directory
> /usr/local/naviserver-4.99.1/install-sh -c -m 644 contrib/index.adp
> /usr/local/n                    s/pages/
> cp: cannot stat `contrib/index.adp': No such file or directory
> chmod: cannot access `/usr/local/ns/pages/#inst.29222#': No such file
> or directo                    ry
> mv: cannot stat `/usr/local/ns/pages/#inst.29222#': No such file or directory
> /usr/local/naviserver-4.99.1/install-sh -c install-sh /usr/local/ns/bin/
> for i in nsthread nsd nssock nscgi nscp nslog nsperm nsdb nsdbtest; do \
>         (cd $i && make install) || exit 1; \
> done
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] ns_proxy revisited

[Stephen D. <sd...@gm...>]

On 9/14/06, Zoran Vasiljevic <zv...@ar...> wrote:
> Hi!
>
> Motivated by the Vlad's recent report, I took another
> tour arround the code and did yet-another cleanup phase.
> Amazing howmany things you can find when you revisit the
> code after some time...
>
> I have fixed numerous MT-safeness problems, removed the
> "-minslaves" config option and added new [ns_proxy pools]
> command. I also updated the test-suite to check more cases.
> Also the problem reported by Vlad should be fixed as well.
>
> Altogether, this thing now seems pretty usable :-)
>
> If you discover something what you can undoubtfully
> classify as a bug, plese tell me as soon as possible
> as I would like to get this code integrated in our product
> (having more testers is an obvious plus for us, hehe).
>


This code looks wrong:

  /*
   * It is pain in the neck to get a satisfactory definition of
   * u_int_XX_t or uintXX_t as different OS'es do that in different
   * header files and sometimes even do not define such types at all.
   * We choose to define them ourselves here and stop the blues.
   */

  typedef unsigned int   uint32;
  typedef unsigned short uint16;


What if int's are 64 bit?  There is this code in the Tcl example extension:


  #ifdef HAVE_INTTYPES_H
  #   include <inttypes.h>
  typedef uint32_t sha_uint32_t;
  #else
  #   if ((1<<31)<0)
  typedef unsigned long sha_uint32_t;
  #   else
  typedef unsigned int sha_uint32_t;
  #   endif
  #endif

Re: [naviserver-devel] ns_proxy revisited

[Stephen D. <sd...@gm...>]

What are your rules for checking for a long running process?

If the ns_proxy command is for isolating badly behaved code, it seems
to me there could usefully be a few more built in checks for bad
behaviour, rather than having to implement these checks manually.

For example, how about a switch which defines the maximum running time
for the command? You can handle this now as Zoran points out by
waiting for the result with a timeout.  But this is awkward to handle
if you have a 'detached' proxy job.

Explicitly passing the runtime-timeout would also eliminate some race
conditions. Let's say you start a job, then in some other thread get a
list of jobs including pid, figure out a job has taken too long, then
kill the pid.  The job could exit between the time you get the pid and
the time your decide to kill it. Pid's recycle (eventualy), and you
kill some completely unrelated proces.

Runtime limits could be implemented with a timer and signal handler in
the proxy process.

One common bad behaviour other than lack of thread safety is memory
leaks.  How about some switches which set rlimits for the process,
maximum heap memory and such?  This would also be a nice safety
feature -- say you're converting images or movies uploaded by some
untrusted source and you want to ensure a flaw in your converter
library doesn't compromise your server.

Maybe a chroot switch would be useful?



On 9/14/06, Vlad Seryakov <vl...@cr...> wrote:
> Can i do this from other thread, spawn in one and check in another,
> actually this is what i need. I can have multiple processes running,
> then according to some rules i will check which one is taking too long
> and kill it.
>
> Zoran Vasiljevic wrote:
> > On 14.09.2006, at 19:50, Vlad Seryakov wrote:
> >
> >> The only immediate request i would have for ns_proxy to be able to
> >> query
> >> pid for active proxies, may be ns_proxy active could return pid or
> >> other
> >> command, that will be useful for forcing process exit if necessary.
> >
> > You can do:
> >
> >    ns_proxy clear ?pool?
> >
> > to forcefully clear _all_ slave processes (or just
> > ones for the given pool).
> >
> > But look here:
> >
> >    server1:nscp 1> set proxy [ns_proxy get pool]
> >
> > zvpb:~ zoran$ ps -ax | grep proxy
> > 13423  p0  S+     0:00.02 /usr/local/ns/bin/nsproxy pool pool-7
> > 13425  p2  S+     0:00.01 grep proxy
> >
> > Now we have a slave process running. So:
> >
> >    server1:nscp 2> ns_proxy send $proxy "exec sleep 10000"
> >    server1:nscp 3> ns_proxy wait $proxy 1000
> >    could not wait for proxy "pool-7": timeout waiting for evaluation
> >
> > zvpb:~ zoran$ ps -ax | grep proxy
> > 13423  p0  S+     0:00.02 /usr/local/ns/bin/nsproxy pool pool-7 {exec
> > sleep 10000}
> >
> > Now you have the slave "gone wild". How to get rid
> > of it?  Simply:
> >
> >    server1:nscp 4> ns_proxy put $proxy
> >
> > and hopla:
> >
> > [14/Sep/2006:19:58:11][13419.41968128][-nsproxy:reap-] Warning:
> > [pool]: pid 13423 won't die, send signal 15
> > [14/Sep/2006:19:58:12][13419.41968128][-nsproxy:reap-] Error: process
> > 13423 killed with signal 15 (Terminated)
> >
> > and:
> >
> > zvpb:~ zoran$ ps -ax | grep proxy
> > 13430  p2  R+     0:00.00 grep proxy
> >
> > This is NEW in the code. If your proxy is just taking too
> > much time, you return it back to the pool. The code will
> > see that you are returning an non-idle proxy back to pool
> > and will assure to kill the slave.
> >
> > Is this what you are looking for?
> >
> > Cheers,
> > Zoran
> >
> >
> >
> > -------------------------------------------------------------------------
> > Using Tomcat but need to do more? Need to support web services, security?
> > Get stuff done quickly with pre-integrated technology to make your job easier
> > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> > _______________________________________________
> > naviserver-devel mailing list
> > nav...@li...
> > https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> >
>
> --
> Vlad Seryakov
> 571 262-8608 office
> vl...@cr...
> http://www.crystalballinc.com/vlad/
>
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] ns_proxy revisited

[Vlad S. <vl...@cr...>]

then the case is closed (for me), once ready i will use it

Zoran Vasiljevic wrote:
> On 14.09.2006, at 20:25, Vlad Seryakov wrote:
> 
>> Yes, this looks good, can you put pid there as well?
> 
> yes.
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] ns_proxy revisited

[Zoran V. <zv...@ar...>]

On 14.09.2006, at 20:25, Vlad Seryakov wrote:

> Yes, this looks good, can you put pid there as well?

yes.

Re: [naviserver-devel] ns_proxy revisited

[Vlad S. <vl...@cr...>]

Yes, this looks good, can you put pid there as well?

Zoran Vasiljevic wrote:
> On 14.09.2006, at 20:12, Zoran Vasiljevic wrote:
> 
>> server1:nscp 7> ns_proxy active pool
>> pool-7 {exec sleep 10000}
> 
> Alternatively, we can change the return value
> of the command to be:
> 
>    {proxyId timestamp script} {proxyId timestamp script} ...
> 
> i.e. a list of lists, where each sublist has variable
> number of arguments as above.
> Or, even better:
> 
>    {id proxyId time timestamp script script}
> 
> a key/value list so it is easy to put in the array.
> 
> The "timestamp" would be Ns_Timestamp, time when
> the command was submitted for execution.
> 
> What do you think?
> 
> 
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] ns_proxy revisited

[Zoran V. <zv...@ar...>]

On 14.09.2006, at 20:12, Zoran Vasiljevic wrote:

> server1:nscp 7> ns_proxy active pool
> pool-7 {exec sleep 10000}

Alternatively, we can change the return value
of the command to be:

   {proxyId timestamp script} {proxyId timestamp script} ...

i.e. a list of lists, where each sublist has variable
number of arguments as above.
Or, even better:

   {id proxyId time timestamp script script}

a key/value list so it is easy to put in the array.

The "timestamp" would be Ns_Timestamp, time when
the command was submitted for execution.

What do you think?

Re: [naviserver-devel] ns_proxy revisited

[Vlad S. <vl...@cr...>]

Thanks, no hurry

Zoran Vasiljevic wrote:
> On 14.09.2006, at 20:13, Vlad Seryakov wrote:
> 
>> That is what i need
> 
> I was afraid of you saying this!
> 
> Because I had that code already done and then
> have removed it because I believed nobody would
> use it anyway....
> Allright, if you want (and I see no reasons why not)
> I can plug that in again. Tomorrow.
> 
> Cheers,
> Zoran
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] ns_proxy revisited

[Zoran V. <zv...@ar...>]

On 14.09.2006, at 20:13, Vlad Seryakov wrote:

> That is what i need

I was afraid of you saying this!

Because I had that code already done and then
have removed it because I believed nobody would
use it anyway....
Allright, if you want (and I see no reasons why not)
I can plug that in again. Tomorrow.

Cheers,
Zoran

Re: [naviserver-devel] ns_proxy revisited

[Vlad S. <vl...@cr...>]

That is what i need

Zoran Vasiljevic wrote:
> On 14.09.2006, at 20:03, Vlad Seryakov wrote:
> 
>> Can i do this from other thread, spawn in one and check in another,
>> actually this is what i need. I can have multiple processes running,
>> then according to some rules i will check which one is taking too long
>> and kill it.
> 
> Currently no as you can't operate on a proxy from other thread then
> from the one you obtained it.
> 
> But I'm thinking loud...
> 
> There is a command [ns_proxy active pool]
> It will give you such thing back:
> 
> server1:nscp 5> set proxy [ns_proxy get pool]
> pool-7
> server1:nscp 6> ns_proxy send $proxy "exec sleep 10000"
> server1:nscp 7> ns_proxy active pool
> pool-7 {exec sleep 10000}
> 
> A list of poolId/script. If we would expand the
> [ns_proxy clear ?pool?]
> to be
> [ns_proxy clear ?pool ?handle??]
> then you'd be able to:
> 
>     ns_proxy clear pool pool-7
> 
> and this will bruteforce close the slave. Any potential
> waiter will exit with Tcl error and can examine errorCode
> to see why it got kicked out.
> This could be done from any thread.
> 
> Is this something suitable?
> 
> Cheers
> Zoran
> 
> 
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] ns_proxy revisited

[Zoran V. <zv...@ar...>]

On 14.09.2006, at 20:03, Vlad Seryakov wrote:

> Can i do this from other thread, spawn in one and check in another,
> actually this is what i need. I can have multiple processes running,
> then according to some rules i will check which one is taking too long
> and kill it.

Currently no as you can't operate on a proxy from other thread then
from the one you obtained it.

But I'm thinking loud...

There is a command [ns_proxy active pool]
It will give you such thing back:

server1:nscp 5> set proxy [ns_proxy get pool]
pool-7
server1:nscp 6> ns_proxy send $proxy "exec sleep 10000"
server1:nscp 7> ns_proxy active pool
pool-7 {exec sleep 10000}

A list of poolId/script. If we would expand the
[ns_proxy clear ?pool?]
to be
[ns_proxy clear ?pool ?handle??]
then you'd be able to:

    ns_proxy clear pool pool-7

and this will bruteforce close the slave. Any potential
waiter will exit with Tcl error and can examine errorCode
to see why it got kicked out.
This could be done from any thread.

Is this something suitable?

Cheers
Zoran

Re: [naviserver-devel] ns_proxy revisited

[Vlad S. <vl...@cr...>]

Can i do this from other thread, spawn in one and check in another, 
actually this is what i need. I can have multiple processes running, 
then according to some rules i will check which one is taking too long 
and kill it.

Zoran Vasiljevic wrote:
> On 14.09.2006, at 19:50, Vlad Seryakov wrote:
> 
>> The only immediate request i would have for ns_proxy to be able to  
>> query
>> pid for active proxies, may be ns_proxy active could return pid or  
>> other
>> command, that will be useful for forcing process exit if necessary.
> 
> You can do:
> 
>    ns_proxy clear ?pool?
> 
> to forcefully clear _all_ slave processes (or just
> ones for the given pool).
> 
> But look here:
> 
>    server1:nscp 1> set proxy [ns_proxy get pool]
> 
> zvpb:~ zoran$ ps -ax | grep proxy
> 13423  p0  S+     0:00.02 /usr/local/ns/bin/nsproxy pool pool-7
> 13425  p2  S+     0:00.01 grep proxy
> 
> Now we have a slave process running. So:
> 
>    server1:nscp 2> ns_proxy send $proxy "exec sleep 10000"
>    server1:nscp 3> ns_proxy wait $proxy 1000
>    could not wait for proxy "pool-7": timeout waiting for evaluation
> 
> zvpb:~ zoran$ ps -ax | grep proxy
> 13423  p0  S+     0:00.02 /usr/local/ns/bin/nsproxy pool pool-7 {exec  
> sleep 10000}
> 
> Now you have the slave "gone wild". How to get rid
> of it?  Simply:
> 
>    server1:nscp 4> ns_proxy put $proxy
> 
> and hopla:
> 
> [14/Sep/2006:19:58:11][13419.41968128][-nsproxy:reap-] Warning:  
> [pool]: pid 13423 won't die, send signal 15
> [14/Sep/2006:19:58:12][13419.41968128][-nsproxy:reap-] Error: process  
> 13423 killed with signal 15 (Terminated)
> 
> and:
> 
> zvpb:~ zoran$ ps -ax | grep proxy
> 13430  p2  R+     0:00.00 grep proxy
> 
> This is NEW in the code. If your proxy is just taking too
> much time, you return it back to the pool. The code will
> see that you are returning an non-idle proxy back to pool
> and will assure to kill the slave.
> 
> Is this what you are looking for?
> 
> Cheers,
> Zoran
> 
> 
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] ns_proxy revisited

[Zoran V. <zv...@ar...>]

On 14.09.2006, at 19:50, Vlad Seryakov wrote:

> The only immediate request i would have for ns_proxy to be able to  
> query
> pid for active proxies, may be ns_proxy active could return pid or  
> other
> command, that will be useful for forcing process exit if necessary.

You can do:

   ns_proxy clear ?pool?

to forcefully clear _all_ slave processes (or just
ones for the given pool).

But look here:

   server1:nscp 1> set proxy [ns_proxy get pool]

zvpb:~ zoran$ ps -ax | grep proxy
13423  p0  S+     0:00.02 /usr/local/ns/bin/nsproxy pool pool-7
13425  p2  S+     0:00.01 grep proxy

Now we have a slave process running. So:

   server1:nscp 2> ns_proxy send $proxy "exec sleep 10000"
   server1:nscp 3> ns_proxy wait $proxy 1000
   could not wait for proxy "pool-7": timeout waiting for evaluation

zvpb:~ zoran$ ps -ax | grep proxy
13423  p0  S+     0:00.02 /usr/local/ns/bin/nsproxy pool pool-7 {exec  
sleep 10000}

Now you have the slave "gone wild". How to get rid
of it?  Simply:

   server1:nscp 4> ns_proxy put $proxy

and hopla:

[14/Sep/2006:19:58:11][13419.41968128][-nsproxy:reap-] Warning:  
[pool]: pid 13423 won't die, send signal 15
[14/Sep/2006:19:58:12][13419.41968128][-nsproxy:reap-] Error: process  
13423 killed with signal 15 (Terminated)

and:

zvpb:~ zoran$ ps -ax | grep proxy
13430  p2  R+     0:00.00 grep proxy

This is NEW in the code. If your proxy is just taking too
much time, you return it back to the pool. The code will
see that you are returning an non-idle proxy back to pool
and will assure to kill the slave.

Is this what you are looking for?

Cheers,
Zoran

Re: [naviserver-devel] ns_proxy revisited

[Vlad S. <vl...@cr...>]

The only immediate request i would have for ns_proxy to be able to query 
pid for active proxies, may be ns_proxy active could return pid or other 
command, that will be useful for forcing process exit if necessary.

Zoran Vasiljevic wrote:
> Hi!
> 
> Motivated by the Vlad's recent report, I took another
> tour arround the code and did yet-another cleanup phase.
> Amazing howmany things you can find when you revisit the
> code after some time...
> 
> I have fixed numerous MT-safeness problems, removed the
> "-minslaves" config option and added new [ns_proxy pools]
> command. I also updated the test-suite to check more cases.
> Also the problem reported by Vlad should be fixed as well.
> 
> Altogether, this thing now seems pretty usable :-)
> 
> If you discover something what you can undoubtfully
> classify as a bug, plese tell me as soon as possible
> as I would like to get this code integrated in our product
> (having more testers is an obvious plus for us, hehe).
> 
>  From this point on we may think of the ways how to
> eventually improve the beast. But, even as-is, it proved
> _immensely_ valuable for us, as we can peacefully hit an
> external program with a relative ease and encapsulate
> non-mt-safe code otherwise unthinkable to run in the
> context of the nsd process. Very Cool!
> 
> Cheers,
> Zoran
> 
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

[naviserver-devel] ns_proxy revisited

[Zoran V. <zv...@ar...>]

Hi!

Motivated by the Vlad's recent report, I took another
tour arround the code and did yet-another cleanup phase.
Amazing howmany things you can find when you revisit the
code after some time...

I have fixed numerous MT-safeness problems, removed the
"-minslaves" config option and added new [ns_proxy pools]
command. I also updated the test-suite to check more cases.
Also the problem reported by Vlad should be fixed as well.

Altogether, this thing now seems pretty usable :-)

If you discover something what you can undoubtfully
classify as a bug, plese tell me as soon as possible
as I would like to get this code integrated in our product
(having more testers is an obvious plus for us, hehe).

 From this point on we may think of the ways how to
eventually improve the beast. But, even as-is, it proved
_immensely_ valuable for us, as we can peacefully hit an
external program with a relative ease and encapsulate
non-mt-safe code otherwise unthinkable to run in the
context of the nsd process. Very Cool!

Cheers,
Zoran

Re: [naviserver-devel] test suite

[Vlad S. <vl...@cr...>]

fixed but still that encoding/chunked patch breaks a lot of tests

Zoran Vasiljevic wrote:
> This seem rather broken:
> 
> Tests ended at Wed Sep 13 19:50:31 CEST 2006
> all.tcl:        Total   748     Passed  567     Skipped 5        
> Failed  176
> Sourced 43 Test Files.
> Files with failing tests: cookies.test encoding.test errordoc.test  
> http.test http_byteranges.test http_chunked.test ns_adp_compress.test  
> ns_cache.test ns_conn_host.test ns_limits.test ns_pagepath.test  
> ns_register_filter.test ns_register_proc.test ns_serverpath.test  
> ns_urlencode.test tclresp.test timeout.test url2file.test
> 
> Please can somebody take a look why is this so?
> By glancing over the output I'Ve seen such:
> 
> errorInfo: invalid command name "nstest_http"
> (lots of)
> 
> then
> ---- Result was:
> default l1
> ---- Result should have been (exact matching):
> confLimit1 confLimit2 default l1
> ==== limits-2.4 FAILED
> 
> then
> errorInfo(cleanup): invalid command name "ns_memoize_flush"
> errorInfo: invalid command name "ns_memoize"
> 
> Any ideas?
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/