naviserver-devel — Developer discussion

Re: [naviserver-devel] ns_proxy crash

[Vlad S. <vl...@cr...>]

No hurry, i am doing this the old-fashioned way using ns_thread for now 
but ns_job-like way of handling proxies would be nice

Zoran Vasiljevic wrote:
> On 13.09.2006, at 21:52, Vlad Seryakov wrote:
> 
>> ok, for example i schedule encoder
>>
>> ns_proxy configure channel1 -minslaves 0 -maxslaves 1
>> set h [ns_proxy get channel1]
>> ns_proxy send $h "/usr/bin/mencoder ...."
>>
>> now proxy is running and pool channel1 is taken, in another thread i
>> cannot just issue ns_proxy get channel1, it will hang.
>>
>> I can run ns_proxy active or ns_proxy handles but how i would know  
>> about
>> this particular process in order to use ns_kill?
> 
> I believe that ns_proxy as-implemtented now is not entirely
> correctly, hence it opens whole lotta questions
> as yours above. I'm gonna have to (partly) rewrite it tomorrow.
> But before I do that, the answers:
> 
> Normally a proxy pool is just a collection of proxy-structures.
> Each proxy structure is used to communicate with (and control)
> an process instance. Each thread can acquire a number of such
> structures from the pool. After doing so, it has sole ownership
> of the proxy(es) and other threads have no access to them.
> When a thread is done with the proxy, it can return it to pool
> in which case some other threads can acquire it.
> So, the proxy structure cannot be shared among threads. There
> is no locking that would support that.
> 
> Now we may think about adding this, but to be honest I haven't
> thougth about that at all yet. My first goal is to have a properly
> working ns_proxy module. After we get this stable (which it still
> is not) we can think about extending it.
> 
> Cheers,
> Zoran
> 
> 
>   
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] ns_proxy crash

[Zoran V. <zv...@ar...>]

On 13.09.2006, at 21:52, Vlad Seryakov wrote:

> ok, for example i schedule encoder
>
> ns_proxy configure channel1 -minslaves 0 -maxslaves 1
> set h [ns_proxy get channel1]
> ns_proxy send $h "/usr/bin/mencoder ...."
>
> now proxy is running and pool channel1 is taken, in another thread i
> cannot just issue ns_proxy get channel1, it will hang.
>
> I can run ns_proxy active or ns_proxy handles but how i would know  
> about
> this particular process in order to use ns_kill?

I believe that ns_proxy as-implemtented now is not entirely
correctly, hence it opens whole lotta questions
as yours above. I'm gonna have to (partly) rewrite it tomorrow.
But before I do that, the answers:

Normally a proxy pool is just a collection of proxy-structures.
Each proxy structure is used to communicate with (and control)
an process instance. Each thread can acquire a number of such
structures from the pool. After doing so, it has sole ownership
of the proxy(es) and other threads have no access to them.
When a thread is done with the proxy, it can return it to pool
in which case some other threads can acquire it.
So, the proxy structure cannot be shared among threads. There
is no locking that would support that.

Now we may think about adding this, but to be honest I haven't
thougth about that at all yet. My first goal is to have a properly
working ns_proxy module. After we get this stable (which it still
is not) we can think about extending it.

Cheers,
Zoran


  

Re: [naviserver-devel] ns_proxy crash

[Vlad S. <vl...@cr...>]

ok, for example i schedule encoder

ns_proxy configure channel1 -minslaves 0 -maxslaves 1
set h [ns_proxy get channel1]
ns_proxy send $h "/usr/bin/mencoder ...."

now proxy is running and pool channel1 is taken, in another thread i 
cannot just issue ns_proxy get channel1, it will hang.

I can run ns_proxy active or ns_proxy handles but how i would know about 
this particular process in order to use ns_kill?

Zoran Vasiljevic wrote:
> On 13.09.2006, at 16:45, Vlad Seryakov wrote:
> 
>> Can nsproxy be extended to be able to kill any particular process
>> without issuing nsproxy get first, get hangs if no available  
>> process are
>>   in the pool. It would be nice to be able to query how many are  
>> running
>> and be able to kill by pid or name, does not matter.
> 
> Any particular process??? What do you mean about that?
> There is ns_kill command for that.
> But, I'd say I doi not understand the question...
> Can you put in more detail?
> 
> Cheers
> Zoran
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] ns_proxy crash

[Zoran V. <zv...@ar...>]

On 13.09.2006, at 16:45, Vlad Seryakov wrote:

> Can nsproxy be extended to be able to kill any particular process
> without issuing nsproxy get first, get hangs if no available  
> process are
>   in the pool. It would be nice to be able to query how many are  
> running
> and be able to kill by pid or name, does not matter.

Any particular process??? What do you mean about that?
There is ns_kill command for that.
But, I'd say I doi not understand the question...
Can you put in more detail?

Cheers
Zoran

[naviserver-devel] Relesase Build Problem

[Ian H. <iha...@de...>]

I downloaded the tarballs from sourceforge, and there seem to be a
couple things missing.

The contrib directory is not present, and sample-config.tcl does not
exist.  Other than that, things seem to go OK...

-----------

for i in contrib/nsd.tcl contrib/mimetypes.tcl sample-config.tcl; do \
        /usr/local/naviserver-4.99.1/install-sh -c -m 644 $i
/usr/local/ns/conf/                    ; \
done
cp: cannot stat `contrib/nsd.tcl': No such file or directory
chmod: cannot access `/usr/local/ns/conf/#inst.29201#': No such file
or director                    y
mv: cannot stat `/usr/local/ns/conf/#inst.29201#': No such file or directory
cp: cannot stat `contrib/mimetypes.tcl': No such file or directory
chmod: cannot access `/usr/local/ns/conf/#inst.29208#': No such file
or director                    y
mv: cannot stat `/usr/local/ns/conf/#inst.29208#': No such file or directory
cp: cannot stat `sample-config.tcl': No such file or directory
chmod: cannot access `/usr/local/ns/conf/#inst.29215#': No such file
or director                    y
mv: cannot stat `/usr/local/ns/conf/#inst.29215#': No such file or directory
/usr/local/naviserver-4.99.1/install-sh -c -m 644 contrib/index.adp
/usr/local/n                    s/pages/
cp: cannot stat `contrib/index.adp': No such file or directory
chmod: cannot access `/usr/local/ns/pages/#inst.29222#': No such file
or directo                    ry
mv: cannot stat `/usr/local/ns/pages/#inst.29222#': No such file or directory
/usr/local/naviserver-4.99.1/install-sh -c install-sh /usr/local/ns/bin/
for i in nsthread nsd nssock nscgi nscp nslog nsperm nsdb nsdbtest; do \
        (cd $i && make install) || exit 1; \
done

[naviserver-devel] test suite

[Zoran V. <zv...@ar...>]

This seem rather broken:

Tests ended at Wed Sep 13 19:50:31 CEST 2006
all.tcl:        Total   748     Passed  567     Skipped 5        
Failed  176
Sourced 43 Test Files.
Files with failing tests: cookies.test encoding.test errordoc.test  
http.test http_byteranges.test http_chunked.test ns_adp_compress.test  
ns_cache.test ns_conn_host.test ns_limits.test ns_pagepath.test  
ns_register_filter.test ns_register_proc.test ns_serverpath.test  
ns_urlencode.test tclresp.test timeout.test url2file.test

Please can somebody take a look why is this so?
By glancing over the output I'Ve seen such:

errorInfo: invalid command name "nstest_http"
(lots of)

then
---- Result was:
default l1
---- Result should have been (exact matching):
confLimit1 confLimit2 default l1
==== limits-2.4 FAILED

then
errorInfo(cleanup): invalid command name "ns_memoize_flush"
errorInfo: invalid command name "ns_memoize"

Any ideas?

[naviserver-devel] ns_proxy is still broken

[Zoran V. <zv...@ar...>]

Hi!

Apologize for this but the ns_proxy still has some
issues I've found after examining the core that
Vlad sent yesterday. I'm working on that now...

Cheers,
Zoran

Re: [naviserver-devel] ns_proxy crash

[Vlad S. <vl...@cr...>]

The other night i was trying to use nsproxy for long running processes, 
like encoding with ability to work with them like ns_job queues,
define min. max, submit in the background, check who is running, kill if 
have to.

Can nsproxy be extended to be able to kill any particular process 
without issuing nsproxy get first, get hangs if no available process are 
  in the pool. It would be nice to be able to query how many are running 
and be able to kill by pid or name, does not matter.

Zoran Vasiljevic wrote:
> On 13.09.2006, at 04:55, Vlad Seryakov wrote:
> 
>> Had ns_proxy running, typed in nscp ns_shutdown and nsd crahsed
> 
> Eh... this is true. In the attempt to cleanup as much as possible
> I prematurely cleaned a pool having references to it.
> Actually, pools still having references to them should not be
> touched and should rather leak.
> OTOH, at the shutdown, there should be no references as all
> threads should have already been exited. I guess at this point
> we have still some trouble...
> 
> The immediate and not clean solution is just to skip such pools
> and let them leak. I'm emitting the log message in such case as
> the reminder:
> 
> [13/Sep/2006:10:22:37][1132.2684415384][-main-] Warning: nsproxy:  
> pool "test" not destroyed, has 1 active slaves
> 
> I do not think it is possible to avoid this, after thinking about
> this for a while. But still, lets leave this here as we might be
> more clever in the future.
> 
> Please checkout and try again.
> 
> Cheers
> Zoran
> 
> 
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] ns_proxy crash

[Zoran V. <zv...@ar...>]

On 13.09.2006, at 04:55, Vlad Seryakov wrote:

> Had ns_proxy running, typed in nscp ns_shutdown and nsd crahsed

Eh... this is true. In the attempt to cleanup as much as possible
I prematurely cleaned a pool having references to it.
Actually, pools still having references to them should not be
touched and should rather leak.
OTOH, at the shutdown, there should be no references as all
threads should have already been exited. I guess at this point
we have still some trouble...

The immediate and not clean solution is just to skip such pools
and let them leak. I'm emitting the log message in such case as
the reminder:

[13/Sep/2006:10:22:37][1132.2684415384][-main-] Warning: nsproxy:  
pool "test" not destroyed, has 1 active slaves

I do not think it is possible to avoid this, after thinking about
this for a while. But still, lets leave this here as we might be
more clever in the future.

Please checkout and try again.

Cheers
Zoran

[naviserver-devel] ns_proxy crash

[Vlad S. <vl...@cr...>]

Had ns_proxy running, typed in nscp ns_shutdown and nsd crahsed

0xffffe410 in __kernel_vsyscall ()
(gdb) bt
#0  0xffffe410 in __kernel_vsyscall ()
#1  0xb7d5b486 in __nanosleep_nocancel () from /lib/libc.so.6
#2  0xb7d5b2ab in sleep () from /lib/libc.so.6
#3  0xb62d701c in DnsPanic (fmt=0xb7f2d340 "nsthreads: %s failed in %s: 
%s") at nsdns.c:237
#4  0xb7eee03e in Tcl_PanicVA () from /usr/lib/libtcl8.4.so
#5  0xb7eee0d7 in Tcl_Panic () from /usr/lib/libtcl8.4.so
#6  0xb7f2b1fd in NsThreadFatal (func=0xb7f2db3a "Ns_CondDestroy",
     osfunc=0xb7f2db25 "pthread_cond_destroy", err=16) at error.c:64
#7  0xb7f2cb1c in Ns_CondDestroy (cond=0xb7f2e0f8) at pthread.c:553
#8  0xb4614855 in Ns_ProxyExit (arg=0xbf889858) at nsproxylib.c:2621
#9  0xb7f4bcef in RunCallbacks (cbPtr=<value optimized out>, reverse=0) 
at callbacks.c:490
#10 0xb7f5feea in Ns_Main (argc=8, argv=0xbf88ae44, initProc=0x8048620 
<ServerInit>)
     at nsmain.c:838
#11 0x0804866f in main (argc=0, argv=0x1e313f8) at main.c:64


-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] nsfreetds

[Vlad S. <vl...@cr...>]

it is nsdbtds now under modules/

Ian Harding wrote:
> Where did nsfreetds go?  I don't see it in the modules tarball 8^(
>
> I really need to get at the file because the original version at
> aolserver is unmaintained.
>
> Thanks!
>
> - Ian
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>   

[naviserver-devel] nsfreetds

[Ian H. <har...@gm...>]

Where did nsfreetds go?  I don't see it in the modules tarball 8^(

I really need to get at the file because the original version at
aolserver is unmaintained.

Thanks!

- Ian

Re: [naviserver-devel] Second set of documentation

[Zoran V. <zv...@ar...>]

On 09.09.2006, at 17:21, Vlad Seryakov wrote:

> Ok, then i will start polishing it and will import into CVS under  
> html/

Perhaps others may comment as well before you invest more time...
Let the people say what they think. If nobody objects (or has
some other breath-taking idea) then continue...

Cheers
Zoran

Re: [naviserver-devel] Second set of documentation

[Vlad S. <vl...@cr...>]

Ok, then i will start polishing it and will import into CVS under html/

Zoran Vasiljevic wrote:
> On 09.09.2006, at 16:31, Vlad Seryakov wrote:
> 
>> This is what we can base our documentation on, the question is: do we
>> convert it into doctools or keep as html?
> 
> Why would you want to convert it in doctools? Such high-level docs is
> perfectly OK to be kept in HTML mode...
> Allright, if we'd to make pdf's one day, then it is OK to have it in
> neutral (doctools) format, but I doubt this is now worth the effort.
> Different is reference docs (man pages) because it is regularily used
> as help during programming. Now, I always have to go to sources to
> look what the Ns_BlaBlah is doing... with "man Ns_BlahBlah" I can
> refresh myself easily. Tcl has such man-based docs and I find it
> _INVALUABLE_ for development.
> 
> Cheers
> Zoran
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 


-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Second set of documentation

[Zoran V. <zv...@ar...>]

On 09.09.2006, at 16:31, Vlad Seryakov wrote:

> This is what we can base our documentation on, the question is: do we
> convert it into doctools or keep as html?

Why would you want to convert it in doctools? Such high-level docs is
perfectly OK to be kept in HTML mode...
Allright, if we'd to make pdf's one day, then it is OK to have it in
neutral (doctools) format, but I doubt this is now worth the effort.
Different is reference docs (man pages) because it is regularily used
as help during programming. Now, I always have to go to sources to
look what the Ns_BlaBlah is doing... with "man Ns_BlahBlah" I can
refresh myself easily. Tcl has such man-based docs and I find it
_INVALUABLE_ for development.

Cheers
Zoran

Re: [naviserver-devel] Second set of documentation

[Vlad S. <vl...@cr...>]

>
> Wow! It is great to see how much you (and Bernd)
> have done in so little time... Congratulations!
>   

This is what we can base our documentation on, the question is: do we 
convert it into doctools or keep as html?

Re: [naviserver-devel] Second set of documentation

[Zoran V. <zv...@ar...>]

On 09.09.2006, at 00:18, Vlad Seryakov wrote:

>
> http://www.crystalballinc.com/vlad/nsdocs/
>
> A little bit converted but still in HTML

Wow! It is great to see how much you (and Bernd)
have done in so little time... Congratulations!

A side note: anybody still usin ns_share?

   NOTE: The use of ns_share is deprecated. Careless use of ns_share
   commands can severely limit the scalability of an application.
   The nsv commands solve this and this document describes how to
   migrate your ns_share code to use nsv.

I would drop that immediately.

Cheers
Zoran

[naviserver-devel] Second set of documentation

[Vlad S. <vl...@cr...>]

http://www.crystalballinc.com/vlad/nsdocs/

A little bit converted but still in HTML

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] How can I help?

[Michael A. C. <cle...@gm...>]

On 9/8/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> You are now added as developer.
> Welcome on board.
>
> Zoran

Thanks :-)

Michael

Re: [naviserver-devel] How can I help?

[Zoran V. <zv...@ar...>]

On 08.09.2006, at 22:53, Michael A. Cleverly wrote:

>
> Yes, "cleverly" is my SF username.


You are now added as developer.
Welcome on board.

Zoran

Re: [naviserver-devel] How can I help?

[Michael A. C. <cle...@gm...>]

On 9/8/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 08.09.2006, at 22:12, Mike wrote:
>
> > Zoran,
> >   We do not have a consistent, complete, documented, standalone,
> > high-level database API.  Sourceforge archives are crap, but you can
> > see the relevant thread here:
> > http://sourceforge.net/mailarchive/message.php?msg_id=30716597
>
> Allright... as said. Vlad will definitely have to say
> something about that since he wrote some db modules
> as I can see from the modules directory.
> But, I'm all for it, if this helps make the server
> more feature complete and easier to use.
>
> Michael, do you have account on SF so I can
> add you to the project?

Yes, "cleverly" is my SF username.

> Normally. we do not have a high-barrier on commiters
> but we expect one, when having new ideas, to first present
> them to others, discuss them and if nobody is objecting
> he's free to add new stuff.

This is a good policy.

> If you discover a bug, you can fix it alone or if
> you do not know how, post a bug report on SF so somebody
> else will take care about it.
> Altogether, we try to be pretty non-buerocratic in this
> respect.

Sounds good, thanks!

Michael

Re: [naviserver-devel] How can I help?

[Zoran V. <zv...@ar...>]

On 08.09.2006, at 22:12, Mike wrote:

> Zoran,
>   We do not have a consistent, complete, documented, standalone,
> high-level database API.  Sourceforge archives are crap, but you can
> see the relevant thread here:
> http://sourceforge.net/mailarchive/message.php?msg_id=30716597

Allright... as said. Vlad will definitely have to say
something about that since he wrote some db modules
as I can see from the modules directory.
But, I'm all for it, if this helps make the server
more feature complete and easier to use.

Michael, do you have account on SF so I can
add you to the project?

Normally. we do not have a high-barrier on commiters
but we expect one, when having new ideas, to first present
them to others, discuss them and if nobody is objecting
he's free to add new stuff.
If you discover a bug, you can fix it alone or if
you do not know how, post a bug report on SF so somebody
else will take care about it.
Altogether, we try to be pretty non-buerocratic in this
respect.

Cheers
Zoran

Re: [naviserver-devel] How can I help?

[Mike <nee...@gm...>]

On 9/8/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 08.09.2006, at 21:57, Zoran Vasiljevic wrote:
>
> > How is this different/better than what we have now?
>
> FYI... I'm kind of a "special" server user as we do not
> use the DB interface, nor ADP (we have developed our
> own db-intgerace and tdom-based templating)...
> Hence the question...
>
> But Vlad and Stepheh do use it so they will definitely
> have more questions. I just wanted to get a brid's-eye view,
> kind of "executive summary"...

Zoran,
  We do not have a consistent, complete, documented, standalone,
high-level database API.  Sourceforge archives are crap, but you can
see the relevant thread here:
http://sourceforge.net/mailarchive/message.php?msg_id=30716597

Re: [naviserver-devel] How can I help?

[Michael A. C. <cle...@gm...>]

On 9/8/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 08.09.2006, at 21:52, Mike wrote:
>
> >   I think Michael failed to properly introduce himself.  :)  Michael
> > is the author of nstcl: http://wiki.tcl.tk/3666 ...  I can see no more
> > perfect candidate for resolving the database interface issue...  I
> > think getting an ACS-like db interface has been on the wishlist of
> > AOLserver for years.  This would be a huge step forward for
> > NaviServer.  (again, my opinion only)
>
> I have no problem with that. Just out of curiosity...
> How is this different/better than what we have now?

At the lower level (ns_db) obviously what NaviServer has is the best
(after all it is the API that nstcl has sought to emulate since as far
back as the AOLserver 3 era when libnsd was still just a
dream/vaporware).

Above that is the ACS-like database API (db_foreach, db_multirow,
db_string, etc.) that are built on top of the ns_db and ns_set APIs.
nstcl has this code, probably more easily extractable then OACS, which
I'd be willing to package as a .tcl module (or a .tcl support file to
go alon with the existing nsdb module).  I've also got separate
stand-alone man pages for each of the high-level database API
functions already written.

Michael

Re: [naviserver-devel] How can I help?

[Zoran V. <zv...@ar...>]

On 08.09.2006, at 21:57, Zoran Vasiljevic wrote:

> How is this different/better than what we have now?

FYI... I'm kind of a "special" server user as we do not
use the DB interface, nor ADP (we have developed our
own db-intgerace and tdom-based templating)...
Hence the question...

But Vlad and Stepheh do use it so they will definitely
have more questions. I just wanted to get a brid's-eye view,
kind of "executive summary"...

Cheers
Zoran