naviserver-devel — Developer discussion

Re: [naviserver-devel] ns_perm module bug fix

[Daniel S. <moo...@av...>]

On Sat, May 10, 2008 at 8:32 AM, Vlad Seryakov <vl...@cr...> wrote:
> patched, submitted. Thanks

I just committed it to aolserver cvs also.

Daniel

-- 
| ---------------------------------------------------------------
| Daniel P. Stasinski | http://www.saidsimple.com
| moo...@av... | http://www.disabilities-r-us.com
| XMMP: moo...@av... | http://www.avenues.org
| Google Talk: mooooooo | http://www.scriptkitties.com

[naviserver-devel] sample config

[Vasiljevic Z. <zv...@ar...>]

Hi !

In the process of a spring-cleanup I wanted to attack
the gobal sample-config.tcl file that we deliver
with the server. The purpose of that file is obviously
dual:

  o. get a config file people can start with
  o. declare and document each and every config option

As it seems, we fail to meet any of those goals _really_.
One can use this file to start the server, right, but
thats about it. Any attempt to understand it, is not
going to bear fruits...

I will reorganize this file as follows:

   o. write man page about the overal layout of the file
   o. write man page explaining server internal parameters
   o. expand man pages for each module and put explanations
      of module paramaters there
   o. rewrite this file to include sample for configuring
      two virtual servers

Since we now have man pages in place, logical place to
put all possible documentation and description of options
is there. In the config file we can put only really necessary
stuff that is needed to startup some example configurations
and put cross references to the documentation.

Vlad already started this right by providing a minimal
no-fuss config file for somebody that needs to pull up
the server fast with minimal config. What we need is
something similar to that for virtual servers in perhaps
2 or 3 sample configurations. But we should NOT put docs
in that files. We should write docs in man and put only
refs to that in the config files.

Any other ideas? Suggestions?

Zoran

Re: [naviserver-devel] moonshadow

[Vasiljevic Z. <zv...@ar...>]

On 10.05.2008, at 18:15, Vlad Seryakov wrote:

> What will happen if 100 threads will send commands at the same time  
> and
> those commands will be more complicated, not just return?
>
> Main thread will serialize them, i guess.
> Does not it seems like global lock in Lua?
> Will it scale the way you want it to scale?


I believe it will. It is all compromise and it
sacrifies speed for memory, obviously. But you
could decide how much memory you burn and how
fast the system is by tuning the number of
workers. At the moment we cannot tune much.
All our interps are created equal and all are
fully loaded. I would create a number of those
heavy interps upfront and make "connection"
threads as lightweight as possible (just basic
Tcl commands plus ns_ commands). I would
instrument the unknown command in them so all
that gets executed in the connection thread,
gets really excuted in one of those pre-started
things, unless it is the ns_ or regular Tcl command.
This requires certain conventions in the app but it
would I believe scale very well.

Re: [naviserver-devel] moonshadow

[Vlad S. <vl...@cr...>]

You may also check out SpiderMonkey javascript. It is thread safe and 
runtime is not very big.

Vasiljevic Zoran wrote:
> On 10.05.2008, at 06:13, Andrew Piskorski wrote:
> 
>> On Fri, May 09, 2008 at 06:06:05PM +0200, Vasiljevic Zoran wrote:
>>
>>>> I saw somewhere that Lua can have global and per-thread state
>>>> and that you can provide your own locking primitives they use
>>>> to lock their own code, but I haven't dig deeper there...
>>> Damn again. This is a global lock that serializes everything.
>>> A no go :-(
>>> Same applies to others (Ruby, Phyton, Perl).
>> Zoran, are you SURE?  Because from what I've read about Lua, it does
>> NOT have any sort of single global lock limiting true parallelism to
>> one cpu/core at a time the way Python and Ruby do.  Lua's default
>> behavior is coroutines cooperatively multitasking on just a single
>> cpu, but AFAIK the Lua implementation is not inherently limited to
>> that.
> 
> Coroutines are something similar, (purists outthere, note: _similar_)
> to Tcl event loop in terms that they work on a single processor.
> This is not real MP multithreading that we need in order to scale.
>  From the programing perspective, they are indeed very interesting
> because they don't force me into the "event-loop" shoe. I have
> nothing against event-loop. It is just that it is really targeted to
> a GUI type of apps.
> I was greping to "lua_lock" calls arround the code and found mostly:
> 
>     something () {
>        lua_lock(L);
>        //...
>        lua_unlock(L);
>     }
> 
> OK, this must _not_ mean they are absolutely serialized, but it
> really appears to me like that on the first glance.
> What I plan to do is go to Lua developers and ask couple of
> questions...
> 
>>
>> Your application clearly wants to use many thousands of stateful
>> lightweight processes.  AFAIK Lua and especially Erlang both support
>> that out of the box.  Do you ALSO need to run 2 or 8 or more of those
>> threads in parallel (at the exact same time) on a single multi-cpu
>> server?  Recent versions of Erlang definitely support that.  I'm not
>> sure whether or not Lua supports that out of the box, but it may be
>> feasible to make it work.
> 
> Yep, Erlang is something that looks very good as well. I just stumbled
> across that yesterday.  Clever, clever... still I have to see how
> they works internally and what are the gotchas of their message-passing.
> 
> Cheers
> Zoran
> 
> 
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
> Don't miss this year's exciting event. There's still time to save $100. 
> Use priority code J8TL2D2. 
> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] moonshadow

[Vlad S. <vl...@cr...>]

What will happen if 100 threads will send commands at the same time and 
those commands will be more complicated, not just return?

Main thread will serialize them, i guess.
Does not it seems like global lock in Lua?
Will it scale the way you want it to scale?

Vasiljevic Zoran wrote:
> On 10.05.2008, at 17:50, Vasiljevic Zoran wrote:
> 
>>  I think
>> it is the most one can do with Tcl.
> 
> 
> Hm ... not that bad consider the
> trivialiy of the example.
> 
> package req Thread
> set tp [tpool::create -initcmd {
>      proc sayhello args {
>          return hello-[thread::id]
>      }
> }]
> 
> proc unknown args {
>      global tp
>      set id [tpool::post $tp $args]
>      tpool::wait $tp $id
>      tpool::get  $tp $id
> }
> 
> % time {sayhello} 100
> 48 microseconds per iteration
> % sayhello
> hello-tid0xb0103000
> % thread::id
> tid0xa0111fa0
> 
> As you see, unknown triggers the sayhello that is declared
> on pool threads, but not in my thread. And the execution
> time is not really bad. And this is just generic stuff.
> Done in C really tight, I can go make that 1/5 most probably.
> 
> 
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
> Don't miss this year's exciting event. There's still time to save $100. 
> Use priority code J8TL2D2. 
> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] moonshadow

[Vasiljevic Z. <zv...@ar...>]

On 10.05.2008, at 17:50, Vasiljevic Zoran wrote:

>  I think
> it is the most one can do with Tcl.


Hm ... not that bad consider the
trivialiy of the example.

package req Thread
set tp [tpool::create -initcmd {
     proc sayhello args {
         return hello-[thread::id]
     }
}]

proc unknown args {
     global tp
     set id [tpool::post $tp $args]
     tpool::wait $tp $id
     tpool::get  $tp $id
}

% time {sayhello} 100
48 microseconds per iteration
% sayhello
hello-tid0xb0103000
% thread::id
tid0xa0111fa0

As you see, unknown triggers the sayhello that is declared
on pool threads, but not in my thread. And the execution
time is not really bad. And this is just generic stuff.
Done in C really tight, I can go make that 1/5 most probably.

Re: [naviserver-devel] moonshadow

[Vasiljevic Z. <zv...@ar...>]

On 10.05.2008, at 06:23, Andrew Piskorski wrote:

> So you probably have 10 MB or so of proc definitions, which are 95+%
> or even 100% identical in every thread, but Tcl keeps a completely
> separate 10 MB copy in each and every thread.  What you really want is
> to just have a single read-only 10 MB copy of all those Tcl proc
> definitions, and have all 1000 otherwise completely independent Tcl
> interps use those same procs.

_RIGHTO_

>
>
> Zoran, have you talked to Jeff Hobbs about this?  He has mentioned
> possibly adding "interp cloning" and other sorts of related stuff on
> the AOLserver list before.  I understand that the compiled Tcl byte
> code is somehow specific to the interpretor where it was compiled, so
> it doesn't (currently) work to somehow have just one copy that you use
> from multiple interps in multiple threads.  But perhaps there's some
> good way to improve that...

No go. I know about that work already. It was done I believe for
Cisco as contract-work and is living in Tcl CVS under some branch.
The problem is: this is just the single-thread interp-clone. Nothing
that can be shared.

Since some years I've been trying various approaches how to tackle
this problem but I really found no realy good and universal way.
I believe there isn't one, after all those years. I will try just this
one more idea with unknown-overloading and dispatching unknown command
to a battery of preloaded interps living in different threads. Although
not 100% transparent (people must code with couple of assumptions) I  
think
it is the most one can do with Tcl. And, I do not think this will  
improve
over time.

Hopefully I will soon find piece of mind with this issue and stop
going on other people's nervers and saturate email lists :-)
Thanks for following so far!


Cheers
Zoran

Re: [naviserver-devel] ns_perm module bug fix

[Vlad S. <vl...@cr...>]

patched, submitted. Thanks

Daniel Stasinski wrote:
> Here is an interesting one that has been broken since AOLserver 4.0.
> 
> In hosts.allow, you can use either a full or partial hostname, or
> ipaddr/netmask.   Not all work.
> 
> 192.168.0.10/255.255.255.255  <- Gives error "Invalid address or
> hostname "192.168.0.10". should be ipaddr/netmask or hostname"
> 192.168.0.10/255.255.255.0 <- This fixes the above -BUT- it allows in
> the whole class C, which may not really be what is wanted.
> bob.someserver.com  <- this works (though haven't looked deep enough
> to see if it does sanity check on forward + reverse lookup to
> guarantee it is really the right host)
> 
> The error is caused by inet_addr() returning INADDR_NONE on
> 255.255.255.255 .  Using inet_aton() resolves this issue.   A patch to
> nsperm.c at around line #739 follows:
> 
>             *slash = '\0';
>             if (inet_aton(net, &ip) == 0 || inet_aton(slash+1, &mask) == 0) {
>                 Tcl_AppendResult(interp, "invalid address or hostname \"",
>                                  net, "\". " "should be ipaddr/netmask
> or hostname", NULL);
>                 goto fail;
>             }
> 
> Daniel
> 

-- 
Vlad Seryakov
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] moonshadow

[Vasiljevic Z. <zv...@ar...>]

On 09.05.2008, at 20:44, Vasiljevic Zoran wrote:

> What MIGHT work is kind of this: create some numbers of
> fully loaded interps, each sitting in its own thread.
> A compute-farm, so to say. Then every other thread just
> creates a slave interp in one of those and runs all its
> scripts in his private slave in one of those parked threads
> in the compute-farm, using some kind of message passing.
> The slaves have all the commands aliased into the main
> one. Not sure what would happen with uplevels and upvars
> here (need to think) but this is roughly what I'm
> contemplating about now.

Upvars/Uplevels will not work. No way.

> So when a thread exits, its
> thin-slave is just deleted completely. And created it is
> also fast. Just alias all commands from the master. Not
> sure about the global variables... this is still open.

No access to variables...
So: no generic solution as I was looking after.

FWIW...

For those interested: on a Mac PB, using Tcl8.4 and
current threading extension, I can load 1000 threads
with virgin interp and this costs me about 700MB virtual
memory. Dont know about the thread stacks, if I reduce
the default, perhaps it will be less (platform specific
anyway). But this is acceptable footprint. After all
there are 1000 units of execution! If I now just blindly
override the [::unknown] command in each of them and go
look in a round-robin fashion for a idle "mother" interp
(one loaded with all that fancy app state) and execute
the "real" command there, instead in my tiny little
per-user thread, then this could work fine.
Admitently, this is _similar_ to [ns_job] or [therad::send]
but it is more "implicit" in the sense that you need not
program with [ns_job] of [thread] Tcl API.
One can just simply program in plain Tcl, provided the
application allows following limitations:

   no upvars
   no uplevels
   no direct variable access
   no references (channel handles, or other types of handles)

Just plain command dispatching. The command may only return
things "per-value" (lists, scalars) i.e. something that can be
cheaply converted to string. References/hndles are not allowed.

Those limitation are actually not bad: they force the programmer
to encapsulate the app logic within manageable self-contained blocks.

So, at the end... the only "trick" is to create a virgin interp,
overload the [::unknown] command and re-route the execution to a farm
of pre-parked and loaded interps using in-memory message-passing.
Those are all created "equal" and can be created and destroyed on
as-needed basis i.e. dynamically (actually, the Naviserver already
provides this in form of ns_job and threading extension in form
of thread-pools).

I will give this a second thought... If this holds water I will
see if I can do this in a reasonably generic fashion for general
(purpose) usage.

Cheers
Zoran

Re: [naviserver-devel] moonshadow

[Vasiljevic Z. <zv...@ar...>]

On 10.05.2008, at 06:13, Andrew Piskorski wrote:

> On Fri, May 09, 2008 at 06:06:05PM +0200, Vasiljevic Zoran wrote:
>
>>> I saw somewhere that Lua can have global and per-thread state
>>> and that you can provide your own locking primitives they use
>>> to lock their own code, but I haven't dig deeper there...
>>
>> Damn again. This is a global lock that serializes everything.
>> A no go :-(
>> Same applies to others (Ruby, Phyton, Perl).
>
> Zoran, are you SURE?  Because from what I've read about Lua, it does
> NOT have any sort of single global lock limiting true parallelism to
> one cpu/core at a time the way Python and Ruby do.  Lua's default
> behavior is coroutines cooperatively multitasking on just a single
> cpu, but AFAIK the Lua implementation is not inherently limited to
> that.

Coroutines are something similar, (purists outthere, note: _similar_)
to Tcl event loop in terms that they work on a single processor.
This is not real MP multithreading that we need in order to scale.
 From the programing perspective, they are indeed very interesting
because they don't force me into the "event-loop" shoe. I have
nothing against event-loop. It is just that it is really targeted to
a GUI type of apps.
I was greping to "lua_lock" calls arround the code and found mostly:

    something () {
       lua_lock(L);
       //...
       lua_unlock(L);
    }

OK, this must _not_ mean they are absolutely serialized, but it
really appears to me like that on the first glance.
What I plan to do is go to Lua developers and ask couple of
questions...

>
>
> Your application clearly wants to use many thousands of stateful
> lightweight processes.  AFAIK Lua and especially Erlang both support
> that out of the box.  Do you ALSO need to run 2 or 8 or more of those
> threads in parallel (at the exact same time) on a single multi-cpu
> server?  Recent versions of Erlang definitely support that.  I'm not
> sure whether or not Lua supports that out of the box, but it may be
> feasible to make it work.

Yep, Erlang is something that looks very good as well. I just stumbled
across that yesterday.  Clever, clever... still I have to see how
they works internally and what are the gotchas of their message-passing.

Cheers
Zoran

Re: [naviserver-devel] moonshadow

[Andrew P. <at...@pi...>]

On Fri, May 09, 2008 at 05:38:38PM +0200, Vasiljevic Zoran wrote:

> Recently we launched internal development of the module being
> able to handle 1000+ "users" (in our product-terms, workstations)
> with a stock 4 GB memory off-the-shelf machine like xserve or
> a Mac Pro. If we'd to make a thread-per-workstation, we'd need
> 1000 * at_least_10MB of virtual memory!

> What Tcl really lacks is a per-thread and global state.
> And I see _no_way_ how this can be made... Grmpf.

So you probably have 10 MB or so of proc definitions, which are 95+%
or even 100% identical in every thread, but Tcl keeps a completely
separate 10 MB copy in each and every thread.  What you really want is
to just have a single read-only 10 MB copy of all those Tcl proc
definitions, and have all 1000 otherwise completely independent Tcl
interps use those same procs.

Zoran, have you talked to Jeff Hobbs about this?  He has mentioned
possibly adding "interp cloning" and other sorts of related stuff on
the AOLserver list before.  I understand that the compiled Tcl byte
code is somehow specific to the interpretor where it was compiled, so
it doesn't (currently) work to somehow have just one copy that you use
from multiple interps in multiple threads.  But perhaps there's some
good way to improve that...

-- 
Andrew Piskorski <at...@pi...>
http://www.piskorski.com/

Re: [naviserver-devel] moonshadow

[Andrew P. <at...@pi...>]

On Fri, May 09, 2008 at 06:06:05PM +0200, Vasiljevic Zoran wrote:

> > I saw somewhere that Lua can have global and per-thread state
> > and that you can provide your own locking primitives they use
> > to lock their own code, but I haven't dig deeper there...
> 
> Damn again. This is a global lock that serializes everything.
> A no go :-(
> Same applies to others (Ruby, Phyton, Perl).

Zoran, are you SURE?  Because from what I've read about Lua, it does
NOT have any sort of single global lock limiting true parallelism to
one cpu/core at a time the way Python and Ruby do.  Lua's default
behavior is coroutines cooperatively multitasking on just a single
cpu, but AFAIK the Lua implementation is not inherently limited to
that.

Your application clearly wants to use many thousands of stateful
lightweight processes.  AFAIK Lua and especially Erlang both support
that out of the box.  Do you ALSO need to run 2 or 8 or more of those
threads in parallel (at the exact same time) on a single multi-cpu
server?  Recent versions of Erlang definitely support that.  I'm not
sure whether or not Lua supports that out of the box, but it may be
feasible to make it work.

(Erlang's message passing design should also mean that you can easily
start running stuff across two or more independent server boxes too.
Lua has no equilvalent, AFAIK, although there must be people in the
Lua community thinking about that sort of stuff.)

-- 
Andrew Piskorski <at...@pi...>
http://www.piskorski.com/

[naviserver-devel] ns_perm module bug fix

[Daniel S. <moo...@av...>]

Here is an interesting one that has been broken since AOLserver 4.0.

In hosts.allow, you can use either a full or partial hostname, or
ipaddr/netmask.   Not all work.

192.168.0.10/255.255.255.255  <- Gives error "Invalid address or
hostname "192.168.0.10". should be ipaddr/netmask or hostname"
192.168.0.10/255.255.255.0 <- This fixes the above -BUT- it allows in
the whole class C, which may not really be what is wanted.
bob.someserver.com  <- this works (though haven't looked deep enough
to see if it does sanity check on forward + reverse lookup to
guarantee it is really the right host)

The error is caused by inet_addr() returning INADDR_NONE on
255.255.255.255 .  Using inet_aton() resolves this issue.   A patch to
nsperm.c at around line #739 follows:

            *slash = '\0';
            if (inet_aton(net, &ip) == 0 || inet_aton(slash+1, &mask) == 0) {
                Tcl_AppendResult(interp, "invalid address or hostname \"",
                                 net, "\". " "should be ipaddr/netmask
or hostname", NULL);
                goto fail;
            }

Daniel

-- 
| ---------------------------------------------------------------
| Daniel P. Stasinski | http://www.saidsimple.com
| moo...@av... | http://www.disabilities-r-us.com
| XMMP: moo...@av... | http://www.avenues.org
| Google Talk: mooooooo | http://www.scriptkitties.com

Re: [naviserver-devel] ns_register_adp

[Daniel S. <moo...@av...>]

On Fri, May 9, 2008 at 11:19 AM, Vasiljevic Zoran <zv...@ar...> wrote:
> ... you may perhaps maintain a "transition log" explaining
> what you needed to change in your code? This might help
> other potential "switchers" as well.

Excellent idea.  The only "problems" so far have been a few proc name
clashes and the nsssl problem that you fixed (thank you.)   This is
the first time I have had to edit the code in almost 10 years.  I had
also written a variety of C modules but NaviServer now includes most
of their functionality so all I had to keep was my PKE module.

The one thing I do miss is the use of NS/Admin modules that I ported
over from AOLserver 2.x.   I'm sure I could get it all to work again
but I decided to just emulate the live tcl code editor part of it and
for database management, I'm content to use PgAdmin.

One addition that has kept me smiling is Stephen Deasey's nsdbi code.
 What a wonderful addition.  I've added an extra option and fixed what
appears to be a quirk and will submit a diff to him shortly.

Daniel

-- 
| ---------------------------------------------------------------
| Daniel P. Stasinski | http://www.saidsimple.com
| moo...@av... | http://www.disabilities-r-us.com
| XMMP: moo...@av... | http://www.avenues.org
| Google Talk: mooooooo | http://www.scriptkitties.com

Re: [naviserver-devel] moonshadow

[Vasiljevic Z. <zv...@ar...>]

On 09.05.2008, at 20:29, Vlad Seryakov wrote:

> So what you need is the ability to create "lightweight" Tcl thread  
> with
> minimum commands, but with couple "important" commands to access  
> special
> "global state" or ability to submit/retrieve data from "global state".
> Is this correct?

Yes. The lightweight state should be light (a wonder!)
in terms it is private, easily created, easily destroyed
and small in memory footprint. The closest "thing" in the
Tcl universe would be a child interp ([interp create]).

>
>
> But that "monster" thread, how do you execute in it?
> Using ns_proxy or messaging, or some other way?

Well messaging is first that comes to mind. Threading
extension already does that. The gotcha's are mostly
upvars and uplevels :-(

>
>
> I guess you create regular pthread, call Tcl_CreateInterp,
> register several command and execute your Tcl script.
> Why do you think it is not generic enough?

Well, kind of :-) It is like: I guess you go sit in the
rocket, turn on the boosters and fly to moon! Nothing
simpler that that. Unfortunately. I'm not NASA!

Our app consist of roghly 350.000 lines of Tcl code and
loads about 20+ external modules, all containing internal
state.  The command set is 500+ top-level commands and
they all depend on the internal state.

What MIGHT work is kind of this: create some numbers of
fully loaded interps, each sitting in its own thread.
A compute-farm, so to say. Then every other thread just
creates a slave interp in one of those and runs all its
scripts in his private slave in one of those parked threads
in the compute-farm, using some kind of message passing.
The slaves have all the commands aliased into the main
one. Not sure what would happen with uplevels and upvars
here (need to think) but this is roughly what I'm
contemplating about now.  So when a thread exits, its
thin-slave is just deleted completely. And created it is
also fast. Just alias all commands from the master. Not
sure about the global variables... this is still open.

Re: [naviserver-devel] moonshadow

[Vlad S. <vl...@cr...>]

> 
> Yes. Won't work. I mean it _could_ but I need to program
> for that _extra_. Ideally, I would start 10,100,1000 threads
> each with its own small "universe" that actually contains
> just dark matter (nothing) whereas all the good-stuff is
> located centraly (a large mother-interpreter?).
> Ideally, all the Tcl commands would be automagically executed
> "in the mother" but have access to my "local" space (for
> uplevel and upvar and friends).

So what you need is the ability to create "lightweight" Tcl thread with 
minimum commands, but with couple "important" commands to access special 
"global state" or ability to submit/retrieve data from "global state".
Is this correct?

But that "monster" thread, how do you execute in it?
Using ns_proxy or messaging, or some other way?

I guess you create regular pthread, call Tcl_CreateInterp,
register several command and execute your Tcl script.
Why do you think it is not generic enough?

Re: [naviserver-devel] ns_register_adp

[Vasiljevic Z. <zv...@ar...>]

On 09.05.2008, at 18:51, Daniel Stasinski wrote:

> While still migrating from aolserver to naviserver


... you may perhaps maintain a "transition log" explaining
what you needed to change in your code? This might help
other potential "switchers" as well.

Cheers
Zoran

Re: [naviserver-devel] moonshadow

[Vasiljevic Z. <zv...@ar...>]

On 09.05.2008, at 20:00, Vlad Seryakov wrote:

>> Effectively, what I need is a upside-down of the
>> Tcl model: one interp any many threads.
>> Don't tell me: go shop Java. This does not compute
>> (for various unrelated reasons).
>
> You can use ns_job, you define how may Tcl interp/threads you may use
> and then just submit tasks, each task will get state and do the work.
>
>
> I guess you already explored that.

Yes. Won't work. I mean it _could_ but I need to program
for that _extra_. Ideally, I would start 10,100,1000 threads
each with its own small "universe" that actually contains
just dark matter (nothing) whereas all the good-stuff is
located centraly (a large mother-interpreter?).
Ideally, all the Tcl commands would be automagically executed
"in the mother" but have access to my "local" space (for
uplevel and upvar and friends).
>
> You may combine methods, like let one thread to handle several
> connections sequentially if that will work but i do not have exact
> details so i will stop now.

You can of course design a 1001 model that will work for the
particular case. What I need is a _generic_ scalable model
that will work always. Imagine a Tcl interp that costs nothing
to create and consumes just 5-10 KB virtual memory! Yet it has
"access" to (or "behaves" like) a full-blown beast that sourced
thousand+ files and loaded hundred+ extensions...

The blues that we are against is that we must source 1000s
and load 100s of things for each and every connection thread.
Given connection times are short, and one re-uses conn threads
all computes well. BUT.. in my case, connection times are long
(hours, days)... So, upside down. Hence a new methodology is
needed. I tried to reduce it to event-loop type, but this is
entirely different programing technique and will not work with
the code we already have.

I will have to digest this alone. If anything _generic_ comes
out of that I will give it away. But it is good at least to
express the idea here, in case somebody has some magic already
done up his sleeve :-)

Cheers
Zoran

Re: [naviserver-devel] ns_register_adp

[Vlad S. <vl...@cr...>]

yes, this situation was not handled properly,
fixed in CVS

Daniel Stasinski wrote:
> While still migrating from aolserver to naviserver, I have found that
> when using ns_register_adp to register an adp that is outside of the
> page directory, naviserver returns a 500 error.  Nothing is logged.
> 
> Good --> ns_register_adp GET /hellogood "adp/hello.adp"
> Bad --> ns_register_adp GET /hellobad "/home/nsadmin/hello.adp"
> 
> Daniel
> 

Re: [naviserver-devel] moonshadow

[Vlad S. <vl...@cr...>]

> Effectively, what I need is a upside-down of the
> Tcl model: one interp any many threads.
> Don't tell me: go shop Java. This does not compute
> (for various unrelated reasons).

You can use ns_job, you define how may Tcl interp/threads you may use 
and then just submit tasks, each task will get state and do the work.


I guess you already explored that.

>> and, do why do you need all separate 1000 threads for every user, is  
>> the
>> requirement to be connected to all at the same time? just wondering
> 
> Yes. 1000's of workstations may be connected all the time
> and over a long time. And all of them are pumping data to
> the server, mostly. Doing this in a event-loop mode is
> possible but not feasible. I must isolate everybody in a
> thread because of the potential (complex) state that each
> connection may have.

You may combine methods, like let one thread to handle several 
connections sequentially if that will work but i do not have exact 
details so i will stop now.

Re: [naviserver-devel] moonshadow

[Vasiljevic Z. <zv...@ar...>]

On 09.05.2008, at 19:44, Vlad Seryakov wrote:

> Can you explain me about this global-state, please.
>

Ideally, one would have all stuff that is needed
loaded once. Stuff that is local to thread is
handled locally (procedures, vars that that one
thread creates). OTOH, the whole "knowledge" about
the application is loaded and held once.

Effectively, what I need is a upside-down of the
Tcl model: one interp any many threads.
Don't tell me: go shop Java. This does not compute
(for various unrelated reasons).


> and, do why do you need all separate 1000 threads for every user, is  
> the
> requirement to be connected to all at the same time? just wondering

Yes. 1000's of workstations may be connected all the time
and over a long time. And all of them are pumping data to
the server, mostly. Doing this in a event-loop mode is
possible but not feasible. I must isolate everybody in a
thread because of the potential (complex) state that each
connection may have.

Re: [naviserver-devel] moonshadow

[Vlad S. <vl...@cr...>]

Can you explain me about this global-state, please.

and, do why do you need all separate 1000 threads for every user, is the 
requirement to be connected to all at the same time? just wondering

Vasiljevic Zoran wrote:
> On 09.05.2008, at 17:38, Vasiljevic Zoran wrote:
> 
>> I saw somewhere that Lua can have global and per-thread state
>> and that you can provide your own locking primitives they use
>> to lock their own code, but I haven't dig deeper there...
> 
> Damn again. This is a global lock that serializes everything.
> A no go :-(
> Same applies to others (Ruby, Phyton, Perl).
> 
> FWIW, seems like there isn't a script language which can handle
> MT well. Must we write a new language? I knew it... We are
> always re-inventing the wheel; over and over again.
> 
> I see that I must dig even deeper that I thought initially...
> 
> 
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
> Don't miss this year's exciting event. There's still time to save $100. 
> Use priority code J8TL2D2. 
> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

[naviserver-devel] ns_register_adp

[Daniel S. <moo...@av...>]

While still migrating from aolserver to naviserver, I have found that
when using ns_register_adp to register an adp that is outside of the
page directory, naviserver returns a 500 error.  Nothing is logged.

Good --> ns_register_adp GET /hellogood "adp/hello.adp"
Bad --> ns_register_adp GET /hellobad "/home/nsadmin/hello.adp"

Daniel

-- 
| ---------------------------------------------------------------
| Daniel P. Stasinski | http://www.saidsimple.com
| moo...@av... | http://www.disabilities-r-us.com
| XMMP: moo...@av... | http://www.avenues.org
| Google Talk: mooooooo | http://www.scriptkitties.com

Re: [naviserver-devel] moonshadow

[Vasiljevic Z. <zv...@ar...>]

On 09.05.2008, at 17:38, Vasiljevic Zoran wrote:

> I saw somewhere that Lua can have global and per-thread state
> and that you can provide your own locking primitives they use
> to lock their own code, but I haven't dig deeper there...

Damn again. This is a global lock that serializes everything.
A no go :-(
Same applies to others (Ruby, Phyton, Perl).

FWIW, seems like there isn't a script language which can handle
MT well. Must we write a new language? I knew it... We are
always re-inventing the wheel; over and over again.

I see that I must dig even deeper that I thought initially...

Re: [naviserver-devel] moonshadow

[Vasiljevic Z. <zv...@ar...>]

On 09.05.2008, at 17:04, Vlad Seryakov wrote:

> But Tcl still is the core language and server will always be linked to
> Tcl library. And we still have problem with Tcl initialization.

Damn, I know that! This bugs us for about 8 years _constantly_.
The cost of a fully loaded interp is outrageous.

Recently we launched internal development of the module being
able to handle 1000+ "users" (in our product-terms, workstations)
with a stock 4 GB memory off-the-shelf machine like xserve or
a Mac Pro. If we'd to make a thread-per-workstation, we'd need
1000 * at_least_10MB of virtual memory!
No go.
So I had to write all in plain C. This of course limited the
functionality as we had to drop lots of it to buy time :-(

What Tcl really lacks is a per-thread and global state.
And I see _no_way_ how this can be made... Grmpf.

I saw somewhere that Lua can have global and per-thread state
and that you can provide your own locking primitives they use
to lock their own code, but I haven't dig deeper there...
I think I will make it a weekend project.

Cheers
Zoran