Menu
▾
▴
naviserver-devel
|
From: Zoran V. <zv...@ar...> - 2006-10-04 10:05:54
|
On 03.10.2006, at 01:01, Stephen Deasey wrote:
> ns_serialize {
> # whatever
> }
I've been thinking about that...
Basically, what you try to avoid is to replicate common
C-idioms to Tcl because Tcl is not C after all.
Would this include common threading paradigms like
mutex lock
while (true) {
cond wait cond mutex
}
mutex ulock
?
To what extent are you prepared to go?
More specifically, how would you code this
particular thing in a Tcl-like way?
|
|
From: Andrew P. <at...@pi...> - 2006-10-04 14:52:40
|
On Wed, Oct 04, 2006 at 12:05:48PM +0200, Zoran Vasiljevic wrote: > I've been thinking about that... > Basically, what you try to avoid is to replicate common > C-idioms to Tcl because Tcl is not C after all. > > Would this include common threading paradigms like > cond wait cond mutex ns_cond has exactly the same semantics and usage style as the C pthread_cond_*() functions which underly it, and thus is precisely as confusing and low-level as they are. This is annoying, especially when using ns_cond for the first time. However, it also has the important ADVANTAGE that the ns_cond implementation is simple, and that you know that ns_cond* and pthread_cond_* are in fact intended to work EXACTLY the same way. ns_cond has been extremely useful to me on the rare occasions when I needed it, but the average Naviserver user probably never uses it even once. And AOLserver (and thus Naviserver) has had ns_conf for many, many years. I bet the original ns_cond implementor couldn't think of a much better design, so he did the obvious simple thing: just wrap the ugly C APIs as is. Designing better, high level, Tcl-ish APIs is non-trivial. And if you're going to design some better high level API for waking up a thread, why even pollute your thinking with low-level "condition variable" stuff at all? Zoran, I think you've already got some nice message passing style stuff in your Threads extension, maybe it would make more sense to add some friendlier "Heh thread X, wake up, you've got work to do now!" style API there? And if so, I'd say leave ns_cond alone, it's just a low-level wrapper around pthread_cond_*, and that's fine. Futzing with it sounds likely to be wasted effort at best, and perhaps a source of very tricky destabalizing bugs at worst. -- Andrew Piskorski <at...@pi...> http://www.piskorski.com/ |
|
From: Zoran V. <zv...@ar...> - 2006-10-04 15:09:33
|
On 04.10.2006, at 16:52, Andrew Piskorski wrote: > Designing better, high level, Tcl-ish APIs is non-trivial. And if > you're going to design some better high level API for waking up a > thread, why even pollute your thinking with low-level "condition > variable" stuff at all? Zoran, I think you've already got some nice > message passing style stuff in your Threads extension, maybe it would > make more sense to add some friendlier "Heh thread X, wake up, you've > got work to do now!" style API there? > > And if so, I'd say leave ns_cond alone, it's just a low-level wrapper > around pthread_cond_*, and that's fine. Futzing with it sounds likely > to be wasted effort at best, and perhaps a source of very tricky > destabalizing bugs at worst. Every word true. I just gave it as an example to illustrate what Stephen ment by "doing the Tcl way". I know that exposing the low level C API to the Tcl level isn't going to please everybody. And this is true. Just, to get "some" things right, you need to step out of the circle and offer someting completely different. Out of the experience with the tcl threading extension I can say that people *mostly* just use it in a most-simple fashion: post scripts to threads and wait for results. But, ocasionally, they need "alternate" way of synchronization and that is where mutexes and condvars come into play. And, I must say, I really do not know how to "solve" that in a different way. I'm not even going to try. Speaking of that... the naviserver users (either current or future) may really benefit of the "alternate" thread communication I did for the Tcl threading extension (basically message-passing thing), but this is another story. |
|
From: Stephen D. <sd...@gm...> - 2006-10-04 15:51:41
|
On 10/4/06, Andrew Piskorski <at...@pi...> wrote: > On Wed, Oct 04, 2006 at 12:05:48PM +0200, Zoran Vasiljevic wrote: > > I've been thinking about that... > > Basically, what you try to avoid is to replicate common > > C-idioms to Tcl because Tcl is not C after all. > > > > Would this include common threading paradigms like > > > cond wait cond mutex > > ns_cond has exactly the same semantics and usage style as the C > pthread_cond_*() functions which underly it, and thus is precisely as > confusing and low-level as they are. This is annoying, especially > when using ns_cond for the first time. > > However, it also has the important ADVANTAGE that the ns_cond > implementation is simple, and that you know that ns_cond* and > pthread_cond_* are in fact intended to work EXACTLY the same way. > > ns_cond has been extremely useful to me on the rare occasions when I > needed it, but the average Naviserver user probably never uses it even > once. And AOLserver (and thus Naviserver) has had ns_conf for many, > many years. I bet the original ns_cond implementor couldn't think of > a much better design, so he did the obvious simple thing: just wrap > the ugly C APIs as is. > > Designing better, high level, Tcl-ish APIs is non-trivial. And if > you're going to design some better high level API for waking up a > thread, why even pollute your thinking with low-level "condition > variable" stuff at all? Zoran, I think you've already got some nice > message passing style stuff in your Threads extension, maybe it would > make more sense to add some friendlier "Heh thread X, wake up, you've > got work to do now!" style API there? > > And if so, I'd say leave ns_cond alone, it's just a low-level wrapper > around pthread_cond_*, and that's fine. ns_cond is a *literal* implementation of the underlying C API. An ns_cond which takes a condition name rather than a handle is a *faithful* implementation of the underlying C API. Which is entirely different than the hypothetical ns_serialize command which has no C equivalent. The reason that a name based ns_cond makes sense as a low level building block, deviating from the C tradition of passing a handle around, is that Tcl is a different environment. Each thread has it's own Tcl interp, yet each thread must use the same condition variable. In C this is not a problem. Memory is shared so you can use a global variable to hold the condition var, and this is really easy. In Tcl, you have to some how communicate the handle to each interp, and this is painful. Which leads to the crazy situation where it's is easier to use condition variables from C than it is in Tcl..! So yes, we need to be careful when creating APIs that when we strive for simplicity we do not make it difficult or impossible to do things a little out of the ordinary. But we also need to remember that Tcl and C are different so that a low-level command does not necessarily have to be a literal translation of the C API. |
|
From: Zoran V. <zv...@ar...> - 2006-10-06 20:11:52
|
On 06.10.2006, at 21:25, Stephen Deasey wrote:
>
> But what I'm wondering is, why you need to do this with proxy slaves?
> It seems like they don't have the same state problem that a series of
> database statements do.
>
> It's possible to send multiple Tcl commands to a single proxy slave at
> the same time:
>
> ns_proxy eval {
> set f [do_foo ...]
> set b [do_bar ...]
> return [list $f $b]
> }
>
> (You can't do that with databases because they often forbid using a
> statement separating semicolon when using prepared statements and/or
> bind variables. But Tcl doesn't have that restriction.)
I do not need it. As, as you say, I can simply send all of them
in one script. That's trivial of course.
I give you another reason pro-handle: I'd like to allocate
the handle in order to have it always (for a longer time)
available. I "reserve" it, so to speak. If I can't do that
than my code is less "predictive" in terms that I might wait
potentially for a long time to get the chance to do something
as all proxies might be doing something else. Understand?
>
> For example, any time budget you have for the statements as a whole
> must be split between each. So, if you decide each call should take
> only 30 secs, and the first call takes 1 sec but the second takes 31
> secs, you will get a timeout error with almost half your total time
> budget still to spend. But you would have been perfectly happy to wait
> 58 seconds in total, spread evenly between the two calls.
>
> Another problem might be the code you run in between the two eval
> calls. In the case of explicit handle management or the withhandle
> command, the handle is kept open even when it is not being used. If
> the code that runs between the two eval calls takes some time --
> perhaps because it blocks on IO, which may not be apparent to the
> caller -- then other threads may be prevented from using a proxy slave
> because the pool is empty. Handles which could have been returned to
> the pool and used by other callers are sitting idle in threads busy
> doing other things. This is a (temporary, mostly) resource leak.
>
Yes. This is true. And this is obviously a contra-handle as it
may lead to starvation.
> So, apart from state (if this is needed), the withhandle command is a
> speed optimisation. If you know for a fact that you're going to have
> to make sequential calls to the proxy system, you can take the pool
> lock just once. Otherwise, with implicit handle management, you take
> and release the pool lock for each evaluation.
>
> Regardless, the withhandle command allows implicit handle management
> in the common case, and a nice, clear syntax for when you explicitly
> want to manage the handle for performance or state reasons.
I must think this over...
>
>
>> For b.
>> I do not care how we call it. We can call it ns_cocacola if you like.
>> The name contest is open...
>
>
> It's an annoying thing to have to even bother about... But it is
> important. If it's not clear, people will be confused, we've seen a
> lot of that. Confused people take longer to get up to speed. People
> like new hires, which costs real money.
So, how would we call the baby? Why not just simply ns_exec?
Actually, we can build this into the core server and not as
module... The ns_exec could exec the same executable with
different command-line args that would select other main
function and not Ns_Main, for example. I'm just thinking "loud"...
>
>
>> For c.
>> I'd rather stick to explicit pool naming. I'd leave this "default"
>> to the programmer. The programmer might do something like
>> (not 100% right but serves the illustration purpose):
>>
>> ns_proxy config default
>> rename ::exec tcl::exec
>> proc ::exec args {
>> ns_proxy eval default $args
>> }
>>
>> This covers overloading of Tcl exec command. If you can convince me
>> that there are other benefits of having the default pool I can
>> think about them. I just do not see any at this point.
>
>
> A default only makes sense if it's built in. If it isn't, no on can
> rely on it and all code that uses proxy pools will have to create
> their own.
OK.
>
> Even with a built-in default, user code can certainly still create
> it's own proxy pool(s). Nothing is being taken away.
>
True.
> If it was unlikely that the default would work for much/most code,
> then it would be wrong to have one. That would be hiding something
> from programmers that they should be paying attention to. It looks to
> me though like a default pool would work for most code. But this is
> just a convenience.
>
> To manage resources efficiently you need the default pool. I can
> imagine 'exec' using the default pool, and 3rd party modules doing
> something like:
>
> ns_proxy eval -pool [ns_configvalue ... pool default] {
> ....
> }
>
> Which will just work. The site administrator can then allocate
> resources in a more fine grained way, if needed, according to the
> local situation.
>
And if we put the ns_exec into the server and make it like this:
ns_exec eval arg ?arg...?
ns_exec eval -pool thepool arg ?arg ...?
ns_exec config thepool option ?value option value ...?
etc ? Or ns_slave?
ns_slave eval arg
ns_slave config thepool option ?value option value ...?
Hm... that's not bad. What do you think?
I think ns_slave would be "opportune". The people will of course
immediately ask: where is ns_master? But I guess you can't dance
on all weddings...
Still, I will have to think about the "handle" issue for some time
(over the weekend) as I'm still not 100% convinced...
Would you integrate that in the core code or would you leave this
as a module? Actually, I keeep asking mylself why we still stick to
modules when some fuctionality is obviously needed all the time
(nslog or nscp for example)...
|
|
From: Stephen D. <sd...@gm...> - 2006-10-06 20:42:23
|
On 10/6/06, Zoran Vasiljevic <zv...@ar...> wrote: > > etc ? Or ns_slave? > > ns_slave eval arg > ns_slave config thepool option ?value option value ...? > > Hm... that's not bad. What do you think? > I think ns_slave would be "opportune". The people will of course > immediately ask: where is ns_master? But I guess you can't dance > on all weddings... That's not a bad idea. Tcl already has a concept of slave interps. The difference being that ours run out-of-process. There's a whole bunch of other stuff that goes along with Tcl slave interps, but the core is the same: you send a script to evaluate and a result comes back. > Would you integrate that in the core code or would you leave this > as a module? Actually, I keeep asking mylself why we still stick to > modules when some fuctionality is obviously needed all the time > (nslog or nscp for example)... I think we need more modules, not less... :-) |
|
From: Stephen D. <sd...@gm...> - 2006-10-06 20:32:49
|
On 10/6/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 06.10.2006, at 21:25, Stephen Deasey wrote:
> >
>
> > But what I'm wondering is, why you need to do this with proxy slaves?
> > It seems like they don't have the same state problem that a series of
> > database statements do.
> >
> > It's possible to send multiple Tcl commands to a single proxy slave at
> > the same time:
> >
> > ns_proxy eval {
> > set f [do_foo ...]
> > set b [do_bar ...]
> > return [list $f $b]
> > }
> >
> > (You can't do that with databases because they often forbid using a
> > statement separating semicolon when using prepared statements and/or
> > bind variables. But Tcl doesn't have that restriction.)
>
> I do not need it. As, as you say, I can simply send all of them
> in one script. That's trivial of course.
>
> I give you another reason pro-handle: I'd like to allocate
> the handle in order to have it always (for a longer time)
> available. I "reserve" it, so to speak. If I can't do that
> than my code is less "predictive" in terms that I might wait
> potentially for a long time to get the chance to do something
> as all proxies might be doing something else. Understand?
In this case, wouldn't you just size the pool so that there was always
a free handle?
If the pool doesn't have enough handles, and you do hang on to handles
in threads indefinitely, then you've basically got a first come, first
served system. The threads which ask first will get handles and the
rest will be permanently disappointed...
Or, maybe you need a special pool for special threads, and a default
pool for generic threads. If there are only a couple of special
threads then the pool can be small -- you don't need to size the pool
according to the many generic threads just to make sure there's always
one left over for a special thread.
|
|
From: Zoran V. <zv...@ar...> - 2006-10-06 20:36:59
|
On 06.10.2006, at 22:21, Vlad Seryakov wrote: > I vote for ns_exec and putting it into the core OK. Couple of simple thoughts agout that: ns_exec ?-pool poolname? script ?arg ...? is nice and short. It uses the default pool (as Stephen is advocating). But... how would we create pools, configure pool-wide options etc pp? As I could not figure that out, I thought it would be better NOT to specify the action (i.e. exec) but the vehicle (the slave process) hence ns_slave. Now you can easly replace ns_proxy with ns_slave and all (most) subcommands read nice... So we have now: ns_process ns_exec ns_slave ns_slave and ns_process specify the thing and subcommands the action. it is trivial to expand to include pool management commands but api is rather "large" or "clumsy". ns_exec specifies an action only. it is difficult to fit in any other "action" for pool management for example but is small and compact. Any other idea? (Sometimes it can ve REALLY difficult to give baby a name...) |
|
From: Stephen D. <sd...@gm...> - 2006-10-06 20:52:07
|
On 10/6/06, Zoran Vasiljevic <zv...@ar...> wrote: > > On 06.10.2006, at 22:21, Vlad Seryakov wrote: > > > I vote for ns_exec and putting it into the core > > OK. Couple of simple thoughts agout that: > > ns_exec ?-pool poolname? script ?arg ...? > > is nice and short. It uses the default pool > (as Stephen is advocating). > > But... how would we create pools, configure pool-wide > options etc pp? As I could not figure that out, I thought > it would be better NOT to specify the action (i.e. exec) > but the vehicle (the slave process) hence ns_slave. > Now you can easly replace ns_proxy with ns_slave and > all (most) subcommands read nice... > > So we have now: > > ns_process > ns_exec > ns_slave > > ns_slave and ns_process specify the thing and subcommands > the action. it is trivial to expand to include pool > management commands but api is rather "large" or "clumsy". > > ns_exec specifies an action only. it is difficult to fit > in any other "action" for pool management for example > but is small and compact. > > Any other idea? > > (Sometimes it can ve REALLY difficult to give baby a name...) > ns_exec works for me as well. I guess the wording doesn't work so well for the sub-commands, but it does explicitly say what's special about this command, whereas you'd have to read the docs for ns_salve. ns_process is a good idea, but it feels like a very generic word to me. It could imply an individual OS process, but I also think of it more as an action: to process: 'do stuff' to this thing: 'process the loan application, Miss Higgins...!' |
|
From: Zoran V. <zv...@ar...> - 2006-10-06 20:53:30
|
On 06.10.2006, at 22:32, Stephen Deasey wrote: > > Or, maybe you need a special pool for special threads, and a default > pool for generic threads. If there are only a couple of special > threads then the pool can be small -- you don't need to size the pool > according to the many generic threads just to make sure there's always > one left over for a special thread. So, if I can read between the lines, you say we should get rid of the handles in ns_whateverwenamethething ? Completely? Irreversibly? Just plain trash handles altogether and always lock pool, get slave, send command, wait for result, read result, lock pool, put slave back. You loose "reservation" but gain "no starvation". This is a trade-off. In such situations it is more oportune to have both. So you can do it with or w/o handles. You can "reserve" and run your commands in the handle, or you can run directly, w/o handle. This can all be the part of the API. I simply hate NOT to be able to reserve a process, open a file in it and use that descriptor for long time. I'd easily imagine how I can profit from that and trashing handles altogether just cuts me from this capability. OK. So far I can go. I could imagine an API without explicit handle usage. I can't however imagine scratching them (handles) altogether. The default pool is still giving me headaches. How would you configure default pool options? |
|
From: Zoran V. <zv...@ar...> - 2006-10-06 20:56:16
|
On 06.10.2006, at 22:53, Zoran Vasiljevic wrote: > OK. So far I can go. I could imagine an API without explicit > handle usage. I can't however imagine scratching them (handles) > altogether. What I mean by that is that the API should be able to allow both types of usage. |
|
From: Zoran V. <zv...@ar...> - 2006-10-07 15:51:46
|
On 07.10.2006, at 17:42, Vlad Seryakov wrote: > to use which, i do not think they are mutual exclusive This is the impression what I have after all those emails.... Although it sounds pretty "opportune" in the bad sense of the word, I think that a simple no-nonsense handle-free API + a handle based one are of popular benefit. |
|
From: Stephen D. <sd...@gm...> - 2006-10-07 17:00:04
|
On 10/7/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 07.10.2006, at 17:42, Vlad Seryakov wrote:
>
> > to use which, i do not think they are mutual exclusive
>
> This is the impression what I have after all those
> emails.... Although it sounds pretty "opportune" in
> the bad sense of the word, I think that a simple
> no-nonsense handle-free API + a handle based one
> are of popular benefit.
A *result* handle is still needed, but you don't need a handle to a
slave before you can send it a command.
You need a result handle for the case where you want to start the
slave running but not wait for the result immediately. So:
set id [ns_exec_queue -timeout 5 {
# long running stuff...
}]
# do other stuff here...
# later:
set result [ns_exec_wait $id]
(The above is ns_job-like: queue, wait, waitany)
But the case where you need to serially execute jobs in the slave can
be handled by either passing them all at once, or by using a
'withhandle' command.
What do you think of the withhandle command? Can you get away with
passing everything to the slave at once, and if not, does the
withhandle command cover this?
|
|
From: Zoran V. <zv...@ar...> - 2006-10-07 17:26:10
|
On 07.10.2006, at 19:00, Stephen Deasey wrote: > But the case where you need to serially execute jobs in the slave can > be handled by either passing them all at once, or by using a > 'withhandle' command. > > What do you think of the withhandle command? Can you get away with > passing everything to the slave at once, and if not, does the > withhandle command cover this? On the first glance, I'd say yes. But you know me... It may change the next second. So, let me think this over again... |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X