naviserver-devel — Developer discussion

Re: [naviserver-devel] nsproxy API

[Vlad S. <vl...@cr...>]

I like the idea of extending ns_job to handle processes, this way when i 
create queue i define what kind it is: threads or processes and then use 
it same way, no need in eval command, ns_job queue submites script to be 
evaluated to the next avail thread/process

Zoran Vasiljevic wrote:
> On 18.09.2006, at 20:55, Stephen Deasey wrote:
> 
>> API:
>>
>> result ns_exec_eval               ?-pool p? ?-timeout t?  script
>>
>> Id     ns_exec_queue  ?-detached? ?-pool p? ?-timeout t?  script
>>        ns_exec_wait                         ?-timeout t?  Id ?Id ...?
>>        ns_exec_cancel
>>
>>        ns_exec_set    ?-opt val ...?                      pool
>>        ns_exec_stats                                      pool
>>
>> Example:
>>
>>        set result [ns_exec_eval {
>>            set x y
>>            exec /bin/blah $x
>>        }]
>>
>> Notes:
>>
>>        Create pool 'default' on startup.
>>
>>        Reuse identical processes (same binary, ulimits, etc.) among  
>> pools, in
>>        the same way that threads are shared globally among ns_job  
>> queues.
>>
>>        'maxexecs' parameter, after which processes exit and are  
>> respawned.
> 
> First impression:
> 
>    a. I'd rather use ns_cmdtobenamed queue pool ...
>                      ns_cmdtobenamed eval  pool ...
> 
>       Two things here: ns_exec_command I do not like. The "Tcl" way  
> would
>       be actually to use a namespace to encapsulate all commands but  
> that
>       would break the current style.
>       The oher thing, I do not like the "default" pool. I think the pool
>       (being the place to put common things shared across all slaves)
>       should be given explicitly.
> 
>    b. If we go that far then ns_exec_xxx is also not a suitable name as
>       we are not exec'ing anything really.
> 
> But lets go two steps back...
> 
> You say you wanted to get ns_job and ns_proxy together. You also say
> that this is not feasible because of the "eval" command.
> Why?
> The ns_job could be expanded easily to integrate the "eval" command,
> but it would not make much sense (i.e. practical use). OTOH, the
> ns_proxy eval has much sense (i.e. practical use). Given that, in my
> opinion, [ns_job eval] would not harm anybody, why _not_ integrating  
> both?
> Instead of aueue of threads, youd create queue of processes. Most (if
> not all) of other ns_job commands will just be "blind" in terms of
> threads vs. processes. Each process would actually be the nsd itself
> but using alternate Ns_Main function. It would load the libnsd.so, read
> the same (or alternate) configuration file, etc. pp.
> 
> This way the API question is solved. The work to be done is pretty clear
> (rewrite ns_job internals to allow handling of processes in addition to
> threads).
> 
> Is there a very good reason why not going that way instead of
> maintaining the proxy module ?
> 
> 
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] nsproxy API

[Zoran V. <zv...@ar...>]

On 18.09.2006, at 20:55, Stephen Deasey wrote:

> API:
>
> result ns_exec_eval               ?-pool p? ?-timeout t?  script
>
> Id     ns_exec_queue  ?-detached? ?-pool p? ?-timeout t?  script
>        ns_exec_wait                         ?-timeout t?  Id ?Id ...?
>        ns_exec_cancel
>
>        ns_exec_set    ?-opt val ...?                      pool
>        ns_exec_stats                                      pool
>
> Example:
>
>        set result [ns_exec_eval {
>            set x y
>            exec /bin/blah $x
>        }]
>
> Notes:
>
>        Create pool 'default' on startup.
>
>        Reuse identical processes (same binary, ulimits, etc.) among  
> pools, in
>        the same way that threads are shared globally among ns_job  
> queues.
>
>        'maxexecs' parameter, after which processes exit and are  
> respawned.

First impression:

   a. I'd rather use ns_cmdtobenamed queue pool ...
                     ns_cmdtobenamed eval  pool ...

      Two things here: ns_exec_command I do not like. The "Tcl" way  
would
      be actually to use a namespace to encapsulate all commands but  
that
      would break the current style.
      The oher thing, I do not like the "default" pool. I think the pool
      (being the place to put common things shared across all slaves)
      should be given explicitly.

   b. If we go that far then ns_exec_xxx is also not a suitable name as
      we are not exec'ing anything really.

But lets go two steps back...

You say you wanted to get ns_job and ns_proxy together. You also say
that this is not feasible because of the "eval" command.
Why?
The ns_job could be expanded easily to integrate the "eval" command,
but it would not make much sense (i.e. practical use). OTOH, the
ns_proxy eval has much sense (i.e. practical use). Given that, in my
opinion, [ns_job eval] would not harm anybody, why _not_ integrating  
both?
Instead of aueue of threads, youd create queue of processes. Most (if
not all) of other ns_job commands will just be "blind" in terms of
threads vs. processes. Each process would actually be the nsd itself
but using alternate Ns_Main function. It would load the libnsd.so, read
the same (or alternate) configuration file, etc. pp.

This way the API question is solved. The work to be done is pretty clear
(rewrite ns_job internals to allow handling of processes in addition to
threads).

Is there a very good reason why not going that way instead of
maintaining the proxy module ?

Re: [naviserver-devel] SVN repository

[Bernd E. <eid...@we...>]

Hi Vlad,

> Of course i almost all the time work with
> HEAD so i am the simplest case.

At the office we followed the "default" SVN approach with a directory 
structure of

# svn list file:///repository/projectname
branches/
tags/
trunk/

It's really nice to simply copy from trunk (or local working copy) into a new 
directory below "tags/" and have a "tag". Or a "branch". It's a cheap copy, 
only activated when you commit to it later on.
It's not all peaches and cream, but I like the easy way of moving files and 
directories around or the transactional approach.

Bernd.

Re: [naviserver-devel] Simple http proxy handler

[Michael A. C. <cle...@gm...>]

On 9/18/06, Stephen Deasey <sd...@gm...> wrote:
> I was wondering why registering a proxy handler is different than a
> regular registered proc.  Do they have to be different?

As I've been reading the NaviServer sources to acquaint myself I was
wondering the same thing.

> The main difference seems to be that you can register a proxy for a
> protocol, i.e. http or https, but you don't have that option with
> regular registered procs.
>
> Is this an important distinction to make though?  Registered procs can
> still check the protocol, manually, if they care.

I don't think it is an important decision to make because the
registered proc can already check in the cases it matters.  Back in my
ACS 4.2 days I'd regularly introspect whether the conn was ssl or not
in a registered proc.

> Re having to register for each method, that also is sometimes a
> problem for registered procs. Maybe allowing something like:
>
>   ns_register_proc * /* callback
>
> Where the only supported symbol is '*', meaning 'default, if no more
> specific method is registered'.  Perhaps that's too confusing --
> people will expect full glob support.  I don't think glob support is
> possible, or even desirable here.  Url handlers are supposed to be
> deterministic, only one will match some URL (unlike filters).

I think the "*" would confuse peope (re: globbing).  What about the
empty string? After all, a regular expression of "" will match any
input, so there is a conceptual linkage available.

> That's what occurs to me off the top of my head...

Ditto.

Michael

[naviserver-devel] SVN repository

[Vlad S. <vl...@cr...>]

I converted and loaded current CVS HEAD into naviserver SVN repository, 
CVS is still primary but we can play with SVN and decide to switch or 
keep CVS. Personally, i switched to SVN more than a year ago in all my 
projects and very happy with it, much better than CVS, especially with 
directory and files handling. Of course i almost all the time work with 
HEAD so i am the simplest case.


Migrating process may take some time, so i think tomorrow SVN repo will 
be ready to use.

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] nsproxy API

[Zoran V. <zv...@ar...>]

On 18.09.2006, at 23:03, Stephen Deasey wrote:

>
> I mean that
>
>     Ns_ProxyGet()
>
> might be appropriate for C programmers, but that doesn't mean that
>
>     ns_proxy get
>
> is for Tcl programmers.  It's not just about level of difficulty
> either. Mapping the low level C APIs directly seems to add a lot of
> difficulty, because you have to take account of the Tcl environment
> and all the crazy things that you just don't allow in the C API.

Aha. Well, in that particular case, the [ns_proxy get] is not bad at  
all!
You get the handle of "something" that you use to do all kinds of  
things.
Later on, when you're done with it, you [ns_proxy put] itback, allowing
others to use the same "something"

How would you make this in a "Tcl way"?

After  all, you [open] the channel, then you [puts] to it and then
[close] it. What is not OK here?

I think I know where are you heading and I can understand the idea
but this particular example may not be optimal.

I believe you are thinking about something like this:

   ns_mutex lock $lock
   do bla bla bla
   ns_mutex unlock $lock

The "optimal" way to do that would really be:

    ns_mutex lock $lock {
       do bla bla bla
    }

as it will save you the trouble if "do bla bla" throws error.
In the latter case you must ugly things like

   ns_mutex lock $lock
   if {[catch {do bla bla} err]} {
       ns_mutex unlock $lock
       error $err
   }
   ns_mutex unlock $lock

which is just plain rubbish (because of which we had to invent
our own do/catch/finally command)...

I will take more time to answer on your nsproxy email
as this two are really comming close...

Re: [naviserver-devel] hosting java-script server-side?

[Zoran V. <zv...@ar...>]

On 18.09.2006, at 22:59, Stephen Deasey wrote:

>
> Yeah, and how popular are the Netscape servers?

Well, not much, but this is not because of the JS
as we all know.

>
> Tcl sucks, that's for sure. But it rules in these three very  
> important areas:
>
> * It's thread safe (and thread friendly)
> * It's utf-8 safe
> * You can write domain specific languages in it.

I find Tcl actually very nice. Admitently, it could be
facelifted to allow OO programming, as this helps a lot
in some types of applications (we use XOTcl for that).

But lets stop at this point, as we do not want to
start yet another language-flame. Our sister-project
is responsible for that kind of flames :-)

Re: [naviserver-devel] hosting java-script server-side?

[Vlad S. <vl...@cr...>]

> 
> People are writing a lot of sucky Tcl for AOLserver.  Here's an
> example from a couple of days ago:
> 
> 
> proc startworker {name} {
>        nsv_set $name mutex [ns_mutex create]
>         nsv_set $name cond  [ns_cond create]
>        nsv_set $name queue {}
>        nsv_set $name tid [ns_thread begindetached [list workermain $name]]
> }
> 
> Jumping through hoops with low level mutexes and and variables etc.,
> you see it all the time.  Why can't we have code that looks like this?
> 
>     ns_serialize {
>         # Only one copy of this code runs at a time
>     }
> 
> 
> Which brings us back to nsproxy and handle management... :-)


You just showed that it is not Tcl sucks but a programmer that writes 
bad code in Tcl without using it properly or not in full. That example 
of your shows that programmer that wrote it has more experience in C and 
  uses Tcl in pure procedure way, which is not that bad, the code works.

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] nsproxy API

[Stephen D. <sd...@gm...>]

On 9/18/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 18.09.2006, at 20:55, Stephen Deasey wrote:
>
> >  it's the explicit handle management that's making
> > things different.  Low level C APIs just don't map directly into C.
>
> What C API's ??
>
> Ns_ProxyGet
> Ns_ProxyPut
> Ns_ProxyEval
>
> are the only API's that C programmer might use.
> They are perfectly symetric and most simple.
> You Ns_ProxyGet a proxy, then you Ns_ProxyEval
> something in it (possibly many times) and then
> you Ns_ProxyPut it back. No locking. What could
> be simpler than that?


I mean that

    Ns_ProxyGet()

might be appropriate for C programmers, but that doesn't mean that

    ns_proxy get

is for Tcl programmers.  It's not just about level of difficulty
either. Mapping the low level C APIs directly seems to add a lot of
difficulty, because you have to take account of the Tcl environment
and all the crazy things that you just don't allow in the C API.

Re: [naviserver-devel] Simple http proxy handler

[Vlad S. <vl...@cr...>]

It will require 2 hash calls but currently it results in error, so 2 
calls will not be that bad if default handler will be able to process 
request.

Stephen Deasey wrote:
> On 9/18/06, Vlad Seryakov <vl...@cr...> wrote:
>>> I don't know, I'm asking...
>>>
>>> I was wondering why registering a proxy handler is different than a
>>> regular registered proc.  Do they have to be different?
>>>
>>> The main difference seems to be that you can register a proxy for a
>>> protocol, i.e. http or https, but you don't have that option with
>>> regular registered procs.
>> correct
>>
>>> Is this an important distinction to make though?  Registered procs can
>>> still check the protocol, manually, if they care.
>> right
>>
>>> Re having to register for each method, that also is sometimes a
>>> problem for registered procs. Maybe allowing something like:
>>>
>>>   ns_register_proc * /* callback
>>>
>>> Where the only supported symbol is '*', meaning 'default, if no more
>>> specific method is registered'.  Perhaps that's too confusing --
>>> people will expect full glob support.  I don't think glob support is
>>> possible, or even desirable here.  Url handlers are supposed to be
>>> deterministic, only one will match some URL (unlike filters).
>> Agree, i will take a look. but glob does not work with url-specific
>> hashes and we want to keep them, they are very fast
>>
> 
> Agree. Maybe it was confusing to suggest using '*' syntax to mean
> 'default' and not 'globbing pattern'.
> 
> What do you think to allowing defaults for methods?
> 
> i.e. If a GET request comes in, and a GET handler is registered, run
> it. This will be just as quick as now.
> 
> If a PROPFIND request comes in, there is no PROPFIND handler, but
> there is a default handler, run the default handler.
> 
> I'm not sure if this *is* feasable.  But sometimes it seems like this
> would be useful.
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] hosting java-script server-side?

[Stephen D. <sd...@gm...>]

On 9/18/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 18.09.2006, at 19:58, Stephen Deasey wrote:
>
> >
> >
> > Are you running a fever?
>
> Ah no! I'm just trying to see what others think.
> Well, all responses so far were not exactly
> in favour of that but that doesn't matter.
> I'm not going to start working on that in short
> nor in mid term.  But, this MIGHT ease newbie
> acceptance, don't you think?
> After all, Netscape did that already some years
> ago.... So it is not just out of the blue...
>

Yeah, and how popular are the Netscape servers?

Tcl sucks, that's for sure. But it rules in these three very important areas:

* It's thread safe (and thread friendly)
* It's utf-8 safe
* You can write domain specific languages in it.

I think it's the last point which makes up for the suckage, and which
is also the main selling point of Ruby.  When you wite a Rails
application, what you end up with *looks* like it does what it does --
a python program always seem to look like just another python program.

People are writing a lot of sucky Tcl for AOLserver.  Here's an
example from a couple of days ago:


proc startworker {name} {
       nsv_set $name mutex [ns_mutex create]
        nsv_set $name cond  [ns_cond create]
       nsv_set $name queue {}
       nsv_set $name tid [ns_thread begindetached [list workermain $name]]
}

Jumping through hoops with low level mutexes and and variables etc.,
you see it all the time.  Why can't we have code that looks like this?

    ns_serialize {
        # Only one copy of this code runs at a time
    }


Which brings us back to nsproxy and handle management... :-)

Re: [naviserver-devel] nsproxy API

[Zoran V. <zv...@ar...>]

On 18.09.2006, at 20:55, Stephen Deasey wrote:

>  it's the explicit handle management that's making
> things different.  Low level C APIs just don't map directly into C.

What C API's ??

Ns_ProxyGet
Ns_ProxyPut
Ns_ProxyEval

are the only API's that C programmer might use.
They are perfectly symetric and most simple.
You Ns_ProxyGet a proxy, then you Ns_ProxyEval
something in it (possibly many times) and then
you Ns_ProxyPut it back. No locking. What could
be simpler than that?

Re: [naviserver-devel] Simple http proxy handler

[Stephen D. <sd...@gm...>]

On 9/18/06, Vlad Seryakov <vl...@cr...> wrote:
> >
> > I don't know, I'm asking...
> >
> > I was wondering why registering a proxy handler is different than a
> > regular registered proc.  Do they have to be different?
> >
> > The main difference seems to be that you can register a proxy for a
> > protocol, i.e. http or https, but you don't have that option with
> > regular registered procs.
>
> correct
>
> > Is this an important distinction to make though?  Registered procs can
> > still check the protocol, manually, if they care.
>
> right
>
> > Re having to register for each method, that also is sometimes a
> > problem for registered procs. Maybe allowing something like:
> >
> >   ns_register_proc * /* callback
> >
> > Where the only supported symbol is '*', meaning 'default, if no more
> > specific method is registered'.  Perhaps that's too confusing --
> > people will expect full glob support.  I don't think glob support is
> > possible, or even desirable here.  Url handlers are supposed to be
> > deterministic, only one will match some URL (unlike filters).
>
> Agree, i will take a look. but glob does not work with url-specific
> hashes and we want to keep them, they are very fast
>

Agree. Maybe it was confusing to suggest using '*' syntax to mean
'default' and not 'globbing pattern'.

What do you think to allowing defaults for methods?

i.e. If a GET request comes in, and a GET handler is registered, run
it. This will be just as quick as now.

If a PROPFIND request comes in, there is no PROPFIND handler, but
there is a default handler, run the default handler.

I'm not sure if this *is* feasable.  But sometimes it seems like this
would be useful.

Re: [naviserver-devel] Simple http proxy handler

[Vlad S. <vl...@cr...>]

> 
> I don't know, I'm asking...
> 
> I was wondering why registering a proxy handler is different than a
> regular registered proc.  Do they have to be different?
> 
> The main difference seems to be that you can register a proxy for a
> protocol, i.e. http or https, but you don't have that option with
> regular registered procs.

correct

> Is this an important distinction to make though?  Registered procs can
> still check the protocol, manually, if they care.

right

> Re having to register for each method, that also is sometimes a
> problem for registered procs. Maybe allowing something like:
> 
>   ns_register_proc * /* callback
> 
> Where the only supported symbol is '*', meaning 'default, if no more
> specific method is registered'.  Perhaps that's too confusing --
> people will expect full glob support.  I don't think glob support is
> possible, or even desirable here.  Url handlers are supposed to be
> deterministic, only one will match some URL (unlike filters).

Agree, i will take a look. but glob does not work with url-specific 
hashes and we want to keep them, they are very fast


-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] hosting java-script server-side?

[Zoran V. <zv...@ar...>]

On 18.09.2006, at 19:58, Stephen Deasey wrote:

>
>
> Are you running a fever?

Ah no! I'm just trying to see what others think.
Well, all responses so far were not exactly
in favour of that but that doesn't matter.
I'm not going to start working on that in short
nor in mid term.  But, this MIGHT ease newbie
acceptance, don't you think?
After all, Netscape did that already some years
ago.... So it is not just out of the blue...

Re: [naviserver-devel] Simple http proxy handler

[Stephen D. <sd...@gm...>]

On 9/18/06, Vlad Seryakov <vl...@cr...> wrote:
> >
> > Because this has never been exposed to Tcl before, and because seems
> > to be (to me) a genuine bug which could bight us as we add more HTTP
> > 1.1 support, it seems like now would be the time to take a step back
> > and look at what we really want in this API.
> >
> > It's only one call to register, one to unregister, and some Tcl
> > wrappers.  Do you think it's not worth it?
> >
>
> Ok, what is the new API and logic you want to add? Or i missed it?
>


I don't know, I'm asking...

I was wondering why registering a proxy handler is different than a
regular registered proc.  Do they have to be different?

The main difference seems to be that you can register a proxy for a
protocol, i.e. http or https, but you don't have that option with
regular registered procs.

Is this an important distinction to make though?  Registered procs can
still check the protocol, manually, if they care.

Re having to register for each method, that also is sometimes a
problem for registered procs. Maybe allowing something like:

  ns_register_proc * /* callback

Where the only supported symbol is '*', meaning 'default, if no more
specific method is registered'.  Perhaps that's too confusing --
people will expect full glob support.  I don't think glob support is
possible, or even desirable here.  Url handlers are supposed to be
deterministic, only one will match some URL (unlike filters).

That's what occurs to me off the top of my head...

Re: [naviserver-devel] nsproxy API

[Zoran V. <zv...@ar...>]

On 18.09.2006, at 20:55, Stephen Deasey wrote:

>
> Naming is easy -- it's the explicit handle management that's making
> things different.  Low level C APIs just don't map directly into C.
> For example, you never expect a C programmer to grab a handle then
> leave it dangling, that's just broken code.  But with Tcl this sort of
> thing is expected.  You have to clean up, take extra precautions with
> locking, and so on.
>
> As far as I can see, the handles aren't needed.  How about  
> something like this:
>
>
> API:
>
> result ns_exec_eval               ?-pool p? ?-timeout t?  script
>
> Id     ns_exec_queue  ?-detached? ?-pool p? ?-timeout t?  script
>        ns_exec_wait                         ?-timeout t?  Id ?Id ...?
>        ns_exec_cancel
>
>        ns_exec_set    ?-opt val ...?                      pool
>        ns_exec_stats                                      pool
>
> Example:
>
>        set result [ns_exec_eval {
>            set x y
>            exec /bin/blah $x
>        }]
>
> Notes:
>
>        Create pool 'default' on startup.
>
>        Reuse identical processes (same binary, ulimits, etc.) among  
> pools, in
>        the same way that threads are shared globally among ns_job  
> queues.
>
>        'maxexecs' parameter, after which processes exit and are  
> respawned.
>
>
>
> As well as not being very descriptive, the name 'proxy' also clashes
> with genuine HTTP proxy functions such as ns_register_proxy.  The
> proxy functionality that Vlad's been adding recently obviously should
> be in a a module called nsproxy...
>
> Some things are definitely version 2, ulimits for example.  It would
> be a pain to get stuck with an awkward API though.
>
> What do you think?

All kinds of good ideas... I will comment on that tomorrow
when I'm back to work.

>
>
> I noticed a couple of things while looking through the code:
>
>
>
> nsproxymod.c: SrvMod structure is statically allocated, one per
> process, but is assigned to each time the module is loaded,
> potentially x times per virtual server.

Yes. Incorrect. Must fix that. There is no harm except
memory leaking... but this is bad enough.

>
>
> nsproxylib.c, ProxyPop(): Extra call to Ns_CondBroadcast() without
> lock being held.
>
>
>     if (err != ENone) {
>         while ((proxyPtr = firstPtr) != NULL) {
>             firstPtr = proxyPtr->nextPtr;
>             PushProxy(proxyPtr);
>         }
>         Ns_CondBroadcast(&poolPtr->cond);
>         return TCL_ERROR;
>     }

?? You need not necessarily hold a locked mutex to broadcast the  
condition
varible. In this case the PushProxy is making necessary locks.
The Ns_CB is called here to release waiters at line 2067 or 2070
after we have retuned pop'ed proxies back to the pool in case of error.

>
>
> nsproxylib.c, ProxyPush(): Ns_CondBroadcast() called whether proxy is
> returned to pool or not.

OK. This is true. Actually, we only need this one. The other Ns_CB  
(above)
can be removed. Will make necessary modifications.

>
>     Ns_MutexLock(&poolPtr->lock);
>     poolPtr->nused--;
>     poolPtr->nfree++;
>     if ((poolPtr->nused + poolPtr->nfree) <= poolPtr->max) {
>         proxyPtr->nextPtr = poolPtr->firstPtr;
>         poolPtr->firstPtr = proxyPtr;
>         if (proxyPtr->procPtr) {
>             SetExpire(proxyPtr->procPtr, proxyPtr->timers.tidle);
>         }
>         proxyPtr->timers = poolPtr->timers;
>         proxyPtr = NULL;
>     } else {
>         poolPtr->nfree--;
>     }
>     Ns_CondBroadcast(&poolPtr->cond);
>     Ns_MutexUnlock(&poolPtr->lock);
>
>
> GetPool(): Proxy pool is created if name pool name does not already
> exist. What about typos? This is error prone. Alternative: always
> create a pool 'default' on start up and make specifying pool names
> optional?

This was the original code I just took as-is and made "usable" and
working. Of yourse, if you change this then all other options are
open.... But more on that tomorrow.

>
> The term 'procPtr' is confusing when used to mean pointer to Proxy
> Slave. In the rest of the code base, procPtr means pointer to C
> function, i.e. a callback.


How about slavePtr?

Cheers
Zoran

Re: [naviserver-devel] Simple http proxy handler

[Vlad S. <vl...@cr...>]

> 
> Because this has never been exposed to Tcl before, and because seems
> to be (to me) a genuine bug which could bight us as we add more HTTP
> 1.1 support, it seems like now would be the time to take a step back
> and look at what we really want in this API.
> 
> It's only one call to register, one to unregister, and some Tcl
> wrappers.  Do you think it's not worth it?
> 

Ok, what is the new API and logic you want to add? Or i missed it?

Re: [naviserver-devel] Simple http proxy handler

[Stephen D. <sd...@gm...>]

On 9/18/06, Vlad Seryakov <vl...@cr...> wrote:
> More code i meant to implement the whole full-blown proxy, was not
> clear. As for activation, this is how the core works, everything is a
> callback using url-specific hash entry

Yeah, and I'm saying it's broken because the HTTP 1.1 spec says that
normal clients, *not* expecting to be talking to a proxy, are supposed
to send the full protocol and host.  In our code, that will activate a
*proxy* callback, but it should just be a a normal HTTP response for
whatever the server has for that URL

At least, that's how I read the RFC.

That's why I say that although adding Tcl access to this stuff is
good, it's going in the wrong direction by exposing a buggy
implementation.

The concept of the current (before you) code is also weak, due to the
need to register a callback for each and every method to be proxied.

Because this has never been exposed to Tcl before, and because seems
to be (to me) a genuine bug which could bight us as we add more HTTP
1.1 support, it seems like now would be the time to take a step back
and look at what we really want in this API.

It's only one call to register, one to unregister, and some Tcl
wrappers.  Do you think it's not worth it?



> so current proxy implementation
> works the same way, just using protocol/method instead of method/url for
> regular HTTP. So i am not sure if we need to re-write this.
>

Re: [naviserver-devel] Simple http proxy handler

[Vlad S. <vl...@cr...>]

More code i meant to implement the whole full-blown proxy, was not 
clear. As for activation, this is how the core works, everything is a 
callback using url-specific hash entry, so current proxy implementation 
works the same way, just using protocol/method instead of method/url for 
regular HTTP. So i am not sure if we need to re-write this.

Name clash exists, but the module can be named as nshttpproxy

Stephen Deasey wrote:
> 
> Well, no.  You could transparent proxy right now by using
> ns_register_proc and ns_http to forward the request.  It's *less*
> code.
> 
> There's two separate issues: where the HTTP proxy code lives; and how
> it is activated.  The only think stopping you moving the code to the
> nsproxy module -- which is the right thing to do even if you don't
> plan to immediately add more to the simple implementation you have now
> -- is the naming clash with Zoran's bundled nsproxy module.  The
> activation stuff I guess needs some thought.
> 
> Exposing this as Tcl commands is definitely a good move. But with the
> activation bug and naming confusion it's sort of going in the wrong
> direction.  It would be great to set things off on the right foot so
> that people could more easily contribute.
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Simple http proxy handler

[Stephen D. <sd...@gm...>]

On 9/18/06, Vlad Seryakov <vl...@cr...> wrote:
> It started as small proxy to handle special cases when we need access to
> private LAN inside our LAN to provision some servers via WEB. I never
> intended to make it full HTTP proxy but even in current state it does
> work and we access our appliance boxes though this module.
>
> To make it more usefull, it needs to be in separate module, i agree.
> Currently it uses internal proxy which is in the core, that's why
> specific register by protocol method, this is how it defined and used in
> the op.c and used in queue.c.
>
> Not sure about transparent proxy on Tcl level, this needs a lot of code
>   it is more like proxy driver as i see it.


Well, no.  You could transparent proxy right now by using
ns_register_proc and ns_http to forward the request.  It's *less*
code.

There's two separate issues: where the HTTP proxy code lives; and how
it is activated.  The only think stopping you moving the code to the
nsproxy module -- which is the right thing to do even if you don't
plan to immediately add more to the simple implementation you have now
-- is the naming clash with Zoran's bundled nsproxy module.  The
activation stuff I guess needs some thought.

Exposing this as Tcl commands is definitely a good move. But with the
activation bug and naming confusion it's sort of going in the wrong
direction.  It would be great to set things off on the right foot so
that people could more easily contribute.

Re: [naviserver-devel] Simple http proxy handler

[Vlad S. <vl...@cr...>]

It started as small proxy to handle special cases when we need access to 
private LAN inside our LAN to provision some servers via WEB. I never 
intended to make it full HTTP proxy but even in current state it does 
work and we access our appliance boxes though this module.

To make it more usefull, it needs to be in separate module, i agree.
Currently it uses internal proxy which is in the core, that's why 
specific register by protocol method, this is how it defined and used in 
the op.c and used in queue.c.

Not sure about transparent proxy on Tcl level, this needs a lot of code 
  it is more like proxy driver as i see it.

Stephen Deasey wrote:
> tcl/http.tcl:
> 
> if {[ns_config -bool ns/server/[ns_info server] enablehttpproxy off]} {
>     ns_register_proxy GET  http ns_proxy_handler_http
>     ns_register_proxy POST http ns_proxy_handler_http
>     nsv_set ns:proxy allow [ns_config ns/server/[ns_info server] allowhttpproxy]
> }
> 
> 
> This is going to break on the first HEAD request...
> 
> This really should be a module of it's own, in the external modules
> directory.  So far the functionality is pretty limited and arbitrary,
> but obviously really useful.  But there's a lot that needs to be added
> to be a fully useful proxy implementation, and probably a lot of
> people who would be interested.  This really would benefit from being
> an external module, with it's own release cycle etc.
> 
> Some thought needs to go into the API for registering proxy handlers.
> 
> At the moment the handler is triggered exclusively by the client
> sending the full protocol and host  in the request line, i.e.:
> 
>     GET http://foo.com/blah.html HTTP/1.0
> 
> The problem is that the spec says HTTP 1.1 clients are supposed to
> send the full protocol and host on each request.  If we move it full
> HTTP 1.1 compliance, this is going tp break.
> 
> As it stands, the proxy code is only triggered if the client is
> explicitly configured to treat the server as it's proxy, but it's also
> useful to have a transparent proxy, e.g. to send some section of the
> URL hierarchy to a legacy Apache server :-)
> 
> So, removing the requirement that the full protocol and host needed to
> trigger the proxy code isn't such a problem, it would enable
> transparent proxies.
> 
> Registering proxy handlers to a specific method / protocol combination
> seems wrong to me.  Even if you added HEAD to the list of methods in
> the configuration, what happens when a client send something else?
> Shouldn't it be passed through to the back end server, which perhaps
> does know how to handle the method?
> 
> 
> We needs some ideas for what the API for registering proxy handlers
> should look like.  Do we even need one?  Is the existing
> ns_register_proc not sufficient?  Perhaps if we added the ability to
> register a proc for a 'default' method, *, that would do?
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

[naviserver-devel] Simple http proxy handler

[Stephen D. <sd...@gm...>]

tcl/http.tcl:

if {[ns_config -bool ns/server/[ns_info server] enablehttpproxy off]} {
    ns_register_proxy GET  http ns_proxy_handler_http
    ns_register_proxy POST http ns_proxy_handler_http
    nsv_set ns:proxy allow [ns_config ns/server/[ns_info server] allowhttpproxy]
}


This is going to break on the first HEAD request...

This really should be a module of it's own, in the external modules
directory.  So far the functionality is pretty limited and arbitrary,
but obviously really useful.  But there's a lot that needs to be added
to be a fully useful proxy implementation, and probably a lot of
people who would be interested.  This really would benefit from being
an external module, with it's own release cycle etc.

Some thought needs to go into the API for registering proxy handlers.

At the moment the handler is triggered exclusively by the client
sending the full protocol and host  in the request line, i.e.:

    GET http://foo.com/blah.html HTTP/1.0

The problem is that the spec says HTTP 1.1 clients are supposed to
send the full protocol and host on each request.  If we move it full
HTTP 1.1 compliance, this is going tp break.

As it stands, the proxy code is only triggered if the client is
explicitly configured to treat the server as it's proxy, but it's also
useful to have a transparent proxy, e.g. to send some section of the
URL hierarchy to a legacy Apache server :-)

So, removing the requirement that the full protocol and host needed to
trigger the proxy code isn't such a problem, it would enable
transparent proxies.

Registering proxy handlers to a specific method / protocol combination
seems wrong to me.  Even if you added HEAD to the list of methods in
the configuration, what happens when a client send something else?
Shouldn't it be passed through to the back end server, which perhaps
does know how to handle the method?


We needs some ideas for what the API for registering proxy handlers
should look like.  Do we even need one?  Is the existing
ns_register_proc not sufficient?  Perhaps if we added the ability to
register a proc for a 'default' method, *, that would do?

[naviserver-devel] nsproxy API

[Stephen D. <sd...@gm...>]

The API for nsproxy looks like a bit of an accident to me. The module
is based on the external db driver interface, hence the name 'proxy'
and the explicit handle management. But 'proxy' really doesn't apply
here, and the worst part of the nsdb module has always been the handle
management...

Also, the module presents an async API but doesn't follow the lead of
the existing ns_job (very similar aims) or ns_http.  For comparison,
here's ns_proxy, ns_job, ns_http and some threading commands:


   *** Proxy Pools ***

ns_proxy get       pool ?-opt val ...? (get proxy handle from pool)
ns_proxy put       handle              (return proxy handle to pool)
ns_proxy release   handle              (identical to put)
ns_proxy ping      handle              (is slave responding?)
ns_proxy cleanup                       (release all handles for interp)

ns_proxy eval      handle script       (send script, wait for and recv result)

ns_proxy send      handle script       (send script to slave)
ns_proxy wait      handle ?timeout?    (wait for result from salve)
ns_proxy recv      handle              (get the result from slave)

ns_proxy pools                         (list all pools for all servers)
ns_proxy configure pool ?opt val ...?  (create new / cfg existing pool)

ns_proxy free      pool                (list free slaves in pool)
ns_proxy active    pool                (list busy slaves in pool)
ns_proxy handles                       (list allocated handles for interp)
ns_proxy clear     pool ?handle?       (close all slaves in all pools/pool)
ns_proxy stop      pool ?handle?       (close running slaves in all pools/pool)

(NB: 'pools', 'clear' and 'stop' sub-commands not virtual-server safe)


   *** Job Queues ***

ns_job create  ?-desc d?    queueId ?maxT? (create new q)
ns_job delete               queueId        (delete q when jobs complete)

ns_job queues                              (list of all queues)
ns_job queuelist                           (list of all q's w/q-info)
ns_job jobs                 queueId        (list of jobIds for q)
ns_job joblist              queueId        (list of jobIds w/job-info for q)
ns_job threadlist                          (return min/max threads info)

ns_job queue   ?-detached?  queueId script (send script to queue, ret jobid)
ns_job wait    ?-timeout t? queueId jobId  (fech job result)
ns_job waitany ?-timeout t? queueId        (fetch next result from q)
ns_job cancel               queueId jobId  (discard pending result)

ns_job genid                               (generate a unique q name)

(q names are not virtual-server safe)


   *** Async http requests ***

ns_http queue  ?qflags? url  (open conn to server, send task to io thread)
ns_http wait   ?wflags? hId  (wait for http opp result from io thread)

ns_http run    ?qflags? url  (queue and wait)

ns_http cancel          hId  (abort given http opp)

ns_http cleanup              (abort all opps in current interp)


    *** Threads ***

ns_thread begin         script (run script)
ns_thread begindetached script (run script, discard result, exit thread)
ns_thread wait          tId    (get result of non-detached thread)



The ns_job command isn't great -- it shares the same brokenness with
ns_proxy in that job queue names are global, like proxy pool names.
There's some ideas though:

- 'queue' initiates an action, returns an opaque handle which can be waited
  upon for the result.
- 'wait' waits for the result of a queued action.
- 'cancel' discards the result of a pending action.
- 'detached' actions can not be waited upon -- they do not have any result (or
  the result is discarded)


I was thinking that as ns_job and ns_proxy were so similar they could
be merged, but there is a significant difference: the 'eval' command
makes sense for ns_proxy but not for ns_job.

Naming is easy -- it's the explicit handle management that's making
things different.  Low level C APIs just don't map directly into C.
For example, you never expect a C programmer to grab a handle then
leave it dangling, that's just broken code.  But with Tcl this sort of
thing is expected.  You have to clean up, take extra precautions with
locking, and so on.

As far as I can see, the handles aren't needed.  How about something like this:


API:

result ns_exec_eval               ?-pool p? ?-timeout t?  script

Id     ns_exec_queue  ?-detached? ?-pool p? ?-timeout t?  script
       ns_exec_wait                         ?-timeout t?  Id ?Id ...?
       ns_exec_cancel

       ns_exec_set    ?-opt val ...?                      pool
       ns_exec_stats                                      pool

Example:

       set result [ns_exec_eval {
           set x y
           exec /bin/blah $x
       }]

Notes:

       Create pool 'default' on startup.

       Reuse identical processes (same binary, ulimits, etc.) among pools, in
       the same way that threads are shared globally among ns_job queues.

       'maxexecs' parameter, after which processes exit and are respawned.



As well as not being very descriptive, the name 'proxy' also clashes
with genuine HTTP proxy functions such as ns_register_proxy.  The
proxy functionality that Vlad's been adding recently obviously should
be in a a module called nsproxy...

Some things are definitely version 2, ulimits for example.  It would
be a pain to get stuck with an awkward API though.

What do you think?


I noticed a couple of things while looking through the code:



nsproxymod.c: SrvMod structure is statically allocated, one per
process, but is assigned to each time the module is loaded,
potentially x times per virtual server.


nsproxylib.c, ProxyPop(): Extra call to Ns_CondBroadcast() without
lock being held.


    if (err != ENone) {
        while ((proxyPtr = firstPtr) != NULL) {
            firstPtr = proxyPtr->nextPtr;
            PushProxy(proxyPtr);
        }
        Ns_CondBroadcast(&poolPtr->cond);
        return TCL_ERROR;
    }


nsproxylib.c, ProxyPush(): Ns_CondBroadcast() called whether proxy is
returned to pool or not.

    Ns_MutexLock(&poolPtr->lock);
    poolPtr->nused--;
    poolPtr->nfree++;
    if ((poolPtr->nused + poolPtr->nfree) <= poolPtr->max) {
        proxyPtr->nextPtr = poolPtr->firstPtr;
        poolPtr->firstPtr = proxyPtr;
        if (proxyPtr->procPtr) {
            SetExpire(proxyPtr->procPtr, proxyPtr->timers.tidle);
        }
        proxyPtr->timers = poolPtr->timers;
        proxyPtr = NULL;
    } else {
        poolPtr->nfree--;
    }
    Ns_CondBroadcast(&poolPtr->cond);
    Ns_MutexUnlock(&poolPtr->lock);


GetPool(): Proxy pool is created if name pool name does not already
exist. What about typos? This is error prone. Alternative: always
create a pool 'default' on start up and make specifying pool names
optional?

The term 'procPtr' is confusing when used to mean pointer to Proxy
Slave. In the rest of the code base, procPtr means pointer to C
function, i.e. a callback.

Re: [naviserver-devel] hosting java-script server-side?

[Stephen D. <sd...@gm...>]

On 9/17/06, Zoran Vasiljevic <zv...@ar...> wrote:
> Hi !
>
> Following a recent discussion on the sister-project, I just wanted
> to check what would you think about integrating javascript as an
> alternate language to Tcl ?
>
> I'm long time user of Tcl (over 12 years) and I still preffer it
> to any other scripting language, buI I see that most of the
> younger people are actually already fluent in javascript
> whereas they are not fluent in (mostly even do not know about) Tcl.
>
> This is at the moment not an issue for us but it may be one in the
> future....
>
> I do not know what would need to be done and to what extent a
> "foreign" language could/should be integrated in the server;
> I just wanted to hear what people would think about this.
>


Are you running a fever?