naviserver-devel — Developer discussion

Re: [naviserver-devel] Cache compiled Tcl page bytecode

[Stephen D. <sd...@gm...>]

On 7/13/06, Andrew Piskorski <at...@pi...> wrote:
> On Thu, Jul 13, 2006 at 08:44:17PM +0100, Stephen Deasey wrote:
>
> > Byte code can only be cached per-interp and our caches are
> > server-wide.  Currently, it's just caching the read of the Tcl source
> > from disk.
>
> I thought the way Rob Mayoff's original feature worked was that it
> "compiled" *.tcl pages to Tcl procs, then cached the compiled bytecode
> of those procs, presumably in per-thread caches.  Does that sound
> right?


No.  Tcl code is compiled by running it.  It's is compiled on demand.

Except... Rob found a neat way to cheat by passing the script as an
arg to a do-nothing 'for' command.  'for' compiles the script for you.
 Very smart and hellish to read...


> What happens for ordinary Tcl procs in ordinary Naviserver Tcl
> libraries?  When do they get compiled to bytecode, and how does
> Naviserver avoid constantly recompiling them?


They're compiled when they're first called.

Remember, the naviserver threading model is to run Tcl interps in
different threads. This is the only way to do it as that's the only
promise of thread safety Tcl provides.

Everything else needs special treatment, hence the nsv_ interface
which communicates between interps.

So, if you have one virtual server defined, and 10 threads, each proc
will be compiled 10 times, one each for each interp run by a conn
thread. (and again for sched threads, jobs threads, etc.).

The byte code is reused after that of course, unless the string
representation of a proc changes, i.e. you redefine it.


> Btw, is the byte code for a given Tcl snippet actually different
> somehow for Tcl interps in different conn threads?


Yes.  The should be equal, but were compiled individually from
separate copies of the source.


> > I think the way to go here is to merge Tcl pages in with the ADP page
> > code.  What's the difference between an ADP page like this:
>
> What would that accomplish?  Is there already cacheing of the
> bytecodes of Tcl source included in ADP pages?


Yes.  And also, the bit below...


> > The ADP stuff in aolserver 4.5 has *output* caching.  Tcl pages would
>
> Huh?  Caching of the output generated by Tcl code is totally different
> than cacheing the compiled Tcl bytecodes.  Both are useful, neither is
> a replacement for the other.


Right, output caching is a different thing.  But wouldn't it be cool
to have it working for ADP pages, Tcl pages, registered procs etc.?

I don't want to implement that 3 separate times.

Re: [naviserver-devel] running callbacks

[Zoran V. <zv...@ar...>]

Am 13.07.2006 um 22:16 schrieb Stephen Deasey:

>  Maybe
> have a look if this particular cases is tested for...

ok. will do.

Re: [naviserver-devel] ns_cache_forever

[Zoran V. <zv...@ar...>]

Am 13.07.2006 um 21:35 schrieb Stephen Deasey:

>
> I'm happy...

I'm delighted, as long as it stays so :-)

Cheers
Zoran

Re: [naviserver-devel] running callbacks

[Stephen D. <sd...@gm...>]

On 7/13/06, Zoran Vasiljevic <zv...@ar...> wrote:
> Hi !
>
> There is a call in tclcallbacks.c:
>
> Ns_TclCallbackArgProc(Tcl_DString *dsPtr, void *arg)
> {
>      int             ii;
>      Ns_TclCallback *cbPtr = arg;
>
>      Tcl_DStringAppendElement(dsPtr, cbPtr->script);
>      for (ii = 0; ii < cbPtr->argc; ii++) {
>          Tcl_DStringAppendElement(dsPtr, cbPtr->argv[ii]);
>      }
> }
>
>
> This way of constructing Tcl script works only if the
> script is a single word, like a procedure name. If the
> script consists of two or more "words" then it won't.
> Common examples are in OO sysytems where you would specify
> the class and the object to be the callback.
> Good example can be made using XOTcl like this:
>
> Class Logger
> Logger log args {
>     puts stderr $args
> }
>
>    ns_logctl register "Logger log" arg1 arg2
>
> This above would actually build the Tcl line as:
>
>    {Logger log} arg1 arg2
>
> and this will of course not work as Tcl will not
> find the {Logger log} command.
>
> It should have rather be:
>
>     Logger log arg1 arg2
>
> So the preferred way would actually look like:
>
>      Tcl_DStringAppend(dsPtr, cbPtr->script);
>      for (ii = 0; ii < cbPtr->argc; ii++) {
>          Tcl_DStringAppendElement(dsPtr, cbPtr->argv[ii]);
>      }
>
> Any objections to chaging this behaviour in the current code?
> I do not know if this will break anything, therefore I ask.
>


Ns_TclCallbackArgProc is only for reporting purposes, so it doesn't
much matter. There should be lots of tests for this in the test suite
for all the existing systems which use Tcl level callbacks.  Maybe
have a look if this particular cases is tested for...



> More...
>
> During the general callback registration and un-registration
> it would be extremely helpful if the callbakck would "know"
> that is being registered or called regularily or un-registered.
> This way it could perform some tasks like internal caching
> of resources (on registration) or destroying the caches (on
> un-registration).


Why can't you initialise resources when you register the callback?  I
don't think I understand the problem here...



> So, I plan to do that with the log callbacks. For that I will
> "invent" the:
>
> typedef enum {
>      unknown,
>      register,
>      execute,
>      unregister
> } Ns_CallbackOperation;
>
> and would put this into the Ns_TclCallback structure as one
> additional element. Also I would add two wrapper functions
>
> Ns_TclCallbackSetRegister
> Ns_TclCallbackSetUnregister
> Ns_TclCallbackSetExecute
>
> which will just flip the bit. This is of course not a change
> we would like to propagate to existing callbacks as this will
> mean breaking the compatiblity with existing scripts which is
> always a pain in the neck. But we can use this in any new
> callbacks that we define the [ns_logctl register] being the
> obvious first candidate.
>
> In this way the Tcl script will always be called with at least
> one argument: callback operation, followed by any number of
> other required arguments.
>
> I hope I managed to bring you the whole picture.
> Are there any objections or better ideas how to do that?
>
> Cheers
> Zoran
>
>
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] Cache compiled Tcl page bytecode

[Andrew P. <at...@pi...>]

On Thu, Jul 13, 2006 at 08:44:17PM +0100, Stephen Deasey wrote:

> Byte code can only be cached per-interp and our caches are
> server-wide.  Currently, it's just caching the read of the Tcl source
> from disk.

I thought the way Rob Mayoff's original feature worked was that it
"compiled" *.tcl pages to Tcl procs, then cached the compiled bytecode
of those procs, presumably in per-thread caches.  Does that sound
right?

What happens for ordinary Tcl procs in ordinary Naviserver Tcl
libraries?  When do they get compiled to bytecode, and how does
Naviserver avoid constantly recompiling them?

Btw, is the byte code for a given Tcl snippet actually different
somehow for Tcl interps in different conn threads?

> I think the way to go here is to merge Tcl pages in with the ADP page
> code.  What's the difference between an ADP page like this:

What would that accomplish?  Is there already cacheing of the
bytecodes of Tcl source included in ADP pages?

> The ADP stuff in aolserver 4.5 has *output* caching.  Tcl pages would

Huh?  Caching of the output generated by Tcl code is totally different
than cacheing the compiled Tcl bytecodes.  Both are useful, neither is
a replacement for the other.

-- 
Andrew Piskorski <at...@pi...>
http://www.piskorski.com/

Re: [naviserver-devel] Cache compiled Tcl page bytecode

[Stephen D. <sd...@gm...>]

On 7/13/06, Vlad Seryakov <vl...@cr...> wrote:
> This is from file.tcl,
>
>      # And here's the magic part. We're using "for" here to translate the
>      # text source file into bytecode, which will be associated with the
>      # Tcl_Obj we just cached (as its internal representation).  "eval"
>      # doesn't do this as the eval provided in Tcl uses the TCL_EVAL_DIRECT
>      # flag, and hence interprets the text directly.
>      #
>
>      uplevel [for [lindex $pair 1] {0} {} {}]
>


It's all lies without per-interp caches...

Re: [naviserver-devel] Cache compiled Tcl page bytecode

[Vlad S. <vl...@cr...>]

This is from file.tcl,

     # And here's the magic part. We're using "for" here to translate the
     # text source file into bytecode, which will be associated with the
     # Tcl_Obj we just cached (as its internal representation).  "eval"
     # doesn't do this as the eval provided in Tcl uses the TCL_EVAL_DIRECT
     # flag, and hence interprets the text directly.
     #

     uplevel [for [lindex $pair 1] {0} {} {}]


Stephen Deasey wrote:
> On 7/13/06, Vlad Seryakov <vl...@cr...> wrote:
>> I meant .tcl file cacheing, which is in tcl/file.tcl when
>> enabletclpages=on then all requests to .tcl are cached in global cache.
> 
> Right, the Tcl is read from disk and cached in memory.  But only the
> Tcl *source*.  It needs to be byte code compiled before use.  This is
> what Andrew is talking about.
> 
> Our cache stringifys everything that goes into it -- you can't share
> Tcl objects between threads.  Therefore, every time you retrieve the
> Tcl page from cache (not disk), you still have to byte code compile
> it.
> 
> aolserver3.x-ad13 cached the byte code per-interp.
> 
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Cache compiled Tcl page bytecode

[Stephen D. <sd...@gm...>]

On 7/13/06, Vlad Seryakov <vl...@cr...> wrote:
> I meant .tcl file cacheing, which is in tcl/file.tcl when
> enabletclpages=on then all requests to .tcl are cached in global cache.

Right, the Tcl is read from disk and cached in memory.  But only the
Tcl *source*.  It needs to be byte code compiled before use.  This is
what Andrew is talking about.

Our cache stringifys everything that goes into it -- you can't share
Tcl objects between threads.  Therefore, every time you retrieve the
Tcl page from cache (not disk), you still have to byte code compile
it.

aolserver3.x-ad13 cached the byte code per-interp.

Re: [naviserver-devel] Cache compiled Tcl page bytecode

[Vlad S. <vl...@cr...>]

I meant .tcl file cacheing, which is in tcl/file.tcl when 
enabletclpages=on then all requests to .tcl are cached in global cache.

Stephen Deasey wrote:
> On 7/12/06, Andrew Piskorski <at...@pi...> wrote:
>> On Wed, Jul 12, 2006 at 04:03:37PM -0400, Vlad Seryakov wrote:
>>> Naviserver does it automatically, it was ported a long time ago
>> Awesome, that's good to know.
>>
> 
> It doesn't actually...
> 
> Byte code can only be cached per-interp and our caches are
> server-wide.  Currently, it's just caching the read of the Tcl source
> from disk.
> 
> I think the way to go here is to merge Tcl pages in with the ADP page
> code.  What's the difference between an ADP page like this:
> 
> <%
>     # Entire page is Tcl...
> %>
> 
> ...and a *.tcl page under the page root?  Nothing, practically.
> They're both 'ADP pages' with one Tcl chunk.
> 
> The ADP stuff in aolserver 4.5 has *output* caching.  Tcl pages would
> be able to take advantage of output caching if they were just
> one-chunk ADP pages.
> 
> This is on my todo list, but not for a while, so this might make a
> nice project for someone. Someone needs to write a few tests for the
> ADP stuff first -- aolserver seems to have broken the ADP stuff during
> development.  A nice small project for someone...
> 
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Cache compiled Tcl page bytecode

[Stephen D. <sd...@gm...>]

On 7/12/06, Andrew Piskorski <at...@pi...> wrote:
> On Wed, Jul 12, 2006 at 04:03:37PM -0400, Vlad Seryakov wrote:
> > Naviserver does it automatically, it was ported a long time ago
>
> Awesome, that's good to know.
>

It doesn't actually...

Byte code can only be cached per-interp and our caches are
server-wide.  Currently, it's just caching the read of the Tcl source
from disk.

I think the way to go here is to merge Tcl pages in with the ADP page
code.  What's the difference between an ADP page like this:

<%
    # Entire page is Tcl...
%>

...and a *.tcl page under the page root?  Nothing, practically.
They're both 'ADP pages' with one Tcl chunk.

The ADP stuff in aolserver 4.5 has *output* caching.  Tcl pages would
be able to take advantage of output caching if they were just
one-chunk ADP pages.

This is on my todo list, but not for a while, so this might make a
nice project for someone. Someone needs to write a few tests for the
ADP stuff first -- aolserver seems to have broken the ADP stuff during
development.  A nice small project for someone...

Re: [naviserver-devel] ns_cache_forever

[Stephen D. <sd...@gm...>]

On 7/11/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> Am 11.07.2006 um 22:13 schrieb Stephen Deasey:
>
> > The API works for me...
> >
>
> OK. Perhaps I was not clear enough:
> We still have "ns_cache_create -ttl" syntax
> The new code does not support this (-ttl) option.
> I will now change our code to adjust to current
> Tcl API and make new release. I will be not very
> happy if after that the option gets renamed, dropped
> or otherwise changed w/o very good (whatever)
> reason.
>
> So, the "API works for me" is:
>
>    ... and I do not plan to change it in near future
>
> or
>
>    ... but I plan to add/drop X because of Y
>
>
> ??
>


No no, I'm happy with it as is.  Unless someone has some other ideas?
I'm happy to look at other ideas, but I guess it has to be quick
because Zoran needs this nailed down.

Is everyone happy with -expires and -timeout taking both an absolute
time and an offset from now, working out which it is by the size of
the value given?

I'm happy...

Re: [naviserver-devel] Why was NaviServer forked?

[Andrew P. <at...@pi...>]

On Wed, Jul 12, 2006 at 04:49:38PM +0200, Zoran Vasiljevic wrote:
> 
> Am 12.07.2006 um 16:22 schrieb Andrew Piskorski:

> > I would have started actively and frequently committing code to the
> > AOLserver CVS, as Dossy explicitly asked/challenged you to do, rather
> > than merely talking about doing so.
> 
> Ah, this goes again... I did that, was publicly exposed as one who
> "broke" some internally used AOL API which (almost) resulted in all

No, the argument you had with Dossy came AFTER (and because of) that
incident, as did Dossy's saying more or less, 'You win, commit more
code.'  Note, I purposely said, "start actively and frequently
committing code", not "commit one change once and then give up".

I know you WANTED to commit code, but (for various reasons, all
social) you didn't.  If you were so convinced Dossy would in the
future gratuitously revert your new contributions, fine, let him do
it.  Then at least you'd have shown that a fork really was required.
My point is that as far as I could tell from the public record, you
never did.

> And not everything was going over the public list.

Since I know nothing about that, I can only comment on what was
public.

> > Look, Zoran, I know that's not your personal style, but as far as I
> > could tell you basically won your last public argument with Dossy, and
> > he must have known it.  He more or less threw up his hands and said,
> > 'Ok, just start committing code, we'll work it out from there.'  And
> > then you walked away!
> 
> Well, this is a very good observation. The reason was simple: I did not
> have the *time* to try again. My company is relying on the code  

Your frustration was quite understandable, however, technically
speaking, if you had the time to fork and then commit your changes to
Naviserver, you also had the time to commit your changes to AOLserver.
You were using CVS in both cases, same task...

Yes, you may have not had that time to then enter into NEW arguments
on the AOLserver list.  But one, you assumed that a new argument would
happen, it might not have.  And two, even if a new argument did
happen, it seems to me that there wouldn't be much forcing you to
spend time on it.  If necessary you could have simply ignored it, and
walked away THEN.

I've been in project lead situations before, and to my surprise, I
noted in my own behavior how easy it was to get all hyper and
overreact to small worries and problems, especially the first time or
three they came up.

And that's all I really saw in Dossy's response to whatever bug
exactly was introduced with your keyed list commit (which you then
fixed really fast).  Yes, he overreacted, and he could and should have
been a hell of a lot clearer and more proactive in admitting that.
But as far as I could tell, that's all that that was really going on.

A silly thing to lose half or so of AOLserver's active core developers
over, and a silly thing to fork over as well...  All IMO, of course.

-- 
Andrew Piskorski <at...@pi...>
http://www.piskorski.com/

Re: [naviserver-devel] running callbacks

[Zoran V. <zv...@ar...>]

Am 13.07.2006 um 16:44 schrieb Vlad Seryakov:

> Is it possible to retrieve callback operation from within callback
> without providing required first argument all the time.
>
> something line ns_callback operation, ns_callback script,  
> ns_callback ....
>
> and in the callback and ns_callback command use special tls which will
> be set with callback struct. this way we keep compatibility and have a
> lot of ways to extend this

hm... good idea... I will try to see how this can be done.
I was also not very pleased with that additional arg but
this is first what came to mind. Lets see how we can cope
with that...

Thanks for the idea!
Zoran

Re: [naviserver-devel] running callbacks

[Vlad S. <vl...@cr...>]

Is it possible to retrieve callback operation from within callback 
without providing required first argument all the time.

something line ns_callback operation, ns_callback script, ns_callback ....

and in the callback and ns_callback command use special tls which will 
be set with callback struct. this way we keep compatibility and have a 
lot of ways to extend this

Zoran Vasiljevic wrote:
> Hi !
> 
> There is a call in tclcallbacks.c:
> 
> Ns_TclCallbackArgProc(Tcl_DString *dsPtr, void *arg)
> {
>      int             ii;
>      Ns_TclCallback *cbPtr = arg;
> 
>      Tcl_DStringAppendElement(dsPtr, cbPtr->script);
>      for (ii = 0; ii < cbPtr->argc; ii++) {
>          Tcl_DStringAppendElement(dsPtr, cbPtr->argv[ii]);
>      }
> }
> 
> 
> This way of constructing Tcl script works only if the
> script is a single word, like a procedure name. If the
> script consists of two or more "words" then it won't.
> Common examples are in OO sysytems where you would specify
> the class and the object to be the callback.
> Good example can be made using XOTcl like this:
> 
> Class Logger
> Logger log args {
>     puts stderr $args
> }
> 
>    ns_logctl register "Logger log" arg1 arg2
> 
> This above would actually build the Tcl line as:
> 
>    {Logger log} arg1 arg2
> 
> and this will of course not work as Tcl will not
> find the {Logger log} command.
> 
> It should have rather be:
> 
>     Logger log arg1 arg2
> 
> So the preferred way would actually look like:
> 
>      Tcl_DStringAppend(dsPtr, cbPtr->script);
>      for (ii = 0; ii < cbPtr->argc; ii++) {
>          Tcl_DStringAppendElement(dsPtr, cbPtr->argv[ii]);
>      }
> 
> Any objections to chaging this behaviour in the current code?
> I do not know if this will break anything, therefore I ask.
> 
> More...
> 
> During the general callback registration and un-registration
> it would be extremely helpful if the callbakck would "know"
> that is being registered or called regularily or un-registered.
> This way it could perform some tasks like internal caching
> of resources (on registration) or destroying the caches (on
> un-registration).
> So, I plan to do that with the log callbacks. For that I will
> "invent" the:
> 
> typedef enum {
>      unknown,
>      register,
>      execute,
>      unregister
> } Ns_CallbackOperation;
> 
> and would put this into the Ns_TclCallback structure as one
> additional element. Also I would add two wrapper functions
> 
> Ns_TclCallbackSetRegister
> Ns_TclCallbackSetUnregister
> Ns_TclCallbackSetExecute
> 
> which will just flip the bit. This is of course not a change
> we would like to propagate to existing callbacks as this will
> mean breaking the compatiblity with existing scripts which is
> always a pain in the neck. But we can use this in any new
> callbacks that we define the [ns_logctl register] being the
> obvious first candidate.
> 
> In this way the Tcl script will always be called with at least
> one argument: callback operation, followed by any number of
> other required arguments.
> 
> I hope I managed to bring you the whole picture.
> Are there any objections or better ideas how to do that?
> 
> Cheers
> Zoran
> 
> 
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] running callbacks

[Zoran V. <zv...@ar...>]

Am 13.07.2006 um 10:17 schrieb Zoran Vasiljevic:

> Class Logger
> Logger log args {
>     puts stderr $args
> }

Logger proc log args {
     puts stderr $args
}

that is...

Zoran

[naviserver-devel] running callbacks

[Zoran V. <zv...@ar...>]

Hi !

There is a call in tclcallbacks.c:

Ns_TclCallbackArgProc(Tcl_DString *dsPtr, void *arg)
{
     int             ii;
     Ns_TclCallback *cbPtr = arg;

     Tcl_DStringAppendElement(dsPtr, cbPtr->script);
     for (ii = 0; ii < cbPtr->argc; ii++) {
         Tcl_DStringAppendElement(dsPtr, cbPtr->argv[ii]);
     }
}


This way of constructing Tcl script works only if the
script is a single word, like a procedure name. If the
script consists of two or more "words" then it won't.
Common examples are in OO sysytems where you would specify
the class and the object to be the callback.
Good example can be made using XOTcl like this:

Class Logger
Logger log args {
    puts stderr $args
}

   ns_logctl register "Logger log" arg1 arg2

This above would actually build the Tcl line as:

   {Logger log} arg1 arg2

and this will of course not work as Tcl will not
find the {Logger log} command.

It should have rather be:

    Logger log arg1 arg2

So the preferred way would actually look like:

     Tcl_DStringAppend(dsPtr, cbPtr->script);
     for (ii = 0; ii < cbPtr->argc; ii++) {
         Tcl_DStringAppendElement(dsPtr, cbPtr->argv[ii]);
     }

Any objections to chaging this behaviour in the current code?
I do not know if this will break anything, therefore I ask.

More...

During the general callback registration and un-registration
it would be extremely helpful if the callbakck would "know"
that is being registered or called regularily or un-registered.
This way it could perform some tasks like internal caching
of resources (on registration) or destroying the caches (on
un-registration).
So, I plan to do that with the log callbacks. For that I will
"invent" the:

typedef enum {
     unknown,
     register,
     execute,
     unregister
} Ns_CallbackOperation;

and would put this into the Ns_TclCallback structure as one
additional element. Also I would add two wrapper functions

Ns_TclCallbackSetRegister
Ns_TclCallbackSetUnregister
Ns_TclCallbackSetExecute

which will just flip the bit. This is of course not a change
we would like to propagate to existing callbacks as this will
mean breaking the compatiblity with existing scripts which is
always a pain in the neck. But we can use this in any new
callbacks that we define the [ns_logctl register] being the
obvious first candidate.

In this way the Tcl script will always be called with at least
one argument: callback operation, followed by any number of
other required arguments.

I hope I managed to bring you the whole picture.
Are there any objections or better ideas how to do that?

Cheers
Zoran

Re: [naviserver-devel] Encoding

[Bernd E. <eid...@we...>]

Hi Vlad,

> Or you can try to replace ReturnCharData in return.c and see if this
> works (i did not test it):

yes, it seems to work:

  HTTP/1.1 200 OK
  MIME-Version: 1.0
  Accept-Ranges: bytes
  Date: Thu, 13 Jul 2006 06:45:55 GMT
  Server: NaviServer/4.99.2
  Content-Type: text/html; charset=iso8859-15
  Transfer-encoding: chunked
  Connection: close
(Length: not specified [text/html])

I think we should somehow solve this and compute a correct Content-Length, 
reconsidering also the note from Stephen concerning open bug nr. 1377059.

Bernd.

Re: [naviserver-devel] Encoding

[Vlad S. <vl...@cr...>]

Or you can try to replace ReturnCharData in return.c and see if this 
works (i did not test it):

static int
ReturnCharData(Ns_Conn *conn, int status, CONST char *data, int len,
                CONST char *type, int sendRaw)
{
     Conn        *connPtr = (Conn *) conn;
     int          result;
     Tcl_Encoding enc;
     Tcl_DString  type_ds;
     int          new_type = NS_FALSE;

     if (conn->flags & NS_CONN_SKIPBODY) {
         data = NULL;
         len = 0;
     }
     if (len < 0) {
         len = data ? strlen(data) : 0;
     }
     if (len > 0 && !sendRaw) {
         /*
          * Make sure we know what output encoding (if any) to use.
          */
         NsComputeEncodingFromType(type, &enc, &new_type, &type_ds);
         if (new_type) {
             type = Tcl_DStringValue(&type_ds);
         }
         if (enc != NULL) {
             connPtr->encoding = enc;
             conPtr->flags |= NS_CONN_WRITE_CHUNKED;
         } else if (connPtr->encoding == NULL) {
             sendRaw = NS_TRUE;
         }
     }
     Ns_ConnSetRequiredHeaders(conn, type, len);
     Ns_ConnQueueHeaders(conn, status);
     if (sendRaw) {
         result = Ns_WriteConn(conn, data, len);
     } else {
         result = Ns_WriteCharConn(conn, data, len);
     }
     if (result == NS_OK) {
         result = Ns_ConnClose(conn);
     }
     if (new_type) {
         Tcl_DStringFree(&type_ds);
     }

     return result;
}

Vlad Seryakov wrote:
> I think the only solution here will be to use Chunked-Encoding to output 
> encoded content, in this case Content-Length should be removed fromthe 
> headers.
> 
> Ns_ConnWriteVChars handles that, so it shoudl be chnaged i guess.
> 
> Bernd Eidenschink wrote:
>> Am Mittwoch, 12. Juli 2006 15:23 schrieb Bernd Eidenschink:
>>> it looks like the mechanism of changing the encoding works as expected, but
>>> the computation of the correct contentlength not.
>> ok. I think this is going on:
>>
>> 1. NsTclReturnObjCmd
>> If ns_return is called _without_ -binary switch, Ns_ConnReturnCharData is 
>> called. The length is computed from the UTF-8 representation.
>>
>> 2. Ns_ConnReturnCharData
>> calls
>>
>> 3. ReturnCharData
>> Knows what output encoding to use. If not UTF-8, it will call
>> Ns_WriteCharConn
>>
>> ...BUT BEFORE...
>>
>> 4.
>> Ns_ConnSetRequiredHeaders
>> calls Ns_ConnSetLengthHeader
>> (Set the Content-Length output header)
>>
>> for
>>
>> 5.
>> Ns_ConnQueueHeaders
>> calls NS_ConnConstructHeaders
>> where
>> "Update the response length value directly from the header to be sent, i.e., 
>> don't trust programmers"
>>
>> So the content-length is set.
>>
>> NOW:
>>
>> 7. Ns_WriteCharConn is called, calls
>> Ns_ConnWriteChars
>> that with the help of Ns_ConnWriteVChars encodes
>> with the help of "Tcl_UtfToExternal" to the final encoding.
>>
>> I can be fundamentally wrong, but it seems to me like the 
>> whatever-it-will-become final encoding length should be computed earlier...
>>
>> What do you think?
>>
>> Bernd.
>>
>>
>> -------------------------------------------------------------------------
>> Using Tomcat but need to do more? Need to support web services, security?
>> Get stuff done quickly with pre-integrated technology to make your job easier
>> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>>
> 
> 


-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Encoding

[Vlad S. <vl...@cr...>]

I think the only solution here will be to use Chunked-Encoding to output 
encoded content, in this case Content-Length should be removed fromthe 
headers.

Ns_ConnWriteVChars handles that, so it shoudl be chnaged i guess.

Bernd Eidenschink wrote:
> Am Mittwoch, 12. Juli 2006 15:23 schrieb Bernd Eidenschink:
>> it looks like the mechanism of changing the encoding works as expected, but
>> the computation of the correct contentlength not.
> 
> ok. I think this is going on:
> 
> 1. NsTclReturnObjCmd
> If ns_return is called _without_ -binary switch, Ns_ConnReturnCharData is 
> called. The length is computed from the UTF-8 representation.
> 
> 2. Ns_ConnReturnCharData
> calls
> 
> 3. ReturnCharData
> Knows what output encoding to use. If not UTF-8, it will call
> Ns_WriteCharConn
> 
> ...BUT BEFORE...
> 
> 4.
> Ns_ConnSetRequiredHeaders
> calls Ns_ConnSetLengthHeader
> (Set the Content-Length output header)
> 
> for
> 
> 5.
> Ns_ConnQueueHeaders
> calls NS_ConnConstructHeaders
> where
> "Update the response length value directly from the header to be sent, i.e., 
> don't trust programmers"
> 
> So the content-length is set.
> 
> NOW:
> 
> 7. Ns_WriteCharConn is called, calls
> Ns_ConnWriteChars
> that with the help of Ns_ConnWriteVChars encodes
> with the help of "Tcl_UtfToExternal" to the final encoding.
> 
> I can be fundamentally wrong, but it seems to me like the 
> whatever-it-will-become final encoding length should be computed earlier...
> 
> What do you think?
> 
> Bernd.
> 
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 


-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Questions about NS details... (scalability, lib changes)

[Zoran V. <zv...@ar...>]

Am 12.07.2006 um 22:04 schrieb Mike:

>
> mmap is good because it kind of allows zero-copy from usercode..
> however it's not useable in my situation directly because of the
> 32-bit address space restriction - can't mmap 65GB (650MB*100) of
> address space...  The solution I've seen others use is to mmap in 16MB
> chunks and move the mmaped chunk as the transfer progresses along the
> file.

I wouldn't be surprised to see some clever sendfile implementations
mmap'ing the file and there you go with the 32-bit problem!
I do not know... I believe you may be good advised to look into
the actual platform implementation of the sendfile.

Cheers
Zoran

Re: [naviserver-devel] Cache compiled Tcl page bytecode

[Andrew P. <at...@pi...>]

On Wed, Jul 12, 2006 at 04:03:37PM -0400, Vlad Seryakov wrote:
> Naviserver does it automatically, it was ported a long time ago

Awesome, that's good to know.

> >   Cache compiled Tcl page bytecode
> >   https://sourceforge.net/tracker/?func=detail&aid=689515&group_id=3152&atid=353152

-- 
Andrew Piskorski <at...@pi...>
http://www.piskorski.com/

Re: [naviserver-devel] Questions about NS details... (scalability, lib changes)

[Mike <nee...@gm...>]

On 7/12/06, Vlad Seryakov <vl...@cr...> wrote:
> I could be wrong, never used sendfile before.
>
> The code you can check is driver.c, WriterThread, and bufsize parameter
> can be used to define buffer.

Excerpt from sendfile(2) on FreeBSD:

     When using a socket marked for non-blocking I/O, sendfile() may send
     fewer bytes than requested.  In this case, the number of bytes success-
     fully written is returned in *sbytes (if specified), and the error EAGAIN
     is returned.

Overall it looks like it would be very easy to change WriterThread()
to use sendfile() instead of read+memmove+(something).  However the
(something) appears to be NsSockRead() which I am guessing is a bit
more complicated than just an indirect call to write()...  since
probably that code-path supports sending through zlib or over an SSL
socket...

> I was considering using mmap, but somewhere i heard that too many
> mmapped files could be a problem for the server.

mmap is good because it kind of allows zero-copy from usercode..
however it's not useable in my situation directly because of the
32-bit address space restriction - can't mmap 65GB (650MB*100) of
address space...  The solution I've seen others use is to mmap in 16MB
chunks and move the mmaped chunk as the transfer progresses along the
file.

Re: [naviserver-devel] Cache compiled Tcl page bytecode

[Vlad S. <vl...@cr...>]

Naviserver does it automatically, it was ported a long time ago

Andrew Piskorski wrote:
> On Wed, Jul 12, 2006 at 03:00:17PM -0400, Mike wrote:
>> Subject: [naviserver-devel] Questions about NS details... (scalability, lib changes)
> 
>> Does there exist a caching mechanism for dynamically generated tcl/adp
>> pages?  If it does, where can I find docs for how it works/when the
>> cache is invalidated?
> 
> Yes, but last I heard no one got around to actually forward porting it
> from AOLserver 3.3+ad13 to 4.x:
> 
>   Cache compiled Tcl page bytecode
>   https://sourceforge.net/tracker/?func=detail&aid=689515&group_id=3152&atid=353152
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Questions about NS details... (scalability, lib changes)

[Bernd E. <eid...@we...>]

Hi Mike,

> The project for which I am considering NS will have a dynamic and very
> large static part.  There will be many very large (650MB) static files
> that people will download.  It is hard for me to judge what the
> performance of serving these files with NS will be.

It depends a little bit on what is "a dynamic" and a "very large
static part" and about what dimensions you talk in general. Usually it
could make sense to delegate uncached, personalized, database
intensive scripting parts to NS and the huge downloads to one or more
(balanced) servers with very optimized capabilities in that context.
"Pound" is very good for that (and brings SSL!, if you don't want to
rely on the availability of nsopenssl).
(And so the connections to the database on another server do not
suffer from the downloads).
Personally, if only authorized people should be allowed to download,
it could be very easy (in terms of development) to do everything with
n-Naviservers and one database simply because you limit the number of
technologies you have to deal with.

How many users will download? What minimum download rate do you want
to offer them? You could limit the parallel number of downloads per
box so you are able to come very close to the hardware dimensions
needed.

>  Of course
> benchmarking would be the ideal route, however I have the pleasure of
> writing the code now, and the hardware for this project won't be
> purchased until a while from now.  Should I use NS for the dynamic
> part and redirect the static requests to something like lighttpd, or
> can I rely on NS to handle such load without problems?

NS can handle it and offers specialized "writerthreads", but for the mere "dumb"
transfer lighttpd might be a very good decision. Use NS for the
scripting.

> Does a mechanism exist to "reload" the tcl libraries on a live server?

Yes, the "evil ns_eval".

And you can activate the "lazyloader" mechanism that speeds up thread
creation as only needed code parts are loaded via the unknown
mechanism during the thread livetime.
http://naviserver.sourceforge.net/wiki/index.php/Nstrace.tcl

Bernd.

[naviserver-devel] Cache compiled Tcl page bytecode

[Andrew P. <at...@pi...>]

On Wed, Jul 12, 2006 at 03:00:17PM -0400, Mike wrote:
> Subject: [naviserver-devel] Questions about NS details... (scalability, lib changes)

> Does there exist a caching mechanism for dynamically generated tcl/adp
> pages?  If it does, where can I find docs for how it works/when the
> cache is invalidated?

Yes, but last I heard no one got around to actually forward porting it
from AOLserver 3.3+ad13 to 4.x:

  Cache compiled Tcl page bytecode
  https://sourceforge.net/tracker/?func=detail&aid=689515&group_id=3152&atid=353152

-- 
Andrew Piskorski <at...@pi...>
http://www.piskorski.com/