naviserver-devel — Developer discussion

Re: [naviserver-devel] Upload Progress

[Gustaf N. <ne...@wu...>]

Zoran Vasiljevic wrote:
>
> But that would occupy the conn thread for ages, right?
> I can imagine several slow-line large-file uploads could
> consume many of those. Wasn't that the reason everything
> was moved in the driver thread for the 4.0 ?
>
> What I have/had in mind is aync writes from the driver thread.
> Most of the OS'es have this feature (kaio) so we can employ it.
> The question of locking, however still remains in such case.
> So the decision has to be made on: what is cheaper? Locking
> or spooling to disk out of the conn thread? I have no real-life
> experience but I'm inclined to believe that spooling out of the
> conn thread would be more costly.
>
> What do you think?
i have not much looked into the c-code, so please forgive, if my
mail here is obviously wrong or naive.

One thing, which i dislike about the aolserver for file uploads is that
there is no hook for early permission or quota checking; one has to
upload the whole file before aolserver is able to give some feedback.
i see this even more important than the progress bar.

What about the following model:

A) HTTP header (up to empty line) -> driver thread
 
B) Then passing control+socket to a separate receiving thread that
  1) issues a callback at the start (e.g. for permission checking) 
  2) does async i/o to receive the message body (e.g. PUT, POST request)
    here, one can issue as well callbacks per async block, and decide
    fom tcl, whether one would like to spool to memory or disc.

C) When end of message received, pass control to connection thread as usual.

The receiving thread is like a connection thread with tcl stuff; when the
receiving thread is implemented via libthtread, there won't be probably
much locking. Only one receiving thread will be necessary.
The overall model could be quite compatible with the aolserver, since the
connection tread is invoked when all data is received. It would be necessary
to pass context information from the receiving thread to the connection 
thread
to avoid double permission checking.

It could as well make sense, to pass control to the receiving thread only
for chunked content. Maybe, the receiving thread is not needed at all,
since the driver thread could handle everything as well.

-gustaf

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 31.12.2005 um 11:58 schrieb Stephen Deasey:

>
> I think we talked about this before, but I can't find it in the
> mailing list archive.  Anyway, the problem with recording the upload
> process is all the locking that's required.  You could minimize this,
> e.g. by only recording uploads above a certain size, or to a certain
> URL.

Hm... not very useful.

>
> It reminds me of a similar problem we had.  Spooling large uploads  
> to disk:
>
> https://sourceforge.net/mailarchive/forum.php? 
> thread_id=7524448&forum_id=43966
>
> Vlad implemented the actual spooling, but moving that work into the
> conn threads, reading lazily, is still to be done.

Yes. This is what I wanted to do but never got to it because
of the internal work... huh.

>
> Lazy uploading is exactly the hook you need to track upload progress.
> The client starts to upload a file.  Read-ahead occurs in the driver
> thread, say 8k.  Control is passed to a conn thread, which then calls
> Ns_ConnContent().  The remaining content is read from the client, in
> the context of the conn thread and so not blocking the driver thread,
> and perhaps the content is spooled to disk.

But that would occupy the conn thread for ages, right?
I can imagine several slow-line large-file uploads could
consume many of those. Wasn't that the reason everything
was moved in the driver thread for the 4.0 ?

What I have/had in mind is aync writes from the driver thread.
Most of the OS'es have this feature (kaio) so we can employ it.
The question of locking, however still remains in such case.
So the decision has to be made on: what is cheaper? Locking
or spooling to disk out of the conn thread? I have no real-life
experience but I'm inclined to believe that spooling out of the
conn thread would be more costly.

What do you think?

Zoran

Re: [naviserver-devel] Naviserver and OACS (compatibility)?

[Zoran V. <zv...@ar...>]

Am 31.12.2005 um 12:54 schrieb Stephen Deasey:

>
> The order:
>
>     TheFilter preauth arg1 arg2
>
> makes the most sense to me. The <when> is not an optional arg, it's
> always appended.  Required args always come before optional args in
> proc specs...

Allright.

>
> If we changed this, we should also change Ns_TclEvalCallback() in the
> same way.  A proc registered with ns_serverrootproc may have the host
> appended as the last arg. It uses Ns_TclEvalCallback() directly.

Yes.

>
> ns_register_proc doesn't append information of it's own.  But it does
> insist on appending a blank arg if none were supplied during
> registration.  I'd like to remove that as it's easy to write code
> which works with either case.

Yes.

Zoran

Re: [naviserver-devel] Naviserver and OACS (compatibility)?

[Stephen D. <sd...@gm...>]

On 12/31/05, Zoran Vasiljevic <zv...@ar...> wrote:
>
> Am 31.12.2005 um 01:15 schrieb Gustaf Neumann:
>
> >
> >  Class Filter
> >  Filter instproc preauth args { .... }
> >  Filter instproc postauth args { .... }
> >
>
> Damn, you are right. In which case we'd do:
>
>      ns_register_filter <reason> GET /junk myfilter myarg1 myarg2 ...
>
> and it would call the filter as:
>
>      myfilter <reason> myarg2 myarg3
>
> Understand. This would allow you to:
>
> Class TheFilter
> TheFilter proc preauth   {arg1 arg2} {...}
> TheFilter proc postauth  {arg1 arg2 arg3} {...}
>
> ns_register_filter preauth  GET /junk TheFilter arg1 arg2
> ns_register_filter postauth GET /junk TheFilter arg1 arg2 arg3
>
> and we'd call
>
> TheFilter preauth arg1 arg2
> TheFilter postauth arg1 arg2 arg3
>
> The proc way would be:
>
> proc TheFilter {reason args} {...}
>
> and it would naturally fit in, as with OO syntax.
>
> Well, this certainly makes sense to me. This is clean and elegant.
> Hm... this will however be an incompatible change to the way AOL
> server is handling the task, but will be way more sexy.
>
> This is what I mean (others can comment): I'd go after this change.
> For the people not using any optional arguments, there would be no
> compatibility issues. For those using the arguments, they'd have to
> rewrite the code. But it is for the better, not worse.
>
> Stephen, Vlad, what do you think? I have no problem with that
> as we are not using any of the optional args in our code.
> We might examine to modify ns_register_proc the same way.


The order:

    TheFilter preauth arg1 arg2

makes the most sense to me. The <when> is not an optional arg, it's
always appended.  Required args always come before optional args in
proc specs...

If we changed this, we should also change Ns_TclEvalCallback() in the
same way.  A proc registered with ns_serverrootproc may have the host
appended as the last arg. It uses Ns_TclEvalCallback() directly.

ns_register_proc doesn't append information of it's own.  But it does
insist on appending a blank arg if none were supplied during
registration.  I'd like to remove that as it's easy to write code
which works with either case.

[naviserver-devel] Upload Progress

[Stephen D. <sd...@gm...>]

On 12/30/05, Vlad Seryakov <vl...@cr...> wrote:
> On that note i have another idea i'd like to discuss before i even start
> coding a prototype. There is thread on aolserver mailing list about
> upload prgoress, so i thought would it be a good idea to have global
> url-specific cache of all uploads, let's say, all requestes with
> content-length > 0. It will have only 2 alues, current size and total
> length and will last only for the time of upload. It will belong to Sock
> structure of the Request, so on close it will be freed as well.
>
> Making it url-specific will give Web developer ability to generate
> uniaue urls for upload and then request statistics, requests with the
> same url will not override each other, server will update statistics for
>   the first created cache only, subsequent uploads with the same url
> will not show anything or show old values which is fine for security
> reasons.
>
> Overhead is minimal and it will add one new commmand like
> ns_upload_stats url. SockRead will handle it all, so no other places are
> affected.
>
> Is it worth trying?


I think we talked about this before, but I can't find it in the
mailing list archive.  Anyway, the problem with recording the upload
process is all the locking that's required.  You could minimize this,
e.g. by only recording uploads above a certain size, or to a certain
URL.

It reminds me of a similar problem we had.  Spooling large uploads to disk:

https://sourceforge.net/mailarchive/forum.php?thread_id=3D7524448&forum_id=
=3D43966

Vlad implemented the actual spooling, but moving that work into the
conn threads, reading lazily, is still to be done.

Lazy uploading is exactly the hook you need to track upload progress.=20
The client starts to upload a file.  Read-ahead occurs in the driver
thread, say 8k.  Control is passed to a conn thread, which then calls
Ns_ConnContent().  The remaining content is read from the client, in
the context of the conn thread and so not blocking the driver thread,
and perhaps the content is spooled to disk.

To implement upload tracking you would register a proc for /upload
which instead of calling Ns_ConnContent(), calls Ns_ConnRead()
multiple times, recording the number of bytes read in the upload
tracking cache, and saving the data to disk or wherever.

A lot more control of the upload process is needed, whether it be to
control size, access, to record stats, or something we haven't thought
of yet.  If we complete the work to get lazy reading from the client
working, an upload tracker will be an easy module to write.

Does this make sense?

Re: [naviserver-devel] Naviserver and OACS (compatibility)?

[Zoran V. <zv...@ar...>]

Am 31.12.2005 um 01:15 schrieb Gustaf Neumann:

>
>  Class Filter
>  Filter instproc preauth args { .... }
>  Filter instproc postauth args { .... }
>

Damn, you are right. In which case we'd do:

     ns_register_filter <reason> GET /junk myfilter myarg1 myarg2 ...

and it would call the filter as:

     myfilter <reason> myarg2 myarg3

Understand. This would allow you to:

Class TheFilter
TheFilter proc preauth   {arg1 arg2} {...}
TheFilter proc postauth  {arg1 arg2 arg3} {...}

ns_register_filter preauth  GET /junk TheFilter arg1 arg2
ns_register_filter postauth GET /junk TheFilter arg1 arg2 arg3

and we'd call

TheFilter preauth arg1 arg2
TheFilter postauth arg1 arg2 arg3

The proc way would be:

proc TheFilter {reason args} {...}

and it would naturally fit in, as with OO syntax.

Well, this certainly makes sense to me. This is clean and elegant.
Hm... this will however be an incompatible change to the way AOL
server is handling the task, but will be way more sexy.

This is what I mean (others can comment): I'd go after this change.
For the people not using any optional arguments, there would be no
compatibility issues. For those using the arguments, they'd have to
rewrite the code. But it is for the better, not worse.

Stephen, Vlad, what do you think? I have no problem with that
as we are not using any of the optional args in our code.
We might examine to modify ns_register_proc the same way.

Cheers,
Zoran

Re: [naviserver-devel] Naviserver and OACS (compatibility)?

[Gustaf N. <ne...@wu...>]

Zoran Vasiljevic wrote:
>
> Am 30.12.2005 um 19:19 schrieb Gustaf Neumann:
>
>>
>> how will be the call, when
>>
>>    ns_register_filter preauth GET /junk myfilter
>>
>> is registered? when
>>
>>   myfilter preauth
>
> Like this.
fine.
>
>>
>> is called, this would be pretty much comaptible with  the aolserver 
>> for the simple cases.
>> this would  as well work with xotcl methods in this case. however, 
>> this would as well be
>> the case, when WHY is placed before the passed arguments.
>>
>> the hyperflexible approach would be to use
>>
>>   ns_register_filter preauth GET /junk [list myfilter arg1] arg2  arg3
>
> I do not understan why bother with all this arguments (arg2 arg3, etc)
> when you can easily construct the script using the [list] command? 
...
> and if you need to pass arguments to "myfilter" just use 
>    ns_register_filter preauth GET /junk [list myfilter $arg1 $arg2 $arg3] 
> Would this be less flexible? I think not.
Flexible, well... Here is a reason: look at the example from my first 
mail. with xotcl
(or other oo approaches) it is the most straightforward to implement the 
filters as methods.

  Class Filter
  Filter instproc preauth args { .... }
  Filter instproc postauth args { .... }

  Filter myfilter

In such cases it is most conveniant, when the WHY argument is first, without
loosing the ability to pass to the handlers. When appending always the WHY
at the end, the arguments become the "methods", unless one has a 
zero-argument filter proc.
When one needs an additional argument, one has to change the way to map he
methods.

i would be as happy with

    ns_register_filter preauth GET /junk [list myfilter $arg1 $arg2 $arg3]

calling

   myfilter WHY arg1 arg2 arg3

Certainly, in pure tcl cases, it does not matter much, where the WHY is 
placed.
When the WHY is always appended to the end - as you suggests - i envision
myself defining the filter like

    ns_register_filter preauth GET /junk [list myfilter preauth $arg1 
$arg2 $arg3]
    ns_register_filter postauth GET /junk [list myfilter postauth $arg1 
$arg2 $arg3]

and not much caring about the WHY argument. certainly doable, but 
slightly strange.

as i said before, the most common case it that the filter cmd is used 
without
arguments. for the other cases, there is always a way to hack around.

decide, whatever you prefer.

all the best

-gustaf

Re: [naviserver-devel] Naviserver and OACS (compatibility)?

[Zoran V. <zv...@ar...>]

Am 30.12.2005 um 19:19 schrieb Gustaf Neumann:

>
> how will be the call, when
>
>    ns_register_filter preauth GET /junk myfilter
>
> is registered? when
>
>   myfilter preauth

Like this.

>
> is called, this would be pretty much comaptible with  the aolserver  
> for the simple cases.
> this would  as well work with xotcl methods in this case. however,  
> this would as well be
> the case, when WHY is placed before the passed arguments.
>
> the hyperflexible approach would be to use
>
>   ns_register_filter preauth GET /junk [list myfilter arg1] arg2  arg3

I do not understan why bother with all this arguments (arg2 arg3, etc)
when you can easily construct the script using the [list] command?

In most of the callbacks I use (or have made in our C-code) I revert
to a single callback script line which must be prepared by the caller,
then registered. The C-implementation would then ADD arguments to it.

>
> which means, cmd as you suggested, appending optional arguments.  
> this means
>
>   ns_register_filter preauth GET /junk myfilter  arg2  arg3

Again, I'd revert to:

    ns_register_filter preauth GET /junk myfilter

and if you need to pass arguments to "myfilter" just use

    ns_register_filter preauth GET /junk [list myfilter $arg1 $arg2  
$arg3]

Would this be less flexible? I think not.

Cheers
Zoran

Re: Fwd: [naviserver-devel] Naviserver and OACS (compatibility)?

[Gustaf N. <ne...@wu...>]

Zoran Vasiljevic wrote:
>
>>
>> Why would we need to process that optional argument in the first place?
>>
>> Lets make the ns_register_filter as:
>>     ns_register_filter when method urlPattern script
>>
>> The "script" should be completely specified, for example
>>
>>    set arg "whatever thing"
>>    ns_register_filter preauth GET /junk [list myfilter $arg]
>>
>> During runtime we'd append the WHY so it will be called as
>>
>>    myfilter "whatever thing" preauth
>>
>> Wouldn't that be better/cleaner altogether? This will not break
>> any code using no optional args (we never used this arg thing)
>> and would be much more logical to understand.
>>
>> Hm?

 how will be the call, when

    ns_register_filter preauth GET /junk myfilter

is registered? when

   myfilter preauth

is called, this would be pretty much comaptible with  the aolserver for 
the simple cases.
this would  as well work with xotcl methods in this case. however, this 
would as well be
the case, when WHY is placed before the passed arguments.

the hyperflexible approach would be to use

   ns_register_filter preauth GET /junk [list myfilter arg1] arg2  arg3

which calls
 
  myfilter arg1 preauth arg1 arg2

which means, cmd as you suggested, appending optional arguments. this means

   ns_register_filter preauth GET /junk myfilter  arg2  arg3

will make perfectly nice xotcl argumnt lists. In the simple case, this 
ist still
comaptible with aolserver, in the complex cases, one has to do rewrite 
the code anyhow.

-gustaf
 


>>
>> Zoran
>>
>>
>>
>> -------------------------------------------------------
>> This SF.net email is sponsored by: Splunk Inc. Do you grep through 
>> log files
>> for problems?  Stop!  Download the new AJAX search engine that makes
>> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
>> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] nacache

[Zoran V. <zv...@ar...>]

Am 30.12.2005 um 18:42 schrieb Vlad Seryakov:

> Unique url or unique parameters, you can upload using /some/url/for/ 
> post?arg=1&arg2=2 urls as well, parameters will be ignored but can  
> be used in upload stats using request->line instead of request- 
> >url. This way it is much easier to develop and support.

yes. this is true. the request line seems more appropriate to me.
Zoran

Re: [naviserver-devel] nacache

[Vlad S. <vl...@cr...>]

Unique url or unique parameters, you can upload using 
/some/url/for/post?arg=1&arg2=2 urls as well, parameters will be ignored 
but can be used in upload stats using request->line instead of 
request->url. This way it is much easier to develop and support.

Zoran Vasiljevic wrote:
> 
> Am 30.12.2005 um 18:33 schrieb Vlad Seryakov:
> 
>> That is the key, if developer is not prepared unique urls, that  
>> statistics is useless because every upload overwrites the same stats.
>>
> 
> OK. I will talk to one of our Web-GUI guys. He's been bugging me about
> that upload progress since we switched to 4.0 server. I will check with
> him about the unique url approach and see if this is feasible. I do not
> see any problem with that but I'm not that much in that corner of the
> world...
> 
> Zoran
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log 
> files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] nacache

[Zoran V. <zv...@ar...>]

Am 30.12.2005 um 18:33 schrieb Vlad Seryakov:

> That is the key, if developer is not prepared unique urls, that  
> statistics is useless because every upload overwrites the same stats.
>

OK. I will talk to one of our Web-GUI guys. He's been bugging me about
that upload progress since we switched to 4.0 server. I will check with
him about the unique url approach and see if this is feasible. I do not
see any problem with that but I'm not that much in that corner of the
world...

Zoran

Re: [naviserver-devel] nacache

[Vlad S. <vl...@cr...>]

That is the key, if developer is not prepared unique urls, that 
statistics is useless because every upload overwrites the same stats.


Zoran Vasiljevic wrote:
> 
> Am 30.12.2005 um 18:19 schrieb Vlad Seryakov:
> 
>> Patch is attached, works great
>>
> 
> Dummy question: what happens I have several uploads at a time
> all handled by the same url?
> 
> Zoran
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log 
> files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] nacache

[Zoran V. <zv...@ar...>]

Am 30.12.2005 um 18:19 schrieb Vlad Seryakov:

> Patch is attached, works great
>

Dummy question: what happens I have several uploads at a time
all handled by the same url?

Zoran

Re: [naviserver-devel] nacache

[Zoran V. <zv...@ar...>]

Am 30.12.2005 um 18:19 schrieb Vlad Seryakov:

> Patch is attached, works great

Why don't you put that in RFE section in SourceForge?
This way get get a nice record for future refs.

Cheers
Zoran

Re: [naviserver-devel] nacache

[Vlad S. <vl...@cr...>]

Patch is attached, works great

Zoran Vasiljevic wrote:
> 
> Am 30.12.2005 um 16:02 schrieb Vlad Seryakov:
> 
>> Is it worth trying?
> 
> 
> Yesterday!
> Zoran
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log 
> files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] nacache

[Zoran V. <zv...@ar...>]

Am 30.12.2005 um 16:02 schrieb Vlad Seryakov:

> Is it worth trying?

Yesterday!
Zoran

Re: [naviserver-devel] nacache

[Vlad S. <vl...@cr...>]

On that note i have another idea i'd like to discuss before i even start 
coding a prototype. There is thread on aolserver mailing list about 
upload prgoress, so i thought would it be a good idea to have global 
url-specific cache of all uploads, let's say, all requestes with 
content-length > 0. It will have only 2 alues, current size and total 
length and will last only for the time of upload. It will belong to Sock 
structure of the Request, so on close it will be freed as well.

Making it url-specific will give Web developer ability to generate 
uniaue urls for upload and then request statistics, requests with the 
same url will not override each other, server will update statistics for 
  the first created cache only, subsequent uploads with the same url 
will not show anything or show old values which is fine for security 
reasons.

Overhead is minimal and it will add one new commmand like 
ns_upload_stats url. SockRead will handle it all, so no other places are 
affected.

Is it worth trying?


Zoran Vasiljevic wrote:
> 
> Am 30.12.2005 um 15:46 schrieb Vlad Seryakov:
> 
>> I have web applications and server applications running on our  server 
>> and in my experience so far i always prefer global caches  over 
>> per-thread cache. If you need something local in long running  thread, 
>> you can use Tcl global variables to keep state during the  processing, 
>> even namespace variables. It is just one Tcl  interpreter anyway.
> 
> 
> True. But the state can be accumulating over time, though,
> leading to extensive memory usage. Hence the cache, as it
> can selecively expire things.
> 
> But...
> 
> I haven't any code needing per-thread caches, yet.
> As a tool in the toolbox, OK, but I will not press
> on that. I'm perfectly happy with global caches.
> 
> Zoran
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log 
> files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_idv37&alloc_id865&op=click
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] nacache

[Zoran V. <zv...@ar...>]

Am 30.12.2005 um 15:46 schrieb Vlad Seryakov:

> I have web applications and server applications running on our =20
> server and in my experience so far i always prefer global caches =20
> over per-thread cache. If you need something local in long running =20
> thread, you can use Tcl global variables to keep state during the =20
> processing, even namespace variables. It is just one Tcl =20
> interpreter anyway.

=10True. But the state can be accumulating over time, though,
leading to extensive memory usage. Hence the cache, as it
can selecively expire things.

But...

I haven't any code needing per-thread caches, yet.
As a tool in the toolbox, OK, but I will not press
on that. I'm perfectly happy with global caches.

Zoran=

Re: [naviserver-devel] nacache

[Vlad S. <vl...@cr...>]

I have web applications and server applications running on our server 
and in my experience so far i always prefer global caches over 
per-thread cache. If you need something local in long running thread, 
you can use Tcl global variables to keep state during the processing, 
even namespace variables. It is just one Tcl interpreter anyway.
Just my 2 cents.

Zoran Vasiljevic wrote:
> 
> Am 30.12.2005 um 08:46 schrieb Stephen Deasey:
> 
>>
>> But I don't know, there may be situations where a per-thread cache is
>> worht while and we should add that...?
> 
> 
> Hm... for a typical web application (whatever that is) where you have
> short lived connections, it is not worth the work, IMO.
> But if you have a long running operation in a thread, a per-thread
> cache seems like a good idea. There you can stuff (and reuse) Tcl
> objects w/o shimmering to/from string.
> To be honest, at the moment I do not have any need for such thing
> but on the general level, an option to create a cache as "global"
> vs. thread-private is a powerful functionality.
> 
> Cheers
> Zoran
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log 
> files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Naviserver and OACS (compatibility)?

[Zoran V. <zv...@ar...>]

Am 30.12.2005 um 12:46 schrieb Stephen Deasey:

> On 12/30/05, Zoran Vasiljevic <zv...@ar...> wrote:
>>
>> Am 30.12.2005 um 03:28 schrieb Vlad Seryakov:
>>
>>>  So, now with your patch it works the same way as aolserver? Then
>>> keep it.
>>
>> We will have to maintain a path between compatibility and cleanness.
>> Not for any price but at places where it seems oportune.
>> I agree with Vlad.
>>
>> Zoran
>
>
> Oh.  Looks like Gustav hacked around this already:
>
> http://cvs.openacs.org/cvs/openacs-4/packages/acs-tcl/tcl/request- 
> processor-procs.tcl? 
> r1=1.82.2.2&r2=1.82.2.3&sortby=date&only_with_tag=oacs-5-2
>
> If that's OK, then we could after all swap the args for filter procs.
> Old code wouldn't work though.  What to do?

I would not require people to use such hacks, if possible.
But I've been looking in the AS code and saw that ugly
      GetNumArgs(Tcl_Interp *interp, Proc *procPtr)
which was used to decide wether an arg is expected or not.
Brrrr.... that is pretty ugly.

Why would we need to process that optional argument in the first place?

Lets make the ns_register_filter as:
     ns_register_filter when method urlPattern script

The "script" should be completely specified, for example

    set arg "whatever thing"
    ns_register_filter preauth GET /junk [list myfilter $arg]

During runtime we'd append the WHY so it will be called as

    myfilter "whatever thing" preauth

Wouldn't that be better/cleaner altogether? This will not break
any code using no optional args (we never used this arg thing)
and would be much more logical to understand.

Hm?

Zoran

Re: [naviserver-devel] nacache

[Zoran V. <zv...@ar...>]

Am 30.12.2005 um 08:46 schrieb Stephen Deasey:

>
> But I don't know, there may be situations where a per-thread cache is
> worht while and we should add that...?

Hm... for a typical web application (whatever that is) where you have
short lived connections, it is not worth the work, IMO.
But if you have a long running operation in a thread, a per-thread
cache seems like a good idea. There you can stuff (and reuse) Tcl
objects w/o shimmering to/from string.
To be honest, at the moment I do not have any need for such thing
but on the general level, an option to create a cache as "global"
vs. thread-private is a powerful functionality.

Cheers
Zoran

Re: [naviserver-devel] Naviserver and OACS (compatibility)?

[Stephen D. <sd...@gm...>]

On 12/30/05, Zoran Vasiljevic <zv...@ar...> wrote:
>
> Am 30.12.2005 um 03:28 schrieb Vlad Seryakov:
>
> >  So, now with your patch it works the same way as aolserver? Then
> > keep it.
>
> We will have to maintain a path between compatibility and cleanness.
> Not for any price but at places where it seems oportune.
> I agree with Vlad.
>
> Zoran


Oh.  Looks like Gustav hacked around this already:

http://cvs.openacs.org/cvs/openacs-4/packages/acs-tcl/tcl/request-processor=
-procs.tcl?r1=3D1.82.2.2&r2=3D1.82.2.3&sortby=3Ddate&only_with_tag=3Doacs-5=
-2

If that's OK, then we could after all swap the args for filter procs.=20
Old code wouldn't work though.  What to do?

Is there a simillar problem with scheduled procs? I haven't investigated.

http://cvs.openacs.org/cvs/openacs-4/packages/acs-tcl/tcl/utilities-procs.t=
cl?r1=3D1.83.2.4&r2=3D1.83.2.5&sortby=3Ddate&only_with_tag=3Doacs-5-2


If bugs are reported we can fix them  :-)

Re: [naviserver-devel] Naviserver and OACS (compatibility)?

[Zoran V. <zv...@ar...>]

Am 30.12.2005 um 03:28 schrieb Vlad Seryakov:

>  So, now with your patch it works the same way as aolserver? Then  
> keep it.

We will have to maintain a path between compatibility and cleanness.
Not for any price but at places where it seems oportune.
I agree with Vlad.

Zoran

Re: [naviserver-devel] nacache

[Stephen D. <sd...@gm...>]

On 12/29/05, Andrew Piskorski <at...@pi...> wrote:
> On Thu, Dec 29, 2005 at 07:11:35PM -0700, Stephen Deasey wrote:
>
> > Still to think about are per-thread Tcl caches.  The main reason for
> > these are that you can put Tcl objects in the cache, not just their
> > string rep, which is particularly useful for already-compiled byte
> > code -- i.e. for *.tcl pages.
>
> Tcl code in *.tcl pages (not in procs) can be, has been, and I hope
> currently still is cached in OpenACS using the mechanism Rob Mayoff
> added to AOLserver 3.3 long ago.  Essentially, all Tcl code in each
> *.tcl page is automatically "compiled" to Tcl procs on first
> execution, and those procs' bytecode is then cached via ns_cache.
> Each connection thread does its own entirely independent cacheing, but
> it worked fine and apparently was a substantial speed up for some ACS
> sites when switching from AOLserver 3.3+ad11 or 3.3+ad12 (I forget
> which) to 3.3+ad13.
>
> Actually, I'm not sure whether that code is currently active or not.
> It sounds as if the feature may have broke during the AOLserver 3.x to
> 4.x transition and never been repaired.  See here for more info:
>
>   http://sourceforge.net/tracker/?func=3Ddetail&aid=3D689515&group_id=3D3=
152&atid=3D353152


ACS hasn't used that for a long time.  It handles *.adp and *.tcl
pages itself.  ACS creates procs named after the file to evaluate at
runtime.  It does this for each thread the first time a *.tcl page is
called.  There is no eviction policy, therefore it's more of a memory
leak than a cache.

Rob's old method was to slurp the contents of the *.tcl file into a
variable and eval it.  The variable would be byte code compiled the
first time it was evaled.  Because of limitations in Tcl, byte code
objects cannot be shared across threads, so to save the byte code it
was necessary to have a per-thread Tcl cache.


> Does Rob's cacheing cover the functionality you want, or are you
> thinking of cacheing of some other sort of Tcl objects?


Rob's per-thread Tcl cache can cache any Tcl object.  It's just a hash
table with an eviction policy (max size or time limit).


> > The best solution for caching byte code might be to share the
> > mechanism with ADP pages.  They currently have a pre-thread cache of
> > script blocks, and a per-server cache of text blocks, for each page.
> > AOLserver 4.5 adds a per-server output cache.  We could use this for
> > *.tcl pages as well, perhaps negating the need for per-thread Tcl
> > caches.  That still leaves Tcl objects like lists or dictionaries that
> > you may not want to stringify on each access.
>
> Ah.  I think Rob's code above solely cached compiled Tcl bytecode (in
> what form I don't know), not Tcl_Obj representations of anything.
> I've also no idea whether it plays nicely with Zoran's Ttrace
> extension or not.
>
> But, we already have centralized server-wide cacheing of Tcl_Obj's via
> Zoran's Tcl Threads Extension, right?  So, is there any real use case
> for a per-thread Tcl_Obj cache?  (Offhand I can't think of one, but
> then I don't think I've ever really used ns_cache directly at all,
> only the simplified util_memoize facility of OpenACS, so I'm not the
> right person to ask.)


Kind of.  The Tcl thread package has nsv arrays, with the added twist
that certain Tcl objects do not have to be serialised to their string
form.  This isn't a cache though -- there's no eviction policy.  Also,
compared to a per-thread Tcl cache there will be locking overhead.

It's a trade off though.  If you have a small amount of data and/or if
the data is expensive to serialise or is to be evaled, it may make
sense to put it in a per-thread cache.  Otherwise, save the memory and
share it between threads.

If you look at the old nscache module you'll see basically two
completely independent code paths for per-thread and shared caches.  I
didn't want to go that far.  I also thought that perhaps the main use
case was for caching byte code, and that might be better done with the
ADP caching mechanism.

But I don't know, there may be situations where a per-thread cache is
worht while and we should add that...?