naviserver-devel

Re: [naviserver-devel] [naviserver-commits] naviserver ChangeLog, 1.405, 1.406

[Stephen D. <sd...@gm...>]

On 6/19/06, Vlad Seryakov <vl...@cr...> wrote:
> Stephen,
>
> I understand you have the "right" vision how to do things but after long
> period of time just removing Tcl API functions (cache set/get) is not
> very polite.
> It breaks a lot of Tcl code and i am not sure why do you think they are
> racey and i need to emulate them in Tcl instead of C.
>
> Very frustrating.


We talked about this before.  If I remember correctly, you didn't
disagree with me that the commands were racy, you said you code base
was large and you didn't have time to figure out what you really
needed.

More and more unnecessary code is being added.  Things are getting out
of control.  However, you'll notice that I did not just remove
everything.  With the new -force switch to ns_cache_eval, _set, _get,
and _info can be trivially emulated in a couple of lines of Tcl, if
you want that kind of racy behaviour.

Everything is of course still available via the C api, and I'd
recommend taking a look at what your real requirements are and
building accordingly.  If it can be generalised, we can get it into
the core.


Sorry this is hurried.  I don't have 'net access.  I'll be travelling
for the next couple of weeks.


> Stephen Deasey wrote:
> > Update of /cvsroot/naviserver/naviserver
> > In directory sc8-pr-cvs7.sourceforge.net:/tmp/cvs-serv27993
> >
> > Modified Files:
> >       ChangeLog
> > Log Message:
> >         * nsd/fastpath.c:
> >         * nsd/adpeval.c: Use Ns_CacheDeleteEntry instead of
> >         Ns_CacheFlushEntry to prevent genuine flush stats being inflated.
> >
> >         * tcl/cache.tcl:
> >         * nsd/nsconf.c:
> >         * nsd/dns.c: Update to new API.
> >
> >         * nsd/nsd.h:
> >         * nsd/server.c: Remove tcl cache timeout param in anticipation of new
> >         limits API.
> >
> >         * nsd/tclcmds.c: Remove ns_cache _get _set _exists _info
> >         commands. The first three are racey and can be emulated with _eval
> >         and the new -force switch.  The new -contents switch to the _stats
> >         command replaces _info, discouraging racey check-and-set usage.
> >
> >         * nsd/tcltime.c:
> >         * nsd/tclobjv.c: New Ns_ObjvTime parse callback for the Ns_Time
> >         type. Handy for passing absolute time deadlines to routines which
> >         timeout.
> >
> >         * include/ns.h:
> >         * nsd/cache.c: New Ns_CacheResetStats routine.
> >
> >         Ns_CacheGetConfig removed -- caches are immutable; the current
> >         settings can be looked up in the config file.
> >
> >         Directly expire entries from within all routines which return one,
> >         but but only if the value is not null, indicating no concurrent
> >         update is in progress.
> >
> >         Use Ns_CacheDeleteEntry to remove an entry from the cache without
> >         updating the stats. Flush stats should reflect explicit flushes
> >         only.
> >
> >         Log stats when a cache is destroyed.
> >
> >         * nsd/tclcache.c: ns_cache_create now takes a -maxentry option
> >         which is the maximum size of an entry allowed in the cache. This
> >         prevents one large entry completely emptying an otherwise usefully
> >         full cache.
> >
> >         The ns_cache_create -timeout switch has been removed in
> >         anticipation of a general purpose limits scheme.
> >
> >         The -timeout option is now expected to be an absolute time in the
> >         future, not an offset from the current time. This is awkward to
> >         use manually, but should not be needed with the above mentioned
> >         limits scheme. The -ttl option has been renames -expires and is
> >         also an absolute time.
> >
> >         ns_cache_eval now takes a -force switch which will unconditioanly
> >         replace any exiting entry, whether it has expired or not. Replaces
> >         _set.
> >
> >         ns_cache_stats -content dumps the size and expirey for each
> >         entry in the cache. Adding the -reset switch to either mode resets
> >         the stats to zero.
> >
> >         * tests/ns_cache.test: Try to exercise the underlying Ns_CacheFind
> >         harder with some more flush tests.
> >
> >
> >
> > Index: ChangeLog
> > ===================================================================
> > RCS file: /cvsroot/naviserver/naviserver/ChangeLog,v
> > retrieving revision 1.405
> > retrieving revision 1.406
> > diff -C2 -d -r1.405 -r1.406
> > *** ChangeLog 19 Jun 2006 19:05:47 -0000      1.405
> > --- ChangeLog 19 Jun 2006 21:16:38 -0000      1.406
> > ***************
> > *** 1,2 ****
> > --- 1,71 ----
> > + 2006-06-19  Stephen Deasey  <sd...@us...>
> > +
> > +     * nsd/fastpath.c:
> > +     * nsd/adpeval.c: Use Ns_CacheDeleteEntry instead of
> > +     Ns_CacheFlushEntry to prevent genuine flush stats being inflated.
> > +
> > +     * tcl/cache.tcl:
> > +     * nsd/nsconf.c:
> > +     * nsd/dns.c: Update to new API.
> > +
> > +     * nsd/nsd.h:
> > +     * nsd/server.c: Remove tcl cache timeout param in anticipation of new
> > +     limits API.
> > +
> > +     * nsd/tclcmds.c: Remove ns_cache _get _set _exists _info
> > +     commands. The first three are racey and can be emulated with _eval
> > +     and the new -force switch.  The new -contents switch to the _stats
> > +     command replaces _info, discouraging racey check-and-set usage.
> > +
> > +     * nsd/tcltime.c:
> > +     * nsd/tclobjv.c: New Ns_ObjvTime parse callback for the Ns_Time
> > +     type. Handy for passing absolute time deadlines to routines which
> > +     timeout.
> > +
> > +     * include/ns.h:
> > +     * nsd/cache.c: New Ns_CacheResetStats routine.
> > +
> > +     Ns_CacheGetConfig removed -- caches are immutable; the current
> > +     settings can be looked up in the config file.
> > +
> > +     Directly expire entries from within all routines which return one,
> > +     but but only if the value is not null, indicating no concurrent
> > +     update is in progress.
> > +
> > +     Use Ns_CacheDeleteEntry to remove an entry from the cache without
> > +     updating the stats. Flush stats should reflect explicit flushes
> > +     only.
> > +
> > +     Log stats when a cache is destroyed.
> > +
> > +     * nsd/tclcache.c: ns_cache_create now takes a -maxentry option
> > +     which is the maximum size of an entry allowed in the cache. This
> > +     prevents one large entry completely emptying an otherwise usefully
> > +     full cache.
> > +
> > +     The ns_cache_create -timeout switch has been removed in
> > +     anticipation of a general purpose limits scheme.
> > +
> > +     The -timeout option is now expected to be an absolute time in the
> > +     future, not an offset from the current time. This is awkward to
> > +     use manually, but should not be needed with the above mentioned
> > +     limits scheme. The -ttl option has been renames -expires and is
> > +     also an absolute time.
> > +
> > +     ns_cache_eval now takes a -force switch which will unconditioanly
> > +     replace any exiting entry, whether it has expired or not. Replaces
> > +     _set.
> > +
> > +     ns_cache_stats -content dumps the size and expirey for each
> > +     entry in the cache. Adding the -reset switch to either mode resets
> > +     the stats to zero.
> > +
> > +
> > +
> > +
> > +
> > +     * tests/ns_cache.test: Try to exercise the underlying Ns_CacheFind
> > +     harder with some more flush tests.
> > +
> >   2006-05-19  Vlad Seryakov  <ser...@us...>
> >
> >
> >
> >
> > _______________________________________________
> > naviserver-commits mailing list
> > nav...@li...
> > https://lists.sourceforge.net/lists/listinfo/naviserver-commits
> >
>
> --
> Vlad Seryakov
> 571 262-8608 office
> vl...@cr...
> http://www.crystalballinc.com/vlad/
>
>
>
> _______________________________________________
> naviserver-commits mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-commits
>

Re: [naviserver-devel] [naviserver-commits] naviserver ChangeLog, 1.405, 1.406

[Vlad S. <vl...@cr...>]

> 
> We talked about this before.  If I remember correctly, you didn't
> disagree with me that the commands were racy, you said you code base
> was large and you didn't have time to figure out what you really
> needed.

That was about tls not cache

> More and more unnecessary code is being added.  Things are getting out
> of control.  However, you'll notice that I did not just remove
> everything.  With the new -force switch to ns_cache_eval, _set, _get,
> and _info can be trivially emulated in a couple of lines of Tcl, if
> you want that kind of racy behaviour.

Here is the thing, if i need to test for entry existence in the cache i 
need touse ns_cache_eval which will create this entry, same with
ns_cache_get, why do i have to create it when i want to test existence 
only, info exists does not create Tcl variables

And why ns_cache_eval -force is not racy but than ns_cache_set is if 
they do the same thing, why cryptic API flags instead of clear 
functions. If have them for nsv_ for example.

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] [naviserver-commits] naviserver ChangeLog, 1.405, 1.406

[Stephen D. <sd...@gm...>]

On 6/21/06, Vlad Seryakov <vl...@cr...> wrote:
> >
> > We talked about this before.  If I remember correctly, you didn't
> > disagree with me that the commands were racy, you said you code base
> > was large and you didn't have time to figure out what you really
> > needed.
>
> That was about tls not cache
>
> > More and more unnecessary code is being added.  Things are getting out
> > of control.  However, you'll notice that I did not just remove
> > everything.  With the new -force switch to ns_cache_eval, _set, _get,
> > and _info can be trivially emulated in a couple of lines of Tcl, if
> > you want that kind of racy behaviour.
>
> Here is the thing, if i need to test for entry existence in the cache i
> need touse ns_cache_eval which will create this entry, same with
> ns_cache_get, why do i have to create it when i want to test existence
> only, info exists does not create Tcl variables


If you need to test for existence, you're doing something wrong.
Between the test and the action taken depending on the result, the
result could have changed.

However:

   ns_cache_eval $cache $key {error "no entry available"}

You can catch the error and do whatever, no new entry will be created.


> And why ns_cache_eval -force is not racy but than ns_cache_set is if
> they do the same thing, why cryptic API flags instead of clear
> functions. If have them for nsv_ for example.


It depends how you use them.  The use case of pre-filling a cache in
bulk at start up for example might be a good use of this.  In combo
with _info is not.  The -force switch was a ~3 line addition.

Re: [naviserver-devel] [naviserver-commits] naviserver ChangeLog, 1.405, 1.406

[Vlad S. <vl...@cr...>]

> 
> If you need to test for existence, you're doing something wrong.
> Between the test and the action taken depending on the result, the
> result could have changed.
> 
> However:
> 
>    ns_cache_eval $cache $key {error "no entry available"}
> 
> You can catch the error and do whatever, no new entry will be created.

Again, you assuming only one way of using caches, in my case i am not 
afraid what happens in between because, i just need to know if somebody 
put anything in the cache, similar to flag. Very simple test, and does 
not need to generate and cache error exception which will make Tcl doe 
look ugly. That's why we have info exists in Tcl instead of error 
handling if we need to test existence.

> 
>> And why ns_cache_eval -force is not racy but than ns_cache_set is if
>> they do the same thing, why cryptic API flags instead of clear
>> functions. If have them for nsv_ for example.
> 
> 
> It depends how you use them.  The use case of pre-filling a cache in
> bulk at start up for example might be a good use of this.  In combo
> with _info is not.  The -force switch was a ~3 line addition.

Right, so it is up to the developer to decide how to use primitive cache 
routines, and i never said i use _info first then _set next, but i can 
see it may be used in environment where serialized jobs are running in 
different threads and i need to know what previous job have done. In 
this case _info then _set or _get is pretty valid, it uses cache as a 
storage in non-concurrent environment.

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] [naviserver-commits] naviserver ChangeLog, 1.405, 1.406

[Stephen D. <sd...@gm...>]

On 6/25/06, Vlad Seryakov <vl...@cr...> wrote:
> >
> > If you need to test for existence, you're doing something wrong.
> > Between the test and the action taken depending on the result, the
> > result could have changed.
> >
> > However:
> >
> >    ns_cache_eval $cache $key {error "no entry available"}
> >
> > You can catch the error and do whatever, no new entry will be created.
>
> Again, you assuming only one way of using caches, in my case i am not
> afraid what happens in between because, i just need to know if somebody
> put anything in the cache, similar to flag. Very simple test, and does
> not need to generate and cache error exception which will make Tcl doe
> look ugly. That's why we have info exists in Tcl instead of error
> handling if we need to test existence.
>
> >
> >> And why ns_cache_eval -force is not racy but than ns_cache_set is if
> >> they do the same thing, why cryptic API flags instead of clear
> >> functions. If have them for nsv_ for example.
> >
> >
> > It depends how you use them.  The use case of pre-filling a cache in
> > bulk at start up for example might be a good use of this.  In combo
> > with _info is not.  The -force switch was a ~3 line addition.
>
> Right, so it is up to the developer to decide how to use primitive cache
> routines, and i never said i use _info first then _set next, but i can
> see it may be used in environment where serialized jobs are running in
> different threads and i need to know what previous job have done. In
> this case _info then _set or _get is pretty valid, it uses cache as a
> storage in non-concurrent environment.

Communication between serialized threads doesn't sound like a cache to me.

Anyway, the idea is that if you need to do weird stuff it should be
possible, and an ns_cache_info equivalent can be writen in ~3 lines of
Tcl.

Re: [naviserver-devel] [naviserver-commits] naviserver ChangeLog, 1.405, 1.406

[Vlad S. <vl...@cr...>]

> 
> Communication between serialized threads doesn't sound like a cache to me.
> Anyway, the idea is that if you need to do weird stuff it should be
> possible, and an ns_cache_info equivalent can be writen in ~3 lines of
> Tcl.

Very sad.

I'd suggest we define what is weird and what is not first.

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] [naviserver-commits] naviserver ChangeLog, 1.405, 1.406

[Zoran V. <zv...@ar...>]

Am 29.06.2006 um 19:21 schrieb Vlad Seryakov:

>>
>> Communication between serialized threads doesn't sound like a  
>> cache to me.
>> Anyway, the idea is that if you need to do weird stuff it should be
>> possible, and an ns_cache_info equivalent can be writen in ~3  
>> lines of
>> Tcl.
>
> Very sad.
>
> I'd suggest we define what is weird and what is not first.

I believe Stephen is trying to say that if you use cache module
to save values between two tasks executed by two serialized
thredas, this is kind of unusual "usage".
I must admit that Stephen is right. But, OTOH, we should not
question anybody's usage that deep. If you or Stephen
say that any feature is needed I will simply trust you and
try to arrange myself accordingly. Of course, this is what I
expect from you in return. I will never ask for a feature I
did not think over and am absolutely sure (and can convince you)
that it is better we add it in the core.

Addind something in the code is easily done but we should NOT
forget that we will have to maintain it for years to come.
So, if something can really be done in a small Tcl wrapper
and IS not performance hog I would vote to do it in Tcl.
Otherwise I would be the last to stand in the way to do it
in the C as part of the common core.

Just my 0.2 cents

Zoran
(who tries to keep the ball rolling)

Re: [naviserver-devel] [naviserver-commits] naviserver ChangeLog, 1.405, 1.406

[Vlad S. <vl...@cr...>]

> 
> I believe Stephen is trying to say that if you use cache module
> to save values between two tasks executed by two serialized
> thredas, this is kind of unusual "usage".

I always thought that shared memory usage is usual as long as i am using 
public API. Limiting API in functionality will lead to workarounds.

Can you image file access API with only open,close,read,write without 
stat call? I can open file, read it and find out the size of course but 
does is look right?

So, i am being told all the time that i do not use the ns_cache in right 
way, then somebody can define what is the correct and the only right way 
of using ns_cache facility in the server environment? I could be lost or 
un-educated enough but i would like to know?

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] [naviserver-commits] naviserver ChangeLog, 1.405, 1.406

[Zoran V. <zv...@ar...>]

Am 29.06.2006 um 22:44 schrieb Vlad Seryakov:

> So, i am being told all the time that i do not use the ns_cache in  
> right
> way, then somebody can define what is the correct and the only  
> right way
> of using ns_cache facility in the server environment? I could be  
> lost or
> un-educated enough but i would like to know?
>

It is not the question of "right" or "wrong"! Lets not misunderstand
each other.
I can't tell you what is "right" or what's "wrong" but I can try to
explain how I see the "usual" cache usage....

Noormally I'd use cache to save data which is rather expensive to
calculate. In that case I will temporary save (hard calculated)
data into some (size-limited) storage and read it from there.
I also expect that this data will not change for some (rather short)
period of time and for that time I will retain it in the cache.

A real-world examnple would be DNS lookup, or SQL selection
or things lie that. In our code, I extensively use caches
to store ID's of elements in a tree structure (similar to
directory) which is emulated in 2 dimension table(s).
It is rather expensive to get that info and since it does not
change that often, I put those things in the cache.

I will however NOT use cache structures to maintain values used
for commumincation or decision purposes. I would use nsv's or
files or similar.

Having said that, I must stress that it is perfectly allright
for me that anybody uses whatever tools he needs to get the job
done. It is just that I see cache module usage in a certain patterns
which may or may not correspond to the view of other people.

So what we are all about here is to understand what EXACTLY you
are after and perhaps see if the cache as-is is the optimal solution
for that, because our experience may be different from yours.
But: if you say: guys, I know what I need, then I'm allright with that
and will support you all the way.

Allright?

Cheers,
Zoran

Re: [naviserver-devel] [naviserver-commits] naviserver ChangeLog, 1.405, 1.406

[Vlad S. <vl...@cr...>]

I am not going to argue, it is just old ns_cache used to have all 
functionality and was very popular and useful. Now we have more limited 
ns_cache_XXX family and standalone ns_cache does not work anymore.
I do not see who benefits from all this.

Zoran Vasiljevic wrote:
> Am 29.06.2006 um 22:44 schrieb Vlad Seryakov:
> 
>> So, i am being told all the time that i do not use the ns_cache in  
>> right
>> way, then somebody can define what is the correct and the only  
>> right way
>> of using ns_cache facility in the server environment? I could be  
>> lost or
>> un-educated enough but i would like to know?
>>
> 
> It is not the question of "right" or "wrong"! Lets not misunderstand
> each other.
> I can't tell you what is "right" or what's "wrong" but I can try to
> explain how I see the "usual" cache usage....
> 
> Noormally I'd use cache to save data which is rather expensive to
> calculate. In that case I will temporary save (hard calculated)
> data into some (size-limited) storage and read it from there.
> I also expect that this data will not change for some (rather short)
> period of time and for that time I will retain it in the cache.
> 
> A real-world examnple would be DNS lookup, or SQL selection
> or things lie that. In our code, I extensively use caches
> to store ID's of elements in a tree structure (similar to
> directory) which is emulated in 2 dimension table(s).
> It is rather expensive to get that info and since it does not
> change that often, I put those things in the cache.
> 
> I will however NOT use cache structures to maintain values used
> for commumincation or decision purposes. I would use nsv's or
> files or similar.
> 
> Having said that, I must stress that it is perfectly allright
> for me that anybody uses whatever tools he needs to get the job
> done. It is just that I see cache module usage in a certain patterns
> which may or may not correspond to the view of other people.
> 
> So what we are all about here is to understand what EXACTLY you
> are after and perhaps see if the cache as-is is the optimal solution
> for that, because our experience may be different from yours.
> But: if you say: guys, I know what I need, then I'm allright with that
> and will support you all the way.
> 
> Allright?
> 
> Cheers,
> Zoran
> 
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] [naviserver-commits] naviserver ChangeLog, 1.405, 1.406

[Bernd E. <eid...@we...>]

Am Freitag, 30. Juni 2006 00:43 schrieb Vlad Seryakov:
> I am not going to argue, it is just old ns_cache used to have all
> functionality and was very popular and useful. Now we have more limited
> ns_cache_XXX family and standalone ns_cache does not work anymore.
> I do not see who benefits from all this.

This is something that should not happen. It might have happend because
a) there is a desire to straighten and improve code and logic of the "legacy" 
where appropriate (good!)
b) there is a desire to include new functionality (good!)
c) there is the need to use existing applications with naviserver "as is" and 
incompatibilities are to avoid. This is also a signal to new naviserver users  
(good!)
and while a,b,c are desirable for themselves it's not always easy or possible 
to achieve everything at once.

We should solve the reliance on a working HEAD so nobody is forced to code in 
god mode.

Tagging could be used more often. So it is possible for everyone of us to tell 
others things like "I use naviserver-pre-5.0-1 (...) for 3 weeks now in 
production and everything works" - and of course rollback if there is need.

There could at a minimum be one branch that allows everyone to implement and 
try new functionality that may even break usage for n days. Of course 
everyone can develop locally but than this effort is invisible and/or it just 
leads to monster commits. There would be no need to discuss everything 
everytime as the main branch is not affected; a merge can then be discussed 
without haste: "I refactored the whole xyz part - give it a try". So the ball 
keeps rolling at full speed.

Bernd.

Re: [naviserver-devel] [naviserver-commits] naviserver ChangeLog, 1.405, 1.406

[Zoran V. <zv...@ar...>]

Am 30.06.2006 um 00:43 schrieb Vlad Seryakov:

> I am not going to argue, it is just old ns_cache used to have all
> functionality and was very popular and useful. Now we have more  
> limited
> ns_cache_XXX family and standalone ns_cache does not work anymore.
> I do not see who benefits from all this.

The point is to have simpler, better and more stable cache code.
Quality of the API does not necessarily relate to the size i.e.
number of calls. So, API with 10 calls is not more limited than
the API with 100 calls, if the 10 calls is all people "really"
need.

But, do not get me wrong. I'm not in favour of stripping functionality.
As far as I', concerned, ns_cache API can consist of only three
calls:

    ns_cache_create
    ns_cache_eval
    ns_cache_flush

More is not needed for the "usual" cache operation. Now, one
can imagine adding some operations on the cache vaules directly
like

    incr
    llength
    lindex

and atomic operations lile

    set
    exists
    get

which is all allright but in some "sense" diverts from the
base idea of cache: just temporary store values resulting
from hard caclulations.

Frankly speeaking, I do not have anything against expanding
this API to accomodate all that. If you look at the threading
extension, you will see the "thread::tsv" (thread shared variables)
which is nsv on steroids. So, I went there and added tons of additional
options and commands because nsv is designed for cross-therad storage
of values. You never go and purge nsv arrays on size and on time
i.e you never go and prune the nsv automatically. So the question is:
wny using nscache when nsv *may* be the more appropriate vehicle?
I'm not saying that it is in your particular usage case. I just say
that I *think* it would serve better. But you should know this better
because you write your own code.

I will going to backport the thread::tsv code from the Tcl
threading extension into the NS and this will bring you much more
power into nsv than ns_cache will ever have. It is just the question:
do you need AUTOMATIC purging (on time or size constraints) for the
values you keep now in ns_cache?

Please do not think that I'm blocking you from anything. I' m just
trying to understand the needs and provide optimal solution. If at the
end we see the need to expand ns_cache to handle all that nsv does
PLUS automatic entry expiry, then we'l do that. But then we'll know the
reason why and this will justify adding and maintaining more code.

So, relax. I'm the last person on the earth to block anybody ideas.
We'll sort all this out. But I just need *some* insights on how
this code is used in your application.  To make a good example,
I will try to write how I see that code from our's perspective i.e.
what we need and what not. This will be a part of a new message
under different subject.

Cheers
Zoran