naviserver-devel — Developer discussion

Re: ns_tls? Was: [naviserver-devel] Re: [naviserver-commits] naviserver/nsd tclcache.c,1.2,1.3 tclcmds.c,1.34,1.35

[Zoran V. <zv...@ar...>]

On 18.04.2006, at 15:52, Vlad Seryakov wrote:

> You convinced me that at this moment ns_tls is not appropriate, i  
> will remove it

Oh?! How did I do that? Apart from giving a trivial
alternative, I think that this interface is ok per-se.
It allows "unified" access from Tcl and C code.
That is, one should be able to do the "same" from C
and Tcl. Isn't it so?

Cheers
Zoran

Re: ns_tls? Was: [naviserver-devel] Re: [naviserver-commits] naviserver/nsd tclcache.c,1.2,1.3 tclcmds.c,1.34,1.35

[Vlad S. <vl...@cr...>]

You convinced me that at this moment ns_tls is not appropriate, i will 
remove it

Zoran Vasiljevic wrote:
> 
> On 16.04.2006, at 19:20, Vlad Seryakov wrote:
> 
>>
>> Using Ns_Tls in C looks very usafull, why not having the same in Tcl 
>> as well?
> 
> I belive what you did not "sell" correctly is the ability
> to have C and Tcl interface to same data in a "simple" manner.
> Because in my eyes, this is the major "pro".
> 
> Normally, I would not buy ns_tls interface if it were just
> a replacement for setting a global variable in Tcl. Because
> you can very simply setup a tiny Tcl wrapper:
> 
> proc ns_tls_set {key val}
>     set private::tls($key) $val
> }
> proc ns_tls_get {key}
>     set private::tls($key)
> }
> 
> (add some more simple code if you'd be using
>  [interp create] or such...)
> 
> But the fact that you can access the same things
> from C code makes ns_tls worth considering (still, you need
> to have some conventions in place about data-types...)
> 
> A good example of this dual interface is ns_config.
> It can be used from C and Tcl, which is why I was contemplating
> of adding more features there (perhaps allowing read/write operations
> after server startup).
> 
> I haven't check the implementation but you need to be careful
> if you plan to store Tcl_Obj's references there. I can't recall eactly
> but there might be problems when you make some Tcl_Obj's jump the
> interpreters...
> 
> Cheers
> Zoran
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live 
> webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

ns_tls? Was: [naviserver-devel] Re: [naviserver-commits] naviserver/nsd tclcache.c,1.2,1.3 tclcmds.c,1.34,1.35

[Zoran V. <zv...@ar...>]

On 16.04.2006, at 19:20, Vlad Seryakov wrote:

>
> Using Ns_Tls in C looks very usafull, why not having the same in  
> Tcl as well?

I belive what you did not "sell" correctly is the ability
to have C and Tcl interface to same data in a "simple" manner.
Because in my eyes, this is the major "pro".

Normally, I would not buy ns_tls interface if it were just
a replacement for setting a global variable in Tcl. Because
you can very simply setup a tiny Tcl wrapper:

proc ns_tls_set {key val}
     set private::tls($key) $val
}
proc ns_tls_get {key}
     set private::tls($key)
}

(add some more simple code if you'd be using
  [interp create] or such...)

But the fact that you can access the same things
from C code makes ns_tls worth considering (still, you need
to have some conventions in place about data-types...)

A good example of this dual interface is ns_config.
It can be used from C and Tcl, which is why I was contemplating
of adding more features there (perhaps allowing read/write operations
after server startup).

I haven't check the implementation but you need to be careful
if you plan to store Tcl_Obj's references there. I can't recall eactly
but there might be problems when you make some Tcl_Obj's jump the
interpreters...

Cheers
Zoran

Re: [naviserver-devel] ns_tls ?

[Vlad S. <vl...@cr...>]

> 
> But for nsv shared variables you *have* to stringify because Tcl objs
> can not pass between threads.  Here your explicitly looking for thread
> local, so stringifying would seem to be an unnecessary slowdown.

Mostly we store strings in the cache so it is not a real problem

> This isn't a cache, there's no eviction policy.
> 

But if i do not need eviction policy, if i want to keep them forever?

> I don't see how ns_tlsthis isn't exactly equivalent to:
> 
>   set ::tcl_MyMache Whatever
> 
> which has the big advantage of keeping Tcl objects in object form.

True, and i am still using it this way, i even use  global arrays to be 
able to get all values easily. But i have a lot of filters and callback 
procedures that are called from different places even from C modules. 
Calling Tcl wrapper all the time is a big slow down especially if i need 
to process several hundreds of rows, tls will be a big improvement in 
this case.

But i agree that functionality-wise it is the same

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Re: [naviserver-commits] naviserver/nsd tclcache.c,1.2,1.3 tclcmds.c,1.34,1.35

[Vlad S. <vl...@cr...>]

> 
> There's no point checking for existence unless you're also going to do
> something with it -- that's a separate operation and that's a race
> condition.

If the cache is used to keep some flag about somehting, other thread can 
just check if flag is set then do something.

> A read only cache doesn't make any sense to me.  If you're not
> entering values at runtime, you're not using the cache feature of
> purging which is the only thing differentiating it from an nsv.  Just
> use nsv's in this case.

When i said readonly i did not mean pure readonly, but not very 
frequently updated cache.

> It's a trade off.  In this case the problem I have is that these extra
> features encourage broken usage.  I really think it's important to
> find the real solution to the problem, which I'm happy to help you do.
>  I'm not keen to add broken features to the code base because yours is
> too big to figure out though.  Hope that doesn't sound too harsh, it's
> not meant to be. 

No offense taken but why is this broken usage, if naviserver would 
propose only one way of doing things it would be very limited and not 
many developers would use it because it would be hard to implement 
different applications. It is not used by many now anyway so this is not 
the point but our application is very different from OpenACS for 
example, it is not only web app but true server backend which implements 
device provisioning, network monitoring and other pretty complicated 
logic. Having ability to do different thing different depending on the 
task is very powerful. If i would have nsv_ arrays only i would never be 
able to implement our system effectively, there is no one right way of 
developing different applications.

Using Ns_Tls in C looks very usafull, why not having the same in Tcl as 
well?


-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] ns_tls ?

[Stephen D. <sd...@gm...>]

On 4/16/06, Vlad Seryakov <vl...@cr...> wrote:
> Yes, it is more convenience but even nsv_ shared variables work the same
> way, keep stringified values and objectifies on the return.

But for nsv shared variables you *have* to stringify because Tcl objs
can not pass between threads.  Here your explicitly looking for thread
local, so stringifying would seem to be an unnecessary slowdown.

> We decided to get rid of thread specific caches and keep global ones
> only, and at that time i was for it but now i think simple thread local
> cache should be in the core to offer more flexibility to the developer.


This isn't a cache, there's no eviction policy.

I don't see how ns_tlsthis isn't exactly equivalent to:

  set ::tcl_MyMache Whatever

which has the big advantage of keeping Tcl objects in object form.

Re: [naviserver-devel] Re: [naviserver-commits] naviserver/nsd tclcache.c,1.2,1.3 tclcmds.c,1.34,1.35

[Stephen D. <sd...@gm...>]

On 4/16/06, Vlad Seryakov <vl...@cr...> wrote:
> No, ns_cache_exists should be kept, it is useful sometimes to check if
> the entry there, for read-only caches for example.

There's no point checking for existence unless you're also going to do
something with it -- that's a separate operation and that's a race
condition.

A read only cache doesn't make any sense to me.  If you're not
entering values at runtime, you're not using the cache feature of
purging which is the only thing differentiating it from an nsv.  Just
use nsv's in this case.

> It's hard to tell exactly the real problem, once your application is
> over 100K lines of Tcl code which implements the whole company's
> OSS/backoffice, there are a lot of specific tasks i have to solve.
> The more flexible and feature rich API is the easier the task is.


It's a trade off.  In this case the problem I have is that these extra
features encourage broken usage.  I really think it's important to
find the real solution to the problem, which I'm happy to help you do.
 I'm not keen to add broken features to the code base because yours is
too big to figure out though.  Hope that doesn't sound too harsh, it's
not meant to be. \

Re: [naviserver-devel] Re: [naviserver-commits] naviserver/nsd tclcache.c,1.2,1.3 tclcmds.c,1.34,1.35

[Vlad S. <vl...@cr...>]

No, ns_cache_exists should be kept, it is useful sometimes to check if 
the entry there, for read-only caches for example.

It's hard to tell exactly the real problem, once your application is 
over 100K lines of Tcl code which implements the whole company's 
OSS/backoffice, there are a lot of specific tasks i have to solve.
The more flexible and feature rich API is the easier the task is.

Stephen Deasey wrote:
> On 4/5/06, Vlad Seryakov <vl...@cr...> wrote:
>> Yes, i use cache and in my case i do need to use get and set in
>> different situations, for example i need to set cache entry regardless
>> of what is there, eval will return me existing value, so i need to flush
>> it first which gets to the same race condition issue.
>> Same with get, i just need to get value if it exists without issuing
>> exists first and then eval, if it is not there i do not want to set it,
>> eval will set it.
> 
> 
> So ns_cache_exists is redundant?  Can we remove that?
> 
> 
> What's the real problem your solving here?  Do you have some custom
> code for invalidating cache entries or something?
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=k&kid0944&bid$1720&dat1642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 


-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] ns_tls ?

[Vlad S. <vl...@cr...>]

Yes, it is more convenience but even nsv_ shared variables work the same 
way, keep stringified values and objectifies on the return.

We decided to get rid of thread specific caches and keep global ones 
only, and at that time i was for it but now i think simple thread local 
cache should be in the core to offer more flexibility to the developer.

But i am not insisting.

Stephen Deasey wrote:
> On 4/14/06, Vlad Seryakov <vl...@cr...> wrote:
>> Yes, i always use Tcl vars but when i have too many different callbacks
>> and filters, using simple mechanism to access named value that is global
>> inside the thread make it easier than performing multiple global
>> statements over the code. If i want to add another var, i need to create
>> Tcl wrapper so i do not need to go over the code and see where i may use
>> it, so in this case ns_tls will make it easier. But in general, global
>> Tcl var is way to go.
> 
> 
> If the main motivation is convenience, can't you just write some
> simple Tcl wrapper around global arrays?  A disadvantage of the way
> you have it now is that everything is stringified before it's stored
> in the set, and re-objectified on the way out.
> 
> This is a pretty specialised function.  I don't think the core is the
> right place for it.
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=k&kid0944&bid$1720&dat1642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 


-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Tcl cache crashes

[Stephen D. <sd...@gm...>]

On 4/14/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 14.04.2006, at 11:58, Zoran Vasiljevic wrote:
>
> > OK. This is the beast:
> >
> > test cache-6.2 {eval ttl} -body {
> >     ns_cache_eval -ttl 1 -- c1 k1 {return a}
> >     after 1500
> >     ns_cache_eval c1 k1 {return b}
> > } -cleanup {
> >     ns_cache_flush c1
> > } -result b
>
> It is now fixed in the CVS. Stephen please read
> the comment I put in the code. I do not know if
> this is really correct as it ignores the expired
> entry and just reuses it. OTOH, if I do it "right"
> i.e. purge then re-create, the test routine starts
> to throw errors. So it is either the test that is
> wrong or the API that is wrong. As it is in CVS now
> it works with the current test routine and does not
> crash. Admitently, expired entries are not expired
> at that place, rather re-used.


There were a couple of errors:  a call to Ns_CacheFreeEntry() crept
into ExpireEntry() when Ns_UnsetValue() was enough.  Looks like a
thinko, mirroring the code of PruneEntry() below it.  Also, cache
flushes were causing Ns_CacheWaitCreateEntry() to fail, which isn't
what callers expect.  The entry is just recreated in that case.  I
split the code between the two similar functions and it seems a lot
clearer what's going on now.

Re: [naviserver-devel] Re: [naviserver-commits] naviserver/nsd tclcache.c,1.2,1.3 tclcmds.c,1.34,1.35

[Stephen D. <sd...@gm...>]

On 4/5/06, Vlad Seryakov <vl...@cr...> wrote:
> Yes, i use cache and in my case i do need to use get and set in
> different situations, for example i need to set cache entry regardless
> of what is there, eval will return me existing value, so i need to flush
> it first which gets to the same race condition issue.
> Same with get, i just need to get value if it exists without issuing
> exists first and then eval, if it is not there i do not want to set it,
> eval will set it.


So ns_cache_exists is redundant?  Can we remove that?


What's the real problem your solving here?  Do you have some custom
code for invalidating cache entries or something?

Re: [naviserver-devel] ns_tls ?

[Stephen D. <sd...@gm...>]

On 4/14/06, Vlad Seryakov <vl...@cr...> wrote:
> Yes, i always use Tcl vars but when i have too many different callbacks
> and filters, using simple mechanism to access named value that is global
> inside the thread make it easier than performing multiple global
> statements over the code. If i want to add another var, i need to create
> Tcl wrapper so i do not need to go over the code and see where i may use
> it, so in this case ns_tls will make it easier. But in general, global
> Tcl var is way to go.


If the main motivation is convenience, can't you just write some
simple Tcl wrapper around global arrays?  A disadvantage of the way
you have it now is that everything is stringified before it's stored
in the set, and re-objectified on the way out.

This is a pretty specialised function.  I don't think the core is the
right place for it.

Re: [naviserver-devel] ns_tls ?

[Vlad S. <vl...@cr...>]

Yes, i always use Tcl vars but when i have too many different callbacks 
and filters, using simple mechanism to access named value that is global 
inside the thread make it easier than performing multiple global 
statements over the code. If i want to add another var, i need to create 
Tcl wrapper so i do not need to go over the code and see where i may use 
it, so in this case ns_tls will make it easier. But in general, global 
Tcl var is way to go.

Zoran Vasiljevic wrote:
> 
> On 14.04.2006, at 18:36, Vlad Seryakov wrote:
> 
>> This is very subjective but it will make my life much easier and 
>> application more cleaner by using Tls in Tcl, so if nobody will object 
>> i would keep ns_tls command if possible.
> 
> Absolutely! I have no problem with that, whatsoever.
> It is just that I was curious *when* you start needing
> such things, as we never did. We need(ed) thread-wide
> things (so nsv_ is ok) but not thread-specific.
> Good to know...
> 
> Cheers
> Zoran
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live 
> webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] ns_tls ?

[Zoran V. <zv...@ar...>]

On 14.04.2006, at 18:36, Vlad Seryakov wrote:

> This is very subjective but it will make my life much easier and  
> application more cleaner by using Tls in Tcl, so if nobody will  
> object i would keep ns_tls command if possible.

Absolutely! I have no problem with that, whatsoever.
It is just that I was curious *when* you start needing
such things, as we never did. We need(ed) thread-wide
things (so nsv_ is ok) but not thread-specific.
Good to know...

Cheers
Zoran

Re: [naviserver-devel] ns_tls ?

[Vlad S. <vl...@cr...>]

Sure, currently i am using global Tcl vars to store thread data but over 
the time code that supports this became kind of ugly, too many things i 
need to store during the connection lifetime, i tried multiple 
variables, then global array but for example in C, i use Ns_Tls and this 
works very good, so i kind of wanted to have the same functionality in 
Tcl as well. For one simple var global Tcl is fine but my application is 
very big right now and i had to create the Tcl wrapper around accessing 
global Tcl vars/arrays from different procs and namespaces. Using Tls 
would simplify this and make it very transparent when i want to access 
thread specific global data.

This is very subjective but it will make my life much easier and 
application more cleaner by using Tls in Tcl, so if nobody will object i 
would keep ns_tls command if possible.

Zoran Vasiljevic wrote:
> Hi Vlad,
> 
> Just out of the curiosity... what you need that
> beast for?
> Why just not address some global array or such?
> Tcl interp is per-thread anyways so you get
> this for free.
> Obviously there is some hidden usage otherwise
> you would not go to that extent. Can you share
> it?
> 
> Cheers
> Zoran
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live 
> webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

[naviserver-devel] ns_tls ?

[Zoran V. <zv...@ar...>]

Hi Vlad,

Just out of the curiosity... what you need that
beast for?
Why just not address some global array or such?
Tcl interp is per-thread anyways so you get
this for free.
Obviously there is some hidden usage otherwise
you would not go to that extent. Can you share
it?

Cheers
Zoran

Re: [naviserver-devel] Tcl cache crashes

[Zoran V. <zv...@ar...>]

On 14.04.2006, at 11:58, Zoran Vasiljevic wrote:

> OK. This is the beast:
>
> test cache-6.2 {eval ttl} -body {
>     ns_cache_eval -ttl 1 -- c1 k1 {return a}
>     after 1500
>     ns_cache_eval c1 k1 {return b}
> } -cleanup {
>     ns_cache_flush c1
> } -result b

It is now fixed in the CVS. Stephen please read
the comment I put in the code. I do not know if
this is really correct as it ignores the expired
entry and just reuses it. OTOH, if I do it "right"
i.e. purge then re-create, the test routine starts
to throw errors. So it is either the test that is
wrong or the API that is wrong. As it is in CVS now
it works with the current test routine and does not
crash. Admitently, expired entries are not expired
at that place, rather re-used.

Cheers
Zoran

Re: [naviserver-devel] Tcl cache crashes

[Zoran V. <zv...@ar...>]

On 14.04.2006, at 11:35, Zoran Vasiljevic wrote:

> We are funking all over the place. Now I have to
> isolate the test which triggers the error. This
> will be a long work as I have to disable test per
> test, reboot the server (takes long time under purify)
> and repeat...

OK. This is the beast:

test cache-6.2 {eval ttl} -body {
     ns_cache_eval -ttl 1 -- c1 k1 {return a}
     after 1500
     ns_cache_eval c1 k1 {return b}
} -cleanup {
     ns_cache_flush c1
} -result b


It immediately results in the series of
invalid writes and reads. I will have to
isolate this in detail but here is the
breakdown of Purify:


       FMR: Free memory read
       This is occurring while in thread 13:
             Push           [cache.c:903]
             Ns_CacheWaitCreateEntry [cache.c:295]
             CreateEntry    [tclcache.c:686]
             NsTclCacheEvalObjCmd [tclcache.c:158]
             TclEvalObjvInternal [tclBasic.c:3085]
             Tcl_EvalEx     [tclBasic.c:3687]
       Reading 4 bytes from 0x15c2dd8 in the heap.
       Address 0x15c2dd8 is 8 bytes into a freed  block at 0x15c2dd0  
of 32 bytes.
       This block was allocated from thread 13:
             malloc         [rtlib.o]
             TclpAlloc      [tclAlloc.c:679]
             Tcl_Alloc      [tclCkalloc.c:1002]
             ns_malloc      [memory.c:67]
             ns_calloc      [memory.c:85]
             Ns_CacheWaitCreateEntry [cache.c:260]
       There have been 0 frees since this block was freed from thread  
13:
             free           [rtlib.o]
             TclpFree       [tclAlloc.c:702]
             Tcl_Free       [tclCkalloc.c:1160]
             ns_free        [memory.c:74]
             Ns_CacheDeleteEntry [cache.c:489]
             ExpireEntry    [cache.c:937]
       FMW: Free memory write
       This is occurring while in thread 13:
             Push           [cache.c:906]
             Ns_CacheWaitCreateEntry [cache.c:295]
             CreateEntry    [tclcache.c:686]
             NsTclCacheEvalObjCmd [tclcache.c:158]
             TclEvalObjvInternal [tclBasic.c:3085]
             Tcl_EvalEx     [tclBasic.c:3687]
       Writing 4 bytes to 0x15c2dd4 in the heap.
       Address 0x15c2dd4 is 4 bytes into a freed  block at 0x15c2dd0  
of 32 bytes.
       This block was allocated from thread 13:
             malloc         [rtlib.o]
             TclpAlloc      [tclAlloc.c:679]
             Tcl_Alloc      [tclCkalloc.c:1002]
             ns_malloc      [memory.c:67]
             ns_calloc      [memory.c:85]
             Ns_CacheWaitCreateEntry [cache.c:260]
       There have been 0 frees since this block was freed from thread  
13:
             free           [rtlib.o]
             TclpFree       [tclAlloc.c:702]
             Tcl_Free       [tclCkalloc.c:1160]
             ns_free        [memory.c:74]
             Ns_CacheDeleteEntry [cache.c:489]
             ExpireEntry    [cache.c:937]
       FMR: Free memory read
       This is occurring while in thread 13:
             Push           [cache.c:907]
             Ns_CacheWaitCreateEntry [cache.c:295]
             CreateEntry    [tclcache.c:686]
             NsTclCacheEvalObjCmd [tclcache.c:158]
             TclEvalObjvInternal [tclBasic.c:3085]
             Tcl_EvalEx     [tclBasic.c:3687]
       Reading 4 bytes from 0x15c2dd8 in the heap.
       Address 0x15c2dd8 is 8 bytes into a freed  block at 0x15c2dd0  
of 32 bytes.
       This block was allocated from thread 13:
             malloc         [rtlib.o]
             TclpAlloc      [tclAlloc.c:679]
             Tcl_Alloc      [tclCkalloc.c:1002]
             ns_malloc      [memory.c:67]
             ns_calloc      [memory.c:85]
             Ns_CacheWaitCreateEntry [cache.c:260]
       There have been 0 frees since this block was freed from thread  
13:
             free           [rtlib.o]
             TclpFree       [tclAlloc.c:702]
             Tcl_Free       [tclCkalloc.c:1160]
             ns_free        [memory.c:74]
             Ns_CacheDeleteEntry [cache.c:489]
             ExpireEntry    [cache.c:937]
       FMW: Free memory write
       This is occurring while in thread 13:
             Push           [cache.c:907]
             Ns_CacheWaitCreateEntry [cache.c:295]
             CreateEntry    [tclcache.c:686]
             NsTclCacheEvalObjCmd [tclcache.c:158]
             TclEvalObjvInternal [tclBasic.c:3085]
             Tcl_EvalEx     [tclBasic.c:3687]
       Writing 4 bytes to 0x15c2dd0 in the heap.
       Address 0x15c2dd0 is at the beginning of a freed block of 32  
bytes.
       This block was allocated from thread 13:
             malloc         [rtlib.o]
             TclpAlloc      [tclAlloc.c:679]
             Tcl_Alloc      [tclCkalloc.c:1002]
             ns_malloc      [memory.c:67]
             ns_calloc      [memory.c:85]
             Ns_CacheWaitCreateEntry [cache.c:260]
       There have been 0 frees since this block was freed from thread  
13:
             free           [rtlib.o]
             TclpFree       [tclAlloc.c:702]
             Tcl_Free       [tclCkalloc.c:1160]
             ns_free        [memory.c:74]
             Ns_CacheDeleteEntry [cache.c:489]
             ExpireEntry    [cache.c:937]
       FMR: Free memory read
       This is occurring while in thread 13:
             Push           [cache.c:908]
             Ns_CacheWaitCreateEntry [cache.c:295]
             CreateEntry    [tclcache.c:686]
             NsTclCacheEvalObjCmd [tclcache.c:158]
             TclEvalObjvInternal [tclBasic.c:3085]
             Tcl_EvalEx     [tclBasic.c:3687]
       Reading 4 bytes from 0x15c2dd8 in the heap.
       Address 0x15c2dd8 is 8 bytes into a freed  block at 0x15c2dd0  
of 32 bytes.
       This block was allocated from thread 13:
             malloc         [rtlib.o]
             TclpAlloc      [tclAlloc.c:679]
             Tcl_Alloc      [tclCkalloc.c:1002]
             ns_malloc      [memory.c:67]
             ns_calloc      [memory.c:85]
             Ns_CacheWaitCreateEntry [cache.c:260]
       There have been 0 frees since this block was freed from thread  
13:
             free           [rtlib.o]
             TclpFree       [tclAlloc.c:702]
             Tcl_Free       [tclCkalloc.c:1160]
             ns_free        [memory.c:74]
             Ns_CacheDeleteEntry [cache.c:489]
             ExpireEntry    [cache.c:937]
       FMR: Free memory read
       This is occurring while in thread 13:
             Push           [cache.c:909]
             Ns_CacheWaitCreateEntry [cache.c:295]
             CreateEntry    [tclcache.c:686]
             NsTclCacheEvalObjCmd [tclcache.c:158]
             TclEvalObjvInternal [tclBasic.c:3085]
             Tcl_EvalEx     [tclBasic.c:3687]
       Reading 4 bytes from 0x15c2dd8 in the heap.
       Address 0x15c2dd8 is 8 bytes into a freed  block at 0x15c2dd0  
of 32 bytes.
       This block was allocated from thread 13:
             malloc         [rtlib.o]
             TclpAlloc      [tclAlloc.c:679]
             Tcl_Alloc      [tclCkalloc.c:1002]
             ns_malloc      [memory.c:67]
             ns_calloc      [memory.c:85]
             Ns_CacheWaitCreateEntry [cache.c:260]
       There have been 0 frees since this block was freed from thread  
13:
             free           [rtlib.o]
             TclpFree       [tclAlloc.c:702]
             Tcl_Free       [tclCkalloc.c:1160]
             ns_free        [memory.c:74]
             Ns_CacheDeleteEntry [cache.c:489]
             ExpireEntry    [cache.c:937]
       FMR: Free memory read
       This is occurring while in thread 13:
             Push           [cache.c:910]
             Ns_CacheWaitCreateEntry [cache.c:295]
             CreateEntry    [tclcache.c:686]
             NsTclCacheEvalObjCmd [tclcache.c:158]
             TclEvalObjvInternal [tclBasic.c:3085]
             Tcl_EvalEx     [tclBasic.c:3687]
       Reading 4 bytes from 0x15c2dd8 in the heap.
       Address 0x15c2dd8 is 8 bytes into a freed  block at 0x15c2dd0  
of 32 bytes.
       This block was allocated from thread 13:
             malloc         [rtlib.o]
             TclpAlloc      [tclAlloc.c:679]
             Tcl_Alloc      [tclCkalloc.c:1002]
             ns_malloc      [memory.c:67]
             ns_calloc      [memory.c:85]
             Ns_CacheWaitCreateEntry [cache.c:260]
       There have been 0 frees since this block was freed from thread  
13:
             free           [rtlib.o]
             TclpFree       [tclAlloc.c:702]
             Tcl_Free       [tclCkalloc.c:1160]
             ns_free        [memory.c:74]
             Ns_CacheDeleteEntry [cache.c:489]
             ExpireEntry    [cache.c:937]
       FMW: Free memory write
       This is occurring while in thread 13:
             Push           [cache.c:904]
             Ns_CacheWaitCreateEntry [cache.c:295]
             Ns_CacheCreateEntry [cache.c:244]
             NsTclCacheEvalObjCmd [tclcache.c:172]
             TclEvalObjvInternal [tclBasic.c:3085]
             Tcl_EvalEx     [tclBasic.c:3687]
       Writing 4 bytes to 0x15c2dd4 in the heap.
       Address 0x15c2dd4 is 4 bytes into a freed  block at 0x15c2dd0  
of 32 bytes.
       This block was allocated from thread 13:
             malloc         [rtlib.o]
             TclpAlloc      [tclAlloc.c:679]
             Tcl_Alloc      [tclCkalloc.c:1002]
             ns_malloc      [memory.c:67]
             ns_calloc      [memory.c:85]
             Ns_CacheWaitCreateEntry [cache.c:260]
       There have been 1 frees since this block was freed from thread  
13:
             free           [rtlib.o]
             TclpFree       [tclAlloc.c:702]
             Tcl_Free       [tclCkalloc.c:1160]
             ns_free        [memory.c:74]
             Ns_CacheDeleteEntry [cache.c:489]
             ExpireEntry    [cache.c:937]
       FMW: Free memory write
       This is occurring while in thread 13:
             Delink         [cache.c:875]
             Ns_CacheDeleteEntry [cache.c:487]
             Ns_CacheFlushEntry [cache.c:517]
             Ns_CacheFlush  [cache.c:548]
             NsTclCacheFlushObjCmd [tclcache.c:587]
             TclEvalObjvInternal [tclBasic.c:3085]
       Writing 4 bytes to 0x15c2dd4 in the heap.
       Address 0x15c2dd4 is 4 bytes into a freed  block at 0x15c2dd0  
of 32 bytes.
       This block was allocated from thread 13:
             malloc         [rtlib.o]
             TclpAlloc      [tclAlloc.c:679]
             Tcl_Alloc      [tclCkalloc.c:1002]
             ns_malloc      [memory.c:67]
             ns_calloc      [memory.c:85]
             Ns_CacheWaitCreateEntry [cache.c:260]
       There have been 23 frees since this block was freed from  
thread 13:
             free           [rtlib.o]
             TclpFree       [tclAlloc.c:702]
             Tcl_Free       [tclCkalloc.c:1160]
             ns_free        [memory.c:74]
             Ns_CacheDeleteEntry [cache.c:489]
             ExpireEntry    [cache.c:937]

Re: [naviserver-devel] Tcl cache crashes

[Zoran V. <zv...@ar...>]

On 14.04.2006, at 11:25, Zoran Vasiljevic wrote:

> Lets see if this reveals anything...
>

Hm.... This code is pretty broken:

       FMR: Free memory read
       FMW: Free memory write
       FMR: Free memory read
       FMW: Free memory write
       FMR: Free memory read
       FMR: Free memory read
       FMR: Free memory read
       FMW: Free memory write
       FMW: Free memory write (4 times)
       FMW: Free memory write (3 times)
       ABW: Array bounds write (4 times)
       ABW: Array bounds write
       ABW: Array bounds write
       FMW: Free memory write (2 times)
       FMW: Free memory write
       ABW: Array bounds write (2 times)
       ABW: Array bounds write
       ABW: Array bounds write (2 times)
       ABW: Array bounds write
       FMW: Free memory write (2 times)
       FMW: Free memory write
       FMW: Free memory write
       FMW: Free memory write
       FMW: Free memory write

We are funking all over the place. Now I have to
isolate the test which triggers the error. This
will be a long work as I have to disable test per
test, reboot the server (takes long time under purify)
and repeat...

Zoran

Re: [naviserver-devel] Tcl cache crashes

[Zoran V. <zv...@ar...>]

On 14.04.2006, at 11:10, Zoran Vasiljevic wrote:
> Unable to get Purify running with the test env, therefore
> I just did the above commands from within the control port
> session:
>
> lexxsrv:nscp 1> ns_cache_create c1 1000
>
> lexxsrv:nscp 2> ns_cache_incr c1 k1
> 1
> lexxsrv:nscp 3> ns_cache_incr c1 k1
> 2
> lexxsrv:nscp 4>  ns_cache_flush c1
> 1
>
> Purify observes no memory violation and all is fine.
> I can repeat the above 1000's of times w/o any side
> effect.
> I'm running against freshly checked-out code
> and with Tcl 8.4.12. Is there any other test
> which would reveal the corruption?

I have seen that tests/ns_crypt.test cores the process.
Although, when I disable ns_cache test, it goes fine.
Presumably, something is really  wrong in the cache code,
I will try to run  (I have to yet figure out how) the
cache tests  from the control port under purify control.
Lets see if this reveals anything...

Cheers
Zoran

Re: [naviserver-devel] Tcl cache crashes

[Zoran V. <zv...@ar...>]

On 14.04.2006, at 00:30, Stephen Deasey wrote:

> Using Tcl 8.4.12.
>
> Create the file tests/x.test:
>
>
> package require tcltest 2.2
> namespace import -force ::tcltest::*
>
> eval ::tcltest::configure $argv
>
> test corruption-1 {cache incr highlights corruption} -setup {
>      ns_cache_create c1 1000
> } -body {
>     ns_cache_incr c1 k1
>     ns_cache_incr c1 k1
> } -cleanup {
>     ns_cache_flush c1
> } -result 2
>
> cleanupTests
>
>
>
> Run with:
>
> make test TCLTESTARGS=3D"-file x.test"
>
>
> You get an error like:
>
> errorInfo: expected integer but got "1aaaaaaaaaaaaZa=A5aaa=069"
>
> Hopefully Purify spots something.  Thanks!
>

Unable to get Purify running with the test env, therefore
I just did the above commands from within the control port
session:

lexxsrv:nscp 1> ns_cache_create c1 1000

lexxsrv:nscp 2> ns_cache_incr c1 k1
1
lexxsrv:nscp 3> ns_cache_incr c1 k1
2
lexxsrv:nscp 4>  ns_cache_flush c1
1

Purify observes no memory violation and all is fine.
I can repeat the above 1000's of times w/o any side
effect.
I'm running against freshly checked-out code
and with Tcl 8.4.12. Is there any other test
which would reveal the corruption?

Cheers
Zoran

Re: [naviserver-devel] Tcl cache crashes

[Stephen D. <sd...@gm...>]

Maybe I'm just confusing myself, but I thought that with a different
configuration, THREAD_ALLOC enabled iirc, the error message was
showing that it was reading one byte *before* the buffer, no just
running off the end because of no NIL byte.  Could be wrong though.=20
I'll take another look at the cache code.

ns_adp_compress-1.2 is failing with one byte different on my new
Fedore Core 5 machine as well...



On 4/13/06, Vlad Seryakov <vl...@cr...> wrote:
> But still make test crashes in
>
> #0  Push (ePtr=3D0x87ff938) at cache.c:904
> 904             ePtr->cachePtr->firstEntryPtr->prevPtr =3D ePtr;
>
> Stephen Deasey wrote:
> > Using Tcl 8.4.12.
> >
> > Create the file tests/x.test:
> >
> >
> > package require tcltest 2.2
> > namespace import -force ::tcltest::*
> >
> > eval ::tcltest::configure $argv
> >
> > test corruption-1 {cache incr highlights corruption} -setup {
> >      ns_cache_create c1 1000
> > } -body {
> >     ns_cache_incr c1 k1
> >     ns_cache_incr c1 k1
> > } -cleanup {
> >     ns_cache_flush c1
> > } -result 2
> >
> > cleanupTests
> >
> >
> >
> > Run with:
> >
> > make test TCLTESTARGS=3D"-file x.test"
> >
> >
> > You get an error like:
> >
> > errorInfo: expected integer but got "1aaaaaaaaaaaaZa=A5aaa=069"
> >
> > Hopefully Purify spots something.  Thanks!
> >
> >
> >
> > On 4/13/06, Zoran Vasiljevic <zv...@ar...> wrote:
> >> On 13.04.2006, at 23:58, Stephen Deasey wrote:
> >>
> >>> Unless anyone has any bright ideas, I'm going to test snapshots from
> >>> CVS until the problem goes away, when I get time...
> >>>
> >> I have Purify. Can you get me a sequence of commands
> >> which expose memory corrruption so I can replay it
> >> under purify control?
> >>
> >> Cheers
> >> Zoran
> >>
> >>
> >> -------------------------------------------------------
> >> This SF.Net email is sponsored by xPML, a groundbreaking scripting lan=
guage
> >> that extends applications into web and mobile media. Attend the live w=
ebcast
> >> and join the prime developer group breaking into this new coding terri=
tory!
> >> http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D110944&bid=3D241720&da=
t=3D121642
> >> _______________________________________________
> >> naviserver-devel mailing list
> >> nav...@li...
> >> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> >>
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by xPML, a groundbreaking scripting lang=
uage
> > that extends applications into web and mobile media. Attend the live we=
bcast
> > and join the prime developer group breaking into this new coding territ=
ory!
> > http://sel.as-us.falkag.net/sel?cmd=3Dk&kid=110944&bid$1720&dat=121642
> > _______________________________________________
> > naviserver-devel mailing list
> > nav...@li...
> > https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> >
>
> --
> Vlad Seryakov
> 571 262-8608 office
> vl...@cr...
> http://www.crystalballinc.com/vlad/
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting langua=
ge
> that extends applications into web and mobile media. Attend the live webc=
ast
> and join the prime developer group breaking into this new coding territor=
y!
> http://sel.as-us.falkag.net/sel?cmdlnk&kid=110944&bid$1720&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] Tcl cache crashes

[Vlad S. <vl...@cr...>]

But still make test crashes in

#0  Push (ePtr=3D0x87ff938) at cache.c:904
904             ePtr->cachePtr->firstEntryPtr->prevPtr =3D ePtr;

Stephen Deasey wrote:
> Using Tcl 8.4.12.
>=20
> Create the file tests/x.test:
>=20
>=20
> package require tcltest 2.2
> namespace import -force ::tcltest::*
>=20
> eval ::tcltest::configure $argv
>=20
> test corruption-1 {cache incr highlights corruption} -setup {
>      ns_cache_create c1 1000
> } -body {
>     ns_cache_incr c1 k1
>     ns_cache_incr c1 k1
> } -cleanup {
>     ns_cache_flush c1
> } -result 2
>=20
> cleanupTests
>=20
>=20
>=20
> Run with:
>=20
> make test TCLTESTARGS=3D"-file x.test"
>=20
>=20
> You get an error like:
>=20
> errorInfo: expected integer but got "1aaaaaaaaaaaaZa=A5aaa=069"
>=20
> Hopefully Purify spots something.  Thanks!
>=20
>=20
>=20
> On 4/13/06, Zoran Vasiljevic <zv...@ar...> wrote:
>> On 13.04.2006, at 23:58, Stephen Deasey wrote:
>>
>>> Unless anyone has any bright ideas, I'm going to test snapshots from
>>> CVS until the problem goes away, when I get time...
>>>
>> I have Purify. Can you get me a sequence of commands
>> which expose memory corrruption so I can replay it
>> under purify control?
>>
>> Cheers
>> Zoran
>>
>>
>> -------------------------------------------------------
>> This SF.Net email is sponsored by xPML, a groundbreaking scripting lan=
guage
>> that extends applications into web and mobile media. Attend the live w=
ebcast
>> and join the prime developer group breaking into this new coding terri=
tory!
>> http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D110944&bid=3D241720&da=
t=3D121642
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>>
>=20
>=20
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting lang=
uage
> that extends applications into web and mobile media. Attend the live we=
bcast
> and join the prime developer group breaking into this new coding territ=
ory!
> http://sel.as-us.falkag.net/sel?cmd=3Dk&kid=110944&bid$1720&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>=20

--=20
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Tcl cache crashes

[Vlad S. <vl...@cr...>]

My fault, commiting the fix

Stephen Deasey wrote:
> Using Tcl 8.4.12.
>=20
> Create the file tests/x.test:
>=20
>=20
> package require tcltest 2.2
> namespace import -force ::tcltest::*
>=20
> eval ::tcltest::configure $argv
>=20
> test corruption-1 {cache incr highlights corruption} -setup {
>      ns_cache_create c1 1000
> } -body {
>     ns_cache_incr c1 k1
>     ns_cache_incr c1 k1
> } -cleanup {
>     ns_cache_flush c1
> } -result 2
>=20
> cleanupTests
>=20
>=20
>=20
> Run with:
>=20
> make test TCLTESTARGS=3D"-file x.test"
>=20
>=20
> You get an error like:
>=20
> errorInfo: expected integer but got "1aaaaaaaaaaaaZa=A5aaa=069"
>=20
> Hopefully Purify spots something.  Thanks!
>=20
>=20
>=20
> On 4/13/06, Zoran Vasiljevic <zv...@ar...> wrote:
>> On 13.04.2006, at 23:58, Stephen Deasey wrote:
>>
>>> Unless anyone has any bright ideas, I'm going to test snapshots from
>>> CVS until the problem goes away, when I get time...
>>>
>> I have Purify. Can you get me a sequence of commands
>> which expose memory corrruption so I can replay it
>> under purify control?
>>
>> Cheers
>> Zoran
>>
>>
>> -------------------------------------------------------
>> This SF.Net email is sponsored by xPML, a groundbreaking scripting lan=
guage
>> that extends applications into web and mobile media. Attend the live w=
ebcast
>> and join the prime developer group breaking into this new coding terri=
tory!
>> http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D110944&bid=3D241720&da=
t=3D121642
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>>
>=20
>=20
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting lang=
uage
> that extends applications into web and mobile media. Attend the live we=
bcast
> and join the prime developer group breaking into this new coding territ=
ory!
> http://sel.as-us.falkag.net/sel?cmd=3Dk&kid=110944&bid$1720&dat=121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>=20

--=20
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Tcl cache crashes

[Stephen D. <sd...@gm...>]

Using Tcl 8.4.12.

Create the file tests/x.test:


package require tcltest 2.2
namespace import -force ::tcltest::*

eval ::tcltest::configure $argv

test corruption-1 {cache incr highlights corruption} -setup {
     ns_cache_create c1 1000
} -body {
    ns_cache_incr c1 k1
    ns_cache_incr c1 k1
} -cleanup {
    ns_cache_flush c1
} -result 2

cleanupTests



Run with:

make test TCLTESTARGS=3D"-file x.test"


You get an error like:

errorInfo: expected integer but got "1aaaaaaaaaaaaZa=A5aaa=069"

Hopefully Purify spots something.  Thanks!



On 4/13/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> On 13.04.2006, at 23:58, Stephen Deasey wrote:
>
> > Unless anyone has any bright ideas, I'm going to test snapshots from
> > CVS until the problem goes away, when I get time...
> >
>
> I have Purify. Can you get me a sequence of commands
> which expose memory corrruption so I can replay it
> under purify control?
>
> Cheers
> Zoran
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting langua=
ge
> that extends applications into web and mobile media. Attend the live webc=
ast
> and join the prime developer group breaking into this new coding territor=
y!
> http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D110944&bid=3D241720&dat=
=3D121642
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>