naviserver-devel — Developer discussion

Re: [naviserver-devel] 2006-06-24 -> FCGI

[Bernd E. <eid...@we...>]

> Yes, that's why i thought about developing fcgi module for ns, just to
> make an option for Rails and other frameworks to run on naviserver. It
> could work like for nginx, but could not, still an option is an option.

True. These frameworks offer so much so nicely out of the box one would never 
want to re-invent in ADP/Tcl.

And on the other hand they need a real webserver doing all the return huge 
files, return lots of cached small files etc. stuff.

Bernd.

Re: [naviserver-devel] 2006-06-24 -> FCGI

[Vlad S. <vl...@cr...>]

Yes, that's why i thought about developing fcgi module for ns, just to 
make an option for Rails and other frameworks to run on naviserver. It 
could work like for nginx, but could not, still an option is an option.

Bernd Eidenschink wrote:
>> BTW... is FCGI still that "alive" after all those years?
>> Do people really use it?
> 
> The usual setup for the Django framework is using mod_python, loading the 
> Python interpreter into each Apache process (forking, not the threading 
> extension). They recommend that.
> 
> The alternative solution is FastCGI, which might reduce memory overhead and 
> allows to run the framework under a different user. You could imagine an 
> Apache setup with many virtual servers talking to separate frameworks running 
> with their specific user accounts. And it solves the issue of the usual 
> life-time-limited apache process.
> 
> But anyway, I'm only interested if there is a possibility at all to deploy
> Django <-> Naviserver
> 
> Only out of interest, I have no real need for a solution at the moment.
> 
> Bernd.
> 
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

Re: [naviserver-devel] 2006-06-24 -> FCGI

[Bernd E. <eid...@we...>]

> BTW... is FCGI still that "alive" after all those years?
> Do people really use it?

The usual setup for the Django framework is using mod_python, loading the 
Python interpreter into each Apache process (forking, not the threading 
extension). They recommend that.

The alternative solution is FastCGI, which might reduce memory overhead and 
allows to run the framework under a different user. You could imagine an 
Apache setup with many virtual servers talking to separate frameworks running 
with their specific user accounts. And it solves the issue of the usual 
life-time-limited apache process.

But anyway, I'm only interested if there is a possibility at all to deploy
Django <-> Naviserver

Only out of interest, I have no real need for a solution at the moment.

Bernd.

Re: [naviserver-devel] 2006-06-24 -> FCGI

[Vasiljevic Z. <zv...@ar...>]

On 09.09.2008, at 09:54, Bernd Eidenschink wrote:

> Is there any FCGI-attempt for AOLserver / Naviserver that may be  
> usable? Or
> how much work would it be?

I *had* extensive FCGI experience back in 1990's ...
This is now some 12 years old... (at that time we had
written our own app server based on FCGI and Apache
as frontend webserver...). Unfortunaltely (or rather,
fortunately) I discovered how Navi/AOL/<younameit>
server can ease my life so I stopped working with
FCGI completely.

So if your <whatever> templating system uses FCGI
permanent proces(es) you would need to write an FCGI
interface (I assume only responder role) to AOL/Naviserver.
I do not know if anybody written that already but I cannot
imagine the need is widespread (can be wrong though)...

One can use the CGI module as a starting point but will
definitely need to improve on that significantly to fit in
the FCGI server-side library.
I cannot take this work on my shoulders as it really does not
bring anything significantly new for us. I could however help
with some advices if need be. Also, I do not think this task
is so complex if you are familiar with C and you have some
web-protocols experience.

BTW... is FCGI still that "alive" after all those years?
Do people really use it?

Cheers
Zoran

Re: [naviserver-devel] 2006-06-24 -> FCGI

[Vlad S. <vl...@cr...>]

At some point i thought about writing fcgi module but then decided not 
to do it because i actually did not have any project to play with.

Bernd Eidenschink wrote:
> Hi!
> 
> I found this in the archives (Zoran):
> 
>> Great! I've been also thinking about adding FCGI by rewriting
>> the nsproxy module (as Stephen suggested). As we do not really
>> need it, it would be just a gift to anybody who might have usage
>> for such thing.
> 
> Is there any FCGI-attempt for AOLserver / Naviserver that may be usable? Or 
> how much work would it be?
> 
> I just came to think about it when looking at Django-Deployment concepts 
> (www.djangoproject.com). 
> 
> Bernd.
> 
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

[naviserver-devel] 2006-06-24 -> FCGI

[Bernd E. <eid...@we...>]

Hi!

I found this in the archives (Zoran):

>Great! I've been also thinking about adding FCGI by rewriting
>the nsproxy module (as Stephen suggested). As we do not really
>need it, it would be just a gift to anybody who might have usage
>for such thing.

Is there any FCGI-attempt for AOLserver / Naviserver that may be usable? Or 
how much work would it be?

I just came to think about it when looking at Django-Deployment concepts 
(www.djangoproject.com). 

Bernd.

Re: [naviserver-devel] naviserver ChangeLog,1.787,1.788

[Stephen D. <sd...@gm...>]

On Fri, Aug 29, 2008 at 2:12 AM, Vlad Seryakov <vl...@cr...> wrote:
>>> +    * tests/http_byteranges.test: Add a couple of tests for ranges on
>>> +    larger files to exercise the file descriptor path rather than the
>>> +    in memory mmap or cache path. Unfortunately, this actually
>>> +    triggers the writer thread and this has no support of ranges,
>>> +    which I guess is where it's actually most needed. TODO...
>>> +
>>
>> I really like last changes, at the same time i need writer support
>> because my tests showed that serving big files with conn threads are
>> waste of resources.
>> At the same time, being able to support range in the writer make sense.
>>
>> And sent email without finishing the thought :-))
>
> Maybe parsing for range should done early enough and saved in the
> request, so whoever then will be sending data will use it.
>
> Just a thought from east coast :-))


I was thinking of adding support for sendfile().

It's different on every platform, so having looked at a few, I was
going to make our interface follow the Solaris senfilev() model which
basically allows an arbitrary number of interleaved memory and file
ranges.  It's an almost exact fit for our byterange code  :-)

Here's the man page:

    http://docs.sun.com/app/docs/doc/816-5172/sendfilev-3ext?a=view

So we'd end up with an array of those sendfilev_t-like structures
describing the data to send. Not sure if they should go in the
conn/request, or...   But maybe it should be a public interface,
unlike the Range struct which is kind of a private thing atm.

Re: [naviserver-devel] naviserver ChangeLog,1.787,1.788

[Vlad S. <vl...@cr...>]

And sent email without finishing the thought :-))

Maybe parsing for range should done early enough and saved in the 
request, so whoever then will be sending data will use it.

Just a thought from east coast :-))

Vlad Seryakov wrote:
>> + 	* tests/http_byteranges.test: Add a couple of tests for ranges on
>> + 	larger files to exercise the file descriptor path rather than the
>> + 	in memory mmap or cache path. Unfortunately, this actually
>> + 	triggers the writer thread and this has no support of ranges,
>> + 	which I guess is where it's actually most needed. TODO...
>> + 
> 
> I really like last changes, at the same time i need writer support 
> because my tests showed that serving big files with conn threads are 
> waste of resources.
> At the same time, being able to support range in the writer make sense.
> 

-- 
Vlad Seryakov
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] naviserver ChangeLog,1.787,1.788

[Vlad S. <vl...@cr...>]

> + 	* tests/http_byteranges.test: Add a couple of tests for ranges on
> + 	larger files to exercise the file descriptor path rather than the
> + 	in memory mmap or cache path. Unfortunately, this actually
> + 	triggers the writer thread and this has no support of ranges,
> + 	which I guess is where it's actually most needed. TODO...
> + 

I really like last changes, at the same time i need writer support 
because my tests showed that serving big files with conn threads are 
waste of resources.
At the same time, being able to support range in the writer make sense.

-- 
Vlad Seryakov
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Log rolling error

[Vlad S. <vl...@cr...>]

Seems to be working on my side, regardless of path.

Are there any other errors in the log?

Daniel Stasinski wrote:
> I've noticed that when I install to the non-default directory, log
> rolling seems to fail.  Before I start digging, anyone else have any
> simple fix?
> 
> Daniel
> 
>   [26/Aug/2008:08:00:00][7928.b7549b90][-sched-] Error: nslog: failed:
> roll '/home/nsadmin/naviserver/servers/myserver/modules/nslog/access.log':
> 'No such file or directory'
> 
>   ns_section     "ns/server/myserver/module/nslog"
>   ns_param        file
> /home/nsadmin/naviserver/servers/myserver/modules/nslog/access.log
>   ns_param        rolllog             true
>   ns_param        rollonsignal        false
>   ns_param        rollhour            8
>   ns_param        maxbackup           45
>   ns_param        logcombined         true
>   ns_param        rollfmt             %Y-%m-%d
> 
> 

[naviserver-devel] Log rolling error

[Daniel S. <moo...@av...>]

I've noticed that when I install to the non-default directory, log
rolling seems to fail.  Before I start digging, anyone else have any
simple fix?

Daniel

  [26/Aug/2008:08:00:00][7928.b7549b90][-sched-] Error: nslog: failed:
roll '/home/nsadmin/naviserver/servers/myserver/modules/nslog/access.log':
'No such file or directory'

  ns_section     "ns/server/myserver/module/nslog"
  ns_param        file
/home/nsadmin/naviserver/servers/myserver/modules/nslog/access.log
  ns_param        rolllog             true
  ns_param        rollonsignal        false
  ns_param        rollhour            8
  ns_param        maxbackup           45
  ns_param        logcombined         true
  ns_param        rollfmt             %Y-%m-%d


-- 
| ---------------------------------------------------------------
| Daniel P. Stasinski | http://www.saidsimple.com
| moo...@av... | http://www.disabilities-r-us.com
| XMMP: moo...@av... | http://www.avenues.org
| Google Talk: mooooooo | http://www.scriptkitties.com

Re: [naviserver-devel] Bug in fastpath cache causes info leak

[Stephen D. <sd...@gm...>]

On Thu, Aug 28, 2008 at 9:52 PM, Vasiljevic Zoran <zv...@ar...> wrote:
>
> On 21.08.2008, at 21:06, Stephen Deasey wrote:
>
>> You used to have to manually edit the makefile to disable zippy. Is
>> this still the case?
>>
>> Perhaps the Tcl default should be to use the system allocator even on
>> threaded builds, and to provide a configure switch to enable zippy.
>> Zoran, you have Tcl cvs access right?
>
> Yes. I just discussed this with Vlad.
> Here is the consensus:
>
> There is Tcl single-threaded allocator and zippy allocator.
>
> If you turn on Tcl allocator it should use
>  single-threaded allocator for non-thread builds
>  zippy allocator for thread builds
>
> If you turn off Tcl allocator it should use
>  system-level malloc regardless of the thread/nonthread builds
>
> This is most "logical" setup. So, either you have custom Tcl
> memory allocator or not. If you dont, then system malloc is used.
> If you do, then it depends on the threaded build.
>
> To aid this, there is (confugure-wise unused) USE_TCLALLOC define.
> Control which (simple one-lock Tcl allocator or AOL's zippy allocator)
> is selected, is done over USE_THREAD_ALLOC.
>
> By allowing the USE_TCLALLOC  to be confugured "from the outside"
> one can decide wether TCL allocator (singlethreaded or zippy)
> or system/OS malloc us used.
>
> The USE_THREAD_ALLOC is implicitly set (reset) when threaded build
> is configured (or not).
>
> This way we would have one "knob" (the USE_TCLALLOC) which we can
> toggle over --enable-tclalloc (default us true). By --disable-tclalloc
> we would default to system level alloc, thread build or not.
>
> This is the path of least resistence and would not change any defaults.
> I could go to the Tcl core list with this suggestion, yes.
>
> Vlad suggested to raise the queston of disabling the zippy even for
> thread builds, per default. I can only second that. But then again,
> we would change the default behaviour from as-is now and that is a
> chance to get some oposition which I do not have time to deal with.
>
> What do you think?


Sounds sensible. Configuration knob first, change defaults later, possibly.

To change the default is going to require proof that one way is better
than another anyway. There's a Tcl performance test suite somewhere,
right?  Some work for somebody...

Re: [naviserver-devel] Bug in fastpath cache causes info leak

[Vasiljevic Z. <zv...@ar...>]

On 21.08.2008, at 21:06, Stephen Deasey wrote:

> You used to have to manually edit the makefile to disable zippy. Is
> this still the case?
>
> Perhaps the Tcl default should be to use the system allocator even on
> threaded builds, and to provide a configure switch to enable zippy.
> Zoran, you have Tcl cvs access right?

Yes. I just discussed this with Vlad.
Here is the consensus:

There is Tcl single-threaded allocator and zippy allocator.

If you turn on Tcl allocator it should use
  single-threaded allocator for non-thread builds
  zippy allocator for thread builds

If you turn off Tcl allocator it should use
  system-level malloc regardless of the thread/nonthread builds

This is most "logical" setup. So, either you have custom Tcl
memory allocator or not. If you dont, then system malloc is used.
If you do, then it depends on the threaded build.

To aid this, there is (confugure-wise unused) USE_TCLALLOC define.
Control which (simple one-lock Tcl allocator or AOL's zippy allocator)
is selected, is done over USE_THREAD_ALLOC.

By allowing the USE_TCLALLOC  to be confugured "from the outside"
one can decide wether TCL allocator (singlethreaded or zippy)
or system/OS malloc us used.

The USE_THREAD_ALLOC is implicitly set (reset) when threaded build
is configured (or not).

This way we would have one "knob" (the USE_TCLALLOC) which we can
toggle over --enable-tclalloc (default us true). By --disable-tclalloc
we would default to system level alloc, thread build or not.

This is the path of least resistence and would not change any defaults.
I could go to the Tcl core list with this suggestion, yes.

Vlad suggested to raise the queston of disabling the zippy even for
thread builds, per default. I can only second that. But then again,
we would change the default behaviour from as-is now and that is a
chance to get some oposition which I do not have time to deal with.

What do you think?

Re: [naviserver-devel] Last build, cache test failure

[Stephen D. <sd...@gm...>]

On Tue, Aug 26, 2008 at 11:36 PM, Vlad Seryakov <vl...@cr...> wrote:
> I spent a lot of time and still cannot resolve the problem with latest
> version ns_cache test failure.
>
> One problem i noticed with latest Tcl 8.5.4, in crashes nsd in thread
> test in 64bit and hangs forever in 32bit Linux.
>
> But ns_caches fails with tests using -expires and i rolled back the
> changed i made recently and it still fails. I cannot see why driver
> change would make it fail, but other sources are the same.
>
> Help please


When doing development, as opposed to building for install, you MUST
run autogen.sh with --prefix=/tmp/ns or some other path that is
unlikely to have a copy of the naviserver libraries.

Otherwise, you will drive yourself insane...  :-)

Re: [naviserver-devel] ns_time Tcl type problem

[Vlad S. <vl...@cr...>]

Sorry, found my own bug in tcltime.c

Vlad Seryakov wrote:
> It looks like the problem with Tcl ns_time type:
> 
> ossweb:nscp 78> ns_time
> 1219798300
> ossweb:nscp 79> ns_time
> 1219798300
> ossweb:nscp 80> ns_time get
> 164905896
> ossweb:nscp 81> ns_time get
> 164905848
> 

-- 
Vlad Seryakov
vl...@cr...
http://www.crystalballinc.com/vlad/

[naviserver-devel] ns_time Tcl type problem

[Vlad S. <vl...@cr...>]

It looks like the problem with Tcl ns_time type:

ossweb:nscp 78> ns_time
1219798300
ossweb:nscp 79> ns_time
1219798300
ossweb:nscp 80> ns_time get
164905896
ossweb:nscp 81> ns_time get
164905848

-- 
Vlad Seryakov
vl...@cr...
http://www.crystalballinc.com/vlad/

[naviserver-devel] Last build, cache test failure

[Vlad S. <vl...@cr...>]

I spent a lot of time and still cannot resolve the problem with latest 
version ns_cache test failure.

One problem i noticed with latest Tcl 8.5.4, in crashes nsd in thread 
test in 64bit and hangs forever in 32bit Linux.

But ns_caches fails with tests using -expires and i rolled back the 
changed i made recently and it still fails. I cannot see why driver 
change would make it fail, but other sources are the same.

Help please

Re: [naviserver-devel] Bug in fastpath cache causes info leak

[Bernd E. <eid...@we...>]

Thanks!

> ./configure --prefix=/usr --enable-threads --disable-64bit
>
> sed -i -e 's/-DUSE_THREAD_ALLOC=1//g' tclConfig.sh
> sed -i -e 's/-DUSE_THREAD_ALLOC=1//g' Makefile

Re: [naviserver-devel] Bug in fastpath cache causes info leak

[Vlad S. <vl...@cr...>]

I mainly use Archlinux so i created package file but basically i added 
just 2 lines to it and overall it looks like it:

./configure --prefix=/usr --enable-threads --disable-64bit

sed -i -e 's/-DUSE_THREAD_ALLOC=1//g' tclConfig.sh
sed -i -e 's/-DUSE_THREAD_ALLOC=1//g' Makefile

make install


Bernd Eidenschink wrote:
>>> You used to have to manually edit the makefile to disable zippy. Is
>>> this still the case?
>> What i do i use sed to replace zippy define after i ran configure, there
>> is no easy way to disable it currently
> 
> Hi Vlad!
> 
> Can you share the way you do it? Thanks!
> 
> Bernd.
> 
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

Re: [naviserver-devel] Bug in fastpath cache causes info leak

[Bernd E. <eid...@we...>]

> > You used to have to manually edit the makefile to disable zippy. Is
> > this still the case?
>
> What i do i use sed to replace zippy define after i ran configure, there
> is no easy way to disable it currently

Hi Vlad!

Can you share the way you do it? Thanks!

Bernd.

Re: [naviserver-devel] Bug in fastpath cache causes info leak

[Vlad S. <vl...@cr...>]

> 
> You used to have to manually edit the makefile to disable zippy. Is
> this still the case?
> 

What i do i use sed to replace zippy define after i ran configure, there 
is no easy way to disable it currently

Re: [naviserver-devel] Bug in fastpath cache causes info leak

[Stephen D. <sd...@gm...>]

On Thu, Aug 21, 2008 at 7:49 PM, Vasiljevic Zoran <zv...@ar...> wrote:
>
> On 21.08.2008, at 18:26, Vlad Seryakov wrote:
>
>> I always recompile Tcl with default memory
>> allocator: it has 2 benefits, 1) not-ever-growing, 2) i can use
>> vtmalloc
>> easily for example if default one does not work like i need
>
> yep. me too.
>


You used to have to manually edit the makefile to disable zippy. Is
this still the case?

Perhaps the Tcl default should be to use the system allocator even on
threaded builds, and to provide a configure switch to enable zippy.
Zoran, you have Tcl cvs access right?

Re: [naviserver-devel] Bug in fastpath cache causes info leak

[Vasiljevic Z. <zv...@ar...>]

On 21.08.2008, at 18:26, Vlad Seryakov wrote:

> I always recompile Tcl with default memory
> allocator: it has 2 benefits, 1) not-ever-growing, 2) i can use  
> vtmalloc
> easily for example if default one does not work like i need

yep. me too.

Re: [naviserver-devel] Bug in fastpath cache causes info leak

[Vlad S. <vl...@cr...>]

I think many issues or questions also are result of defaults in 
Aolserver and Naviserver still being set according to past AOL setup and 
requirements. It can be considered as an extreme case and because a lot 
of functionality was implemented for AOL high-performance requirements 
and setup, everybody else has to know that deeply, change config and/or 
know C code good enough to make knowledgeable decisions.

For example, one of such things is threaded allocator which is default 
in Tcl, nowadays default allocators a good enough, return memory to the 
system, zippy is needed only in very high performance cases and even in 
such cases it needs carefull testing to make sure the application does 
not consume more and more different pieces of memory which will lead to 
even wore fragmentation. I always recompile Tcl with default memory 
allocator: it has 2 benefits, 1) not-ever-growing, 2) i can use vtmalloc 
easily for example if default one does not work like i need

We know the code and we assume everybody else should and many questions 
and problems reported seem like dumb or idiotic but they are not in all 
cases.

I think all functionality AOL developed is very valuable but enabling it 
by default is making everybody else to follow AOL's way of doing things 
with AS/NS.

Jeff Rogers wrote:
> Brett Schwarz wrote:
>> Wow, I am amazed at the different reactions to this issue between
>> this list and the aolserver list. Makes me want to switch to
>> naviserver even more now...
> 
> 90% of the difference is that the issue was introduced here by someone
> saying "I contributed a patch."  That in itself gets a better reaction 
> than hyperbole and insisting that everyone else doesn't know what 
> they're talking about.  On the flipside, some of the reactions were 
> overly harsh and dismissive.
> 
> BTW, the solution in the patch of adding the filename to the hash key
> was rejected by the OP.
> 
> After all the discussion there, I am generally in agreement that there 
> is no real problem other than insufficient documentation, caveats, and 
> examples.  But some options to make the cache less aggressive when 
> necessary can be a good thing.  Other than adding in the filename (which 
> causes more memory to be used) a way to avoid the caching in the first 
> place could be good; either a -nocache flag to ns_returnfile or a 
> different command (to still get the performance of mmap-ing) or my other 
> suggestion of skipping caching for files matching certain patterns 
> (e.g., under a particular directory like /tmp, /var/tmp, or whatever) 
> would avoid uselessly using up the cache memory in the first place.
> 
> As far as the performance impact (of mmap or fastpath in the first 
> place) numbers speak louder than speculation, and testing it isn't that 
> hard.
> 
> 
> -J
> 
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

Re: [naviserver-devel] CVS, just keep it?

[Vasiljevic Z. <zv...@ar...>]

On 20.08.2008, at 00:19, Stephen Deasey wrote:

> I think it's probably better to stay with the familiar cvs. It doesn't
> scare anyone, SF handles everything, and patches are probably too
> infrequent to for the vcs to make much difference anyway.
>
> Agree?

Yes. Lets keep it simple for now.