naviserver-devel — Developer discussion

Re: [naviserver-devel] nsconfigrw

[Vasiljevic Z. <zv...@ar...>]

On 24.10.2007, at 21:31, Stephen Deasey wrote:

> Yes.

It is (now in CVS). We now only have knownBugs
left, but we will never bee case-insensitive
I guess.

Now: where you want to go today?

Couple of issues still left:

  persistence
  wideint type
  parameter introspection
  (you name it)

Cheers
Zoran

Re: [naviserver-devel] nsconfigrw

[Stephen D. <sd...@gm...>]

On 10/24/07, Vasiljevic Zoran <zv...@ar...> wrote:
>
> OK. I buy everything.
>
> So, supported conversion rules:
>
>     Unknown -> any
>     Int -> Unknown
>     Bool-> Unknown


    Unknown -> Int || Bool

Valid for the value stored in the config itself.

On the way out:

    Int || Bool -> no assert when asked for Unknown.

i.e. no conversion is going on, but the checking (assertion) rules are relaxed.

(Just clarifying what's getting converted, where).


> and unsupported:
>
>     Int -> Boolean
>     Boolean -> Int
>
> (more types like floating, wide can be introduced later, if needed).


Yes.

Re: [naviserver-devel] nsconfigrw

[Vasiljevic Z. <zv...@ar...>]

On 24.10.2007, at 20:35, Stephen Deasey wrote:

> On 10/24/07, Vasiljevic Zoran <zv...@ar...> wrote:
>>
>> On 24.10.2007, at 19:09, Vasiljevic Zoran wrote:
>>
>>> So
>>>
>>>    ns_getconfig -bool section param 1
>>>    ns_getconfig -int section param
>>>    configuration parameter is not an integer
>>>
>>> would make you happy?
>>>
>>
>> Lets stop and think about consequences of this...
>>
>> Underneath, some integer will be used to hold
>> the value of the boolean param. After all,we
>> have C underneath. Now somebody will go get
>> integer rep of it and will receive error
>> because the type is boolean. Good for the moment.
>> But consider this:
>>
>>     ns_getconfig section param "1"
>>     ns_getconfig -bool section param
>>
>> How should we react here? The "1" is not a boolean
>> value. It is an string.
>
>
> It should be 'unknown'. i.e. this:
>
>     typedef enum {
>         UnknownType,
>         StrType,
>         IntType
>     } ParamType;
>
> Should be more like this:
>
>     typedef enum {
>         UnknownType,
>         BoolType,
>         IntType
>     } ParamType;
>
>
> The conversion rules would be that you can go from Unknown to Bool or
> Int, but not the other way around, and you can't go from Bool to Int
> or vice versa.
>
> So code like this:
>
>     ns_getconfig section key 42
>
> would set a value of unknown type if not present. Code like this:
>
>     ns_getconfig -int section key 42
>
> would cause it's type to become Int. It's then OK to follow that  
> sequence with:
>
>     ns_getconfig section key 42
>
> again. The underlying C code would actually return a Tcl Integer type,
> but that doesn't matter.
>
> If you don't specify either -int or -bool then you're either setting
> an unknown type or are happy to receive any type. Type conversion and
> checking can still occur, but that's standard Tcl:
>
>     if {[ns_getconfig section key]} { return [expr $key + 2] }
>
> If we were still at step 1 above, they key with type Unknown would be
> returned as a string and Tcl would convert to an int.
>
>
> It *is* kinda weird to build (but easy to use), but that's because
> we're using a command designed for the read-only existing config.
> Explicitly, ns_getconfig with a default param is just shorthand for
> this sort of code:
>
>     if {[catch {
>         set value [ns_getconfig -int section key]
>     } err]} {
>         ns_setconfig -int section key 42
>         set value 42
>     }
>
> The first time the code is called, when there is no record of the key
> or value in the global config, there is an implicit call to set.
>
> We want to record that 'key' is an int because that is useful:
>
>   [-main-] Dev: config section: ns/server/server1
>   [-main-] Dev: config: (null):maxthreads value=(null) min=0 max=100
> default=10 (int)
>
>
> And really we always want this information. So maybe ns_setconfig
> should be more like:
>
>     ns_setstringconfig section key value
>     ns_setboolconfig section key value
>     ns_setintconfig ?-min n? ?-max n? ?--? section key value
>
> And if you do that, then as ns_getconfig is implicitly a set on the
> first call with a default, then maybe ns_getconfig should be more
> like:
>
>     ns_getstringconfig section key ?default?
>     ns_getboolconfig section key ?default?
>     ns_getintconfig ?-min n? ?-max n? ?--? section key ?default?
>
>     ns_getconfig ?section? ?key?
>
> $ ns_getconfig
> ns/foo ns/bar
>
> $ ns_getconfig ns/foo
> a b
>
> $ ns_getconfig ns/foo a
> type int value 42 min 0 max 99
>
>
> Or, considering this common idiom:
>
>     ns_section "ns/foo"
>     ns_param  bar greeble  ;# Oh, Hi!
>
>
>     ns_setstringvalue ?-description d? section key ?default?
>
>
> $ ns_getconfig ns/foo bar
> type string value greeble description "Oh, Hi!"
>
>

OK. I buy everything.

So, supported conversion rules:

    Unknown -> any
    Int -> Unknown
    Bool-> Unknown

and unsupported:

    Int -> Boolean
    Boolean -> Int

(more types like floating, wide can be introduced later, if needed).

Commands ns_getconfig/ns_setconfig ?-type? I would leave as is.
Without the specific type, if the argument is already set,
the conversion is done as any->unknown (AND the parameter
shimmers). If the parameter is not there and default is given
it is recored as Unonown type. In all other cases, type-checking
is introduced.

Do we now have consensus here? I would not like to change the
code every 5 minutes, so lets stick to (any) spec and go from there.
I believe we have something good now that we can build upon.

Re: [naviserver-devel] nsconfigrw

[Stephen D. <sd...@gm...>]

On 10/24/07, Vasiljevic Zoran <zv...@ar...> wrote:
>
> On 24.10.2007, at 19:09, Vasiljevic Zoran wrote:
>
> > So
> >
> >    ns_getconfig -bool section param 1
> >    ns_getconfig -int section param
> >    configuration parameter is not an integer
> >
> > would make you happy?
> >
>
> Lets stop and think about consequences of this...
>
> Underneath, some integer will be used to hold
> the value of the boolean param. After all,we
> have C underneath. Now somebody will go get
> integer rep of it and will receive error
> because the type is boolean. Good for the moment.
> But consider this:
>
>     ns_getconfig section param "1"
>     ns_getconfig -bool section param
>
> How should we react here? The "1" is not a boolean
> value. It is an string.


It should be 'unknown'. i.e. this:

    typedef enum {
        UnknownType,
        StrType,
        IntType
    } ParamType;

Should be more like this:

    typedef enum {
        UnknownType,
        BoolType,
        IntType
    } ParamType;


The conversion rules would be that you can go from Unknown to Bool or
Int, but not the other way around, and you can't go from Bool to Int
or vice versa.

So code like this:

    ns_getconfig section key 42

would set a value of unknown type if not present. Code like this:

    ns_getconfig -int section key 42

would cause it's type to become Int. It's then OK to follow that sequence with:

    ns_getconfig section key 42

again. The underlying C code would actually return a Tcl Integer type,
but that doesn't matter.

If you don't specify either -int or -bool then you're either setting
an unknown type or are happy to receive any type. Type conversion and
checking can still occur, but that's standard Tcl:

    if {[ns_getconfig section key]} { return [expr $key + 2] }

If we were still at step 1 above, they key with type Unknown would be
returned as a string and Tcl would convert to an int.


It *is* kinda weird to build (but easy to use), but that's because
we're using a command designed for the read-only existing config.
Explicitly, ns_getconfig with a default param is just shorthand for
this sort of code:

    if {[catch {
        set value [ns_getconfig -int section key]
    } err]} {
        ns_setconfig -int section key 42
        set value 42
    }

The first time the code is called, when there is no record of the key
or value in the global config, there is an implicit call to set.

We want to record that 'key' is an int because that is useful:

  [-main-] Dev: config section: ns/server/server1
  [-main-] Dev: config: (null):maxthreads value=(null) min=0 max=100
default=10 (int)


And really we always want this information. So maybe ns_setconfig
should be more like:

    ns_setstringconfig section key value
    ns_setboolconfig section key value
    ns_setintconfig ?-min n? ?-max n? ?--? section key value

And if you do that, then as ns_getconfig is implicitly a set on the
first call with a default, then maybe ns_getconfig should be more
like:

    ns_getstringconfig section key ?default?
    ns_getboolconfig section key ?default?
    ns_getintconfig ?-min n? ?-max n? ?--? section key ?default?

    ns_getconfig ?section? ?key?

$ ns_getconfig
ns/foo ns/bar

$ ns_getconfig ns/foo
a b

$ ns_getconfig ns/foo a
type int value 42 min 0 max 99


Or, considering this common idiom:

    ns_section "ns/foo"
    ns_param  bar greeble  ;# Oh, Hi!


    ns_setstringvalue ?-description d? section key ?default?


$ ns_getconfig ns/foo bar
type string value greeble description "Oh, Hi!"



> > A buffer size of 'no' doesn't make any
> > sense.
>
> But
>
>   ns_getconfig section buffer no   (module A)
>   ns_getconfig section buffer 1000 (module B)
>
> is the application problem, not ours.
>
> This is simply clash of config params. I do not
> think we need to take care of that.


The above would not throw an error, they're of Unknown type. It would
throw an error if one specified -bool and the other -int. But that is
an error (of the application).

A single key in a section can't have two types at once. You can treat
it as two different types once you get it into Tcl, but that's just
standard Tcl.

Re: [naviserver-devel] Large file upload dilemma

[Vasiljevic Z. <zv...@ar...>]

On 24.10.2007, at 19:58, Vlad Seryakov wrote:

> Actually i put this for windows as well, only mmap now does not  
> compile
> for window, but i never tested, does not have windows

I have written NsMemMap and NsMemUmap (nsd/nswin.c).
This should work.

Re: [naviserver-devel] Large file upload dilemma

[Vlad S. <vl...@cr...>]

Actually i put this for windows as well, only mmap now does not compile 
for window, but i never tested, does not have windows

Vasiljevic Zoran wrote:
> On 24.10.2007, at 18:13, Vlad Seryakov wrote:
> 
>> ns_conn contentfile will be non empty only in such situation, in  
>> normal
>> cases it is always empty
>>
> 
> So the non-empty [ns_conn contentfile] means we need to parse.
> Otherwise server parses.
> 
> Allright. I will take the freedom to make the changes also
> usable from windows. At the moment all is ifndef win32 and
> this must change.
> 
> 
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

Re: [naviserver-devel] nsconfigrw

[Vasiljevic Z. <zv...@ar...>]

On 24.10.2007, at 19:09, Vasiljevic Zoran wrote:

> So
>
>    ns_getconfig -bool section param 1
>    ns_getconfig -int section param
>    configuration parameter is not an integer
>
> would make you happy?
>

Lets stop and think about consequences of this...

Underneath, some integer will be used to hold
the value of the boolean param. After all,we
have C underneath. Now somebody will go get
integer rep of it and will receive error
because the type is boolean. Good for the moment.
But consider this:

    ns_getconfig section param "1"
    ns_getconfig -bool section param

How should we react here? The "1" is not a boolean
value. It is an string. Should I convert it to boolean?
If the answer is YES, then the top example should
not result in error.
If the answer is NO, then we're going VERY far from
common (Tcl) logic...

  
  

Re: [naviserver-devel] nsconfigrw

[Vasiljevic Z. <zv...@ar...>]

On 24.10.2007, at 18:36, Stephen Deasey wrote:

> ns_getconfig both uses and declares. It declares so that an admin can
> set the correct value. Some code may only need to check whether a
> buffer size is greater than zero (is it true?), but that doesn't make
> it a boolean config option. A buffer size of 'no' doesn't make any
> sense.

So

   ns_getconfig -bool section param 1
   ns_getconfig -int section param
   configuration parameter is not an integer

would make you happy?

Re: [naviserver-devel] nsconfigrw

[Vasiljevic Z. <zv...@ar...>]

On 24.10.2007, at 18:36, Stephen Deasey wrote:

> A buffer size of 'no' doesn't make any
> sense.

But

   ns_getconfig section buffer no   (module A)
   ns_getconfig section buffer 1000 (module B)

is the application problem, not ours.

This is simply clash of config params. I do not
think we need to take care of that. Otherwise we
MUST predefine all upfront and not set implicitly
when somebody provides default value.

Re: [naviserver-devel] Large file upload dilemma

[Vasiljevic Z. <zv...@ar...>]

On 24.10.2007, at 18:13, Vlad Seryakov wrote:

> ns_conn contentfile will be non empty only in such situation, in  
> normal
> cases it is always empty
>

So the non-empty [ns_conn contentfile] means we need to parse.
Otherwise server parses.

Allright. I will take the freedom to make the changes also
usable from windows. At the moment all is ifndef win32 and
this must change.

Re: [naviserver-devel] Large file upload dilemma

[Vlad S. <vl...@cr...>]

ns_conn contentfile will be non empty only in such situation, in normal 
cases it is always empty

Vasiljevic Zoran wrote:
> On 24.10.2007, at 00:34, Vlad Seryakov wrote:
> 
>> Yes, i already tested it and just commited, please review if anybody
>> have time. I am sure it does not break anything and if maxupload
>> parameters is not set will never be activated but it is pretty useful.
>> Now it is possible to upload/download gigantic files without using  
>> conn
>> threads.
> 
> Setting maxupload to 1024*1024*512 will turn on the special processing
> for files larger than 512MB. If I transfer 100MB, the usual
> connthread processing is taking place, right?
> 
> How do I know which situation I'm gonna need to handle in Tcl?
> 
> 
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

Re: [naviserver-devel] nsconfigrw

[Stephen D. <sd...@gm...>]

On 10/24/07, Vasiljevic Zoran <zv...@ar...> wrote:
>
> On 23.10.2007, at 23:52, Vasiljevic Zoran wrote:
>
> > Now that we "solved" that one....
>
> So, it is done. I will shortly check-in the changes.
> But before I do, a question: how "seriously" should
> I take those tests:
>
> test cfg-4.3 {type check: bool != int} -body {
>      ns_getconfig -bool section cfg-4.3 yes
>      ns_getconfig -int section cfg-4.3 42
> } -returnCodes error -result {configuration parameter is not an integer}
>
> test cfg-4.4 {global type check: bool != int} -body {
>      ns_job wait cfgtest [ns_job queue cfgtest {
>          ns_getconfig -bool section cfg-4.4 yes
>      }]
>      ns_getconfig -int section cfg-4.4 42
> } -returnCodes error -result {configuration parameter is not an integer}
>
>
> If this should be true, then something like this should also fail:
>
>    ns_getconfig -bool foo bar 1
>    ns_getconfig -int foo bar
>
> Or not?


Should add that test.


> I mean, we are not Java. Tcl is still C-like:
>
>    set true 42
>    if {$true} {puts yesitis}
>
> Tcl will not jump in your face telling you that
> "true is not a boolean", or?


This works:

  if {[ns_getconfig -int section key 42]} {
      ...
  }

ns_getconfig both uses and declares. It declares so that an admin can
set the correct value. Some code may only need to check whether a
buffer size is greater than zero (is it true?), but that doesn't make
it a boolean config option. A buffer size of 'no' doesn't make any
sense.

Re: [naviserver-devel] nsconfigrw

[Vasiljevic Z. <zv...@ar...>]

On 23.10.2007, at 23:52, Vasiljevic Zoran wrote:

> Now that we "solved" that one....

So, it is done. I will shortly check-in the changes.
But before I do, a question: how "seriously" should
I take those tests:

test cfg-4.3 {type check: bool != int} -body {
     ns_getconfig -bool section cfg-4.3 yes
     ns_getconfig -int section cfg-4.3 42
} -returnCodes error -result {configuration parameter is not an integer}

test cfg-4.4 {global type check: bool != int} -body {
     ns_job wait cfgtest [ns_job queue cfgtest {
         ns_getconfig -bool section cfg-4.4 yes
     }]
     ns_getconfig -int section cfg-4.4 42
} -returnCodes error -result {configuration parameter is not an integer}


If this should be true, then something like this should also fail:

   ns_getconfig -bool foo bar 1
   ns_getconfig -int foo bar

Or not?

I mean, we are not Java. Tcl is still C-like:

   set true 42
   if {$true} {puts yesitis}

Tcl will not jump in your face telling you that
"true is not a boolean", or?

Cheers
Zoran

Re: [naviserver-devel] Large file upload dilemma

[Vasiljevic Z. <zv...@ar...>]

On 24.10.2007, at 00:34, Vlad Seryakov wrote:

> Yes, i already tested it and just commited, please review if anybody
> have time. I am sure it does not break anything and if maxupload
> parameters is not set will never be activated but it is pretty useful.
> Now it is possible to upload/download gigantic files without using  
> conn
> threads.

Setting maxupload to 1024*1024*512 will turn on the special processing
for files larger than 512MB. If I transfer 100MB, the usual
connthread processing is taking place, right?

How do I know which situation I'm gonna need to handle in Tcl?

Re: [naviserver-devel] Large file upload dilemma

[Vlad S. <vl...@cr...>]

> Being able to delegate mpart processing to
> some other "thing" is generally good option.
> I just do not know how you will "communicate"
> that to the connection thread.
> Do you have some plan? 

Yes, i already tested it and just commited, please review if anybody 
have time. I am sure it does not break anything and if maxupload 
parameters is not set will never be activated but it is pretty useful. 
Now it is possible to upload/download gigantic files without using conn 
threads.

Re: [naviserver-devel] nsconfigrw

[Vasiljevic Z. <zv...@ar...>]

On 23.10.2007, at 23:06, Stephen Deasey wrote:
> If you have:
>
>     ns_getconfig section theint 42
>
> and somewhere else you have:
>
>     ns_getconfig section theint 13
>
> and you set it up so that on the first access, if the value does not
> exist in the global config you seed it with the default (as you must
> if you want introspection to work), then the value seeded depends on
> which code runs first.
>
> I don't think this is necessarily an error. By design it's a
> read/write config, so if the value started at 42, maybe someone reset
> it to 13?  It's the same effect.

Not exactly as the second on will ignore the
default, because the first one have set it allright.
But I agree that this may not be considered as
error. The deault is only used to seed the config
if nothing found.
As it is now. it doesn't. It simply parrots the
default back to user w/o setting it in the store.
This is how ns_config works. Not necesarily the
correct way, though.

>
> However, I think this would be an error:
>
>     ns_getconfig -bool section key yes
>     ...
>     ns_getconfig -int section key 1
>
> The two pieces of code share a config key but they have conflicting
> ideas about the type. One of them is wrong.

But we can't recover from that.

>
> The alternative is to always store the string value and do conversion
> on every access. I don't think this is the way to go: it's slow and
> it's just not logically correct.

Storing the string is not going to be very sexy because of
permanent string/type conversions.

I can live with "first default sets the value" in the
config. This is how it was yesterday and I changed it
today to be on-pair with ns_config. Not necessarily a
good idea. I will revert the code again.

Now that we "solved" that one.... do you have any
idea about persistence?

Re: [naviserver-devel] nsconfigrw

[Stephen D. <sd...@gm...>]

On 10/23/07, Vasiljevic Zoran <zv...@ar...> wrote:
>
> On 23.10.2007, at 21:24, Stephen Deasey wrote:
> >
> > But one of the things we want to support is introspection, right?
>
> Right.
>
> >
> >
> >   foreach section [ns_getconfig] {
> >       foreach {k v} [ns_getconfig $section] {
> >           ...
> >       }
> >   }
> >
> >
> > The only way to make it work atm is to pre-declare all config values:
> >
> >
> >   # mylib.tcl
> >
> >   ns_setconfig section enabled true
> >   ...
> >
> >   proc foo {} {
> >       if {[ns_getconfig section enabled]} {
> >           ...
> >       }
> >   }
> >   ...
> >
> >
> > Pre-declaring is a new restriction. Is this the plan?
>
> Well, I went ahead in making it ns_config compatible
> although this may NOT really always be clear.
>
> If you do not set default values, then module A might
> have different defaults then module B, which is right
> and wrong, depending on the position...
> If you however set them in module A, might that have
> side-effects in module B ?


If you have:

    ns_getconfig section theint 42

and somewhere else you have:

    ns_getconfig section theint 13

and you set it up so that on the first access, if the value does not
exist in the global config you seed it with the default (as you must
if you want introspection to work), then the value seeded depends on
which code runs first.

I don't think this is necessarily an error. By design it's a
read/write config, so if the value started at 42, maybe someone reset
it to 13?  It's the same effect.

However, I think this would be an error:

    ns_getconfig -bool section key yes
    ...
    ns_getconfig -int section key 1

The two pieces of code share a config key but they have conflicting
ideas about the type. One of them is wrong.

The alternative is to always store the string value and do conversion
on every access. I don't think this is the way to go: it's slow and
it's just not logically correct.

Re: [naviserver-devel] nsconfigrw

[Vasiljevic Z. <zv...@ar...>]

On 23.10.2007, at 22:28, Stephen Deasey wrote:

>
> int Tcl_GetBoolean(Tcl_Interp *, CONST char *string, int *boolPtr)
>
>   http://www.tcl.tk/man/tcl8.4/TclLib/GetInt.htm

Good tip!

Re: [naviserver-devel] nsconfigrw

[Vasiljevic Z. <zv...@ar...>]

On 23.10.2007, at 22:28, Stephen Deasey wrote:

>
> $ make

Damn. I did not check-in the nsconfigrw.h...

Re: [naviserver-devel] nsconfigrw

[Stephen D. <sd...@gm...>]

On 10/23/07, Vasiljevic Zoran <zv...@ar...> wrote:
>
> On 23.10.2007, at 19:25, Stephen Deasey wrote:
>
> >
> > test cfg-4.1 {global type check bool} -body {
> >     ns_setconfig section cfg-4.1 true
> >     expr {[ns_getconfig -bool section cfg-4.1] ? 1 : 0}
> > } -result 1
>
> Try again...



$ make

gcc -g -Wall -Wno-implicit-int -fPIC  -pipe
-I/home/sd/local/ns-HEADg/include
-I"/home/sd/local/tcl8.4.15g/include"  -DHAVE_CONFIG_H   -c -o
configrw.o configrw.c

configrw.c:219: error: conflicting types for 'Cfg_GetBool'
nsconfigrw.h:58: error: previous declaration of 'Cfg_GetBool' was here
configrw.c:265: error: conflicting types for 'Cfg_GetInt'
nsconfigrw.h:61: error: previous declaration of 'Cfg_GetInt' was here
configrw.c: In function 'Cfg_GetInt':
configrw.c:266: warning: passing argument 3 of 'Cfg_GetRange' discards
qualifiers from pointer target type
configrw.c: At top level:
configrw.c:291: error: conflicting types for 'Cfg_GetRange'
nsconfigrw.h:65: error: previous declaration of 'Cfg_GetRange' was here
make: *** [configrw.o] Error 1



Also:


> static int
> GetBoolFromParam(Param *param, int *result)
> {
>     if (param->type != IntType) {
>         int bool;
>         if (   STREQ (param->value, "1"   )
>             || STRIEQ(param->value, "y"   )
>             || STRIEQ(param->value, "yes" )
>             || STRIEQ(param->value, "on"  )
>             || STRIEQ(param->value, "t"   )
>             || STRIEQ(param->value, "true")) {
>             param->intval = 1;


int Tcl_GetBoolean(Tcl_Interp *, CONST char *string, int *boolPtr)

  http://www.tcl.tk/man/tcl8.4/TclLib/GetInt.htm

(You can pass  it a NULL interp)

Re: [naviserver-devel] Large file upload dilemma

[Vasiljevic Z. <zv...@ar...>]

On 23.10.2007, at 21:57, Vlad Seryakov wrote:

> Yes, i am thinking how to make driver to spool data into regular file,
> not mmap it and just pass to the connection thread. In the conn some
> kind of command like ns_conn tempfile would return name of the file  
> and
> in the script i have to do whatever i want.
>
> This asks for another knob to introduce, something like max upload  
> limit
> if greater, contents will go directly into file without parsing.
>
> The spooler thread can still do the upload work.

Hm... this reires some plumbing, right?

Being able to delegate mpart processing to
some other "thing" is generally good option.
I just do not know how you will "communicate"
that to the connection thread.
Do you have some plan? 

Re: [naviserver-devel] Large file upload dilemma

[Vlad S. <vl...@cr...>]

> 
> Your dilemma ist: how to specifiy that server should NOT preparse
> the uploaded multipart-data?
> 

Yes, i am thinking how to make driver to spool data into regular file, 
not mmap it and just pass to the connection thread. In the conn some 
kind of command like ns_conn tempfile would return name of the file and 
in the script i have to do whatever i want.

This asks for another knob to introduce, something like max upload limit 
if greater, contents will go directly into file without parsing.

The spooler thread can still do the upload work.

Re: [naviserver-devel] Large file upload dilemma

[Vasiljevic Z. <zv...@ar...>]

On 23.10.2007, at 21:26, Vlad Seryakov wrote:

> We are building web site which will accept uploads of very big files,
> 1-2GB is average. Spooler works fine but the problem comes after the
> upload. With big number of users, all my virtual memory will be
> exhausted very quickly by mmap, and after the upload i need to copy
> those files somewhere because they are temp and deleted already.
>
> This makes the whole spooling thing useless by its own, i think it  
> needs
> to be extended so i can somehow tell that those urls needs to go into
> normal files, not mmaped without parsing multipart-data at all, for  
> such
> big files, it is better to do it offline than on the web server.

Your dilemma ist: how to specifiy that server should NOT preparse
the uploaded multipart-data?

Re: [naviserver-devel] nsconfigrw

[Vasiljevic Z. <zv...@ar...>]

On 23.10.2007, at 21:24, Stephen Deasey wrote:
>
> But one of the things we want to support is introspection, right?

Right.

>
>
>   foreach section [ns_getconfig] {
>       foreach {k v} [ns_getconfig $section] {
>           ...
>       }
>   }
>
>
> The only way to make it work atm is to pre-declare all config values:
>
>
>   # mylib.tcl
>
>   ns_setconfig section enabled true
>   ...
>
>   proc foo {} {
>       if {[ns_getconfig section enabled]} {
>           ...
>       }
>   }
>   ...
>
>
> Pre-declaring is a new restriction. Is this the plan?

Well, I went ahead in making it ns_config compatible
although this may NOT really always be clear.

If you do not set default values, then module A might
have different defaults then module B, which is right
and wrong, depending on the position...
If you however set them in module A, might that have
side-effects in module B ?

Perhaps the clearest is to have both, as you say below...

>
> I could also imagine having a hybrid system, where you *could*
> pre-declare, but you could also have ad-hoc calls. Both have their
> benefits:
>
> - Config which is only referenced once, and rarely if ever needs to be
> changed, is ideally handled with inline calls. Everything is in one
> place.
>
> - Config which cannot be guessed, e.g. a host name, or which is
> accessed from multiple pieces of code, would benefit from a
> declaration. This would allow the default to be written once and make
> options for configuring code more obvious.
>
>
> A system of pre-declaring might have benefits for C-code especially.
>
>
> Or..?
>


So we'd change (back) the config to set default values when first
required ?

[naviserver-devel] Large file upload dilemma

[Vlad S. <vl...@cr...>]

We are building web site which will accept uploads of very big files, 
1-2GB is average. Spooler works fine but the problem comes after the 
upload. With big number of users, all my virtual memory will be 
exhausted very quickly by mmap, and after the upload i need to copy 
those files somewhere because they are temp and deleted already.

This makes the whole spooling thing useless by its own, i think it needs 
to be extended so i can somehow tell that those urls needs to go into 
normal files, not mmaped without parsing multipart-data at all, for such 
big files, it is better to do it offline than on the web server.