naviserver-devel — Developer discussion

[naviserver-devel] NsWriterQueue Assertion fails

[David O. <da...@qc...>]

Hi,

We spotted that, if the following conditions are met, then a malformed HTTP
request will make a Naviserver assert fail (4.99.15).

   - Naviserver is compiled with --enable-symbols
   - At least 1 writerthread is running
   - A malformed HTTP request is made to a fastpath file which is bigger
   than writersize

I have 2 questions.

1. Is there a way of compiling Naviserver with debug symbols included, but
with NDEBUG flag set as to not abort when an assert fails?

2. Is the failing assert showing us something interesting?

A test case which demonstrates the assert failure from a fresh Naviserver
build on Debian 8.7 (kernel 3.16.0-4-amd64) is the following:

*SERVER*

hg clone https://bitbucket.org/naviserver/naviserver
cd naviserver
./autogen.sh --enable-rpath *--enable-symbols* --enable-threads
make
make install
chown -R nsd /usr/local/ns/logs/
cd /usr/local/ns/pages
wget www.qcode.co.uk

Edit:
/usr/local/ns/conf/nsd-config.tcl

In the ns/server/default/module/nssock section add:
ns_param        writerthreads   1
ns_param        writersize      1

Run:
/usr/local/ns/bin/nsd -c -u nsd -t /usr/local/ns/conf/nsd-config.tcl

*CLIENT*

echo "GET /index.html" > index.txt
cat index.txt | nc <server-ip> 8080

*SERVER*

[13/Feb/2017:18:06:40][22169.7fb8a27fc700][-driver:nssock:0-] Notice:
pre-HTTP/1.0 request <GET /index.html>
nsd: driver.c:4872: NsWriterQueue: Assertion `wrSockPtr->headerString ==
((void *)0)' failed.
Aborted

The assert that's failing, driver.c:4872 is:
assert(wrSockPtr->headerString == NULL);
https://bitbucket.org/naviserver/naviserver/src/443eb0b47a9fa0eb55d1bdd9f3a90ff92db55aa1/nsd/driver.c?at=default&fileviewer=file-view-default#driver.c-4873
(In the debugger, headerString is showing as "")

Thanks for any help.

Regards,
-- 
David

Re: [naviserver-devel] Access logs logging 2 IPs for single request

[David O. <da...@qc...>]

Ah... We do have checkforproxy enabled.
So while it may hold true that a request can only have 1 peer, the
X-Forwarded-For header could contain a list of proxies after the client
address. Makes sense - thanks



On 24 January 2017 at 10:32, Gustaf Neumann <ne...@wu...> wrote:

> Am 24.01.17 um 11:02 schrieb David Osborne:
>
> Can anyone come up with a scenario where a request could have 2 peer IP
> addresses?
>
> We occasionally see entries in the naviserver access logs like this one
> and I'm really not sure what to make of them:
>
> 10.x.x.x, 37.x.x.x - - [23/Jan/2017:07:55:55 +0000] "GET / HTTP/1.0" 500 671 "https://www.domain.co.uk/index.html" "Opera/9.80 (Android; Opera Mini/7.7.40394/37.9311; U; en) Presto/2.12.423 Version/12.16"
>
> Maybe running behind a proxy and/or parameter "checkforproxy" of
> ns/server/${server}/module/nslog set to true?
> When this parameter is it, it will produce an entry as provided by
> X-Forwarded-For.
>
> -g
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>


-- 
David Osborne
Qcode Software Limited
http://www.qcode.co.uk
T: +44 (0)1463 896484

Re: [naviserver-devel] Access logs logging 2 IPs for single request

[John f. D. E. <jo...@de...>]

I’m guessing that’s Opera caching/compression/proxy technology doing that, unless you tell me that you see this from any other browser besides Opera Mini.  

With Opera mini, you request a page on your phone with the opera browser, and opera.com’s server farm fetches the page for you, compresses it (lowering jpg quality) and sends it to the user.  For whatever reason, Opera’s designers decided to tell you both their IP address and that of the phone as well.

Re: [naviserver-devel] Access logs logging 2 IPs for single request

[Gustaf N. <ne...@wu...>]

Am 24.01.17 um 11:02 schrieb David Osborne:
> Can anyone come up with a scenario where a request could have 2 peer 
> IP addresses?
>
> We occasionally see entries in the naviserver access logs like this 
> one and I'm really not sure what to make of them:
>
> |10.x.x.x, 37.x.x.x - - [23/Jan/2017:07:55:55 +0000] "GET / HTTP/1.0" 
> 500 671 "https://www.domain.co.uk/index.html" "Opera/9.80 (Android; 
> Opera Mini/7.7.40394/37.9311; U; en) Presto/2.12.423 Version/12.16"|
Maybe running behind a proxy and/or parameter "checkforproxy" of 
ns/server/${server}/module/nslog set to true?
When this parameter is it, it will produce an entry as provided by 
X-Forwarded-For.

-g

[naviserver-devel] Access logs logging 2 IPs for single request

[David O. <da...@qc...>]

Hi,

Can anyone come up with a scenario where a request could have 2 peer IP
addresses?

We occasionally see entries in the naviserver access logs like this one and
I'm really not sure what to make of them:

10.x.x.x, 37.x.x.x - - [23/Jan/2017:07:55:55 +0000] "GET / HTTP/1.0"
500 671 "https://www.domain.co.uk/index.html" "Opera/9.80 (Android;
Opera Mini/7.7.40394/37.9311; U; en) Presto/2.12.423 Version/12.16"

Thanks.
-- 
David

Re: [naviserver-devel] ADP without file name extensions

[John f. D. E. <jo...@de...>]

Wouldn't be a solution based on a default rule as suggested by Stephen, where the name with out extension is mapped to the adp processor, which checks for a default extension before retuning 404, an option for your scenario? 
Yes, absolutely, Stephen’s suggestion will work for me.  

Another approach to solving this my problem would be to register a 404 handler, which then checks to see if an.adp version of the request exists and returns that instead.  For efficiency, I could modify ns_register_adp that page after the first 404 happens.  Basically, it’s stephen’s approach combined with a 404 handler.

The advantage of this approach is that I don’t have to pre-register all my ADPs and also if I create new ADPs the 404 handler automatically notices them.

On my bookmooch.com site (running aolserver) I use a 404 handler to return book cover images and file-system-cache the results. The first time a URL such as http://bookmooch.com/img/23423423.jpg is called, the file doesn’t exist, so the 404 handler makes the jpg (by proxying from amazon.com) and returns it, and then saves the contents as a file, for fast future delivery.  This sort of hack is why I love navisever/aolserver, as doing this sort of thing on any other web platform I can think of would be really hard.

So, now I have at least two good solutions to my problem, thank you, and so if you don’t want to mess up the source with new options I’d be ok with that. :D

-john

Re: [naviserver-devel] ADP without file name extensions

[Gustaf N. <ne...@wu...>]

Am 17.01.17 um 23:53 schrieb Stephen:
>> That is a neat idea! The only disadvantage is, that the order of
>> registration is important,
>> which can can lead to fiddling with the config file and to some not easy
>> predictable behavior.
> Does it make any difference in practice?
Don't know. The config files are designed having separation of concerns 
in mind, the plain NaviServer user has typically no knowledge about the 
registration order and how to influence it.
> This test looks looks good. Does it still work if /* and /*.* are
> registered after the others?
yes, by looking at the code and some small tests. But the fact, that you 
are asking the question show,  that the precedence order is not fully 
clear. However, for this case, only a little control is needed.

By these experiments, i would tend to think, that we do not need 
negative patterns. However, getting more control over the URLspace (e.g. 
noinherit) would be nice, but that can be as well achieved via the 
programmatic interface.

-g

Re: [naviserver-devel] ADP without file name extensions

[Gustaf N. <ne...@wu...>]

Am 18.01.17 um 03:45 schrieb John from Decent Espresso:
>
> from what I’m understanding, implementing a 
> anything-without-an-extension-is-adp would be a filter like this?
>
>>      ns_register_adp GET /*^*.*
> Is that right?
> Can you confirm a file name of "test.situation.adp" would work with this filter?
It depends, what "work" means and in which order the entries are defined 
(as discussed in the other thread).
In the following scenario, "/foo" is interpreted as ADP and "/foo.1" as 
FILE (along with the more obvious cases.

=======================================================================
test ns_urlspace-5.3 {wild-card-filter with negative patterns} -setup {
     ns_urlspace set -key 5.3 /*.*   FILE
     ns_urlspace set -key 5.3 /*.adp ADP
     ns_urlspace set -key 5.3 /*.tcl TCL
     ns_urlspace set -key 5.3 /*^*.* ADP
} -body {
     foreach url {
	/foo.adp /foo.tcl /foo.txt /foo
	/foo.1.adp /foo.1.tcl /foo.1.txt /foo.1 	
     } {
	lappend _ [ns_urlspace get -key 5.3 $url]
     }
     set _
} -cleanup {
     unset -nocomplain _
} -result {ADP TCL FILE ADP ADP TCL FILE FILE}
=======================================================================

However, when the first mapping is be changed from "/*.*" to "/*", this 
will - in my current, uncommited implementation of negative patterns - 
subsume the line with the negative pattern, and "/foo" will be mapped to 
"FILE" as well (these subsumption interactions have to be looked into it 
in detail, as indicated earlier).

Notice as well, that the NaviServer "filters" (registered via 
"ns_register_filter") are an implementation completely independent of 
the urlspace (used e.g. for "ns_register_proc", "ns_register_adp", ...). 
Filters are collected in a linear list (not in a trie like in urlspace), 
filters have a "continue" interface (e.g. should more filters be 
executed?) and they use only simple tcl match on the method and url. So, 
when the urlspace machinery is extended, this does not affect filters. 
The urlspace trie is designed to scale and to work with large amounts of 
url patterns (see as well the section "similar, but significantly 
different" in [1]).

Wouldn't be a solution based on a default rule as suggested by Stephen, 
where the name with out extension is mapped to the adp processor, which 
checks for a default extension before retuning 404, an option for your 
scenario? I've added the change for handling such a default extension to 
bitbucket [2].

-g

[1] https://naviserver.sourceforge.io/n/naviserver/files/ns_register.html
[2] 
https://bitbucket.org/naviserver/naviserver/commits/f563564dfc30f52a88483bda54f2c0887c17474f

Re: [naviserver-devel] ADP without file name extensions

[John f. D. E. <jo...@de...>]

Some of these premises overlap.
One option would be to switch from glob string matching to regexp,
but that would be significantly slower and would require a larger
changes to avoid frequent regexp compilations.
I agree with your reasoning, and would also argue that moving to regexp would potentially break existing filters, as well as a big speed penalty. That’s why I coded my initial implementation in C. If someone wants regexp filename matching, they can just filter every page request and do the regexp in Tcl.

The implemented approach allows to specify in addition to a
(positive) match pattern (such as "foo*") optionally a negative match
pattern (such as ^*.*), which can be used to say "match which such
It’s a slightly weird pattern match syntax, but as it’s efficient and easily implemented, I like it!

from what I’m understanding, implementing a anything-without-an-extension-is-adp would be a filter like this?

    ns_register_adp GET /*^*.*
Is that right?
Can you confirm a file name of "test.situation.adp" would work with this filter? 
i.e.:
http://localhost/test.situation
If not, that’s an ok limitation, but I wanted to call it out.

-john

Re: [naviserver-devel] ADP without file name extensions

[Stephen <yo...@gr...>]

On Tue, Jan 17, 2017 at 9:43 PM, Gustaf Neumann <ne...@wu...> wrote:
>
> Dear Stephen and all,
>
> That is a neat idea! The only disadvantage is, that the order of
> registration is important,
> which can can lead to fiddling with the config file and to some not easy
> predictable behavior.


Does it make any difference in practice?  Looks like the filters are
stored sorted according to this order:

  https://bitbucket.org/naviserver/naviserver/src/57c84d3671e7127ae42f3754463dead32b04f256/nsd/urlspace.c?at=default&fileviewer=file-view-default#urlspace.c-1141

Which tends to match more specific filters first.

This test looks looks good. Does it still work if /* and /*.* are
registered after the others?


> The behavior can be neatly demonstrated with ns_urlspace
> (using X as a special key for testing purposes).
>
> =============================================================
> test ns_urlspace-5.2 {wild-card-filter with ordering} -setup {
>      ns_urlspace set -key X /*        ADP
>      ns_urlspace set -key X /*.adp    ADP
>      ns_urlspace set -key X /*.tcl    TCL
>      ns_urlspace set -key X /*.*      FILE
>      ns_urlspace set -key X /static/* FILE
> } -body {
>      foreach url {
>         /test.adp /test.txt /test.tcl /test
>         /static/test.adp /static/test.txt /static/test.tcl /static/test
>      } {
>         lappend _ [ns_urlspace get -key X $url]
>      }
>      set _
> } -cleanup {
>      unset -nocomplain _
> } -result {ADP FILE TCL ADP FILE FILE FILE FILE}
> =============================================================

Re: [naviserver-devel] ADP without file name extensions

[Gustaf N. <ne...@wu...>]

Dear Stephen and all,

That is a neat idea! The only disadvantage is, that the order of 
registration is important,
which can can lead to fiddling with the config file and to some not easy 
predictable behavior.

The behavior can be neatly demonstrated with ns_urlspace
(using X as a special key for testing purposes).

=============================================================
test ns_urlspace-5.2 {wild-card-filter with ordering} -setup {
     ns_urlspace set -key X /*        ADP
     ns_urlspace set -key X /*.adp    ADP
     ns_urlspace set -key X /*.tcl    TCL
     ns_urlspace set -key X /*.*      FILE
     ns_urlspace set -key X /static/* FILE
} -body {
     foreach url {
	/test.adp /test.txt /test.tcl /test
	/static/test.adp /static/test.txt /static/test.tcl /static/test	
     } {
	lappend _ [ns_urlspace get -key X $url]
     }
     set _
} -cleanup {
     unset -nocomplain _
} -result {ADP FILE TCL ADP FILE FILE FILE FILE}
=============================================================

The urls "/test.adp"  and "/test" are mapped to ADP, therefore these can be
mapped to "ns_register_adp" and the other ones to either 
"ns_register_tcl" or
"ns_register_fasturl2file". Just the tiny change for adding the default 
extension
in ADP PageRequest() is needed to achieve what John asked for....

-g

Am 17.01.17 um 18:21 schrieb Stephen:
> Try something like:
>
>      ns_register_url2file /* my_no_adp_ext_url2file_proc
>
>      foreach ext {adp html txt xml jpg gif} {
>          ns_register_fasturl2file "/*.$ext"
>      }
>      ns_register_fasturl2file /static/*
>
>
> ns_register_url2file handles only url-to-file lookups so you can still
> use filters and registered procs as usual
>
> my_no_adp_ext_url2file_proc only has to handle the case where no
> extension is present even though it is registered for the whole
> urlspace, because...
>
> the filter /*.adp is more specific than /*, so the default fast
> url2file handler will run for any url with a common extension in the
> global urlspace, plus all urls under /static/*
>
>
>
> On Tue, Jan 17, 2017 at 12:12 PM, Gustaf Neumann <ne...@wu...> wrote:
>> Dear John,
>>
>> I've prototyped something along these lines, which looks promising so far.
>>
>> Premises:
>> - fit into urlspace mechanism
>> - not limited in applicability to .adp
>> - no full rewrite of urlspace (which is already complex enough)
>> - keep speed of existing implementation
>> - provide a mostly conservative extensions
>>
>> Some of these premises overlap.
>> One option would be to switch from glob string matching to regexp,
>> but that would be significantly slower and would require a larger
>> changes to avoid frequent regexp compilations.
>>
>> The implemented approach allows to specify in addition to a
>> (positive) match pattern (such as "foo*") optionally a negative match
>> pattern (such as ^*.*), which can be used to say "match which such
>> names except with some others". One can specify e.g. "foo*^*.*"
>> which means files starting with "foo" but which must not have
>> an extension (must not match with "*.*"). One can use as well
>> rules like "must not contain a dash", or "must not start with an upper
>> case character" or whatever. Both match pattern are applied with
>> Tcl's string match implementation, which is quite efficient.
>>
>> To see this in action, look at the following test case, where
>> an urlspace entry s set with an extension (foo*.tcl), one other is
>> set with a wild card (/bar*) and one is set with a wild card and
>> a negative pattern. These urlspace entries are matched with
>> URLs with and without extension. Either
>> - only the URL with the extension is matched (first two cases), or
>> - both URLs are matched (second two cases), or
>> - just the entry without the extension is matched (last two cases)
>>
>> =======================================================================
>> test ns_urlspace-5.1 {wild-card-filter with negative patterns} -setup {
>>      ns_urlspace set /foo*.tcl 1
>>      ns_urlspace set /bar*     2
>>      ns_urlspace set /baz*^*.* 3
>> } -body {
>>      foreach url {
>> 	/foo1.tcl /foo1
>> 	/bar1.tcl /bar1
>> 	/baz1.tcl /baz1
>>      } {
>> 	lappend _ [ns_urlspace get $url]
>>      }
>>      set _
>> } -cleanup {
>>      unset -nocomplain _
>> } -result {1 {} 2 2 {} 3}
>> =======================================================================
>>
>> The path matching is untouched, as well as the features noinherit, exact,
>> etc.
>>
>> To apply this approach to the .adp case, one can now call ns_register_adp
>> (in addition to the standard cases) with a negative pattern, add optionally
>> in the ADP PageRequest() the .adp if the first match fails, and we are done.
>>
>> In the test cases, there exists a /helloworld.adp file, which can be called
>> now via /helloworld, whereas /hellow returns a 404 like usual.
>>
>> =======================================================================
>> test adp-2.1 {ADP page map} -setup {
>>      ns_register_adp GET /hello*^*.*
>> } -body {
>>      nstest::http -getbody 1 GET /helloworld
>> } -cleanup {
>>      ns_unregister_op GET /hello*^*.*
>> } -result {200 {Hello World!}}
>> =======================================================================
>>
>> I have not looked at the consequences for subsumption of patterns, or
>> deletion/querying of patterns etc, but this seems doable.
>>
>> Syntax-wise, we could consider as well the following candidates
>>
>>     {/hello*^[*.*]}
>>     "/hello* ^*.*"
>>      /hello*^{*.*}
>>      /hello*{^}*.*
>>      /hello*{NOT}*.*
>>     "/hello* -not *.*"
>>     "/hello* -not *.* -noinherit"
>>
>> However, i am not too happy about requiring quoting with double quotes or
>> braces.
>> The single character "^" has advantages for escaping, when it should be
>> taken literally
>> (and not as start character for a negative pattern).
>>
>> The last syntax option has the advantage to specify also "-noinherit" in the
>> string
>> provided to e.g. map in the config file, which is currently not possible
>> (meaning:
>> do not apply this matching rule for the underlying directories).
>>
>> OTOH, the -noinherit etc. is already supported by the API, so, why reinvent
>> some
>> (redundant) syntax, we could as well use instead of "map" a "mapcmd" with a
>> generalized "ns_register" (setting per default the methods GET POST HEAD)
>>
>>     ns_section ns/server/default/adp
>>     ns_param   mapcmd  {ns_register  ... -pattern /*.adp -noinherit}
>>     ns_param   mapcmd  {ns_register  ... -pattern /* -except *.*}
>>
>> this way, we get standardized error messages, etc... The old "map" syntax
>> can be supported as well for a while... The same can be made as well for
>> other "maps" (e.g. connection pool mappings) Comments are always welcome.
>> all the best -g
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel


-- 
Univ.Prof. Dr. Gustaf Neumann
WU Vienna
Institute of Information Systems and New Media
Welthandelsplatz 1, A-1020 Vienna, Austria

Re: [naviserver-devel] ADP without file name extensions

[Stephen <yo...@gr...>]

Try something like:

    ns_register_url2file /* my_no_adp_ext_url2file_proc

    foreach ext {adp html txt xml jpg gif} {
        ns_register_fasturl2file "/*.$ext"
    }
    ns_register_fasturl2file /static/*


ns_register_url2file handles only url-to-file lookups so you can still
use filters and registered procs as usual

my_no_adp_ext_url2file_proc only has to handle the case where no
extension is present even though it is registered for the whole
urlspace, because...

the filter /*.adp is more specific than /*, so the default fast
url2file handler will run for any url with a common extension in the
global urlspace, plus all urls under /static/*



On Tue, Jan 17, 2017 at 12:12 PM, Gustaf Neumann <ne...@wu...> wrote:
> Dear John,
>
> I've prototyped something along these lines, which looks promising so far.
>
> Premises:
> - fit into urlspace mechanism
> - not limited in applicability to .adp
> - no full rewrite of urlspace (which is already complex enough)
> - keep speed of existing implementation
> - provide a mostly conservative extensions
>
> Some of these premises overlap.
> One option would be to switch from glob string matching to regexp,
> but that would be significantly slower and would require a larger
> changes to avoid frequent regexp compilations.
>
> The implemented approach allows to specify in addition to a
> (positive) match pattern (such as "foo*") optionally a negative match
> pattern (such as ^*.*), which can be used to say "match which such
> names except with some others". One can specify e.g. "foo*^*.*"
> which means files starting with "foo" but which must not have
> an extension (must not match with "*.*"). One can use as well
> rules like "must not contain a dash", or "must not start with an upper
> case character" or whatever. Both match pattern are applied with
> Tcl's string match implementation, which is quite efficient.
>
> To see this in action, look at the following test case, where
> an urlspace entry s set with an extension (foo*.tcl), one other is
> set with a wild card (/bar*) and one is set with a wild card and
> a negative pattern. These urlspace entries are matched with
> URLs with and without extension. Either
> - only the URL with the extension is matched (first two cases), or
> - both URLs are matched (second two cases), or
> - just the entry without the extension is matched (last two cases)
>
> =======================================================================
> test ns_urlspace-5.1 {wild-card-filter with negative patterns} -setup {
>     ns_urlspace set /foo*.tcl 1
>     ns_urlspace set /bar*     2
>     ns_urlspace set /baz*^*.* 3
> } -body {
>     foreach url {
> 	/foo1.tcl /foo1
> 	/bar1.tcl /bar1
> 	/baz1.tcl /baz1
>     } {
> 	lappend _ [ns_urlspace get $url]
>     }
>     set _
> } -cleanup {
>     unset -nocomplain _
> } -result {1 {} 2 2 {} 3}
> =======================================================================
>
> The path matching is untouched, as well as the features noinherit, exact,
> etc.
>
> To apply this approach to the .adp case, one can now call ns_register_adp
> (in addition to the standard cases) with a negative pattern, add optionally
> in the ADP PageRequest() the .adp if the first match fails, and we are done.
>
> In the test cases, there exists a /helloworld.adp file, which can be called
> now via /helloworld, whereas /hellow returns a 404 like usual.
>
> =======================================================================
> test adp-2.1 {ADP page map} -setup {
>     ns_register_adp GET /hello*^*.*
> } -body {
>     nstest::http -getbody 1 GET /helloworld
> } -cleanup {
>     ns_unregister_op GET /hello*^*.*
> } -result {200 {Hello World!}}
> =======================================================================
>
> I have not looked at the consequences for subsumption of patterns, or
> deletion/querying of patterns etc, but this seems doable.
>
> Syntax-wise, we could consider as well the following candidates
>
>    {/hello*^[*.*]}
>    "/hello* ^*.*"
>     /hello*^{*.*}
>     /hello*{^}*.*
>     /hello*{NOT}*.*
>    "/hello* -not *.*"
>    "/hello* -not *.* -noinherit"
>
> However, i am not too happy about requiring quoting with double quotes or
> braces.
> The single character "^" has advantages for escaping, when it should be
> taken literally
> (and not as start character for a negative pattern).
>
> The last syntax option has the advantage to specify also "-noinherit" in the
> string
> provided to e.g. map in the config file, which is currently not possible
> (meaning:
> do not apply this matching rule for the underlying directories).
>
> OTOH, the -noinherit etc. is already supported by the API, so, why reinvent
> some
> (redundant) syntax, we could as well use instead of "map" a "mapcmd" with a
> generalized "ns_register" (setting per default the methods GET POST HEAD)
>
>    ns_section ns/server/default/adp
>    ns_param   mapcmd  {ns_register  ... -pattern /*.adp -noinherit}
>    ns_param   mapcmd  {ns_register  ... -pattern /* -except *.*}
>
> this way, we get standardized error messages, etc... The old "map" syntax
> can be supported as well for a while... The same can be made as well for
> other "maps" (e.g. connection pool mappings) Comments are always welcome.
> all the best -g
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] ADP without file name extensions

[Gustaf N. <ne...@wu...>]

Dear John,

I've prototyped something along these lines, which looks promising so far.

Premises:
- fit into urlspace mechanism
- not limited in applicability to .adp
- no full rewrite of urlspace (which is already complex enough)
- keep speed of existing implementation
- provide a mostly conservative extensions

Some of these premises overlap.
One option would be to switch from glob string matching to regexp,
but that would be significantly slower and would require a larger
changes to avoid frequent regexp compilations.

The implemented approach allows to specify in addition to a
(positive) match pattern (such as "foo*") optionally a negative match
pattern (such as ^*.*), which can be used to say "match which such
names except with some others". One can specify e.g. "foo*^*.*"
which means files starting with "foo" but which must not have
an extension (must not match with "*.*"). One can use as well
rules like "must not contain a dash", or "must not start with an upper
case character" or whatever. Both match pattern are applied with
Tcl's string match implementation, which is quite efficient.

To see this in action, look at the following test case, where
an urlspace entry s set with an extension (foo*.tcl), one other is
set with a wild card (/bar*) and one is set with a wild card and
a negative pattern. These urlspace entries are matched with
URLs with and without extension. Either
- only the URL with the extension is matched (first two cases), or
- both URLs are matched (second two cases), or
- just the entry without the extension is matched (last two cases)

=======================================================================
test ns_urlspace-5.1 {wild-card-filter with negative patterns} -setup {
     ns_urlspace set /foo*.tcl 1
     ns_urlspace set /bar*     2
     ns_urlspace set /baz*^*.* 3
} -body {
     foreach url {
	/foo1.tcl /foo1
	/bar1.tcl /bar1
	/baz1.tcl /baz1
     } {
	lappend _ [ns_urlspace get $url]
     }
     set _
} -cleanup {
     unset -nocomplain _
} -result {1 {} 2 2 {} 3}
=======================================================================

The path matching is untouched, as well as the features noinherit, 
exact, etc.

To apply this approach to the .adp case, one can now call ns_register_adp
(in addition to the standard cases) with a negative pattern, add optionally
in the ADP PageRequest() the .adp if the first match fails, and we are 
done.

In the test cases, there exists a /helloworld.adp file, which can be called
now via /helloworld, whereas /hellow returns a 404 like usual.

=======================================================================
test adp-2.1 {ADP page map} -setup {
     ns_register_adp GET /hello*^*.*
} -body {
     nstest::http -getbody 1 GET /helloworld
} -cleanup {
     ns_unregister_op GET /hello*^*.*
} -result {200 {Hello World!}}
=======================================================================

I have not looked at the consequences for subsumption of patterns, or
deletion/querying of patterns etc, but this seems doable.

Syntax-wise, we could consider as well the following candidates

    {/hello*^[*.*]}
    "/hello* ^*.*"
     /hello*^{*.*}
     /hello*{^}*.*
     /hello*{NOT}*.*
    "/hello* -not *.*"
    "/hello* -not *.* -noinherit"

However, i am not too happy about requiring quoting with double quotes 
or braces.
The single character "^" has advantages for escaping, when it should be 
taken literally
(and not as start character for a negative pattern).

The last syntax option has the advantage to specify also "-noinherit" in 
the string
provided to e.g. map in the config file, which is currently not possible 
(meaning:
do not apply this matching rule for the underlying directories).

OTOH, the -noinherit etc. is already supported by the API, so, why 
reinvent some
(redundant) syntax, we could as well use instead of "map" a "mapcmd" with a
generalized "ns_register" (setting per default the methods GET POST HEAD)

    ns_section ns/server/default/adp
    ns_param   mapcmd  {ns_register ... -pattern /*.adp -noinherit}
    ns_param   mapcmd  {ns_register ... -pattern /* -except *.*}

this way, we get standardized error messages, etc... The old "map" 
syntax can be supported as well for a while... The same can be made as 
well for other "maps" (e.g. connection pool mappings) Comments are 
always welcome. all the best -g

Re: [naviserver-devel] gzipping static content - how to do?

[Gustaf N. <ne...@wu...>]

Suggestions for an improved wording in the documentation are welcome.
-g

Am 17.01.17 um 04:51 schrieb John from Decent Espresso:
>> As David mentioned, one has to make a initial pick which files should 
>> be delivered via static gzip by gzipping these. As you mentioned, the 
>> the rules, what exactly should be delivered via gzip and what not 
>> might be complex, so, we do not want to add overhead at the first place.
>
> Ok, that actually makes a lot of sense feature-wise, I just did not 
> understand that important point from the existing documentation. So, 
> it appears my system is now running the gzip compression correctly.
>

Re: [naviserver-devel] gzipping static content - how to do?

[John f. D. E. <jo...@de...>]

As David mentioned, one has to make a initial pick which files should be delivered via static gzip by gzipping these. As you mentioned, the the rules, what exactly should be delivered via gzip and what not might be complex, so, we do not want to add overhead at the first place.
Ok, that actually makes a lot of sense feature-wise, I just did not understand that important point from the existing documentation. So, it appears my system is now running the gzip compression correctly.

-john

Re: [naviserver-devel] ADP without file name extensions

[John f. D. E. <jo...@de...>]

You have asked this question before:
Sorry Gustaf, I thought I had asked aolserver before, not naviserver. At any rate, still no good solution to this, other than adding a tcl filter that captures and potentially rewrites all requests.

However, one does not want to run the filter on every requests, 
Agreed.

and the approach should be compatible with the way how NaviServer configuration works for choosing on which files the ADP parser should be used. This is happening now via ns_register_adp (see [1]) using a config section like:
   ns_section     "ns/server/default/adp"
   ns_param        map              /*.adp
   ns_param        disabled         false    
Another approach would be for that map to be defined for no-extension files, perhaps something like this:

   ns_param        map              /*?^[\.]
but such pattern matching isn't (to my knowledge) supported.


One approach would be to allow mapping rules for the URLspace machinery for empty extensions (something, which does not exist yet). Such a mapping rule could be used for a filter for only urls without extensions, or it could be used to map adp pages with empty extensions with "ns_register_adp" like above and let the ADP "PageRequest" check, if a file with ".adp" exists. Such a syntactic rule for for the URLspace could be used as well in other operations on the URLspace (for registering procs,  connection pool mapping, ns_urlspace, etc.). This would be backward compatible and avoid overall overhead....
I quite like your suggestion, which I assume would be something like this:

   ns_section     "ns/server/default/adp"
   ns_param        enable_when_no_extension              true
   ns_param        disabled         false    
that’s more or less what I’ve implemented with my hack.
PS: As we are using on our main sites OpenACS, we do not have this problem, since OpenACS defines its own "request processor" which handles every URL.
I see.

-john

Re: [naviserver-devel] ADP without file name extensions

[Gustaf N. <ne...@wu...>]

Am 16.01.17 um 10:17 schrieb John from Decent Espresso:
>
> A different topic. I use the code patch below to get rid of the need 
> for ".adp" on the end of my URLs.
>
> In other words, naviserver transparently rewrites
> https://decentespresso.com/cart
> as
> https://decentespresso.com/cart.adp
>
> and doesn’t display the .adp to the user in the browser.  This makes 
> for very clear urls.
>
> This works for me, but would you recommend a less hard-wired hack to 
> accomplish this same thing?

You have asked this question before:

   [1] https://sourceforge.net/p/naviserver/mailman/message/32586538/

I don't like a hard-coded approach - the C code is agnostic on the .adp 
extension. One would probably want to have the following behavior on an 
URL ending with "/file" or "/file.adp":

    called   "file exists" "file.adp exists"  action
    =====================================================================
    file        1             1             deliver "file"
    file        1             0             deliver "file"
    file        0             1             deliver "file.adp"
    file        0             0             error
    file.adp    1             1             deliver "file.adp"
    file.adp    1             0             deliver "file"
    file.adp    0             1             deliver "file.adp"
    file.adp    0             0             error


A filter can achieve this behavior (see jeffs solution in the thread 
above), which checks at runtime e.g. if "file" exists, and if not to try 
"file.adp".

However, one does not want to run the filter on every requests, and the 
approach should be compatible with the way how NaviServer configuration 
works for choosing on which files the ADP parser should be used. This is 
happening now via ns_register_adp (see [1]) using a config section like:

    ns_section     "ns/server/default/adp"
    ns_param        map              /*.adp
    ns_param        disabled         false


One approach would be to allow mapping rules for the URLspace machinery 
for empty extensions (something, which does not exist yet). Such a 
mapping rule could be used for a filter for only urls without 
extensions, or it could be used to map adp pages with empty extensions 
with "ns_register_adp" like above and let the ADP "PageRequest" check, 
if a file with ".adp" exists. Such a syntactic rule for for the URLspace 
could be used as well in other operations on the URLspace (for 
registering procs, connection pool mapping, ns_urlspace, etc.). This 
would be backward compatible and avoid overall overhead....

-g
PS: As we are using on our main sites OpenACS, we do not have this 
problem, since OpenACS defines its own "request processor" which handles 
every URL.


https://bitbucket.org/naviserver/naviserver/src/a6cf227cad8d0c03dc3dff3c643b704e06ea1238/tcl/config.tcl?at=default&fileviewer=file-view-default#config.tcl-120

Re: [naviserver-devel] gzipping static content - how to do?

[Gustaf N. <ne...@wu...>]

Am 16.01.17 um 10:17 schrieb John from Decent Espresso:
> Thanks Gustaf (Hahaha) that is the cause of the mime-type problem. 
> I’ve upgraded to ns 4.99.15 and the MIME type bug is now gone.
>
> However, I’m still not having gzipped files automatically made.  I’ve 
> "chmod 777 ." and also "chmod 666 *.css" to try to find the root.
>
> —
>
> It appears that ns_gzipfile is not being called at all, as I modded it 
> with this:
>
> proc ns_gzipfile {source target} {
>     puts "ns_gzipfile $source $target"
>
> and it’s not ever called.
>
> How would you recommend debugging this problem?
>
> Is there some other nsd.tcl parameter needed indicate that (say) .js 
> and .css files should be gzipped, but .jpg and .gif should not?
As David mentioned, one has to make a initial pick which files should be 
delivered via static gzip by gzipping these. As you mentioned, the the 
rules, what exactly should be delivered via gzip and what not might be 
complex, so, we do not want to add overhead at the first place.

Ages ago, i wrote the little script below as a kick-starter for gzipping 
for some filename patterns in some directories.
Maybe, someone will find this useful.

-g

###############################################################
#!/bin/bash
#
# This script checks a list of directories for a list of extensions and
# generated gzipped versions of the files that are found
#
# If the modification date of a file is newer than its gzipped version
# then the gzip file is regenerated
  
# Specify filetypes like *.css or a filename like index.html
FILETYPES="*.css *.js *.html"
  
# Specify a list of directories to check recursively
DIRECTORIES="/var/www/openacs.org/"

for currentdir in $DIRECTORIES
do
    for extension in $FILETYPES
    do
       find $currentdir -iname $extension -exec bash -c 'PLAINFILE={};GZIPPEDFILE={}.gz; \
          if [ -e $GZIPPEDFILE ]; \
          then   if [ `stat --printf=%Y $PLAINFILE` -gt `stat --printf=%Y $GZIPPEDFILE` ]; \
                 then    echo "$GZIPPEDFILE outdated, regenerating"; \
                         gzip -9 -f -c $PLAINFILE > $GZIPPEDFILE; \
                         chown nsadmin:nsadmin $GZIPPEDFILE; \
                  fi; \
          else echo "$GZIPPEDFILE is missing, creating it"; \
               gzip -9 -c $PLAINFILE > $GZIPPEDFILE; \
               chown nsadmin:nsadmin $GZIPPEDFILE; \
          fi' \;
    done
done

Re: [naviserver-devel] gzipping static content - how to do?

[John f. D. E. <jo...@de...>]

Not sure if you picked up that the gzip files aren't automatically created for you within Naviserver. But they are automatically refreshed. So you need to manually create an initial gzipped version of all the files you want to deliver gzip encoded.

Once the initial gzipped version exists, naviserver will deliver it, and re-gzip if the static file is updated.

Daniel's filter is one way of achieving the initial gzipping.
Ah ha!  No, missed that important detail! Thank you!

DONE!

find . -name '*.css' -exec gzip -9 -f -k {} \;

Re: [naviserver-devel] gzipping static content - how to do?

[David O. <da...@qc...>]

On 16 January 2017 at 09:17, John from Decent Espresso <
jo...@de...> wrote:

> Thanks Gustaf (Hahaha) that is the cause of the mime-type problem. I’ve
> upgraded to ns 4.99.15 and the MIME type bug is now gone.
>
> However, I’m still not having gzipped files automatically made.  I’ve
> "chmod 777 ." and also "chmod 666 *.css" to try to find the root.
>

Not sure if you picked up that the gzip files aren't automatically created
for you within Naviserver. But they are automatically refreshed. So you
need to manually create an initial gzipped version of all the files you
want to deliver gzip encoded.

Once the initial gzipped version exists, naviserver will deliver it, and
re-gzip if the static file is updated.

Daniel's filter is one way of achieving the initial gzipping.

Re: [naviserver-devel] gzipping static content - how to do?

[John f. D. E. <jo...@de...>]

Thanks Gustaf (Hahaha) that is the cause of the mime-type problem. I’ve upgraded to ns 4.99.15 and the MIME type bug is now gone.

However, I’m still not having gzipped files automatically made.  I’ve "chmod 777 ." and also "chmod 666 *.css" to try to find the root.

—

It appears that ns_gzipfile is not being called at all, as I modded it with this:

proc ns_gzipfile {source target} {
    puts "ns_gzipfile $source $target"

and it’s not ever called.

How would you recommend debugging this problem?

Is there some other nsd.tcl parameter needed indicate that (say) .js and .css files should be gzipped, but .jpg and .gif should not?


—

A different topic. I use the code patch below to get rid of the need for ".adp" on the end of my URLs.

In other words, naviserver transparently rewrites
https://decentespresso.com/cart
as
https://decentespresso.com/cart.adp

and doesn’t display the .adp to the user in the browser.  This makes for very clear urls.

This works for me, but would you recommend a less hard-wired hack to accomplish this same thing?

nsd/request.c
    if (*url != '\0' && url[strlen(url) - 1] == '/') {
        Ns_DStringAppend(&ds2, "/");
    }
     
/* john buckman added 4/14/06 */
/* check if should add default filename extension of .adp */
/* only if no / on end of url which indicates a directory */
char * dotpos;
if (ds2.string[ds2.length - 1] != '/') {
    /* if not . in the entire url, or if there is a dot before the final / (indicating a . in a
       directory name, which is ok, then add the default filename extension */
    dotpos = strrchr(ds2.string, '.');
    if ((dotpos == NULL) || (strchr(dotpos, '/') != NULL)) {
        Ns_DStringAppend(&ds2, ".adp");
/*        Ns_Log(Notice, "added default extension to get '%s'", ds2.string); */
    }
}
/* end john buckman added */

Re: [naviserver-devel] gzipping static content - how to do?

[Gustaf N. <ne...@wu...>]

Dear John,

What version of NaviServer are you using?
There was a related bug in NaviServer 4.99.12, which was fixed in Sept 
2016 by commit [2],
included in releases since 4.99.13 (Oct 2016)

all the best
-gustaf

[1] https://bitbucket.org/naviserver/naviserver/src/23966c68ab0a758f9d3127b5aaa9282de7289977/NEWS?at=default&fileviewer=file-view-default#NEWS-451
[2] https://bitbucket.org/naviserver/naviserver/commits/e26c913dd15c96ee0827a0cbf084936ff34bea49



Am 16.01.17 um 05:49 schrieb John from Decent Espresso:
> Hi Gustaf,
>
> When I create a .gz file for the CSS file (by hand, to test) the 
> problem I encounter is that naviserver changes the MIME type to 
> application/gzip, which makes chrome and firefox very unhappy:
> > skel.min.js:2 Resource interpreted as Stylesheet but transferred 
> with MIME type application/gzip: 
> "https://decentespresso.com/css/style-xlarge.css".
>
> -john
>

Re: [naviserver-devel] gzipping static content - how to do?

[John f. D. E. <jo...@de...>]

Hi Gustaf,

When I create a .gz file for the CSS file (by hand, to test) the problem I encounter is that naviserver changes the MIME type to application/gzip, which makes chrome and firefox very unhappy:
> skel.min.js:2 Resource interpreted as Stylesheet but transferred with MIME type application/gzip: "https://decentespresso.com/css/style-xlarge.css".

-john

Re: [naviserver-devel] gzipping static content - how to do?

[Gustaf N. <ne...@wu...>]

Dear John,

The option "gzip_static" requires, that the files to be delivered are gzip
compressed in the file system. In order to have "gzip_rerfresh" working
(automatically re-compress the file when the version has changed) requires
that the server has write permissions on the .gz file and that "gzip_cmd"
points to the gzip program.

Below is something some data from a life site (openacs.org) having
static gzip delivery enabled with a snippet from the config file.

There is no extra filter necessary to get this working.

best regards
-gustaf neumann

URL: 
http://www.openacs.org/resources/openacs-bootstrap3-theme/css/all.min.css

Snippet from the config file:

#---------------------------------------------------------------------
# Global fastpath parameters
#---------------------------------------------------------------------
ns_section      "ns/fastpath"
     #ns_param        cache               true       ;# default: false
     #ns_param        cachemaxsize        10240000   ;# default: 1024*10000
     #ns_param        cachemaxentry       100000     ;# default: 8192
     #ns_param        mmap                true       ;# default: false
     ns_param        gzip_static         true       ;# check for static gzip; default: false
     ns_param        gzip_refresh        true       ;# refresh stale .gz files on the fly using ::ns_gzipfile
     ns_param        gzip_cmd            "/bin/gzip -9"  ;# use for re-compressing


files in the file system
/var/www/openacs.org/packages/openacs-bootstrap3-theme/www/resources/css# ls -ltr
-rw-rw-r-- 1 openacs  nsadmin 115102 Jul 15  2016 all.min.css
-rw-rw-r-- 1 openacs  nsadmin  20213 Jul 15  2016 all.min.css.gz
...

Reply header fields:

Accept-Ranges	|bytes|
Connection	|keep-alive|
Content-Encoding	|gzip|
Content-Length	|20213|
Content-Type	|text/css|
Date	|Sat, 14 Jan 2017 15:01:46 GMT|
Expires	|Mon, 13 Feb 2017 15:01:46 GMT|
Last-Modified	|Fri, 15 Jul 2016 15:39:21 GMT|
Server	|NaviServer/4.99.15|
Vary	|Accept-Encoding|
X-Content-Type-Options	|nosniff|
X-Frame-Options	|SAMEORIGIN|
X-XSS-Protection	|1; mode=block|

  

Am 14.01.17 um 06:10 schrieb John from Decent Espresso:
> I’m struggling a bit to get naviserver to gzip static content (.js 
> and.css). ADP gzipping is working.
> According to the docs, it’d seem that all I need to do is enable all 
> the fastpath config stuff below, and it’d work, but that’s not doing 
> the trick.  Perhaps there is additional fast path configuring to do to 
> enable certain directories?
> According to this code by Daniel Clark, a fast path filter is needed 
> to make this work, but it seems that (a) his code may not be current 
> and (b) his filter doesn’t work because he doesn’t supply the other 
> dependent functions his code calls.
> https://www.qcode.co.uk/post/121
> I’ve looked at the fastpath.c/fastpath.tcl as the docs indicated, but 
> was not illuminated.
> I probably just don’t know where to look, but after about 2h of 
> banging on this, I thought I’d ask for help….
> -john
> ns_section"ns/fastpath"
>    # Enable cache for normal URLs. Optional, default is false.
>    ns_paramcachetrue
>    # Size of fast path cache. Optional, default is ~10M.
>    ns_paramcachemaxsize[expr 1024*10000]
>    # Largest file size allowed in cache. Optional, default is 8K
>    ns_paramcachemaxentry8192
>    # Use mmap() for cache. Optional, default is false.
>    ns_parammmaptrue
>    # Return gzip-ed variant, if available and allowed by client 
> (default false)
>    ns_paramgzip_statictrue
>    # Refresh stale .gz files on the fly using ::ns_gzipfile (default 
> false)
>    ns_paramgzip_refreshtrue
>    # Return the specified command for re-compressing when gzip file is 
> outdated
>    ns_paramgzip_cmd"/usr/bin/gzip -9"
>
> ------------------------------------------------------------------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today. http://sdm.link/xeonphi
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
WU Vienna
Institute of Information Systems and New Media
Welthandelsplatz 1, A-1020 Vienna, Austria

Re: [naviserver-devel] gzipping static content - how to do?

[Ben B. <tek...@ya...>]

Hi John,

One has to gzip the files for naviserver to serve.

 From the url you referred to:
"Once you have initially seeded the filesystem with .gz versions of 
static files.."

cd to the dir with files static files:
gzip -k *

iirc one has to restart naviserver for the compressed files to be 
registered for use.

cheers,
Ben

On 01/13/2017 09:10 PM, John from Decent Espresso wrote:
> I’m struggling a bit to get naviserver to gzip static content (.js
> and.css). ADP gzipping is working.
>
> According to the docs, it’d seem that all I need to do is enable all the
> fastpath config stuff below, and it’d work, but that’s not doing the
> trick.  Perhaps there is additional fast path configuring to do to
> enable certain directories?
>
> According to this code by Daniel Clark, a fast path filter is needed to
> make this work, but it seems that (a) his code may not be current and
> (b) his filter doesn’t work because he doesn’t supply the other
> dependent functions his code calls.
> https://www.qcode.co.uk/post/121
>
> I’ve looked at the fastpath.c/fastpath.tcl as the docs indicated, but
> was not illuminated.
>
> I probably just don’t know where to look, but after about 2h of banging
> on this, I thought I’d ask for help….
>
> -john
>
>
>
> ns_section"ns/fastpath"
>
>    # Enable cache for normal URLs. Optional, default is false.
>    ns_paramcachetrue
>
>    # Size of fast path cache. Optional, default is ~10M.
>    ns_paramcachemaxsize[expr 1024*10000]
>
>    # Largest file size allowed in cache. Optional, default is 8K
>    ns_paramcachemaxentry8192
>
>    # Use mmap() for cache. Optional, default is false.
>    ns_parammmaptrue
>
>    # Return gzip-ed variant, if available and allowed by client (default
> false)
>    ns_paramgzip_statictrue
>
>    # Refresh stale .gz files on the fly using ::ns_gzipfile (default false)
>    ns_paramgzip_refreshtrue
>
>    # Return the specified command for re-compressing when gzip file is
> outdated
>    ns_paramgzip_cmd"/usr/bin/gzip -9"
>
>
> ------------------------------------------------------------------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today. http://sdm.link/xeonphi
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>