naviserver-devel — Developer discussion

[naviserver-devel] test ns_config-7.4.1, Stack around the variable 'filter' was corrupted

[Andrew P. <at...@pi...>]

On Tue, Mar 14, 2023 at 04:44:21PM -0400, Andrew Piskorski wrote:

>   Debug Error!
>   Program:  C:\web\ns-fork-pub\naviserver\nsd\libnsd.dll
>   Run-Time Check Failure #2 - Stack around the variable 'filter' was corrupted.
> 
> I never see the usual "all.tcl" test summary output.  Maybe nsd is
> hitting the above "Debug Error!" before getting there?

On Windows, running the "ns_config.test" tests triggers that one.  All
the tests through ns_config-7.4.0, pass, then it stops with no further
output.  Yep, the next test, ns_config-7.4.1, is sufficient to trigger
the problem all by itself.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] how to check if nsd has zlib?

[Gustaf N. <ne...@wu...>]

On 15.03.23 19:50, Andrew Piskorski wrote:
> On Windows, what's the best (simple?) way to check if my nsd is
> actually linked correctly with zlib?

If zlib is not linked correctly, you would get errors during linking.

The easiest thing to test, whether compression via zlib works fine
is to test this via browser (e.g., browse the file
tests/testserver/pages/ns_adp_compress.adp)

Since the errors you get are from "nstest::http-0.9", i would
not be surprised, if this is related with the last problem you reported,
also using the old-style regression test interface.

The code in "nstest::http-0.9" is quite old, it has not been changed
since ages. Are you sure, these tests were working before? It can
also be that this the one-of-content detection was never correct
but happened to work due to different buffering in tcl/c-library/OS/....

all the best

-g

Re: [naviserver-devel] Windows test failures, http_chunked

[Gustaf N. <ne...@wu...>]

On 15.03.23 19:39, Andrew Piskorski wrote:
> But despite that change, I still get those same 4 failures on Windows.

The change your are citing has nothing to do with windows.

If i see correctly, all these tests use "nstest::http-0.9", which is the 
old-style
regression test interface using ns_sockopen and tcl/IO (fconfigure, 
read, ...).
I would expect an error there.

The newer regression test interface is based on ns_http.
The failing tests use the old interface, since one gets there the raw
output from the server, while ns_http returns the parsed output.

-gn

[naviserver-devel] how to check if nsd has zlib?

[Andrew P. <at...@pi...>]

On Windows, what's the best (simple?) way to check if my nsd is
actually linked correctly with zlib?

Reason I ask, is on Windows I was always getting warnings like this:

  Warning: init server test: compress is enabled, but no zlib support built in

I've now installed zlib-1.2.13, and am playing with the build to try
and make Naviserver use it.  That made the, "no zlib support built in"
warning go away, but I'm not sure if I'm linking zlib correctly.  I
still get the three "compress" test failures below, but I don't know
if this means my zlib support is completely missing or broken, or some
other problem.


==== compress-3.3 ns_write streaming, compressed FAILED
==== Contents of test case:

    set b [nstest::http-0.9  -http 1.0  -getbinary 1  -encoding binary  -setheaders {Accept-Encoding gzip}  -getheaders {Content-Encoding Vary}  GET /compress]
    list {*}[lrange $b 0 2] [llength [lindex $b end]] [lrange [lindex $b end] end-15 end]

---- Result was:
200 gzip Accept-Encoding 37 {ff ff 52 48 54 28 49 2d 2e e1 02 00 00 00 ff ff}
---- Result should have been (exact matching):
200 gzip Accept-Encoding 47 {02 00 00 00 ff ff 03 00 12 13 05 72 0f 00 00 00}
==== compress-3.3 FAILED


==== compress-3.4 streaming adp, compressed FAILED
==== Contents of test case:

    set b [nstest::http-0.9  -http 1.0  -getbinary 1  -encoding binary  -setheaders {Accept-Encoding gzip}  -getheaders {Content-Encoding Vary}  GET /ns_adp_compress.adp?stream=1]
    list {*}[lrange $b 0 2] [llength [lindex $b end]] [lrange [lindex $b end] end-20 end]

---- Result was:
200 gzip Accept-Encoding 30 {0a 2a c9 c8 2c 56 00 a2 44 85 92 d4 e2 12 2e 00 00 00 00 ff ff}
---- Result should have been (exact matching):
200 gzip Accept-Encoding 40 {92 d4 e2 12 2e 00 00 00 00 ff ff 03 00 12 13 05 72 0f 00 00 00}
==== compress-3.4 FAILED


==== compress-3.5 ns_write streaming + HTTP 1.1 chunking, compressed FAILED
==== Contents of test case:

    set b [nstest::http-0.9  -http 1.1  -getbinary 1  -encoding binary  -setheaders {Accept-Encoding gzip}  -getheaders {Content-Encoding Vary Transfer-Encoding Content-Length}  GET /compress]
    list {*}[lrange $b 0 3] [llength [lindex $b end]] [lrange [lindex $b end] end-30 end]

---- Result was:
200 gzip Accept-Encoding chunked 44 {2a c9 c8 2c 56 c8 2c 06 00 00 00 ff ff 0a 65 0a 52 48 54 28 49 2d 2e e1 02 00 00 00 ff ff 0a}
---- Result should have been (exact matching):
200 gzip Accept-Encoding chunked 60 {52 48 54 28 49 2d 2e e1 02 00 00 00 ff ff 0a 61 0a 03 00 12 13 05 72 0f 00 00 00 0a 30 0a 0a}
==== compress-3.5 FAILED

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] Windows test failures, http_chunked

[Andrew P. <at...@pi...>]

On Tue, Mar 14, 2023 at 04:44:21PM -0400, Andrew Piskorski wrote:
> I also see 4 "http_chunked" test failures, below:

> ==== http_chunked-1.1 ADP streaming w/chunks to HTTP/1.1 client FAILED
> ==== http_chunked-1.1 FAILED
> ==== http_chunked-1.3 ADP with longer partly-buffered response and auto-streaming FAILED
> ==== http_chunked-1.3 FAILED
> ==== http_chunked-2.1 Tcl streaming w/chunks to HTTP/1.1 client FAILED
> ==== http_chunked-2.1 FAILED
> ==== http_chunked-2.1.1 Tcl streaming multiple binary buffers w/chunks to HTTP/1.1 client FAILED
> ==== http_chunked-2.1.1 FAILED

I am now using this change:

  https://bitbucket.org/naviserver/naviserver/commits/7e65faade74d4da66dcffd0f7c271d73f748b3a2
  commit 7e65faade74d4da66dcffd0f7c271d73f748b3a2
  Author: Gustaf Neumann <ne...@wu...>
  Date:   2023-03-14 10:02:49 +0100 Tue
  ns_http: Fixed behavior of HEAD request for persistent connections.

But despite that change, I still get those same 4 failures on Windows.
Output looks the same, the output my nsd is sending ends in "01234\n",
but the test is expecting it to be "01234\n0\n".

-- 
Andrew Piskorski <at...@pi...>

[naviserver-devel] Windows test failures, http_chunked

[Andrew P. <at...@pi...>]

On Tue, Mar 14, 2023 at 02:54:54PM -0400, Andrew Piskorski wrote:

>>   [14/Mar/2023:13:17:45][16420.4aa0][-main:test-] Error: modload: Z:/src/web/ns-fork-pub/naviserver/tests/../nslog/nslog.dll: couldn't load library "Z:/src/web/ns-fork-pub/naviserver/tests/../nslog/nslog.dll": permission denied
>>   [14/Mar/2023:13:17:45][16420.4aa0][-main:test-] Fatal: modload: failed to load module 'Z:/src/web/ns-fork-pub/naviserver/tests/../nslog/nslog.dll'

> In my case, trying to run on a network drive (over Samba) seems to be
> causing the "permission denied" problem.

Embarassingly, the problem there was simply that my compiled Windows
*.dll files, located on the Samba drive, lacked execute permission.
Some Samba configurations enforce that and some don't.  With that
fixed I can actually look at real test failures.

I am still getting the pop-up I mentioned before:

  Debug Error!
  Program:  C:\web\ns-fork-pub\naviserver\nsd\libnsd.dll
  Run-Time Check Failure #2 - Stack around the variable 'filter' was corrupted.

I never see the usual "all.tcl" test summary output.  Maybe nsd is
hitting the above "Debug Error!" before getting there?

In the test output, I have lots of "compress" failures, but that's
probably because I don't have zlib installed (yet) on this machine.

I also see 4 "http_chunked" test failures, below:


==== http_chunked-1.1 ADP streaming w/chunks to HTTP/1.1 client FAILED
==== Contents of test case:

    nstest::http-0.9 -http 1.1 -setheaders {Connection keep-alive}  -getbody 1 -getheaders {Transfer-Encoding Connection Content-Length}  GET /http_chunked.adp?stream=1

---- Result was:
200 chunked keep-alive {} {a
0123456789
5
01234
}
---- Result should have been (exact matching):
200 chunked keep-alive {} {a
0123456789
5
01234
0

}
==== http_chunked-1.1 FAILED


==== http_chunked-1.3 ADP with longer partly-buffered response and auto-streaming FAILED
==== Contents of test case:

    nstest::http-0.9 -http 1.1 -setheaders {Connection keep-alive}  -getheaders {Transfer-Encoding Connection Content-Length} -getbody t  GET /http_chunked.adp?stream=0&bufsize=8

---- Result was:
200 chunked keep-alive {} {a
0123456789
5
01234
}
---- Result should have been (exact matching):
200 chunked keep-alive {} {a
0123456789
5
01234
0

}
==== http_chunked-1.3 FAILED


==== http_chunked-2.1 Tcl streaming w/chunks to HTTP/1.1 client FAILED
==== Contents of test case:

    nstest::http-0.9 -http 1.1 -getheaders {Transfer-Encoding Content-Length} -getbody 1  GET /tclchunked

---- Result was:
200 chunked {} {a
0123456789
5
01234
}
---- Result should have been (exact matching):
200 chunked {} {a
0123456789
5
01234
0

}
==== http_chunked-2.1 FAILED


==== http_chunked-2.1.1 Tcl streaming multiple binary buffers w/chunks to HTTP/1.1 client FAILED
==== Contents of test case:

    nstest::http-0.9 -http 1.1 -getheaders {Transfer-Encoding Content-Length} -getbody 1  GET /tclchunked

---- Result was:
200 chunked {} {a
0123456789
5
01234
}
---- Result should have been (exact matching):
200 chunked {} {a
0123456789
5
01234
0

}
==== http_chunked-2.1.1 FAILED

Re: [naviserver-devel] cannot run Windows tests, mysterious nslog.dll permission denied

[Andrew P. <at...@pi...>]

On Tue, Mar 14, 2023 at 01:46:34PM -0400, Andrew Piskorski wrote:

>   nmake -f Makefile.win32 _test_one
> 
> Now, I can't get that to work, and I'm stuck trying to figure out why.

In my case, trying to run on a network drive (over Samba) seems to be
causing the "permission denied" problem.  (I'm still trying to figure
out why.)

If I simply copy everything to a local Windows disk, then the tests do
run.  They eventually crash nsd with this popup error message:

  Debug Error!
  Program:  C:\web\ns-fork-pub\naviserver\nsd\libnsd.dll
  Run-Time Check Failure #2 - Stack around the variable 'filter' was corrupted.

I haven't tried to debug that yet.

-- 
Andrew Piskorski <at...@pi...>

[naviserver-devel] cannot run Windows tests, mysterious nslog.dll permission denied

[Andrew P. <at...@pi...>]

Back in 2020, we had the regression tests successfully running on
Windows, from the command line with these sorts of commands:

  nmake -f Makefile.win32 _test
  nmake -f Makefile.win32 _test_one

Now, I can't get that to work, and I'm stuck trying to figure out why.
Note, I DON'T mean that the tests are failing - they're not.  Rather,
I currently can't get the test nsd to start up and run the tests at all!

When run from nmake like the above, I get no useful output anywhere,
just a "return code '0xc0000022'" message from nmake.

If I copy that generated command line and run the it manually at the
Windows command prompt, it then DOES write to "tests/nsd.log", where I
see errors like this:

  [14/Mar/2023:13:17:45][16420.4aa0][-main:test-] Error: modload: Z:/src/web/ns-fork-pub/naviserver/tests/../nslog/nslog.dll: couldn't load library "Z:/src/web/ns-fork-pub/naviserver/tests/../nslog/nslog.dll": permission denied
  [14/Mar/2023:13:17:45][16420.4aa0][-main:test-] Fatal: modload: failed to load module 'Z:/src/web/ns-fork-pub/naviserver/tests/../nslog/nslog.dll'

Yes, it says "permission denied", but I have no idea why!  AFAICT that
nslog.dll' file *IS* readable by everybody in every other context.

Note that in that same Windows Command Prompt I can run tclsh and see
that the file IS readable:

  Z:\src\web\ns-fork-pub\naviserver> tclsh
  % info patchlevel
  8.6.12
  % file readable Z:/src/web/ns-fork-pub/naviserver/tests/../nslog/nslog.dll
  1
  % exit

I suspect I'm overlooking something.  Any debugging suggestions for me?

Btw, once I install Naviserver I can start it up using the
"simple-config.tcl" just fine, but I don't remember if it's feasible
to run the tests that way.  In the past I definitely ran them using
the build directory like the above, not after installing.

-- 
Andrew Piskorski <at...@pi...>

[naviserver-devel] LC_COLLATE, Windows, nsmain: system configuration mismatch

[Andrew P. <at...@pi...>]

Since c. 2022-04-10, code at line 614 of nsd/nsmain.c calls Ns_Fatal()
with message like this:

  OpenSSL system configuration mismatch.
  The provided locale *****1 based on the system-wide default locale is not installed on this system.

With latest Naviserver code on Windows 10, I was getting that error
every time I tried to start Naviserver.  In my string "locale *****1"
above, the "*" symbols were actually weird rectangles, completely
illegible.  The only readable part was the "1" at the end.

I set a System environment variable LC_COLLATE = en_US.UTF-8, and that
fixed it, now Naviserver actually starts!  I'm confused about why that
was necessary, but so far it seems to work.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] aead::encrypt test failures

[Gustaf N. <ne...@wu...>]

The problem with ns_crypto::aead::encrypt/decrypt test under
OpenSSL 1.1.1 (OpenSSL 1.1.1-1ubuntu2.1~18.04.21) on Ubuntu 18.04.4
is now fixed in the repositotry.

In short, the problem was that with this version of OpenSSL, setting
empty additional authenticated data (AAD) behaved differently from
other versions, namely it was clearing incorrectly the information
that the initialization vector (IV) was already set. An upgrade of
OpenSSL fixed the problem. However, with these changes, also the stock
version of OpenSSL can be used.

Fixing this was more tricky as already apprehend, but solving a riddle
is also rewarding.

all the best

-g

Re: [naviserver-devel] aead::encrypt test failures

[Gustaf N. <ne...@wu...>]

On 09.03.23 17:30, Andrew Piskorski wrote:
> In the meantime, how widely used within NaviServer is aead::encrypt?
> Is it necessary for basic serving of https pages, or just an extra API
> programmers can optionally use?  Do even the latest versions of
> OpenACS depend on it?  (In other words, I'm wondering if these two
> aead::encrypt test failures actually matter for me.)
the aead::* functions are not used anywhere in OpenACS, these
are not used for serving pages via TLS/...  or other kind of "regular"
or "internal" usage.

The only public code that i am aware of is the nswebpush module
(optional naviserver module). But of course, every tailored
OpenACS application might use this...

all the best

-g

Re: [naviserver-devel] aead::encrypt test failures

[Andrew P. <at...@pi...>]

On Thu, Mar 09, 2023 at 12:27:46PM +0100, Gustaf Neumann wrote:

> My first suspicion is the version of OpenSSL in use. OpenSSL is a moving 
> target.
> If i see correctly, there is a version "1.1.1-1ubuntu2.1~18.04.21" in 
> place for Ubuntu 18.04 will all updates. It this what you are using?

Yes, I am.  Hm, clearly the best overall approach is for me to upgrade
this server, Ubuntu 18.04.4 is very old.

In the meantime, how widely used within NaviServer is aead::encrypt?
Is it necessary for basic serving of https pages, or just an extra API
programmers can optionally use?  Do even the latest versions of
OpenACS depend on it?  (In other words, I'm wondering if these two
aead::encrypt test failures actually matter for me.)

On that old server, I'm currently using an old version of NaviServer
with code from 2020-06-15.  It had zero failed tests, and still seems
to be working fine.

Btw, these are the package versions I see on Ubuntu:

Ubuntu 18.04.4: libssl-dev version 1.1.1-1ubuntu2.1~18.04.21
Ubuntu 20.04.1: libssl-dev version 1.1.1f-1ubuntu2.17
Ubuntu 22.04.2: libssl-dev version 3.0.2-0ubuntu1.8

Ubuntu 22.04 stopped shipping OpenSSL 1.1.x entirely, and replaced it
with 3.0.2.  And it looks like the newer OpenSSL 3.x is NOT included
at all in the older 18.04 and 20.04 distributions of Ubuntu.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] aead::encrypt test failures

[Gustaf N. <ne...@wu...>]

Small update: with Ubuntu 18.04 +  "OpenSSL 1.1.1-1ubuntu2.1~18.04.21",
also older versions of naviserver do not work (went back until 4.99.20, 
jan 2021).
so, in case this is really needed, it requires detailed debugging, including
comparing the results of OpenSSL API calls.

-gn

On 09.03.23 12:27, Gustaf Neumann wrote:
>
> On 08.03.23 21:52, Andrew Piskorski wrote:
>> Building the NaviServer head (latest code from 2023-03-02), I'm
>> getting two "make test" failures, both from aead::encrypt (below).
>> Any advice for me on what the problem might be,
>
> My first suspicion is the version of OpenSSL in use. OpenSSL is a 
> moving target.
> If i see correctly, there is a version "1.1.1-1ubuntu2.1~18.04.21" in 
> place for
> Ubuntu 18.04 will all updates. It this what you are using?
>
> Options:
> - Upgrade of OpenSSL (i have just tested an installation with 18.04 + 
> OpenSSL 3)
> - Upgrade of Ubuntu (18.04 is not the youngest)
> - Downgrade of NaviServer: there were changes between NaviServer 
> 4.99.23 and .24
>    concerning support of OpenSSL 3.0 - using newer API calls - so maybe
>    these newer API calls are available in this version of OpenSSL, but
>    not fully functioning yet.
>
> ... and of course, provide a fix to "make it work" also in your 
> combination.
>
> -gn
>
>
>> or how I should
>> further track it down?  (Thanks!)
>>
>> This is on an old Ubuntu 18.04.4 LTS machine, using gcc 8.4.0, and Tcl
>> 8.6.13.  My built-from-source Tcl includes nsf 2.4.0, Thread 2.8.9,
>> tdom 0.9.3, and tcllib 1.20.
>>
>>
>> ## Excerpts from "make test" output:
>>
>> [08/Mar/2023:15:23:16][11421.7f2097bfc700][-command-] Notice: 
>> SSL_shutdown(33) has failed: error:14094123:SSL 
>> routines:ssl3_read_bytes:application data after close notify
>> ns_crypt.test
>> ns_crypto.test
>>
>> ==== aead-1.0 aead::encrypt FAILED
>> ---- Result was:
>> bytes 0 tag 32
>> ---- Result should have been (exact matching):
>> bytes 22 tag 32
>> ==== aead-1.0 FAILED
>>
>> ==== aead-1.1 aead::encrypt and decrypt FAILED
>> ---- Test generated error; Return code was: 1
>> ---- Return code should have been one of: 0 2
>> ==== aead-1.1 FAILED
>>
>> Tests ended at Wed Mar 08 15:24:13 EST 2023
>> all.tcl:    Total    1998    Passed    1971    Skipped    25 Failed    2
>> Sourced 71 Test Files.
>> Files with failing tests: ns_driver.test
>> Number of tests skipped for each constraint:
>>     19    !usingExternalToUtf
>>     2    binaryMismatch
>>     1    copyAliasBug
>>     2    knownBug
>>     1    stress
>> Makefile:236: recipe for target 'test' failed
>> make: *** [test] Error 130
>>
-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

Re: [naviserver-devel] aead::encrypt test failures

[Gustaf N. <ne...@wu...>]

On 08.03.23 21:52, Andrew Piskorski wrote:
> Building the NaviServer head (latest code from 2023-03-02), I'm
> getting two "make test" failures, both from aead::encrypt (below).
> Any advice for me on what the problem might be,

My first suspicion is the version of OpenSSL in use. OpenSSL is a moving 
target.
If i see correctly, there is a version "1.1.1-1ubuntu2.1~18.04.21" in 
place for
Ubuntu 18.04 will all updates. It this what you are using?

Options:
- Upgrade of OpenSSL (i have just tested an installation with 18.04 + 
OpenSSL 3)
- Upgrade of Ubuntu (18.04 is not the youngest)
- Downgrade of NaviServer: there were changes between NaviServer 4.99.23 
and .24
    concerning support of OpenSSL 3.0 - using newer API calls - so maybe
    these newer API calls are available in this version of OpenSSL, but
    not fully functioning yet.

... and of course, provide a fix to "make it work" also in your combination.

-gn


> or how I should
> further track it down?  (Thanks!)
>
> This is on an old Ubuntu 18.04.4 LTS machine, using gcc 8.4.0, and Tcl
> 8.6.13.  My built-from-source Tcl includes nsf 2.4.0, Thread 2.8.9,
> tdom 0.9.3, and tcllib 1.20.
>
>
> ## Excerpts from "make test" output:
>
> [08/Mar/2023:15:23:16][11421.7f2097bfc700][-command-] Notice: SSL_shutdown(33) has failed: error:14094123:SSL routines:ssl3_read_bytes:application data after close notify
> ns_crypt.test
> ns_crypto.test
>
> ==== aead-1.0 aead::encrypt FAILED
> ---- Result was:
> bytes 0 tag 32
> ---- Result should have been (exact matching):
> bytes 22 tag 32
> ==== aead-1.0 FAILED
>
> ==== aead-1.1 aead::encrypt and decrypt FAILED
> ---- Test generated error; Return code was: 1
> ---- Return code should have been one of: 0 2
> ==== aead-1.1 FAILED
>
> Tests ended at Wed Mar 08 15:24:13 EST 2023
> all.tcl:	Total	1998	Passed	1971	Skipped	25	Failed	2
> Sourced 71 Test Files.
> Files with failing tests: ns_driver.test
> Number of tests skipped for each constraint:
> 	19	!usingExternalToUtf
> 	2	binaryMismatch
> 	1	copyAliasBug
> 	2	knownBug
> 	1	stress
> Makefile:236: recipe for target 'test' failed
> make: *** [test] Error 130
>
-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

[naviserver-devel] aead::encrypt test failures

[Andrew P. <at...@pi...>]

Building the NaviServer head (latest code from 2023-03-02), I'm
getting two "make test" failures, both from aead::encrypt (below).
Any advice for me on what the problem might be, or how I should
further track it down?  (Thanks!)

This is on an old Ubuntu 18.04.4 LTS machine, using gcc 8.4.0, and Tcl
8.6.13.  My built-from-source Tcl includes nsf 2.4.0, Thread 2.8.9,
tdom 0.9.3, and tcllib 1.20.


## Excerpts from "make test" output:

[08/Mar/2023:15:23:16][11421.7f2097bfc700][-command-] Notice: SSL_shutdown(33) has failed: error:14094123:SSL routines:ssl3_read_bytes:application data after close notify
ns_crypt.test
ns_crypto.test

==== aead-1.0 aead::encrypt FAILED
---- Result was:
bytes 0 tag 32
---- Result should have been (exact matching):
bytes 22 tag 32
==== aead-1.0 FAILED

==== aead-1.1 aead::encrypt and decrypt FAILED
---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
==== aead-1.1 FAILED

Tests ended at Wed Mar 08 15:24:13 EST 2023
all.tcl:	Total	1998	Passed	1971	Skipped	25	Failed	2
Sourced 71 Test Files.
Files with failing tests: ns_driver.test
Number of tests skipped for each constraint:
	19	!usingExternalToUtf
	2	binaryMismatch
	1	copyAliasBug
	2	knownBug
	1	stress
Makefile:236: recipe for target 'test' failed
make: *** [test] Error 130

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] Ns_conn location questions

[Gustaf N. <ne...@wu...>]

On 29.01.23 19:36, Maksym Zinchenko wrote:
> About the second question, I didn't notice this before :) Thank you 
> for your information.

When you fetch a new version of the nsshell module, it supports now 
"ns_conn location" (it's an emulaton, but probably for most situations 
good enough).

all the best

-g

Re: [naviserver-devel] Ns_conn location questions

[Maksym Z. <siq...@gm...>]

Thank you, I realised that it comes from my config  < docker.
Now I pass --hostname=${HOSTNAME} to docker run and set it up in
nsd-config.tcl as


set server [ns_info hostname]
    ns_section ns/module/nssock/servers {
        ns_param default    ${server}
    }

About the second question, I didn't notice this before :) Thank you
for your information.


On Sun, Jan 29, 2023 at 3:02 PM Gustaf Neumann <ne...@wu...> wrote:

> > My first question is where this localhost comes from?
>
> I would think, this comes from your configuration file and/of from the
> request.
>
> If one starts e.g. with the sample configuration file nsd-config.tcl, one
> sees entries like:
>
>     [29/Jan/2023:16:35:24][54720.100490580][-main:default-] Notice: nssock:0: adding virtual host entry for host <localhost:8080> location: http://localhost:8080 mapped to server: default ctx 0x0
>     [29/Jan/2023:16:35:24][54720.100490580][-main:default-] Notice: nssock:0: adding virtual host entry for host <MacBook-Pro-6.local:8080> location: http://MacBook-Pro-6.local:8080 mapped to server: default ctx 0x0
>
> These log-entries are coming e.g. from the following section
>
>     ns_section ns/module/nssock/servers {
>         ns_param default    localhost
>         ns_param default    [ns_info hostname]
>     }
>
> from the configuration file that define the mapping of hostnames for the
> "default" server. When there are incoming requests following HTTP/1.1,
> these come with a "Host:" header field, which might be in your case as well
> "localhost".
>
>
> > When I try to run ns_conn location in nsshell it gives me error
> bad_option "location".
>
> This is a know limitation (see first line of "Current shortcomings" on
> https://bitbucket.org/naviserver/nsshell/src/main/
>
> The reason for this is that nsshell communicates with a "kernel" thread in
> the background that keeps the state of your nsshell session (e.g. to be
> able to obtain the variable value for later requests, if one types "set x
> 1" in nsshell). Since this kernel runs in the background, it is no
> connection thread, and has no connection information available.
>
> Theoretically, one could stretch the limits of nsshell further, but it is
> quite hard to make the background job look completely like the a connection
> thread.
>
> -gn
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] Ns_conn location questions

[Gustaf N. <ne...@wu...>]

 > My first question is where this localhost comes from?

I would think, this comes from your configuration file and/of from the 
request.

If one starts e.g. with the sample configuration file nsd-config.tcl, 
one sees entries like:

     [29/Jan/2023:16:35:24][54720.100490580][-main:default-] Notice: nssock:0: adding virtual host entry for host <localhost:8080> location:http://localhost:8080  mapped to server: default ctx 0x0
     [29/Jan/2023:16:35:24][54720.100490580][-main:default-] Notice: nssock:0: adding virtual host entry for host <MacBook-Pro-6.local:8080> location:http://MacBook-Pro-6.local:8080  mapped to server: default ctx 0x0

These log-entries are coming e.g. from the following section

     ns_section ns/module/nssock/servers {
         ns_param default    localhost
         ns_param default    [ns_info hostname]
     }

from the configuration file that define the mapping of hostnames for the 
"default" server. When there are incoming requests following HTTP/1.1, 
these come with a "Host:" header field, which might be in your case as 
well "localhost".


 > When I try to run ns_conn location in nsshell it gives me error 
bad_option "location".

This is a know limitation (see first line of "Current shortcomings" on
https://bitbucket.org/naviserver/nsshell/src/main/

The reason for this is that nsshell communicates with a "kernel" thread 
in the background that keeps the state of your nsshell session (e.g. to 
be able to obtain the variable value for later requests, if one types 
"set x 1" in nsshell). Since this kernel runs in the background, it is 
no connection thread, and has no connection information available.

Theoretically, one could stretch the limits of nsshell further, but it 
is quite hard to make the background job look completely like the a 
connection thread.

-gn

[naviserver-devel] Ns_conn location questions

[Maksym Z. <siq...@gm...>]

Hello, I have some questions about ns_conn location.
According to manual "The location is determined via the following means:

1. if *ns_locationproc
<https://naviserver.sourceforge.io/n/naviserver/files/ns_locationproc.html>*
is configured, its result is returned.
2. if virtual hosting is enabled, and the "Host:" header field is provided
and valid, it returns its content.
3. If everything above fails, it is determined by virtual hosts mapping
table (as defined in the "ns/module/nssock/servers" or
"ns/module/nsssl/servers" section in the configuration file).
4. If everything above fails, and a connection is open, it is determined by
the current socket address.
5. If everything above fails, it is determined by configuration values of
the driver."

First and second is not my case, because i'm not using virtual hosting, in
my config i have part like that:

ns_section "ns/module/nsssl" {
ns_param defaultserver $hostname
ns_param address $ip_addr
ns_param port $ssl_port
ns_param hostnamens_section "ns/module/nsssl" {
ns_param defaultserver $hostname
ns_param certificate /opt/ns/modules/nsssl/daidze.pem
ns_param address $ip_addr
ns_param port $ssl_port
ns_param hostname $hostname
...
}

So I'm assuming i'm gonna get $hostname:$ssl_port when i run [ns_conn
location]
I'm running Naviserver inside Docker container, and instead getting
localhost.

My first question is where this localhost comes from?

And second question when I try to run ns_conn location in nsshell it gives
me error bad_option "location". Actually it gives this error when I try to
run any subcommand of ns_conn inside nsshell.

Thank you

Re: [naviserver-devel] ns_adp_registerproc and attributes

[Gustaf N. <ne...@wu...>]

Dear Wolfgang,

After Maksym Zinchenko, we found as well a problem with the modified ADP 
parser (inside an embedded minified JavaScript code)

The situation is more tricky than expected, since we have
- a language mix in ADP files (Tcl, HTML, JavaScript) and special ADP 
tags and semantics
- Some of these problems are addressed by HTML, which treats <script>
    context as CDATA (e.g. tags inside JavaScript)

      <script type="text/javascript">document.write("<script>alert('Hi')</script>");</script>
      <script type="text/javascript">x = '<br>';</script>

   In addition, ADP allows as well inline code blocks

     <script type="text/javascript"> x= '<%= [expr {2 + 2}] %>';</script>

  such that CDATA is not sufficient. Some of the <script>s can be 
executed on the client side, some on the server side (runat=server), and 
ADP also supports tag substitutions in tag definitions as well

   <html<if @doc.lang@ not nil> lang="@doc.lang;literal@"</if>>
   <script type="@type@"<if @src@ not nil> src="@src@"</if>><if @content@ not nil>@content@</if></script>

These are examples from OpenACS, which as an <if> tag defined and has 
also special substitution rules. The usage of ADP tags inside the 
starting HTML tags is not really documented, but used all over.

The semantics of the ADP parser are quite complex to handle these 
situations in some meaningful way, and i would not be surprised, if some 
rough edges limit full orthogonality would pop up. However, the version 
committed today addresses all known issues, and has a substantially 
extended regression test. It required some refactoring of the ADP 
parser. The performance should be slightly faster than the classical parser.

Please test if possible,

-g

Re: [naviserver-devel] invalid command name "detach" used within example documentation

[Gustaf N. <ne...@wu...>]

Dear David,

this is a documentation bug. What i can see by digging into old 
versions, the command "detach" was removed between AOLserver 3 and 4.... 
but survived in the documentation.

Not sure, someone has an example of a server based on ns_socket, but i 
have written several years ago a (partial) implementation of an ftp 
server based on ns_connchan (see attachment). We do not have this in 
active use, but maybe you can use this as a blueprint for your project.

all the best

-gn

On 15.01.23 20:22, D.Fox wrote:
> Greetings, I was trying to setup a socket server within NaviServer.
>
> Following the example listed under ns_sockcallback within the 
> documentation. The example uses an detach procedure although when I 
> attempt to use the syntax I get an invalid command name error
>
> https://naviserver.sourceforge.io/n/naviserver/files/ns_sockcallback.html
>
> failed: invalid command name "detach"
> :    TCL LOOKUP COMMAND detach
> :    invalid command name "detach"
> :        while executing
> :    "detach $sock"
> :        (file "Init.tcl" line 25)
> :        invoked from within
> :    "source $file"
>
> If I remove the detach, it creates the socket server however this only 
> allows one connection and then refuses to initiate any other 
> connections there after.
> Any insight would be great on how to allow multiple connections.
>
> Many Thanks,
> David Fox
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

[naviserver-devel] invalid command name "detach" used within example documentation

[D.Fox <un...@cr...>]

Greetings, I was trying to setup a socket server within NaviServer. 

Following the example listed under ns_sockcallback within the documentation. The example uses an detach procedure although when I attempt to use the syntax I get an invalid command name error 

[ https://naviserver.sourceforge.io/n/naviserver/files/ns_sockcallback.html | https://naviserver.sourceforge.io/n/naviserver/files/ns_sockcallback.html ] 

failed: invalid command name "detach" 
: TCL LOOKUP COMMAND detach 
: invalid command name "detach" 
: while executing 
: "detach $sock" 
: (file "Init.tcl" line 25) 
: invoked from within 
: "source $file" 

If I remove the detach, it creates the socket server however this only allows one connection and then refuses to initiate any other connections there after. 
Any insight would be great on how to allow multiple connections. 

Many Thanks, 
David Fox 

Re: [naviserver-devel] nsshell

[Gustaf N. <ne...@wu...>]

Hi Maksym,

This was due to the change of yesterday triggered by Wolfgang's report.
NaviServer's parser handles <% ... %> blocks also as tags, but in such
cases, the quote handling should not be applied. For now, I've added a
quick fix to fall back to the old behavior in such blocks, ... and added
a test case for such condition. Probably, the code should be more
refactored, when I have some more time.

Please update, nsshell should work again as before.

all the best

-g

On 06.01.23 11:44, Maksym Zinchenko wrote:
> Hello, I'm having trouble after update naviserver with shell, it shows
...

[naviserver-devel] nsshell

[Maksym Z. <siq...@gm...>]

Hello, I'm having trouble after update naviserver with shell, it shows

>
> <% # Get hostname without protocol (i.e. "localhost") set host [lindex
> [split [ns_conn location] "/"] 2] set wsConfigPath "ns/server/[ns_info
> server]/module/websocket/shell" set shConfigPath "ns/server/[ns_info
> server]/module/nsshell" # Get WebSocket Protocol (i.e. "ws") set wsProtocol
> [expr {[ns_conn protocol] eq "http" ? "ws" : "wss"}] # Get Shell URL from
> module (i.e. "nsshell") set shellURL [ns_config $wsConfigPath urls] set
> heartBeat [ns_config $shConfigPath kernel_heartbeat 4] # Generate Base URL
> (i.e. "localhost/shell") set baseURL [string trimright $host/$shellURL] #
> Generate WebSocket URL (i.e. ws://localhost/shell/connect) set wsUri
> $wsProtocol://$baseURL/connect # Generate kernelId # Combine uuid with
> connection id and encrpyt with sha-1 set kernelID [ns_sha1 "[ns_uuid]
> [ns_conn id]"] # Remember ns_conn settings from the start of this shell #
> in a nsv variable for this kernel. foreach subcommand { acceptedcompression
> auth authpassword authuser contentfile contentlength contentsentlength
> driver files flags form headers host id isconnected location method
> outputheaders peeraddr peerport pool port protocol query partialtimes
> request server sock start timeout url urlc urlv version zipaccepted } {
> nsv_set shell_conn $kernelID,$subcommand [ns_conn $subcommand] } # # Check,
> if we are connected via the WebSocket interface or via XHR # if
> {${shellURL} ne "" && [string match ${shellURL}* [ns_conn url]]} { ns_log
> notice "nsshell uses WebSocket interface" # # Flag WebSocket usage by
> setting xhrURL empty. # set xhrURL "" # If kernelID is specified on URL,
> change kernelId to the one # specified in the URL if {"kernel" in [ns_conn
> urlv]} { set kernelID [string trim [lindex [ns_conn urlv] end]] # KernelID
> cannot empty if {$kernelID eq ""} { ns_returnredirect [ns_conn
> location]/$shellURL } } else { # If kernel isn't specified on URL, redirect
> with the generated kernelId ns_returnredirect [ns_conn
> location]/$shellURL/kernel/$kernelID } } else { ns_log notice "nsshell uses
> XHR interface" # # Flag XHR usage by setting wsUri empty. # set wsUri ""
> set xhrURL [ns_conn url] # If kernel is specified on URL, change kernelId
> to the URL one set query_kernelID [ns_queryget kernelID ""] if
> {$query_kernelID eq ""} { ns_returnredirect "[ns_conn
> url]?kernelID=$kernelID" } else { set kernelID $query_kernelID } } %>
>

In the browser window

Re: [naviserver-devel] ns_adp_registerproc and attributes

[Wolfgang W. <wol...@di...>]

Aloha!

I'll test it over the weekend, including performance tests.

Mahalo and have a nice time in Maui!

Regards,

Wolfgang

Am 05.01.23 um 07:08 schrieb Gustaf Neumann:
>
> Dear Wolfgang,
>
> The HTML Living Standard allows the greater than sign inside single or 
> double quoted
> attribute values, so i would regard the old behavior as a bug (i think 
> to remember
> that at a certain time, one had to use entities for these).
>
> With the change, all old test continue to work, although the testing set
> is not very extensive. Please test.
>
> https://bitbucket.org/naviserver/naviserver/commits/6f7b322d0f45daa2154d702ca763442bb2be9fac
>
> all the best from Maui,
>
> -gustaf
>
> On 04.01.23 15:19, Wolfgang Winkler via naviserver-devel wrote:
>>
>> Hello!
>>
>> When we try to use custom tags with attributes, we encounter a 
>> problem when passing html strings, e.g:
>>
>> proc::dummy_tag_proc {params} {
>> return[ns_set array$params]
>> }
>> ns_adp_registerscript dummy_tag ::dummy_tag_proc
>>
>> <dummy_tag title="<i class='fal fa-link'></i>">
>>
>> Outputs:
>>
>> title {"<i class='fal fa-icon'>}
>>
>> Everything after the first ">" is truncated. Is there a safe way to 
>> prevent this behaviour?
>>
>> Regards,
>>
>> Wolfgang
>>
>> -- 
>>
>> *Wolfgang Winkler*
>> Geschäftsführung
>> wol...@di...
>> mobil +43.699.19971172
>>
>> dc:*büro*
>> digital concepts Novak Winkler OG
>> Software & Design
>> Landstraße 68, 5. Stock, 4020 Linz
>> www.digital-concepts.com <http://www.digital-concepts.com>
>> tel +43.732.997117.72
>> tel +43.699.1997117.2
>>
>> Firmenbuchnummer: 192003h
>> Firmenbuchgericht: Landesgericht Linz
>>
>>
>>
>>
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> -- 
> Univ.Prof. Dr. Gustaf Neumann
> Head of the Institute of Information Systems and New Media
> of Vienna University of Economics and Business
> Program Director of MSc "Information Systems"
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz