naviserver-devel — Developer discussion

[naviserver-devel] Some questions about Docker images for Naviserver

[Brian F. <bri...@ai...>]

Hello all

We're exploring reviving the Linux version of our Naviserver product. As part of this we'd like to investigate the feasibility of using Docker, initially for ease of development, and potentially later for production systems. I have a couple of questions that would be good to get the community's responses to.

1. Is there anyone using Docker Naviserver in production? If so would you recommend it, and are there any peculiarities that we should know about?

2. There are quite a few Naviserver images on on https://hub.docker.com, but I don't see an official one. It seems from posts on OpenACS.org <https://openacs.org/xowiki/setup-with-docker-s6> that Vlad V's build has had a lot of community input, and seems to be the de facto build. Would it be worth trying to get that version tagged as an official Naviserver Docker<https://docs.docker.com/docker-hub/official_images/> build to give newcomers clarity? I must admit to some confusion regarding the difference between the S6 and the non-S6 versions - should we as a community pick one as the official one?

3. For those of us using Oracle, what would be involved in getting Oracle driver support added as a build/run option? I'd be happy to help with this in any way I can.

thanks in advance
Brian

[naviserver-devel] Support for Argon2

[Gustaf N. <ne...@wu...>]

Dear all,

For all security hungry NaviServer users:

NaviServer supports now Argon2, which is currently the best known
password hashing function (well more than this, it is a key derivation
algorithm). For details, see [1]. It may take still some time until
OpenSSL 3.2 is available in the main Linux distributions.

With this change, NaviServer provides direct support for the two most
recommended password hashing algorithms of the OWASP project [2],
namely Argon2 and scrypt, along with SCRAM-sha-256 (actually PBKDF2)
which is the most secure algorithm implemented in PostgreSQL.

All the best

-g

[1] 
https://bitbucket.org/naviserver/naviserver/commits/4d634d54b77d1ce6b61f07944871f3dcf1a330a5
[2] 
https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html#password-hashing-algorithms

Re: [naviserver-devel] nscp segfaults server when missing module/nscp/users

[Gustaf N. <ne...@wu...>]

Thanks for the report. The problem could be repeated and is fixed in the
public repository [1]. The problem was introduced in 2022-07-11, but
not included in any released version.

all the best -g

[1] 
https://bitbucket.org/naviserver/naviserver/commits/168eebbe67d0d631df5948e1647a36e6e0d9baf0

On 18.03.23 20:07, Andrew Piskorski wrote:
> Probably nscp shouldn't segfault if it's missing that section?

[naviserver-devel] nscp segfaults server when missing module/nscp/users

[Andrew P. <at...@pi...>]

Using the Naviserver head on Linux, the installed "conf/nsd-config.tcl"
and "conf/simple-config.tcl" config fieles both work fine for testing
that a newly compliled server starts up correctly.  However, simple-config.tcl
includes a commented-out line loading the nscp module, like so:

    #ns_param   nscp            nscp

If you uncomment that line, then the server crashes every time on startup.
If you ALSO copy this "module/nscp/users" section from nsd-config.tcl,
then the crash goes away and server again starts fine:

ns_section ns/server/default/module/nscp/users {
    ns_param user "::"
}

Probably nscp shouldn't segfault if it's missing that section?  Or
maybe we should either just add that section into simple-config.tcl,
or remove the commented-out nscp line entirely.

When the server crashes because of the missing nscp configuration, the
backtrace looked like this:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f4a13c2c859 in __GI_abort () at abort.c:79
#2  0x00007f4a13ea4288 in Panic () from /usr/local/pkg/nsd-20230317-1/lib/libnsd.so
#3  0x00007f4a13af358d in Tcl_PanicVA (format=<optimized out>, argList=argList@entry=0x7ffcecfc0b40) at /home/nobackup/co/tcl/tcl-core-8-6-branch/generic/tclPanic.c:99
#4  0x00007f4a13af36ff in Tcl_Panic (format=<optimized out>) at /home/nobackup/co/tcl/tcl-core-8-6-branch/generic/tclPanic.c:160
#5  0x00007f4a13f096c7 in Abort () from /usr/local/pkg/nsd-20230317-1/lib/libnsd.so
#6  <signal handler called>
#7  0x00007f4a13ebd18d in Ns_SetFindCmp () from /usr/local/pkg/nsd-20230317-1/lib/libnsd.so
#8  0x00007f4a13ebd287 in Ns_SetFind () from /usr/local/pkg/nsd-20230317-1/lib/libnsd.so
#9  0x00007f4a13ebc951 in Ns_SetUpdateSz () from /usr/local/pkg/nsd-20230317-1/lib/libnsd.so
#10 0x00007f4a13e22a6e in LoadUsers () from /home/local-20.04/pkg/nsd-20230317-1/bin/nscp.so
#11 0x00007f4a13e22eb0 in Ns_ModuleInit () from /home/local-20.04/pkg/nsd-20230317-1/bin/nscp.so
#12 0x00007f4a13ea6e61 in Ns_ModuleLoad () from /usr/local/pkg/nsd-20230317-1/lib/libnsd.so
#13 0x00007f4a13ea7141 in NsTclModuleLoadObjCmd () from /usr/local/pkg/nsd-20230317-1/lib/libnsd.so

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] test ns_config-7.4.1, Stack around the variable 'filter' was corrupted

[Andrew P. <at...@pi...>]

On Thu, Mar 16, 2023 at 02:36:27PM -0400, Andrew Piskorski wrote:
> > >   Debug Error!
> > >   Program:  C:\web\ns-fork-pub\naviserver\nsd\libnsd.dll
> > >   Run-Time Check Failure #2 - Stack around the variable 'filter' was corrupted.

> I put the same code that the ns_config-7.4.1 test runs into a simple
> *.tcl page like this:
> 
>   set xx [ns_set array [ns_configsection -filter "unread" ns/testconfig]]
>   ns_return 200 {text/plain} "Result:  $xx"

I found this helpful article on trying to find the source of stack
corruption via memory access breakpoints:
 
  https://www.timdbg.com/posts/debugger-lies-part-1/#memory-access-breakpoints

Below is my attempt to do that; I got stuck without clear results.
My memory access breakpoint fired, but the stack trace said we were
still in NsTclConfigSectionObjCmd(), seemingly still at the beginning
of that function, with just this cryptic (to me) output:

  (41d0.f7c): Break instruction exception - code 80000003 (first chance)
  libnsd!failwithmessage+0x234:
  00007ff8`c7101364 cc              int     3

I'm not very skilled at using WinDbg, so it's likely I missed or
misinterpreted something.


Here's what I did to get that far:
------------------------------------------------------------
## I downloaded and installed Windows SDK 10.0.22621:
https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/

## That gave me WinDbg installed here:
C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\windbg.exe

## Build NaviServer on Windows my usual way:
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat"
z: & cd Z:\src\web\ns-fork-pub\naviserver
nmake -f Makefile.win32 clean-core clean-mod
nmake -f Makefile.win32 all-core all-mod
# Fix permissions if necessary!  e.g.:
#   find . -type f \( -name "*.exe" -o -name "*.dll" \) -print | sudo xargs chmod 775
nmake -f Makefile.win32 _install
nmake -f Makefile.win32 _test

## Start up nsd.exe under WinDbg:
"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\windbg.exe" -g C:\P\nsd\nsd-fp_2023-03-15-2\bin\nsd.exe -t Z:/src/web/ns-fork-pub/naviserver/tests/test.nscfg

## Tell WinDbg about my Naviserver code:
.srcpath Z:\src\web\ns-fork-pub\naviserver
.sympath C:\P\nsd\nsd-fp_2023-03-15-2\lib;C:\P\Tcl-64-8.6.12\lib;SRV*C:\MyLocalSymbols*http://msdl.microsoft.com/download/symbols
!sym noisy
lml
.reload libnsd.dll
.reload nsd.exe

## Set breakpoint 0 on NsTclConfigSectionObjCmd():
bp NsTclConfigSectionObjCmd
g

## Trigger the bug.  I like to use Cygwin like so:
wget -O - -q http://localhost:8000/atp-crash.tcl

## Now in WinDbg were are in NsTclConfigSectionObjCmd().
## Show stack:
k
## Show current stack pointer:
dx @$csp
## Set memory access breakpoint 1 on the stack pointer:
ba w 8 @$csp
bl
g

## This is the cryptic output I got:
(41d0.f7c): Break instruction exception - code 80000003 (first chance)
libnsd!failwithmessage+0x234:
00007ff8`c7101364 cc              int     3

## Disable the stack pointer breakpoint:
bd 1
------------------------------------------------------------

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] test ns_config-7.4.1, Stack around the variable 'filter' was corrupted

[Andrew P. <at...@pi...>]

On Wed, Mar 15, 2023 at 03:57:40PM -0400, Andrew Piskorski wrote:
> On Tue, Mar 14, 2023 at 04:44:21PM -0400, Andrew Piskorski wrote:
> 
> >   Debug Error!
> >   Program:  C:\web\ns-fork-pub\naviserver\nsd\libnsd.dll
> >   Run-Time Check Failure #2 - Stack around the variable 'filter' was corrupted.

> On Windows, running the "ns_config.test" tests triggers that one.  All
> the tests through ns_config-7.4.0, pass, then it stops with no further
> output.  Yep, the next test, ns_config-7.4.1, is sufficient to trigger
> the problem all by itself.

I put the same code that the ns_config-7.4.1 test runs into a simple
*.tcl page like this:

  set xx [ns_set array [ns_configsection -filter "unread" ns/testconfig]]
  ns_return 200 {text/plain} "Result:  $xx"

Hitting that web page, I the ns_return never runs, because nsd.exe has
broken before getting that far.

WinDbg says the stack corruption is happening in NsTclConfigSectionObjCmd()
(in "nsd/tclconf.c"), but it doesn't notice until it gets to the end
of that function.  I think most of the work there is in Ns_ParseObjv(),
so maybe that, or something it calls, is the most likely place for
some sort of array bounds overrun to be hiding.  I don't know though,
it could be anything that NsTclConfigSectionObjCmd() calls.

I don't know how else to further track down this bug, but if anybody
has further suggestions I'm willing to try them.

-- 
Andrew Piskorski <at...@pi...>

[naviserver-devel] test ns_config-7.4.1, Stack around the variable 'filter' was corrupted

[Andrew P. <at...@pi...>]

On Tue, Mar 14, 2023 at 04:44:21PM -0400, Andrew Piskorski wrote:

>   Debug Error!
>   Program:  C:\web\ns-fork-pub\naviserver\nsd\libnsd.dll
>   Run-Time Check Failure #2 - Stack around the variable 'filter' was corrupted.
> 
> I never see the usual "all.tcl" test summary output.  Maybe nsd is
> hitting the above "Debug Error!" before getting there?

On Windows, running the "ns_config.test" tests triggers that one.  All
the tests through ns_config-7.4.0, pass, then it stops with no further
output.  Yep, the next test, ns_config-7.4.1, is sufficient to trigger
the problem all by itself.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] how to check if nsd has zlib?

[Gustaf N. <ne...@wu...>]

On 15.03.23 19:50, Andrew Piskorski wrote:
> On Windows, what's the best (simple?) way to check if my nsd is
> actually linked correctly with zlib?

If zlib is not linked correctly, you would get errors during linking.

The easiest thing to test, whether compression via zlib works fine
is to test this via browser (e.g., browse the file
tests/testserver/pages/ns_adp_compress.adp)

Since the errors you get are from "nstest::http-0.9", i would
not be surprised, if this is related with the last problem you reported,
also using the old-style regression test interface.

The code in "nstest::http-0.9" is quite old, it has not been changed
since ages. Are you sure, these tests were working before? It can
also be that this the one-of-content detection was never correct
but happened to work due to different buffering in tcl/c-library/OS/....

all the best

-g

Re: [naviserver-devel] Windows test failures, http_chunked

[Gustaf N. <ne...@wu...>]

On 15.03.23 19:39, Andrew Piskorski wrote:
> But despite that change, I still get those same 4 failures on Windows.

The change your are citing has nothing to do with windows.

If i see correctly, all these tests use "nstest::http-0.9", which is the 
old-style
regression test interface using ns_sockopen and tcl/IO (fconfigure, 
read, ...).
I would expect an error there.

The newer regression test interface is based on ns_http.
The failing tests use the old interface, since one gets there the raw
output from the server, while ns_http returns the parsed output.

-gn

[naviserver-devel] how to check if nsd has zlib?

[Andrew P. <at...@pi...>]

On Windows, what's the best (simple?) way to check if my nsd is
actually linked correctly with zlib?

Reason I ask, is on Windows I was always getting warnings like this:

  Warning: init server test: compress is enabled, but no zlib support built in

I've now installed zlib-1.2.13, and am playing with the build to try
and make Naviserver use it.  That made the, "no zlib support built in"
warning go away, but I'm not sure if I'm linking zlib correctly.  I
still get the three "compress" test failures below, but I don't know
if this means my zlib support is completely missing or broken, or some
other problem.


==== compress-3.3 ns_write streaming, compressed FAILED
==== Contents of test case:

    set b [nstest::http-0.9  -http 1.0  -getbinary 1  -encoding binary  -setheaders {Accept-Encoding gzip}  -getheaders {Content-Encoding Vary}  GET /compress]
    list {*}[lrange $b 0 2] [llength [lindex $b end]] [lrange [lindex $b end] end-15 end]

---- Result was:
200 gzip Accept-Encoding 37 {ff ff 52 48 54 28 49 2d 2e e1 02 00 00 00 ff ff}
---- Result should have been (exact matching):
200 gzip Accept-Encoding 47 {02 00 00 00 ff ff 03 00 12 13 05 72 0f 00 00 00}
==== compress-3.3 FAILED


==== compress-3.4 streaming adp, compressed FAILED
==== Contents of test case:

    set b [nstest::http-0.9  -http 1.0  -getbinary 1  -encoding binary  -setheaders {Accept-Encoding gzip}  -getheaders {Content-Encoding Vary}  GET /ns_adp_compress.adp?stream=1]
    list {*}[lrange $b 0 2] [llength [lindex $b end]] [lrange [lindex $b end] end-20 end]

---- Result was:
200 gzip Accept-Encoding 30 {0a 2a c9 c8 2c 56 00 a2 44 85 92 d4 e2 12 2e 00 00 00 00 ff ff}
---- Result should have been (exact matching):
200 gzip Accept-Encoding 40 {92 d4 e2 12 2e 00 00 00 00 ff ff 03 00 12 13 05 72 0f 00 00 00}
==== compress-3.4 FAILED


==== compress-3.5 ns_write streaming + HTTP 1.1 chunking, compressed FAILED
==== Contents of test case:

    set b [nstest::http-0.9  -http 1.1  -getbinary 1  -encoding binary  -setheaders {Accept-Encoding gzip}  -getheaders {Content-Encoding Vary Transfer-Encoding Content-Length}  GET /compress]
    list {*}[lrange $b 0 3] [llength [lindex $b end]] [lrange [lindex $b end] end-30 end]

---- Result was:
200 gzip Accept-Encoding chunked 44 {2a c9 c8 2c 56 c8 2c 06 00 00 00 ff ff 0a 65 0a 52 48 54 28 49 2d 2e e1 02 00 00 00 ff ff 0a}
---- Result should have been (exact matching):
200 gzip Accept-Encoding chunked 60 {52 48 54 28 49 2d 2e e1 02 00 00 00 ff ff 0a 61 0a 03 00 12 13 05 72 0f 00 00 00 0a 30 0a 0a}
==== compress-3.5 FAILED

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] Windows test failures, http_chunked

[Andrew P. <at...@pi...>]

On Tue, Mar 14, 2023 at 04:44:21PM -0400, Andrew Piskorski wrote:
> I also see 4 "http_chunked" test failures, below:

> ==== http_chunked-1.1 ADP streaming w/chunks to HTTP/1.1 client FAILED
> ==== http_chunked-1.1 FAILED
> ==== http_chunked-1.3 ADP with longer partly-buffered response and auto-streaming FAILED
> ==== http_chunked-1.3 FAILED
> ==== http_chunked-2.1 Tcl streaming w/chunks to HTTP/1.1 client FAILED
> ==== http_chunked-2.1 FAILED
> ==== http_chunked-2.1.1 Tcl streaming multiple binary buffers w/chunks to HTTP/1.1 client FAILED
> ==== http_chunked-2.1.1 FAILED

I am now using this change:

  https://bitbucket.org/naviserver/naviserver/commits/7e65faade74d4da66dcffd0f7c271d73f748b3a2
  commit 7e65faade74d4da66dcffd0f7c271d73f748b3a2
  Author: Gustaf Neumann <ne...@wu...>
  Date:   2023-03-14 10:02:49 +0100 Tue
  ns_http: Fixed behavior of HEAD request for persistent connections.

But despite that change, I still get those same 4 failures on Windows.
Output looks the same, the output my nsd is sending ends in "01234\n",
but the test is expecting it to be "01234\n0\n".

-- 
Andrew Piskorski <at...@pi...>

[naviserver-devel] Windows test failures, http_chunked

[Andrew P. <at...@pi...>]

On Tue, Mar 14, 2023 at 02:54:54PM -0400, Andrew Piskorski wrote:

>>   [14/Mar/2023:13:17:45][16420.4aa0][-main:test-] Error: modload: Z:/src/web/ns-fork-pub/naviserver/tests/../nslog/nslog.dll: couldn't load library "Z:/src/web/ns-fork-pub/naviserver/tests/../nslog/nslog.dll": permission denied
>>   [14/Mar/2023:13:17:45][16420.4aa0][-main:test-] Fatal: modload: failed to load module 'Z:/src/web/ns-fork-pub/naviserver/tests/../nslog/nslog.dll'

> In my case, trying to run on a network drive (over Samba) seems to be
> causing the "permission denied" problem.

Embarassingly, the problem there was simply that my compiled Windows
*.dll files, located on the Samba drive, lacked execute permission.
Some Samba configurations enforce that and some don't.  With that
fixed I can actually look at real test failures.

I am still getting the pop-up I mentioned before:

  Debug Error!
  Program:  C:\web\ns-fork-pub\naviserver\nsd\libnsd.dll
  Run-Time Check Failure #2 - Stack around the variable 'filter' was corrupted.

I never see the usual "all.tcl" test summary output.  Maybe nsd is
hitting the above "Debug Error!" before getting there?

In the test output, I have lots of "compress" failures, but that's
probably because I don't have zlib installed (yet) on this machine.

I also see 4 "http_chunked" test failures, below:


==== http_chunked-1.1 ADP streaming w/chunks to HTTP/1.1 client FAILED
==== Contents of test case:

    nstest::http-0.9 -http 1.1 -setheaders {Connection keep-alive}  -getbody 1 -getheaders {Transfer-Encoding Connection Content-Length}  GET /http_chunked.adp?stream=1

---- Result was:
200 chunked keep-alive {} {a
0123456789
5
01234
}
---- Result should have been (exact matching):
200 chunked keep-alive {} {a
0123456789
5
01234
0

}
==== http_chunked-1.1 FAILED


==== http_chunked-1.3 ADP with longer partly-buffered response and auto-streaming FAILED
==== Contents of test case:

    nstest::http-0.9 -http 1.1 -setheaders {Connection keep-alive}  -getheaders {Transfer-Encoding Connection Content-Length} -getbody t  GET /http_chunked.adp?stream=0&bufsize=8

---- Result was:
200 chunked keep-alive {} {a
0123456789
5
01234
}
---- Result should have been (exact matching):
200 chunked keep-alive {} {a
0123456789
5
01234
0

}
==== http_chunked-1.3 FAILED


==== http_chunked-2.1 Tcl streaming w/chunks to HTTP/1.1 client FAILED
==== Contents of test case:

    nstest::http-0.9 -http 1.1 -getheaders {Transfer-Encoding Content-Length} -getbody 1  GET /tclchunked

---- Result was:
200 chunked {} {a
0123456789
5
01234
}
---- Result should have been (exact matching):
200 chunked {} {a
0123456789
5
01234
0

}
==== http_chunked-2.1 FAILED


==== http_chunked-2.1.1 Tcl streaming multiple binary buffers w/chunks to HTTP/1.1 client FAILED
==== Contents of test case:

    nstest::http-0.9 -http 1.1 -getheaders {Transfer-Encoding Content-Length} -getbody 1  GET /tclchunked

---- Result was:
200 chunked {} {a
0123456789
5
01234
}
---- Result should have been (exact matching):
200 chunked {} {a
0123456789
5
01234
0

}
==== http_chunked-2.1.1 FAILED

Re: [naviserver-devel] cannot run Windows tests, mysterious nslog.dll permission denied

[Andrew P. <at...@pi...>]

On Tue, Mar 14, 2023 at 01:46:34PM -0400, Andrew Piskorski wrote:

>   nmake -f Makefile.win32 _test_one
> 
> Now, I can't get that to work, and I'm stuck trying to figure out why.

In my case, trying to run on a network drive (over Samba) seems to be
causing the "permission denied" problem.  (I'm still trying to figure
out why.)

If I simply copy everything to a local Windows disk, then the tests do
run.  They eventually crash nsd with this popup error message:

  Debug Error!
  Program:  C:\web\ns-fork-pub\naviserver\nsd\libnsd.dll
  Run-Time Check Failure #2 - Stack around the variable 'filter' was corrupted.

I haven't tried to debug that yet.

-- 
Andrew Piskorski <at...@pi...>

[naviserver-devel] cannot run Windows tests, mysterious nslog.dll permission denied

[Andrew P. <at...@pi...>]

Back in 2020, we had the regression tests successfully running on
Windows, from the command line with these sorts of commands:

  nmake -f Makefile.win32 _test
  nmake -f Makefile.win32 _test_one

Now, I can't get that to work, and I'm stuck trying to figure out why.
Note, I DON'T mean that the tests are failing - they're not.  Rather,
I currently can't get the test nsd to start up and run the tests at all!

When run from nmake like the above, I get no useful output anywhere,
just a "return code '0xc0000022'" message from nmake.

If I copy that generated command line and run the it manually at the
Windows command prompt, it then DOES write to "tests/nsd.log", where I
see errors like this:

  [14/Mar/2023:13:17:45][16420.4aa0][-main:test-] Error: modload: Z:/src/web/ns-fork-pub/naviserver/tests/../nslog/nslog.dll: couldn't load library "Z:/src/web/ns-fork-pub/naviserver/tests/../nslog/nslog.dll": permission denied
  [14/Mar/2023:13:17:45][16420.4aa0][-main:test-] Fatal: modload: failed to load module 'Z:/src/web/ns-fork-pub/naviserver/tests/../nslog/nslog.dll'

Yes, it says "permission denied", but I have no idea why!  AFAICT that
nslog.dll' file *IS* readable by everybody in every other context.

Note that in that same Windows Command Prompt I can run tclsh and see
that the file IS readable:

  Z:\src\web\ns-fork-pub\naviserver> tclsh
  % info patchlevel
  8.6.12
  % file readable Z:/src/web/ns-fork-pub/naviserver/tests/../nslog/nslog.dll
  1
  % exit

I suspect I'm overlooking something.  Any debugging suggestions for me?

Btw, once I install Naviserver I can start it up using the
"simple-config.tcl" just fine, but I don't remember if it's feasible
to run the tests that way.  In the past I definitely ran them using
the build directory like the above, not after installing.

-- 
Andrew Piskorski <at...@pi...>

[naviserver-devel] LC_COLLATE, Windows, nsmain: system configuration mismatch

[Andrew P. <at...@pi...>]

Since c. 2022-04-10, code at line 614 of nsd/nsmain.c calls Ns_Fatal()
with message like this:

  OpenSSL system configuration mismatch.
  The provided locale *****1 based on the system-wide default locale is not installed on this system.

With latest Naviserver code on Windows 10, I was getting that error
every time I tried to start Naviserver.  In my string "locale *****1"
above, the "*" symbols were actually weird rectangles, completely
illegible.  The only readable part was the "1" at the end.

I set a System environment variable LC_COLLATE = en_US.UTF-8, and that
fixed it, now Naviserver actually starts!  I'm confused about why that
was necessary, but so far it seems to work.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] aead::encrypt test failures

[Gustaf N. <ne...@wu...>]

The problem with ns_crypto::aead::encrypt/decrypt test under
OpenSSL 1.1.1 (OpenSSL 1.1.1-1ubuntu2.1~18.04.21) on Ubuntu 18.04.4
is now fixed in the repositotry.

In short, the problem was that with this version of OpenSSL, setting
empty additional authenticated data (AAD) behaved differently from
other versions, namely it was clearing incorrectly the information
that the initialization vector (IV) was already set. An upgrade of
OpenSSL fixed the problem. However, with these changes, also the stock
version of OpenSSL can be used.

Fixing this was more tricky as already apprehend, but solving a riddle
is also rewarding.

all the best

-g

Re: [naviserver-devel] aead::encrypt test failures

[Gustaf N. <ne...@wu...>]

On 09.03.23 17:30, Andrew Piskorski wrote:
> In the meantime, how widely used within NaviServer is aead::encrypt?
> Is it necessary for basic serving of https pages, or just an extra API
> programmers can optionally use?  Do even the latest versions of
> OpenACS depend on it?  (In other words, I'm wondering if these two
> aead::encrypt test failures actually matter for me.)
the aead::* functions are not used anywhere in OpenACS, these
are not used for serving pages via TLS/...  or other kind of "regular"
or "internal" usage.

The only public code that i am aware of is the nswebpush module
(optional naviserver module). But of course, every tailored
OpenACS application might use this...

all the best

-g

Re: [naviserver-devel] aead::encrypt test failures

[Andrew P. <at...@pi...>]

On Thu, Mar 09, 2023 at 12:27:46PM +0100, Gustaf Neumann wrote:

> My first suspicion is the version of OpenSSL in use. OpenSSL is a moving 
> target.
> If i see correctly, there is a version "1.1.1-1ubuntu2.1~18.04.21" in 
> place for Ubuntu 18.04 will all updates. It this what you are using?

Yes, I am.  Hm, clearly the best overall approach is for me to upgrade
this server, Ubuntu 18.04.4 is very old.

In the meantime, how widely used within NaviServer is aead::encrypt?
Is it necessary for basic serving of https pages, or just an extra API
programmers can optionally use?  Do even the latest versions of
OpenACS depend on it?  (In other words, I'm wondering if these two
aead::encrypt test failures actually matter for me.)

On that old server, I'm currently using an old version of NaviServer
with code from 2020-06-15.  It had zero failed tests, and still seems
to be working fine.

Btw, these are the package versions I see on Ubuntu:

Ubuntu 18.04.4: libssl-dev version 1.1.1-1ubuntu2.1~18.04.21
Ubuntu 20.04.1: libssl-dev version 1.1.1f-1ubuntu2.17
Ubuntu 22.04.2: libssl-dev version 3.0.2-0ubuntu1.8

Ubuntu 22.04 stopped shipping OpenSSL 1.1.x entirely, and replaced it
with 3.0.2.  And it looks like the newer OpenSSL 3.x is NOT included
at all in the older 18.04 and 20.04 distributions of Ubuntu.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] aead::encrypt test failures

[Gustaf N. <ne...@wu...>]

Small update: with Ubuntu 18.04 +  "OpenSSL 1.1.1-1ubuntu2.1~18.04.21",
also older versions of naviserver do not work (went back until 4.99.20, 
jan 2021).
so, in case this is really needed, it requires detailed debugging, including
comparing the results of OpenSSL API calls.

-gn

On 09.03.23 12:27, Gustaf Neumann wrote:
>
> On 08.03.23 21:52, Andrew Piskorski wrote:
>> Building the NaviServer head (latest code from 2023-03-02), I'm
>> getting two "make test" failures, both from aead::encrypt (below).
>> Any advice for me on what the problem might be,
>
> My first suspicion is the version of OpenSSL in use. OpenSSL is a 
> moving target.
> If i see correctly, there is a version "1.1.1-1ubuntu2.1~18.04.21" in 
> place for
> Ubuntu 18.04 will all updates. It this what you are using?
>
> Options:
> - Upgrade of OpenSSL (i have just tested an installation with 18.04 + 
> OpenSSL 3)
> - Upgrade of Ubuntu (18.04 is not the youngest)
> - Downgrade of NaviServer: there were changes between NaviServer 
> 4.99.23 and .24
>    concerning support of OpenSSL 3.0 - using newer API calls - so maybe
>    these newer API calls are available in this version of OpenSSL, but
>    not fully functioning yet.
>
> ... and of course, provide a fix to "make it work" also in your 
> combination.
>
> -gn
>
>
>> or how I should
>> further track it down?  (Thanks!)
>>
>> This is on an old Ubuntu 18.04.4 LTS machine, using gcc 8.4.0, and Tcl
>> 8.6.13.  My built-from-source Tcl includes nsf 2.4.0, Thread 2.8.9,
>> tdom 0.9.3, and tcllib 1.20.
>>
>>
>> ## Excerpts from "make test" output:
>>
>> [08/Mar/2023:15:23:16][11421.7f2097bfc700][-command-] Notice: 
>> SSL_shutdown(33) has failed: error:14094123:SSL 
>> routines:ssl3_read_bytes:application data after close notify
>> ns_crypt.test
>> ns_crypto.test
>>
>> ==== aead-1.0 aead::encrypt FAILED
>> ---- Result was:
>> bytes 0 tag 32
>> ---- Result should have been (exact matching):
>> bytes 22 tag 32
>> ==== aead-1.0 FAILED
>>
>> ==== aead-1.1 aead::encrypt and decrypt FAILED
>> ---- Test generated error; Return code was: 1
>> ---- Return code should have been one of: 0 2
>> ==== aead-1.1 FAILED
>>
>> Tests ended at Wed Mar 08 15:24:13 EST 2023
>> all.tcl:    Total    1998    Passed    1971    Skipped    25 Failed    2
>> Sourced 71 Test Files.
>> Files with failing tests: ns_driver.test
>> Number of tests skipped for each constraint:
>>     19    !usingExternalToUtf
>>     2    binaryMismatch
>>     1    copyAliasBug
>>     2    knownBug
>>     1    stress
>> Makefile:236: recipe for target 'test' failed
>> make: *** [test] Error 130
>>
-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

Re: [naviserver-devel] aead::encrypt test failures

[Gustaf N. <ne...@wu...>]

On 08.03.23 21:52, Andrew Piskorski wrote:
> Building the NaviServer head (latest code from 2023-03-02), I'm
> getting two "make test" failures, both from aead::encrypt (below).
> Any advice for me on what the problem might be,

My first suspicion is the version of OpenSSL in use. OpenSSL is a moving 
target.
If i see correctly, there is a version "1.1.1-1ubuntu2.1~18.04.21" in 
place for
Ubuntu 18.04 will all updates. It this what you are using?

Options:
- Upgrade of OpenSSL (i have just tested an installation with 18.04 + 
OpenSSL 3)
- Upgrade of Ubuntu (18.04 is not the youngest)
- Downgrade of NaviServer: there were changes between NaviServer 4.99.23 
and .24
    concerning support of OpenSSL 3.0 - using newer API calls - so maybe
    these newer API calls are available in this version of OpenSSL, but
    not fully functioning yet.

... and of course, provide a fix to "make it work" also in your combination.

-gn


> or how I should
> further track it down?  (Thanks!)
>
> This is on an old Ubuntu 18.04.4 LTS machine, using gcc 8.4.0, and Tcl
> 8.6.13.  My built-from-source Tcl includes nsf 2.4.0, Thread 2.8.9,
> tdom 0.9.3, and tcllib 1.20.
>
>
> ## Excerpts from "make test" output:
>
> [08/Mar/2023:15:23:16][11421.7f2097bfc700][-command-] Notice: SSL_shutdown(33) has failed: error:14094123:SSL routines:ssl3_read_bytes:application data after close notify
> ns_crypt.test
> ns_crypto.test
>
> ==== aead-1.0 aead::encrypt FAILED
> ---- Result was:
> bytes 0 tag 32
> ---- Result should have been (exact matching):
> bytes 22 tag 32
> ==== aead-1.0 FAILED
>
> ==== aead-1.1 aead::encrypt and decrypt FAILED
> ---- Test generated error; Return code was: 1
> ---- Return code should have been one of: 0 2
> ==== aead-1.1 FAILED
>
> Tests ended at Wed Mar 08 15:24:13 EST 2023
> all.tcl:	Total	1998	Passed	1971	Skipped	25	Failed	2
> Sourced 71 Test Files.
> Files with failing tests: ns_driver.test
> Number of tests skipped for each constraint:
> 	19	!usingExternalToUtf
> 	2	binaryMismatch
> 	1	copyAliasBug
> 	2	knownBug
> 	1	stress
> Makefile:236: recipe for target 'test' failed
> make: *** [test] Error 130
>
-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

[naviserver-devel] aead::encrypt test failures

[Andrew P. <at...@pi...>]

Building the NaviServer head (latest code from 2023-03-02), I'm
getting two "make test" failures, both from aead::encrypt (below).
Any advice for me on what the problem might be, or how I should
further track it down?  (Thanks!)

This is on an old Ubuntu 18.04.4 LTS machine, using gcc 8.4.0, and Tcl
8.6.13.  My built-from-source Tcl includes nsf 2.4.0, Thread 2.8.9,
tdom 0.9.3, and tcllib 1.20.


## Excerpts from "make test" output:

[08/Mar/2023:15:23:16][11421.7f2097bfc700][-command-] Notice: SSL_shutdown(33) has failed: error:14094123:SSL routines:ssl3_read_bytes:application data after close notify
ns_crypt.test
ns_crypto.test

==== aead-1.0 aead::encrypt FAILED
---- Result was:
bytes 0 tag 32
---- Result should have been (exact matching):
bytes 22 tag 32
==== aead-1.0 FAILED

==== aead-1.1 aead::encrypt and decrypt FAILED
---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
==== aead-1.1 FAILED

Tests ended at Wed Mar 08 15:24:13 EST 2023
all.tcl:	Total	1998	Passed	1971	Skipped	25	Failed	2
Sourced 71 Test Files.
Files with failing tests: ns_driver.test
Number of tests skipped for each constraint:
	19	!usingExternalToUtf
	2	binaryMismatch
	1	copyAliasBug
	2	knownBug
	1	stress
Makefile:236: recipe for target 'test' failed
make: *** [test] Error 130

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] Ns_conn location questions

[Gustaf N. <ne...@wu...>]

On 29.01.23 19:36, Maksym Zinchenko wrote:
> About the second question, I didn't notice this before :) Thank you 
> for your information.

When you fetch a new version of the nsshell module, it supports now 
"ns_conn location" (it's an emulaton, but probably for most situations 
good enough).

all the best

-g

Re: [naviserver-devel] Ns_conn location questions

[Maksym Z. <siq...@gm...>]

Thank you, I realised that it comes from my config  < docker.
Now I pass --hostname=${HOSTNAME} to docker run and set it up in
nsd-config.tcl as


set server [ns_info hostname]
    ns_section ns/module/nssock/servers {
        ns_param default    ${server}
    }

About the second question, I didn't notice this before :) Thank you
for your information.


On Sun, Jan 29, 2023 at 3:02 PM Gustaf Neumann <ne...@wu...> wrote:

> > My first question is where this localhost comes from?
>
> I would think, this comes from your configuration file and/of from the
> request.
>
> If one starts e.g. with the sample configuration file nsd-config.tcl, one
> sees entries like:
>
>     [29/Jan/2023:16:35:24][54720.100490580][-main:default-] Notice: nssock:0: adding virtual host entry for host <localhost:8080> location: http://localhost:8080 mapped to server: default ctx 0x0
>     [29/Jan/2023:16:35:24][54720.100490580][-main:default-] Notice: nssock:0: adding virtual host entry for host <MacBook-Pro-6.local:8080> location: http://MacBook-Pro-6.local:8080 mapped to server: default ctx 0x0
>
> These log-entries are coming e.g. from the following section
>
>     ns_section ns/module/nssock/servers {
>         ns_param default    localhost
>         ns_param default    [ns_info hostname]
>     }
>
> from the configuration file that define the mapping of hostnames for the
> "default" server. When there are incoming requests following HTTP/1.1,
> these come with a "Host:" header field, which might be in your case as well
> "localhost".
>
>
> > When I try to run ns_conn location in nsshell it gives me error
> bad_option "location".
>
> This is a know limitation (see first line of "Current shortcomings" on
> https://bitbucket.org/naviserver/nsshell/src/main/
>
> The reason for this is that nsshell communicates with a "kernel" thread in
> the background that keeps the state of your nsshell session (e.g. to be
> able to obtain the variable value for later requests, if one types "set x
> 1" in nsshell). Since this kernel runs in the background, it is no
> connection thread, and has no connection information available.
>
> Theoretically, one could stretch the limits of nsshell further, but it is
> quite hard to make the background job look completely like the a connection
> thread.
>
> -gn
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] Ns_conn location questions

[Gustaf N. <ne...@wu...>]

 > My first question is where this localhost comes from?

I would think, this comes from your configuration file and/of from the 
request.

If one starts e.g. with the sample configuration file nsd-config.tcl, 
one sees entries like:

     [29/Jan/2023:16:35:24][54720.100490580][-main:default-] Notice: nssock:0: adding virtual host entry for host <localhost:8080> location:http://localhost:8080  mapped to server: default ctx 0x0
     [29/Jan/2023:16:35:24][54720.100490580][-main:default-] Notice: nssock:0: adding virtual host entry for host <MacBook-Pro-6.local:8080> location:http://MacBook-Pro-6.local:8080  mapped to server: default ctx 0x0

These log-entries are coming e.g. from the following section

     ns_section ns/module/nssock/servers {
         ns_param default    localhost
         ns_param default    [ns_info hostname]
     }

from the configuration file that define the mapping of hostnames for the 
"default" server. When there are incoming requests following HTTP/1.1, 
these come with a "Host:" header field, which might be in your case as 
well "localhost".


 > When I try to run ns_conn location in nsshell it gives me error 
bad_option "location".

This is a know limitation (see first line of "Current shortcomings" on
https://bitbucket.org/naviserver/nsshell/src/main/

The reason for this is that nsshell communicates with a "kernel" thread 
in the background that keeps the state of your nsshell session (e.g. to 
be able to obtain the variable value for later requests, if one types 
"set x 1" in nsshell). Since this kernel runs in the background, it is 
no connection thread, and has no connection information available.

Theoretically, one could stretch the limits of nsshell further, but it 
is quite hard to make the background job look completely like the a 
connection thread.

-gn

[naviserver-devel] Ns_conn location questions

[Maksym Z. <siq...@gm...>]

Hello, I have some questions about ns_conn location.
According to manual "The location is determined via the following means:

1. if *ns_locationproc
<https://naviserver.sourceforge.io/n/naviserver/files/ns_locationproc.html>*
is configured, its result is returned.
2. if virtual hosting is enabled, and the "Host:" header field is provided
and valid, it returns its content.
3. If everything above fails, it is determined by virtual hosts mapping
table (as defined in the "ns/module/nssock/servers" or
"ns/module/nsssl/servers" section in the configuration file).
4. If everything above fails, and a connection is open, it is determined by
the current socket address.
5. If everything above fails, it is determined by configuration values of
the driver."

First and second is not my case, because i'm not using virtual hosting, in
my config i have part like that:

ns_section "ns/module/nsssl" {
ns_param defaultserver $hostname
ns_param address $ip_addr
ns_param port $ssl_port
ns_param hostnamens_section "ns/module/nsssl" {
ns_param defaultserver $hostname
ns_param certificate /opt/ns/modules/nsssl/daidze.pem
ns_param address $ip_addr
ns_param port $ssl_port
ns_param hostname $hostname
...
}

So I'm assuming i'm gonna get $hostname:$ssl_port when i run [ns_conn
location]
I'm running Naviserver inside Docker container, and instead getting
localhost.

My first question is where this localhost comes from?

And second question when I try to run ns_conn location in nsshell it gives
me error bad_option "location". Actually it gives this error when I try to
run any subcommand of ns_conn inside nsshell.

Thank you