naviserver-devel — Developer discussion

Re: [naviserver-devel] Problem with new pthread code: Resource temporarily unavailable

[Gustaf N. <ne...@wu...>]

Just a quick followup: I've tested what I've suggested below with Debian 
buster (== Debian stable)

     root@buster# uname -a
     Linux buster 4.19.0-13-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28) x86_64 GNU/Linux

     root@buster# lsb_release -d
     Description:    Debian GNU/Linux 10 (buster)

and everything looks ok

all the best

-gn

On 27.04.21 11:52, Gustaf Neumann wrote:
> Dear Oscar,
>
> Can you make please a quick check:
> when you run [1] the installation says at the end:
>
>    You can now run plain NaviServer by typing the following command:
>    sudo /usr/local/ns/bin/nsd -f -u nsadmin -g nsadmin -t 
> /usr/local/ns/conf/nsd-config.tcl
>
> When you start this instance of naviserver, does it work?
> If you, we know the libraries on your machine are fine, the problem is
> either your installation process or the used configuration file.
>
> all the best
>
> -gn
>
> [1] https://github.com/gustafn/install-ns

Re: [naviserver-devel] Problem with new pthread code: Resource temporarily unavailable

[Gustaf N. <ne...@wu...>]

Dear Oscar,

Can you make please a quick check:
when you run [1] the installation says at the end:

    You can now run plain NaviServer by typing the following command:
    sudo /usr/local/ns/bin/nsd -f -u nsadmin -g nsadmin -t 
/usr/local/ns/conf/nsd-config.tcl

When you start this instance of naviserver, does it work?
If you, we know the libraries on your machine are fine, the problem is
either your installation process or the used configuration file.

all the best

-gn

[1] https://github.com/gustafn/install-ns

[naviserver-devel] Problem with new pthread code: Resource temporarily unavailable

[Oscar R. F. <oro...@vr...>]

Dear all,

First I'd like to  thank all devs for your great work in naviserver.

I had been using aolserver 4.5 for one legacy project for a very long
time. A couple of years ago I replaced it for naviserver and has been
very happy for the great improvement in stability and
performance since then. Last version that is working for me is 4.19

I built the current released version and am stuck with a problem that
happens when openning the first connection. The server initializes
all right but on first connection it crashes.

The last lines of the logs are (last three are triggered with first
connection):

[27/Apr/2021:00:29:28][22153.7f3b49ffb700][-
conn:myserver:default:8:0-] Notice: update interpreter myserver to
epoch 1, trace none, time 0.055877 secs concurrent 2
[27/Apr/2021:00:29:28][22153.7f3b4affd700][-
conn:myserver:default:7:0-] Notice: update interpreter myserver to
epoch 1, trace none, time 0.056772 secs concurrent 2
[27/Apr/2021:00:29:28][22153.7f3b49ffb700][-
conn:myserver:default:8:0-] Notice: thread initialized (0.094352
secs)
[27/Apr/2021:00:29:28][22153.7f3b4affd700][-
conn:myserver:default:7:0-] Notice: thread initialized (0.094376
secs)
[27/Apr/2021:00:29:37][22153.7f3b497fa700][-driver:nssock:0-] Debug:
nssock(5): option TCP_NODELAY activated
[27/Apr/2021:00:29:37][22153.7f3b497fa700][-driver:nssock:0-] Debug:
=== length < avail (length 0, avail 0) tfd 0 tfile (nil)
chunkStartOff
0                                                                    
                                                              
[27/Apr/2021:00:29:37][22153.7f3b49ffb700][-
conn:myserver:default:8:0-] Fatal: nsthreads:
pthread_rwlock_tryrdlock failed in Ns_RWLockRdLock: Resource
temporarily unavailable   
  
AFAIK 4.19 was the last version without the new pthreads code so it
may be related.

Wierdly enough nsthreadtest program seems to work all right.

I've built it under debian stable with a backported tcl 8.6.11.

¿Can it be a configuration problem? ¿Any tips on how to debug this?

Any help is very much appreciated.

Best regards.
Óscar

[naviserver-devel] NaviServer 4.99.20 available

[Gustaf N. <ne...@wu...>]

Dear all,

The release of NaviServer 4.99.21 is now available on sourceforge [1].

This is essentially a bug-fix release and cleans everything up, what was
reported after the larger release of 4.99.20 as an issue. I will make as 
well
as joint release annouce of 4.99.20 and .21 at the Tcl forum.

The code is tested at least with Ubuntu 20.04, macOS 10.14.6, 11.2.3,
FreeBSD-12.2, OpenBSD 6.8. Below is a summary of changes since the
last release.

All the best
-gustaf
[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.21/

=======================================
NaviServer 4.99.21, released 2021-04-09
=======================================

  64 files changed, 939 insertions(+), 536 deletions(-)

New Features:
-------------

  - Logging improvements: Added parameter "logsec" and "logthread" for
    section "ns/parameter" to make logging of full timestamps and
    thread info optional. This give more control to a developer to
    shorten the log entries e.g. on development instances by excluding
    certain values.

  - Made "ns_http ... -decompress ... "obsolete in favor of new option
    "-raw".  This fixes as well a crash.


Bug Fixes:
----------

  - Fixed computation of "location" of a request for cases where the
    network driver is not installed globally. The value of [ns_conn
    location] (and the redirect target) is often, but not always
    determined by the host header field as used for virtual servers.

  - Fixed handling of sendBodyLength in ns_http

  - Fixed handling of user-ids in nscp.

  - Fixed potential segmentation violation in debug output, when (a) the
    network driver is installed locally, and (b) the content of host
    header field cannot be located in virtual server map (fallback to
    validation via driver data).

  - Fixed potential segmentation violation when nsd is called with '-c'
    and '-t' and the config file contains no "home" parameter in the
    config file. The fix simplifies the logic, and performs same
    configuration of nsconf.home whenever "-c" is used in the command
    line.

  - Increase sysadmin-friendliness of OCSP handling in error cases: In
    case OCSP (Online Certificate Status Protocol) is activated and the
    connection to the AIA-server is blocked, do not reject all traffic
    but behave like in a configuration without having OCSP activated,
    and complain the system log.  Background: Sometimes requests to the
    let's encrypt AIA server seems to be blocked by
    firewalls. Previously, in such stituations, all traffic was
    blocked.

  - Relax bailing out on malloc(0), since at least the OpenSSL 3a*
    versions use this frequently on certain configurations. Calls of
    malloc(0) are essentially useless and dangerous, but we cannot
    avoid these.

Performance improvements:
-------------------------

  - Added support for TCP_NODELAY for https sockets via configuration
    option "nodelay" in the nsssl driver section. For whatever reasons,
    NaviServer handled so far this option. The option can reduce the
    time of connection calls via https (clearly measurable in benchmark
    setups).


Documentation improvements:
---------------------------

  - improved the following man pages

         doc/src/manual/admin-config.man
         doc/src/manual/admin-tuning.man
         doc/src/manual/main-history.man
         doc/src/naviserver/ns_http.man
         doc/src/naviserver/ns_http.man
         doc/src/naviserver/ns_write.man
         nsdb/doc/mann/ns_db.man

Configuration Changes:
----------------------

  - New options "logsec" and "logthread" for section "ns/parameter"
    (see above)


Code Changes:
-------------
  - Extended regression test to cover e.g. the error cases from above.

  - Code Cleanup
     . Removed obsolete and deprecated functions ctime_r()
       and asctime_r() in favor of strftime().

     . Removed calls to deprecated Tcl functions

     - Don't pass (implementation-defined) NULL as last argument of a
       variadic function.

     - Usual cleanup, such as reduced implicit type conversions,
       removed uselass casts, removed dead assignments

  - Improve comments, fixed typos

[naviserver-devel] Towards NaviServer 4.99.21

[Gustaf N. <ne...@wu...>]

Dear all,

on sourceforge is a release candidate for NaviServer 4.99.21 [1].

This is essentially a bug-fix release and cleans everything up, what was
reported after the larger release of 4.99.20 as an issue. The in-depth
look of the changes of oleg (as reported before the release of 4.99.20)
are still on our agenda.

I have tested the code with Ubuntu 20.04, macOS 10.14.6, FreeBSD-12.2, 
OpenBSD 6.8.
The code is already running on openacs.org.

Below is a preliminary summary of changes. Please test if possible.
The release should be in about one week.

-gustaf
[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.21/

=======================================
NaviServer 4.99.21, released 2021-01-XX
=======================================

  59 files changed, 808 insertions(+), 499 deletions(-)

New Features:
-------------

  - Logging improvements: Added parameter "logsec" and "logthread" for
    section "ns/parameter" to make logging of full timestamps and
    thread info optional. This give more control to a developer to
    shorten the log entries e.g. on development instances by excluding
    certain values.

  - Made "ns_http ... -decompress ... "obsolete in favour of new option
    "-raw".  This fixes as well a crash.


Bug Fixes:
----------

  - Fixed computation of "location" of a request for cases, where the
    network driver is not installed globally. The value of [ns_conn
    location] (and the redirect target) is often, but not always
    determined by the host header field as used for virtual servers.

  - Fixed handling of sendBodyLength in ns_http

  - Fixed handling of user-ids in nscp.

  - Fixed potential segmentation violation in debug output, when (a) the
    network driver is installed locally, and (b) the content of host
    header field cannot be located in virtual server map (fallback to
    validation via driver data).

  - Fixed potential segmentation violation when nsd is called with '-c'
    and '-t' and the config file contains no "home" parameter in the
    config file. The fix simplifies the logic, and performs same
    configuration of nsconf.home whenever "-c" is used in the command
    line.

  - Increase sysadmin-friendliness of OCSP handling in error cases: In
    case OCSP (Online Certificate Status Protocol) is activated and the
    connection to the AIA-server is blocked, do not reject all traffic
    but behave like in a configuration without having OCSP activated,
    and complain the system log.  Background: Sometimes requests to the
    let's encrypt AIA server seems to be blocked by
    firewalls. Previously, in such stituations, all traffic was
    blocked.

  - Relax bailing out on malloc(0), since at least the OpenSSL 3a*
    versions use this frequently on certain configurations. Calls of
    malloc(0) are essentially useless and dangerous, but we cannot
    avoid these.

Performance improvements:
-------------------------

  - Added support for TCP_NODELAY for https sockets via configuration
    option "nodelay" in the nsssl driver section. For whatever reasons,
    NaviServer handled so far this option. The option can reduce the
    time of connection calls via https (clearly measurable in benchmark
    setups).


Documentation improvements:
---------------------------

  - improved the following man pages

         doc/src/manual/admin-config.man
         doc/src/manual/admin-tuning.man
         doc/src/manual/main-history.man
         doc/src/naviserver/ns_http.man
         doc/src/naviserver/ns_http.man
         doc/src/naviserver/ns_write.man
         nsdb/doc/mann/ns_db.man

Configuration Changes:
----------------------

  - New options "logsec" and "logthread" for section "ns/parameter"
    (see above)


Code Changes:
-------------
  - Extended regression test to cover e.g. the error cases from above.

  - Code Cleanup
     . Removed obsolete and deprecated functions ctime_r()
       and asctime_r() in favor of strftime().

     . Removed calls to deprecated Tcl functions

     - Don't pass (implementation-defined) NULL as last argument of a
       variadic function.

     - Usual cleanup, such as reduced implicit type conversions,
       removed uselass casts, removed dead assignments

  - Improve comments, fixed typos

Re: [naviserver-devel] Recommended approach to building Naviserver on Windows?

[Brian F. <Bri...@qu...>]

Many thanks to both Andrew and Maksym for such detailed replies - really helpful!

It seems that Visual Studio is the most popular way to go (at least from my sample size of N=2). Would be good to hear from Msys + Mingw/gcc people too.

I promise we will document our process in tedious detail once we have something running.

thanks again
Brian

________________________________
From: Andrew Piskorski <at...@pi...>
Sent: Thursday 4 March 2021 15:44
To: nav...@li... <nav...@li...>
Subject: Re: [naviserver-devel] Recommended approach to building Naviserver on Windows?

On Thu, Mar 04, 2021 at 10:24:42AM +0000, Brian Fenton wrote:

> I took a look at the install documentation here https://bitbucket.org/naviserver/naviserver/src/master/ and I noticed that there are 3 approaches described for compiling and installing on Windows.
> Which one of the 3 is the preferred approach within the community?

The only approach I have extensive experience with is using the
Microsoft C compiler natively on Windows.  That works.

I installed "Visual Studio 2019 Community Edition 16.5", and haven't
had any reason to look for a newer version yet.  In the installer, I
checked only the "Desktop development with C++" Workload, and
otherwise left everything else set to defaults.

Here are my notes to Compile and Install NaviServer on Windows, using
the command-line MSVCC compiler:  (I believe this info was up-to-date
around 2020-07, using the NaviServer head code from c. 2020-06-14.)

1. To compile we need "include/nsversion.h", which is auto-generated
by "include/nsversion.h.in", so ou have two choices:

Either (bizarrely), run configure on Linux before building on Windows!
Surprisingly, this works and is how I did it for a long time.
Or if you don't like that, or don't have Linux available, use the
"win32-util/configure.tcl" script instead.  Actually you don't need to
run it yourself at all, as "Makefile.win32" does it for you.

2. If you already built Linux NaviServer in that same directory, run
make clean there to remove the Linux object files.  (TODO: It would be
so much nicer to have separate Build-* subdirectories...)

3. As the instructions in "naviserver/Makefile.win32" say, make sure
you edit "include/Makefile.build, changing the "Makefile.module" line
to say "Makefile.win32" instead!  If you do not you will get cryptic
nmake errors.

4. In a Windows Command Prompt window, Build NaviServer like this, for 64-bit:

  "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat"
  z: & cd Z:\src\web\ns-fork-myfork\naviserver
  nmake -f Makefile.win32  clean-core clean-mod  all-core all-mod

5. Again from the Command Prompt, run this little script to properly
copy all the files to their installed locations:

  cd Z:\src\web\ns-fork-myfork\naviserver\
  tclsh ./win32-util/install-nsd.tcl -i

6. Test plain NaviServer in the foreground, e.g. like so:

  C:\web\nsd4-myfork-20200411-1\bin\nsd.exe -t
  C:\web\nsd4-myfork-20200411-1\conf\simple-config.tcl -f

Note that by default the "install-nsd.tcl" script above does NOT
overwrite the installed binaries, instead it installs to a similar
directory alongside it, with a timestamp in the name, e.g.,
"nsd4-myfork-20040927T190908\".  You are free to rename that directory
later to whatever you want (because nothing inside it embeds the
directory name).

You can also use the "install-nsd.tcl" script's "-f" and "-t" switches
to control what directory the install script copies from and to, but
normally the defaults are what you want.  If run without any
arguments, "install-nsd.tcl" will show you its defaults.

--
Andrew Piskorski <at...@pi...>


_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Recommended approach to building Naviserver on Windows?

[Maksym Z. <siq...@gm...>]

As I remember naviserver needs zlib also, to support compression. I had
some issues with that too, I used https://github.com/kiyolee/zlib-win-build
this guy project to make it working.
Cheers

On Thu, Mar 4, 2021 at 4:05 PM Maksym Zinchenko <siq...@gm...> wrote:

> I did something like:
> Compile Tcl 8.6.11
> ----------------------
>
> 1. Download sources and extract to D:\NSWIN\SRC\tcl8.6.11
> 2. Run Visual Studio "x64 Native Tools Command Prompt for VS 2019" as
> Administrator
> 3. Change directory to D:\NSWIN\SRC\tcl8.6.11\win
> 4. nmake -f makefile.vc INSTALLDIR=C:\NS\Tcl8.6.11
> 5. nmake -f makefile.vc install INSTALLDIR=C:\NS\Tcl8.6.11
>
> Compile Tk 8.6.11
> ----------------------
> 1. Download sources and extract to D:\NSWIN\SRC\tk8.6.11
> 2. Run Visual Studio "x64 Native Tools Command Prompt for VS 2019" as
> Administrator
> 3. Change directory to D:\NSWIN\SRC\tk8.6.11\win
> 4. nmake -f makefile.vc INSTALLDIR=C:\NS\Tcl8.6.11
> 5. nmake -f makefile.vc install INSTALLDIR=C:\NS\Tcl8.6.11
>
> OpenSSL
> ----------------------
> 1. Download sources and extract to D:\NSWIN\SRC\openssl-master
> 2. Download and install Strawberry Perl (dont use ActivePerl, something
> wrong with that)
> 3. Download and install NASM
> 4. Add Perl and NASM to the windows Path in Windows System Properties,
> reboot to be sure :)
> 5. Run Visual Studio "x64 Native Tools Command Prompt for VS 2019" as
> Administrator
> 6. Change directory to D:\NSWIN\SRC\openssl-master
> 7. Install some Perl dependencies:
> a) cpan -i Text::Template
> b) cpan -i Test::More
> 8. perl Configure VC-WIN64A --prefix=C:\NS\OpenSSL
> --openssldir=C:\NS\OpenSSL\SSL
> 9. nmake (and go grab some coffee)
> 10. nmake test (More Coffee!!)
> 11. nmake install_sw
>
> After that downloaded naviserver source, edit Makefile.win32,
> include\Makefile.win32, changed path to tcl and openssl and compiled with
> nmake
>
> On Thu, Mar 4, 2021 at 2:56 PM Andrew Piskorski <at...@pi...> wrote:
>
>>
>> The Tcl version should not really matter, I just install whatever the
>> latest stable ActiveTcl is for Windows.  I believe the latest I've
>> used was 8.6.x, which I installed into "C:\P\Tcl-64-8.6\".  It should
>> also work to compile Tcl yourself on Windows, but I haven't tried that
>> in many years.
>>
>>
>> For Oracle on Windows, I installed the Oracle "Instant Client" version
>> 19.6.0.0.0 a while back.  This was 4 separate zip files:
>>
>>   instantclient-basiclite-windows.x64-19.6.0.0.0dbru.zip
>>   instantclient-sdk-windows.x64-19.6.0.0.0dbru.zip
>>   instantclient-sqlplus-windows.x64-19.6.0.0.0dbru.zip
>>   instantclient-tools-windows.x64-19.6.0.0.0dbru.zip
>>
>> To install, I simply unzipped all of those into a tmp directory, which
>> created an "instantclient_19_6" directory.  Then I copied the whole
>> "instantclient_19_6" directory into "C:\P\oracle\".
>> The "nsoracle/Makefile.win32" has this line to use it:
>>
>>   ORACLE_HOME = C:\P\oracle\instantclient_19_6
>>
>>
>> For OpenSSL, back in Windows XP days I did compile it on Windows, and
>> as far as I recall that worked, although the compile was VERY slow,
>> and I had to install various support tools first (NASM Assembler,
>> etc.).  More recently I've only installed binary OpenSSL builds on
>> Windows.
>>
>> I've used this OpenSSL for Windows:
>>
>>   http://www.slproweb.com/products/Win32OpenSSL.html
>>
>> There are also binaries for the older 1.0.2 OpenSSL line, here, but I
>> don't think I ever actually used them:
>>
>>   https://indy.fulgan.com/SSL/
>>
>> See also OpenSSL's own list of Windows Binary Builds:
>>
>>   https://wiki.openssl.org/index.php/Binaries
>>
>> Old Win32OpenSSL instructions said to install the Visual C++ 2008
>> Redistributable Package via its "vcredist_x64.exe" and
>> "vcredist_x86.exe" web installers:
>>
>>   x64: http://www.microsoft.com/en-us/download/details.aspx?id=15336
>>   x86: http://www.microsoft.com/en-us/download/details.aspx?id=29
>>
>> But of course these days should probably be using the 64-bit "Visual
>> C++ 2017 Redistributable" or something even newer instead.  (Read
>> their current instuctions.)
>>
>> Around 2019-01, I installed the (at the time) latest
>> "Win64OpenSSL-1_1_1a.exe" and "Win32OpenSSL-1_1_1a.exe" into
>> "C:\P\OpenSSL-Win64\".  When it asked my where to "Copy OpenSSL DLLs
>> to:" I always pick "The OpenSSL binaries (/bin) directory", NOT the
>> default "Windows system directory".
>>
>> --
>> Andrew Piskorski <at...@pi...>
>>
>>
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>>
>

Re: [naviserver-devel] Recommended approach to building Naviserver on Windows?

[Maksym Z. <siq...@gm...>]

I did something like:
Compile Tcl 8.6.11
----------------------

1. Download sources and extract to D:\NSWIN\SRC\tcl8.6.11
2. Run Visual Studio "x64 Native Tools Command Prompt for VS 2019" as
Administrator
3. Change directory to D:\NSWIN\SRC\tcl8.6.11\win
4. nmake -f makefile.vc INSTALLDIR=C:\NS\Tcl8.6.11
5. nmake -f makefile.vc install INSTALLDIR=C:\NS\Tcl8.6.11

Compile Tk 8.6.11
----------------------
1. Download sources and extract to D:\NSWIN\SRC\tk8.6.11
2. Run Visual Studio "x64 Native Tools Command Prompt for VS 2019" as
Administrator
3. Change directory to D:\NSWIN\SRC\tk8.6.11\win
4. nmake -f makefile.vc INSTALLDIR=C:\NS\Tcl8.6.11
5. nmake -f makefile.vc install INSTALLDIR=C:\NS\Tcl8.6.11

OpenSSL
----------------------
1. Download sources and extract to D:\NSWIN\SRC\openssl-master
2. Download and install Strawberry Perl (dont use ActivePerl, something
wrong with that)
3. Download and install NASM
4. Add Perl and NASM to the windows Path in Windows System Properties,
reboot to be sure :)
5. Run Visual Studio "x64 Native Tools Command Prompt for VS 2019" as
Administrator
6. Change directory to D:\NSWIN\SRC\openssl-master
7. Install some Perl dependencies:
a) cpan -i Text::Template
b) cpan -i Test::More
8. perl Configure VC-WIN64A --prefix=C:\NS\OpenSSL
--openssldir=C:\NS\OpenSSL\SSL
9. nmake (and go grab some coffee)
10. nmake test (More Coffee!!)
11. nmake install_sw

After that downloaded naviserver source, edit Makefile.win32,
include\Makefile.win32, changed path to tcl and openssl and compiled with
nmake

On Thu, Mar 4, 2021 at 2:56 PM Andrew Piskorski <at...@pi...> wrote:

>
> The Tcl version should not really matter, I just install whatever the
> latest stable ActiveTcl is for Windows.  I believe the latest I've
> used was 8.6.x, which I installed into "C:\P\Tcl-64-8.6\".  It should
> also work to compile Tcl yourself on Windows, but I haven't tried that
> in many years.
>
>
> For Oracle on Windows, I installed the Oracle "Instant Client" version
> 19.6.0.0.0 a while back.  This was 4 separate zip files:
>
>   instantclient-basiclite-windows.x64-19.6.0.0.0dbru.zip
>   instantclient-sdk-windows.x64-19.6.0.0.0dbru.zip
>   instantclient-sqlplus-windows.x64-19.6.0.0.0dbru.zip
>   instantclient-tools-windows.x64-19.6.0.0.0dbru.zip
>
> To install, I simply unzipped all of those into a tmp directory, which
> created an "instantclient_19_6" directory.  Then I copied the whole
> "instantclient_19_6" directory into "C:\P\oracle\".
> The "nsoracle/Makefile.win32" has this line to use it:
>
>   ORACLE_HOME = C:\P\oracle\instantclient_19_6
>
>
> For OpenSSL, back in Windows XP days I did compile it on Windows, and
> as far as I recall that worked, although the compile was VERY slow,
> and I had to install various support tools first (NASM Assembler,
> etc.).  More recently I've only installed binary OpenSSL builds on
> Windows.
>
> I've used this OpenSSL for Windows:
>
>   http://www.slproweb.com/products/Win32OpenSSL.html
>
> There are also binaries for the older 1.0.2 OpenSSL line, here, but I
> don't think I ever actually used them:
>
>   https://indy.fulgan.com/SSL/
>
> See also OpenSSL's own list of Windows Binary Builds:
>
>   https://wiki.openssl.org/index.php/Binaries
>
> Old Win32OpenSSL instructions said to install the Visual C++ 2008
> Redistributable Package via its "vcredist_x64.exe" and
> "vcredist_x86.exe" web installers:
>
>   x64: http://www.microsoft.com/en-us/download/details.aspx?id=15336
>   x86: http://www.microsoft.com/en-us/download/details.aspx?id=29
>
> But of course these days should probably be using the 64-bit "Visual
> C++ 2017 Redistributable" or something even newer instead.  (Read
> their current instuctions.)
>
> Around 2019-01, I installed the (at the time) latest
> "Win64OpenSSL-1_1_1a.exe" and "Win32OpenSSL-1_1_1a.exe" into
> "C:\P\OpenSSL-Win64\".  When it asked my where to "Copy OpenSSL DLLs
> to:" I always pick "The OpenSSL binaries (/bin) directory", NOT the
> default "Windows system directory".
>
> --
> Andrew Piskorski <at...@pi...>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] Recommended approach to building Naviserver on Windows?

[Andrew P. <at...@pi...>]

The Tcl version should not really matter, I just install whatever the
latest stable ActiveTcl is for Windows.  I believe the latest I've
used was 8.6.x, which I installed into "C:\P\Tcl-64-8.6\".  It should
also work to compile Tcl yourself on Windows, but I haven't tried that
in many years.


For Oracle on Windows, I installed the Oracle "Instant Client" version
19.6.0.0.0 a while back.  This was 4 separate zip files:

  instantclient-basiclite-windows.x64-19.6.0.0.0dbru.zip
  instantclient-sdk-windows.x64-19.6.0.0.0dbru.zip
  instantclient-sqlplus-windows.x64-19.6.0.0.0dbru.zip
  instantclient-tools-windows.x64-19.6.0.0.0dbru.zip

To install, I simply unzipped all of those into a tmp directory, which
created an "instantclient_19_6" directory.  Then I copied the whole
"instantclient_19_6" directory into "C:\P\oracle\".
The "nsoracle/Makefile.win32" has this line to use it:

  ORACLE_HOME = C:\P\oracle\instantclient_19_6


For OpenSSL, back in Windows XP days I did compile it on Windows, and
as far as I recall that worked, although the compile was VERY slow,
and I had to install various support tools first (NASM Assembler,
etc.).  More recently I've only installed binary OpenSSL builds on
Windows.

I've used this OpenSSL for Windows:

  http://www.slproweb.com/products/Win32OpenSSL.html

There are also binaries for the older 1.0.2 OpenSSL line, here, but I
don't think I ever actually used them:

  https://indy.fulgan.com/SSL/

See also OpenSSL's own list of Windows Binary Builds:

  https://wiki.openssl.org/index.php/Binaries

Old Win32OpenSSL instructions said to install the Visual C++ 2008
Redistributable Package via its "vcredist_x64.exe" and
"vcredist_x86.exe" web installers:

  x64: http://www.microsoft.com/en-us/download/details.aspx?id=15336
  x86: http://www.microsoft.com/en-us/download/details.aspx?id=29

But of course these days should probably be using the 64-bit "Visual
C++ 2017 Redistributable" or something even newer instead.  (Read
their current instuctions.)

Around 2019-01, I installed the (at the time) latest
"Win64OpenSSL-1_1_1a.exe" and "Win32OpenSSL-1_1_1a.exe" into
"C:\P\OpenSSL-Win64\".  When it asked my where to "Copy OpenSSL DLLs
to:" I always pick "The OpenSSL binaries (/bin) directory", NOT the
default "Windows system directory".

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] Recommended approach to building Naviserver on Windows?

[Andrew P. <at...@pi...>]

On Thu, Mar 04, 2021 at 10:24:42AM +0000, Brian Fenton wrote:

> I took a look at the install documentation here https://bitbucket.org/naviserver/naviserver/src/master/ and I noticed that there are 3 approaches described for compiling and installing on Windows.
> Which one of the 3 is the preferred approach within the community?

The only approach I have extensive experience with is using the
Microsoft C compiler natively on Windows.  That works.

I installed "Visual Studio 2019 Community Edition 16.5", and haven't
had any reason to look for a newer version yet.  In the installer, I
checked only the "Desktop development with C++" Workload, and
otherwise left everything else set to defaults.

Here are my notes to Compile and Install NaviServer on Windows, using
the command-line MSVCC compiler:  (I believe this info was up-to-date
around 2020-07, using the NaviServer head code from c. 2020-06-14.)

1. To compile we need "include/nsversion.h", which is auto-generated
by "include/nsversion.h.in", so ou have two choices:

Either (bizarrely), run configure on Linux before building on Windows!
Surprisingly, this works and is how I did it for a long time.
Or if you don't like that, or don't have Linux available, use the
"win32-util/configure.tcl" script instead.  Actually you don't need to
run it yourself at all, as "Makefile.win32" does it for you.

2. If you already built Linux NaviServer in that same directory, run
make clean there to remove the Linux object files.  (TODO: It would be
so much nicer to have separate Build-* subdirectories...)

3. As the instructions in "naviserver/Makefile.win32" say, make sure
you edit "include/Makefile.build, changing the "Makefile.module" line
to say "Makefile.win32" instead!  If you do not you will get cryptic
nmake errors.

4. In a Windows Command Prompt window, Build NaviServer like this, for 64-bit:

  "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat"
  z: & cd Z:\src\web\ns-fork-myfork\naviserver
  nmake -f Makefile.win32  clean-core clean-mod  all-core all-mod

5. Again from the Command Prompt, run this little script to properly
copy all the files to their installed locations:

  cd Z:\src\web\ns-fork-myfork\naviserver\
  tclsh ./win32-util/install-nsd.tcl -i

6. Test plain NaviServer in the foreground, e.g. like so:

  C:\web\nsd4-myfork-20200411-1\bin\nsd.exe -t
  C:\web\nsd4-myfork-20200411-1\conf\simple-config.tcl -f

Note that by default the "install-nsd.tcl" script above does NOT
overwrite the installed binaries, instead it installs to a similar
directory alongside it, with a timestamp in the name, e.g.,
"nsd4-myfork-20040927T190908\".  You are free to rename that directory
later to whatever you want (because nothing inside it embeds the
directory name).

You can also use the "install-nsd.tcl" script's "-f" and "-t" switches
to control what directory the install script copies from and to, but
normally the defaults are what you want.  If run without any
arguments, "install-nsd.tcl" will show you its defaults.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] Recommended approach to building Naviserver on Windows?

[Brian F. <Bri...@qu...>]

Hi Maksym

thanks for the reply! Could you maybe go into some detail about how you did this, if you have a few minutes?
For example, how did you build TCL and OpenSSL etc?

thanks
Brian

________________________________
From: Maksym Zinchenko <siq...@gm...>
Sent: Thursday 4 March 2021 12:33
To: nav...@li... <nav...@li...>
Subject: Re: [naviserver-devel] Recommended approach to building Naviserver on Windows?


This message's attachments contains at least one web link. This is often used for phishing attempts. Please only interact with this attachment if you know its source and that the content is safe. If in doubt, confirm the legitimacy with the sender by phone.

Hello. I've managed to compile Naviserver 4.99.20 64bits on Windows 10 with Visual Studio 2019 (Express), used 8.6 Tcl sources and PostgreSQL, had some problems with OpenSSL 1.1.1, but I'm not really good at it. But it's doable.
Cheers
Maksym

On Thu, Mar 4, 2021 at 9:57 AM Brian Fenton <Bri...@qu...<mailto:Bri...@qu...>> wrote:
Hi

we are looking to start doing our own Windows builds of Naviserver. We have been fortunate to have used Maurizio's great work in the past, but our management now see it as a priority that we do our own builds in-house.
I see that a lot of good work has been done recently by the community on the Windows front. I just have a few questions to help us get started as we don't have much experience of this kind of thing, and there seem to be multiple ways of doing some of the steps.

I took a look at the install documentation here https://bitbucket.org/naviserver/naviserver/src/master/ and I noticed that there are 3 approaches described for compiling and installing on Windows. <https://bitbucket.org/naviserver/naviserver/src/master/>
Which one of the 3 is the preferred approach within the community?
Which TCL version is currently recommended of the two versions currently available (8.6.11 and 8.7a3 from https://www.tcl-lang.org/software/tcltk/download.html )?
Are there Naviserver-specific instructions for configuring, building and installing TCL on Windows (apart from the standard instructions here https://www.tcl-lang.org/doc/howto/compile.html#win)?
Are there Windows-specific instructions for compiling the latest Oracle driver? I see https://bitbucket.org/naviserver/nsoracle/src/master/ has recent commits, but the documentation still references AOLserver so it's not clear how up to date that is.
What is the recommended version of OpenSSL (or LibreSSL?) for Naviserver running on Windows (seems to be quite a few choices for OpenSSL here https://wiki.openssl.org/index.php/Binaries )? Are there Naviserver-specific instructions for getting either of these running on Windows too?

thanks
Brian
_______________________________________________
naviserver-devel mailing list
nav...@li...<mailto:nav...@li...>
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Recommended approach to building Naviserver on Windows?

[Maksym Z. <siq...@gm...>]

Hello. I've managed to compile Naviserver 4.99.20 64bits on Windows 10 with
Visual Studio 2019 (Express), used 8.6 Tcl sources and PostgreSQL, had some
problems with OpenSSL 1.1.1, but I'm not really good at it. But it's doable.
Cheers
Maksym

On Thu, Mar 4, 2021 at 9:57 AM Brian Fenton <Bri...@qu...> wrote:

> Hi
>
> we are looking to start doing our own Windows builds of Naviserver. We
> have been fortunate to have used Maurizio's great work in the past, but our
> management now see it as a priority that we do our own builds in-house.
> I see that a lot of good work has been done recently by the community on
> the Windows front. I just have a few questions to help us get started as we
> don't have much experience of this kind of thing, and there seem to be
> multiple ways of doing some of the steps.
>
> I took a look at the install documentation here
> https://bitbucket.org/naviserver/naviserver/src/master/ and I
> noticed that there are 3 approaches described for compiling and installing
> on Windows.  <https://bitbucket.org/naviserver/naviserver/src/master/>
> Which one of the 3 is the preferred approach within the community?
> Which TCL version is currently recommended of the two versions currently
> available (8.6.11 and 8.7a3 from
> https://www.tcl-lang.org/software/tcltk/download.html )?
> Are there Naviserver-specific instructions for configuring, building and
> installing TCL on Windows (apart from the standard instructions here
> https://www.tcl-lang.org/doc/howto/compile.html#win)?
> Are there Windows-specific instructions for compiling the latest Oracle
> driver? I see https://bitbucket.org/naviserver/nsoracle/src/master/ has
> recent commits, but the documentation still references AOLserver so it's
> not clear how up to date that is.
> What is the recommended version of OpenSSL (or LibreSSL?) for Naviserver
> running on Windows (seems to be quite a few choices for OpenSSL here
> https://wiki.openssl.org/index.php/Binaries )? Are there Naviserver
> -specific instructions for getting either of these running on Windows too?
>
> thanks
> Brian
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

[naviserver-devel] Recommended approach to building Naviserver on Windows?

[Brian F. <Bri...@qu...>]

Hi

we are looking to start doing our own Windows builds of Naviserver. We have been fortunate to have used Maurizio's great work in the past, but our management now see it as a priority that we do our own builds in-house.
I see that a lot of good work has been done recently by the community on the Windows front. I just have a few questions to help us get started as we don't have much experience of this kind of thing, and there seem to be multiple ways of doing some of the steps.

I took a look at the install documentation here https://bitbucket.org/naviserver/naviserver/src/master/ and I noticed that there are 3 approaches described for compiling and installing on Windows. <https://bitbucket.org/naviserver/naviserver/src/master/>
Which one of the 3 is the preferred approach within the community?
Which TCL version is currently recommended of the two versions currently available (8.6.11 and 8.7a3 from https://www.tcl-lang.org/software/tcltk/download.html )?
Are there Naviserver-specific instructions for configuring, building and installing TCL on Windows (apart from the standard instructions here https://www.tcl-lang.org/doc/howto/compile.html#win)?
Are there Windows-specific instructions for compiling the latest Oracle driver? I see https://bitbucket.org/naviserver/nsoracle/src/master/ has recent commits, but the documentation still references AOLserver so it's not clear how up to date that is.
What is the recommended version of OpenSSL (or LibreSSL?) for Naviserver running on Windows (seems to be quite a few choices for OpenSSL here https://wiki.openssl.org/index.php/Binaries )? Are there Naviserver-specific instructions for getting either of these running on Windows too?

thanks
Brian

Re: [naviserver-devel] Content-Disposition: form-data - Apparently wrong format

[Gustaf N. <ne...@wu...>]

Dear Iuri

"ns_http" does not do magic transformation on the POST data just by 
providing different header fields. You have to provide the correct 
formdata yourself. You are providing apparently the data in the wrong 
format.

However, since you are using OpenACS, you should be able to use

     util::http::post ... -formvars formvars ... -multipart ...

See [1] for details

-gn

[1] https://openacs.org/api-doc/proc-view?proc=util::http::post&source_p=1

On 24.02.21 19:08, Iuri de Araujo Sampaio wrote:
> Hello there,
> How would I achieve such format for a POST data request, using ns_procs?
>
>
> [24/Feb/2021:14:52:13][30433.7efbf3d70700][-conn:qonteo:default:0:162-] 
> Notice: COENTNT ----------------------------674349094811372605992454\x0d
> :   Content-Disposition: form-data; name="To"\x0d
> :   \x0d
> :   whatsapp:+18474048771\x0d
> :   ----------------------------674349094811372605992454\x0d
> :   Content-Disposition: form-data; name="From"\x0d
> :   \x0d
> :   whatsapp:+14155238886\x0d
> :   ----------------------------674349094811372605992454\x0d
> :   Content-Disposition: form-data; name="Body"\x0d
> :   \x0d
> :   Your appointment is coming up on July 21 at 3PM\x0d
> :   ----------------------------674349094811372605992454--\x0d
> :
>

Re: [naviserver-devel] What's the replacement of [curl -u] in [ns_http]

[Gustaf N. <ne...@wu...>]

On 24.02.21 17:46, Iuri de Araujo Sampaio wrote:
> Even though, I still have no idea why that happens.

please re-read the following lines of my last answer. What is unclear 
about  this?
Since it works now, it is clear, that the line-breaks caused you troubles.

The line-breaks are usual for base64 encoders (GNU uuencode, tcllib 
base64::encode, ...), so ns_base64encode is just compatible with these.

The code in OpenACS, that David has pointed to, has therefore exactly 
the same problem, when the encoded string starts to contain line breaks. 
Usually userid+password are not so long as in your case, so this bug was 
not noticed before.

-gn

>
>> On Raj. 13, 1442 AH, at 00:03, Gustaf Neumann <ne...@wu... 
>> <mailto:ne...@wu...>> wrote:
>>
>> Dear Iuri,
>>
>> The problem, you are probably facing is probably that ns_base64encode 
>> produces line breaks as required for MIME encoded text strings [1].
>>
>> % ns_base64encode Afrewf564DFSFSF54jgnfhgGDGdfRGRT43:7584fjhfjhf84jkrugrefAFFD9449474
>> QWZyZXdmNTY0REZTRlNGNTRqZ25maGdHREdkZlJHUlQ0Mzo3NTg0ZmpoZmpo
>> Zjg0amtydWdyZWZBRkZEOTQ0OTQ3NA==
>>
>> Converting to and from a string with these linebreaks should be fine 
>> for all base64 codecs (you see, colon is not the problem)
>>
>> % ns_base64decode [ns_base64encode Afrewf564DFSFSF54jgnfhgGDGdfRGRT43:7584fjhfjhf84jkrugrefAFFD9449474]
>> Afrewf564DFSFSF54jgnfhgGDGdfRGRT43:7584fjhfjhf84jkrugrefAFFD9449474
>> But probably, the header field has problems with linebreaks. 
>> Therefore, simply remove the newlines by e.g.
>> set credentials [join [ns_base64encode Afrewf564DFSFSF54jgnfhgGDGdfRGRT43:7584fjhfjhf84jkrugrefAFFD9449474] ""]
>> set req_headers [ns_set create headers Authorization "Basic $credentials"]
>>
>> All the best
>> -g
>>
>> PS: In order to see, what is sent via and and/or node, use tcpdump 
>> and/or wireshark, then you do not have to do any guessing, what is 
>> exactly sent by which command in which implementation....
>>
>> [1] 
>> https://unix.stackexchange.com/questions/447191/whats-the-right-way-to-base64-encode-a-binary-file-on-centos-7#:~:text=base64%20inserts%20line%20breaks%20after,Transfer%2DEncoding%20for%20emails).
>>
>>
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li... 
>> <mailto:nav...@li...>
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"

[naviserver-devel] Content-Disposition: form-data - Apparently wrong format

[Iuri de A. S. <iu...@iu...>]

Hello there,
How would I achieve such format for a POST data request, using ns_procs?


[24/Feb/2021:14:52:13][30433.7efbf3d70700][-conn:qonteo:default:0:162-] Notice: COENTNT ----------------------------674349094811372605992454\x0d
:    Content-Disposition: form-data; name="To"\x0d
:    \x0d
:    whatsapp:+18474048771\x0d
:    ----------------------------674349094811372605992454\x0d
:    Content-Disposition: form-data; name="From"\x0d
:    \x0d
:    whatsapp:+14155238886\x0d
:    ----------------------------674349094811372605992454\x0d
:    Content-Disposition: form-data; name="Body"\x0d
:    \x0d
:    Your appointment is coming up on July 21 at 3PM\x0d
:    ----------------------------674349094811372605992454--\x0d
:    



Following official documentation, I’ve changed Content-Type to multipart/form-data,  however the body of the request got a different format. So the API on the other side complains about that fields have not been passed correctly,

[24/Feb/2021:14:52:23][30433.7efbf3d70700][-conn:qonteo:default:0:167-] Notice: COENTNT To=whatsapp:%2b+18474048771 From=whatsapp:%2b14155238886 Body=%22Your&Yummy&Cupcakes&Company&order&of&1&dozen&frosted&cupcakes&has&shipped&and&shoul&d&be&delivered%22

#######################
 # construct POST data
 #######################
 set post_data {}
 foreach {key value} {q NaviServer} {
   lappend post_data "[ns_urlencode $key]=[ns_urlencode $value]"
 }
 set post_data [join $post_data &]
#######################
 # submit POST request
 #######################
 set requestHeaders [ns_set create]
 set replyHeaders [ns_set create]
 ns_set update $requestHeaders "Content-type" "multipart/form-data"
set h [ns_http queue -method POST \
   -headers $requestHeaders \
   -timeout 10.0 \
   -body $post_data https://duckduckgo.com/]
 set r [ns_http wait $h]

Re: [naviserver-devel] What's the replacement of [curl -u] in [ns_http]

[Iuri de A. S. <iu...@iu...>]

Thanks Gustaf,

[join $hash “”] was the solution.  It works now!

Even though, I still have no idea why that happens. I’m using ns_* procs,  simple and straight forward. As in:

>   % set h [ns_set create]
>   % ns_set update $h Authorization "Basic [ns_base64encode guest:guest]"
>   % ns_http run -headers $h https://jigsaw.w3.org/HTTP/Basic/ <https://jigsaw.w3.org/HTTP/Basic/> 
> <https://jigsaw.w3.org/HTTP/Basic/ <https://jigsaw.w3.org/HTTP/Basic/>>


the only different is because strings are longer than “guest:guest"


Best wishes,
I


> On Raj. 13, 1442 AH, at 00:03, Gustaf Neumann <ne...@wu...> wrote:
> 
> Dear Iuri,
> 
> The problem, you are probably facing is probably that ns_base64encode produces line breaks as required for MIME encoded text strings [1].
> 
> % ns_base64encode Afrewf564DFSFSF54jgnfhgGDGdfRGRT43:7584fjhfjhf84jkrugrefAFFD9449474
> QWZyZXdmNTY0REZTRlNGNTRqZ25maGdHREdkZlJHUlQ0Mzo3NTg0ZmpoZmpo
> Zjg0amtydWdyZWZBRkZEOTQ0OTQ3NA==
> Converting to and from a string with these linebreaks should be fine for all base64 codecs (you see, colon is not the problem)
> 
> % ns_base64decode [ns_base64encode Afrewf564DFSFSF54jgnfhgGDGdfRGRT43:7584fjhfjhf84jkrugrefAFFD9449474]
> Afrewf564DFSFSF54jgnfhgGDGdfRGRT43:7584fjhfjhf84jkrugrefAFFD9449474
> But probably, the header field has problems with linebreaks. Therefore, simply remove the newlines by e.g.
> set credentials [join [ns_base64encode Afrewf564DFSFSF54jgnfhgGDGdfRGRT43:7584fjhfjhf84jkrugrefAFFD9449474] ""]
> set req_headers [ns_set create headers Authorization "Basic $credentials"]
> All the best
> -g
> 
> PS: In order to see, what is sent via and and/or node, use tcpdump and/or wireshark, then you do not have to do any guessing, what is exactly sent by which command in which implementation....
> 
> [1] https://unix.stackexchange.com/questions/447191/whats-the-right-way-to-base64-encode-a-binary-file-on-centos-7#:~:text=base64%20inserts%20line%20breaks%20after,Transfer%2DEncoding%20for%20emails <https://unix.stackexchange.com/questions/447191/whats-the-right-way-to-base64-encode-a-binary-file-on-centos-7#:~:text=base64%20inserts%20line%20breaks%20after,Transfer%2DEncoding%20for%20emails>).
> 
> 
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] What's the replacement of [curl -u] in [ns_http]

[Gustaf N. <ne...@wu...>]

Dear Iuri,

The problem, you are probably facing is probably that ns_base64encode 
produces line breaks as required for MIME encoded text strings [1].

% ns_base64encode Afrewf564DFSFSF54jgnfhgGDGdfRGRT43:7584fjhfjhf84jkrugrefAFFD9449474

QWZyZXdmNTY0REZTRlNGNTRqZ25maGdHREdkZlJHUlQ0Mzo3NTg0ZmpoZmpo
Zjg0amtydWdyZWZBRkZEOTQ0OTQ3NA==

Converting to and from a string with these linebreaks should be fine for 
all base64 codecs (you see, colon is not the problem)

% ns_base64decode [ns_base64encode Afrewf564DFSFSF54jgnfhgGDGdfRGRT43:7584fjhfjhf84jkrugrefAFFD9449474]

Afrewf564DFSFSF54jgnfhgGDGdfRGRT43:7584fjhfjhf84jkrugrefAFFD9449474

But probably, the header field has problems with linebreaks. Therefore, 
simply remove the newlines by e.g.

set credentials [join [ns_base64encode Afrewf564DFSFSF54jgnfhgGDGdfRGRT43:7584fjhfjhf84jkrugrefAFFD9449474] ""]
set req_headers [ns_set create headers Authorization "Basic $credentials"]

All the best
-g

PS: In order to see, what is sent via and and/or node, use tcpdump 
and/or wireshark, then you do not have to do any guessing, what is 
exactly sent by which command in which implementation....

[1] 
https://unix.stackexchange.com/questions/447191/whats-the-right-way-to-base64-encode-a-binary-file-on-centos-7#:~:text=base64%20inserts%20line%20breaks%20after,Transfer%2DEncoding%20for%20emails).

Re: [naviserver-devel] What's the replacement of [curl -u] in [ns_http]

[Iuri de A. S. <iu...@iu...>]

Hi everyone, 

So far, It seems ns_base64encode generates a different hash key, and the colon is the problem somehow. 


A valid request from Postman or CURL, has the following Authorization block within it :

Authorization {Basic HGTRFHTYhththtyhotyhtyjotjytjojg956456346n64ui5n4436346nu4y4iuyn4iutGRENDM3OWQwNmFkMWsFmNg==}

Meaning the authorization hash is:

HGTRFHTYhththtyhotyhtyjotjytjojg956456346n64ui5n4436346nu4y4iuyn4iutGRENDM3OWQwNmFkMWsFmNg==




While the Authorization block generated by [ns_base64encode] is: 

Authorization {Basic HGTRFHTYhththtyhotyhtyjotjytjojg956456346n64ui5n4436346nu4y4} 

So, it’s only the first part. 
HGTRFHTYhththtyhotyhtyjotjytjojg956456346n64ui5n4436346nu4y4


and the last part, after ":” has been missing (i.e. password) 
iuyn4iutGRENDM3OWQwNmFkMWsFmNg==



However, the same credentials were used to generate the 64 bits hash key encoded. 
set req_headers [ns_set create]
ns_set update $req_headers Authorization "Basic [ns_base64encode Afrewf564DFSFSF54jgnfhgGDGdfRGRT43:7584fjhfjhf84jkrugrefAFFD9449474]"


My guess is that [ns_base64encode] breaks in the colon char, and encodes only the first part as a 64 bits hash key.

How would the entire string  be encode as a 64 bits hash then?

How would the proc ns_base64encode support colon “:" (if the colon is the problem actually)?


p.s. logs are attached bellow.


Best wishes,
I





A valid request from Postman or CURL, contains the entire 64 bits hash key encoded 

[23/Feb/2021:15:59:08][23531.7efbf356f700][-conn:qonteo:default:1:0-] Notice: HEADER 
:     t0
[23/Feb/2021:15:59:08][23531.7efbf356f700][-conn:qonteo:default:1:0-] Notice: HEADERS 11
[23/Feb/2021:15:59:08][23531.7efbf356f700][-conn:qonteo:default:1:0-] Notice: Host dashboard.qonteo.com X-Real-IP 187.127.207.76 Connection close Content-Length 454 Authorization {Basic HGTRFHTYhththtyhotyhtyjotjytjojg956456346n64ui5n4436346nu4y4iuyn4iutGRENDM3OWQwNmFkMWsFmNg==  User-Agent PostmanRuntime/7.26.8 Accept */* Cache-Control no-cache Postman-Token 6bd5b753-bc8e-4447-bc85-155f517d6c72 Content-Type {multipart/form-data; boundary=--------------------------494740493183890794196511} Cookie {ad_locale="es_ES"; ad_session_id="34040084%2c0%2c0%2c1614106611%20{362%201614107811%201133CEC432ACA1B0529F80D7603EFD5D38289E85}”}





The request from Naviserver, contains a different Authorization 64 bits hash key encoded. The has key is only correspondent to the first part of the authorization (i.e. username).
 
[23/Feb/2021:15:58:01][1207.7efbb67fc700][-conn:qonteo:default:3:3251-] Notice: HEADER 
:     t0
[23/Feb/2021:15:58:01][1207.7efbb67fc700][-conn:qonteo:default:3:3251-] Notice: HEADERS 6
[23/Feb/2021:15:58:01][1207.7efbb67fc700][-conn:qonteo:default:3:3251-] Notice: Host dashboard.qonteo.com X-Real-IP 192.199.241.132 Connection close Content-Length 178 Authorization {Basic HGTRFHTYhththtyhotyhtyjotjytjojg956456346n64ui5n4436346nu4y4}  User-Agent NaviServer/4.99.19







> On Raj. 11, 1442 AH, at 16:08, Iuri de Araujo Sampaio <iu...@iu...> wrote:
> 
> So far, I've found a previous post, from myself :-), regarding quite the same problem.
> 
> https://www.mail-archive.com/nav...@li.../msg03818.html <https://www.mail-archive.com/nav...@li.../msg03818.html>
> 
> > 
> > Ok, now add some Authorization. As defined by RFC 7617, one has to add an
> > Authorization request header field with user:id:password encoded in base64.
> >   % set h [ns_set create]
> >   % ns_set update $h Authorization "Basic [ns_base64encode guest:guest]"
> >   % ns_http run -headers $h https://jigsaw.w3.org/HTTP/Basic/ <https://jigsaw.w3.org/HTTP/Basic/> 
> > <https://jigsaw.w3.org/HTTP/Basic/ <https://jigsaw.w3.org/HTTP/Basic/>>
> 
> 
> However, I’m still having a hard time to figure out what would be the replacement of the following NodeJS sample for a Basic Auth header on Naviserver, proc
>  ns_http 
> 
> 
> NodeJS sample
> 
> ...
>  axios
>         .post(url, new URLSearchParams(messageBody), {
>             auth: {
>                 username: authSID,
>                 password: authToken
>             }
>         })
> 
> ...
> 
> 
> 
> TCL Sample
> 
> ...
> set req_headers [ns_set create]
> ns_set update $req_headers Authorization "Basic [ns_base64encode ${username}:${token}]"
> 
> set data [list]
> lappend "[ns_urlencode To]=[ns_urlencode whatsapp:+55454545571]"
> lappend "[ns_urlencode From]=[ns_urlencode whatsapp:+14155238886]"
> lappend "[ns_urlencode Body]=[ns_urlencode \"Your Yummy Cupcakes Company order of 1 dozen frosted cupcakes has shipped and should be delivered\"]"
> 
> 
> set res [ns_http run -method POST -headers $req_headers -body $data $url]
> ns_log Notice "RES2 $res"
> 
> 
> 
> Based on this TCL sample, it returns 401 , meaning authentication step had failure.
> 
> 
> 22/Feb/2021:16:07:21][1207.7efbb6ffd700][-conn:qonteo:default:2:1647-] Notice: RES2 status 401 time 0:203450 headers d5 body {{"code": 20003, "detail": "", "message": "Authenticate", "more_info": "https://www.twilio.com/docs/errors/20003 <https://www.twilio.com/docs/errors/20003>", "status": 401}} https {sslversion TLSv1.2 cipher ECDHE-RSA-AES128-GCM-SHA256}
> 
> 
> 
> 
> Best wishes,
> I
> 
> 
> 
> 
>> On Raj. 10, 1442 AH, at 20:14, Iuri de Araujo Sampaio <iu...@iu... <mailto:iu...@iu...>> wrote:
>> 
>> Hello there,
>> I’m trying to rewrite a CURL request to Naviserver. 
>> 
>> curl 'https://api.twilio.com/2010-04-01/Accounts/ACe13c882f57e87c4b4db37/Messages.json <https://api.twilio.com/2010-04-01/Accounts/ACe13c882f57e87c4b4db37/Messages.json>' -X POST \
>> --data-urlencode 'To=whatsapp:+5511998865465' \
>> --data-urlencode 'From=whatsapp:+14155238886' \
>> --data-urlencode 'Body=Your Yummy Cupcakes Company order of 1 dozen frosted cupcakes has shipped and should be delivered on July 10, 2019. Details: http://www.yummycupcakes.com/' <http://www.yummycupcakes.com/'> \
>> -u [Authkey]:[AuthToken]
>> 
>> 
>> man url 
>>  -u, --user <user:password>
>>               Specify the user name and password to use for server authentication. Overrides -n, --netrc and --netrc-optional.
>> 
>>               If you simply specify the user name, curl will prompt for a password.
>> …
>> 
>> 
>> 
>> So basically, data-urlencode becomes posta_data. How could -u argument can be rewritten to [ns_http run …] arguments
>> 
>> 
>> set post_data [list]
>> lappend "[ns_urlencode to]=[ns_urlencode whatsapp:+5511998865465]"
>> lappend "[ns_urlencode From]=[ns_urlencode whatsapp:+14155238886]"
>> lappend "[ns_urlencode Body]=[ns_urlencode \"Your Yummy Cupcakes Company order of 1 dozen frosted cupcakes has shipped and should be deli\
>> vered\"]"
>> 
>> set url "https://api.twilio.com/2010-04-01/Accounts/ACe13c431f82f57e87c4b4db37/Messages.json <https://api.twilio.com/2010-04-01/Accounts/ACe13c431f82f57e87c4b4db37/Messages.json>"
>> 
>> set res [ns_http run -method POST -headers $req_headers -body $post_data $url]
>> ns_log Notice "RES2 $res"
>> 
>> 
>> 
>> I was looking for references in the documentation, however I haven’t found any matches. What's the replacement of curl -u in ns_http?
>> 
>> References: 
>> https://naviserver.sourceforge.io/n/naviserver/files/ns_http.html <https://naviserver.sourceforge.io/n/naviserver/files/ns_http.html>
>> 
>> 
>> Best wishes,
>> I
> 
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] What's the replacement of [curl -u] in [ns_http]

[Iuri de A. S. <iu...@iu...>]

So far, I've found a previous post, from myself :-), regarding quite the same problem.

https://www.mail-archive.com/nav...@li.../msg03818.html <https://www.mail-archive.com/nav...@li.../msg03818.html>

> 
> Ok, now add some Authorization. As defined by RFC 7617, one has to add an
> Authorization request header field with user:id:password encoded in base64.
>   % set h [ns_set create]
>   % ns_set update $h Authorization "Basic [ns_base64encode guest:guest]"
>   % ns_http run -headers $h https://jigsaw.w3.org/HTTP/Basic/ <https://jigsaw.w3.org/HTTP/Basic/> 
> <https://jigsaw.w3.org/HTTP/Basic/ <https://jigsaw.w3.org/HTTP/Basic/>>


However, I’m still having a hard time to figure out what would be the replacement of the following NodeJS sample for a Basic Auth header on Naviserver, proc
 ns_http 


NodeJS sample

...
 axios
        .post(url, new URLSearchParams(messageBody), {
            auth: {
                username: authSID,
                password: authToken
            }
        })

...



TCL Sample

...
set req_headers [ns_set create]
ns_set update $req_headers Authorization "Basic [ns_base64encode ${username}:${token}]"

set data [list]
lappend "[ns_urlencode To]=[ns_urlencode whatsapp:+55454545571]"
lappend "[ns_urlencode From]=[ns_urlencode whatsapp:+14155238886]"
lappend "[ns_urlencode Body]=[ns_urlencode \"Your Yummy Cupcakes Company order of 1 dozen frosted cupcakes has shipped and should be delivered\"]"


set res [ns_http run -method POST -headers $req_headers -body $data $url]
ns_log Notice "RES2 $res"



Based on this TCL sample, it returns 401 , meaning authentication step had failure.


22/Feb/2021:16:07:21][1207.7efbb6ffd700][-conn:qonteo:default:2:1647-] Notice: RES2 status 401 time 0:203450 headers d5 body {{"code": 20003, "detail": "", "message": "Authenticate", "more_info": "https://www.twilio.com/docs/errors/20003", "status": 401}} https {sslversion TLSv1.2 cipher ECDHE-RSA-AES128-GCM-SHA256}




Best wishes,
I




> On Raj. 10, 1442 AH, at 20:14, Iuri de Araujo Sampaio <iu...@iu...> wrote:
> 
> Hello there,
> I’m trying to rewrite a CURL request to Naviserver. 
> 
> curl 'https://api.twilio.com/2010-04-01/Accounts/ACe13c882f57e87c4b4db37/Messages.json <https://api.twilio.com/2010-04-01/Accounts/ACe13c882f57e87c4b4db37/Messages.json>' -X POST \
> --data-urlencode 'To=whatsapp:+5511998865465' \
> --data-urlencode 'From=whatsapp:+14155238886' \
> --data-urlencode 'Body=Your Yummy Cupcakes Company order of 1 dozen frosted cupcakes has shipped and should be delivered on July 10, 2019. Details: http://www.yummycupcakes.com/' <http://www.yummycupcakes.com/'> \
> -u [Authkey]:[AuthToken]
> 
> 
> man url 
>  -u, --user <user:password>
>               Specify the user name and password to use for server authentication. Overrides -n, --netrc and --netrc-optional.
> 
>               If you simply specify the user name, curl will prompt for a password.
> …
> 
> 
> 
> So basically, data-urlencode becomes posta_data. How could -u argument can be rewritten to [ns_http run …] arguments
> 
> 
> set post_data [list]
> lappend "[ns_urlencode to]=[ns_urlencode whatsapp:+5511998865465]"
> lappend "[ns_urlencode From]=[ns_urlencode whatsapp:+14155238886]"
> lappend "[ns_urlencode Body]=[ns_urlencode \"Your Yummy Cupcakes Company order of 1 dozen frosted cupcakes has shipped and should be deli\
> vered\"]"
> 
> set url "https://api.twilio.com/2010-04-01/Accounts/ACe13c431f82f57e87c4b4db37/Messages.json <https://api.twilio.com/2010-04-01/Accounts/ACe13c431f82f57e87c4b4db37/Messages.json>"
> 
> set res [ns_http run -method POST -headers $req_headers -body $post_data $url]
> ns_log Notice "RES2 $res"
> 
> 
> 
> I was looking for references in the documentation, however I haven’t found any matches. What's the replacement of curl -u in ns_http?
> 
> References: 
> https://naviserver.sourceforge.io/n/naviserver/files/ns_http.html <https://naviserver.sourceforge.io/n/naviserver/files/ns_http.html>
> 
> 
> Best wishes,
> I

Re: [naviserver-devel] What's the replacement of [curl -u] in [ns_http]

[David O. <da...@qc...>]

Hi Luri,
There may be a better way but you could construct the Basic Auth header by
hand and set in the outgoing $req_headers.
Something similar in here:
https://openacs.org/api-doc/proc-view?proc=util::http::basic_auth&source_p=1

On Sun, 21 Feb 2021 at 23:34, Iuri de Araujo Sampaio <iu...@iu...> wrote:

> Hello there,
> I’m trying to rewrite a CURL request to Naviserver.
>
> curl '
> https://api.twilio.com/2010-04-01/Accounts/ACe13c882f57e87c4b4db37/Messages.json
> ' -X POST \ --data-urlencode 'To=whatsapp:+5511998865465' \
> --data-urlencode 'From=whatsapp:+14155238886' \ --data-urlencode 'Body=Your
> Yummy Cupcakes Company order of 1 dozen frosted cupcakes has shipped and
> should be delivered on July 10, 2019. Details:
> http://www.yummycupcakes.com/' \ -u [Authkey]:[AuthToken]
>
>
> man url
>  -u, --user <user:password>
>               Specify the user name and password to use for server
> authentication. Overrides -n, --netrc and --netrc-optional.
>
>               If you simply specify the user name, curl will prompt for a
> password.
> …
>
>
>
> So basically, data-urlencode becomes posta_data. How could -u argument can
> be rewritten to [ns_http run …] arguments
>
>
> set post_data [list]
> lappend "[ns_urlencode to]=[ns_urlencode whatsapp:+5511998865465]"
> lappend "[ns_urlencode From]=[ns_urlencode whatsapp:+14155238886]"
> lappend "[ns_urlencode Body]=[ns_urlencode \"Your Yummy Cupcakes Company
> order of 1 dozen frosted cupcakes has shipped and should be deli\
> vered\"]"
>
> set url "
> https://api.twilio.com/2010-04-01/Accounts/ACe13c431f82f57e87c4b4db37/Messages.json
> "
>
> set res [ns_http run -method POST -headers $req_headers -body $post_data $
> url]
> ns_log Notice "RES2 $res"
>
>
>
> I was looking for references in the documentation, however I haven’t found
> any matches. What's the replacement of curl -u in ns_http?
>
> References:
> https://naviserver.sourceforge.io/n/naviserver/files/ns_http.html
>
>
> Best wishes,
> I
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>


-- 

*David Osborne | Software Engineer*
Qcode Software, Castle House, Fairways Business Park, Inverness, IV2 6AA
*Email:* da...@qc... | *Phone:* 01463 896 484
www.qcode.co.uk

[naviserver-devel] What's the replacement of [curl -u] in [ns_http]

[Iuri de A. S. <iu...@iu...>]

Hello there,
I’m trying to rewrite a CURL request to Naviserver. 

curl 'https://api.twilio.com/2010-04-01/Accounts/ACe13c882f57e87c4b4db37/Messages.json' -X POST \
--data-urlencode 'To=whatsapp:+5511998865465' \
--data-urlencode 'From=whatsapp:+14155238886' \
--data-urlencode 'Body=Your Yummy Cupcakes Company order of 1 dozen frosted cupcakes has shipped and should be delivered on July 10, 2019. Details: http://www.yummycupcakes.com/' \
-u [Authkey]:[AuthToken]


man url 
 -u, --user <user:password>
              Specify the user name and password to use for server authentication. Overrides -n, --netrc and --netrc-optional.

              If you simply specify the user name, curl will prompt for a password.
…



So basically, data-urlencode becomes posta_data. How could -u argument can be rewritten to [ns_http run …] arguments


set post_data [list]
lappend "[ns_urlencode to]=[ns_urlencode whatsapp:+5511998865465]"
lappend "[ns_urlencode From]=[ns_urlencode whatsapp:+14155238886]"
lappend "[ns_urlencode Body]=[ns_urlencode \"Your Yummy Cupcakes Company order of 1 dozen frosted cupcakes has shipped and should be deli\
vered\"]"

set url "https://api.twilio.com/2010-04-01/Accounts/ACe13c431f82f57e87c4b4db37/Messages.json"

set res [ns_http run -method POST -headers $req_headers -body $post_data $url]
ns_log Notice "RES2 $res"



I was looking for references in the documentation, however I haven’t found any matches. What's the replacement of curl -u in ns_http?

References: 
https://naviserver.sourceforge.io/n/naviserver/files/ns_http.html <https://naviserver.sourceforge.io/n/naviserver/files/ns_http.html>


Best wishes,
I

Re: [naviserver-devel] ns_param location - change of behaviour for ns_returnredirect

[David O. <da...@qc...>]

Thank you again Gustaf,

I've done a build of the latest version and the behaviour we were seeing
previously with "location" has been restored.


On Thu, 28 Jan 2021 at 18:40, Gustaf Neumann <ne...@wu...> wrote:

> Dear David,
>
> The i could reproduce the problems, which showed up only for
> configurations with local network drivers (loading the driver
> in the server specific section, not in the global section).
>
> In general, the definition over the global section is recommended,
> since it supports virtual hosting. Nevertheless, also the "old style"
> local definitions should continue to work. The computation is
> somewhat complex, since one cannot trust the content of the
> host header field blindly, and since a single driver can listen
> on many IP addresses and ports and connect to multiple virtual
> servers.
>
> The updated version in the public repository computes now the value
> of [ns_conn location] (and the redirect target) is computed by
> the following rules.
>
> Globally defined network driver (recommended installation, supports
> virtual hosting)
> a) Use content of host header field, when the value is configured as an
>     accepted host for one of the servers.
> b) If the host value is different, fall back to the default server
>     (a default server has to be configured).
> c) If the request has no host header field, fall back to the default
> server.
>
> Per-server network driver:
> a) Use content of host header field, when the valued is configured as an
>     accepted host for the server.
> b) If the host value is different or missing (allowed in HTTP/1.0),
>     fall back to the value as configured in the driver parameter
> "location".
> c) If the driver parameter "location" is not given, use the actual
>     local IP address and port for determining the location.
>
> Please check whether this works as well with your configurations.
>
> all the best
>
> -gn
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>


-- 

*David Osborne | Software Engineer*
Qcode Software, Castle House, Fairways Business Park, Inverness, IV2 6AA
*Email:* da...@qc... | *Phone:* 01463 896 484
www.qcode.co.uk

Re: [naviserver-devel] ns_param location - change of behaviour for ns_returnredirect

[Gustaf N. <ne...@wu...>]

Dear David,

The i could reproduce the problems, which showed up only for
configurations with local network drivers (loading the driver
in the server specific section, not in the global section).

In general, the definition over the global section is recommended,
since it supports virtual hosting. Nevertheless, also the "old style"
local definitions should continue to work. The computation is
somewhat complex, since one cannot trust the content of the
host header field blindly, and since a single driver can listen
on many IP addresses and ports and connect to multiple virtual
servers.

The updated version in the public repository computes now the value
of [ns_conn location] (and the redirect target) is computed by
the following rules.

Globally defined network driver (recommended installation, supports 
virtual hosting)
a) Use content of host header field, when the value is configured as an
    accepted host for one of the servers.
b) If the host value is different, fall back to the default server
    (a default server has to be configured).
c) If the request has no host header field, fall back to the default server.

Per-server network driver:
a) Use content of host header field, when the valued is configured as an
    accepted host for the server.
b) If the host value is different or missing (allowed in HTTP/1.0),
    fall back to the value as configured in the driver parameter "location".
c) If the driver parameter "location" is not given, use the actual
    local IP address and port for determining the location.

Please check whether this works as well with your configurations.

all the best

-gn

Re: [naviserver-devel] ns_param location - change of behaviour for ns_returnredirect

[Gustaf N. <ne...@wu...>]

On 25.01.21 17:42, David Osborne wrote:
> This is part of the commit which seems to be changing the behaviour in 
> our case in SockSetServer in driver.c..
> sockPtr->location would have previously been set to drvPtr->location 
> here and passed on to connPtr->location in NsQueueConn I think.

Dear David,

i will look into the details the next days.

In general, the "location" in the driver is a problematic field in case
virtual hosting is involved (you do not want e.g. to redirect always
to a fixed location with a fixed port), but this decision should be
driven by the host header field. There is also some support
(ns_locationproc) in NaviServer for providing custom location
handlers, since the default behavior is known to be problematic.

I will come back to you later this week after doing some
investigations based on your provided configuration example.

all the best
-g

https://naviserver.sourceforge.io/n/naviserver/files/ns_locationproc.html