naviserver-devel — Developer discussion

[naviserver-devel] undefined symbol: _Ns_ModuleInit

[Sassy N. <sa...@gm...>]

Hi Group,

I was trying to install naviserver 4.99.31-1 on my ubuntu 24.04 server.
I can provide to the community a debian package I have built to support
this project (see attached picture).

However, It seems that I have an issue with the naviserver-nsdbpg module.

I'm using postgresql-17 (17.2-1.pgdg24.04) on this server, and I was
successfully build the nsdbpg module using "make
PGLIB=$/usr/lib/postgresql/17/lib/ PGINCLUDE=/usr/include/postgresql/".

The nsdbpg.so file was created successfully with no error on the
compilation and linkage.

When trying to load naviserver I'm getting the following error message:

 Error: modload: /usr/lib/x86_64-linux-gnu/naviserver/bin/nsdbpg.so: cannot
find symbol "Ns_ModuleInit":
/usr/lib/x86_64-linux-gnu/naviserver/bin/nsdbpg.so: undefined symbol:
_Ns_ModuleInit

Running ldd on the file /usr/lib/x86_64-linux-gnu/naviserver/bin/nsdbpg.so
provide this output:

        linux-vdso.so.1 (0x00007fff676fc000)
        libnsdb-4.99.31.so.1 => /lib/x86_64-linux-gnu/libnsdb-4.99.31.so.1
(0x00007ddd751af000)
        libpq.so.5 => /lib/x86_64-linux-gnu/libpq.so.5 (0x00007ddd75158000)
        libnsthread-4.99.31.so.1 =>
/lib/x86_64-linux-gnu/libnsthread-4.99.31.so.1 (0x00007ddd7514c000)
        libnsd-4.99.31.so.1 => /lib/x86_64-linux-gnu/libnsd-4.99.31.so.1
(0x00007ddd75047000)
        libtcl8.6.so => /lib/x86_64-linux-gnu/libtcl8.6.so
(0x00007ddd74e9a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ddd74c00000)
        libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3
(0x00007ddd74b56000)
        libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3
(0x00007ddd74600000)
        libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2
(0x00007ddd74e44000)
        libldap.so.2 => /lib/x86_64-linux-gnu/libldap.so.2
(0x00007ddd745a3000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007ddd74e26000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1
(0x00007ddd74b1c000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ddd744ba000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ddd751d5000)
        libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3
(0x00007ddd743f1000)
        libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3
(0x00007ddd743c5000)
        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2
(0x00007ddd74e1e000)
        libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0
(0x00007ddd743b8000)
        liblber.so.2 => /lib/x86_64-linux-gnu/liblber.so.2
(0x00007ddd743a8000)
        libsasl2.so.2 => /lib/x86_64-linux-gnu/libsasl2.so.2
(0x00007ddd7438e000)
        libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30
(0x00007ddd74194000)
        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1
(0x00007ddd74e15000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2
(0x00007ddd74181000)
        libp11-kit.so.0 => /lib/x86_64-linux-gnu/libp11-kit.so.0
(0x00007ddd73fdd000)
        libidn2.so.0 => /lib/x86_64-linux-gnu/libidn2.so.0
(0x00007ddd73fbb000)
        libunistring.so.5 => /lib/x86_64-linux-gnu/libunistring.so.5
(0x00007ddd73e0e000)
        libtasn1.so.6 => /lib/x86_64-linux-gnu/libtasn1.so.6
(0x00007ddd73df8000)
        libnettle.so.8 => /lib/x86_64-linux-gnu/libnettle.so.8
(0x00007ddd73da3000)
        libhogweed.so.6 => /lib/x86_64-linux-gnu/libhogweed.so.6
(0x00007ddd73d5b000)
        libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10
(0x00007ddd73cd7000)
        libffi.so.8 => /lib/x86_64-linux-gnu/libffi.so.8
(0x00007ddd73ccb000)


Running nm (list symbols from object files) on the
/usr/lib/x86_64-linux-gnu/naviserver/bin/nsdbpg.so doesn't provide any
feedback about having the the _Ns_ModuleInit symbol.

See here:
..
..
..
..

0000000000004d50 t AddCmds
0000000000004260 t BindRow
00000000000077f0 t blob_dml_file
0000000000007200 t blob_get
00000000000075e0 t blob_put
00000000000072e0 t blob_send_to_stream
00000000000041e0 t CloseDb
                 U close@GLIBC_2.2.5
000000000000b068 b completed.0
                 U __ctype_b_loc@GLIBC_2.3
                 w __cxa_finalize@GLIBC_2.2.5
000000000000b070 b dateStyle
0000000000006870 t DbFail
0000000000003f50 t DbType
0000000000007c70 t decode3
0000000000003ce0 t deregister_tm_clones
0000000000003d50 t __do_global_dtors_aux
000000000000a8d8 d __do_global_dtors_aux_fini_array_entry
000000000000b000 d __dso_handle
000000000000aa30 d _DYNAMIC
0000000000007b60 t encode3
0000000000007b30 t enc_one
                 U __errno_location@GLIBC_2.2.5
0000000000004370 t Exec
0000000000007d5c t _fini
0000000000004980 t Flush
0000000000003d90 t frame_dummy
000000000000a8d0 d __frame_dummy_init_array_entry
00000000000093a0 r __FRAME_END__
0000000000006fb0 t get_blob_tuples
                 U getenv@GLIBC_2.2.5
0000000000007c50 t get_one
0000000000004740 t GetRow
0000000000004940 t GetRowCount
000000000000ac60 d _GLOBAL_OFFSET_TABLE_
                 w __gmon_start__
0000000000008b50 r __GNU_EH_FRAME_HDR
000000000000b078 b id
0000000000003000 t _init
000000000000b080 b intTypePtr
                 U __isoc23_strtol@GLIBC_2.38
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
0000000000007a80 t linkedListElement_new
0000000000007ae0 t LinkedList_free_list
0000000000007ab0 t LinkedList_len
0000000000005370 t ListElementExternal
                 U ns_calloc
                 U Ns_ConfigGetValue
                 U Ns_ConnContentSent
                 U Ns_ConnWriteData
                 U Ns_Db0or1Row
                 U Ns_Db1Row
                 U Ns_DbDML
0000000000003da0 T Ns_DbDriverInit
                 U Ns_DbDriverName
                 U Ns_DbExec
                 U Ns_DbGetMinDuration
00000000000052c4 N nsdbpg.c.6eb8844b
                 U Ns_DbRegisterDriver
                 U Ns_DbSelect
                 U Ns_DbSetException
                 U Ns_DiffTime
                 U Ns_DStringExport
                 U Ns_DStringPrintf
                 U NS_EMPTY_STRING
                 U ns_free
                 U Ns_GetTime
                 U Ns_HttpParseHost2
                 U Ns_Log
                 U Ns_LogSeverityEnabled
                 U Ns_LogSqlDebug
                 U ns_malloc
0000000000008000 R Ns_ModuleVersion
                 U Ns_ObjvArgs
                 U Ns_ObjvObj
                 U Ns_ObjvSet
                 U Ns_ObjvString
                 U Ns_ParseObjv
0000000000004d00 T Ns_PgServerInit
                 U Ns_SetClearValues
                 U Ns_SetFree
                 U Ns_SetGet
                 U Ns_SetPutSz
                 U Ns_SetPutValueSz
                 U Ns_SubcmdObjv
                 U Ns_TclDbGetHandle
                 U Ns_TclEnterSet
                 U Ns_TclGetConn
                 U Ns_TclPrintfResult
                 U Ns_TclRegisterTrace
0000000000003f60 t OpenDb
                 U open@GLIBC_2.2.5
0000000000006a30 t parse_bind_variables
0000000000005260 t ParsedSQLDupInternalRep
00000000000051f0 t ParsedSQLFreeInternalRep
000000000000b040 d ParsedSQLObjType
00000000000052d0 t ParsedSQLSetFromAny
00000000000058a0 t PgBindDmlObjCmd
0000000000005fd0 t PgBindExecObjCmd
00000000000061e0 t PgBindObjCmd
0000000000005a50 t PgBindOneRowObjCmd
0000000000005e10 t PgBindSelectObjCmd
0000000000005c20 t PgBindZeroOrOneRowObjCmd
000000000000b020 D pgDbName
0000000000004dc0 t PgObjCmd
0000000000006440 t PgPrepareObjCmd
                 U PQbackendPID
                 U PQclear
                 U PQcmdTuples
                 U PQdb
                 U PQerrorMessage
                 U PQexec
                 U PQfinish
                 U PQfname
                 U PQfreemem
                 U PQftype
                 U PQgetlength
                 U PQgetvalue
                 U PQhost
                 U PQlibVersion
                 U PQnfields
                 U PQntuples
                 U PQoptions
                 U PQport
                 U PQresultErrorMessage
                 U PQresultStatus
                 U PQsetdbLogin
                 U PQstatus
                 U PQunescapeBytea
000000000000a960 d procs
                 U read@GLIBC_2.2.5
0000000000003d10 t register_tm_clones
00000000000049f0 t ResetHandle
0000000000004a90 t SetTransactionState
                 U __snprintf_chk@GLIBC_2.3.4
                 U __sprintf_chk@GLIBC_2.3.4
00000000000053c0 t SqlObjToString
                 U __stack_chk_fail@GLIBC_2.4
                 U strcasecmp@GLIBC_2.2.5
                 U __strcat_chk@GLIBC_2.3.4
                 U strchr@GLIBC_2.2.5
                 U strcmp@GLIBC_2.2.5
                 U __strcpy_chk@GLIBC_2.3.4
                 U strerror@GLIBC_2.2.5
                 U strlen@GLIBC_2.2.5
                 U strncasecmp@GLIBC_2.2.5
000000000000a8e0 d subcmds.0
00000000000062e1 N tclcmds.c.1cb70ca1
                 U Tcl_ConvertToType
                 U Tcl_CreateObjCommand
                 U Tcl_DictObjPut
                 U Tcl_DStringAppend
                 U Tcl_DStringFree
                 U Tcl_DStringInit
                 U Tcl_DStringResult
                 U Tcl_DStringSetLength
                 U Tcl_ExternalToUtfDString
                 U Tcl_GetByteArrayFromObj
                 U Tcl_GetIndexFromObjStruct
                 U Tcl_GetObjType
                 U Tcl_GetString
                 U Tcl_GetStringFromObj
                 U Tcl_GetVar2Ex
                 U Tcl_ListObjAppendElement
                 U Tcl_NewByteArrayObj
                 U Tcl_NewDictObj
                 U Tcl_NewIntObj
                 U Tcl_NewListObj
                 U Tcl_NewStringObj
                 U Tcl_Panic
                 U Tcl_SetObjResult
                 U Tcl_UtfToExternalDString
                 U Tcl_WrongNumArgs
000000000000b068 d __TMC_END__
                 U write@GLIBC_2.2.5
0000000000007560 t write_to_stream


Am I missing something ? or there is an issue with the Code?
[image: image.png]


Thanks
Regards,

Sassy Natan
972-(0)54-2203702

Re: [naviserver-devel] ns_set reform

[Brian F. <bri...@ai...>]

Thank you Gustaf for all your incredible work.

Regarding the commands you wish to deprecate, I see that we are still using ns_formvalueput in some very old, but still used, code.

All the best
Brian
________________________________
From: Gustaf Neumann (sslmail) <ne...@wu...>
Sent: Friday 22 November 2024 10:19 am
To: nav...@li... <nav...@li...>
Subject: Re: [naviserver-devel] ns_set reform

Dear all,

>From the previously state goals:

>   1) all commands tested in the regression test should be documented
>   2) all documented commands should be tested (at least syntax-tested)
>   3) all implemented commands should be tested
>   4) the parameters of the implemented commands should be documented and vice versa.
>   5) deprecated commands should be listed in the deprecated section of the command listing in the documentation, but not advertised in the documentation

At least 1 and 2 are finished by today. This week, i have added over 700 tests to the regression test, which was hard work and took all my resources. We have now 2608 tests in the NaviServer 5 regression test vs. 1893 in the newest 4.99 branch:

 % egrep -r --include=*.test '^test ' /usr/local/src/naviserver/ |wc -l
 2608

 % egrep -r --include=*.test '^test ' /usr/local/src/naviserver-4.99/ |wc -l
 1893

The current state is not perfect, but a progress.

Tested commands not documented
------------> 706 tested commands, 72 deprecated,  648 documented, 0 NOT documented

Documented commands not tested
------------> 648 documented commands, 648 tested, 0 NOT tested

Deprecated commands which are documented
------------> 0 deprecated commands are documented

The next goal is to check the alignment of the argument lists of the implementation with the documentation.

On my wishlist is also a non-deprecated mode switchable via configuration file, where NaviServer offers only non-deprecated commands.

In the meanwhile, something to think about for you: We have still a bunch of commands, which have a questionable future. These commands are in a "TBD" state for 19 years.

I think, we should deprecate these commands (see below for the list) in the NaviServer 5 release.
Please speak up, if your applications need these:

    ns_browsermatch
    ns_choosecharset
    ns_cookiecharset
    ns_formfieldcharset
    ns_formvalueput
    ns_paren
    ns_tagelement
    ns_tagelementset

All the best
-g

Re: [naviserver-devel] ns_set reform

[Sassy N. <sa...@gm...>]

Thank you.

This project is so impressive and was really introduced things head off its
time.
I’m working with Navi since 2002 when node js, python and all other new
programming languages was never adapted the method of being the web server
embedded in the language itself.
AOL or Navi was from day one like this.

When is the plan to release version 5.0?


Regards,

Sassy Natan
972-(0)54-2203702


On Fri, 22 Nov 2024 at 12:25 Maksym Zinchenko <siq...@gm...> wrote:

> Hello, thank you for hard work. And I think nobody using this commands any
> more.
>
> On Fri, 22 Nov 2024, 09:20 Gustaf Neumann (sslmail), <ne...@wu...>
> wrote:
>
>> Dear all,
>>
>> From the previously state goals:
>>
>> >   1) all commands tested in the regression test should be documented
>> >   2) all documented commands should be tested (at least syntax-tested)
>> >   3) all implemented commands should be tested
>> >   4) the parameters of the implemented commands should be documented
>> and vice versa.
>> >   5) deprecated commands should be listed in the deprecated section of
>> the command listing in the documentation, but not advertised in the
>> documentation
>>
>> At least 1 and 2 are finished by today. This week, i have added over 700
>> tests to the regression test, which was hard work and took all my
>> resources. We have now 2608 tests in the NaviServer 5 regression test vs.
>> 1893 in the newest 4.99 branch:
>>
>>  % egrep -r --include=*.test '^test ' /usr/local/src/naviserver/ |wc -l
>>  2608
>>
>>  % egrep -r --include=*.test '^test ' /usr/local/src/naviserver-4.99/ |wc
>> -l
>>  1893
>>
>>
>> The current state is not perfect, but a progress.
>>
>> Tested commands not documented
>> ------------> 706 tested commands, 72 deprecated,  648 documented, 0 NOT
>> documented
>>
>> Documented commands not tested
>> ------------> 648 documented commands, 648 tested, 0 NOT tested
>>
>> Deprecated commands which are documented
>> ------------> 0 deprecated commands are documented
>>
>>
>> The next goal is to check the alignment of the argument lists of the
>> implementation with the documentation.
>>
>> On my wishlist is also a non-deprecated mode switchable via configuration
>> file, where NaviServer offers only non-deprecated commands.
>>
>> In the meanwhile, something to think about for you: We have still a bunch
>> of commands, which have a questionable future. These commands are in a
>> "TBD" state for 19 years.
>>
>> I think, we should deprecate these commands (see below for the list) in
>> the NaviServer 5 release.
>> Please speak up, if your applications need these:
>>
>>     ns_browsermatch
>>     ns_choosecharset
>>     ns_cookiecharset
>>     ns_formfieldcharset
>>     ns_formvalueput
>>     ns_paren
>>     ns_tagelement
>>     ns_tagelementset
>>
>> All the best
>> -g
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] ns_set reform

[Maksym Z. <siq...@gm...>]

Hello, thank you for hard work. And I think nobody using this commands any
more.

On Fri, 22 Nov 2024, 09:20 Gustaf Neumann (sslmail), <ne...@wu...>
wrote:

> Dear all,
>
> From the previously state goals:
>
> >   1) all commands tested in the regression test should be documented
> >   2) all documented commands should be tested (at least syntax-tested)
> >   3) all implemented commands should be tested
> >   4) the parameters of the implemented commands should be documented and
> vice versa.
> >   5) deprecated commands should be listed in the deprecated section of
> the command listing in the documentation, but not advertised in the
> documentation
>
> At least 1 and 2 are finished by today. This week, i have added over 700
> tests to the regression test, which was hard work and took all my
> resources. We have now 2608 tests in the NaviServer 5 regression test vs.
> 1893 in the newest 4.99 branch:
>
>  % egrep -r --include=*.test '^test ' /usr/local/src/naviserver/ |wc -l
>  2608
>
>  % egrep -r --include=*.test '^test ' /usr/local/src/naviserver-4.99/ |wc
> -l
>  1893
>
>
> The current state is not perfect, but a progress.
>
> Tested commands not documented
> ------------> 706 tested commands, 72 deprecated,  648 documented, 0 NOT
> documented
>
> Documented commands not tested
> ------------> 648 documented commands, 648 tested, 0 NOT tested
>
> Deprecated commands which are documented
> ------------> 0 deprecated commands are documented
>
>
> The next goal is to check the alignment of the argument lists of the
> implementation with the documentation.
>
> On my wishlist is also a non-deprecated mode switchable via configuration
> file, where NaviServer offers only non-deprecated commands.
>
> In the meanwhile, something to think about for you: We have still a bunch
> of commands, which have a questionable future. These commands are in a
> "TBD" state for 19 years.
>
> I think, we should deprecate these commands (see below for the list) in
> the NaviServer 5 release.
> Please speak up, if your applications need these:
>
>     ns_browsermatch
>     ns_choosecharset
>     ns_cookiecharset
>     ns_formfieldcharset
>     ns_formvalueput
>     ns_paren
>     ns_tagelement
>     ns_tagelementset
>
> All the best
> -g
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] ns_set reform

[Gustaf N. (sslmail) <ne...@wu...>]

Dear all,

From the previously state goals:

>   1) all commands tested in the regression test should be documented
>   2) all documented commands should be tested (at least syntax-tested)
>   3) all implemented commands should be tested
>   4) the parameters of the implemented commands should be documented and vice versa.
>   5) deprecated commands should be listed in the deprecated section of the command listing in the documentation, but not advertised in the documentation

At least 1 and 2 are finished by today. This week, i have added over 700 tests to the regression test, which was hard work and took all my resources. We have now 2608 tests in the NaviServer 5 regression test vs. 1893 in the newest 4.99 branch:

 % egrep -r --include=*.test '^test ' /usr/local/src/naviserver/ |wc -l
 2608

 % egrep -r --include=*.test '^test ' /usr/local/src/naviserver-4.99/ |wc -l
 1893

The current state is not perfect, but a progress.

Tested commands not documented
------------> 706 tested commands, 72 deprecated,  648 documented, 0 NOT documented

Documented commands not tested
------------> 648 documented commands, 648 tested, 0 NOT tested

Deprecated commands which are documented
------------> 0 deprecated commands are documented

The next goal is to check the alignment of the argument lists of the implementation with the documentation.

On my wishlist is also a non-deprecated mode switchable via configuration file, where NaviServer offers only non-deprecated commands.

In the meanwhile, something to think about for you: We have still a bunch of commands, which have a questionable future. These commands are in a "TBD" state for 19 years.

I think, we should deprecate these commands (see below for the list) in the NaviServer 5 release.
Please speak up, if your applications need these:

    ns_browsermatch
    ns_choosecharset
    ns_cookiecharset
    ns_formfieldcharset
    ns_formvalueput
    ns_paren
    ns_tagelement
    ns_tagelementset

All the best
-g

Re: [naviserver-devel] ns_set reform

[Gustaf N. (sslmail) <ne...@wu...>]

> However, based on a quick grep, there are still several hundred syntax tests missing. 

Some more precise data: We have in NaviServer altogether 638 (!) documented commands including subcommands,

Especially there are still many subcommands, which are not syntax tested. Yesterday and today, i added about 100 further tests to the regression test.
Current State: there are 422 documented commands tested, and 216 still not (syntax) tested. There is probably some more legacy stuff in the documentation that should be removed.

all the best
-g

Re: [naviserver-devel] ns_set reform

[Gustaf N. (sslmail) <ne...@wu...>]

Dear all,

i’ve implemented the changes with case-insensitive ns_sets, and they are working nicely.  These changes are highly backwards compatible, no changes were e.g. necessary for OpenACS.

The new version is still verbose, when e.g. a “get” or “update” operation happens on a case-insensitive set (e.g. HTTP headers) to ease fixing such cases also for older NaviServer versions, but that will go away for the release,

Below are the old and new man pages for “ns_set”, the new man page is mostly a rewrite.

   NaviServer 4.99: https://naviserver.sourceforge.io/4.99/naviserver/files/ns_set.html
   NaviServer 5: https://naviserver.sourceforge.io/5.0/naviserver/files/ns_set.html

There are more places, where case-insensitive ns_sets should be used (ADP parser for attributes since HTML attributes are case-insensitive, or query-results from SQL), but I have not investigated enough for these.

Inspired by the previous state of the documentation of ns_set, I addressed the long-standing issue, that in usage messages (when commands a called incorrectly), did not differentiate between literal command elements and placeholders. In the new version, placeholders are enclosed in slashes (/), making it evident which parts are variable inputs.

   OLD: ns_ip properties ipaddr
   NEW: ns_ip properties /ipaddr/

Similarly, the arguments of parameters in the usage messages were often not informative about possible values, which are now included were we have internally the information

   OLD: ns_getcookie ?-all all? ?-include_set_cookies include_set_cookies? ?--? name ?default? 
   NEW: ns_getcookie ?-all true|false? ?-include_set_cookies true|false? ?--? /name/ ?default?

   OLD: ns_deletecookie ... ?-samesite samesite? … 
   NEW: ns_deletecookie ... ?-samesite strict|lax|none? …

For more details, see commit [1] . While i was at this area, i found many things in the documentation which should be fixed before the release. The goal for the release should be:

   1) all commands tested in the regression test should be documented
   2) all documented commands should be tested (at least syntax-tested)
   3) all implemented commands should be tested
   4) the parameters of the implemented commands should be documented and vice versa.
   5) deprecated commands should be listed in the deprecated section of the command listing in the documentation, but not advertised in the documentation

The state in previous releases (including AOLserver releases) is really not good. For example, we have commands documented which were removed at least 22 years ago from NaviServer(AOLserver), such as 

     ns_db close
     ns_db open
 
I have already added over 50 new tests to the regression test suite to cover usage messages. However, based on a quick grep, there are still several hundred syntax tests missing. If we don't address1-5 now, before the release of the new major version, it's highly likely that we never will.

All the best.
-g

[1] https://github.com/naviserver-project/naviserver/commit/ffbd32774db1fe4a5b24a11b0032f144bec45cf7

> On 08.11.2024, at 17:35, Brian Fenton <bri...@ai...> wrote:
> 
> This change sounds very welcome. I think we've all experienced the frustration of case-insensitive searching.
> 
> One suggestion I have for the man page would be a beginner friendly example of iterating through the contents of an ns_set e.g.
> 
> set headers [ns_conn headers]
> set output ""
> foreach {key value} [ns_set array $headers] {
>   append output "$key = $value \n"    
> }
> set output
> 
> I know we have ns_set print, but the first time I encountered sets an example like the above would have been helpful.

Re: [naviserver-devel] ns_set reform

[Brian F. <bri...@ai...>]

This change sounds very welcome. I think we've all experienced the frustration of case-insensitive searching.

One suggestion I have for the man page would be a beginner friendly example of iterating through the contents of an ns_set e.g.

set headers [ns_conn headers]
set output ""
foreach {key value} [ns_set array $headers] {
  append output "$key = $value \n"
}
set output

I know we have ns_set print, but the first time I encountered sets an example like the above would have been helpful.

All the best
Brian
________________________________
From: Gustaf Neumann (sslmail) <ne...@wu...>
Sent: Thursday 7 November 2024 10:13 am
To: nav...@li... <nav...@li...>
Subject: [naviserver-devel] ns_set reform

Dear all,

The upcoming release of NaviServer 5 includes several enhancements to "ns_set", such as more efficient storage management and the addition of the "-all" option for "ns_set get" and "ns_set iget". I also plan to implement further improvements.

One common problem area is the handling of "ns_set" collections with case-insensitive keys (e.g., HTTP request and reply headers, or configuration values). Currently, developers must use the i* variants of commands whenever accessing or updating elements with case-insensitive keys.

This current approach has two main issues:

a) Semantic Issues: When a developer forgets to use the i* variant for such sets errors will occur that are difficult to debug. These errors may happen at both the C level and the Tcl level, affecting the main NaviServer code as well as the NaviServer modules. In the past few days, I have fixed more than ten such cases, which is why I am now addressing this issue.

b) Performance Issues: Using the i* variants (e.g., "ns_set iget”) leads to keys being searched sequentially using strcasecmp(). For a set with N elements, an average of N/2 strcasecmp() operations are performed per lookup or update operation. In other words, during each get/update/delete operation, approximately half of the dictionary is converted to lowercase. Additionally, the input key is repeatedly converted to lowercase. Benchmark comparisons indicate that strcasecmp() is 20 to 30 times slower than strcmp(). The heavy reliance on strcasecmp() in the implementation of Ns_Set certainly impacts performance.

To address these problems, I have added a property to the Ns_Set structure that allows it to be declared case-insensitive. With this property enabled, keys are stored in lowercase, enabling the use of
strcmp() instead of strcasemp() for comparisons. This change not only improves performance but also reduces the likelihood of semantic errors, as developers no longer need to remember to use the i* variants.

So far, I have made the configuration value sets and HTTP header sets case-insensitive:

 - HTTP Headers: According to the RFC, the names of header fields are case-insensitive. Now, these headers are stored in lowercase, ensuring consistent behavior.

 - Configuration Sets: Most operations were already case-insensitive,  but a few were not, which I regard as bugs.

The Ns_Sets for both cases are created from the C level. The only behavioral change is that when obtaining all keys from such a set, they are now returned in lowercase. For case-insensitive sets, the operations "ns_set get" and "ns_set iget will" return the same results.

In future versions, we might consider other options for ns_sets, such as enforcing uniqueness or sorting. However, I do not regard these as important as the "-nocase" option for now.

Some preliminary data: For a single HTTP request in OpenACS, I count 1,672 strcasecmp() operations. By utilizing the "-nocase" options on HTTP headers and ns_config data, this number is reduced for the same request to just 195 operations per request. On some of our busier systems that process up to 1,000 requests per second, the nocase option on sets saves nearly 1.5 million strcasecmp() operations per second.

I am also considering adding a "-nocase" flag to the base operations (i.e. allowing "ns_set get -nocase ..." in addition to "ns_set iget ...").  The "-nocase" flag is common in the Tcl world and appears more familiar than using strange names like: "icput", "idelkey", "ifind”, "iget", "imerge", "iunique", and "iupdate",

Finally, the man page for "ns_set" is not very user-friendly, as the base case and the "-nocase" variant should be presented together and not require excessive scrolling. The i* variants will continue to work and might be marked as deprecated in the future.

As always, your input is welcome. If you have suggestions about maybe more annoying legacy operations, please let me know, the release change is a chance to address such things.

All the best
-g

_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] ns_set reform

[Gustaf N. (sslmail) <ne...@wu...>]

Dear all,

The upcoming release of NaviServer 5 includes several enhancements to "ns_set", such as more efficient storage management and the addition of the "-all" option for "ns_set get" and "ns_set iget". I also plan to implement further improvements.

One common problem area is the handling of "ns_set" collections with case-insensitive keys (e.g., HTTP request and reply headers, or configuration values). Currently, developers must use the i* variants of commands whenever accessing or updating elements with case-insensitive keys.

This current approach has two main issues:

a) Semantic Issues: When a developer forgets to use the i* variant for such sets errors will occur that are difficult to debug. These errors may happen at both the C level and the Tcl level, affecting the main NaviServer code as well as the NaviServer modules. In the past few days, I have fixed more than ten such cases, which is why I am now addressing this issue.

b) Performance Issues: Using the i* variants (e.g., "ns_set iget”) leads to keys being searched sequentially using strcasecmp(). For a set with N elements, an average of N/2 strcasecmp() operations are performed per lookup or update operation. In other words, during each get/update/delete operation, approximately half of the dictionary is converted to lowercase. Additionally, the input key is repeatedly converted to lowercase. Benchmark comparisons indicate that strcasecmp() is 20 to 30 times slower than strcmp(). The heavy reliance on strcasecmp() in the implementation of Ns_Set certainly impacts performance.

To address these problems, I have added a property to the Ns_Set structure that allows it to be declared case-insensitive. With this property enabled, keys are stored in lowercase, enabling the use of
strcmp() instead of strcasemp() for comparisons. This change not only improves performance but also reduces the likelihood of semantic errors, as developers no longer need to remember to use the i* variants.

So far, I have made the configuration value sets and HTTP header sets case-insensitive:

 - HTTP Headers: According to the RFC, the names of header fields are case-insensitive. Now, these headers are stored in lowercase, ensuring consistent behavior.

 - Configuration Sets: Most operations were already case-insensitive,  but a few were not, which I regard as bugs. 

The Ns_Sets for both cases are created from the C level. The only behavioral change is that when obtaining all keys from such a set, they are now returned in lowercase. For case-insensitive sets, the operations "ns_set get" and "ns_set iget will" return the same results.

In future versions, we might consider other options for ns_sets, such as enforcing uniqueness or sorting. However, I do not regard these as important as the "-nocase" option for now.

Some preliminary data: For a single HTTP request in OpenACS, I count 1,672 strcasecmp() operations. By utilizing the "-nocase" options on HTTP headers and ns_config data, this number is reduced for the same request to just 195 operations per request. On some of our busier systems that process up to 1,000 requests per second, the nocase option on sets saves nearly 1.5 million strcasecmp() operations per second.

I am also considering adding a "-nocase" flag to the base operations (i.e. allowing "ns_set get -nocase ..." in addition to "ns_set iget ...").  The "-nocase" flag is common in the Tcl world and appears more familiar than using strange names like: "icput", "idelkey", "ifind”, "iget", "imerge", "iunique", and "iupdate",

Finally, the man page for "ns_set" is not very user-friendly, as the base case and the "-nocase" variant should be presented together and not require excessive scrolling. The i* variants will continue to work and might be marked as deprecated in the future.

As always, your input is welcome. If you have suggestions about maybe more annoying legacy operations, please let me know, the release change is a chance to address such things.

All the best
-g

Re: [naviserver-devel] Fwd: [TCLCORE] Monday noon TCL/Tk meeting report

[Zoran V. <zv...@ar...>]

On 29.10.24 18:27, Gustaf Neumann (sslmail) wrote:
> see below for the current ambitious plans for the Tcl roadmap.
> The core-team puts a strong emphasis on Tcl9 and names already end-of-support times for Tcl 8.*
> 

This is VERY ambitious in the sense that most of those (TCT)
people do not existentialy rely on the code quality, as we do.
We have a company and large installation base (mainly large
media companies) and cannot afford destabilisation.
So, for us, the 8.6 is here to stay for a long(er) time.
If needed we will "cook our own soup" i.e. completely
detach from the current/future developments. I see that comming.

Cheers,
Zoran

[naviserver-devel] Fwd: [TCLCORE] Monday noon TCL/Tk meeting report

[Gustaf N. (sslmail) <ne...@wu...>]

Dear NaviServer community,

see below for the current ambitious plans for the Tcl roadmap.
The core-team puts a strong emphasis on Tcl9 and names already end-of-support times for Tcl 8.*

all the best
-g

> Begin forwarded message:
> 
> From: Harald Oehlmann <har...@el...>
> Subject: [TCLCORE] Monday noon TCL/Tk meeting report
> Date: 28. October 2024 at 14:33:20 GMT+1
> To: Tcl Core List <tcl...@li...>
> 
> Dear TCL/Tk/OpenACS/Naviserver team,
> 
> we had a great meeting today with: Don, Jan, Rolf, Paul, Brian, Massimo, Ashok, Andreas Kupries, Andreas Leitgreb, Mark, Steve, Harald.
> 
> The results are:
> 
> R1) TCL/Tk 9.0.0 release is great ! Thanks for all contributors !
> 
> R2) TCL/Tk 9.0.1 is the next step
>  - release process starts in December 2024
>  - developed in main branches
>  - address double-distribution of 8.6 and 9.0 by Linux maintainers
>  - TIP 701 (tilde API) included
>  - TIP 700 (MD documentation) eventually started
> 
> R3) TCL 8.6 support
>  - proposed support time: until end 2025
>  - all major packages and distributions should be ported to 9.0
>  - but some will only switch, if 8.6 declared as unsupported
>  - specially Tk 9 offers so much practical value, that Tk8.6 is seen as obsolete (scaled interface, updated themes)
>  - no new features for 8.6, sorry, only bug back-porting
> 
> R4) TCL 8.7 release
>  - discussion point
>  - most just want to focus on 9.0
>  - 8.6 is easy for migration for 8.6 packages:
>    - TCL_CHAR size with 16 bit interface is available
>    - TCL_CHAR size with 32 bit interface (like 9.0) is also available
>    - No surrogates (like 8.6.11+)
>  - 8.7 is better than 8.6 as compatible 9.0 extensions are included (full Unicode)
>  - major player like AndroWish are supposed to directly go to 9.0
>  - if released, 8.6 should be taken out of maintenance to avoid double-burden
>  - first possible release in 1 year. All bugfixes of 9.0 are back-ported, so 8.7. is getting better
>  - Debian co-maintainer pointed out, that distributions do not have resources for multiple versions. So, only one version is best, max 2, not 3.
> 
> R5) TCL 9.1 release
>  - will be started when 9.0 is stable (not now)
>  - TIP 626 (long number of command arguments) is binary incompatible and will wait for 9.1
>  - now we have long strings and lists, we should work on its performance.
>  - decodings currently also have performance issues and are sometimes issued multiple times
>  - drop support for ITCL
> 
> R6) Decoupling of TCL and Tk release
>  - There is a strong support to decouple TCL and Tk (minor) releases
>  - Don is ready to release Tk separately at any time
>  - Major Tk people were missing in the call
>  - Eventually own infra-structure: TIP, Team
>  It would be great to build-up a Tk team and a working way.
>  Any people for a task force?
> 
> R7) Documentation
>  - The new documentation project is seen as important to get wider audience
>  - Help is proposed to advance - Torsten must not do all the work
> 
> R8) Extensions
>  - Porting all extensions to 9 is a main step to put energy in
>  - Get tests for larger strings/lists
>  - make valgrind cleaner for extensions
>  - Massimo is the Debian maintainer for TDBC. He proposed (before) to be the tdbc packages maintainer. I would say: yes, just do it, great, applause!
> 
> It is proposed to repeat those meetings each two weeks on Mondays at 12:00 UTC. The next meeting will take place 11th of November 2025.
> 
> Thank you all, you guys rock !
> Harald
> 
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

[naviserver-devel] regression tests now also with tcl 8.5

[Gustaf N. (sslmail) <ne...@wu...>]

The regression test on github runs now also the tests on tcl8.5 (in addition to 8.6.15, 8.7a5, 9.0). 
This makes sure, that recent changes keep 8.5 compatibility.

Below are the rest running after every push operation.
core-8-5-branch is a post 8.5.19 version containing several fixes.

all the best
-g


          - os: ubuntu-latest
            compiler: gcc-10
            tcltag: core-8-6-15
            nsf_version: 2.3.0
            tdom_version: 0.9.1

          - os: ubuntu-latest
            compiler: gcc-12
            tcltag: core-8-7-a5
            ns_modules: nsdbpg nsdbi nsdbipg nsudp nscoap nssmtpd
            nsf_version: 2.4.0
            tdom_version: 0.9.4

          - os: ubuntu-latest
            compiler: gcc-12
            tcltag: core-8-5-branch
            nsf_version: HEAD
            tcllib_version: "2.0"
            ns_modules: nsdbpg nsdbi nsdbipg nsudp nscoap nssmtpd nsloopctl
            tdom_version: 0.9.4

          - os: ubuntu-latest
            compiler: gcc-12
            tcltag: 9.0.0
            nsf_version: HEAD
            tcllib_version: "2.0"
            ns_modules: nsdbpg nsdbi nsdbipg nsudp nscoap nssmtpd nsloopctl
            tdom_version: 0.9.4

Re: [naviserver-devel] ns_mktemp

[Zoran V. <zv...@ar...>]

On 08.10.24 17:49, Andrew Piskorski wrote:
> Have you heard of anybody
> who still really WANTS to stay on Tcl 8.5 for some reason?

I do :-)
But I am not in any way married to the server's public
version so I guess I do not count. At some (later) point
we will move one tick up the ladder and use Tcl 8.6 and
will stay there for some years to come.

Cheers
Zoran

Re: [naviserver-devel] ns_mktemp

[Andrew P. <at...@pi...>]

On Tue, Oct 08, 2024 at 04:58:11PM +0200, Gustaf Neumann (sslmail) wrote:

> What speaks against using the Tcl scripted approach is that so far, we support NaviServer with tcl8.5. ???file tempfile??? was introduced in Tcl 8.6.

Ah, I hadn't realized 'file tempfile' was that new.  Maybe just wait
until NaviServer requires Tcl 8.6 or later?  Have you heard of anybody
who still really WANTS to stay on Tcl 8.5 for some reason?

> Furthermore, TclUnixOpenTemporaryFile() is just for Unix, and not part of the public API (starting with ???Tcl_???). 

Well you'd never directly call TclUnixOpenTemporaryFile() of course.
That's just how TclFileTemporaryCmd() and TclpOpenTemporaryFile() are
implemented underneath.  The Windows implemenation uses GetTempPathW()
instead of mkstemp(), but presumably works the same at the Tcl level,
and in C when calling TclpOpenTemporaryFile().

> But providing this support is not a big thing, since we have in NaviServer since a while support for mkstemp() also for windows (named ns_mkstemp)), since we use this call internally on several places. I???ll cook something up which is most version and platform independent, backward compatible, providing depreciation messages, which can be turned off for ???hopeless cases??? when admins get annoyed about to many depreciation messages.

That sounds great!  Thanks for taking care of this so proactively.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] ns_mktemp

[Gustaf N. (sslmail) <ne...@wu...>]

On 07.10.2024, at 17:59, Andrew Piskorski <at...@pi...> wrote:
> 
> On Mon, Oct 07, 2024 at 12:54:32PM +0200, Gustaf Neumann (sslmail) wrote:
> 
>> However, there are many cases, where existing programs use "ns_mkstemp", which cannot be replaced easily. When looking at OpenACS, I see 33 cases like
>> 
>> - the temporary name is passed to an external program (e.g. "tar", "zip", image creation), or
>> - the temporary name is passed to a Tcl function expecting a filename (e.g. "file copy").
> 
> Interesting.  It sounds like are no 100% good solutions for this, and
> in fact there CAN'T be.

This is right in the sense of “there can’t be a solution free of possible race conditions unless the code is refactored” (e.g. let the NaviServer application write to the temp file rather than the external program)

> 
> I'd lean towards turning ns_mktemp into a wrapper around Tcl's "file
> tempfile".  That calls TclUnixOpenTemporaryFile() and thus mkstemp()
> or mkstemps(), but it looks like Tcl has already done the necessary
> work

What speaks against using the Tcl scripted approach is that so far, we support NaviServer with tcl8.5. “file tempfile” was introduced in Tcl 8.6. Furthermore, TclUnixOpenTemporaryFile() is just for Unix, and not part of the public API (starting with “Tcl_”). 

But providing this support is not a big thing, since we have in NaviServer since a while support for mkstemp() also for windows (named ns_mkstemp)), since we use this call internally on several places. I’ll cook something up which is most version and platform independent, backward compatible, providing depreciation messages, which can be turned off for “hopeless cases” when admins get annoyed about to many depreciation messages.

all the best
-g

Re: [naviserver-devel] ns_mktemp

[Wolfgang W. <wol...@di...>]

I like the solution suggested by Andrew as well. This should allow an 
easy transition of existing code without sacrificing security.

Am 07.10.24 um 17:59 schrieb Andrew Piskorski:
> On Mon, Oct 07, 2024 at 12:54:32PM +0200, Gustaf Neumann (sslmail) wrote:
>
>> However, there are many cases, where existing programs use "ns_mkstemp", which cannot be replaced easily. When looking at OpenACS, I see 33 cases like
>>
>> - the temporary name is passed to an external program (e.g. "tar", "zip", image creation), or
>> - the temporary name is passed to a Tcl function expecting a filename (e.g. "file copy").
> Interesting.  It sounds like are no 100% good solutions for this, and
> in fact there CAN'T be.
>
>> - Call the safe function (e.g. mkstemp()) and delete the file, while
>>    producing a depreciation message? This could also be done on the Tcl-level.
> I guess so.  Probably with a switch to control whether the file is
> created and remains (new-style behavior), or gets quickly deleted
> (more fully backward compatible).
>
> I'd lean towards turning ns_mktemp into a wrapper around Tcl's "file
> tempfile".  That calls TclUnixOpenTemporaryFile() and thus mkstemp()
> or mkstemps(), but it looks like Tcl has already done the necessary
> work to easily make the wrapper backwards compatible with the
> old-style ns_mktemp.  And having the compatibility wrapper in Tcl
> instead of C makes it a lot easier for NaviServer users to adjust it
> to their own needs if necessary.
>

Re: [naviserver-devel] ns_mktemp

[Brian F. <bri...@ai...>]

I agree with Andrew's suggested approach here.

Brian
________________________________
From: Andrew Piskorski <at...@pi...>
Sent: Monday 7 October 2024 4:59 pm
To: nav...@li... <nav...@li...>
Subject: Re: [naviserver-devel] ns_mktemp

On Mon, Oct 07, 2024 at 12:54:32PM +0200, Gustaf Neumann (sslmail) wrote:

> However, there are many cases, where existing programs use "ns_mkstemp", which cannot be replaced easily. When looking at OpenACS, I see 33 cases like
>
> - the temporary name is passed to an external program (e.g. "tar", "zip", image creation), or
> - the temporary name is passed to a Tcl function expecting a filename (e.g. "file copy").

Interesting.  It sounds like are no 100% good solutions for this, and
in fact there CAN'T be.

> - Call the safe function (e.g. mkstemp()) and delete the file, while
>   producing a depreciation message? This could also be done on the Tcl-level.

I guess so.  Probably with a switch to control whether the file is
created and remains (new-style behavior), or gets quickly deleted
(more fully backward compatible).

I'd lean towards turning ns_mktemp into a wrapper around Tcl's "file
tempfile".  That calls TclUnixOpenTemporaryFile() and thus mkstemp()
or mkstemps(), but it looks like Tcl has already done the necessary
work to easily make the wrapper backwards compatible with the
old-style ns_mktemp.  And having the compatibility wrapper in Tcl
instead of C makes it a lot easier for NaviServer users to adjust it
to their own needs if necessary.

--
Andrew Piskorski <at...@pi...>


_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] ns_mktemp

[Georg L. <jor...@ma...>]

On 10/7/24 12:54, Gustaf Neumann (sslmail) wrote:
> Dear all.
> [..] However, there are many cases, where existing programs use 
> "ns_mkstemp", which cannot be replaced easily. When looking at 
> OpenACS, I see 33 cases like - the temporary name is passed to an 
> external program (e.g. "tar", "zip", image creation), or - the 
> temporaryname is passed to a Tcl function expecting a filename (e.g. 
> "file copy"). So, dropping the support for "ns_mkstemp" fully is not a 
> good option. Also, providing a "home-cooked" version of "ns_mktemp" is 
> not good either (both in Tcl or in C), since technically speaking, 
> this will not be better than the original function having the same 
> problems. Ignoring the compilation warning is not good either, since 
> sooner or later, the deprecated function will be removed. What should 
> we do? - place "ns_mktemp" into an external module? NaviServer will 
> compile nicely, but applications like OpenACS will have to load the 
> module, making administration and migration to NaviServer 5 less 
> smooth. - Call the safe function (e.g. mkstemp()) and delete the file, 
> while producing a depreciation message? This could also be done on the 
> Tcl-level.

I like this option best. It maintains backward compatibility for the 
application, encourages update to more secure approaches, discourages 
future use - especially when accompanied by respective hints in the 
documentation - and removes the warnings for up-to-date applications.

At some time in the future, the wrapped ns_mktemp could then be 
deprecated and moved out into an external module, which still allows 
legacy operations to continue using it, while raising the bar.

Best Regards,

   Georg

> Other options? Opinions? All the best
> -g [1] 
> https://pubs.opengroup.org/onlinepubs/009695399/functions/mktemp.html 
> [1] https://man.openbsd.org/OpenBSD-7.5/mkstemp.3
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] ns_mktemp

[Zoran V. <zv...@ar...>]

On 07.10.24 12:54, Gustaf Neumann (sslmail) wrote:
> So, dropping the support for "ns_mkstemp" fully is not a good option.

You mean ns_mktemp?

Re: [naviserver-devel] ns_mktemp

[Zoran V. <zv...@ar...>]

On 07.10.24 17:59, Andrew Piskorski wrote:
> I'd lean towards turning ns_mktemp into a wrapper around Tcl's "file
> tempfile".  That calls TclUnixOpenTemporaryFile() and thus mkstemp()
> or mkstemps(), but it looks like Tcl has already done the necessary
> work to easily make the wrapper backwards compatible with the
> old-style ns_mktemp.

Well, if this is so as stated above, then it is a no-brainer!

Re: [naviserver-devel] ns_mktemp

[Andrew P. <at...@pi...>]

On Mon, Oct 07, 2024 at 12:54:32PM +0200, Gustaf Neumann (sslmail) wrote:

> However, there are many cases, where existing programs use "ns_mkstemp", which cannot be replaced easily. When looking at OpenACS, I see 33 cases like
> 
> - the temporary name is passed to an external program (e.g. "tar", "zip", image creation), or
> - the temporary name is passed to a Tcl function expecting a filename (e.g. "file copy").

Interesting.  It sounds like are no 100% good solutions for this, and
in fact there CAN'T be.

> - Call the safe function (e.g. mkstemp()) and delete the file, while
>   producing a depreciation message? This could also be done on the Tcl-level.

I guess so.  Probably with a switch to control whether the file is
created and remains (new-style behavior), or gets quickly deleted
(more fully backward compatible).

I'd lean towards turning ns_mktemp into a wrapper around Tcl's "file
tempfile".  That calls TclUnixOpenTemporaryFile() and thus mkstemp()
or mkstemps(), but it looks like Tcl has already done the necessary
work to easily make the wrapper backwards compatible with the
old-style ns_mktemp.  And having the compatibility wrapper in Tcl
instead of C makes it a lot easier for NaviServer users to adjust it
to their own needs if necessary.

-- 
Andrew Piskorski <at...@pi...>

[naviserver-devel] ns_mktemp

[Gustaf N. (sslmail) <ne...@wu...>]

Dear all.

Since Tcl9 is out there, it becomes time for working towards the NaviServer 5 release. The most recent version of NaviServer from the main branch works nicely with the release of Tcl9 (for trying, you might consider the docker image "gustafn/naviserver/latest-tcl9-bookworm” from https://hub.docker.com/repository/docker/gustafn/naviserver/).

One of the open topics for the NaviServer 5 release is the following: The function “ns_mktemp" exists in NaviServer since ages. The function is based on the POSIX call mktemp() [1], which is unfortunately deprecated since a few years. When compiling NaviServer, one sees messages like the following:

tclfile.c:255:51: warning: 'mktemp' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of mktemp(3), it is highly recommended that you use mkstemp(3) instead. [-Wdeprecated-declarations]
  255 |         Tcl_SetObjResult(interp, Tcl_NewStringObj(mktemp(buffer), TCL_INDEX_NONE));
      |                                                   ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/_stdlib.h:210:1: note: 'mktemp' has been explicitly marked deprecated here
  210 | __deprecated_msg("This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of mktemp(3), it is highly recommended that you use mkstemp(3) instead.")
      | ^

Even worse, newer versions of OpenBSD produce warning messages whenever code is linked containing calls to mktemp() [2]. So, there is no future for mktemp() in NaviServer.

The problem with the C-level function mktemp() is that its usage leads to a race condition, opening space to attacks.  The problematic case happens when an application generates a file name with the mktemp() function and then create a file using this name. Unfortunately, this is not secure, because a different process may create a file with this name in the time between the call to mktemp() and the subsequent attempt to create the file by the first process. A malicious user can predict the name of the temporary file, resulting in other files being accessed, modified, corrupted, or deleted.

The recommended way is to combine the two steps and create the file in an atomic operation (POSIX mkstemp() for files and mkdtemp() for directories). These functions are used by Tcl "file tempfile" and by Tcl9 "file tempdir" and in NaviServer 5 via "ns_mkdtemp".

However, there are many cases, where existing programs use "ns_mkstemp", which cannot be replaced easily. When looking at OpenACS, I see 33 cases like

- the temporary name is passed to an external program (e.g. "tar", "zip", image creation), or
- the temporary name is passed to a Tcl function expecting a filename (e.g. "file copy").

So, dropping the support for "ns_mkstemp" fully is not a good option. Also, providing a "home-cooked" version of "ns_mktemp" is not good either (both in Tcl or in C), since technically speaking, this will not be better than the original function having the same problems. Ignoring the compilation warning is not good either, since sooner or later, the deprecated function will be removed.

What should we do?
- place "ns_mktemp" into an external module?
  NaviServer will compile nicely, but applications like OpenACS 
  will have to load the module, making administration and migration
  to NaviServer 5 less smooth.
- Call the safe function (e.g. mkstemp()) and delete the file, while
  producing a depreciation message? This could also be done on the Tcl-level.

Other options? Opinions?

All the best
-g

[1] https://pubs.opengroup.org/onlinepubs/009695399/functions/mktemp.html
[1] https://man.openbsd.org/OpenBSD-7.5/mkstemp.3

[naviserver-devel] NaviServer 4.99.31 available

[Gustaf N. <ne...@wu...>]

Dear all,

I am glad to announce that the release of NaviServer 4.99.30 is 
available at SourceForge [1] and on GitHub [2]. In essence, this release 
contains a few small bugfixes and some backported features from the 
NaviServer 5 developement (auto SNI and Improved handling of large file 
uploads, portability improvements). Since we want to release NaviServer 
5 after Tcl9, and Tcl9 is still not out, some of the backports become 
more urgent.


All the best!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.31/
[2] https://github.com/orgs/naviserver-project/repositories

=======================================
NaviServer 4.99.31, released 2024-08-11
=======================================

  71 files changed, 897 insertions(+), 303 deletions(-)

New Features:
-------------

   - Added auto-SNI support to "ns_http" and "ns_connchan open": When
     opening an TLS-connection to named target, the used hostname is
     automatically used as a default value for the SNI-host.  The
     automatically derived values has a lower priority than the
     parameter "-hostname" of "ns_http run" or "ns_connchan open".

     This change eases of use with cloud services and affects all
     packages using, e.g., "ns_http", including reverse proxies.

   - Introduced experimental command "ns_fseekchars": This changes
     Improves file-based parsing of multipart/form-data, supporting
     files larger than 4GB, reducing memory usage, and speeding up
     processing..


Bug Fixes:
----------

     - Fixed invalid memory reallocation of pool descriptors array
       [commit 7efaafe4].

     - Fixed potential issue for move operation with overlapping memory
       areas. This change is which was critical for ARM architecture
       when musl is used (Alpine Linux) [commit 20438ed4].

     - Fixed potential crash in "ns_conn copy" to handle cases where an
       incoming request contains no content [commit 79b8cb8a].


Changes in Sample Configuration Files:
--------------------------------------

     - Sample configuration file updates for handling container
       mappings and whitelisting domain names [commits ece849ac,
       9a102a97, 86a176ab].

     - Backported from NaviServer 5: Includes multiple changes to
       configuration files and scripts [commit 0d32e5ea].

Misc Improvements:
------------------

     - Improved forward compatibility to ease backports [commit ef8b4386].

     - Improved spelling and documentation across multiple files to
       enhance readability and understanding [multiple commits e.g.,
       a2cdaa10, 15253120].

     - Improved portability by not assuming that type "char" is the
       same as "unsigned char", important for compatibility with
       different architectures [commit 7635ef00].

     - Made testing more robust for docker images with partial IPv6
       setups to improve reliability in varied deployment environments
       [commit 516ca38c].

     - Removed misleading and unneeded "-encoding binary": This command
       was a no-op since many Tcl releases and was removed in Tcl9 at
       all. This change clarifies code usage [commit a36d15b1].

Re: [naviserver-devel] Problem with a Tcl extension and NaviServer

[Matthew B. <mm...@gm...>]

After doing a little digging, I tracked it down to a line in the USB
library; sure enough, IOHIDDeviceOpen returns a "privilege violation"
error. And just to confirm, if I run NaviServer as root, it all works fine.
So thanks again Gustaf for pointing me in the right direction.

Now I just need to puzzle out why there's no privilege violation when I run
the code via tclsh from the command line. Hopefully that will give me some
ideas for a fix since running NaviServer as root isn't really a practical
option.

Regards,

Matt



On Sat, Jun 29, 2024 at 8:48 AM <
nav...@li...> wrote:

> Date: Fri, 28 Jun 2024 12:41:30 -0400
> From: Matthew Burke <mm...@gm...>
> To: nav...@li...
> Subject: Re: [naviserver-devel] Problem with a Tcl extension and NaviServer
> Message-ID:
>         <
> CAC...@ma...>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Gustaf,
>
> Thanks for the reply. I'll start looking into permissions issues and see if
> anything there looks promising. I should also instrument the USB library
> and try to get a better sense of what, exactly, is failing.
>
> Regards,
>
> Matt
>
>
> On Thu, Jun 27, 2024 at 9:15?AM <
> nav...@li...> wrote:
>
> >
> > Dear Mathew,
> >
> > The answer for your this problem is probably in the call of
> > blink1_openById(devid);
> >
> > Without looking into this call, I would suspect first from my experience
> > with NaviServer IoT projects a permission problem when accessing the
> > hardware interface.
> > For security reasons, NaviServer switches to a nonprivileged user ...
> >
> > all the best -g
> >
> > On 25.06.24 16:18, Matthew Burke wrote:
> > > Hello!
> > >
> > > I am writing a Tcl extension in C to wrap a driver for a USB LED
> > > status light. The extension runs fine from tclsh but there are issues
> > > running it from a Tcl ?page under NaviServer.
> > >
> > > The extension implements one command via Tcl_CreateObjCommand. The
> > > command proc implements a number of sub-commands broken into two
> > > categories: device-independent functionality, e.g. count the number of
> > > status lights that are plugged in, and device-dependent functionality,
> > > e.g. attach to a status light, display a particular color, etc
> > >
> > > The device-independent sub-commands work when run in a Tcl page. For
> > > example, this page runs fine when served from NaviServer:
> > >
> > >
> > > package require blink
> > >
> > > set n [blink enumerate]
> > >
> > > ns_return 200 text/html "
> > > There are <strong>$n</strong> devices plugged in.
> > > "
> > >
> > >
> > > In order to have a device display a color, etc., you first have to run
> > > the "open" command. And that's not working. So here's an example Tcl
> > page:
> > >
> > >
> > > package require blink
> > >
> > > set d [blink open 0]
> > > set q [ns_conn query]
> > > set s [ns_parsequery $q]
> > >
> > > set red [ns_set get $s red 255]
> > > set green [ns_set get $s green 0]
> > > set blue [ns_set get $s blue 0]
> > >
> > > blink set $d $red $green $blue
> > >
> > > ns_return 200 text/html "
> > > ? ? <p>Query: [ns_conn query]</p>
> > >
> > > ? ? <ul>
> > > ? ? <li>Red: $red</li>
> > > ? ? <li>Green: $green</li>
> > > ? ? <li>Blue: $blue</li>
> > > ? ? </ul>
> > > "
> > >
> > >
> > > When this page is served, I get an error in the line: set d [blink
> > > open 0].
> > >
> > > There is a C struct, Blinker, ?that contains data about the attached
> > > device. My C code first allocates
> > > memory for the struct
> > >
> > >
> > > blinkPtr = (Blinker *)Tcl_Alloc(sizeof(Blinker));
> > >
> > >
> > > and this seems to work -- blinkPtr is non-NULL after the assignment.
> > > Then there is a function from the driver library that actually
> > > attaches to the device:
> > >
> > > blinkPtr->device = blink1_openById(devid);
> > >
> > > It should return the memory address of an allocated structure that's
> > > part of the USB library. But the?function?returns NULL. (Just to
> > > reiterate, this works fine when run from tclsh and returns a non-NULL
> > > value.)
> > >
> > > I'm assuming there is some sort of memory protection or similar that I
> > > need to take into account. Any suggestions on what to look at would be
> > > greatly appreciated.
> > >
> > > Thanks,
> > >
> > > Matt
> > >
> > > P.S. This is on macOS 14.3. I'm compiling the extension as follows:
> > >
> > > clang -dynamiclib -DUSE_TCL_STUBS -I/usr/local/ns/include
> > > -L/usr/local/ns/lib -ltclstub8.6 -o blink.dylib blink.c
> > > -L/usr/local/lib -lBlink1
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > naviserver-devel mailing list
> > > nav...@li...
> > > https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> >
> >
> >
> > ------------------------------
> >
> >
> >
> > ------------------------------
> >
> > Subject: Digest Footer
> >
> > _______________________________________________
> > naviserver-devel mailing list
> > nav...@li...
> > https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> >
> >
> > ------------------------------
> >
> > End of naviserver-devel Digest, Vol 162, Issue 3
> > ************************************************
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
>
> ------------------------------
>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>
> ------------------------------
>
> End of naviserver-devel Digest, Vol 162, Issue 4
> ************************************************
>

Re: [naviserver-devel] Problem with a Tcl extension and NaviServer

[Matthew B. <mm...@gm...>]

Hi Gustaf,

Thanks for the reply. I'll start looking into permissions issues and see if
anything there looks promising. I should also instrument the USB library
and try to get a better sense of what, exactly, is failing.

Regards,

Matt


On Thu, Jun 27, 2024 at 9:15 AM <
nav...@li...> wrote:

>
> Dear Mathew,
>
> The answer for your this problem is probably in the call of
> blink1_openById(devid);
>
> Without looking into this call, I would suspect first from my experience
> with NaviServer IoT projects a permission problem when accessing the
> hardware interface.
> For security reasons, NaviServer switches to a nonprivileged user ...
>
> all the best -g
>
> On 25.06.24 16:18, Matthew Burke wrote:
> > Hello!
> >
> > I am writing a Tcl extension in C to wrap a driver for a USB LED
> > status light. The extension runs fine from tclsh but there are issues
> > running it from a Tcl ?page under NaviServer.
> >
> > The extension implements one command via Tcl_CreateObjCommand. The
> > command proc implements a number of sub-commands broken into two
> > categories: device-independent functionality, e.g. count the number of
> > status lights that are plugged in, and device-dependent functionality,
> > e.g. attach to a status light, display a particular color, etc
> >
> > The device-independent sub-commands work when run in a Tcl page. For
> > example, this page runs fine when served from NaviServer:
> >
> >
> > package require blink
> >
> > set n [blink enumerate]
> >
> > ns_return 200 text/html "
> > There are <strong>$n</strong> devices plugged in.
> > "
> >
> >
> > In order to have a device display a color, etc., you first have to run
> > the "open" command. And that's not working. So here's an example Tcl
> page:
> >
> >
> > package require blink
> >
> > set d [blink open 0]
> > set q [ns_conn query]
> > set s [ns_parsequery $q]
> >
> > set red [ns_set get $s red 255]
> > set green [ns_set get $s green 0]
> > set blue [ns_set get $s blue 0]
> >
> > blink set $d $red $green $blue
> >
> > ns_return 200 text/html "
> > ? ? <p>Query: [ns_conn query]</p>
> >
> > ? ? <ul>
> > ? ? <li>Red: $red</li>
> > ? ? <li>Green: $green</li>
> > ? ? <li>Blue: $blue</li>
> > ? ? </ul>
> > "
> >
> >
> > When this page is served, I get an error in the line: set d [blink
> > open 0].
> >
> > There is a C struct, Blinker, ?that contains data about the attached
> > device. My C code first allocates
> > memory for the struct
> >
> >
> > blinkPtr = (Blinker *)Tcl_Alloc(sizeof(Blinker));
> >
> >
> > and this seems to work -- blinkPtr is non-NULL after the assignment.
> > Then there is a function from the driver library that actually
> > attaches to the device:
> >
> > blinkPtr->device = blink1_openById(devid);
> >
> > It should return the memory address of an allocated structure that's
> > part of the USB library. But the?function?returns NULL. (Just to
> > reiterate, this works fine when run from tclsh and returns a non-NULL
> > value.)
> >
> > I'm assuming there is some sort of memory protection or similar that I
> > need to take into account. Any suggestions on what to look at would be
> > greatly appreciated.
> >
> > Thanks,
> >
> > Matt
> >
> > P.S. This is on macOS 14.3. I'm compiling the extension as follows:
> >
> > clang -dynamiclib -DUSE_TCL_STUBS -I/usr/local/ns/include
> > -L/usr/local/ns/lib -ltclstub8.6 -o blink.dylib blink.c
> > -L/usr/local/lib -lBlink1
> >
> >
> >
> >
> > _______________________________________________
> > naviserver-devel mailing list
> > nav...@li...
> > https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>
>
> ------------------------------
>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>
> ------------------------------
>
> End of naviserver-devel Digest, Vol 162, Issue 3
> ************************************************
>