Menu
▾
▴
naviserver-devel — Developer discussion
You can subscribe to this list here.
| 2005 |
Jan
|
Feb
(53) |
Mar
(62) |
Apr
(88) |
May
(55) |
Jun
(204) |
Jul
(52) |
Aug
|
Sep
(1) |
Oct
(94) |
Nov
(15) |
Dec
(68) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2006 |
Jan
(130) |
Feb
(105) |
Mar
(34) |
Apr
(61) |
May
(41) |
Jun
(92) |
Jul
(176) |
Aug
(102) |
Sep
(247) |
Oct
(69) |
Nov
(32) |
Dec
(140) |
| 2007 |
Jan
(58) |
Feb
(51) |
Mar
(11) |
Apr
(20) |
May
(34) |
Jun
(37) |
Jul
(18) |
Aug
(60) |
Sep
(41) |
Oct
(105) |
Nov
(19) |
Dec
(14) |
| 2008 |
Jan
(3) |
Feb
|
Mar
(7) |
Apr
(5) |
May
(123) |
Jun
(5) |
Jul
(1) |
Aug
(29) |
Sep
(15) |
Oct
(21) |
Nov
(51) |
Dec
(3) |
| 2009 |
Jan
|
Feb
(36) |
Mar
(29) |
Apr
|
May
|
Jun
(7) |
Jul
(4) |
Aug
|
Sep
(4) |
Oct
|
Nov
(13) |
Dec
|
| 2010 |
Jan
|
Feb
|
Mar
(9) |
Apr
(11) |
May
(16) |
Jun
|
Jul
|
Aug
|
Sep
(1) |
Oct
|
Nov
|
Dec
|
| 2011 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(1) |
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2012 |
Jan
(7) |
Feb
(3) |
Mar
|
Apr
|
May
|
Jun
(3) |
Jul
|
Aug
|
Sep
|
Oct
(92) |
Nov
(28) |
Dec
(16) |
| 2013 |
Jan
(9) |
Feb
(2) |
Mar
|
Apr
(4) |
May
(4) |
Jun
(6) |
Jul
(14) |
Aug
(12) |
Sep
(4) |
Oct
(13) |
Nov
(1) |
Dec
(6) |
| 2014 |
Jan
(23) |
Feb
(19) |
Mar
(10) |
Apr
(14) |
May
(11) |
Jun
(6) |
Jul
(11) |
Aug
(15) |
Sep
(41) |
Oct
(95) |
Nov
(23) |
Dec
(11) |
| 2015 |
Jan
(3) |
Feb
(9) |
Mar
(19) |
Apr
(3) |
May
(1) |
Jun
(3) |
Jul
(11) |
Aug
(1) |
Sep
(15) |
Oct
(5) |
Nov
(2) |
Dec
|
| 2016 |
Jan
(7) |
Feb
(11) |
Mar
(8) |
Apr
(1) |
May
(3) |
Jun
(17) |
Jul
(12) |
Aug
(3) |
Sep
(5) |
Oct
(19) |
Nov
(12) |
Dec
(6) |
| 2017 |
Jan
(30) |
Feb
(23) |
Mar
(12) |
Apr
(32) |
May
(27) |
Jun
(7) |
Jul
(13) |
Aug
(16) |
Sep
(6) |
Oct
(11) |
Nov
|
Dec
(12) |
| 2018 |
Jan
(1) |
Feb
(5) |
Mar
(6) |
Apr
(7) |
May
(23) |
Jun
(3) |
Jul
(2) |
Aug
(1) |
Sep
(6) |
Oct
(6) |
Nov
(10) |
Dec
(3) |
| 2019 |
Jan
(26) |
Feb
(15) |
Mar
(9) |
Apr
|
May
(8) |
Jun
(14) |
Jul
(10) |
Aug
(10) |
Sep
(4) |
Oct
(2) |
Nov
(20) |
Dec
(10) |
| 2020 |
Jan
(10) |
Feb
(14) |
Mar
(29) |
Apr
(11) |
May
(25) |
Jun
(21) |
Jul
(23) |
Aug
(12) |
Sep
(19) |
Oct
(6) |
Nov
(8) |
Dec
(12) |
| 2021 |
Jan
(29) |
Feb
(9) |
Mar
(8) |
Apr
(8) |
May
(2) |
Jun
(2) |
Jul
(9) |
Aug
(9) |
Sep
(3) |
Oct
(4) |
Nov
(12) |
Dec
(13) |
| 2022 |
Jan
(4) |
Feb
|
Mar
(4) |
Apr
(12) |
May
(15) |
Jun
(7) |
Jul
(10) |
Aug
(2) |
Sep
|
Oct
(1) |
Nov
(8) |
Dec
|
| 2023 |
Jan
(15) |
Feb
|
Mar
(23) |
Apr
(1) |
May
(2) |
Jun
(10) |
Jul
|
Aug
(22) |
Sep
(19) |
Oct
(2) |
Nov
(20) |
Dec
|
| 2024 |
Jan
(1) |
Feb
|
Mar
(16) |
Apr
(15) |
May
(6) |
Jun
(4) |
Jul
(1) |
Aug
(1) |
Sep
|
Oct
(13) |
Nov
(18) |
Dec
(6) |
| 2025 |
Jan
(12) |
Feb
|
Mar
(2) |
Apr
(1) |
May
(11) |
Jun
(5) |
Jul
(4) |
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Gustaf N. <ne...@wu...> - 2019-03-02 20:29:23
|
Dear Ben,
i have still not managed to reproduce an error. I have extended
the hello-world.cgi script to include an image (see below) and tested
this from the main www folder (R1) and from a mapped cgi-bin
folder (R2) with a configuration without "allowstaticresources" set (C1)
and once with this parameter set (C2).
The results:
- The main request of the script returns always the expected HTML
code with the HTML content type
- The image is shown in most cases
R1+C1: yes
R1+C2: yes
R2+C1: no
R2+C2: yes
- When the image requests returns success (200), the image mime type
is correctly set. When the image request fails, a 404 error with the
error page is returned
All this is with NaviServer 4.99.18 on macOS.
Do you get on your platform different results for these tests?
-g
REQUEST R1: http://localhost:8100/hello-world.cgi
REQUEST R2: http://localhost:8100/cgi-bin/hello-world.cgi
CONFIG C1:
ns_section "ns/server/${server}/module/nscgi" {
ns_param map "GET /*.cgi"
ns_param map "GET /cgi-bin /usr/local/cgi-bin"
ns_param map "GET /awstats /opt/local/libexec/w3m/cgi-bin"
ns_param Interps cgi
}
CONFIG C2
ns_section "ns/server/${server}/module/nscgi" {
ns_param map "GET /*.cgi"
ns_param map "GET /cgi-bin /usr/local/cgi-bin"
ns_param map "GET /awstats /opt/local/libexec/w3m/cgi-bin"
ns_param Interps cgi
ns_param allowstaticresources true
}
====== SCRIPT hello.cgi ======================================
puts "HTTP/1.0 200 OK
Date: Monday, 06-Nov-95 17:50:15 GMT
Content-Type: text/html
<html>
<body>
Hello World <img src='DotLrnLogo.gif'>
</body>
"
==============================================================
On 01.03.19 14:04, Ben Brink via naviserver-devel wrote:
> Hi Gustaf,
>
> Thank you for the example and the info in thread:
> https://openacs.org/forums/message-view?message_id=4268249#msg_4268538
>
> Your test case works so I'll debug the legacy scripts.
>
>
>
> Still, static files are returned without a content-type value, and
> without content.
>
> The ns_param allowstaticresources does not change this, whether
> omitted, or set to true or false. For example, if allowstaticresources
> is true, and content/file is:
>
> test6.txt
> ==================================================================
> Hello World
>
> ==================================================================
>
> or any *.gif or *.png file.
>
> Static content is returned as expected when not in cgi-bin.
>
>
> The legacy perl cgi scripts I'm testing produce the same results ie no
> content and no content type, which is why I decided to test static files.
>
>
> For static content, the log shows errors like:
>
> Debug(cgi): nscgi: interp '(null)' exec '/usr/local/cgi-bin/test6.txt'
> Error: exec /usr/local/cgi-bin/test4.txt: execve() failed: Unknown
> error 1275110264
> Debug(cgi): nscgi: execute cgi script in directory
> '/usr/local/cgi-bin' returned pid -3
> Error: waitpid(-3) failed: No child processes
> Error: nscgi: wait for /usr/local/cgi-bin/test4.txt failed: No child
> processes
>
> ..when debug logging is invoked nscgi in section of config.tcl using:
>
> ns_logctl severity Debug(pl) on
> ns_logctl severity Debug(cgi) on
>
>
> cheers,
> Ben
>
|
|
From: Ben B. <tek...@ya...> - 2019-03-01 13:04:51
|
Hi Gustaf, Thank you for the example and the info in thread: https://openacs.org/forums/message-view?message_id=4268249#msg_4268538 Your test case works so I'll debug the legacy scripts. Still, static files are returned without a content-type value, and without content. The ns_param allowstaticresources does not change this, whether omitted, or set to true or false. For example, if allowstaticresources is true, and content/file is: test6.txt ================================================================== Hello World ================================================================== or any *.gif or *.png file. Static content is returned as expected when not in cgi-bin. The legacy perl cgi scripts I'm testing produce the same results ie no content and no content type, which is why I decided to test static files. For static content, the log shows errors like: Debug(cgi): nscgi: interp '(null)' exec '/usr/local/cgi-bin/test6.txt' Error: exec /usr/local/cgi-bin/test4.txt: execve() failed: Unknown error 1275110264 Debug(cgi): nscgi: execute cgi script in directory '/usr/local/cgi-bin' returned pid -3 Error: waitpid(-3) failed: No child processes Error: nscgi: wait for /usr/local/cgi-bin/test4.txt failed: No child processes ..when debug logging is invoked nscgi in section of config.tcl using: ns_logctl severity Debug(pl) on ns_logctl severity Debug(cgi) on cheers, Ben On 3/1/19 2:29 AM, Gustaf Neumann wrote: > > On 28.02.19 21:59, Ben Brink via naviserver-devel wrote: >> >> Is there a way to set a default mime-type for nscgi similar to how >> other naviserver modules use the default mime-types associated by file >> suffix, or in any other way, or have I goofed the config? >> >> With 4.99.18 and these settings, no mime-type is provided. Browsers >> aren't recognizing such content and nscgi isn't sending the content in >> these cases as far as I can tell. > > cgi-scripts have to provide the content type. > > Here is a very short example, where one can verify via browser developer > tools, that the server sends the provided content type correctly (when > browsing to "..../hello-word.cgi". > > Do you see different results? > > -gn > > > hello-world.cgi The case you provide > ================================================================== > puts "HTTP/1.0 200 OK > Date: Monday, 06-Nov-95 17:50:15 GMT > Content-Type: text/html > > <html> > <body> > Hello World > </body> > " > ================================================================== > > > config snippet > ================================================================== > # CGI > # > ns_section "ns/server/${server}/module/nscgi" { > ns_param map "GET /*.cgi" > ns_param Interps cgi > } > > ns_section "ns/interps/cgi" { > ns_param .cgi "/usr/local/ns/bin/tclsh" > } > ================================================================== > > > > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel > |
|
From: Gustaf N. <ne...@wu...> - 2019-03-01 10:29:30
|
On 28.02.19 21:59, Ben Brink via naviserver-devel wrote:
>
> Is there a way to set a default mime-type for nscgi similar to how
> other naviserver modules use the default mime-types associated by file
> suffix, or in any other way, or have I goofed the config?
>
> With 4.99.18 and these settings, no mime-type is provided. Browsers
> aren't recognizing such content and nscgi isn't sending the content in
> these cases as far as I can tell.
cgi-scripts have to provide the content type.
Here is a very short example, where one can verify via browser developer
tools, that the server sends the provided content type correctly (when
browsing to "..../hello-word.cgi".
Do you see different results?
-gn
hello-world.cgi
==================================================================
puts "HTTP/1.0 200 OK
Date: Monday, 06-Nov-95 17:50:15 GMT
Content-Type: text/html
<html>
<body>
Hello World
</body>
"
==================================================================
config snippet
==================================================================
# CGI
#
ns_section "ns/server/${server}/module/nscgi" {
ns_param map "GET /*.cgi"
ns_param Interps cgi
}
ns_section "ns/interps/cgi" {
ns_param .cgi "/usr/local/ns/bin/tclsh"
}
==================================================================
|
|
From: Ben B. <tek...@ya...> - 2019-02-28 20:59:15
|
Hello,
Is there a way to set a default mime-type for nscgi similar to how other naviserver modules use the default mime-types associated by file suffix, or in any other way, or have I goofed the config?
With 4.99.18 and these settings, no mime-type is provided. Browsers aren't recognizing such content and nscgi isn't sending the content in these cases as far as I can tell.
Varying value of ns_param allowstaticresources doesn't seem to change anything (after restarting nsd).
ns_section "ns/server/${server}/module/nscgi"
ns_param map "GET /cgi-bin /usr/local/cgi-bin"
ns_param map "POST /cgi-bin /usr/local/cgi-bin"
ns_param Interps CGIinterps
ns_param allowstaticresources true ;# default: false
#
# CGI
#
ns_section "ns/Interps/CGIinterps"
ns_param .cgi "/usr/local/ns/bin/tclsh"
ns_param .pl "/usr/bin/perl"
FYI, for testing, I made these url/files:/cgi-bin/test.pl - dynamically generated with mime-type
/cgi-bin/test-wo-mimetype.pl - dynamically generated without mime-type
/cgi-bin/test2.cgi -dynamically generated with mime-type (same as case provided here: Forum OpenACS Q&A: Naviserver + nscgi )
|
|
| |
Forum OpenACS Q&A: Naviserver + nscgi
|
|
|
/cgi-bin/test3.cgi -dynamically generated without mime-type (similar to test2.cgi less the mime-type declaration)
/cgi-bin/test4.txt -static "hello world" text file
/cgi-bin/test5.html -static "hello world" html file
/test4.txt -static "hello world" text file
/test5.html -static "hello world" html file
and tested cases using this "t" tcl/adp pair in an OpenACS test site:
t.adp:<pre>
@content_html;noquote@
</pre>
t.tcl:
set test_names_list [list cgi-bin/test.pl \
cgi-bin/test-wo-mimetype.pl \
cgi-bin/test2.cgi \
cgi-bin/test3.cgi \
cgi-bin/test4.txt \
cgi-bin/test5.html \
test4.txt \
test5.html ]
set content_html ""
foreach n $test_names_list {
set u "http://localhost:8000/"
append u $n
set result_dict [util::http::get -url $u ]
append content_html $u " : \n"
dict for {k v} $result_dict {
append content_html " " $k " : " $v "\n"
}
append content_html "\n"
}
Here's an example of test page results using OpenACS' util::http::get:
http://localhost:8000/cgi-bin/test.pl :
headers : server NaviServer/4.99.18 date {Thu, 28 Feb 2019 03:16:50 GMT} x-frame-options SAMEORIGIN x-content-type-options nosniff x-xss-protection {1; mode=block} referrer-policy strict-origin set-cookie {ad_session_id="210001%2c0%2c0%2c1551323810%20{744%201551325010%201605F86CC89B6BA5C0830FD3E4C413A955563B44}"; Expires=Fri, 01-Jan-2035 01:00:00 GMT; Path=/; Discard; HttpOnly} content-type text/plain connection close
page : hello world
file :
status : 200
modified :
http://localhost:8000/cgi-bin/test-wo-mimetype.pl :
headers : server NaviServer/4.99.18 date {Thu, 28 Feb 2019 03:16:50 GMT} x-frame-options SAMEORIGIN x-content-type-options nosniff x-xss-protection {1; mode=block} referrer-policy strict-origin set-cookie {ad_session_id="210002%2c0%2c0%2c1551323810%20{128%201551325010%2000B49E88A7DD72C3E8840DB3BBD40C043E870469}"; Expires=Fri, 01-Jan-2035 01:00:00 GMT; Path=/; Discard; HttpOnly} connection close
page :
file :
status : 200
modified :
http://localhost:8000/cgi-bin/test2.cgi :
headers : server NaviServer/4.99.18 date {Thu, 28 Feb 2019 03:16:50 GMT} x-frame-options SAMEORIGIN x-content-type-options nosniff x-xss-protection {1; mode=block} referrer-policy strict-origin set-cookie {ad_session_id="210003%2c0%2c0%2c1551323810%20{133%201551325010%2017B9AE6B9466893882B47FEF24895721ADFFCA9E}"; Expires=Fri, 01-Jan-2035 01:00:00 GMT; Path=/; Discard; HttpOnly} {http/1.0 200 okdate} {Monday, 06-Nov-95 17:50:15 GMT} content-type text/html connection close
page :
Hello World
file :
status : 200
modified :
http://localhost:8000/cgi-bin/test3.cgi :
headers : server NaviServer/4.99.18 date {Thu, 28 Feb 2019 03:16:51 GMT} x-frame-options SAMEORIGIN x-content-type-options nosniff x-xss-protection {1; mode=block} referrer-policy strict-origin set-cookie {ad_session_id="210004%2c0%2c0%2c1551323810%20{126%201551325010%208E8037ACE4FC0A043ACCF888D561EEA18CDCDF87}"; Expires=Fri, 01-Jan-2035 01:00:00 GMT; Path=/; Discard; HttpOnly} {http/1.0 200 okdate} {Monday, 06-Nov-95 17:50:15 GMT} connection close
page :
Hello World
file :
status : 200
modified :
http://localhost:8000/cgi-bin/test4.txt :
headers : server NaviServer/4.99.18 date {Thu, 28 Feb 2019 03:16:51 GMT} x-frame-options SAMEORIGIN x-content-type-options nosniff x-xss-protection {1; mode=block} referrer-policy strict-origin set-cookie {ad_session_id="210005%2c0%2c0%2c1551323811%20{133%201551325011%205D826D56888E7909B34A73FA2299141C5E681A0E}"; Expires=Fri, 01-Jan-2035 01:00:00 GMT; Path=/; Discard; HttpOnly} connection close
page :
file :
status : 200
modified :
http://localhost:8000/cgi-bin/test5.html :
headers : server NaviServer/4.99.18 date {Thu, 28 Feb 2019 03:16:51 GMT} x-frame-options SAMEORIGIN x-content-type-options nosniff x-xss-protection {1; mode=block} referrer-policy strict-origin set-cookie {ad_session_id="210006%2c0%2c0%2c1551323811%20{134%201551325011%2038F08ED3FADFC1410681FE5A34B8101B0FD194BE}"; Expires=Fri, 01-Jan-2035 01:00:00 GMT; Path=/; Discard; HttpOnly} connection close
page :
file :
status : 200
modified :
http://localhost:8000/test4.txt :
headers : server NaviServer/4.99.18 date {Thu, 28 Feb 2019 03:16:51 GMT} x-frame-options SAMEORIGIN x-content-type-options nosniff x-xss-protection {1; mode=block} referrer-policy strict-origin set-cookie {ad_session_id="210007%2c0%2c0%2c1551323811%20{129%201551325011%20921A7B089760E84FCCDE96D4057070B2705B87CC}"; Expires=Fri, 01-Jan-2035 01:00:00 GMT; Path=/; Discard; HttpOnly} last-modified {Thu, 28 Feb 2019 03:04:56 GMT} content-type text/plain accept-ranges bytes content-length 22 connection close
page : Hello World. Really!
file :
status : 200
modified : Thu, 28 Feb 2019 03:04:56 GMT
http://localhost:8000/test5.html :
headers : server NaviServer/4.99.18 date {Thu, 28 Feb 2019 03:16:51 GMT} x-frame-options SAMEORIGIN x-content-type-options nosniff x-xss-protection {1; mode=block} referrer-policy strict-origin set-cookie {ad_session_id="210008%2c0%2c0%2c1551323811%20{128%201551325011%208B6E0EE2521A4AC57006FCF7A3F6C437E26FAB54}"; Expires=Fri, 01-Jan-2035 01:00:00 GMT; Path=/; Discard; HttpOnly} last-modified {Thu, 28 Feb 2019 03:05:48 GMT} content-type text/html accept-ranges bytes content-length 101 connection close
page :
Hello World
file : status : 200 modified : Thu, 28 Feb 2019 03:05:48 GMT
kind regards,Ben
|
|
From: Gustaf N. <ne...@wu...> - 2019-02-24 19:06:53
|
Dear all, on sourceforge is now the release of NaviServer 4.99.18 [1]. This release is primarily a bug-fix release for installations still using Tcl 8.5, or installations having a half-working IPv6 installations. Upgrade from 4.99.17 is recommended. Below is a summary of changes. Many thanks to all contributors! all the best -gustaf [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.18/ ======================================= NaviServer 4.99.18, released 2019-02-24 ======================================= 133 files changed, 3314 insertions(+), 2106 deletions(-) New Features: ------------- - Added ability to provide memory sizes via memory units (kB, MB, GB, KiB, MiB, GiB) in the configuration files and commands. We follow the approach taken by e.g. PostgreSQL and use 1024 as multiplier, which is e.g. useful for buffer sizes. All memory sizes in the config files can be specified in memory units. Currently, the only commands supporting memory units are ns_cache_create, ns_cache_configure and ns_http. The configuration based on integers without memory units continues to work. - Portable config files (windows compatibility) Rationale: unixoid platforms use traditionally ".so" as extension of binary modules, no matter what "info sharedlibextension" is. On macOS, the sharedlibextension returns ".dylib", while the module is named ".so". Under windows, it seems that the extension of the module has to be ".dll", so we can't write portable config files without extra effort. The new code tries now first to load the library as specified for backward compatibility (maybe containing a file extension). When this fails, it tries to append either ".so" in unixoid platforms or ".dll" under windows. - new API call: ns_db currenthandles Return information about handles currently allocated to the thread performing this call. The result contains a dict with the pools, from which handles were allocated, the names of the handles and their "active" state (a handle is active between a "ns_db select" and the last "ns_db getrow" statement). If no handles are allocated by this thread, empty is returned. For example, a possible result is "main {nsdb0 1} subquery {nsdb1 0}" indicating that the current thread has allocated handles from pool "main" and "subquery", but the handle from pool "main" (i.e. "nsdb0") is currently active (during a select). This feature requires Tcl 8.5 or newer. - More security features: SameSite support for cookies The command ns_setcookie supports now the flag "-samesite" with permissible values "strict|lax|none". When the flag is set it prevents the browser from sending this cookie along with cross-site requests to mitigate cross site scripting attacks. Permissible values are [term strict], [term lax], or [term none] (default). While the value [term strict] prevents sending the cookie to the target site in all cross-site browsing context, the value of [term lax] allows sending the cookie when the user clicks on regular links. For details, see https://www.owasp.org/index.php/SameSite The "SameSite" cookie flag is not yet part of an RFC, but most major browsers support it. Browsers that do not support it, ignore the flag silently (see https://caniuse.com/#search=samesite). Although most cookies should probably use the flags, in order to provide backward compatibility, the flag can't be activated by default on all cookies. - Better entries in server log for scheduled procedures. With this change, one can determine exactly, which scheduled thread was executing an instance of which scheduled procedure. Similarly as for connection threads, one can now distinguish, whether two log-entries are from the same instance of a scheduled proc or from different ones. - nsdb stats: Return as well number of currently connected DB connections. Performance Improvements: ------------------------- - nsproxy: Fixed block in reaper thread that led to excessive time spent in the main reaper loop, thus blocking the whole proxy interface. This change improves concurrency and reduces potentially long locks of the full nsproxy machinery. Bug Fixes: ---------- - Improved combability with Tcl 8.5 * make sure package "try" is used in regression test cases where needed - ns_http * Fixed potential problem with partial writes on HTTP requests with large bodies (e.g. POST of larger files) * Aligned implementation with documentation (if "wait" operation has no timeout, use timeout of queue operation) * Improved client connect() operations with half-configured IPv6 installations. This change addresses especially a problem, where IPv6 is activated on a server, but it cannot be used to connect to the outside world. In these situations, connect operations to hosts with domain names having both IPv4 and IPv6 addresses would fail, although IPv4 connections are possible. In this cases, one should actually either deactivate IPv6 on the machine or probably deactivate IPv6 name lookups in bind (option "-4", filter-aaaa-on-v4 yes;). With this change, this should be necessary. This change fixes also a memory and socket leak in this situation. - Cache transactions: make "ns_cache flush $cache" behave like "ns_cache_flush -glob $cache *" on "ns_cache_transaction_rollback". Previously, when the full cache was flushed, the rollback was not performed properly. - tcl_crypto: * align code with documentation (provide a default for option "-digest") - nscgi: * fixed potential memory leak - ns_sourceproc: fix deprecated function in cases in error cases, make clear in documentation that ns_register_procns_register_tcl is the preferred approach (many thanks to Russell Sorensen for reporting) - tcl/nstrace.tcl: undo change introduced with 4.99.17 to exclude content of ::tcl namespace since this causes problems with Tcl 8.5 (problems with clock or "package require try" when executed during initial load phase. - Ns_VarUnset(): Deleting the entire nsv array when last value is deleted (many thanks to Andrew Piskorski). Documentation improvements: --------------------------- - All documentation: use memory units when appropriate - doc/src/naviserver/ns_job.man: aligned documentation with implementaton; use itemized list for describing entries in the result lists. - doc/src/manual/admin-maintenance.man: Improved sample service file - nsssl/doc/mann/nsssl.man: Improved sample service file - doc/src/naviserver/ns_log.man Added sample configuration for configuring logging behavior - doc/src/naviserver/ns_register.man, doc/src/naviserver/ns_shortcut_filter.man: provide links (and hints) between "ns_shortcut_filter" and "ns_register_filter" - Rework man pages for scheduled procedures: group all related information into a single man page and improve cross referencing, fix broken example code. - doc/src/naviserver/ns_job.man: Fix wrong argument order in example command. - Made markup in documentation more consistent doc/src/naviserver/commandlist.man, doc/src/naviserver/ns_adp_ctl.man, doc/src/naviserver/ns_cond.man, doc/src/naviserver/ns_critsec.man, doc/src/naviserver/ns_env.man, doc/src/naviserver/ns_event.man, doc/src/naviserver/ns_info.man, doc/src/naviserver/ns_rwlock.man, doc/src/naviserver/ns_sema.man, doc/src/naviserver/ns_writer.man, nslog/doc/mann/ns_accesslog.man - Various improvements on several other man pages: - Removed man page for non-existing command "ns_cookietime" Tcl API Changes: ---------------- - Mark ns_cancel explicitly as deprecated, before its use was just discouraged by the man page. Use ns_unschedule_proc instead. Configuration Changes: ---------------------- - Turn off Nagle algorithm (parameter nodelay) per default since this leads in current Linux systems to bad performance (e.g. seeing with Apache bench ~40ms per request latency when keepalive is used). The default is now set to false in the C code as well in nsd-config.tcl and openacs-config.tcl - New parameter "logusecdiff": include time difference since last log entry with microsecond (usec) resolution. This option is useful for determine latencies with minimal effort. - All sample config files: * Remove file extensions for binary modules * Improved documentation - sample-config.tcl: * Added sample for DNS configuration * Added sample server log configuration - openacs-config.tcl: * Deactivated SSLv3 by default * Adjusted comments to values from source code * Add sorting to use always pick the library with the highest version number Code Changes: ------------- - Regressions testing: * Removed false positive * Extended tests: http.test, ns_striphtml.test, ns_cache.test * Replaced deprecated "ns_adp_eval" in test by "ns_adp_parse" - Improved portability: * fixed compilation issues for versions of OpenSSL before 1.0.2 * Windows changes (Many thanks to Andrew Piskorski): + Fixed compilation issues with nsssl under windows + Improved makefiles + use "I64u" or "I32u" instead of c99 conventions, since these seem to be broken in many (all?) MSC versions. - Reduced code security smells: add *CLOEXEC to file descriptor open operation. This fixes no real issue (forks are very limited in nsd), but silences static analyzers. - Replaced UTF-8 string literals by hexadecimal-escape-sequences to avoid potential encoding limitations from older compilers - Improved code locality - Removed unused macro - Adding more declarations for PURE and CONST functions - Eased live of static analysis programs - Added "const" declarations, made symbol table static - Reduced variable scopes - Reduced dead assignments - Reduce assignments in expressions Modules: -------- - nsstats: * Added information about spooler threads to "process" page * Added running ns_jobs to the "process" page * Added running scheduled procedures to the "process" page * Improved HTML layout of configparams page - nsloopctl: * Fixed compilation with current versions |
|
From: Gustaf N. <ne...@wu...> - 2019-02-17 12:09:46
|
The version *b2 uploaded to sourceforge contains also support for the "SameSite" flag for cookies, a means to mitigate CSRF attacks. all the best -gn ns_setcookie: add flag "-samesite" with values "strict|lax|none" When the flag is set it prevents the browser from sending this cookie along with cross-site requests to mitigate cross site scripting attacks. Permissible values are [term strict], [term lax], or [term none] (default). While the value [term strict] prevents sending the cookie to the target site in all cross-site browsing context, the value of [term lax] allows sending the cookie when the user clicks on regular links. For details, see https://www.owasp.org/index.php/SameSite This cookie flag is not yet part of an RFC, but most major browsers support it. Browsers that do not support it, ignore the flag silently (see https://caniuse.com/#search=samesite). Although most cookies should probably use the flags, in order to provide backward compatibility, the flag can't be activated by default on all cookies. On 16.02.19 20:08, Gustaf Neumann wrote: > Dear all, > > on sourceforge is a release candidate for NaviServer 4.99.18 [1]. > > This release is primarily a bug-fix release for installations still > using Tcl 8.5, or installations having a half-working IPv6 installations. > > Below is a - preliminary - summary of changes; especially the > summary of changes in the modules is missing. In case you have more > changes for this release, please commit or get in touch with me. > If everything goes well, the release should follow in about one week. > > Please test if possible. > > best regards > > -gustaf > [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.18/ > > ======================================= > NaviServer 4.99.18, released 2019-XX-XX > ======================================= > > 87 files changed, 2615 insertions(+), 1476 deletions(-) > > New Features: > > - Added ability to provide memory sizes via memory units (kB, MB, GB, > KiB, MiB, GiB) in the configuration files and commands. We follow > the approach taken by e.g. PostgreSQL and use 1024 as > multiplicator, which is e.g. useful for buffer sizes. All memory > sizes in the config files can be specified in memory > units. Currently, the only commands supporting memory units are > ns_cache_create, ns_cache_configure and ns_http. The configuration > based on integers without memory units continues to work. > > > - Portable config files (windows compatibility) > > Rationale: unixoid platforms use traditionally ".so" as extension > of binary modules, no matter what "info sharedlibextension" is. On > macOS, the sharedlibextension returns ".dylib", while the module is > named ".so". Under windows, it seems that the extension of the module > has to be ".dll", so we can't write portable config > files without extra effort. > > The new code tries now first to load the library as specified for > backward compatibility (maybe containing a file extension). When > this fails, it tries to append either ".so" in unixoid platforms or > ".dll" under windows. > > > - new API call: ns_db currenthandles > > Return information about handles currently allocated to the thread > performing this call. The result contains a dict with the pools, > from which handles were allocated, the names of the handles and > their "active" state (a handle is active between a "ns_db select" > and the last "ns_db getrow" statement). If no handles are allocated > by this thread, empty is returned. > > For example, a possible result is "main {nsdb0 1} subquery {nsdb1 > 0}" indicating that the current thread has allocated handles from > pool "main" and "subquery", but the handle from pool "main" (i.e. > "nsdb0") is currently active (during a select). > > This feature requires Tcl 8.5 or newer. > > > Performance Improvements: > > - nsproxy: Fixed block in reaper thread that led to excessive time > spent in the main reaper loop, thus blocking the whole proxy > interface. This change improves concurrency and reduces > potentially long > locks of the full nsproxy machinery. > > > Bug Fixes: > > - improved compatility with Tcl 8.5 > * make sure package "try" is used in regression test cases where > needed > > - ns_http > * Fixed potential problem with partial writes on HTTP requests > with large bodies (e.g. POST of larger files) > * Aligned implementation with documentation (if "wait" > operation has no > timeout, use timeout of queue operation) > * Improved client connect() operations with half-configured > IPv6 > installations. This change addresses especially a problem, where > IPv6 is activated on a server, but it cannot be used to connect > to the outside world. In this situations, connect operations to > hosts with domain names having both IPv4 and IPv6 addresses > would fail, although IPv4 connections are possible. In this > cases, one should actually either deactivate IPv6 on the machine > or probably deactivate IPv6 name lookups in bind (option "-4", > filter-aaaa-on-v4 yes;). With this change, this should be be > necessary. This change fixes also a memory and socket leak in > this situation. > > - Cache transactions: make "ns_cache flush $cache" behave like > "ns_cache_flush -glob $cache *" on "ns_cache_transaction_rollback". > Previously, when the full cache was flushed, the rollback was not > performed properly. > - tcl_crypto: > * align code with documentation (provide a default for option > "-digest") > > - nscgi: > * fixed potential memory leak > > - ns_sourceproc: fix deprecated function in cases in error cases, > make clear in documentation that ns_register_procns_register_tcl > is the preferred approach (many thanks to Russell Sorensen for > reporting) > > - tcl/nstrace.tcl: undo change introduced with 4.99.17 to exclude > content of ::tcl namespace since this causes problems with Tcl 8.5 > (problems with clock or "package require try" when executed during > initial load phase. > > - Ns_VarUnset(): Deleting the entire nsv array when last value is > deleted (many thanks to Andrew Piskorski). > > > Documentation improvements: > > - all documentation: use memory units when appropriate > - doc/src/naviserver/ns_job.man: aligned documentation with > implementaton; use itemized list for describing entries in the > result lists. > - doc/src/manual/admin-maintenance.man: > Improved sample service file > - nsssl/doc/mann/nsssl.man: > Improved sample service file > - doc/src/naviserver/ns_log.man > Added sample configuration for configuring logging behavior > > - doc/src/naviserver/ns_register.man, > doc/src/naviserver/ns_shortcut_filter.man: provide links (and > hints) between "ns_shortcut_filter" and "ns_register_filter" > > - made markup in documentation more consistent > doc/src/naviserver/commandlist.man, > doc/src/naviserver/ns_adp_ctl.man, doc/src/naviserver/ns_cond.man, > doc/src/naviserver/ns_critsec.man, doc/src/naviserver/ns_env.man, > doc/src/naviserver/ns_event.man, doc/src/naviserver/ns_info.man, > doc/src/naviserver/ns_rwlock.man, doc/src/naviserver/ns_sema.man, > doc/src/naviserver/ns_writer.man, nslog/doc/mann/ns_accesslog.man > - various improvements in the following man pages > nsssl/doc/mann/nsssl.man > doc/src/manual/tcl-lib-file.man > > Tcl API Changes: > > C API Changes: > > Incompatible API Changes: > > Configuration Changes: > - Turn off Nagle algorithm (parameter nodelay) per default > since this leads in current Linux systems to bad performance > (e.g. seeing with Apache bench ~40ms per request latencency > when keepalive is used). The default is now set to fasel > in the C code as wel in nsd-config.tcl and openacs-config.tcl > > - New parameter "logusecdiff": include time difference since last > log entry with microsecond (usec) resolution. This option is useful > for determine latencies with minimal effort. > > - All sample config files: > * Remove file extensions for binary modules > * Improved documentation > > - sample-config.tcl: > * Added sample for DNS configuration > * Added sample server log configuration > - openacs-config.tcl: > * Deactivated SSLv3 by default > * Adjusted comments to values from source code > * Add sorting to use always pick the library with the highest > version number > > > Command Line Changes: > > Code Changes: > - Regressions testing: > * removed false positive > * extended tests: http.test, ns_striphtml.test, ns_cache.test > - Improved portability: > * fixed compilation issues for versions of OpenSSL before 1.0.2 > * Windows changes (Many thanks to Andrew Piskorski): > + Fixed compilation issues with nsssl under windows > + Improved makefiles > + use "I64u" or "I32u" instead of c99 conventions, since > these seem to be broken in many (all?) MSC versions. > - Reduced code security smells: add *CLOEXEC to file descriptor > open > operation. This fixes no real issue (forks are very limited in > nsd), but silences static analyzers. > > - Replaced UTF-8 string literals by hexadecimal-escape-sequences to > avoid potential encoding limitations from older compilers > > - Improved code locality > - Removed unused macro > - Adding more declarations for PURE and CONST functions > - Eased live of static analysis programs > - Added "const" declarations, made symbol table static > - Reduced variable scopes > - Reduced dead assignments |
|
From: Gustaf N. <ne...@wu...> - 2019-02-16 19:09:27
|
Dear all, on sourceforge is a release candidate for NaviServer 4.99.18 [1]. This release is primarily a bug-fix release for installations still using Tcl 8.5, or installations having a half-working IPv6 installations. Below is a - preliminary - summary of changes; especially the summary of changes in the modules is missing. In case you have more changes for this release, please commit or get in touch with me. If everything goes well, the release should follow in about one week. Please test if possible. best regards -gustaf [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.18/ ======================================= NaviServer 4.99.18, released 2019-XX-XX ======================================= 87 files changed, 2615 insertions(+), 1476 deletions(-) New Features: - Added ability to provide memory sizes via memory units (kB, MB, GB, KiB, MiB, GiB) in the configuration files and commands. We follow the approach taken by e.g. PostgreSQL and use 1024 as multiplicator, which is e.g. useful for buffer sizes. All memory sizes in the config files can be specified in memory units. Currently, the only commands supporting memory units are ns_cache_create, ns_cache_configure and ns_http. The configuration based on integers without memory units continues to work. - Portable config files (windows compatibility) Rationale: unixoid platforms use traditionally ".so" as extension of binary modules, no matter what "info sharedlibextension" is. On macOS, the sharedlibextension returns ".dylib", while the module is named ".so". Under windows, it seems that the extension of the module has to be ".dll", so we can't write portable config files without extra effort. The new code tries now first to load the library as specified for backward compatibility (maybe containing a file extension). When this fails, it tries to append either ".so" in unixoid platforms or ".dll" under windows. - new API call: ns_db currenthandles Return information about handles currently allocated to the thread performing this call. The result contains a dict with the pools, from which handles were allocated, the names of the handles and their "active" state (a handle is active between a "ns_db select" and the last "ns_db getrow" statement). If no handles are allocated by this thread, empty is returned. For example, a possible result is "main {nsdb0 1} subquery {nsdb1 0}" indicating that the current thread has allocated handles from pool "main" and "subquery", but the handle from pool "main" (i.e. "nsdb0") is currently active (during a select). This feature requires Tcl 8.5 or newer. Performance Improvements: - nsproxy: Fixed block in reaper thread that led to excessive time spent in the main reaper loop, thus blocking the whole proxy interface. This change improves concurrency and reduces potentially long locks of the full nsproxy machinery. Bug Fixes: - improved compatility with Tcl 8.5 * make sure package "try" is used in regression test cases where needed - ns_http * Fixed potential problem with partial writes on HTTP requests with large bodies (e.g. POST of larger files) * Aligned implementation with documentation (if "wait" operation has no timeout, use timeout of queue operation) * Improved client connect() operations with half-configured IPv6 installations. This change addresses especially a problem, where IPv6 is activated on a server, but it cannot be used to connect to the outside world. In this situations, connect operations to hosts with domain names having both IPv4 and IPv6 addresses would fail, although IPv4 connections are possible. In this cases, one should actually either deactivate IPv6 on the machine or probably deactivate IPv6 name lookups in bind (option "-4", filter-aaaa-on-v4 yes;). With this change, this should be be necessary. This change fixes also a memory and socket leak in this situation. - Cache transactions: make "ns_cache flush $cache" behave like "ns_cache_flush -glob $cache *" on "ns_cache_transaction_rollback". Previously, when the full cache was flushed, the rollback was not performed properly. - tcl_crypto: * align code with documentation (provide a default for option "-digest") - nscgi: * fixed potential memory leak - ns_sourceproc: fix deprecated function in cases in error cases, make clear in documentation that ns_register_procns_register_tcl is the preferred approach (many thanks to Russell Sorensen for reporting) - tcl/nstrace.tcl: undo change introduced with 4.99.17 to exclude content of ::tcl namespace since this causes problems with Tcl 8.5 (problems with clock or "package require try" when executed during initial load phase. - Ns_VarUnset(): Deleting the entire nsv array when last value is deleted (many thanks to Andrew Piskorski). Documentation improvements: - all documentation: use memory units when appropriate - doc/src/naviserver/ns_job.man: aligned documentation with implementaton; use itemized list for describing entries in the result lists. - doc/src/manual/admin-maintenance.man: Improved sample service file - nsssl/doc/mann/nsssl.man: Improved sample service file - doc/src/naviserver/ns_log.man Added sample configuration for configuring logging behavior - doc/src/naviserver/ns_register.man, doc/src/naviserver/ns_shortcut_filter.man: provide links (and hints) between "ns_shortcut_filter" and "ns_register_filter" - made markup in documentation more consistent doc/src/naviserver/commandlist.man, doc/src/naviserver/ns_adp_ctl.man, doc/src/naviserver/ns_cond.man, doc/src/naviserver/ns_critsec.man, doc/src/naviserver/ns_env.man, doc/src/naviserver/ns_event.man, doc/src/naviserver/ns_info.man, doc/src/naviserver/ns_rwlock.man, doc/src/naviserver/ns_sema.man, doc/src/naviserver/ns_writer.man, nslog/doc/mann/ns_accesslog.man - various improvements in the following man pages nsssl/doc/mann/nsssl.man doc/src/manual/tcl-lib-file.man Tcl API Changes: C API Changes: Incompatible API Changes: Configuration Changes: - Turn off Nagle algorithm (parameter nodelay) per default since this leads in current Linux systems to bad performance (e.g. seeing with Apache bench ~40ms per request latencency when keepalive is used). The default is now set to fasel in the C code as wel in nsd-config.tcl and openacs-config.tcl - New parameter "logusecdiff": include time difference since last log entry with microsecond (usec) resolution. This option is useful for determine latencies with minimal effort. - All sample config files: * Remove file extensions for binary modules * Improved documentation - sample-config.tcl: * Added sample for DNS configuration * Added sample server log configuration - openacs-config.tcl: * Deactivated SSLv3 by default * Adjusted comments to values from source code * Add sorting to use always pick the library with the highest version number Command Line Changes: Code Changes: - Regressions testing: * removed false positive * extended tests: http.test, ns_striphtml.test, ns_cache.test - Improved portability: * fixed compilation issues for versions of OpenSSL before 1.0.2 * Windows changes (Many thanks to Andrew Piskorski): + Fixed compilation issues with nsssl under windows + Improved makefiles + use "I64u" or "I32u" instead of c99 conventions, since these seem to be broken in many (all?) MSC versions. - Reduced code security smells: add *CLOEXEC to file descriptor open operation. This fixes no real issue (forks are very limited in nsd), but silences static analyzers. - Replaced UTF-8 string literals by hexadecimal-escape-sequences to avoid potential encoding limitations from older compilers - Improved code locality - Removed unused macro - Adding more declarations for PURE and CONST functions - Eased live of static analysis programs - Added "const" declarations, made symbol table static - Reduced variable scopes - Deduced dead assignments |
|
From: Zoran V. <zv...@ar...> - 2019-02-01 17:04:19
|
On Fri, 1 Feb 2019 11:50:58 -0500 Andrew Piskorski <at...@pi...> wrote: > work around the "Nsvs only store strings" limitation This "limitation" is rather the only generic and safe option! And it is pretty trivial in implementation. To see the other possibility, look in the tsv in the threading extension. It is much more complicated but allows one to "share" objects between threads w/o constant to/from string conversion. But it is black magic (rather complex) and limited (by the very nature of Tcl_Objs) to some well known simple object types. Cheers, Zoran |
|
From: Zoran V. <zv...@ar...> - 2019-02-01 16:54:58
|
On Fri, 1 Feb 2019 11:15:46 -0500 Andrew Piskorski <at...@pi...> wrote: > since I'm not using virtual servers anyway, how do I say > "all of them, process-wide"? You can't. The nsv buckets are stored on per-server basis. Naviserver may run a number of virtual servers. You declare all this in the configuration file. See "ns/servers" part in your configuration file. It is your module that, when loaded in the Naviserver, gets the name of the current server it is loaded into. To be loaded as extension, your code needs to provide Ns_ModuleInit() call that gets called by the module loader when a virtual server is getting initialized. The system passes the name of the server and name of the module to your Ns_ModuleInt(). You need to store it somewhere for later use. HTH, ZOran |
|
From: Andrew P. <at...@pi...> - 2019-02-01 16:51:10
|
On Fri, Feb 01, 2019 at 05:28:50PM +0100, Zoran Vasiljevic wrote: > OTOH, if you use ns:addr object, you need not take care about > locking and all the other trivia at all. It is the "rest" of > the system that needs to do that. > > Some of the stuff in "nsd/tclobj.c" seems be primarily concerned with > > the "address object type" (IP addresses?), but I get the sense that > It's (memory-)addresses == pointers. Ah! And you've already written the code to correctly convert between the pointer and string representations, so the ns:addr object approach is also a way to work around the "Nsvs only store strings" limitation. Thanks you, Zoran. I'm not sure which way I'll go in my implementation, but the choices available to me are making more sense now. -- Andrew Piskorski <at...@pi...> |
|
From: Zoran V. <zv...@ar...> - 2019-02-01 16:29:01
|
On Fri, 1 Feb 2019 10:57:00 -0500 Andrew Piskorski <at...@pi...> wrote: > What if you don't care much about Tcl objects? A C pointer is already > easily shareable across threads (you just need the proper mutex > locking when using it), the trick is finding it again using some > handle from Tcl. For simplicity, should I just completely ignore the > whole Tcl_Obj machinery when doing that? Of course. You can always use a lock, hash-table and key your pointers, passing the key to the Tcl. But you asked for some other way to do it :-) > > Or SHOULD I (always?) make my handle into a Tcl_Obj? If so, can you > go into a bit more detail on WHY that's a better approach? Even > despite the fact that this Tcl_Obj thing is now more difficult to > share across threads? By having a Tcl object that contains ptr rep, you can safely toss it arround Tcl as it will be a "natural" Tcl thing. You can treat it everywhere as a regular Tcl object. Whereas if you opt for the above approach, you need to "know" what that key points to. If you are only consumer of that key it is pretty OK to do it. But if you are passing ptr's arround the "unknown" code you might be better with object. This is why we did the ns:addr object, so that we do not have to care. It is really the same thing but other way around. > > If I take an ordinary sort of Tcl string or integer and hand it to my > C code, that works fine as the key for an nsv or Tcl_HashTable lookup. > Of course, the code doesn't "know" that that particular string is > supposed to be a handle, but there's nothing stopping me from simply > using it as one in the C code, and throwing an error if I can't find a > hash entry for it. Exactly. > > I suspect that the above is NOT the recommended approach, but I don't > fully understand why. It seems "simpler" to me... But maybe that's > only because I don't yet understand the Tcl_Obj integration approach, > and what benefits it provides. It is "simpler" because you are used to it. But in essence, you need to make a shared hash table, invent a key for your pointers and make sure all is always properly locked. So it isn't per-se simpler. It is still a devil, but that devil isn't that black because you already know it :-) OTOH, if you use ns:addr object, you need not take care about locking and all the other trivia at all. It is the "rest" of the system that needs to do that. > Hm, that has Ns_TclSetTwoPtrValue(), Ns_TclGetOpaqueFromObj(), and > other black magic (or so it seems to me). Are there docs on these C > APIs anywhere? No. But it is really trivial. If you want to wrap a ptr into the ns:addr object you can do: Tcl_Obj *myObj = Tcl_NewObj(); char *myType = "at-ptr"; /* Name it as you want */ MyDataStruct *myStruct = &MyGlobalStruct; Ns_TclSetAddrObj(myObj, myType, (void*)myStruct); Now you can pass the myObj to Tcl (put it in result, bury it in some list, store it in nsv array etc.pp.). Your code may later access the address/ptr value: Tcl_Obj *myObj = objv[1]; /* For example */ MyDataStruct *myStruct; Ns_TclGetOpaqueFromObj(myObj, myType, (void**)&myStruct); so now you have (or don't if GetOpaque retuns TCL_ERROR) your initial *myStruct value. > > Some of the stuff in "nsd/tclobj.c" seems be primarily concerned with > the "address object type" (IP addresses?), but I get the sense that > much of that code is much more general purpose than that. I see that > "nsd/tcltime.c" uses the tclobj.c functions to implement an Ns_Time > Tcl_Obj type, but I definitely don't understand how or why yet! > It's (memory-)addresses == pointers. HTH, Zoran |
|
From: Andrew P. <at...@pi...> - 2019-02-01 16:15:56
|
On Fri, Feb 01, 2019 at 04:25:20PM +0100, Zoran Vasiljevic wrote: > > I'm a little confused by (e.g.) Ns_VarGet(), though. Why is its first > > arg a string for the "server"? What does that mean, and where am I > > supposed to get it from? > > This is the identification of the Naviserver's virtual server > that is implicitly "known" to the Tcl code. The nsv_XXX is > thus bound to a virtual server since your Tcl code executes > within a specific virtual server. But on the C-level you need > to specify the virtual server explicitly because your C code > may operate for all known virtual servers or just bunch of them. Ok. But where do I get the server string from, in C? Or since I'm not using virtual servers anyway, how do I say "all of them, process-wide"? -- Andrew Piskorski <at...@pi...> |
|
From: Andrew P. <at...@pi...> - 2019-02-01 15:57:08
|
On Fri, Feb 01, 2019 at 11:44:45AM +0100, Zoran Vasiljevic wrote: > The only safe way to share data between threads > in Tcl is over strings. One can go to some extent > and try making Tcl objects shareable but for that > you need to know the object internals which is a > great PITA and does not always work well, except for > a limited set of object types. What if you don't care much about Tcl objects? A C pointer is already easily shareable across threads (you just need the proper mutex locking when using it), the trick is finding it again using some handle from Tcl. For simplicity, should I just completely ignore the whole Tcl_Obj machinery when doing that? Or SHOULD I (always?) make my handle into a Tcl_Obj? If so, can you go into a bit more detail on WHY that's a better approach? Even despite the fact that this Tcl_Obj thing is now more difficult to share across threads? If I take an ordinary sort of Tcl string or integer and hand it to my C code, that works fine as the key for an nsv or Tcl_HashTable lookup. Of course, the code doesn't "know" that that particular string is supposed to be a handle, but there's nothing stopping me from simply using it as one in the C code, and throwing an error if I can't find a hash entry for it. I suspect that the above is NOT the recommended approach, but I don't fully understand why. It seems "simpler" to me... But maybe that's only because I don't yet understand the Tcl_Obj integration approach, and what benefits it provides. On Fri, Feb 01, 2019 at 10:29:47AM +0100, Zoran Vasiljevic wrote: > You might want to look into: > > nsd/tclobj.c Hm, that has Ns_TclSetTwoPtrValue(), Ns_TclGetOpaqueFromObj(), and other black magic (or so it seems to me). Are there docs on these C APIs anywhere? Some of the stuff in "nsd/tclobj.c" seems be primarily concerned with the "address object type" (IP addresses?), but I get the sense that much of that code is much more general purpose than that. I see that "nsd/tcltime.c" uses the tclobj.c functions to implement an Ns_Time Tcl_Obj type, but I definitely don't understand how or why yet! -- Andrew Piskorski <at...@pi...> |
|
From: Zoran V. <zv...@ar...> - 2019-02-01 15:25:31
|
On Fri, 1 Feb 2019 09:59:09 -0500 Andrew Piskorski <at...@pi...> wrote: > So they do the exact same thing, and are completely interoperable? > Meaning I can have C and Tcl talk to each other via Ns_Var* in C and > nsv_* in Tcl? Great! Yes. > > I'm a little confused by (e.g.) Ns_VarGet(), though. Why is its first > arg a string for the "server"? What does that mean, and where am I > supposed to get it from? This is the identification of the Naviserver's virtual server that is implicitly "known" to the Tcl code. The nsv_XXX is thus bound to a virtual server since your Tcl code executes within a specific virtual server. But on the C-level you need to specify the virtual server explicitly because your C code may operate for all known virtual servers or just bunch of them. > > Also, why are NsTclNsvGetObjCmd() and Ns_VarGet() > implemented independently, rather than NsTclNsvGetObjCmd() calling > Ns_VarGet()? I assume there's some good reason for this, but I'd like > to understand what it is... Well, simplicity, that is. In essence, one could have done that but it would be more complicated. The former operates mainly with Tcl objects from within one Tcl script whereas the latter is mainly (supposed to be) used from C code that has no relation to any Tcl interp/script. If you compare the guts of both calls you will see something like LockArray/CreateHashEntry which is what "counts". Everything else is glue. HTH, Zoran |
|
From: Andrew P. <at...@pi...> - 2019-02-01 14:59:17
|
On Fri, Feb 01, 2019 at 10:22:46AM +0100, Zoran Vasiljevic wrote: > Andrew Piskorski <at...@pi...> wrote: > > > I know that the Tcl nsv_get command is implemented by > > NsTclNsvGetObjCmd(). But in the NaviServer C code, what is > > Ns_VarGet() for? I don't see it used anywhere. Is Ns_VarGet() a C > > interface to exact same nsv_get functionality, or is it for something > > completely different? > > The Ns_VarXXX is a programatic interface (from C) > to the nsv (Naviserver Shared Variables) storage. So they do the exact same thing, and are completely interoperable? Meaning I can have C and Tcl talk to each other via Ns_Var* in C and nsv_* in Tcl? Great! I'm a little confused by (e.g.) Ns_VarGet(), though. Why is its first arg a string for the "server"? What does that mean, and where am I supposed to get it from? Also, why are NsTclNsvGetObjCmd() and Ns_VarGet() implemented independently, rather than NsTclNsvGetObjCmd() calling Ns_VarGet()? I assume there's some good reason for this, but I'd like to understand what it is... -- Andrew Piskorski <at...@pi...> |
|
From: Zoran V. <zv...@ar...> - 2019-02-01 10:44:59
|
On Thu, 31 Jan 2019 15:46:59 -0500 Andrew Piskorski <at...@pi...> wrote: > And I already know how to call > NsTclNsvGetObjCmd() etc. from C. But the Nsv functions all operate > ONLY on strings... Oh yes, never use NsXXX calls in your code as these are not exported and may change anytime. Always use Ns_XXX instead. The only safe way to share data between threads in Tcl is over strings. One can go to some extent and try making Tcl objects shareable but for that you need to know the object internals which is a great PITA and does not always work well, except for a limited set of object types. Cheers, Zoran |
|
From: Zoran V. <zv...@ar...> - 2019-02-01 09:30:00
|
On Thu, 31 Jan 2019 15:46:59 -0500 Andrew Piskorski <at...@pi...> wrote: > In general, what's the recommended way for Tcl and/or NaviServer > programmers to allocate handles to C-level pointers, and then access > those handles and pointers from more than one Tcl thread? Can anyone > point me to examples of code doing it the right way? You might want to look into: nsd/tclobj.c Cheers, Zoran |
|
From: Zoran V. <zv...@ar...> - 2019-02-01 09:22:58
|
On Thu, 31 Jan 2019 14:48:11 -0500 Andrew Piskorski <at...@pi...> wrote: > I know that the Tcl nsv_get command is implemented by > NsTclNsvGetObjCmd(). But in the NaviServer C code, what is > Ns_VarGet() for? I don't see it used anywhere. Is Ns_VarGet() a C > interface to exact same nsv_get functionality, or is it for something > completely different? The Ns_VarXXX is a programatic interface (from C) to the nsv (Naviserver Shared Variables) storage. Cheers, Zoran |
|
From: Andrew P. <at...@pi...> - 2019-01-31 20:47:09
|
I'm using a C library that allocates some opaque structure, and returns a pointer to it. I'd like to save that pointer in a safe process-wide place, generate my own unique integer key (easy), and then return that simple key as a handle to my Tcl code. I also need to be able to find and use the C pointer from multiple threads. In the past I've done that sort of thing with my own Tcl_HashTable in C, but those are a bit low level, not that friendly to deal with. I'd rather use nsv_set, nsv_get, etc.! And I already know how to call NsTclNsvGetObjCmd() etc. from C. But the Nsv functions all operate ONLY on strings... I suppose I should be able to (very carefully) convert my pointer value to a string and back each time, but that seems awfully kludgy. Is there a better way? Something that lets me store the binary pointer values as is, without string conversions? The Tcl_HashTable C code DOES let me do that. In general, what's the recommended way for Tcl and/or NaviServer programmers to allocate handles to C-level pointers, and then access those handles and pointers from more than one Tcl thread? Can anyone point me to examples of code doing it the right way? Is Tcl_HashTable the right tool to use, or is some nicer Nsv-like interface available? (Nsv itself is implemented using Tcl_HashTable, of course.) Does Zoran's Thread tsv package have any additional functionality to help with this, beyond what NaviServer's built-in nsv commands can do? I found these wiki pages, but these (handles as values) approaches seem to basically just be fancier versions of what I've done before with Tcl_HashTable entries, and don't address use from multiple threads at all: https://wiki.tcl-lang.org/page/Tcl+Handles https://wiki.tcl-lang.org/page/Creating+and+Using+Tcl+Handles+in+C+Extensions (Thanks in advance for your help and advice!) -- Andrew Piskorski <at...@pi...> |
|
From: Andrew P. <at...@pi...> - 2019-01-31 19:48:20
|
I know that the Tcl nsv_get command is implemented by NsTclNsvGetObjCmd(). But in the NaviServer C code, what is Ns_VarGet() for? I don't see it used anywhere. Is Ns_VarGet() a C interface to exact same nsv_get functionality, or is it for something completely different? -- Andrew Piskorski <at...@pi...> |
|
From: Andrew P. <at...@pi...> - 2019-01-30 21:44:18
|
On Tue, Jan 29, 2019 at 12:28:38PM -0500, Andrew Piskorski wrote: > Hm, Ns_ThreadId() returns type uintptr_t, so why does LogToDString() > use PRIxPTR in the first place? Shouldn't it use PRIuPTR, with a "u" > for unsigned? I think so, so I switched to that. Nope, I was confused. PRIxPTR takes an unsigned int and displays it as hexadecimal output, which is what NaviServer wants. PRIuPTR also takes an unsigned int, but displays it as decimal, so by using that I inadvertently changed the log output. So I reverted that (trivial) change in my BitBucket fork. Here's a nice cheat sheet listing PRIxPTR and the other C99 macros for integers: https://en.cppreference.com/w/c/types/integer > Here's the fix I'm using now: > > https://bitbucket.org/apiskors/naviserver/commits/16e6658319ae9595e7d874e68086e482f2f74f85 > Fix garbled log output on Windows, PRIuPTR now gives I64u or I32u there. As Gustaf explained to me, Microsoft added inttypes.h with PRIxPTR and other (limited) C99 support way back in 2013: https://blogs.msdn.microsoft.com/vcblog/2013/07/19/c99-library-support-in-visual-studio-2013/ So my fix for PRIxPTR on Windows wouldn't be necessary if I was using a recent Microsoft compiler and libraries with proper inttypes.h support. However, it's a good and useful change for people stuck with older compilers, and should not hurt anything for newer compilers. Btw, "naviserver/include/nsthread.h" has "#ifndef HAVE_CONFIG_H" definitions in it for MSVC, c. line 280. I found that if I add a "#define HAVE_INTTYPES_H 1" there, then the Windows build definitely tries to use inttypes.h - but in my case, fails because it can't find the file. -- Andrew Piskorski <at...@pi...> |
|
From: Andrew P. <at...@pi...> - 2019-01-29 17:28:46
|
On Tue, Jan 29, 2019 at 12:02:23AM +0100, Gustaf Neumann wrote: > In case, the name is for unknown reason not initialized, the change [1] > should help. > https://bitbucket.org/naviserver/naviserver/commits/a1a07e7f1397e71809c74ca3a4bf93649affd8b2 That didn't make any difference. > The function Ns_ThreadId() returns uintptr_t, which is printed with > format-code PRIxPTR, maybe here is something > wrong. There is some indication [2], that the C99 prefix "ll" does not > work in all versions of MSC. > https://stackoverflow.com/questions/18107426/printf-format-for-unsigned-int64-on-windows Ah ha, thanks! I'm building NaviServer 32-bit on Windows. The garbled output was due to using that "PRIxPTR" macro to format the ThreadId. If I replace that format specifier with a either plain "%u" or "%I32u", then everything works fine! Hm, Ns_ThreadId() returns type uintptr_t, so why does LogToDString() use PRIxPTR in the first place? Shouldn't it use PRIuPTR, with a "u" for unsigned? I think so, so I switched to that. Here's the fix I'm using now: https://bitbucket.org/apiskors/naviserver/commits/16e6658319ae9595e7d874e68086e482f2f74f85 Fix garbled log output on Windows, PRIuPTR now gives I64u or I32u there. And here's a pull request that includes that, plus a couple other (very minor) fixes: https://bitbucket.org/naviserver/naviserver/pull-requests/20/fix-garbled-log-output-on-windows/diff -- Andrew Piskorski <at...@pi...> |
|
From: Gustaf N. <ne...@wu...> - 2019-01-29 15:29:21
|
On 29.01.19 13:49, Gustaf Neumann wrote: > > I see, ... i can reproduce this, hopefully i can look into this today > evening or tomorrow. > just a quick thought, although it does not solve the not-for-me reproducible issue of [2] the change [1] seem to fix all known problems problems mentioned here. So, [2] is then probably again open, but the issues mentioned here are more serious. We should stick for this solution at least for Tcl 8.5. -gn [1] https://bitbucket.org/naviserver/naviserver/commits/24796ea1438eddfdbe3288891775f702c9b30a63 [2] https://stackoverflow.com/questions/49031790/how-can-i-debug-a-tcl-8-6-error-in-clock-tcl -- Univ.Prof. Dr. Gustaf Neumann WU Vienna Institute of Information Systems and New Media Welthandelsplatz 1, A-1020 Vienna, Austria |
|
From: Gustaf N. <ne...@wu...> - 2019-01-29 12:49:42
|
On 29.01.19 13:13, David Osborne wrote:
> The output you have pasted is the same as mine up to that point. I
> wouldn't have seen an error by then either.
> It's the *subsequent* calls to "clock scan" and "::try {} on error .."
> which then fail.
I see, ... i can reproduce this, hopefully i can look into this today
evening or tomorrow.
-gn
[29/Jan/2019:13:45:13][22500.7efe1caec740][-main-] Notice: nsmain: NaviServer/4.99.17 (bb78064b1137+ default tip) starting
...
[29/Jan/2019:13:45:13][22500.7efe0efa3700][-conn:default:0:0-] Notice: thread initialized (0.003295 secs)
%
% ::try { puts trying } on error [list message options] { puts error }
can't read "magicCodes": no such variable
%
|
|
From: David O. <da...@qc...> - 2019-01-29 12:13:26
|
Thanks for looking at that Gustaf.
I can definitely relate to Zoran's comment:
> "In the moment all is pretty "spooky" and it is difficult to grasp..."
Your change doesn't seem to help the problems I'm looking at here
unfortunately.
The output you have pasted is the same as mine up to that point. I wouldn't
have seen an error by then either.
It's the *subsequent* calls to "clock scan" and "::try {} on error .."
which then fail.
To illustrate I made a bash script (which needs run by sudo) which starts
up naviserver to a prompt:
#!/bin/bash
hg clone https://bitbucket.org/naviserver/naviserver
cd naviserver
./autogen.sh --disable-ipv6 --with-tcl=/usr/lib/tcl8.5 --enable-rpath
--enable-threads --enable-symbols
make
make install
cat <<EOF >/usr/local/ns/tcl/test.tcl
package require try
::try {
ns_log Notice "[clock scan "+365 days"]"
} on error [list message options] {
ns_log Notice "$message"
ns_log Notice "$options"
}
EOF
chown nsd:nsd -R /usr/local/ns/logs
/usr/local/ns/bin/nsd -u nsd -t /usr/local/ns/conf/nsd-config.tcl -c
After this bash script is run, on Debian Stretch, the following commands
will fail in the Naviserver interpreter:
% ::try { puts trying } on error [list message options] { puts error }
can't read "magicCodes": no such variable
% clock scan "+365 days"
invalid command name "::tcl::clock::scan"
It's quite a contrived testcase - the same errors are occurring more
naturally within our "full" environment.
--
David
|
14 messages has been excluded from this view by a project administrator.
