Menu
▾
▴
Re: [naviserver-devel] NaviServer 4.99.25 available
|
From: Gustaf N. <ne...@wu...> - 2023-06-14 10:19:25
|
Dear Sassy,
You are referring to a change that was released more than 3 years ago
(4.99.19).
Log file sanitizing works as expected, namely it prints invisible
characters in a hex representation (in your case tab characters).
Sanitizing was required by security audits, since otherwise, it would
be possible to execute code by looking into the log file, one could
obfuscate the log file, and confuse log file analyzer that alarm when
suspect activities are noticed, etc.
Certainly, the harm caused by a tab character alone is limited, but when
debugging and analyze problems, it is often important to distinguish
between a tab and some spaces.
You can easily replace the hex notation in the log file via "sed", such
as e.g.
cat log/error.log | sed -En 's/\\x09/\t/p' | more
or the like.
all the best
-gn
On 13.06.23 20:51, Sassy Natan wrote:
> Hi All,
>
> After upgrading my server to the latest version, my log file is broken.
>
> For example I see UTF-8 as special characters.
>
> I check the readme but the sanitizelogfiles 2 doesn't seems to work as
> expected.
> I have in my configuration:
>
> set debug true
> ns_logctl severity "Debug(sql)" on
>
> Any ideas?
>
> Here is example:
> : ],is_break_pcols)) as hrs_wo_break
> : FROM wt_payroll_analysis_pp
> : join (select sid,agg_array(case when is_break_pcol=true then
> 99999 else 0 end) as is_break_pcols
> : \x09\x09\x09from ( select * from wt_et_cols_defs where sid=410000
> order by pcol_number )pc_defs group by sid) pdefs using (sid)
> : \x09\x09\x09\x09WHERE sid=410000 and employee_id =40599
> : \x09\x09\x09\x09and
> to_date(pp_year||'-'||pp||'-01','yyyy-mm-dd') between
> to_date('2023-6-01','yyyy-mm-dd')- interval '6 month'
> : \x09\x09\x09\x09\x09\x09\x09and (
> to_date('2023-6-01','yyyy-mm-dd') - interval '1 day' )::date
> : \x09\x09\x09\x09UNION
> : \x09\x09\x09\x09 select
> sid,employee_id,pp,pp_year,ppid,to_date(pp_year||'-'||pp||'-01','yyyy-mm-dd')
> as day,null2zero(hrs_wo_break)*3600
> : \x09\x09\x09\x09 from pp_pa_table
> : \x09\x09\x09\x09) ggg
> : \x09\x09\x09\x09group by sid,employee_id,pp,pp_year
>
>
> Thanks
> Sassy
>
> On Mon, May 1, 2023 at 6:55 PM Gustaf Neumann <ne...@wu...> wrote:
>
> Dear all,
>
> I am glad to announce that the release of NaviServer 4.99.25 is
> available at SourceForge [1]. This release is mostly a bug-fix
> release. The forthcoming version 5.0 of NaviServer will contain
> several new features omitted in this bug-fix branch. In case, you are
> building NaviServer from the Bitbucket repository, please note that
> the release 4.99.25 is in the branch release/4.99 (bug fix branch for
> the NaviServer 4.99 family). New development happens in the "main"
> branch of the repository, leading to NaviServer 5.*.
>
> See below for a summary of the changes.
>
> Many thanks to the contributors of this release:
>
> Andrew Piskorski
> Antonio Pisano
> Brian Fenton
> Gustaf Neumann
> Hector Romojaro
> Joe Oldak
> Khy Huang
> Oleg Oleinick
> Zoran Vasiljevic
>
> All the best!
>
> -gustaf neumann
>
> [1]
> https://sourceforge.net/projects/naviserver/files/naviserver/4.99.25/
>
> =======================================
> NaviServer 4.99.25, released 2023-05-01
> =======================================
>
> 132 files changed, 3957 insertions(+), 2068 deletions(-)
>
> New Features:
> -------------
>
> - Added meta-information to configuration values
>
> NaviServer can now report, what configuration values provided in
> the configuration file were actually used, what their default
> values are, and whether these values were specified or not (using
> the default values). With this information, administration
> (e.g. migration) becomes easier. The NaviServer module "nsstats"
> shows this meta information via the web interface.
>
> This functionality is provided via the new option "-filter" for the
> command "ns_configsection ... section". When the "-filter" option
> is used, different kinds of information about the parameters is
> returned from the specified section.
>
> "-filter unread":
>
> Returns the parameters, which were set during configuration (i.e.,
> in the configuration file) but which were not read in the startup
> phase of the server. This option is useful to determine
> e.g. typographical errors of specified parameter names.
>
> "-filter defaulted":
>
> Returns the parameters, from which the default values were read
> (i.e., which were not explicitly set)
>
> "-filter defaults":
>
> Returns the defaults of the parameter. This is useful for
> contrasting the actual values with the default values for
> Parameters, e.g. in a web based interface.
>
> - ns_set reform (per default deactivated in 4.99, but activated in 5.*)
>
> The classical implementation for ns_sets uses separately malloced
> storage for every attribute name and attribute value. So, e.g., for
> 1000 ns_sets with 20 members each, this means 1,000*20*2 = 40,000
> malloc/free operations, e.g., for a single db query! Although the
> malloc implementations have improved over the years, these will
> require many lock operations, especially under load, where many
> threads might perform many concurrent malloc operations. One other
> consequence is that the allocated memory will be scattered over
> address space, which has bad implications for CPU caching.
>
> The new implementation uses for one "ns_set" a single Tcl_DString
> keeping all attribute names and attribute values. This reduces the
> malloc operations and improves memory locality, such that cache
> hits will improve.
>
> One caveat of this change is that modules using "ns_set" have to be
> recompiled, since the full C-level data structure of the "ns_set"
> is exposed. Therefore, adding a member causes a binary
> incompatibility. One other potential problem is that C-level
> modules using the Ns_Set* API have to make sure that long-living
> string values are copied (this was necessary before as well, but
> was in many cases no problem, when the "ns_sets" were seldom
> updated).
>
> For high compatibility, this feature is deactivated per default in
> the 4.99.* series and can be activated by setting the compile-time
> C macro "NS_SET_DSTRING".
>
>
>
> API changes:
> ------------
>
> API extensions:
> - Provide a new interface ending with *Sz to provide string sizes.
> This reduces the need of strlen() operations.
> * Ns_SetCreateSz()
> * Ns_SetIUpdateSz()
> * Ns_SetPutSz()
> * Ns_SetPutValueSz()
> * Ns_SetUpdateSz()
>
> - New API calls for "ns_set" reform
> * Ns_SetClearValues(): clear the values for all keys
> * Ns_SetDataPrealloc(): creating ns_sets with preallocated values
> to avoid resize operations
> * NsSetResize()
> * NsHeaderSetGet()
>
> - Ns_ConfigSet(const char *section, const char *key, const char *name)
> The last argument is new and allows one to create named sets
> (previously, all such sets were unnamed)
>
> - NsHexPrint(): Print the potentially binary content of a buffer
> in human-readable form.
>
> - Ns_RelativeTime(Ns_Time *relTimePtr, Ns_Time *timePtr)
> This call implements the inverse operation of Ns_AbsoluteTime(),
> and is used mostly to make debug messages eye-friendly.
>
>
> Performance Improvements:
> -------------------------
>
> - Replaced malloc operation per log entry by thread local variable in
> system log implementation.
>
> - When NaviServer 4.99.25 is compiled with NS_SET_DSTRING supportm
> the following preliminary performance results were measured from
> the "ns_set" reform (see above). The tests were performed on
> openacs.org <http://openacs.org> (Xeon Gold 6226R CPU @ 2.90GHz, 32 cores,
> hyper-threading enabled). The test executes the SQL query
>
> select * from acs_objects limit 1000
>
> 100 times in sequence. This test is run in 1 to 30 concurrent
> threads. With 30 threads, 3mio tuples are retrieved, and 72 mio
> malloc/free operations are needed alone for the retrieved values.
>
> Before (classical ns_set with many mallocs):
>
> threads 1 total 4606.787 ms avg 3285.25 ms
> threads 5 total 4595.358 ms avg 3493.07 ms
> threads 10 total 4804.193 ms avg 3755.93 ms
> threads 20 total 6279.524 ms avg 4569.16 ms
> threads 30 total 8966.427 ms avg 6618.58 ms
>
> After reform (using common Tcl_DString per tuple):
>
> threads 1 total 4524.645 ms avg 3242.54 ms
> threads 5 total 4251.266 ms avg 3450.09 ms
> threads 10 total 4656.795 ms avg 3665.31 ms
> threads 20 total 5934.105 ms avg 4671.38 ms
> threads 30 total 7384.591 ms avg 5642.76 ms
>
> To summarize, the improvement increases under higher load (with
> more parallel threads). E.g., with 30 threads, the total time
> improved by 17%.... leading also to a smaller RSS. These tests were
> not performed under "clinical" conditions.
>
> The new Tcl API call "ns_set size" can be used to pre-allocate
> larger ns_sets, such that the usual Tcl_DString growing policy does
> not kick in, reducing further realloc() operations.
>
>
> Bug Fixes:
> ----------
>
> - Fixed potential crash in "ns_accesslog extendedheaders XXXX".
> Setting extended headers via configuration file was correct, but
> changing it dynamically via "ns_accesslog extendedheaders .." was
> broken. (Issuehttps://sourceforge.net/p/naviserver/bugs/91/)
>
> - "ns_conn location": Fixed potential race condition
>
> It was possible that "ns_conn location" could return inconsistent
> results in a single request, when the underlying sockPtr was
> aggressively reused. Now, the value of the location member is
> copied to the connection structure instead of being shared with
> the socket structure (as before).
>
> - "ns_cache_eval -force": Fixed potential race condition
>
> There was a problem with "ns_cache_eval -force", where the system
> relied on the existence of a pre-existing entry, but in case the
> entry was flushed in the meantime problems a crash might have
> happened. Now the value during the "-force" call is cleared exactly
> like in the case of an unset operation. The null-value operations
> are already protected until these are finished in various places
> in the code.
>
> - "ns_socknread": Fix potentially wrong result for buffered channels
>
> - Bug fixes for problems showing up under MS Windows:
>
> * Make sure that the output variable of Ns_ObjvIndex() is always an
> integer. Previously, the output variable was in two places a
> character variable, causing crashes under MS Windows.
>
> * Handle incompatibility in setlocale() under MS Windows. Under
> MS Windows, later calls to setlocale() overwrite the string
> returned by former calls. So, it is necessary to copy of the
> returned string of a setlocale() call under MS Windows.
>
> * Handling linking problems: MS Windows requires explicit handling
> when importing symbols from .dll files (Ns_LogSqlDebug,
> NS_intTypePtr)
>
> * Pass error codes from low-level function SockRecv() and
> SockSend() via variables. This change makes sure the real error
> code (immediately after the I/O operation) is passed to the
> caller to avoid missed error cases and weird error message
> (e.g. under windows).
>
>
> - Bug fixes for ADP parser:
>
> * Support for greater than sign ">" inside attribute values.
>
> Previously, NaviServer determined the terminating end-of-tag
> character as literally the first greater than sign, no matter if
> this was used as attribute values between single or double
> quotes.
>
> The new version supports such values, since the "Living Standard
> of HTML" [1] allows the use of less than "<" and greater than
> ">" signs inside attribute values as long these are between
> single or double quotes. The guide [2] just recommends using
> character escapes for "<", ">" and "&".
>
> [1]https://html.spec.whatwg.org/multipage/syntax.html#syntax-attribute-value
> [2]https://www.w3.org/International/questions/qa-escapes
>
> This problem was reported by Wolfgang Winkler
>
> * Clear ADP flags in case of errors (this error was present for
> many years)
>
> Previously, The following ADP page could lead to a full
> breakdown of the server, since the error states of the server
> were never cleared, and subsequent requests served be the same
> interpreter could lead to the old error states.
>
> Many thanks to Oleg Oleinick for reporting and the great test
> cases.
>
> - Provide better HTTP status code for early errors: When the driver
> terminates already a request, it might be the case that the error
> flags were not read out. In such cases, we provide now more
> specific status codes rather than 400.
>
> - ns_http:
>
> * Improved robustness with domain names resolving against many IP
> addresses: When a domain name is resolved against many IP
> addresses, and all these IP addresses block (connect returns "in
> progress") then the old code might have looped infinitely. Now
> the code respects the provided timeout (default set to 5s) per
> resolved IP address.
>
> Many thanks to Joe Oldak for pointing out the problem and
> leading us to the solution.
>
> * Fixed timeout handling during TLS handshake: This problem could
> appear, when the TCP connection to a server succeeded quickly,
> but the TLS handshake was taking a long time, without a raising
> a timeout exception, although the timeout time has expired.
>
> - Set the default server before the init-scripts is called to make it
> accessible from there
>
> - Provide compatibility between in-memory and file-based form-data
> handling for invalid characters: Since the file based
> implementation uses a fallback-charset of iso8859-1 when parsing
> form data in POST requests (in order to be able to extract
> "_charset_" data), the in-memory based variant does now the
> same. The decision of which parser is used is taken based on the
> size of the form data.... and should therefore be consistent.
>
> - nscp: Fixed problem, when the nscp module is activated but no
> "users" section is specified. Many thanks to Andrew Piskorski for
> reporting the problem.
>
> - Security Improvement: Added protection against sneaking in fake
> NAME.tmpfile entries into form fields, when performing file upload
> operations.
>
>
> Documentation improvements:
> ---------------------------
>
> - Improved the following man pages:
>
> doc/src/manual/adp-overview.man
> doc/src/manual/tcl-lib-nstrace.man
> doc/src/naviserver/commandlist.man
> doc/src/naviserver/ns_adp.man
> doc/src/naviserver/ns_adp_ctl.man
> doc/src/naviserver/ns_config.man
> doc/src/naviserver/ns_conn.man
> doc/src/naviserver/ns_connchan.man
> doc/src/naviserver/ns_cookie.man
> doc/src/naviserver/ns_crypto.man
> doc/src/naviserver/ns_http.man
> doc/src/naviserver/ns_ictl.man
> doc/src/naviserver/ns_locationproc.man
> doc/src/naviserver/ns_parsehostport.man
> doc/src/naviserver/ns_return.man
> doc/src/naviserver/ns_set.man
> doc/src/naviserver/ns_setprivileges.man
>
>
> Configuration Files:
> --------------------
>
> - New feature for the sample configuration file openacs-config.tcl
>
> The following variables (with prefix oacs_) can be taken from the
> shell variables. This makes it easier to run similar variants of
> NaviServer instances from a single configuration file, while
> providing variables from the command line. This is e.g. useful for
> Docker or cluster setups.
>
> CookieNamespace
> bindir
> cachingmode
> db_host
> db_name
> db_port
> db_user
> homedir
> hostname
> httpport
> httpsport
> ipaddress
> logroot
> nscpport
> server
> serverroot
> smtpdport
>
> One can use e.g.the following command to change some ports and the
> log file during startup
>
> oacs_httpport=8000 oacs_httpsport=8443 oacs_smtpdport=2526 \
> oacs_logroot=/var/www/XXXX/log-node1/ \
> /usr/local/ns/bin/nsd -t /var/www/XXXX/etc/openacs-config.tcl ...
>
> - further updates for openacs-config.tcl:
> * Added sample nssmtpd configuration
> * Added placeholder for ClusterSecret
> * Reflect recent Oracle requirements (tested with Oracle 19c)
> * Added documentation for "StaticCSP", "CookieNamespace",
> "NsShutdownWithNonZeroExitCode", "LogIncludeUserId"
>
> - Updated all sample configuration files
>
>
> Code Changes:
> -------------
>
> - Added and updated predefined MIME types based on
> https://www.iana.org/assignments/media-types/media-types.xhtml
>
> - Added compile time macro NS_VERBOSE_MALLOC to identify frequent
> *alloc operations
>
> - Print version of zlib during startup
>
> - OpenSSL maintenance:
>
> * Improved robustness for OCSP with OpenSSL 3.*
>
> The previous version crashed with OpenSSL 3.*, when OCSP was
> tried on self-signed certificates. Aside of the fact that OCSP
> does not make sense for self-signed certificates, the server
> should not crash in such situations.
>
> * Fixed ns_crypto::aead::encrypt/decrypt test under OpenSSL 1.1.1
> (as shipped per default, e.g. on Ubuntu 18.04.4)
>
> In short, the problem was that with this version of OpenSSL,
> setting empty additional authenticated data (AAD) behaved
> differently from other versions, namely it was clearing
> incorrectly (forgetting) the information that the initialization
> vector (IV) was already set. An upgrade of OpenSSL fixed the
> problem. However, with these changes, also the stock version of
> OpenSSL can be used. As a byproduct, better error messages are
> now produced, the code received more cleanup (e.g. explicit
> initialization, etc.)
>
>
> - Automated testing:
>
> * Setup if Bitbucket + GitHub pipelines for automated regression
> testing with multiple versions of components
>
> For NaviServer 4.99 the current setup performs tests with gcc-10
> + gcc-11, Tcl 8.6.13 + 8.7a5, NSF 2.4,0 + 2.4.0, tDOM 0.9.1 +
> 0.9.3, extra modules: nsdbpg nsdbi nsdbipg nsudp nscoap nssmtpd.
>
> https://github.com/nm-wu/naviserver-mirror/actions
>
> * Improved robustness of regression test when running with the
> docker networking setup
>
> * Force nonzero exit code when regression test fails
>
> * Added parameter "-timeout" to call of regression test cases
> Previously, the timeout was hard-wired to 3 seconds. One can now
> call a test with e.g. "nstest::http -timeout 1s ... GET ..."
>
> * adp_compress.test: removed trailing newline to ease
> cross-platform regression tests
>
> * Prefer standard Tcl test constraint "macOrUnix" over own solution
>
> * Extended regression tests with more test cases
>
> - Code management:
> * Changed name of branch from "master" to "main"
>
> - Code Cleanup
> * Improved type cleanness
> * Removed deprecated calls to "sprintf"
> * Improved portability for Tcl 8.7* (handling of binary data)
>
> - Improved comments, fixed typos
>
>
> Changes in NaviServer Modules:
> ==============================
>
> 39 files changed, 9658 insertions(+), 1781 deletions(-)
>
> General:
> Adjust to necessary API changes in NaviServer (contains as well
> support for the forthcoming release of NaviServer 5)
>
>
> nsdbpg:
> -------
>
> - new pg-driver specific command: ns_pg_prepare /sql/
>
> Return a dict building a prepared statement for the passed-in SQL
> statement. The dict contains the keys "sql" and "args". The
> function is used by e.g. OpenACS to generate prepared statements
> from SQL commands with bind variables.
>
> - Raise exception when a value for a bind variable contains a NUL character.
> This value is explicitly forbidden in text strings passed to PostgreSQL.
>
> - Let "ns_pg" report available subcommands even when handle is not
> specified. This makes the command compatible with the "icanuse"
> feature in OpenACS.
>
>
> nsstats:
> --------
>
> - HTTP client log analysis:
> * Provide charts for performance (using highcharts via CDN)
> * Provide charts on request frequency (using highcharts via CDN)
> * Provide a summary table for HTTP client requests
> * Improved robustness against invalid URLs (containing unescaped
> spaces)
> * Added support for selection of different HTTP client log files
> via web interface
>
> - "Process" page:
> * Added percentage of request distribution over connection pools
> * Added information about the connected client
> * Added more detailed version information
>
> - Added cache configuration to output when looking at a single cache
>
> - Improved "log file" analysis
> * Automated stripping of color codes
>
> * Added filter option. The filter can be used to grep for (ID)
> strings in both the system and access logs, providing a summary
> for the traces of a request in the access log and system log on
> a single place.
>
> - Added default and usage information to "Config Parameters" page
>
>
> nsoracle:
> ---------
>
> - Fixed bug when streaming LOB content to connection. The old code
> did not distinguish between binary and non-binary content. This
> bug was discussed in
> https://openacs.org/forums/message-view?message_id=5693661
>
> Bumped version number to 2.9
>
> - switched to plain Debug handling for debugging the driver
>
> The handling of Ns_LogSqlDebug is performed inside nsdb, including
> also the printout of (long) SQL statements. Previously, the driver
> was too chatty when Debug(sql) was turned on.
>
> - Added support for output columns of type SQLT_TIMESTAMP or SQLT_TIMESTAMP_TZ
>
> This change fixes a bug, where SQL queries of the form
>
> SELECT TO_TIMESTAMP(sysdate) FROM dual
>
> lead to errors for the form:
>
> Database operation "getrow" failed (exception 1406, "nsoracle.c:3659:Ns_OracleGetRow:
> error in `OCIStmtFetch ()': ORA-01406: fetched column value was truncated
>
> The driver needs for several output types special rules, where the
> timestamp cases were not supported so far. It is also recommended
> to set the according environment variables specifying the output
> format in the configuration server of NaviServer, such as e.g.
>
> set ::env(NLS_TIMESTAMP_FORMAT) "YYYY-MM-DD HH24:MI:SS.FF6"
> set ::env(NLS_TIMESTAMP_TZ_FORMAT) "YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM"
>
> For testing in you local Oracle installation, you might test the
> output formats (and the required sizes with the following snippet
> for sqlplus:
>
> COLUMN localtimestamp format a40
> COLUMN systimestamp format a40
> COLUMN ts_bytes format a80
>
> alter session set nls_timestamp_format='YYYY-MM-DD HH24:MI:SS.FF6';
> select localtimestamp, length(localtimestamp), dump(localtimestamp) ts_bytes from dual;
>
> alter session set nls_timestamp_tz_format='YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM';
> select systimestamp, length(systimestamp), dump(systimestamp) ts_bytes from dual;
>
> alter session set nls_timestamp_tz_format='YYYY-MM-DD HH24:MI:SS.FF6 TZR';
> select systimestamp, length(systimestamp), dump(systimestamp) ts_bytes from dual;
>
>
> letsencrypt:
> ------------
>
> - Added option to produce certificates with ECDSA:
>
> Prior to this change, all certificates were using RSA keys.
> Since a while, keys based on elliptic curves became the preference
> of letsencrypt.
>
>
> nsshell:
> --------
>
> - Fixed a bug in "ns_conn" emulation, when the "kernel" was not correctly identified
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>
>
> --
> Regards,
>
> Sassy Natan
> 972-(0)54-2203702
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
--
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"
|
