Menu
▾
▴
naviserver-devel — Developer discussion
You can subscribe to this list here.
| 2005 |
Jan
|
Feb
(53) |
Mar
(62) |
Apr
(88) |
May
(55) |
Jun
(204) |
Jul
(52) |
Aug
|
Sep
(1) |
Oct
(94) |
Nov
(15) |
Dec
(68) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2006 |
Jan
(130) |
Feb
(105) |
Mar
(34) |
Apr
(61) |
May
(41) |
Jun
(92) |
Jul
(176) |
Aug
(102) |
Sep
(247) |
Oct
(69) |
Nov
(32) |
Dec
(140) |
| 2007 |
Jan
(58) |
Feb
(51) |
Mar
(11) |
Apr
(20) |
May
(34) |
Jun
(37) |
Jul
(18) |
Aug
(60) |
Sep
(41) |
Oct
(105) |
Nov
(19) |
Dec
(14) |
| 2008 |
Jan
(3) |
Feb
|
Mar
(7) |
Apr
(5) |
May
(123) |
Jun
(5) |
Jul
(1) |
Aug
(29) |
Sep
(15) |
Oct
(21) |
Nov
(51) |
Dec
(3) |
| 2009 |
Jan
|
Feb
(36) |
Mar
(29) |
Apr
|
May
|
Jun
(7) |
Jul
(4) |
Aug
|
Sep
(4) |
Oct
|
Nov
(13) |
Dec
|
| 2010 |
Jan
|
Feb
|
Mar
(9) |
Apr
(11) |
May
(16) |
Jun
|
Jul
|
Aug
|
Sep
(1) |
Oct
|
Nov
|
Dec
|
| 2011 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(1) |
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2012 |
Jan
(7) |
Feb
(3) |
Mar
|
Apr
|
May
|
Jun
(3) |
Jul
|
Aug
|
Sep
|
Oct
(92) |
Nov
(28) |
Dec
(16) |
| 2013 |
Jan
(9) |
Feb
(2) |
Mar
|
Apr
(4) |
May
(4) |
Jun
(6) |
Jul
(14) |
Aug
(12) |
Sep
(4) |
Oct
(13) |
Nov
(1) |
Dec
(6) |
| 2014 |
Jan
(23) |
Feb
(19) |
Mar
(10) |
Apr
(14) |
May
(11) |
Jun
(6) |
Jul
(11) |
Aug
(15) |
Sep
(41) |
Oct
(95) |
Nov
(23) |
Dec
(11) |
| 2015 |
Jan
(3) |
Feb
(9) |
Mar
(19) |
Apr
(3) |
May
(1) |
Jun
(3) |
Jul
(11) |
Aug
(1) |
Sep
(15) |
Oct
(5) |
Nov
(2) |
Dec
|
| 2016 |
Jan
(7) |
Feb
(11) |
Mar
(8) |
Apr
(1) |
May
(3) |
Jun
(17) |
Jul
(12) |
Aug
(3) |
Sep
(5) |
Oct
(19) |
Nov
(12) |
Dec
(6) |
| 2017 |
Jan
(30) |
Feb
(23) |
Mar
(12) |
Apr
(32) |
May
(27) |
Jun
(7) |
Jul
(13) |
Aug
(16) |
Sep
(6) |
Oct
(11) |
Nov
|
Dec
(12) |
| 2018 |
Jan
(1) |
Feb
(5) |
Mar
(6) |
Apr
(7) |
May
(23) |
Jun
(3) |
Jul
(2) |
Aug
(1) |
Sep
(6) |
Oct
(6) |
Nov
(10) |
Dec
(3) |
| 2019 |
Jan
(26) |
Feb
(15) |
Mar
(9) |
Apr
|
May
(8) |
Jun
(14) |
Jul
(10) |
Aug
(10) |
Sep
(4) |
Oct
(2) |
Nov
(20) |
Dec
(10) |
| 2020 |
Jan
(10) |
Feb
(14) |
Mar
(29) |
Apr
(11) |
May
(25) |
Jun
(21) |
Jul
(23) |
Aug
(12) |
Sep
(19) |
Oct
(6) |
Nov
(8) |
Dec
(12) |
| 2021 |
Jan
(29) |
Feb
(9) |
Mar
(8) |
Apr
(8) |
May
(2) |
Jun
(2) |
Jul
(9) |
Aug
(9) |
Sep
(3) |
Oct
(4) |
Nov
(12) |
Dec
(13) |
| 2022 |
Jan
(4) |
Feb
|
Mar
(4) |
Apr
(12) |
May
(15) |
Jun
(7) |
Jul
(10) |
Aug
(2) |
Sep
|
Oct
(1) |
Nov
(8) |
Dec
|
| 2023 |
Jan
(15) |
Feb
|
Mar
(23) |
Apr
(1) |
May
(2) |
Jun
(10) |
Jul
|
Aug
(22) |
Sep
(19) |
Oct
(2) |
Nov
(20) |
Dec
|
| 2024 |
Jan
(1) |
Feb
|
Mar
(16) |
Apr
(15) |
May
(6) |
Jun
(4) |
Jul
(1) |
Aug
(1) |
Sep
|
Oct
(13) |
Nov
(18) |
Dec
(6) |
| 2025 |
Jan
(12) |
Feb
|
Mar
(2) |
Apr
(1) |
May
(11) |
Jun
(5) |
Jul
(4) |
Aug
(2) |
Sep
(2) |
Oct
(6) |
Nov
|
Dec
|
|
From: Brian F. <bri...@ai...> - 2023-08-16 14:11:08
|
Hi Gustaf
many thanks for that. Our version of OpenACS didn't have the tracing feature, but I was able to add it.
We don't issue any "ns_set cleanup" ourselves, but when I added your recommendation of tracing "ns_set", I saw plenty in the logs. The docs say that "This command is autoamtically executed by ns_cleanup, which runs after every request, freeing all sets created via ns_set", so I presume that is where the calls come from.
Is there something in particular I should be looking for in the trace output? There is an enormous amount of information the logs.
I ran another backtrace and this time the error was a little bit different ("error: Cannot access memory at address 0x2" - see below). I will continue to try and find out which ns_set is the source of the issue (but there may be multiple).
Do you have any theory what change in 4.99.25 caused this to stop working?
thanks
Brian
[16/Aug/2023:14:05:13][13608.7ffff4b6f640][-conn:gustaf:default:0:1-] Notice: trace: ns_set iget t7 content-type
[16/Aug/2023:14:05:13][13608.7ffff4b6f640][-conn:gustaf:default:0:1-] Notice: trace: ns_set iget t7 content-type
[16/Aug/2023:14:05:13][13608.7ffff4b6f640][-conn:gustaf:default:0:1-] Notice: trace: ns_set ifind t7 cache-control
[16/Aug/2023:14:05:13][13608.7ffff4b6f640][-conn:gustaf:default:0:1-] Notice: trace: ns_set ifind t7 expires
[16/Aug/2023:14:05:13][13608.7ffff4b6f640][-conn:gustaf:default:0:1-] Notice: trace: ns_set size t7
[16/Aug/2023:14:05:13][13608.7ffff4b6f640][-conn:gustaf:default:0:1-] Notice: trace: ns_set key t7 0
[16/Aug/2023:14:05:13][13608.7ffff4b6f640][-conn:gustaf:default:0:1-] Notice: trace: ns_set size t7
[16/Aug/2023:14:05:13][13608.7ffff4b6f640][-conn:gustaf:default:0:1-] Notice: trace: ns_set update t7 Expires Wed, 16 Aug 2023 13:05:13 GMT
[16/Aug/2023:14:05:13][13608.7ffff4b6f640][-conn:gustaf:default:0:1-] Notice: trace: ns_set put t7 Pragma no-cache
[16/Aug/2023:14:05:13][13608.7ffff4b6f640][-conn:gustaf:default:0:1-] Notice: trace: ns_set put t7 Cache-Control no-cache
[16/Aug/2023:14:05:13][13608.7ffff4b6f640][-conn:gustaf:default:0:1-] Notice: trace: ns_set cleanup
free(): invalid next size (fast)
Thread 4 "nsd" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff4b6f640 (LWP 13801)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140737299019328) at ./nptl/pthread_kill.c:44
44 ./nptl/pthread_kill.c: No such file or directory.
(gdb) backtrace
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737299019328) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=140737299019328) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=140737299019328, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3 0x00007ffff7c7d476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4 0x00007ffff7c637f3 in __GI_abort () at ./stdlib/abort.c:79
#5 0x00007ffff7cc46f6 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7e16b8c "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#6 0x00007ffff7cdbd7c in malloc_printerr (str=str@entry=0x7ffff7e19740 "free(): invalid next size (fast)") at ./malloc/malloc.c:5664
#7 0x00007ffff7cddb1d in _int_free (av=0x7fffdc000030, p=0x7fffdd865320, have_lock=0) at ./malloc/malloc.c:4522
#8 0x00007ffff7ce04d3 in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3391
#9 0x00007ffff7bdb1e5 in ns_free (ptr=0x7fffdd865330) at memory.c:94
#10 0x00007ffff7f09b64 in Ns_SetFree (set=0x7fffdde381a0) at set.c:397
#11 0x00007ffff7f3e119 in NsTclSetObjCmd (clientData=0x7fffdc035570, interp=0x7fffdc005250, objc=2, objv=0x7fffdc245650) at tclset.c:330
#12 0x00007ffff79cb18e in Dispatch (data=0x7fffdc34afc8, interp=0x7fffdc005250, result=0) at /usr/local/src/tcl8.6.13/generic/tclBasic.c:4467
#13 0x00007ffff79cb21f in TclNRRunCallbacks (interp=0x7fffdc005250, result=0, rootPtr=0x0) at /usr/local/src/tcl8.6.13/generic/tclBasic.c:4503
#14 0x00007ffff79ca949 in Tcl_EvalObjv (interp=0x7fffdc005250, objc=1, objv=0x7fffdc2453f0, flags=2097168) at /usr/local/src/tcl8.6.13/generic/tclBasic.c:4226
#15 0x00007ffff79cd384 in TclEvalEx (interp=0x7fffdc005250, script=0x7ffff4b6e880 "ns_cleanup", numBytes=10, flags=0, line=1, clNextOuter=0x0,
outerScript=0x7ffff4b6e880 "ns_cleanup") at /usr/local/src/tcl8.6.13/generic/tclBasic.c:5372
#16 0x00007ffff79cc5d9 in Tcl_EvalEx (interp=0x7fffdc005250, script=0x7ffff4b6e880 "ns_cleanup", numBytes=10, flags=0) at /usr/local/src/tcl8.6.13/generic/tclBasic.c:5037
#17 0x00007ffff7f18c02 in Ns_TclEvalCallback (interp=0x7fffdc005250, cbPtr=0x5555556a49d0, resultDString=0x0) at tclcallbacks.c:186
#18 0x00007ffff7f29764 in NsTclTraceProc (interp=0x7fffdc005250, arg=0x5555556a49d0) at tclinit.c:1913
#19 0x00007ffff7f2a158 in RunTraces (itPtr=0x7fffdc035570, why=NS_TCL_TRACE_DEALLOCATE) at tclinit.c:2375
#20 0x00007ffff7f29976 in PushInterp (itPtr=0x7fffdc035570) at tclinit.c:2026
#21 0x00007ffff7f29717 in NsFreeConnInterp (connPtr=0x555555630950) at tclinit.c:1885
#22 0x00007ffff7efdf11 in ConnRun (connPtr=0x555555630950) at queue.c:2648
#23 0x00007ffff7efd0de in NsConnThread (arg=0x55555564bea0) at queue.c:2211
#24 0x00007ffff7bdd734 in NsThreadMain (arg=0x555558577960) at thread.c:232
#25 0x00007ffff7bdf6f5 in ThreadMain (arg=0x555558577960) at pthread.c:870
#26 0x00007ffff7ccfb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#27 0x00007ffff7d61a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb) bt full
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737299019328) at ./nptl/pthread_kill.c:44
tid = <optimized out>
ret = 0
pd = 0x7ffff4b6f640
old_mask = {__val = {68118949824, 0, 140737349500896, 140737349117410, 140737299013472, 140736886756960, 140736887828416, 15, 140737299013552, 140737348535152,
140736886756976, 140736884396624, 140736886755600, 140736884396624, 140736884381136, 140736886756960}}
ret = <optimized out>
pd = <optimized out>
old_mask = <optimized out>
ret = <optimized out>
tid = <optimized out>
ret = <optimized out>
resultvar = <optimized out>
resultvar = <optimized out>
__arg3 = <optimized out>
__arg2 = <optimized out>
__arg1 = <optimized out>
_a3 = <optimized out>
_a2 = <optimized out>
_a1 = <optimized out>
__futex = <optimized out>
resultvar = <optimized out>
__arg3 = <optimized out>
__arg2 = <optimized out>
__arg1 = <optimized out>
_a3 = <optimized out>
_a2 = <optimized out>
_a1 = <optimized out>
__futex = <optimized out>
__private = <optimized out>
__oldval = <optimized out>
result = <optimized out>
#1 __pthread_kill_internal (signo=6, threadid=140737299019328) at ./nptl/pthread_kill.c:78
No locals.
#2 __GI___pthread_kill (threadid=140737299019328, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
No locals.
#3 0x00007ffff7c7d476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
ret = <optimized out>
#4 0x00007ffff7c637f3 in __GI_abort () at ./stdlib/abort.c:79
save_stage = 1
act = {__sigaction_handler = {sa_handler = 0x7fffdc245a30, sa_sigaction = 0x7fffdc245a30}, sa_mask = {__val = {140736886757216, 140737299014196, 140736884396624,
140736886756976, 140736886757568, 140736886757664, 140736886757840, 140736886757936, 140736886756704, 2203128881648, 140737299013943, 18446744069414584324,
140737349538724, 140733193388042, 140737299014196, 18446744069414584328}}, sa_flags = -600526912, sa_restorer = 0x7ffff4b6e239}
--Type <RET> for more, q to quit, c to continue without paging--c
sigs = {__val = {32, 93825009984944, 0, 0, 3, 140733193388035, 85899345920, 12884901888, 0, 0, 8589934591, 140736886757664, 140736886757664, 140737299014201, 140736886757840, 140736886757936}}
#5 0x00007ffff7cc46f6 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7e16b8c "%s\n") at ../sysdeps/posix/libc_fatal.c:155
ap = {{gp_offset = 24, fp_offset = 3, overflow_arg_area = 0x7ffff4b6e240, reg_save_area = 0x7ffff4b6e1d0}}
fd = <optimized out>
list = <optimized out>
nlist = <optimized out>
cp = <optimized out>
#6 0x00007ffff7cdbd7c in malloc_printerr (str=str@entry=0x7ffff7e19740 "free(): invalid next size (fast)") at ./malloc/malloc.c:5664
No locals.
#7 0x00007ffff7cddb1d in _int_free (av=0x7fffdc000030, p=0x7fffdd865320, have_lock=0) at ./malloc/malloc.c:4522
fail = <optimized out>
idx = <optimized out>
old = <optimized out>
old2 = <optimized out>
size = 32
fb = <optimized out>
nextchunk = <optimized out>
nextsize = <optimized out>
nextinuse = <optimized out>
prevsize = <optimized out>
bck = <optimized out>
fwd = <optimized out>
__PRETTY_FUNCTION__ = "_int_free"
#8 0x00007ffff7ce04d3 in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3391
ar_ptr = <optimized out>
p = <optimized out>
err = 25
#9 0x00007ffff7bdb1e5 in ns_free (ptr=0x7fffdd865330) at memory.c:94
No locals.
#10 0x00007ffff7f09b64 in Ns_SetFree (set=0x7fffdde381a0) at set.c:397
i = 8
__PRETTY_FUNCTION__ = "Ns_SetFree"
#11 0x00007ffff7f3e119 in NsTclSetObjCmd (clientData=0x7fffdc035570, interp=0x7fffdc005250, objc=2, objv=0x7fffdc245650) at tclset.c:330
key = 0x7fffdc3aa070 "d6"
itPtr = 0x7fffdc035570
set = 0x7fffdde381a0
ds = {string = 0x2 <error: Cannot access memory at address 0x2>, length = -601598384, spaceAvl = 32767, staticSpace = "\300\344\266\364\377\177\000\000PV$\334\377\177\000\000\v\350b\336\002\000\000\000\200}\021\334\377\177\000\000H\345\266\364\377\177\000\000PR\000\334\377\177\000\000\000\345\266\364\377\177\000\000\025", '\000' <repeats 15 times>, "PR\000\334\377\177\000\000\220n\004\334\377\177\000\000\340\266\062\334\377\177\000\000@\207\236ZUU\000\000\000%9\210\226wv\035\300\345\266\364\377\177\000\000\230\256\234\367\377\177\000\000`\345\266\364\377\177\000\000p\203\252\367\000\000\000\000PR\000\334\377\177\000\000\021\334\377\177\000\000\000\017\000\334\377\177\000\000\000\000\000\000\020\000 \000\002\000\000\000\000\000\000\000\220n\004\334\377\177\000\000\000\000\000\000\000\000\000"}
tablePtr = 0x7fffdc035740
hPtr = 0x7fffdc3aa050
search = {tablePtr = 0x7fffdc035740, nextIndex = 3, nextEntryPtr = 0x7fffda7c9790}
opt = 1
result = 0
opts = {0x7ffff7f89745 "array", 0x7ffff7f8974b "cleanup", 0x7ffff7f89753 "copy", 0x7ffff7f89758 "cput", 0x7ffff7f8975d "create", 0x7ffff7f89764 "delete", 0x7ffff7f8976b "delkey", 0x7ffff7f89772 "find", 0x7ffff7f89777 "free", 0x7ffff7f8977c "get", 0x7ffff7f89780 "icput", 0x7ffff7f89786 "idelkey", 0x7ffff7f8978e "ifind", 0x7ffff7f89794 "iget", 0x7ffff7f89799 "imerge", 0x7ffff7f897a0 "isnull", 0x7ffff7f897a7 "iunique", 0x7ffff7f897af "iupdate", 0x7ffff7f897b7 "key", 0x7ffff7f897bb "keys", 0x7ffff7f897c0 "list", 0x7ffff7f897c5 "merge", 0x7ffff7f897cb "move", 0x7ffff7f897d0 "name", 0x7ffff7f897d5 "new", 0x7ffff7f897d9 "print", 0x7ffff7f897df "put", 0x7ffff7f897e3 "size", 0x7ffff7f897e8 "split", 0x7ffff7f897ee "truncate", 0x7ffff7f897f7 "unique", 0x7ffff7f897fe "update", 0x7ffff7f89805 "value", 0x7ffff7f8980b "values", 0x0}
SArrayIdx = SArrayIdx
SCleanupIdx = SCleanupIdx
SCopyIdx = SCopyIdx
SCPutIdx = SCPutIdx
SCreateidx = SCreateidx
SDeleteIdx = SDeleteIdx
SDelkeyIdx = SDelkeyIdx
SFindIdx = SFindIdx
SFreeIdx = SFreeIdx
SGetIdx = SGetIdx
SICPutIdx = SICPutIdx
SIDelkeyIdx = SIDelkeyIdx
SIFindIdx = SIFindIdx
SIGetIdx = SIGetIdx
SIMergeIdx = SIMergeIdx
SIsNullIdx = SIsNullIdx
SIUniqueIdx = SIUniqueIdx
SIUpdateIdx = SIUpdateIdx
SKeyIdx = SKeyIdx
SKeysIdx = SKeysIdx
SListIdx = SListIdx
SMergeIdx = SMergeIdx
SMoveIdx = SMoveIdx
sINameIdx = sINameIdx
SNewIdx = SNewIdx
SPrintIdx = SPrintIdx
SPutIdx = SPutIdx
SSizeIdx = SSizeIdx
SSplitIdx = SSplitIdx
STruncateIdx = STruncateIdx
SUniqueIdx = SUniqueIdx
SUpdateIdx = SUpdateIdx
SValueIdx = SValueIdx
SValuesIdx = SValuesIdx
__PRETTY_FUNCTION__ = "NsTclSetObjCmd"
#12 0x00007ffff79cb18e in Dispatch (data=0x7fffdc34afc8, interp=0x7fffdc005250, result=0) at /usr/local/src/tcl8.6.13/generic/tclBasic.c:4467
objProc = 0x7ffff7f3df2d <NsTclSetObjCmd>
clientData = 0x7fffdc035570
objc = 2
objv = 0x7fffdc245650
iPtr = 0x7fffdc005250
#13 0x00007ffff79cb21f in TclNRRunCallbacks (interp=0x7fffdc005250, result=0, rootPtr=0x0) at /usr/local/src/tcl8.6.13/generic/tclBasic.c:4503
callbackPtr = 0x7fffdc34afc0
procPtr = 0x7ffff79cb10e <Dispatch>
iPtr = 0x7fffdc005250
#14 0x00007ffff79ca949 in Tcl_EvalObjv (interp=0x7fffdc005250, objc=1, objv=0x7fffdc2453f0, flags=2097168) at /usr/local/src/tcl8.6.13/generic/tclBasic.c:4226
result = 0
rootPtr = 0x0
#15 0x00007ffff79cd384 in TclEvalEx (interp=0x7fffdc005250, script=0x7ffff4b6e880 "ns_cleanup", numBytes=10, flags=0, line=1, clNextOuter=0x0, outerScript=0x7ffff4b6e880 "ns_cleanup") at /usr/local/src/tcl8.6.13/generic/tclBasic.c:5372
wordLine = 1
wordCLNext = 0x0
objectsNeeded = 1
wordStart = 0x7ffff4b6e880 "ns_cleanup"
numWords = 1
iPtr = 0x7fffdc005250
p = 0x7ffff4b6e880 "ns_cleanup"
next = 0x1f4b6e820 <error: Cannot access memory at address 0x1f4b6e820>
minObjs = 20
objv = 0x7fffdc2453f0
objvSpace = 0x7fffdc2453f0
expand = 0x7fffdc2454a0
lines = 0x7fffdc245500
lineSpace = 0x7fffdc245500
tokenPtr = 0x7fffdc2451d0
commandLength = 32767
bytesLeft = 10
expandRequested = 0
code = 0
savedVarFramePtr = 0x7fffdc001550
allowExceptions = 0
gotParse = 1
i = 4105627376
objectsUsed = 1
parsePtr = 0x7fffdc245140
eeFramePtr = 0x7fffdc245390
stackObjArray = 0x7fffdc2453f0
expandStack = 0x7fffdc2454a0
linesStack = 0x7fffdc245500
clNext = 0x0
#16 0x00007ffff79cc5d9 in Tcl_EvalEx (interp=0x7fffdc005250, script=0x7ffff4b6e880 "ns_cleanup", numBytes=10, flags=0) at /usr/local/src/tcl8.6.13/generic/tclBasic.c:5037
No locals.
#17 0x00007ffff7f18c02 in Ns_TclEvalCallback (interp=0x7fffdc005250, cbPtr=0x5555556a49d0, resultDString=0x0) at tclcallbacks.c:186
arg = 0x0
ii = 0
ap = {{gp_offset = 32, fp_offset = 48, overflow_arg_area = 0x7ffff4b6ea10, reg_save_area = 0x7ffff4b6e950}}
ds = {string = 0x7ffff4b6e880 "ns_cleanup", length = 10, spaceAvl = 200, staticSpace = "ns_cleanup\000\367\377\177\000\000\300\350\266\364\377\177\000\000P\351\266\364\377\177\000\000\210\060jUUU\000\000@\351\266\364\377\177\000\000\000\351\266\364\377\177\000\000\340\tcU\001\001\001\000\340\350\266\364\377\177\000\000\360\350\266\364\377\177\000\000\020\351\266\364\377\177\000\000\270\060jUUU\000\000\020\351\266\364\377\177\000\000\332\356\275\367\377\177\000\000\211\311\334d\000\000\000\000 1jUUU\000\000\000\000\000\000\000\000\000\000\270\060jU\005\000\000\000\220\351\266\364\377\177\000\000\023\275\275\367\377\177\000\000\060\352\266\364\377\177\000\000¢\362\367\377\177\000\000\211\311\334d\000\000\000\000p\fJ\334\b\000\000\000\060JjUUU\000"}
deallocInterp = false
status = 1
__PRETTY_FUNCTION__ = "Ns_TclEvalCallback"
#18 0x00007ffff7f29764 in NsTclTraceProc (interp=0x7fffdc005250, arg=0x5555556a49d0) at tclinit.c:1913
cbPtr = 0x5555556a49d0
result = 0
#19 0x00007ffff7f2a158 in RunTraces (itPtr=0x7fffdc035570, why=NS_TCL_TRACE_DEALLOCATE) at tclinit.c:2375
tracePtr = 0x5555556a4a30
servPtr = 0x55555562b470
__PRETTY_FUNCTION__ = "RunTraces"
#20 0x00007ffff7f29976 in PushInterp (itPtr=0x7fffdc035570) at tclinit.c:2026
interp = 0x7fffdc005250
ok = true
__PRETTY_FUNCTION__ = "PushInterp"
#21 0x00007ffff7f29717 in NsFreeConnInterp (connPtr=0x555555630950) at tclinit.c:1885
itPtr = 0x7fffdc035570
#22 0x00007ffff7efdf11 in ConnRun (connPtr=0x555555630950) at queue.c:2648
sockPtr = 0x7fffd8001d80
conn = 0x555555630950
servPtr = 0x55555562b470
status = NS_OK
auth = 0x0
__PRETTY_FUNCTION__ = "ConnRun"
#23 0x00007ffff7efd0de in NsConnThread (arg=0x55555564bea0) at queue.c:2211
argPtr = 0x55555564bea0
poolPtr = 0x555555630660
servPtr = 0x55555562b470
connPtr = 0x555555630950
wait = {sec = 1692191231, usec = 653861}
timePtr = 0x7ffff4b6ec20
threadId = 0
duringShutdown = 220
fromQueue = false
cpt = 1000
ncons = 999
current = 2
status = NS_OK
timeout = {sec = 120, usec = 0}
exitMsg = 0x7fffdc000b70 ""
joinThread = 0x7ffff4b6f640
threadsLockPtr = 0x5555556306d0
tqueueLockPtr = 0x555555630718
wqueueLockPtr = 0x5555556306a8
__PRETTY_FUNCTION__ = "NsConnThread"
#24 0x00007ffff7bdd734 in NsThreadMain (arg=0x555558577960) at thread.c:232
thrPtr = 0x555558577960
#25 0x00007ffff7bdf6f5 in ThreadMain (arg=0x555558577960) at pthread.c:870
No locals.
#26 0x00007ffff7ccfb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
ret = <optimized out>
pd = <optimized out>
out = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737488346768, 7820541724095505063, 140737299019328, 0, 140737350793296, 140737488347120, -7820557727598026073, -7820559187479928153}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#27 0x00007ffff7d61a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
No locals.
________________________________
From: Gustaf Neumann <ne...@wu...>
Sent: Tuesday 15 August 2023 6:16 pm
To: nav...@li... <nav...@li...>
Subject: Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu
Brian,
many thanks, the backtrace gives some insights:
The problem happens in a Ns_SetFree operation if set "d8"
triggered by an "ns_set cleanup" during the cleanup of the request.
Something is broken with this nsset.
Can it be that your application package issues "ns_set cleanup" as well?
Do you have a recent version of OpenACS? If so, you can turn on tracing
of the "ns_set" command by adding it to the "traced_cmds".
-g
=================================================================================
--- acs-tcl/tcl/tcltrace-init.tcl 27 Nov 2020 09:52:06 -0000 1.4.2.4
+++ acs-tcl/tcl/tcltrace-init.tcl 15 Aug 2023 17:13:51 -0000
@@ -32,6 +32,7 @@
#set traced_cmds {::ns_setcookie ::ns_getcookie ::ns_deletecookie}
#set traced_cmds {::ns_return ::ns_returnnotfound ::ns_returnfile ::ns_returnmoved}
#set traced_cmds [lsort [info commands ::ns_return*]]
+set traced_cmds {::ns_set}
foreach cmd $traced_cmds {
append trace "\ntrace add execution $cmd enter {::tcltrace::before}"
}
=================================================================================
On 15.08.23 15:51, Brian Fenton wrote:
Hi
I reproduced the problem using the install-ns.sh script running under gdb. Here's the output of backtrace and bt full. I'm new to using gdb so please let me know if you'd like to see some other info.
|
|
From: Gustaf N. <ne...@wu...> - 2023-08-15 17:16:35
|
Brian,
many thanks, the backtrace gives some insights:
The problem happens in a Ns_SetFree operation if set "d8"
triggered by an "ns_set cleanup" during the cleanup of the request.
Something is broken with this nsset.
Can it be that your application package issues "ns_set cleanup" as well?
Do you have a recent version of OpenACS? If so, you can turn on tracing
of the "ns_set" command by adding it to the "traced_cmds".
-g
=================================================================================
--- acs-tcl/tcl/tcltrace-init.tcl 27 Nov 2020 09:52:06 -0000 1.4.2.4
+++ acs-tcl/tcl/tcltrace-init.tcl 15 Aug 2023 17:13:51 -0000
@@ -32,6 +32,7 @@
#set traced_cmds {::ns_setcookie ::ns_getcookie ::ns_deletecookie}
#set traced_cmds {::ns_return ::ns_returnnotfound ::ns_returnfile ::ns_returnmoved}
#set traced_cmds [lsort [info commands ::ns_return*]]
+set traced_cmds {::ns_set}
foreach cmd $traced_cmds {
append trace "\ntrace add execution $cmd enter {::tcltrace::before}"
}
=================================================================================
On 15.08.23 15:51, Brian Fenton wrote:
> Hi
>
> I reproduced the problem using the install-ns.sh script running under
> gdb. Here's the output of backtrace and bt full. I'm new to using gdb
> so please let me know if you'd like to see some other info.
|
|
From: Brian F. <bri...@ai...> - 2023-08-15 15:24:14
|
Hi
I reproduced the problem using the install-ns.sh script running under gdb. Here's the output of backtrace and bt full. I'm new to using gdb so please let me know if you'd like to see some other info.
[15/Aug/2023:13:56:52][13147.7fffe35fe640][-driver:nsssl:0-] Notice: ... sockAccept accepted 2 connections
free(): invalid next size (fast)
Thread 4 "nsd" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff4ad8640 (LWP 13651)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140737298400832) at ./nptl/pthread_kill.c:44
44 ./nptl/pthread_kill.c: No such file or directory.
(gdb) backtrace
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737016493632) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=140737016493632) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=140737016493632, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3 0x00007ffff7c7d476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4 0x00007ffff7c637f3 in __GI_abort () at ./stdlib/abort.c:79
#5 0x00007ffff7cc46f6 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7e16b8c "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#6 0x00007ffff7cdbd7c in malloc_printerr (str=str@entry=0x7ffff7e19230 "munmap_chunk(): invalid pointer") at ./malloc/malloc.c:5664
#7 0x00007ffff7cdc05c in munmap_chunk (p=<optimized out>) at ./malloc/malloc.c:3060
#8 0x00007ffff7ce051a in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3381
#9 0x00007ffff7bdb1e5 in ns_free (ptr=0x7fffd4de0ba0) at memory.c:94
#10 0x00007ffff7f09b64 in Ns_SetFree (set=0x7fffd5886210) at set.c:397
#11 0x00007ffff7f3e119 in NsTclSetObjCmd (clientData=0x7fffd403d590, interp=0x7fffd4005250, objc=2, objv=0x7fffd453a510) at tclset.c:330
#12 0x00007ffff79cb18e in Dispatch (data=0x7fffd410e3b8, interp=0x7fffd4005250, result=0) at /usr/local/src/tcl8.6.13/generic/tclBasic.c:4467
#13 0x00007ffff79cb21f in TclNRRunCallbacks (interp=0x7fffd4005250, result=0, rootPtr=0x0) at /usr/local/src/tcl8.6.13/generic/tclBasic.c:4503
#14 0x00007ffff79ca949 in Tcl_EvalObjv (interp=0x7fffd4005250, objc=1, objv=0x7fffd453a2b0, flags=2097168) at /usr/local/src/tcl8.6.13/generic/tclBasic.c:4226
#15 0x00007ffff79cd384 in TclEvalEx (interp=0x7fffd4005250, script=0x7fffe3dfe880 "ns_cleanup", numBytes=10, flags=0, line=1, clNextOuter=0x0,
outerScript=0x7fffe3dfe880 "ns_cleanup") at /usr/local/src/tcl8.6.13/generic/tclBasic.c:5372
#16 0x00007ffff79cc5d9 in Tcl_EvalEx (interp=0x7fffd4005250, script=0x7fffe3dfe880 "ns_cleanup", numBytes=10, flags=0) at /usr/local/src/tcl8.6.13/generic/tclBasic.c:5037
#17 0x00007ffff7f18c02 in Ns_TclEvalCallback (interp=0x7fffd4005250, cbPtr=0x5555556a1b30, resultDString=0x0) at tclcallbacks.c:186
#18 0x00007ffff7f29764 in NsTclTraceProc (interp=0x7fffd4005250, arg=0x5555556a1b30) at tclinit.c:1913
#19 0x00007ffff7f2a158 in RunTraces (itPtr=0x7fffd403d590, why=NS_TCL_TRACE_DEALLOCATE) at tclinit.c:2375
#20 0x00007ffff7f29976 in PushInterp (itPtr=0x7fffd403d590) at tclinit.c:2026
#21 0x00007ffff7f29717 in NsFreeConnInterp (connPtr=0x55555562ebd0) at tclinit.c:1885
#22 0x00007ffff7efdf11 in ConnRun (connPtr=0x55555562ebd0) at queue.c:2648
#23 0x00007ffff7efd0de in NsConnThread (arg=0x555555649030) at queue.c:2211
#24 0x00007ffff7bdd734 in NsThreadMain (arg=0x55555855cdc0) at thread.c:232
#25 0x00007ffff7bdf6f5 in ThreadMain (arg=0x55555855cdc0) at pthread.c:870
#26 0x00007ffff7ccfb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#27 0x00007ffff7d61a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
gdb) bt full
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737016493632) at ./nptl/pthread_kill.c:44
tid = <optimized out>
ret = 0
pd = 0x7fffe3dff640
old_mask = {__val = {140737016487840, 140736755639472, 3823099840, 512, 140737016487920, 140737348535152, 140736755639488, 140736750178896, 140736755638224,
140733193388032, 140736750163408, 140736755639472, 140736755639432, 140736752725904, 93825010219632, 93825010219632}}
ret = <optimized out>
pd = <optimized out>
old_mask = <optimized out>
ret = <optimized out>
tid = <optimized out>
ret = <optimized out>
resultvar = <optimized out>
resultvar = <optimized out>
__arg3 = <optimized out>
__arg2 = <optimized out>
__arg1 = <optimized out>
_a3 = <optimized out>
_a2 = <optimized out>
_a1 = <optimized out>
__futex = <optimized out>
resultvar = <optimized out>
__arg3 = <optimized out>
__arg2 = <optimized out>
__arg1 = <optimized out>
_a3 = <optimized out>
_a2 = <optimized out>
_a1 = <optimized out>
__futex = <optimized out>
__private = <optimized out>
__oldval = <optimized out>
result = <optimized out>
#1 __pthread_kill_internal (signo=6, threadid=140737016493632) at ./nptl/pthread_kill.c:78
No locals.
#2 __GI___pthread_kill (threadid=140737016493632, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
No locals.
#3 0x00007ffff7c7d476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
ret = <optimized out>
#4 0x00007ffff7c637f3 in __GI_abort () at ./stdlib/abort.c:79
save_stage = 1
act = {__sigaction_handler = {sa_handler = 0x600000004, sa_sigaction = 0x600000004}, sa_mask = {__val = {140736789161264, 140733193388042, 140737347688968,
140737016488736, 279037356156, 18446744073709551615, 140736755314240, 140737016488272, 140733193388033, 140736792190256, 140736790551568, 0, 140736755639936,
93825035611088, 140736753589312, 140736756049120}}, sa_flags = 1487610384, sa_restorer = 0x1}
--Type <RET> for more, q to quit, c to continue without paging--
sigs = {__val = {32, 140737350793296, 140737488347040, 140737350862035, 93824993127520, 140736755639576, 8589934656, 93825010219632, 25769803776, 193273528320,
140737016488160, 140737349119905, 3823100240, 4294967296, 2202846355952, 3556773632}}
#5 0x00007ffff7cc46f6 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7e16b8c "%s\n") at ../sysdeps/posix/libc_fatal.c:155
ap = {{gp_offset = 24, fp_offset = 0, overflow_arg_area = 0x7fffe3dfe2a0, reg_save_area = 0x7fffe3dfe230}}
fd = <optimized out>
list = <optimized out>
nlist = <optimized out>
cp = <optimized out>
#6 0x00007ffff7cdbd7c in malloc_printerr (str=str@entry=0x7ffff7e19230 "munmap_chunk(): invalid pointer") at ./malloc/malloc.c:5664
No locals.
#7 0x00007ffff7cdc05c in munmap_chunk (p=<optimized out>) at ./malloc/malloc.c:3060
pagesize = <optimized out>
size = <optimized out>
__PRETTY_FUNCTION__ = "munmap_chunk"
mem = <optimized out>
block = <optimized out>
total_size = <optimized out>
#8 0x00007ffff7ce051a in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3381
ar_ptr = <optimized out>
p = <optimized out>
err = 25
#9 0x00007ffff7bdb1e5 in ns_free (ptr=0x7fffd4de0ba0) at memory.c:94
No locals.
#10 0x00007ffff7f09b64 in Ns_SetFree (set=0x7fffd5886210) at set.c:397
i = 10
__PRETTY_FUNCTION__ = "Ns_SetFree"
#11 0x00007ffff7f3e119 in NsTclSetObjCmd (clientData=0x7fffd403d590, interp=0x7fffd4005250, objc=2, objv=0x7fffd453a510) at tclset.c:330
key = 0x7fffd464eb50 "d8"
itPtr = 0x7fffd403d590
set = 0x7fffd5886210
ds = {string = 0x7fffd6650c80 "%", length = -738176432, spaceAvl = 32767,
staticSpace = "\320\344\337\343\377\177\000\000\240\236t\336\377\177\000\000\260\356\004\324\377\177\000\000PZ\000\324\377\177\000\000\000\345\337\343\377\177\000\000\312Ĝ\367\377\177\000\000\360\357Y\324\377\177\000\000\000\000\000\000\000\000\000\000\200\fe\326\377\177\000\000PR\000\324\377\177\000\000\000\000\000\000\001\000\000\000PR\000\324\377\177\000\000PZ\000\324\377\177\000\000\260\356\004\324\377\177\000\000\300\345\337\343\377\177\000\000Э\234\367\377\177\000\000`\345\337\343\377\177\000\000p\203\252\367\000\000\000\000PR\000\324\377\177\000\000H\003Z\324\377\177\000\000\000\017\000\324\377\177\000\000\000\000\000\000\020\000 \000\002\000\000\000\377\177\000\000\260\356\004\324\377\177\000\000\000\000\000\000\000\000\000"}
tablePtr = 0x7fffd403d760
hPtr = 0x7fffd464eb30
search = {tablePtr = 0x7fffd403d760, nextIndex = 13, nextEntryPtr = 0x0}
opt = 1
result = 0
opts = {0x7ffff7f89745 "array", 0x7ffff7f8974b "cleanup", 0x7ffff7f89753 "copy", 0x7ffff7f89758 "cput", 0x7ffff7f8975d "create", 0x7ffff7f89764 "delete", 0x7ffff7f8976b "delkey", 0x7ffff7f89772 "find", 0x7ffff7f89777 "free", 0x7ffff7f8977c "get", 0x7ffff7f89780 "icput", 0x7ffff7f89786 "idelkey", 0x7ffff7f8978e "ifind", 0x7ffff7f89794 "iget", 0x7ffff7f89799 "imerge", 0x7ffff7f897a0 "isnull", 0x7ffff7f897a7 "iunique", 0x7ffff7f897af "iupdate", 0x7ffff7f897b7 "key", 0x7ffff7f897bb "keys", 0x7ffff7f897c0 "list", 0x7ffff7f897c5 "merge", 0x7ffff7f897cb "move", 0x7ffff7f897d0 "name", 0x7ffff7f897d5 "new", 0x7ffff7f897d9 "print", 0x7ffff7f897df "put", 0x7ffff7f897e3 "size", 0x7ffff7f897e8 "split", 0x7ffff7f897ee "truncate", 0x7ffff7f897f7 "unique", 0x7ffff7f897fe "update", 0x7ffff7f89805 "value", 0x7ffff7f8980b "values", 0x0}
SArrayIdx = SArrayIdx
SCleanupIdx = SCleanupIdx
SCopyIdx = SCopyIdx
SCPutIdx = SCPutIdx
SCreateidx = SCreateidx
SDeleteIdx = SDeleteIdx
SDelkeyIdx = SDelkeyIdx
SFindIdx = SFindIdx
SFreeIdx = SFreeIdx
SGetIdx = SGetIdx
SICPutIdx = SICPutIdx
SIDelkeyIdx = SIDelkeyIdx
SIFindIdx = SIFindIdx
SIGetIdx = SIGetIdx
SIMergeIdx = SIMergeIdx
SIsNullIdx = SIsNullIdx
SIUniqueIdx = SIUniqueIdx
SIUpdateIdx = SIUpdateIdx
SKeyIdx = SKeyIdx
SKeysIdx = SKeysIdx
SListIdx = SListIdx
SMergeIdx = SMergeIdx
SMoveIdx = SMoveIdx
sINameIdx = sINameIdx
SNewIdx = SNewIdx
SPrintIdx = SPrintIdx
SPutIdx = SPutIdx
SSizeIdx = SSizeIdx
SSplitIdx = SSplitIdx
STruncateIdx = STruncateIdx
SUniqueIdx = SUniqueIdx
SUpdateIdx = SUpdateIdx
SValueIdx = SValueIdx
SValuesIdx = SValuesIdx
__PRETTY_FUNCTION__ = "NsTclSetObjCmd"
#12 0x00007ffff79cb18e in Dispatch (data=0x7fffd410e3b8, interp=0x7fffd4005250, result=0) at /usr/local/src/tcl8.6.13/generic/tclBasic.c:4467
objProc = 0x7ffff7f3df2d <NsTclSetObjCmd>
clientData = 0x7fffd403d590
objc = 2
objv = 0x7fffd453a510
iPtr = 0x7fffd4005250
#13 0x00007ffff79cb21f in TclNRRunCallbacks (interp=0x7fffd4005250, result=0, rootPtr=0x0) at /usr/local/src/tcl8.6.13/generic/tclBasic.c:4503
callbackPtr = 0x7fffd410e3b0
procPtr = 0x7ffff79cb10e <Dispatch>
iPtr = 0x7fffd4005250
#14 0x00007ffff79ca949 in Tcl_EvalObjv (interp=0x7fffd4005250, objc=1, objv=0x7fffd453a2b0, flags=2097168) at /usr/local/src/tcl8.6.13/generic/tclBasic.c:4226
result = 0
rootPtr = 0x0
#15 0x00007ffff79cd384 in TclEvalEx (interp=0x7fffd4005250, script=0x7fffe3dfe880 "ns_cleanup", numBytes=10, flags=0, line=1, clNextOuter=0x0, outerScript=0x7fffe3dfe880 "ns_cleanup") at /usr/local/src/tcl8.6.13/generic/tclBasic.c:5372
wordLine = 1
wordCLNext = 0x0
objectsNeeded = 1
wordStart = 0x7fffe3dfe880 "ns_cleanup"
numWords = 1
iPtr = 0x7fffd4005250
p = 0x7fffe3dfe880 "ns_cleanup"
next = 0x1e3dfe820 <error: Cannot access memory at address 0x1e3dfe820>
minObjs = 20
objv = 0x7fffd453a2b0
objvSpace = 0x7fffd453a2b0
expand = 0x7fffd453a360
lines = 0x7fffd453a3c0
lineSpace = 0x7fffd453a3c0
tokenPtr = 0x7fffd453a090
commandLength = 32767
bytesLeft = 10
expandRequested = 0
code = 0
savedVarFramePtr = 0x7fffd4001550
allowExceptions = 0
gotParse = 1
i = 3823101680
objectsUsed = 1
parsePtr = 0x7fffd453a000
eeFramePtr = 0x7fffd453a250
stackObjArray = 0x7fffd453a2b0
expandStack = 0x7fffd453a360
linesStack = 0x7fffd453a3c0
clNext = 0x0
#16 0x00007ffff79cc5d9 in Tcl_EvalEx (interp=0x7fffd4005250, script=0x7fffe3dfe880 "ns_cleanup", numBytes=10, flags=0) at /usr/local/src/tcl8.6.13/generic/tclBasic.c:5037
No locals.
#17 0x00007ffff7f18c02 in Ns_TclEvalCallback (interp=0x7fffd4005250, cbPtr=0x5555556a1b30, resultDString=0x0) at tclcallbacks.c:186
arg = 0x0
ii = 0
ap = {{gp_offset = 32, fp_offset = 48, overflow_arg_area = 0x7fffe3dfea10, reg_save_area = 0x7fffe3dfe950}}
ds = {string = 0x7fffe3dfe880 "ns_cleanup", length = 10, spaceAvl = 200, staticSpace = "ns_cleanup\000\367\377\177\000\000\300\350\337\343\377\177\000\000P\351\337\343\377\177\000\000\210\277jUUU\000\000@\351\337\343\377\177\000\000\000\351\337\343\377\177\000\000`\354bU\001\001\001\000\340\350\337\343\377\177\000\000\360\350\337\343\377\177\000\000\020\351\337\343\377\177\000\000\270\277jUUU\000\000\020\351\337\343\377\177\000\000\332\356\275\367\377\177\000\000\223z\333d\000\000\000\000 \300jUUU\000\000\000\000\000\000\000\000\000\000\270\277jU\005\000\000\000\220\351\337\343\377\177\000\000\023\275\275\367\377\177\000\000\060\352\337\343\377\177\000\000¢\362\367\377\177\000\000\223z\333d\000\000\000\000P\340\025\324\b\000\000\000\220\033jUUU\000"}
deallocInterp = false
status = 1
__PRETTY_FUNCTION__ = "Ns_TclEvalCallback"
#18 0x00007ffff7f29764 in NsTclTraceProc (interp=0x7fffd4005250, arg=0x5555556a1b30) at tclinit.c:1913
cbPtr = 0x5555556a1b30
result = 0
#19 0x00007ffff7f2a158 in RunTraces (itPtr=0x7fffd403d590, why=NS_TCL_TRACE_DEALLOCATE) at tclinit.c:2375
tracePtr = 0x5555556a1b90
servPtr = 0x555555628560
__PRETTY_FUNCTION__ = "RunTraces"
#20 0x00007ffff7f29976 in PushInterp (itPtr=0x7fffd403d590) at tclinit.c:2026
interp = 0x7fffd4005250
ok = true
__PRETTY_FUNCTION__ = "PushInterp"
#21 0x00007ffff7f29717 in NsFreeConnInterp (connPtr=0x55555562ebd0) at tclinit.c:1885
itPtr = 0x7fffd403d590
#22 0x00007ffff7efdf11 in ConnRun (connPtr=0x55555562ebd0) at queue.c:2648
sockPtr = 0x7fffd98f68a0
conn = 0x55555562ebd0
servPtr = 0x555555628560
status = NS_OK
auth = 0x0
__PRETTY_FUNCTION__ = "ConnRun"
#23 0x00007ffff7efd0de in NsConnThread (arg=0x555555649030) at queue.c:2211
argPtr = 0x555555649030
poolPtr = 0x55555562d7c0
servPtr = 0x555555628560
connPtr = 0x55555562ebd0
wait = {sec = 1692105481, usec = 312006}
timePtr = 0x7fffe3dfec20
threadId = 1
duringShutdown = 219
fromQueue = true
cpt = 1000
ncons = 996
current = 2
status = NS_OK
timeout = {sec = 120, usec = 0}
exitMsg = 0x7fffd4000b70 ""
joinThread = 0x7fffe3dff640
threadsLockPtr = 0x55555562d830
tqueueLockPtr = 0x55555562d878
wqueueLockPtr = 0x55555562d808
__PRETTY_FUNCTION__ = "NsConnThread"
#24 0x00007ffff7bdd734 in NsThreadMain (arg=0x55555855cdc0) at thread.c:232
thrPtr = 0x55555855cdc0
#25 0x00007ffff7bdf6f5 in ThreadMain (arg=0x55555855cdc0) at pthread.c:870
No locals.
#26 0x00007ffff7ccfb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
ret = <optimized out>
pd = <optimized out>
out = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737488346688, -3886469656811452993, 140737016493632, 0, 140737350793296, 140737488347040, 3886531503754790335, 3886487635365545407}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#27 0x00007ffff7d61a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
No locals.
thanks
Brian
________________________________
From: Brian Fenton <bri...@ai...>
Sent: Monday 14 August 2023 5:40 pm
To: nav...@li... <nav...@li...>
Subject: Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu
Hi Gustaf
thanks again for the advice. Today I made some more progress on this. There does appear to be some differences between your script and the Oupfiz5 installer e.g. his ns-build.sh script https://github.com/oupfiz5/tcl-build/blob/master/src/builds/ns-build.sh I have reached the conclusion that I will be wasting your time if I can't reproduce this problem using your scripts, so my next task will be to run your script and try to reproduce. I am now seeing the downsides to using a non-official Docker approach!
Today I took the approach of installing (through the APM) our OpenACS packages one by one. For example, we use packages such as Categories, General Comments etc as well as many of our own custom packages. After each package I bounced Naviserver and tested the site. The system worked perfectly until after I installed the last package, which is our main core of our product, very large and old with a lot of features. This makes me very confident that Oracle and nsoracle are working fine. The problem could be some API call in our custom package that maybe changed in 4.99.25.
To answer some of your questions:
* did you run at this state any Oracle queries? Yes, I did. I'm 95% confident that Oracle and nsoracle are working fine.
* did you recompile in the "clean install" also the oracle driver? Yes, I'm building nsoracle from scratch (I am also running the same version of nsoracle in the 4.99.24 build that is working without issue)
* you mean the crash happens in the plain openacs-config.tcl, with no additional drivers etc, no oracle involved? No, this does use Oracle, sorry for not being clear. We have our own heavily modified config file, so I wanted to rule that out by using the openacs-config.tcl that you provide. I just changed the database to Oracle and left everything else as is. The fact that it crashed too means that I can eliminate some strange configuration setting in our custom config file as a possible cause.
* My request in the last mail was to try to reproduce the problem with nsd-config.tcl (i.e. no OpenACS involved). Yes, I replied previously that it runs fine. And also a simple OpenACS install on Oracle runs fine. The problems only start with our custom OpenACS package.
* To be on the safe side, all /usr/local/ns/bin/*.so files should be newly compiled. Yes, these all appear to be freshly compiled.
# ls -l /usr/local/ns/bin/*.so
-rwxr-xr-x 1 nsadmin nsadmin 32560 Aug 10 15:31 /usr/local/ns/bin/nscgi.so
-rwxr-xr-x 1 nsadmin nsadmin 27360 Aug 10 15:31 /usr/local/ns/bin/nscp.so
-rwxr-xr-x 1 nsadmin nsadmin 15808 Aug 10 15:31 /usr/local/ns/bin/nsdb.so
-rwxr-xr-x 1 nsadmin nsadmin 50808 Aug 10 15:31 /usr/local/ns/bin/nsdbpg.so
-rwxr-xr-x 1 nsadmin nsadmin 16176 Aug 10 15:31 /usr/local/ns/bin/nsdbtest.so
-rwxr-xr-x 1 nsadmin nsadmin 32640 Aug 10 15:31 /usr/local/ns/bin/nslog.so
-rwxr-xr-x 1 nsadmin nsadmin 90688 Aug 10 15:42 /usr/local/ns/bin/nsoracle.so
-rwxr-xr-x 1 nsadmin nsadmin 90848 Aug 10 15:42 /usr/local/ns/bin/nsoraclecass.so
-rwxr-xr-x 1 nsadmin nsadmin 31712 Aug 10 15:31 /usr/local/ns/bin/nsperm.so
-rwxr-xr-x 1 nsadmin nsadmin 15888 Aug 10 15:31 /usr/local/ns/bin/nsproxy.so
-rwxr-xr-x 1 nsadmin nsadmin 16536 Aug 10 15:31 /usr/local/ns/bin/nssock.so
-rwxr-xr-x 1 nsadmin nsadmin 26624 Aug 10 15:31 /usr/local/ns/bin/nsssl.so
So my next steps are to try to reproduce the problem using your install-ns.sh script. Then I can compile with debugging and have some fun with gdb.
thanks
Brian
________________________________
From: Gustaf Neumann <ne...@wu...>
Sent: Saturday 12 August 2023 11:55 am
To: nav...@li... <nav...@li...>
Subject: Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu
On 11.08.23 20:15, Brian Fenton wrote:
Hi Gustaf
thanks for the response. I've been looking at this in more detail this afternoon and it does appear to be caused by something in the interaction of our OpenACS application with 4.99.27. As I previously mentioned, it has been running fine on 4.99.24 on the same Ubuntu version. I realise that I may not have been clear on this point on my previous email: this is Naviserver running on Ubuntu in a Docker container. The version of Naviserver is based on this Docker build https://github.com/oupfiz5/naviserver-s6 which I have forked and updated to 4.99.27 (I may well have missed something in updating NS version - maybe I should have waited until oupfiz updates his build).
* I can confirm that nsd-config.tcl runs fine with 4.99.27
* Some good news: I am able to do an OpenACS clean install on Oracle with 4.99.27. I then successfully installed our application using the APM.
did you run at this state any Oracle queries?
did you recompile in the "clean install" also the oracle driver?
* However, once I restart Naviserver the problems start.
* I tried using the openacs-config.tcl that ships with 4.99.27 and the problems are happening with that too.
you mean the crash happens in the plain openacs-config.tcl, with no additional drivers etc, no oracle involved?
this can get us closer to something i might be able to reproduce. My request in the last mail was to try to reproduce the problem with nsd-config.tcl (i.e. no OpenACS involved). If you can reproduce the crash, you should compile with debugging turned on and run nsd under gdb or lldb. First one should get he most simple case causing the crash.
What is odd is that it seems to be able to handle one request before crashing. Eg. I type in the URL, it shows the /register page but then crashes. After restarting, I enter my login details on the register page, press return. It then crashes. After restarting, it successfully logs me, then crashes again.
the memory errors or normally hinting on some buffer overflow, or a mixture between 32bit and 64bit compilation, etc.
There is no clear pattern in the logs. I thought it might be related to OCSP and disabled that, but the problems continued to occur.
if you suspect nsssl, then one potential problem might be a mixture during of different OpenSSL versions during compilation (when using install_ns.sh, this will not happen).
Turning on debug hasn't helped - but maybe there is so much information in the log that I have missed something important.
What drivers are you referring to in your question?
actually all naviserver modules you are using, including the db drivers (since you mentioned nsoracle, which is not part of the regular regression tests). To be on the safe side, all /usr/local/ns/bin/*.so files should be newly compiled.
all the best
-gn
thanks
Brian
________________________________
From: Gustaf Neumann <ne...@wu...><mailto:ne...@wu...>
Sent: Thursday 10 August 2023 7:27 pm
To: nav...@li...<mailto:nav...@li...> <nav...@li...><mailto:nav...@li...>
Subject: Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu
Hi Brian,
The new NaviServer versions are running fine on Ubuntu 22.04. Have you recompiled the drivers you are using with the updated version?
A good test for the NaviServer binary is to test it with one of the packaged configuration files, e.g. nsd-config.tcl.
all the best
-gn
On 10.08.23 18:23, Brian Fenton wrote:
Hello
we have been testing out our OpenACS application on Ubuntu 22.04.2 LTS (previously we only ran on Windows). It was working great with Naviserver 4.99.24 but I have been getting constant crashes on more recent versions.
I get this error on 4.99.25, 4.99.26 and today I also got it on 4.99.27. The server runs fine until I click on a page, then it immediately crashes.
The log has only the following error:
free(): invalid size
and today I got this one:
[10/Aug/2023:15:02:23][303.7fa3a64ee640][-conn:openacs:default:1:119-] Fatal: received fatal signal 11
We have an Oracle application and are using the latest nsoracle driver, which might be a factor here.
We have been running it with a pretty old OpenACS config file, so I am currently looking to merge in all the latest changes to ensure that is not an issue.
Also note that I am running Naviserver on Docker on Windows, but as mentioned it was running great on 4.99.24.
thanks for any help
Brian
_______________________________________________
naviserver-devel mailing list
nav...@li...<mailto:nav...@li...>
https://lists.sourceforge.net/lists/listinfo/naviserver-devel
--
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"
|
|
From: Brian F. <bri...@ai...> - 2023-08-14 17:14:21
|
Hi Gustaf thanks again for the advice. Today I made some more progress on this. There does appear to be some differences between your script and the Oupfiz5 installer e.g. his ns-build.sh script https://github.com/oupfiz5/tcl-build/blob/master/src/builds/ns-build.sh I have reached the conclusion that I will be wasting your time if I can't reproduce this problem using your scripts, so my next task will be to run your script and try to reproduce. I am now seeing the downsides to using a non-official Docker approach! Today I took the approach of installing (through the APM) our OpenACS packages one by one. For example, we use packages such as Categories, General Comments etc as well as many of our own custom packages. After each package I bounced Naviserver and tested the site. The system worked perfectly until after I installed the last package, which is our main core of our product, very large and old with a lot of features. This makes me very confident that Oracle and nsoracle are working fine. The problem could be some API call in our custom package that maybe changed in 4.99.25. To answer some of your questions: * did you run at this state any Oracle queries? Yes, I did. I'm 95% confident that Oracle and nsoracle are working fine. * did you recompile in the "clean install" also the oracle driver? Yes, I'm building nsoracle from scratch (I am also running the same version of nsoracle in the 4.99.24 build that is working without issue) * you mean the crash happens in the plain openacs-config.tcl, with no additional drivers etc, no oracle involved? No, this does use Oracle, sorry for not being clear. We have our own heavily modified config file, so I wanted to rule that out by using the openacs-config.tcl that you provide. I just changed the database to Oracle and left everything else as is. The fact that it crashed too means that I can eliminate some strange configuration setting in our custom config file as a possible cause. * My request in the last mail was to try to reproduce the problem with nsd-config.tcl (i.e. no OpenACS involved). Yes, I replied previously that it runs fine. And also a simple OpenACS install on Oracle runs fine. The problems only start with our custom OpenACS package. * To be on the safe side, all /usr/local/ns/bin/*.so files should be newly compiled. Yes, these all appear to be freshly compiled. # ls -l /usr/local/ns/bin/*.so -rwxr-xr-x 1 nsadmin nsadmin 32560 Aug 10 15:31 /usr/local/ns/bin/nscgi.so -rwxr-xr-x 1 nsadmin nsadmin 27360 Aug 10 15:31 /usr/local/ns/bin/nscp.so -rwxr-xr-x 1 nsadmin nsadmin 15808 Aug 10 15:31 /usr/local/ns/bin/nsdb.so -rwxr-xr-x 1 nsadmin nsadmin 50808 Aug 10 15:31 /usr/local/ns/bin/nsdbpg.so -rwxr-xr-x 1 nsadmin nsadmin 16176 Aug 10 15:31 /usr/local/ns/bin/nsdbtest.so -rwxr-xr-x 1 nsadmin nsadmin 32640 Aug 10 15:31 /usr/local/ns/bin/nslog.so -rwxr-xr-x 1 nsadmin nsadmin 90688 Aug 10 15:42 /usr/local/ns/bin/nsoracle.so -rwxr-xr-x 1 nsadmin nsadmin 90848 Aug 10 15:42 /usr/local/ns/bin/nsoraclecass.so -rwxr-xr-x 1 nsadmin nsadmin 31712 Aug 10 15:31 /usr/local/ns/bin/nsperm.so -rwxr-xr-x 1 nsadmin nsadmin 15888 Aug 10 15:31 /usr/local/ns/bin/nsproxy.so -rwxr-xr-x 1 nsadmin nsadmin 16536 Aug 10 15:31 /usr/local/ns/bin/nssock.so -rwxr-xr-x 1 nsadmin nsadmin 26624 Aug 10 15:31 /usr/local/ns/bin/nsssl.so So my next steps are to try to reproduce the problem using your install-ns.sh script. Then I can compile with debugging and have some fun with gdb. thanks Brian ________________________________ From: Gustaf Neumann <ne...@wu...> Sent: Saturday 12 August 2023 11:55 am To: nav...@li... <nav...@li...> Subject: Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu On 11.08.23 20:15, Brian Fenton wrote: Hi Gustaf thanks for the response. I've been looking at this in more detail this afternoon and it does appear to be caused by something in the interaction of our OpenACS application with 4.99.27. As I previously mentioned, it has been running fine on 4.99.24 on the same Ubuntu version. I realise that I may not have been clear on this point on my previous email: this is Naviserver running on Ubuntu in a Docker container. The version of Naviserver is based on this Docker build https://github.com/oupfiz5/naviserver-s6 which I have forked and updated to 4.99.27 (I may well have missed something in updating NS version - maybe I should have waited until oupfiz updates his build). * I can confirm that nsd-config.tcl runs fine with 4.99.27 * Some good news: I am able to do an OpenACS clean install on Oracle with 4.99.27. I then successfully installed our application using the APM. did you run at this state any Oracle queries? did you recompile in the "clean install" also the oracle driver? * However, once I restart Naviserver the problems start. * I tried using the openacs-config.tcl that ships with 4.99.27 and the problems are happening with that too. you mean the crash happens in the plain openacs-config.tcl, with no additional drivers etc, no oracle involved? this can get us closer to something i might be able to reproduce. My request in the last mail was to try to reproduce the problem with nsd-config.tcl (i.e. no OpenACS involved). If you can reproduce the crash, you should compile with debugging turned on and run nsd under gdb or lldb. First one should get he most simple case causing the crash. What is odd is that it seems to be able to handle one request before crashing. Eg. I type in the URL, it shows the /register page but then crashes. After restarting, I enter my login details on the register page, press return. It then crashes. After restarting, it successfully logs me, then crashes again. the memory errors or normally hinting on some buffer overflow, or a mixture between 32bit and 64bit compilation, etc. There is no clear pattern in the logs. I thought it might be related to OCSP and disabled that, but the problems continued to occur. if you suspect nsssl, then one potential problem might be a mixture during of different OpenSSL versions during compilation (when using install_ns.sh, this will not happen). Turning on debug hasn't helped - but maybe there is so much information in the log that I have missed something important. What drivers are you referring to in your question? actually all naviserver modules you are using, including the db drivers (since you mentioned nsoracle, which is not part of the regular regression tests). To be on the safe side, all /usr/local/ns/bin/*.so files should be newly compiled. all the best -gn thanks Brian ________________________________ From: Gustaf Neumann <ne...@wu...><mailto:ne...@wu...> Sent: Thursday 10 August 2023 7:27 pm To: nav...@li...<mailto:nav...@li...> <nav...@li...><mailto:nav...@li...> Subject: Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu Hi Brian, The new NaviServer versions are running fine on Ubuntu 22.04. Have you recompiled the drivers you are using with the updated version? A good test for the NaviServer binary is to test it with one of the packaged configuration files, e.g. nsd-config.tcl. all the best -gn On 10.08.23 18:23, Brian Fenton wrote: Hello we have been testing out our OpenACS application on Ubuntu 22.04.2 LTS (previously we only ran on Windows). It was working great with Naviserver 4.99.24 but I have been getting constant crashes on more recent versions. I get this error on 4.99.25, 4.99.26 and today I also got it on 4.99.27. The server runs fine until I click on a page, then it immediately crashes. The log has only the following error: free(): invalid size and today I got this one: [10/Aug/2023:15:02:23][303.7fa3a64ee640][-conn:openacs:default:1:119-] Fatal: received fatal signal 11 We have an Oracle application and are using the latest nsoracle driver, which might be a factor here. We have been running it with a pretty old OpenACS config file, so I am currently looking to merge in all the latest changes to ensure that is not an issue. Also note that I am running Naviserver on Docker on Windows, but as mentioned it was running great on 4.99.24. thanks for any help Brian _______________________________________________ naviserver-devel mailing list nav...@li...<mailto:nav...@li...> https://lists.sourceforge.net/lists/listinfo/naviserver-devel -- Univ.Prof. Dr. Gustaf Neumann Head of the Institute of Information Systems and New Media of Vienna University of Economics and Business Program Director of MSc "Information Systems" |
|
From: Gustaf N. <ne...@wu...> - 2023-08-12 10:55:25
|
On 11.08.23 20:15, Brian Fenton wrote: > Hi Gustaf > > thanks for the response. I've been looking at this in more detail this > afternoon and it does appear to be caused by something in the > interaction of our OpenACS application with 4.99.27. As I previously > mentioned, it has been running fine on 4.99.24 on the same Ubuntu > version. I realise that I may not have been clear on this point on my > previous email: this is Naviserver running on Ubuntu in a Docker > container. The version of Naviserver is based on this Docker build > https://github.com/oupfiz5/naviserver-s6 > <https://github.com/oupfiz5/naviserver-s6> which I have forked and > updated to 4.99.27 (I may well have missed something in updating NS > version - maybe I should have waited until oupfiz updates his build). > > * I can confirm that nsd-config.tcl runs fine with 4.99.27 > * Some good news: I am able to do an OpenACS clean install on Oracle > with 4.99.27. I then successfully installed our application using > the APM. > did you run at this state any Oracle queries? did you recompile in the "clean install" also the oracle driver? > > * However, once I restart Naviserver the problems start. > * I tried using the openacs-config.tcl that ships with 4.99.27 and > the problems are happening with that too. > you mean the crash happens in the plain openacs-config.tcl, with no additional drivers etc, no oracle involved? this can get us closer to something i might be able to reproduce. My request in the last mail was to try to reproduce the problem with nsd-config.tcl (i.e. no OpenACS involved). If you can reproduce the crash, you should compile with debugging turned on and run nsd under gdb or lldb. First one should get he most simple case causing the crash. > What is odd is that it seems to be able to handle one request before > crashing. Eg. I type in the URL, it shows the /register page but then > crashes. After restarting, I enter my login details on the register > page, press return. It then crashes. After restarting, it successfully > logs me, then crashes again. the memory errors or normally hinting on some buffer overflow, or a mixture between 32bit and 64bit compilation, etc. > > There is no clear pattern in the logs. I thought it might be related > to OCSP and disabled that, but the problems continued to occur. if you suspect nsssl, then one potential problem might be a mixture during of different OpenSSL versions during compilation (when using install_ns.sh, this will not happen). > Turning on debug hasn't helped - but maybe there is so much > information in the log that I have missed something important. > > What drivers are you referring to in your question? actually all naviserver modules you are using, including the db drivers (since you mentioned nsoracle, which is not part of the regular regression tests). To be on the safe side, all /usr/local/ns/bin/*.so files should be newly compiled. all the best -gn > > thanks > Brian > > ------------------------------------------------------------------------ > *From:* Gustaf Neumann <ne...@wu...> > *Sent:* Thursday 10 August 2023 7:27 pm > *To:* nav...@li... > <nav...@li...> > *Subject:* Re: [naviserver-devel] Crashing on all versions >4.99.24 on > Ubuntu > > Hi Brian, > > > The new NaviServer versions are running fine on Ubuntu 22.04. Have you > recompiled the drivers you are using with the updated version? > > > A good test for the NaviServer binary is to test it with one of the > packaged configuration files, e.g. nsd-config.tcl. > > > all the best > > -gn > > > On 10.08.23 18:23, Brian Fenton wrote: >> Hello >> >> we have been testing out our OpenACS application on Ubuntu 22.04.2 >> LTS (previously we only ran on Windows). It was working great with >> Naviserver 4.99.24 but I have been getting constant crashes on more >> recent versions. >> >> I get this error on 4.99.25, 4.99.26 and today I also got it on >> 4.99.27. The server runs fine until I click on a page, then it >> immediately crashes. >> The log has only the following error: >> free(): invalid size >> >> and today I got this one: >> [10/Aug/2023:15:02:23][303.7fa3a64ee640][-conn:openacs:default:1:119-] >> Fatal: received fatal signal 11 >> >> We have an Oracle application and are using the latest nsoracle >> driver, which might be a factor here. >> We have been running it with a pretty old OpenACS config file, so I >> am currently looking to merge in all the latest changes to ensure >> that is not an issue. >> Also note that I am running Naviserver on Docker on Windows, but as >> mentioned it was running great on 4.99.24. >> >> thanks for any help >> Brian >> > > > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel -- Univ.Prof. Dr. Gustaf Neumann Head of the Institute of Information Systems and New Media of Vienna University of Economics and Business Program Director of MSc "Information Systems" |
|
From: Brian F. <bri...@ai...> - 2023-08-11 19:29:30
|
Hi Gustaf thanks for the response. I've been looking at this in more detail this afternoon and it does appear to be caused by something in the interaction of our OpenACS application with 4.99.27. As I previously mentioned, it has been running fine on 4.99.24 on the same Ubuntu version. I realise that I may not have been clear on this point on my previous email: this is Naviserver running on Ubuntu in a Docker container. The version of Naviserver is based on this Docker build https://github.com/oupfiz5/naviserver-s6 which I have forked and updated to 4.99.27 (I may well have missed something in updating NS version - maybe I should have waited until oupfiz updates his build). * I can confirm that nsd-config.tcl runs fine with 4.99.27 * Some good news: I am able to do an OpenACS clean install on Oracle with 4.99.27. I then successfully installed our application using the APM. However, once I restart Naviserver the problems start. * I tried using the openacs-config.tcl that ships with 4.99.27 and the problems are happening with that too. Here is a selection of the kinds of errors I'm seeing: * munmap_chunk(): invalid pointer * free(): invalid next size (fast) * corrupted size vs. prev_size * malloc(): unaligned tcache chunk detected What is odd is that it seems to be able to handle one request before crashing. Eg. I type in the URL, it shows the /register page but then crashes. After restarting, I enter my login details on the register page, press return. It then crashes. After restarting, it successfully logs me, then crashes again. There is no clear pattern in the logs. I thought it might be related to OCSP and disabled that, but the problems continued to occur. Turning on debug hasn't helped - but maybe there is so much information in the log that I have missed something important. What drivers are you referring to in your question? thanks Brian ________________________________ From: Gustaf Neumann <ne...@wu...> Sent: Thursday 10 August 2023 7:27 pm To: nav...@li... <nav...@li...> Subject: Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu Hi Brian, The new NaviServer versions are running fine on Ubuntu 22.04. Have you recompiled the drivers you are using with the updated version? A good test for the NaviServer binary is to test it with one of the packaged configuration files, e.g. nsd-config.tcl. all the best -gn On 10.08.23 18:23, Brian Fenton wrote: Hello we have been testing out our OpenACS application on Ubuntu 22.04.2 LTS (previously we only ran on Windows). It was working great with Naviserver 4.99.24 but I have been getting constant crashes on more recent versions. I get this error on 4.99.25, 4.99.26 and today I also got it on 4.99.27. The server runs fine until I click on a page, then it immediately crashes. The log has only the following error: free(): invalid size and today I got this one: [10/Aug/2023:15:02:23][303.7fa3a64ee640][-conn:openacs:default:1:119-] Fatal: received fatal signal 11 We have an Oracle application and are using the latest nsoracle driver, which might be a factor here. We have been running it with a pretty old OpenACS config file, so I am currently looking to merge in all the latest changes to ensure that is not an issue. Also note that I am running Naviserver on Docker on Windows, but as mentioned it was running great on 4.99.24. thanks for any help Brian |
|
From: Gustaf N. <ne...@wu...> - 2023-08-10 18:27:37
|
Hi Brian, The new NaviServer versions are running fine on Ubuntu 22.04. Have you recompiled the drivers you are using with the updated version? A good test for the NaviServer binary is to test it with one of the packaged configuration files, e.g. nsd-config.tcl. all the best -gn On 10.08.23 18:23, Brian Fenton wrote: > Hello > > we have been testing out our OpenACS application on Ubuntu 22.04.2 LTS > (previously we only ran on Windows). It was working great with > Naviserver 4.99.24 but I have been getting constant crashes on more > recent versions. > > I get this error on 4.99.25, 4.99.26 and today I also got it on > 4.99.27. The server runs fine until I click on a page, then it > immediately crashes. > The log has only the following error: > free(): invalid size > > and today I got this one: > [10/Aug/2023:15:02:23][303.7fa3a64ee640][-conn:openacs:default:1:119-] > Fatal: received fatal signal 11 > > We have an Oracle application and are using the latest nsoracle > driver, which might be a factor here. > We have been running it with a pretty old OpenACS config file, so I am > currently looking to merge in all the latest changes to ensure that is > not an issue. > Also note that I am running Naviserver on Docker on Windows, but as > mentioned it was running great on 4.99.24. > > thanks for any help > Brian > |
|
From: Brian F. <bri...@ai...> - 2023-08-10 16:56:31
|
Hello we have been testing out our OpenACS application on Ubuntu 22.04.2 LTS (previously we only ran on Windows). It was working great with Naviserver 4.99.24 but I have been getting constant crashes on more recent versions. I get this error on 4.99.25, 4.99.26 and today I also got it on 4.99.27. The server runs fine until I click on a page, then it immediately crashes. The log has only the following error: free(): invalid size and today I got this one: [10/Aug/2023:15:02:23][303.7fa3a64ee640][-conn:openacs:default:1:119-] Fatal: received fatal signal 11 We have an Oracle application and are using the latest nsoracle driver, which might be a factor here. We have been running it with a pretty old OpenACS config file, so I am currently looking to merge in all the latest changes to ensure that is not an issue. Also note that I am running Naviserver on Docker on Windows, but as mentioned it was running great on 4.99.24. thanks for any help Brian |
|
From: Gustaf N. <ne...@wu...> - 2023-08-09 10:47:42
|
Many thanks, David, for figuring this out! Many thanks, David, for figuring this out! The change is incorporated in the nswebpush module on Bitbucket. Against my own rules, I've updated the just released tar file for the modules naviserver-4.99.27-modules.tar.gz to include this change. all the best! -g On 09.08.23 12:19, David Osborne wrote: > Thanks Gustaf - replies inline... > > On Wed, 9 Aug 2023 at 10:38, Gustaf Neumann <ne...@wu...> wrote: > > Hi David, > > We do not have nswebpush somewhere in production. Can you tell > more precisely, what "suddenly" means? > > About lunchtime on 2nd Aug! > > Does this mean, that you have not changed anything in your > environment, but google started to refuse it? > > Yes exactly... > > We've worked out what was angering Google - it was a version of this > code in our case: > https://bitbucket.org/naviserver/nswebpush/src/1e412c76626b29a4573b595a069a8ea10feece8a/webpush-procs.tcl#lines-607 > > Construction of the json from the claim dict was treating "exp" as a > string rather than numeric. > Just as an illustration, this quick hack makes the "make test" run > cleanly in the nswebpush codebase: > > proc dictToJson {dict} { > # > # Serializes a Tcl dict to compact JSON. No testing for > # nested dicts or arrays, these will be simply added as a > # string the JSON is in compact form, meaning no whitespaces > # and newlines between keys/values. > > set pairs {} > dict for {key value} $dict { > regsub -all \" $key "\\\"" key > regsub -all \" $value "\\\"" value > if { $key eq "exp"} { > lappend pairs [subst {"$key":$value}] > } else { > lappend pairs [subst {"$key":"$value"}] > } > } > return "{[join $pairs ,]}" > } > > > > > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel -- Univ.Prof. Dr. Gustaf Neumann Head of the Institute of Information Systems and New Media of Vienna University of Economics and Business Program Director of MSc "Information Systems" |
|
From: David O. <da...@qc...> - 2023-08-09 10:27:53
|
Thanks Gustaf - replies inline... On Wed, 9 Aug 2023 at 10:38, Gustaf Neumann <ne...@wu...> wrote: > Hi David, > > We do not have nswebpush somewhere in production. Can you tell more > precisely, what "suddenly" means? > About lunchtime on 2nd Aug! > Does this mean, that you have not changed anything in your environment, > but google started to refuse it? > Yes exactly... We've worked out what was angering Google - it was a version of this code in our case: https://bitbucket.org/naviserver/nswebpush/src/1e412c76626b29a4573b595a069a8ea10feece8a/webpush-procs.tcl#lines-607 Construction of the json from the claim dict was treating "exp" as a string rather than numeric. Just as an illustration, this quick hack makes the "make test" run cleanly in the nswebpush codebase: proc dictToJson {dict} { # # Serializes a Tcl dict to compact JSON. No testing for # nested dicts or arrays, these will be simply added as a # string the JSON is in compact form, meaning no whitespaces # and newlines between keys/values. set pairs {} dict for {key value} $dict { regsub -all \" $key "\\\"" key regsub -all \" $value "\\\"" value if { $key eq "exp"} { lappend pairs [subst {"$key":$value}] } else { lappend pairs [subst {"$key":"$value"}] } } return "{[join $pairs ,]}" } > > |
|
From: Georg L. <jor...@ma...> - 2023-08-09 10:11:14
|
Hello, Google changed from JWT to OAuth for the FCM HTTP v1 API https://firebase.google.com/docs/cloud-messaging/auth-server?hl=en Have you upgraded from legacy to v1? Best Regards, Georg On 8/9/23 11:37, Gustaf Neumann wrote: > > Hi David, > > We do not have nswebpush somewhere in production. Can you tell more > precisely, what "suddenly" means? > Does this mean, that you have not changed anything in your > environment, but google started to refuse it? > > The implementation in nswebpush uses for JWT the algorithm ES256 > (based on elliptic curves), which seems not supported by google cloud > endpoints, whereas [2] uses ES256, there is as well support in > firebase/php-jwt [3]. Not sure, where to start to look for helping you. > > -g > > [1] > https://cloud.google.com/endpoints/docs/frameworks/python/troubleshoot-jwt?hl=en > [2] > https://cloud.google.com/iap/docs/signed-headers-howto?hl=en#securing_iap_headers > [3] https://github.com/firebase/php-jwt/blob/main/src/JWT.php > > On 08.08.23 17:32, David Osborne wrote: >> Hi there, >> >> We have a chat implementation based on the Naviserver nswebpush >> module which recently stopped working with Google endpoints (eg. >> https://fcm.googleapis.com/fcm/send...). >> Suddenly it's complaining about invalid JWTs. >> >> We went back to reference the nswebpush code. >> https://bitbucket.org/naviserver/nswebpush/src/main/ >> We installed it on a clean Debian Bullseye server with >> latest Naviserver from bitbucket. >> >> When we ran the "make test" we also get a 403 from Google... more >> specifically, the reply was: >> Webpush failed with reply status 403 time 0:88018 headers d8 body >> {invalid JWT provided } https {sslversion TLSv1.3 cipher >> TLS_AES_256_GCM_SHA384} >> >> Is anyone else experiencing this or can make any suggestions as to >> what has changed? >> >> -- >> >> *David >> * > > > > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel |
|
From: Gustaf N. <ne...@wu...> - 2023-08-09 09:53:47
|
Dear all, I am glad to announce that the release of NaviServer 4.99.27 is available at SourceForge [1]. This release is a pure bug-fix and maintenance release, which fixes a bug annoying for some OpenACS users. See below for a summary of the changes. Furthermore, the online documentation at sourceforge https://naviserver.sourceforge.io/n/toc.html is now more mobile-friendly and contains a simple version switcher for the stable release branch (4.99) and the main branch, which will be released as 5.0. All the best! -gustaf neumann [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.27/ ======================================= NaviServer 4.99.27, released 2023-08-09 ======================================= 46 files changed, 739 insertions(+), 172 deletions(-) New Features: ------------- - None Bug Fixes: ---------- - Bug fix: fixed potential crash when fallbackCharset is not defined in the configuration file (using outdated configuration) - Bug fix for fastpath in connection with "ns_serverrootproc" Earlier versions of NaviServer initialized for fastpath the interpreter with the connection conditionally, and rather late. When "ns_serverrootproc" is configured, the interpreter with its linkage to the connection is needed very early in the request. Now it is ensured, that the interpreted is registered for the connection when the callback is issued. NaviServer 5 will have more changes in this respect. Many thanks to Georg Lehner for reporting this bug and testing. - Fixed various spelling errors in source code and documentation - Documentation: * fixed documentation bugs * modernized examples - Ease debugging, how values for "ns_conn location" are determined Configuration Files: -------------------- - Added sample section for charset mapping to sample-config.tcl Modules: -------- The following list contains just bug fixes, new features will be documented with NaviServer 5.0. - nsdbpg: * Fixed potential crash in Tcl, when Database contains UTF-8 characters invalid to Tcl 8.6. Crash was observed in "string tolower $var". * Fixed potential crash, when bind variables are passed via explicit "-bind" option, but variable binding was missing - nsoracle: * Reduced verbosity: For debug messages of the driver implementation, use again "Debug" severity to avoid too much verbosity, when looking for slow queries ("Debug(sql)" severity of nsdb). |
|
From: Gustaf N. <ne...@wu...> - 2023-08-09 09:37:36
|
Hi David, We do not have nswebpush somewhere in production. Can you tell more precisely, what "suddenly" means? Does this mean, that you have not changed anything in your environment, but google started to refuse it? The implementation in nswebpush uses for JWT the algorithm ES256 (based on elliptic curves), which seems not supported by google cloud endpoints, whereas [2] uses ES256, there is as well support in firebase/php-jwt [3]. Not sure, where to start to look for helping you. -g [1] https://cloud.google.com/endpoints/docs/frameworks/python/troubleshoot-jwt?hl=en [2] https://cloud.google.com/iap/docs/signed-headers-howto?hl=en#securing_iap_headers [3] https://github.com/firebase/php-jwt/blob/main/src/JWT.php On 08.08.23 17:32, David Osborne wrote: > Hi there, > > We have a chat implementation based on the Naviserver nswebpush module > which recently stopped working with Google endpoints (eg. > https://fcm.googleapis.com/fcm/send...). > Suddenly it's complaining about invalid JWTs. > > We went back to reference the nswebpush code. > https://bitbucket.org/naviserver/nswebpush/src/main/ > We installed it on a clean Debian Bullseye server with > latest Naviserver from bitbucket. > > When we ran the "make test" we also get a 403 from Google... more > specifically, the reply was: > Webpush failed with reply status 403 time 0:88018 headers d8 body > {invalid JWT provided } https {sslversion TLSv1.3 cipher > TLS_AES_256_GCM_SHA384} > > Is anyone else experiencing this or can make any suggestions as to > what has changed? > > -- > > *David > * |
|
From: David O. <da...@qc...> - 2023-08-08 16:24:15
|
Hi there, We have a chat implementation based on the Naviserver nswebpush module which recently stopped working with Google endpoints (eg. https://fcm.googleapis.com/fcm/send...). Suddenly it's complaining about invalid JWTs. We went back to reference the nswebpush code. https://bitbucket.org/naviserver/nswebpush/src/main/ We installed it on a clean Debian Bullseye server with latest Naviserver from bitbucket. When we ran the "make test" we also get a 403 from Google... more specifically, the reply was: Webpush failed with reply status 403 time 0:88018 headers d8 body {invalid JWT provided } https {sslversion TLSv1.3 cipher TLS_AES_256_GCM_SHA384} Is anyone else experiencing this or can make any suggestions as to what has changed? -- *David* |
|
From: Gustaf N. <ne...@wu...> - 2023-06-22 08:41:45
|
Dear all At the forthcoming OpenACS + EuroTcl/Tk conference (details below), we are considering a "Wishlist" session, where users can place their needs/demands, such we can consider this with higher priority in the development. This also covers NaviServer. If you have something on your mind, please send a short email with such topic to me. We might discuss these with the community at the conference. All the best -gn June 30th, 2023: Registration ends July 19th, 2023: Meet & greet July 20th - July 21st, 2023: Conference For details, see:https://openacs.org/conf2023/ -- Univ.Prof. Dr. Gustaf Neumann Head of the Institute of Information Systems and New Media of Vienna University of Economics and Business Program Director of MSc "Information Systems" |
|
From: Sassy N. <sa...@gm...> - 2023-06-18 11:18:50
|
You are a magician !
On Sun, Jun 18, 2023 at 1:44 PM Gustaf Neumann <ne...@wu...> wrote:
> On 17.06.23 19:08, Sassy Natan wrote:
>
> Is there any way to keep it as it was?
> So encoding convertfrom utf-8 will work like in the past?
>
> The "encoding convertfrom" has not changed, but its output is now properly
> encoded.
>
> The following will probably work for you: add the following section to your
> configuration file (and for other extensions, when you need this).
>
> ns_section ns/encodings {
> ns_param .adp iso8859-1
> }
>
> I am not sure, what this will mean for Tcl9 though.
>
> all the best
>
> -g
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
--
Regards,
Sassy Natan
972-(0)54-2203702
|
|
From: Gustaf N. <ne...@wu...> - 2023-06-18 10:44:00
|
On 17.06.23 19:08, Sassy Natan wrote:
> Is there any way to keep it as it was?
> So encoding convertfrom utf-8 will work like in the past?
The "encoding convertfrom" has not changed, but its output is now
properly encoded.
The following will probably work for you: add the following section to your
configuration file (and for other extensions, when you need this).
ns_section ns/encodings {
ns_param .adp iso8859-1
}
I am not sure, what this will mean for Tcl9 though.
all the best
-g
|
|
From: Sassy N. <sa...@gm...> - 2023-06-17 17:08:40
|
Thank you! The problem is that in my code I was using the encoding convertfrom utf-8 in many places and since I'm in the process of upgrading my system to the new version of NaviSever I will need to maintain two versions of the code. One for the old server and one for the new server. It is not trivial as it sounds as the development cycle of my product is running in parallel in both versions. Is there any way to keep it as it was? So encoding convertfrom utf-8 will work like in the past? even if there is a security bridge? Thank You On Sat, Jun 17, 2023 at 7:27 PM Gustaf Neumann <ne...@wu...> wrote: > Dear Sassy, > > Why do you say that the output right next to text utf-8 is broken? > > You simply do not need the extra [encoding convertfrom utf-8 ... ] in > versions after 4.99.23 in text pages.The UTF-8 handling in earlier > versions was broken. > > The changes in 4.99.23 were triggered by requests of full emoji support > (in UTF-8 multi-byte characters, up to 4 bytes). The changes effect > also the DB-Interface, such that now such multi-byte characters are > fully supported in display and storing. Look e.g.: > https://openacs.org/xowiki/text-with-hebrew > > > One more thing to note is that the log file which also has UTF-8 > characters is not translated. I see the escape characters instead of the > data. > > This is exactly the same answer as in you earlier question. There are > several known attacks based on Unicode, which can as well be used to > trick the person reading the log-file. > > If you do not care about the log file sanitizing and you prefer the old > behavior, turn sanitizing off. > > all the best > > -gn > > > > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel > -- Regards, Sassy Natan 972-(0)54-2203702 |
|
From: Gustaf N. <ne...@wu...> - 2023-06-17 16:27:10
|
Dear Sassy, Why do you say that the output right next to text utf-8 is broken? You simply do not need the extra [encoding convertfrom utf-8 ... ] in versions after 4.99.23 in text pages.The UTF-8 handling in earlier versions was broken. The changes in 4.99.23 were triggered by requests of full emoji support (in UTF-8 multi-byte characters, up to 4 bytes). The changes effect also the DB-Interface, such that now such multi-byte characters are fully supported in display and storing. Look e.g.: https://openacs.org/xowiki/text-with-hebrew > One more thing to note is that the log file which also has UTF-8 characters is not translated. I see the escape characters instead of the data. This is exactly the same answer as in you earlier question. There are several known attacks based on Unicode, which can as well be used to trick the person reading the log-file. If you do not care about the log file sanitizing and you prefer the old behavior, turn sanitizing off. all the best -gn |
|
From: Sassy N. <sa...@gm...> - 2023-06-17 16:11:59
|
thank you
On Wed, Jun 14, 2023 at 1:19 PM Gustaf Neumann <ne...@wu...> wrote:
> Dear Sassy,
>
> You are referring to a change that was released more than 3 years ago
> (4.99.19).
>
> Log file sanitizing works as expected, namely it prints invisible
> characters in a hex representation (in your case tab characters).
> Sanitizing was required by security audits, since otherwise, it would be
> possible to execute code by looking into the log file, one could obfuscate
> the log file, and confuse log file analyzer that alarm when suspect
> activities are noticed, etc.
>
> Certainly, the harm caused by a tab character alone is limited, but when
> debugging and analyze problems, it is often important to distinguish
> between a tab and some spaces.
>
> You can easily replace the hex notation in the log file via "sed", such as
> e.g.
>
> cat log/error.log | sed -En 's/\\x09/\t/p' | more
>
> or the like.
>
> all the best
>
> -gn
> On 13.06.23 20:51, Sassy Natan wrote:
>
> Hi All,
>
> After upgrading my server to the latest version, my log file is broken.
>
> For example I see UTF-8 as special characters.
>
> I check the readme but the sanitizelogfiles 2 doesn't seems to work as
> expected.
> I have in my configuration:
>
> set debug true
> ns_logctl severity "Debug(sql)" on
>
> Any ideas?
>
> Here is example:
> : ],is_break_pcols)) as hrs_wo_break
> : FROM wt_payroll_analysis_pp
> : join (select sid,agg_array(case when is_break_pcol=true then 99999
> else 0 end) as is_break_pcols
> : \x09\x09\x09from ( select * from wt_et_cols_defs where sid=410000
> order by pcol_number )pc_defs group by sid) pdefs using (sid)
> : \x09\x09\x09\x09WHERE sid=410000 and employee_id =40599
> : \x09\x09\x09\x09and to_date(pp_year||'-'||pp||'-01','yyyy-mm-dd')
> between to_date('2023-6-01','yyyy-mm-dd')- interval '6 month'
> : \x09\x09\x09\x09\x09\x09\x09and ( to_date('2023-6-01','yyyy-mm-dd')
> - interval '1 day' )::date
> : \x09\x09\x09\x09UNION
> : \x09\x09\x09\x09 select
> sid,employee_id,pp,pp_year,ppid,to_date(pp_year||'-'||pp||'-01','yyyy-mm-dd')
> as day,null2zero(hrs_wo_break)*3600
> : \x09\x09\x09\x09 from pp_pa_table
> : \x09\x09\x09\x09) ggg
> : \x09\x09\x09\x09group by sid,employee_id,pp,pp_year
>
>
> Thanks
> Sassy
>
> On Mon, May 1, 2023 at 6:55 PM Gustaf Neumann <ne...@wu...> wrote:
>
>> Dear all,
>>
>> I am glad to announce that the release of NaviServer 4.99.25 is
>> available at SourceForge [1]. This release is mostly a bug-fix
>> release. The forthcoming version 5.0 of NaviServer will contain
>> several new features omitted in this bug-fix branch. In case, you are
>> building NaviServer from the Bitbucket repository, please note that
>> the release 4.99.25 is in the branch release/4.99 (bug fix branch for
>> the NaviServer 4.99 family). New development happens in the "main"
>> branch of the repository, leading to NaviServer 5.*.
>>
>> See below for a summary of the changes.
>>
>> Many thanks to the contributors of this release:
>>
>> Andrew Piskorski
>> Antonio Pisano
>> Brian Fenton
>> Gustaf Neumann
>> Hector Romojaro
>> Joe Oldak
>> Khy Huang
>> Oleg Oleinick
>> Zoran Vasiljevic
>>
>> All the best!
>>
>> -gustaf neumann
>>
>> [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.25/
>>
>> =======================================
>> NaviServer 4.99.25, released 2023-05-01
>> =======================================
>>
>> 132 files changed, 3957 insertions(+), 2068 deletions(-)
>>
>> New Features:
>> -------------
>>
>> - Added meta-information to configuration values
>>
>> NaviServer can now report, what configuration values provided in
>> the configuration file were actually used, what their default
>> values are, and whether these values were specified or not (using
>> the default values). With this information, administration
>> (e.g. migration) becomes easier. The NaviServer module "nsstats"
>> shows this meta information via the web interface.
>>
>> This functionality is provided via the new option "-filter" for the
>> command "ns_configsection ... section". When the "-filter" option
>> is used, different kinds of information about the parameters is
>> returned from the specified section.
>>
>> "-filter unread":
>>
>> Returns the parameters, which were set during configuration (i.e.,
>> in the configuration file) but which were not read in the startup
>> phase of the server. This option is useful to determine
>> e.g. typographical errors of specified parameter names.
>>
>> "-filter defaulted":
>>
>> Returns the parameters, from which the default values were read
>> (i.e., which were not explicitly set)
>>
>> "-filter defaults":
>>
>> Returns the defaults of the parameter. This is useful for
>> contrasting the actual values with the default values for
>> Parameters, e.g. in a web based interface.
>>
>> - ns_set reform (per default deactivated in 4.99, but activated in 5.*)
>>
>> The classical implementation for ns_sets uses separately malloced
>> storage for every attribute name and attribute value. So, e.g., for
>> 1000 ns_sets with 20 members each, this means 1,000*20*2 = 40,000
>> malloc/free operations, e.g., for a single db query! Although the
>> malloc implementations have improved over the years, these will
>> require many lock operations, especially under load, where many
>> threads might perform many concurrent malloc operations. One other
>> consequence is that the allocated memory will be scattered over
>> address space, which has bad implications for CPU caching.
>>
>> The new implementation uses for one "ns_set" a single Tcl_DString
>> keeping all attribute names and attribute values. This reduces the
>> malloc operations and improves memory locality, such that cache
>> hits will improve.
>>
>> One caveat of this change is that modules using "ns_set" have to be
>> recompiled, since the full C-level data structure of the "ns_set"
>> is exposed. Therefore, adding a member causes a binary
>> incompatibility. One other potential problem is that C-level
>> modules using the Ns_Set* API have to make sure that long-living
>> string values are copied (this was necessary before as well, but
>> was in many cases no problem, when the "ns_sets" were seldom
>> updated).
>>
>> For high compatibility, this feature is deactivated per default in
>> the 4.99.* series and can be activated by setting the compile-time
>> C macro "NS_SET_DSTRING".
>>
>>
>>
>> API changes:
>> ------------
>>
>> API extensions:
>> - Provide a new interface ending with *Sz to provide string sizes.
>> This reduces the need of strlen() operations.
>> * Ns_SetCreateSz()
>> * Ns_SetIUpdateSz()
>> * Ns_SetPutSz()
>> * Ns_SetPutValueSz()
>> * Ns_SetUpdateSz()
>>
>> - New API calls for "ns_set" reform
>> * Ns_SetClearValues(): clear the values for all keys
>> * Ns_SetDataPrealloc(): creating ns_sets with preallocated values
>> to avoid resize operations
>> * NsSetResize()
>> * NsHeaderSetGet()
>>
>> - Ns_ConfigSet(const char *section, const char *key, const char *name)
>> The last argument is new and allows one to create named sets
>> (previously, all such sets were unnamed)
>>
>> - NsHexPrint(): Print the potentially binary content of a buffer
>> in human-readable form.
>>
>> - Ns_RelativeTime(Ns_Time *relTimePtr, Ns_Time *timePtr)
>> This call implements the inverse operation of Ns_AbsoluteTime(),
>> and is used mostly to make debug messages eye-friendly.
>>
>>
>> Performance Improvements:
>> -------------------------
>>
>> - Replaced malloc operation per log entry by thread local variable in
>> system log implementation.
>>
>> - When NaviServer 4.99.25 is compiled with NS_SET_DSTRING supportm
>> the following preliminary performance results were measured from
>> the "ns_set" reform (see above). The tests were performed on
>> openacs.org (Xeon Gold 6226R CPU @ 2.90GHz, 32 cores,
>> hyper-threading enabled). The test executes the SQL query
>>
>> select * from acs_objects limit 1000
>>
>> 100 times in sequence. This test is run in 1 to 30 concurrent
>> threads. With 30 threads, 3mio tuples are retrieved, and 72 mio
>> malloc/free operations are needed alone for the retrieved values.
>>
>> Before (classical ns_set with many mallocs):
>>
>> threads 1 total 4606.787 ms avg 3285.25 ms
>> threads 5 total 4595.358 ms avg 3493.07 ms
>> threads 10 total 4804.193 ms avg 3755.93 ms
>> threads 20 total 6279.524 ms avg 4569.16 ms
>> threads 30 total 8966.427 ms avg 6618.58 ms
>>
>> After reform (using common Tcl_DString per tuple):
>>
>> threads 1 total 4524.645 ms avg 3242.54 ms
>> threads 5 total 4251.266 ms avg 3450.09 ms
>> threads 10 total 4656.795 ms avg 3665.31 ms
>> threads 20 total 5934.105 ms avg 4671.38 ms
>> threads 30 total 7384.591 ms avg 5642.76 ms
>>
>> To summarize, the improvement increases under higher load (with
>> more parallel threads). E.g., with 30 threads, the total time
>> improved by 17%.... leading also to a smaller RSS. These tests were
>> not performed under "clinical" conditions.
>>
>> The new Tcl API call "ns_set size" can be used to pre-allocate
>> larger ns_sets, such that the usual Tcl_DString growing policy does
>> not kick in, reducing further realloc() operations.
>>
>>
>> Bug Fixes:
>> ----------
>>
>> - Fixed potential crash in "ns_accesslog extendedheaders XXXX".
>> Setting extended headers via configuration file was correct, but
>> changing it dynamically via "ns_accesslog extendedheaders .." was
>> broken. (Issue https://sourceforge.net/p/naviserver/bugs/91/)
>>
>> - "ns_conn location": Fixed potential race condition
>>
>> It was possible that "ns_conn location" could return inconsistent
>> results in a single request, when the underlying sockPtr was
>> aggressively reused. Now, the value of the location member is
>> copied to the connection structure instead of being shared with
>> the socket structure (as before).
>>
>> - "ns_cache_eval -force": Fixed potential race condition
>>
>> There was a problem with "ns_cache_eval -force", where the system
>> relied on the existence of a pre-existing entry, but in case the
>> entry was flushed in the meantime problems a crash might have
>> happened. Now the value during the "-force" call is cleared exactly
>> like in the case of an unset operation. The null-value operations
>> are already protected until these are finished in various places
>> in the code.
>>
>> - "ns_socknread": Fix potentially wrong result for buffered channels
>>
>> - Bug fixes for problems showing up under MS Windows:
>>
>> * Make sure that the output variable of Ns_ObjvIndex() is always an
>> integer. Previously, the output variable was in two places a
>> character variable, causing crashes under MS Windows.
>>
>> * Handle incompatibility in setlocale() under MS Windows. Under
>> MS Windows, later calls to setlocale() overwrite the string
>> returned by former calls. So, it is necessary to copy of the
>> returned string of a setlocale() call under MS Windows.
>>
>> * Handling linking problems: MS Windows requires explicit handling
>> when importing symbols from .dll files (Ns_LogSqlDebug,
>> NS_intTypePtr)
>>
>> * Pass error codes from low-level function SockRecv() and
>> SockSend() via variables. This change makes sure the real error
>> code (immediately after the I/O operation) is passed to the
>> caller to avoid missed error cases and weird error message
>> (e.g. under windows).
>>
>>
>> - Bug fixes for ADP parser:
>>
>> * Support for greater than sign ">" inside attribute values.
>>
>> Previously, NaviServer determined the terminating end-of-tag
>> character as literally the first greater than sign, no matter if
>> this was used as attribute values between single or double
>> quotes.
>>
>> The new version supports such values, since the "Living Standard
>> of HTML" [1] allows the use of less than "<" and greater than
>> ">" signs inside attribute values as long these are between
>> single or double quotes. The guide [2] just recommends using
>> character escapes for "<", ">" and "&".
>>
>> [1] https://html.spec.whatwg.org/multipage/syntax.html#syntax-attribute-value
>> [2] https://www.w3.org/International/questions/qa-escapes
>>
>> This problem was reported by Wolfgang Winkler
>>
>> * Clear ADP flags in case of errors (this error was present for
>> many years)
>>
>> Previously, The following ADP page could lead to a full
>> breakdown of the server, since the error states of the server
>> were never cleared, and subsequent requests served be the same
>> interpreter could lead to the old error states.
>>
>> Many thanks to Oleg Oleinick for reporting and the great test
>> cases.
>>
>> - Provide better HTTP status code for early errors: When the driver
>> terminates already a request, it might be the case that the error
>> flags were not read out. In such cases, we provide now more
>> specific status codes rather than 400.
>>
>> - ns_http:
>>
>> * Improved robustness with domain names resolving against many IP
>> addresses: When a domain name is resolved against many IP
>> addresses, and all these IP addresses block (connect returns "in
>> progress") then the old code might have looped infinitely. Now
>> the code respects the provided timeout (default set to 5s) per
>> resolved IP address.
>>
>> Many thanks to Joe Oldak for pointing out the problem and
>> leading us to the solution.
>>
>> * Fixed timeout handling during TLS handshake: This problem could
>> appear, when the TCP connection to a server succeeded quickly,
>> but the TLS handshake was taking a long time, without a raising
>> a timeout exception, although the timeout time has expired.
>>
>> - Set the default server before the init-scripts is called to make it
>> accessible from there
>>
>> - Provide compatibility between in-memory and file-based form-data
>> handling for invalid characters: Since the file based
>> implementation uses a fallback-charset of iso8859-1 when parsing
>> form data in POST requests (in order to be able to extract
>> "_charset_" data), the in-memory based variant does now the
>> same. The decision of which parser is used is taken based on the
>> size of the form data.... and should therefore be consistent.
>>
>> - nscp: Fixed problem, when the nscp module is activated but no
>> "users" section is specified. Many thanks to Andrew Piskorski for
>> reporting the problem.
>>
>> - Security Improvement: Added protection against sneaking in fake
>> NAME.tmpfile entries into form fields, when performing file upload
>> operations.
>>
>>
>> Documentation improvements:
>> ---------------------------
>>
>> - Improved the following man pages:
>>
>> doc/src/manual/adp-overview.man
>> doc/src/manual/tcl-lib-nstrace.man
>> doc/src/naviserver/commandlist.man
>> doc/src/naviserver/ns_adp.man
>> doc/src/naviserver/ns_adp_ctl.man
>> doc/src/naviserver/ns_config.man
>> doc/src/naviserver/ns_conn.man
>> doc/src/naviserver/ns_connchan.man
>> doc/src/naviserver/ns_cookie.man
>> doc/src/naviserver/ns_crypto.man
>> doc/src/naviserver/ns_http.man
>> doc/src/naviserver/ns_ictl.man
>> doc/src/naviserver/ns_locationproc.man
>> doc/src/naviserver/ns_parsehostport.man
>> doc/src/naviserver/ns_return.man
>> doc/src/naviserver/ns_set.man
>> doc/src/naviserver/ns_setprivileges.man
>>
>>
>> Configuration Files:
>> --------------------
>>
>> - New feature for the sample configuration file openacs-config.tcl
>>
>> The following variables (with prefix oacs_) can be taken from the
>> shell variables. This makes it easier to run similar variants of
>> NaviServer instances from a single configuration file, while
>> providing variables from the command line. This is e.g. useful for
>> Docker or cluster setups.
>>
>> CookieNamespace
>> bindir
>> cachingmode
>> db_host
>> db_name
>> db_port
>> db_user
>> homedir
>> hostname
>> httpport
>> httpsport
>> ipaddress
>> logroot
>> nscpport
>> server
>> serverroot
>> smtpdport
>>
>> One can use e.g.the following command to change some ports and the
>> log file during startup
>>
>> oacs_httpport=8000 oacs_httpsport=8443 oacs_smtpdport=2526 \
>> oacs_logroot=/var/www/XXXX/log-node1/ \
>> /usr/local/ns/bin/nsd -t /var/www/XXXX/etc/openacs-config.tcl ...
>>
>> - further updates for openacs-config.tcl:
>> * Added sample nssmtpd configuration
>> * Added placeholder for ClusterSecret
>> * Reflect recent Oracle requirements (tested with Oracle 19c)
>> * Added documentation for "StaticCSP", "CookieNamespace",
>> "NsShutdownWithNonZeroExitCode", "LogIncludeUserId"
>>
>> - Updated all sample configuration files
>>
>>
>> Code Changes:
>> -------------
>>
>> - Added and updated predefined MIME types based on
>> https://www.iana.org/assignments/media-types/media-types.xhtml
>>
>> - Added compile time macro NS_VERBOSE_MALLOC to identify frequent
>> *alloc operations
>>
>> - Print version of zlib during startup
>>
>> - OpenSSL maintenance:
>>
>> * Improved robustness for OCSP with OpenSSL 3.*
>>
>> The previous version crashed with OpenSSL 3.*, when OCSP was
>> tried on self-signed certificates. Aside of the fact that OCSP
>> does not make sense for self-signed certificates, the server
>> should not crash in such situations.
>>
>> * Fixed ns_crypto::aead::encrypt/decrypt test under OpenSSL 1.1.1
>> (as shipped per default, e.g. on Ubuntu 18.04.4)
>>
>> In short, the problem was that with this version of OpenSSL,
>> setting empty additional authenticated data (AAD) behaved
>> differently from other versions, namely it was clearing
>> incorrectly (forgetting) the information that the initialization
>> vector (IV) was already set. An upgrade of OpenSSL fixed the
>> problem. However, with these changes, also the stock version of
>> OpenSSL can be used. As a byproduct, better error messages are
>> now produced, the code received more cleanup (e.g. explicit
>> initialization, etc.)
>>
>>
>> - Automated testing:
>>
>> * Setup if Bitbucket + GitHub pipelines for automated regression
>> testing with multiple versions of components
>>
>> For NaviServer 4.99 the current setup performs tests with gcc-10
>> + gcc-11, Tcl 8.6.13 + 8.7a5, NSF 2.4,0 + 2.4.0, tDOM 0.9.1 +
>> 0.9.3, extra modules: nsdbpg nsdbi nsdbipg nsudp nscoap nssmtpd.
>>
>> https://github.com/nm-wu/naviserver-mirror/actions
>>
>> * Improved robustness of regression test when running with the
>> docker networking setup
>>
>> * Force nonzero exit code when regression test fails
>>
>> * Added parameter "-timeout" to call of regression test cases
>> Previously, the timeout was hard-wired to 3 seconds. One can now
>> call a test with e.g. "nstest::http -timeout 1s ... GET ..."
>>
>> * adp_compress.test: removed trailing newline to ease
>> cross-platform regression tests
>>
>> * Prefer standard Tcl test constraint "macOrUnix" over own solution
>>
>> * Extended regression tests with more test cases
>>
>> - Code management:
>> * Changed name of branch from "master" to "main"
>>
>> - Code Cleanup
>> * Improved type cleanness
>> * Removed deprecated calls to "sprintf"
>> * Improved portability for Tcl 8.7* (handling of binary data)
>>
>> - Improved comments, fixed typos
>>
>>
>> Changes in NaviServer Modules:
>> ==============================
>>
>> 39 files changed, 9658 insertions(+), 1781 deletions(-)
>>
>> General:
>> Adjust to necessary API changes in NaviServer (contains as well
>> support for the forthcoming release of NaviServer 5)
>>
>>
>> nsdbpg:
>> -------
>>
>> - new pg-driver specific command: ns_pg_prepare /sql/
>>
>> Return a dict building a prepared statement for the passed-in SQL
>> statement. The dict contains the keys "sql" and "args". The
>> function is used by e.g. OpenACS to generate prepared statements
>> from SQL commands with bind variables.
>>
>> - Raise exception when a value for a bind variable contains a NUL character.
>> This value is explicitly forbidden in text strings passed to PostgreSQL.
>>
>> - Let "ns_pg" report available subcommands even when handle is not
>> specified. This makes the command compatible with the "icanuse"
>> feature in OpenACS.
>>
>>
>> nsstats:
>> --------
>>
>> - HTTP client log analysis:
>> * Provide charts for performance (using highcharts via CDN)
>> * Provide charts on request frequency (using highcharts via CDN)
>> * Provide a summary table for HTTP client requests
>> * Improved robustness against invalid URLs (containing unescaped
>> spaces)
>> * Added support for selection of different HTTP client log files
>> via web interface
>>
>> - "Process" page:
>> * Added percentage of request distribution over connection pools
>> * Added information about the connected client
>> * Added more detailed version information
>>
>> - Added cache configuration to output when looking at a single cache
>>
>> - Improved "log file" analysis
>> * Automated stripping of color codes
>>
>> * Added filter option. The filter can be used to grep for (ID)
>> strings in both the system and access logs, providing a summary
>> for the traces of a request in the access log and system log on
>> a single place.
>>
>> - Added default and usage information to "Config Parameters" page
>>
>>
>> nsoracle:
>> ---------
>>
>> - Fixed bug when streaming LOB content to connection. The old code
>> did not distinguish between binary and non-binary content. This
>> bug was discussed in
>> https://openacs.org/forums/message-view?message_id=5693661
>>
>> Bumped version number to 2.9
>>
>> - switched to plain Debug handling for debugging the driver
>>
>> The handling of Ns_LogSqlDebug is performed inside nsdb, including
>> also the printout of (long) SQL statements. Previously, the driver
>> was too chatty when Debug(sql) was turned on.
>>
>> - Added support for output columns of type SQLT_TIMESTAMP or SQLT_TIMESTAMP_TZ
>>
>> This change fixes a bug, where SQL queries of the form
>>
>> SELECT TO_TIMESTAMP(sysdate) FROM dual
>>
>> lead to errors for the form:
>>
>> Database operation "getrow" failed (exception 1406, "nsoracle.c:3659:Ns_OracleGetRow:
>> error in `OCIStmtFetch ()': ORA-01406: fetched column value was truncated
>>
>> The driver needs for several output types special rules, where the
>> timestamp cases were not supported so far. It is also recommended
>> to set the according environment variables specifying the output
>> format in the configuration server of NaviServer, such as e.g.
>>
>> set ::env(NLS_TIMESTAMP_FORMAT) "YYYY-MM-DD HH24:MI:SS.FF6"
>> set ::env(NLS_TIMESTAMP_TZ_FORMAT) "YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM"
>>
>> For testing in you local Oracle installation, you might test the
>> output formats (and the required sizes with the following snippet
>> for sqlplus:
>>
>> COLUMN localtimestamp format a40
>> COLUMN systimestamp format a40
>> COLUMN ts_bytes format a80
>>
>> alter session set nls_timestamp_format='YYYY-MM-DD HH24:MI:SS.FF6';
>> select localtimestamp, length(localtimestamp), dump(localtimestamp) ts_bytes from dual;
>>
>> alter session set nls_timestamp_tz_format='YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM';
>> select systimestamp, length(systimestamp), dump(systimestamp) ts_bytes from dual;
>>
>> alter session set nls_timestamp_tz_format='YYYY-MM-DD HH24:MI:SS.FF6 TZR';
>> select systimestamp, length(systimestamp), dump(systimestamp) ts_bytes from dual;
>>
>>
>> letsencrypt:
>> ------------
>>
>> - Added option to produce certificates with ECDSA:
>>
>> Prior to this change, all certificates were using RSA keys.
>> Since a while, keys based on elliptic curves became the preference
>> of letsencrypt.
>>
>>
>> nsshell:
>> --------
>>
>> - Fixed a bug in "ns_conn" emulation, when the "kernel" was not correctly identified
>>
>>
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>>
>
>
> --
> Regards,
>
> Sassy Natan
> 972-(0)54-2203702
>
>
> _______________________________________________
> naviserver-devel mailing lis...@li...://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
> --
> Univ.Prof. Dr. Gustaf Neumann
> Head of the Institute of Information Systems and New Media
> of Vienna University of Economics and Business
> Program Director of MSc "Information Systems"
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
--
Regards,
Sassy Natan
972-(0)54-2203702
|
|
From: Sassy N. <sa...@gm...> - 2023-06-16 19:02:56
|
Hi Group, I'm facing an issue on my new Ubuntu 22.04 installation with NaviServer/ 4.99.25. First here is the code of my adp page: <html> <meta charset="utf-8"> <body> encoding system:<%=[encoding system]%><br> text utf-8 שליחת מייל <br> function encoding convertfrom utf-8 : <%=[encoding convertfrom utf-8 "\שלח למייל"]%> <br> </body> </html> In NaviServer/4.99.16 everything is working as expected. However in NaviServer/4.99.25 the encoding convertfrom utf-8 is somehow broken. [image: image.png] I have check the locate setting in both servers and they are the same: LANG=en_US.UTF-8 LANGUAGE= LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL= The naviserver in both server print this on the load: *[-main:conf-] Notice: initialized locale en_US.UTF-8 from environment variable LANG[16/Jun/2023:18:51:58][854016.7f8dc402af80][-main:conf-] Warning: config mimtypes: redefine predefined mime type for .adp value 'text/html' with different value: text/html; charset=UTF-8[16/Jun/2023:18:51:58][854016.7f8dc402af80][-main:conf-] Debug: encoding: loaded: utf-8* I also run curl to check the headers: HTTP/1.1 200 OK *Server: NaviServer/4.99.25* Date: Fri, 16 Jun 2023 18:53:45 GMT Set-Cookie: ad_browser_id=414419283; Path=/; Expires=Fri, 01-Jan-2010 01:00:00 GMT Set-Cookie: ad_session_id=414419284,0,c4rYRb4RyIH2tMMORokgBz44T0g9U6XF,1686941625; Path=/; Max-Age=14400 Content-Type: text/html; charset=UTF-8 Expires: now Content-Length: 0 Connection: keep-alive HTTP/1.1 200 OK *Server: NaviServer/4.99.16* Date: Fri, 16 Jun 2023 18:47:05 GMT Set-Cookie: ad_browser_id=622810778; Path=/; Expires=Fri, 01-Jan-2010 01:00:00 GMT Set-Cookie: ad_session_id=622810779,0,P9pbnqGCncOosO.v5xKMTLosjbGdyLQs,1686941225; Path=/; Max-Age=14400 Content-Type: text/html; charset=UTF-8 Expires: now Content-Length: 0 Connection: keep-alive Running the command in a local tclsh gives the same output so I don't think it is a tcl8.6 issue. tclsh % encoding convertfrom utf-8 "שלח למייל" éÜ× ÜÞÙÙÜ % The *Server: NaviServer/4.99.25 is running * tcl 8.6.12+dfsg vs *Server: NaviServer/4.99.16 tcl * 8.6.8+dfsg-3 One more thing to note is that the log file which also has UTF-8 characters is not translated. I see the escape characters instead of the data. Any ideas? What am I missing? Both servers run the same naviserver configuration. I have build the naviserver as a debing pkgs - I can share the code if needed: [image: image.png] -- Regards, Sassy Natan 972-(0)54-2203702 |
|
From: Gustaf N. <ne...@wu...> - 2023-06-16 09:46:54
|
Dear all, I am glad to announce that the release of NaviServer 4.99.26 is available at SourceForge [1]. This release is a pure bug-fix and maintenance release, which fixes a bug annoying for some OpenACS users (wrong value when hostname is retrieved from the configuration values via "ns_config ..."). See below for a summary of the changes. All the best! -gustaf neumann [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.26/ ======================================= NaviServer 4.99.26, released 2023-06-16 ======================================= 18 files changed, 109 insertions(+), 64 deletions(-) New Features: ------------- - None Bug Fixes: ---------- - Bug fix: don't update the value of an ns_set entry using the identical string. Since the update of an ns_set entry requires a free operation of the old memory and a copy operation of the new value, in case the pointers are identical, the new value might be freed in the first step, leading to an invalid value. This condition can happen only from the C level API. Unfortunately, this happened with the sample configuration file for OpenACS. - Increase portability for older compilers and windows. NaviServer 4.99.* uses now same idiom for NS_BITELEMENTS as in NaviServer 5 - improved error message on socket errors (reported unknown error before) Configuration Files: -------------------- - Modernize recommended cipher lists in OpenSSL configuration files |
|
From: Gustaf N. <ne...@wu...> - 2023-06-14 10:19:25
|
Dear Sassy,
You are referring to a change that was released more than 3 years ago
(4.99.19).
Log file sanitizing works as expected, namely it prints invisible
characters in a hex representation (in your case tab characters).
Sanitizing was required by security audits, since otherwise, it would
be possible to execute code by looking into the log file, one could
obfuscate the log file, and confuse log file analyzer that alarm when
suspect activities are noticed, etc.
Certainly, the harm caused by a tab character alone is limited, but when
debugging and analyze problems, it is often important to distinguish
between a tab and some spaces.
You can easily replace the hex notation in the log file via "sed", such
as e.g.
cat log/error.log | sed -En 's/\\x09/\t/p' | more
or the like.
all the best
-gn
On 13.06.23 20:51, Sassy Natan wrote:
> Hi All,
>
> After upgrading my server to the latest version, my log file is broken.
>
> For example I see UTF-8 as special characters.
>
> I check the readme but the sanitizelogfiles 2 doesn't seems to work as
> expected.
> I have in my configuration:
>
> set debug true
> ns_logctl severity "Debug(sql)" on
>
> Any ideas?
>
> Here is example:
> : ],is_break_pcols)) as hrs_wo_break
> : FROM wt_payroll_analysis_pp
> : join (select sid,agg_array(case when is_break_pcol=true then
> 99999 else 0 end) as is_break_pcols
> : \x09\x09\x09from ( select * from wt_et_cols_defs where sid=410000
> order by pcol_number )pc_defs group by sid) pdefs using (sid)
> : \x09\x09\x09\x09WHERE sid=410000 and employee_id =40599
> : \x09\x09\x09\x09and
> to_date(pp_year||'-'||pp||'-01','yyyy-mm-dd') between
> to_date('2023-6-01','yyyy-mm-dd')- interval '6 month'
> : \x09\x09\x09\x09\x09\x09\x09and (
> to_date('2023-6-01','yyyy-mm-dd') - interval '1 day' )::date
> : \x09\x09\x09\x09UNION
> : \x09\x09\x09\x09 select
> sid,employee_id,pp,pp_year,ppid,to_date(pp_year||'-'||pp||'-01','yyyy-mm-dd')
> as day,null2zero(hrs_wo_break)*3600
> : \x09\x09\x09\x09 from pp_pa_table
> : \x09\x09\x09\x09) ggg
> : \x09\x09\x09\x09group by sid,employee_id,pp,pp_year
>
>
> Thanks
> Sassy
>
> On Mon, May 1, 2023 at 6:55 PM Gustaf Neumann <ne...@wu...> wrote:
>
> Dear all,
>
> I am glad to announce that the release of NaviServer 4.99.25 is
> available at SourceForge [1]. This release is mostly a bug-fix
> release. The forthcoming version 5.0 of NaviServer will contain
> several new features omitted in this bug-fix branch. In case, you are
> building NaviServer from the Bitbucket repository, please note that
> the release 4.99.25 is in the branch release/4.99 (bug fix branch for
> the NaviServer 4.99 family). New development happens in the "main"
> branch of the repository, leading to NaviServer 5.*.
>
> See below for a summary of the changes.
>
> Many thanks to the contributors of this release:
>
> Andrew Piskorski
> Antonio Pisano
> Brian Fenton
> Gustaf Neumann
> Hector Romojaro
> Joe Oldak
> Khy Huang
> Oleg Oleinick
> Zoran Vasiljevic
>
> All the best!
>
> -gustaf neumann
>
> [1]
> https://sourceforge.net/projects/naviserver/files/naviserver/4.99.25/
>
> =======================================
> NaviServer 4.99.25, released 2023-05-01
> =======================================
>
> 132 files changed, 3957 insertions(+), 2068 deletions(-)
>
> New Features:
> -------------
>
> - Added meta-information to configuration values
>
> NaviServer can now report, what configuration values provided in
> the configuration file were actually used, what their default
> values are, and whether these values were specified or not (using
> the default values). With this information, administration
> (e.g. migration) becomes easier. The NaviServer module "nsstats"
> shows this meta information via the web interface.
>
> This functionality is provided via the new option "-filter" for the
> command "ns_configsection ... section". When the "-filter" option
> is used, different kinds of information about the parameters is
> returned from the specified section.
>
> "-filter unread":
>
> Returns the parameters, which were set during configuration (i.e.,
> in the configuration file) but which were not read in the startup
> phase of the server. This option is useful to determine
> e.g. typographical errors of specified parameter names.
>
> "-filter defaulted":
>
> Returns the parameters, from which the default values were read
> (i.e., which were not explicitly set)
>
> "-filter defaults":
>
> Returns the defaults of the parameter. This is useful for
> contrasting the actual values with the default values for
> Parameters, e.g. in a web based interface.
>
> - ns_set reform (per default deactivated in 4.99, but activated in 5.*)
>
> The classical implementation for ns_sets uses separately malloced
> storage for every attribute name and attribute value. So, e.g., for
> 1000 ns_sets with 20 members each, this means 1,000*20*2 = 40,000
> malloc/free operations, e.g., for a single db query! Although the
> malloc implementations have improved over the years, these will
> require many lock operations, especially under load, where many
> threads might perform many concurrent malloc operations. One other
> consequence is that the allocated memory will be scattered over
> address space, which has bad implications for CPU caching.
>
> The new implementation uses for one "ns_set" a single Tcl_DString
> keeping all attribute names and attribute values. This reduces the
> malloc operations and improves memory locality, such that cache
> hits will improve.
>
> One caveat of this change is that modules using "ns_set" have to be
> recompiled, since the full C-level data structure of the "ns_set"
> is exposed. Therefore, adding a member causes a binary
> incompatibility. One other potential problem is that C-level
> modules using the Ns_Set* API have to make sure that long-living
> string values are copied (this was necessary before as well, but
> was in many cases no problem, when the "ns_sets" were seldom
> updated).
>
> For high compatibility, this feature is deactivated per default in
> the 4.99.* series and can be activated by setting the compile-time
> C macro "NS_SET_DSTRING".
>
>
>
> API changes:
> ------------
>
> API extensions:
> - Provide a new interface ending with *Sz to provide string sizes.
> This reduces the need of strlen() operations.
> * Ns_SetCreateSz()
> * Ns_SetIUpdateSz()
> * Ns_SetPutSz()
> * Ns_SetPutValueSz()
> * Ns_SetUpdateSz()
>
> - New API calls for "ns_set" reform
> * Ns_SetClearValues(): clear the values for all keys
> * Ns_SetDataPrealloc(): creating ns_sets with preallocated values
> to avoid resize operations
> * NsSetResize()
> * NsHeaderSetGet()
>
> - Ns_ConfigSet(const char *section, const char *key, const char *name)
> The last argument is new and allows one to create named sets
> (previously, all such sets were unnamed)
>
> - NsHexPrint(): Print the potentially binary content of a buffer
> in human-readable form.
>
> - Ns_RelativeTime(Ns_Time *relTimePtr, Ns_Time *timePtr)
> This call implements the inverse operation of Ns_AbsoluteTime(),
> and is used mostly to make debug messages eye-friendly.
>
>
> Performance Improvements:
> -------------------------
>
> - Replaced malloc operation per log entry by thread local variable in
> system log implementation.
>
> - When NaviServer 4.99.25 is compiled with NS_SET_DSTRING supportm
> the following preliminary performance results were measured from
> the "ns_set" reform (see above). The tests were performed on
> openacs.org <http://openacs.org> (Xeon Gold 6226R CPU @ 2.90GHz, 32 cores,
> hyper-threading enabled). The test executes the SQL query
>
> select * from acs_objects limit 1000
>
> 100 times in sequence. This test is run in 1 to 30 concurrent
> threads. With 30 threads, 3mio tuples are retrieved, and 72 mio
> malloc/free operations are needed alone for the retrieved values.
>
> Before (classical ns_set with many mallocs):
>
> threads 1 total 4606.787 ms avg 3285.25 ms
> threads 5 total 4595.358 ms avg 3493.07 ms
> threads 10 total 4804.193 ms avg 3755.93 ms
> threads 20 total 6279.524 ms avg 4569.16 ms
> threads 30 total 8966.427 ms avg 6618.58 ms
>
> After reform (using common Tcl_DString per tuple):
>
> threads 1 total 4524.645 ms avg 3242.54 ms
> threads 5 total 4251.266 ms avg 3450.09 ms
> threads 10 total 4656.795 ms avg 3665.31 ms
> threads 20 total 5934.105 ms avg 4671.38 ms
> threads 30 total 7384.591 ms avg 5642.76 ms
>
> To summarize, the improvement increases under higher load (with
> more parallel threads). E.g., with 30 threads, the total time
> improved by 17%.... leading also to a smaller RSS. These tests were
> not performed under "clinical" conditions.
>
> The new Tcl API call "ns_set size" can be used to pre-allocate
> larger ns_sets, such that the usual Tcl_DString growing policy does
> not kick in, reducing further realloc() operations.
>
>
> Bug Fixes:
> ----------
>
> - Fixed potential crash in "ns_accesslog extendedheaders XXXX".
> Setting extended headers via configuration file was correct, but
> changing it dynamically via "ns_accesslog extendedheaders .." was
> broken. (Issuehttps://sourceforge.net/p/naviserver/bugs/91/)
>
> - "ns_conn location": Fixed potential race condition
>
> It was possible that "ns_conn location" could return inconsistent
> results in a single request, when the underlying sockPtr was
> aggressively reused. Now, the value of the location member is
> copied to the connection structure instead of being shared with
> the socket structure (as before).
>
> - "ns_cache_eval -force": Fixed potential race condition
>
> There was a problem with "ns_cache_eval -force", where the system
> relied on the existence of a pre-existing entry, but in case the
> entry was flushed in the meantime problems a crash might have
> happened. Now the value during the "-force" call is cleared exactly
> like in the case of an unset operation. The null-value operations
> are already protected until these are finished in various places
> in the code.
>
> - "ns_socknread": Fix potentially wrong result for buffered channels
>
> - Bug fixes for problems showing up under MS Windows:
>
> * Make sure that the output variable of Ns_ObjvIndex() is always an
> integer. Previously, the output variable was in two places a
> character variable, causing crashes under MS Windows.
>
> * Handle incompatibility in setlocale() under MS Windows. Under
> MS Windows, later calls to setlocale() overwrite the string
> returned by former calls. So, it is necessary to copy of the
> returned string of a setlocale() call under MS Windows.
>
> * Handling linking problems: MS Windows requires explicit handling
> when importing symbols from .dll files (Ns_LogSqlDebug,
> NS_intTypePtr)
>
> * Pass error codes from low-level function SockRecv() and
> SockSend() via variables. This change makes sure the real error
> code (immediately after the I/O operation) is passed to the
> caller to avoid missed error cases and weird error message
> (e.g. under windows).
>
>
> - Bug fixes for ADP parser:
>
> * Support for greater than sign ">" inside attribute values.
>
> Previously, NaviServer determined the terminating end-of-tag
> character as literally the first greater than sign, no matter if
> this was used as attribute values between single or double
> quotes.
>
> The new version supports such values, since the "Living Standard
> of HTML" [1] allows the use of less than "<" and greater than
> ">" signs inside attribute values as long these are between
> single or double quotes. The guide [2] just recommends using
> character escapes for "<", ">" and "&".
>
> [1]https://html.spec.whatwg.org/multipage/syntax.html#syntax-attribute-value
> [2]https://www.w3.org/International/questions/qa-escapes
>
> This problem was reported by Wolfgang Winkler
>
> * Clear ADP flags in case of errors (this error was present for
> many years)
>
> Previously, The following ADP page could lead to a full
> breakdown of the server, since the error states of the server
> were never cleared, and subsequent requests served be the same
> interpreter could lead to the old error states.
>
> Many thanks to Oleg Oleinick for reporting and the great test
> cases.
>
> - Provide better HTTP status code for early errors: When the driver
> terminates already a request, it might be the case that the error
> flags were not read out. In such cases, we provide now more
> specific status codes rather than 400.
>
> - ns_http:
>
> * Improved robustness with domain names resolving against many IP
> addresses: When a domain name is resolved against many IP
> addresses, and all these IP addresses block (connect returns "in
> progress") then the old code might have looped infinitely. Now
> the code respects the provided timeout (default set to 5s) per
> resolved IP address.
>
> Many thanks to Joe Oldak for pointing out the problem and
> leading us to the solution.
>
> * Fixed timeout handling during TLS handshake: This problem could
> appear, when the TCP connection to a server succeeded quickly,
> but the TLS handshake was taking a long time, without a raising
> a timeout exception, although the timeout time has expired.
>
> - Set the default server before the init-scripts is called to make it
> accessible from there
>
> - Provide compatibility between in-memory and file-based form-data
> handling for invalid characters: Since the file based
> implementation uses a fallback-charset of iso8859-1 when parsing
> form data in POST requests (in order to be able to extract
> "_charset_" data), the in-memory based variant does now the
> same. The decision of which parser is used is taken based on the
> size of the form data.... and should therefore be consistent.
>
> - nscp: Fixed problem, when the nscp module is activated but no
> "users" section is specified. Many thanks to Andrew Piskorski for
> reporting the problem.
>
> - Security Improvement: Added protection against sneaking in fake
> NAME.tmpfile entries into form fields, when performing file upload
> operations.
>
>
> Documentation improvements:
> ---------------------------
>
> - Improved the following man pages:
>
> doc/src/manual/adp-overview.man
> doc/src/manual/tcl-lib-nstrace.man
> doc/src/naviserver/commandlist.man
> doc/src/naviserver/ns_adp.man
> doc/src/naviserver/ns_adp_ctl.man
> doc/src/naviserver/ns_config.man
> doc/src/naviserver/ns_conn.man
> doc/src/naviserver/ns_connchan.man
> doc/src/naviserver/ns_cookie.man
> doc/src/naviserver/ns_crypto.man
> doc/src/naviserver/ns_http.man
> doc/src/naviserver/ns_ictl.man
> doc/src/naviserver/ns_locationproc.man
> doc/src/naviserver/ns_parsehostport.man
> doc/src/naviserver/ns_return.man
> doc/src/naviserver/ns_set.man
> doc/src/naviserver/ns_setprivileges.man
>
>
> Configuration Files:
> --------------------
>
> - New feature for the sample configuration file openacs-config.tcl
>
> The following variables (with prefix oacs_) can be taken from the
> shell variables. This makes it easier to run similar variants of
> NaviServer instances from a single configuration file, while
> providing variables from the command line. This is e.g. useful for
> Docker or cluster setups.
>
> CookieNamespace
> bindir
> cachingmode
> db_host
> db_name
> db_port
> db_user
> homedir
> hostname
> httpport
> httpsport
> ipaddress
> logroot
> nscpport
> server
> serverroot
> smtpdport
>
> One can use e.g.the following command to change some ports and the
> log file during startup
>
> oacs_httpport=8000 oacs_httpsport=8443 oacs_smtpdport=2526 \
> oacs_logroot=/var/www/XXXX/log-node1/ \
> /usr/local/ns/bin/nsd -t /var/www/XXXX/etc/openacs-config.tcl ...
>
> - further updates for openacs-config.tcl:
> * Added sample nssmtpd configuration
> * Added placeholder for ClusterSecret
> * Reflect recent Oracle requirements (tested with Oracle 19c)
> * Added documentation for "StaticCSP", "CookieNamespace",
> "NsShutdownWithNonZeroExitCode", "LogIncludeUserId"
>
> - Updated all sample configuration files
>
>
> Code Changes:
> -------------
>
> - Added and updated predefined MIME types based on
> https://www.iana.org/assignments/media-types/media-types.xhtml
>
> - Added compile time macro NS_VERBOSE_MALLOC to identify frequent
> *alloc operations
>
> - Print version of zlib during startup
>
> - OpenSSL maintenance:
>
> * Improved robustness for OCSP with OpenSSL 3.*
>
> The previous version crashed with OpenSSL 3.*, when OCSP was
> tried on self-signed certificates. Aside of the fact that OCSP
> does not make sense for self-signed certificates, the server
> should not crash in such situations.
>
> * Fixed ns_crypto::aead::encrypt/decrypt test under OpenSSL 1.1.1
> (as shipped per default, e.g. on Ubuntu 18.04.4)
>
> In short, the problem was that with this version of OpenSSL,
> setting empty additional authenticated data (AAD) behaved
> differently from other versions, namely it was clearing
> incorrectly (forgetting) the information that the initialization
> vector (IV) was already set. An upgrade of OpenSSL fixed the
> problem. However, with these changes, also the stock version of
> OpenSSL can be used. As a byproduct, better error messages are
> now produced, the code received more cleanup (e.g. explicit
> initialization, etc.)
>
>
> - Automated testing:
>
> * Setup if Bitbucket + GitHub pipelines for automated regression
> testing with multiple versions of components
>
> For NaviServer 4.99 the current setup performs tests with gcc-10
> + gcc-11, Tcl 8.6.13 + 8.7a5, NSF 2.4,0 + 2.4.0, tDOM 0.9.1 +
> 0.9.3, extra modules: nsdbpg nsdbi nsdbipg nsudp nscoap nssmtpd.
>
> https://github.com/nm-wu/naviserver-mirror/actions
>
> * Improved robustness of regression test when running with the
> docker networking setup
>
> * Force nonzero exit code when regression test fails
>
> * Added parameter "-timeout" to call of regression test cases
> Previously, the timeout was hard-wired to 3 seconds. One can now
> call a test with e.g. "nstest::http -timeout 1s ... GET ..."
>
> * adp_compress.test: removed trailing newline to ease
> cross-platform regression tests
>
> * Prefer standard Tcl test constraint "macOrUnix" over own solution
>
> * Extended regression tests with more test cases
>
> - Code management:
> * Changed name of branch from "master" to "main"
>
> - Code Cleanup
> * Improved type cleanness
> * Removed deprecated calls to "sprintf"
> * Improved portability for Tcl 8.7* (handling of binary data)
>
> - Improved comments, fixed typos
>
>
> Changes in NaviServer Modules:
> ==============================
>
> 39 files changed, 9658 insertions(+), 1781 deletions(-)
>
> General:
> Adjust to necessary API changes in NaviServer (contains as well
> support for the forthcoming release of NaviServer 5)
>
>
> nsdbpg:
> -------
>
> - new pg-driver specific command: ns_pg_prepare /sql/
>
> Return a dict building a prepared statement for the passed-in SQL
> statement. The dict contains the keys "sql" and "args". The
> function is used by e.g. OpenACS to generate prepared statements
> from SQL commands with bind variables.
>
> - Raise exception when a value for a bind variable contains a NUL character.
> This value is explicitly forbidden in text strings passed to PostgreSQL.
>
> - Let "ns_pg" report available subcommands even when handle is not
> specified. This makes the command compatible with the "icanuse"
> feature in OpenACS.
>
>
> nsstats:
> --------
>
> - HTTP client log analysis:
> * Provide charts for performance (using highcharts via CDN)
> * Provide charts on request frequency (using highcharts via CDN)
> * Provide a summary table for HTTP client requests
> * Improved robustness against invalid URLs (containing unescaped
> spaces)
> * Added support for selection of different HTTP client log files
> via web interface
>
> - "Process" page:
> * Added percentage of request distribution over connection pools
> * Added information about the connected client
> * Added more detailed version information
>
> - Added cache configuration to output when looking at a single cache
>
> - Improved "log file" analysis
> * Automated stripping of color codes
>
> * Added filter option. The filter can be used to grep for (ID)
> strings in both the system and access logs, providing a summary
> for the traces of a request in the access log and system log on
> a single place.
>
> - Added default and usage information to "Config Parameters" page
>
>
> nsoracle:
> ---------
>
> - Fixed bug when streaming LOB content to connection. The old code
> did not distinguish between binary and non-binary content. This
> bug was discussed in
> https://openacs.org/forums/message-view?message_id=5693661
>
> Bumped version number to 2.9
>
> - switched to plain Debug handling for debugging the driver
>
> The handling of Ns_LogSqlDebug is performed inside nsdb, including
> also the printout of (long) SQL statements. Previously, the driver
> was too chatty when Debug(sql) was turned on.
>
> - Added support for output columns of type SQLT_TIMESTAMP or SQLT_TIMESTAMP_TZ
>
> This change fixes a bug, where SQL queries of the form
>
> SELECT TO_TIMESTAMP(sysdate) FROM dual
>
> lead to errors for the form:
>
> Database operation "getrow" failed (exception 1406, "nsoracle.c:3659:Ns_OracleGetRow:
> error in `OCIStmtFetch ()': ORA-01406: fetched column value was truncated
>
> The driver needs for several output types special rules, where the
> timestamp cases were not supported so far. It is also recommended
> to set the according environment variables specifying the output
> format in the configuration server of NaviServer, such as e.g.
>
> set ::env(NLS_TIMESTAMP_FORMAT) "YYYY-MM-DD HH24:MI:SS.FF6"
> set ::env(NLS_TIMESTAMP_TZ_FORMAT) "YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM"
>
> For testing in you local Oracle installation, you might test the
> output formats (and the required sizes with the following snippet
> for sqlplus:
>
> COLUMN localtimestamp format a40
> COLUMN systimestamp format a40
> COLUMN ts_bytes format a80
>
> alter session set nls_timestamp_format='YYYY-MM-DD HH24:MI:SS.FF6';
> select localtimestamp, length(localtimestamp), dump(localtimestamp) ts_bytes from dual;
>
> alter session set nls_timestamp_tz_format='YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM';
> select systimestamp, length(systimestamp), dump(systimestamp) ts_bytes from dual;
>
> alter session set nls_timestamp_tz_format='YYYY-MM-DD HH24:MI:SS.FF6 TZR';
> select systimestamp, length(systimestamp), dump(systimestamp) ts_bytes from dual;
>
>
> letsencrypt:
> ------------
>
> - Added option to produce certificates with ECDSA:
>
> Prior to this change, all certificates were using RSA keys.
> Since a while, keys based on elliptic curves became the preference
> of letsencrypt.
>
>
> nsshell:
> --------
>
> - Fixed a bug in "ns_conn" emulation, when the "kernel" was not correctly identified
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>
>
> --
> Regards,
>
> Sassy Natan
> 972-(0)54-2203702
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
--
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"
|
|
From: Sassy N. <sa...@gm...> - 2023-06-13 18:51:28
|
Hi All,
After upgrading my server to the latest version, my log file is broken.
For example I see UTF-8 as special characters.
I check the readme but the sanitizelogfiles 2 doesn't seems to work as
expected.
I have in my configuration:
set debug true
ns_logctl severity "Debug(sql)" on
Any ideas?
Here is example:
: ],is_break_pcols)) as hrs_wo_break
: FROM wt_payroll_analysis_pp
: join (select sid,agg_array(case when is_break_pcol=true then 99999
else 0 end) as is_break_pcols
: \x09\x09\x09from ( select * from wt_et_cols_defs where sid=410000
order by pcol_number )pc_defs group by sid) pdefs using (sid)
: \x09\x09\x09\x09WHERE sid=410000 and employee_id =40599
: \x09\x09\x09\x09and to_date(pp_year||'-'||pp||'-01','yyyy-mm-dd')
between to_date('2023-6-01','yyyy-mm-dd')- interval '6 month'
: \x09\x09\x09\x09\x09\x09\x09and ( to_date('2023-6-01','yyyy-mm-dd') -
interval '1 day' )::date
: \x09\x09\x09\x09UNION
: \x09\x09\x09\x09 select
sid,employee_id,pp,pp_year,ppid,to_date(pp_year||'-'||pp||'-01','yyyy-mm-dd')
as day,null2zero(hrs_wo_break)*3600
: \x09\x09\x09\x09 from pp_pa_table
: \x09\x09\x09\x09) ggg
: \x09\x09\x09\x09group by sid,employee_id,pp,pp_year
Thanks
Sassy
On Mon, May 1, 2023 at 6:55 PM Gustaf Neumann <ne...@wu...> wrote:
> Dear all,
>
> I am glad to announce that the release of NaviServer 4.99.25 is
> available at SourceForge [1]. This release is mostly a bug-fix
> release. The forthcoming version 5.0 of NaviServer will contain
> several new features omitted in this bug-fix branch. In case, you are
> building NaviServer from the Bitbucket repository, please note that
> the release 4.99.25 is in the branch release/4.99 (bug fix branch for
> the NaviServer 4.99 family). New development happens in the "main"
> branch of the repository, leading to NaviServer 5.*.
>
> See below for a summary of the changes.
>
> Many thanks to the contributors of this release:
>
> Andrew Piskorski
> Antonio Pisano
> Brian Fenton
> Gustaf Neumann
> Hector Romojaro
> Joe Oldak
> Khy Huang
> Oleg Oleinick
> Zoran Vasiljevic
>
> All the best!
>
> -gustaf neumann
>
> [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.25/
>
> =======================================
> NaviServer 4.99.25, released 2023-05-01
> =======================================
>
> 132 files changed, 3957 insertions(+), 2068 deletions(-)
>
> New Features:
> -------------
>
> - Added meta-information to configuration values
>
> NaviServer can now report, what configuration values provided in
> the configuration file were actually used, what their default
> values are, and whether these values were specified or not (using
> the default values). With this information, administration
> (e.g. migration) becomes easier. The NaviServer module "nsstats"
> shows this meta information via the web interface.
>
> This functionality is provided via the new option "-filter" for the
> command "ns_configsection ... section". When the "-filter" option
> is used, different kinds of information about the parameters is
> returned from the specified section.
>
> "-filter unread":
>
> Returns the parameters, which were set during configuration (i.e.,
> in the configuration file) but which were not read in the startup
> phase of the server. This option is useful to determine
> e.g. typographical errors of specified parameter names.
>
> "-filter defaulted":
>
> Returns the parameters, from which the default values were read
> (i.e., which were not explicitly set)
>
> "-filter defaults":
>
> Returns the defaults of the parameter. This is useful for
> contrasting the actual values with the default values for
> Parameters, e.g. in a web based interface.
>
> - ns_set reform (per default deactivated in 4.99, but activated in 5.*)
>
> The classical implementation for ns_sets uses separately malloced
> storage for every attribute name and attribute value. So, e.g., for
> 1000 ns_sets with 20 members each, this means 1,000*20*2 = 40,000
> malloc/free operations, e.g., for a single db query! Although the
> malloc implementations have improved over the years, these will
> require many lock operations, especially under load, where many
> threads might perform many concurrent malloc operations. One other
> consequence is that the allocated memory will be scattered over
> address space, which has bad implications for CPU caching.
>
> The new implementation uses for one "ns_set" a single Tcl_DString
> keeping all attribute names and attribute values. This reduces the
> malloc operations and improves memory locality, such that cache
> hits will improve.
>
> One caveat of this change is that modules using "ns_set" have to be
> recompiled, since the full C-level data structure of the "ns_set"
> is exposed. Therefore, adding a member causes a binary
> incompatibility. One other potential problem is that C-level
> modules using the Ns_Set* API have to make sure that long-living
> string values are copied (this was necessary before as well, but
> was in many cases no problem, when the "ns_sets" were seldom
> updated).
>
> For high compatibility, this feature is deactivated per default in
> the 4.99.* series and can be activated by setting the compile-time
> C macro "NS_SET_DSTRING".
>
>
>
> API changes:
> ------------
>
> API extensions:
> - Provide a new interface ending with *Sz to provide string sizes.
> This reduces the need of strlen() operations.
> * Ns_SetCreateSz()
> * Ns_SetIUpdateSz()
> * Ns_SetPutSz()
> * Ns_SetPutValueSz()
> * Ns_SetUpdateSz()
>
> - New API calls for "ns_set" reform
> * Ns_SetClearValues(): clear the values for all keys
> * Ns_SetDataPrealloc(): creating ns_sets with preallocated values
> to avoid resize operations
> * NsSetResize()
> * NsHeaderSetGet()
>
> - Ns_ConfigSet(const char *section, const char *key, const char *name)
> The last argument is new and allows one to create named sets
> (previously, all such sets were unnamed)
>
> - NsHexPrint(): Print the potentially binary content of a buffer
> in human-readable form.
>
> - Ns_RelativeTime(Ns_Time *relTimePtr, Ns_Time *timePtr)
> This call implements the inverse operation of Ns_AbsoluteTime(),
> and is used mostly to make debug messages eye-friendly.
>
>
> Performance Improvements:
> -------------------------
>
> - Replaced malloc operation per log entry by thread local variable in
> system log implementation.
>
> - When NaviServer 4.99.25 is compiled with NS_SET_DSTRING supportm
> the following preliminary performance results were measured from
> the "ns_set" reform (see above). The tests were performed on
> openacs.org (Xeon Gold 6226R CPU @ 2.90GHz, 32 cores,
> hyper-threading enabled). The test executes the SQL query
>
> select * from acs_objects limit 1000
>
> 100 times in sequence. This test is run in 1 to 30 concurrent
> threads. With 30 threads, 3mio tuples are retrieved, and 72 mio
> malloc/free operations are needed alone for the retrieved values.
>
> Before (classical ns_set with many mallocs):
>
> threads 1 total 4606.787 ms avg 3285.25 ms
> threads 5 total 4595.358 ms avg 3493.07 ms
> threads 10 total 4804.193 ms avg 3755.93 ms
> threads 20 total 6279.524 ms avg 4569.16 ms
> threads 30 total 8966.427 ms avg 6618.58 ms
>
> After reform (using common Tcl_DString per tuple):
>
> threads 1 total 4524.645 ms avg 3242.54 ms
> threads 5 total 4251.266 ms avg 3450.09 ms
> threads 10 total 4656.795 ms avg 3665.31 ms
> threads 20 total 5934.105 ms avg 4671.38 ms
> threads 30 total 7384.591 ms avg 5642.76 ms
>
> To summarize, the improvement increases under higher load (with
> more parallel threads). E.g., with 30 threads, the total time
> improved by 17%.... leading also to a smaller RSS. These tests were
> not performed under "clinical" conditions.
>
> The new Tcl API call "ns_set size" can be used to pre-allocate
> larger ns_sets, such that the usual Tcl_DString growing policy does
> not kick in, reducing further realloc() operations.
>
>
> Bug Fixes:
> ----------
>
> - Fixed potential crash in "ns_accesslog extendedheaders XXXX".
> Setting extended headers via configuration file was correct, but
> changing it dynamically via "ns_accesslog extendedheaders .." was
> broken. (Issue https://sourceforge.net/p/naviserver/bugs/91/)
>
> - "ns_conn location": Fixed potential race condition
>
> It was possible that "ns_conn location" could return inconsistent
> results in a single request, when the underlying sockPtr was
> aggressively reused. Now, the value of the location member is
> copied to the connection structure instead of being shared with
> the socket structure (as before).
>
> - "ns_cache_eval -force": Fixed potential race condition
>
> There was a problem with "ns_cache_eval -force", where the system
> relied on the existence of a pre-existing entry, but in case the
> entry was flushed in the meantime problems a crash might have
> happened. Now the value during the "-force" call is cleared exactly
> like in the case of an unset operation. The null-value operations
> are already protected until these are finished in various places
> in the code.
>
> - "ns_socknread": Fix potentially wrong result for buffered channels
>
> - Bug fixes for problems showing up under MS Windows:
>
> * Make sure that the output variable of Ns_ObjvIndex() is always an
> integer. Previously, the output variable was in two places a
> character variable, causing crashes under MS Windows.
>
> * Handle incompatibility in setlocale() under MS Windows. Under
> MS Windows, later calls to setlocale() overwrite the string
> returned by former calls. So, it is necessary to copy of the
> returned string of a setlocale() call under MS Windows.
>
> * Handling linking problems: MS Windows requires explicit handling
> when importing symbols from .dll files (Ns_LogSqlDebug,
> NS_intTypePtr)
>
> * Pass error codes from low-level function SockRecv() and
> SockSend() via variables. This change makes sure the real error
> code (immediately after the I/O operation) is passed to the
> caller to avoid missed error cases and weird error message
> (e.g. under windows).
>
>
> - Bug fixes for ADP parser:
>
> * Support for greater than sign ">" inside attribute values.
>
> Previously, NaviServer determined the terminating end-of-tag
> character as literally the first greater than sign, no matter if
> this was used as attribute values between single or double
> quotes.
>
> The new version supports such values, since the "Living Standard
> of HTML" [1] allows the use of less than "<" and greater than
> ">" signs inside attribute values as long these are between
> single or double quotes. The guide [2] just recommends using
> character escapes for "<", ">" and "&".
>
> [1] https://html.spec.whatwg.org/multipage/syntax.html#syntax-attribute-value
> [2] https://www.w3.org/International/questions/qa-escapes
>
> This problem was reported by Wolfgang Winkler
>
> * Clear ADP flags in case of errors (this error was present for
> many years)
>
> Previously, The following ADP page could lead to a full
> breakdown of the server, since the error states of the server
> were never cleared, and subsequent requests served be the same
> interpreter could lead to the old error states.
>
> Many thanks to Oleg Oleinick for reporting and the great test
> cases.
>
> - Provide better HTTP status code for early errors: When the driver
> terminates already a request, it might be the case that the error
> flags were not read out. In such cases, we provide now more
> specific status codes rather than 400.
>
> - ns_http:
>
> * Improved robustness with domain names resolving against many IP
> addresses: When a domain name is resolved against many IP
> addresses, and all these IP addresses block (connect returns "in
> progress") then the old code might have looped infinitely. Now
> the code respects the provided timeout (default set to 5s) per
> resolved IP address.
>
> Many thanks to Joe Oldak for pointing out the problem and
> leading us to the solution.
>
> * Fixed timeout handling during TLS handshake: This problem could
> appear, when the TCP connection to a server succeeded quickly,
> but the TLS handshake was taking a long time, without a raising
> a timeout exception, although the timeout time has expired.
>
> - Set the default server before the init-scripts is called to make it
> accessible from there
>
> - Provide compatibility between in-memory and file-based form-data
> handling for invalid characters: Since the file based
> implementation uses a fallback-charset of iso8859-1 when parsing
> form data in POST requests (in order to be able to extract
> "_charset_" data), the in-memory based variant does now the
> same. The decision of which parser is used is taken based on the
> size of the form data.... and should therefore be consistent.
>
> - nscp: Fixed problem, when the nscp module is activated but no
> "users" section is specified. Many thanks to Andrew Piskorski for
> reporting the problem.
>
> - Security Improvement: Added protection against sneaking in fake
> NAME.tmpfile entries into form fields, when performing file upload
> operations.
>
>
> Documentation improvements:
> ---------------------------
>
> - Improved the following man pages:
>
> doc/src/manual/adp-overview.man
> doc/src/manual/tcl-lib-nstrace.man
> doc/src/naviserver/commandlist.man
> doc/src/naviserver/ns_adp.man
> doc/src/naviserver/ns_adp_ctl.man
> doc/src/naviserver/ns_config.man
> doc/src/naviserver/ns_conn.man
> doc/src/naviserver/ns_connchan.man
> doc/src/naviserver/ns_cookie.man
> doc/src/naviserver/ns_crypto.man
> doc/src/naviserver/ns_http.man
> doc/src/naviserver/ns_ictl.man
> doc/src/naviserver/ns_locationproc.man
> doc/src/naviserver/ns_parsehostport.man
> doc/src/naviserver/ns_return.man
> doc/src/naviserver/ns_set.man
> doc/src/naviserver/ns_setprivileges.man
>
>
> Configuration Files:
> --------------------
>
> - New feature for the sample configuration file openacs-config.tcl
>
> The following variables (with prefix oacs_) can be taken from the
> shell variables. This makes it easier to run similar variants of
> NaviServer instances from a single configuration file, while
> providing variables from the command line. This is e.g. useful for
> Docker or cluster setups.
>
> CookieNamespace
> bindir
> cachingmode
> db_host
> db_name
> db_port
> db_user
> homedir
> hostname
> httpport
> httpsport
> ipaddress
> logroot
> nscpport
> server
> serverroot
> smtpdport
>
> One can use e.g.the following command to change some ports and the
> log file during startup
>
> oacs_httpport=8000 oacs_httpsport=8443 oacs_smtpdport=2526 \
> oacs_logroot=/var/www/XXXX/log-node1/ \
> /usr/local/ns/bin/nsd -t /var/www/XXXX/etc/openacs-config.tcl ...
>
> - further updates for openacs-config.tcl:
> * Added sample nssmtpd configuration
> * Added placeholder for ClusterSecret
> * Reflect recent Oracle requirements (tested with Oracle 19c)
> * Added documentation for "StaticCSP", "CookieNamespace",
> "NsShutdownWithNonZeroExitCode", "LogIncludeUserId"
>
> - Updated all sample configuration files
>
>
> Code Changes:
> -------------
>
> - Added and updated predefined MIME types based on
> https://www.iana.org/assignments/media-types/media-types.xhtml
>
> - Added compile time macro NS_VERBOSE_MALLOC to identify frequent
> *alloc operations
>
> - Print version of zlib during startup
>
> - OpenSSL maintenance:
>
> * Improved robustness for OCSP with OpenSSL 3.*
>
> The previous version crashed with OpenSSL 3.*, when OCSP was
> tried on self-signed certificates. Aside of the fact that OCSP
> does not make sense for self-signed certificates, the server
> should not crash in such situations.
>
> * Fixed ns_crypto::aead::encrypt/decrypt test under OpenSSL 1.1.1
> (as shipped per default, e.g. on Ubuntu 18.04.4)
>
> In short, the problem was that with this version of OpenSSL,
> setting empty additional authenticated data (AAD) behaved
> differently from other versions, namely it was clearing
> incorrectly (forgetting) the information that the initialization
> vector (IV) was already set. An upgrade of OpenSSL fixed the
> problem. However, with these changes, also the stock version of
> OpenSSL can be used. As a byproduct, better error messages are
> now produced, the code received more cleanup (e.g. explicit
> initialization, etc.)
>
>
> - Automated testing:
>
> * Setup if Bitbucket + GitHub pipelines for automated regression
> testing with multiple versions of components
>
> For NaviServer 4.99 the current setup performs tests with gcc-10
> + gcc-11, Tcl 8.6.13 + 8.7a5, NSF 2.4,0 + 2.4.0, tDOM 0.9.1 +
> 0.9.3, extra modules: nsdbpg nsdbi nsdbipg nsudp nscoap nssmtpd.
>
> https://github.com/nm-wu/naviserver-mirror/actions
>
> * Improved robustness of regression test when running with the
> docker networking setup
>
> * Force nonzero exit code when regression test fails
>
> * Added parameter "-timeout" to call of regression test cases
> Previously, the timeout was hard-wired to 3 seconds. One can now
> call a test with e.g. "nstest::http -timeout 1s ... GET ..."
>
> * adp_compress.test: removed trailing newline to ease
> cross-platform regression tests
>
> * Prefer standard Tcl test constraint "macOrUnix" over own solution
>
> * Extended regression tests with more test cases
>
> - Code management:
> * Changed name of branch from "master" to "main"
>
> - Code Cleanup
> * Improved type cleanness
> * Removed deprecated calls to "sprintf"
> * Improved portability for Tcl 8.7* (handling of binary data)
>
> - Improved comments, fixed typos
>
>
> Changes in NaviServer Modules:
> ==============================
>
> 39 files changed, 9658 insertions(+), 1781 deletions(-)
>
> General:
> Adjust to necessary API changes in NaviServer (contains as well
> support for the forthcoming release of NaviServer 5)
>
>
> nsdbpg:
> -------
>
> - new pg-driver specific command: ns_pg_prepare /sql/
>
> Return a dict building a prepared statement for the passed-in SQL
> statement. The dict contains the keys "sql" and "args". The
> function is used by e.g. OpenACS to generate prepared statements
> from SQL commands with bind variables.
>
> - Raise exception when a value for a bind variable contains a NUL character.
> This value is explicitly forbidden in text strings passed to PostgreSQL.
>
> - Let "ns_pg" report available subcommands even when handle is not
> specified. This makes the command compatible with the "icanuse"
> feature in OpenACS.
>
>
> nsstats:
> --------
>
> - HTTP client log analysis:
> * Provide charts for performance (using highcharts via CDN)
> * Provide charts on request frequency (using highcharts via CDN)
> * Provide a summary table for HTTP client requests
> * Improved robustness against invalid URLs (containing unescaped
> spaces)
> * Added support for selection of different HTTP client log files
> via web interface
>
> - "Process" page:
> * Added percentage of request distribution over connection pools
> * Added information about the connected client
> * Added more detailed version information
>
> - Added cache configuration to output when looking at a single cache
>
> - Improved "log file" analysis
> * Automated stripping of color codes
>
> * Added filter option. The filter can be used to grep for (ID)
> strings in both the system and access logs, providing a summary
> for the traces of a request in the access log and system log on
> a single place.
>
> - Added default and usage information to "Config Parameters" page
>
>
> nsoracle:
> ---------
>
> - Fixed bug when streaming LOB content to connection. The old code
> did not distinguish between binary and non-binary content. This
> bug was discussed in
> https://openacs.org/forums/message-view?message_id=5693661
>
> Bumped version number to 2.9
>
> - switched to plain Debug handling for debugging the driver
>
> The handling of Ns_LogSqlDebug is performed inside nsdb, including
> also the printout of (long) SQL statements. Previously, the driver
> was too chatty when Debug(sql) was turned on.
>
> - Added support for output columns of type SQLT_TIMESTAMP or SQLT_TIMESTAMP_TZ
>
> This change fixes a bug, where SQL queries of the form
>
> SELECT TO_TIMESTAMP(sysdate) FROM dual
>
> lead to errors for the form:
>
> Database operation "getrow" failed (exception 1406, "nsoracle.c:3659:Ns_OracleGetRow:
> error in `OCIStmtFetch ()': ORA-01406: fetched column value was truncated
>
> The driver needs for several output types special rules, where the
> timestamp cases were not supported so far. It is also recommended
> to set the according environment variables specifying the output
> format in the configuration server of NaviServer, such as e.g.
>
> set ::env(NLS_TIMESTAMP_FORMAT) "YYYY-MM-DD HH24:MI:SS.FF6"
> set ::env(NLS_TIMESTAMP_TZ_FORMAT) "YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM"
>
> For testing in you local Oracle installation, you might test the
> output formats (and the required sizes with the following snippet
> for sqlplus:
>
> COLUMN localtimestamp format a40
> COLUMN systimestamp format a40
> COLUMN ts_bytes format a80
>
> alter session set nls_timestamp_format='YYYY-MM-DD HH24:MI:SS.FF6';
> select localtimestamp, length(localtimestamp), dump(localtimestamp) ts_bytes from dual;
>
> alter session set nls_timestamp_tz_format='YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM';
> select systimestamp, length(systimestamp), dump(systimestamp) ts_bytes from dual;
>
> alter session set nls_timestamp_tz_format='YYYY-MM-DD HH24:MI:SS.FF6 TZR';
> select systimestamp, length(systimestamp), dump(systimestamp) ts_bytes from dual;
>
>
> letsencrypt:
> ------------
>
> - Added option to produce certificates with ECDSA:
>
> Prior to this change, all certificates were using RSA keys.
> Since a while, keys based on elliptic curves became the preference
> of letsencrypt.
>
>
> nsshell:
> --------
>
> - Fixed a bug in "ns_conn" emulation, when the "kernel" was not correctly identified
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
--
Regards,
Sassy Natan
972-(0)54-2203702
|
|
From: Gustaf N. <ne...@wu...> - 2023-05-13 14:33:12
|
Dear all, The deadline for submissions of abstracts for presentation at the forthcoming joint OpenACS / EuroTcl conference is in about one month. Presentations about NaviServer applications are very welcome. Important dates: June 15th, 2023: Deadline for submissions of abstracts (max. 2 pages, min. 250 words); June 19th, 2023: Notification of acceptance June 30th, 2023: Registration ends July 19th, 2023: Meet & greet July 20th - July 21st, 2023: Conference For details, see: https://openacs.org/conf2023/ Gustaf Neumann and Harald Oehlmann -- Univ.Prof. Dr. Gustaf Neumann Head of the Institute of Information Systems and New Media of Vienna University of Economics and Business Program Director of MSc "Information Systems" |
14 messages has been excluded from this view by a project administrator.
