naviserver-devel — Developer discussion

Re: [naviserver-devel] Question re nsdbi

[Iuri S. <iu...@iu...>]

I have no clue about the reason.Perhaps a memory licking that went away after reboot .   ...Or you get "lucky" again and the error appears once more. rsrs

My first guess would be if task’s using free(); or what else.  how does memory get released? 



> On Sep 17, 2019, at 15:57, THORPE MAYES via naviserver-devel <nav...@li...> wrote:
> 
> Hi,
> 
> This happened today:
> 
> I was getting thousands of these lines in my server log:
> 
> [17/Sep/2019:07:59:07][4924.7f82467f4700][-conn:mealdeliverysoftware:19:25314-] Notice: dbipg: prepare dbipg_6242353/0 cols 0: deallocate dbipg_6238059
> [17/Sep/2019:07:59:07][4924.7f82467f4700][-conn:mealdeliverysoftware:19:25314-] Notice: dbipg: prepare dbipg_6242354/0 cols 0: deallocate dbipg_6238060
> [17/Sep/2019:07:59:07][4924.7f82467f4700][-conn:mealdeliverysoftware:19:25314-] Notice: dbipg: prepare dbipg_6242355/0 cols 0: deallocate dbipg_6238061	 
> 
> I restarted the server and the problem went away (maybe).
> 
> My guess is that configuration is either wrong or not complete. Here is the section in my config file that deals with this:
> 
> # database configuration                                                                                                                                                         
> ns_section "ns/server/${servername}/module/xxxxx"
>      ns_param   default        true ;# This is the default pool for mds                                                                                                          
>      ns_param   handles        2    ;# Max open handles to db.                                                                                                                   
>      ns_param   maxwait        10   ;# Seconds to wait if handle unavailable.                                                                                                    
>      ns_param   maxidle        0    ;# Handle closed after maxidle seconds if unused.                                                                                            
>      ns_param   maxopen        0    ;# Handle closed after maxopen seconds, regardles of use.                                                                                     
>      ns_param   maxqueries     0    ;# Handle closed after maxqueries sql queries.                                                                                               
>      ns_param   checkinterval  600  ;# Check for idle handles every 10 minutes.                                                                                                  
> 
>      ns_param   maxhandles     0
>      ns_param   timeout        10
>      ns_param   maxrows        1000
> 
>      ns_param   cachesize     [expr 1024*1024]
> 
> # The following parameters are configured at server-startup.                                                                                                                     
>     ns_param   user          “xxxxxx"
>     ns_param   password      xxxxxxx
>     ns_param   database      “xxxxxx"
>     ns_param   datasource     "user=‘xxxxx' password=‘xxxxx' dbname=‘xxxxxx’"
> 
> 
> This is also in the config file for my legacy code:
> # database pool configuration                                                                                                                                                    
> ns_section "ns/db/pool/${default_pool}"
>     ns_param    driver              postgres
>     ns_param    connections         40
>     ns_param    datasource          localhost:5432:${database_name}
>     ns_param    user                xxxxxx
>     ns_param    password            xxxxxx
>     ns_param    verbose             true
>     ns_param    logsqlerrors        true
>     ns_param    extendedtableinfo   true
>     ns_param    maxidle             1000000000
>     ns_param    maxopen             1000000000
> 
> 
> # Database pools accessible by server                                                                                                                                            
> ns_section "ns/server/${servername}/db"
>     ns_param    pools               "*"
>     ns_param    defaultpool         "${default_pool}"
> 
> 
> 
> I will appreciate any help/direction.
> 
> Thank you,
> 
> Thorpe
> 
> Thorpe Mayes
> 2313 Lockhill-Selma Road, Suite 164
> San Antonio, Texas 78230-3007
> Phone: (512) 394-8766
> 
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] Question re nsdbi

[THORPE M. <ta...@me...>]

Hi,

This happened today:

I was getting thousands of these lines in my server log:

[17/Sep/2019:07:59:07][4924.7f82467f4700][-conn:mealdeliverysoftware:19:25314-] Notice: dbipg: prepare dbipg_6242353/0 cols 0: deallocate dbipg_6238059
[17/Sep/2019:07:59:07][4924.7f82467f4700][-conn:mealdeliverysoftware:19:25314-] Notice: dbipg: prepare dbipg_6242354/0 cols 0: deallocate dbipg_6238060
[17/Sep/2019:07:59:07][4924.7f82467f4700][-conn:mealdeliverysoftware:19:25314-] Notice: dbipg: prepare dbipg_6242355/0 cols 0: deallocate dbipg_6238061	 

I restarted the server and the problem went away (maybe).

My guess is that configuration is either wrong or not complete. Here is the section in my config file that deals with this:

# database configuration                                                                                                                                                          
ns_section "ns/server/${servername}/module/xxxxx"
     ns_param   default        true ;# This is the default pool for mds                                                                                                           
     ns_param   handles        2    ;# Max open handles to db.                                                                                                                    
     ns_param   maxwait        10   ;# Seconds to wait if handle unavailable.                                                                                                     
     ns_param   maxidle        0    ;# Handle closed after maxidle seconds if unused.                                                                                             
     ns_param   maxopen        0    ;# Handle closed after maxopen seconds, regardles of use.                                                                                     
     ns_param   maxqueries     0    ;# Handle closed after maxqueries sql queries.                                                                                                
     ns_param   checkinterval  600  ;# Check for idle handles every 10 minutes.                                                                                                   

     ns_param   maxhandles     0
     ns_param   timeout        10
     ns_param   maxrows        1000

     ns_param   cachesize     [expr 1024*1024]

# The following parameters are configured at server-startup.                                                                                                                      
    ns_param   user          “xxxxxx"
    ns_param   password      xxxxxxx
    ns_param   database      “xxxxxx"
    ns_param   datasource     "user=‘xxxxx' password=‘xxxxx' dbname=‘xxxxxx’"


This is also in the config file for my legacy code:
# database pool configuration                                                                                                                                                     
ns_section "ns/db/pool/${default_pool}"
    ns_param    driver              postgres
    ns_param    connections         40
    ns_param    datasource          localhost:5432:${database_name}
    ns_param    user                xxxxxx
    ns_param    password            xxxxxx
    ns_param    verbose             true
    ns_param    logsqlerrors        true
    ns_param    extendedtableinfo   true
    ns_param    maxidle             1000000000
    ns_param    maxopen             1000000000


# Database pools accessible by server                                                                                                                                             
ns_section "ns/server/${servername}/db"
    ns_param    pools               "*"
    ns_param    defaultpool         "${default_pool}"



I will appreciate any help/direction.

Thank you,

Thorpe

Thorpe Mayes
2313 Lockhill-Selma Road, Suite 164
San Antonio, Texas 78230-3007
Phone: (512) 394-8766

Re: [naviserver-devel] Problem compiling NaviServer 4.99.18 in FreeBSD 11.2

[Gustaf N. <ne...@wu...>]

On 28.08.19 21:38, Roderick wrote:
> Only "make test" made few errors (see below). I suppose nothing bad.

it is at least nothing that causes more than normal hairloss for me.

Reasons:
(a) you were running the unreleased TIP branch, and
(b) have no NSF installed (using install-ns.sh would have taken care for 
this).

I've added constraints to NaviServer to leave these testcases out.

The current head version should run now on a recent FreeBSD whith
results like:

    Files with failing tests: ns_addrbyhost.test
    Number of tests skipped for each constraint:
	2	binaryMismatch
	2	knownBug
	1	notDarwin
	11	nsf
	1	stress

    % [root@freebsd /usr/local/src/naviserver]# uname -a
    FreeBSD freebsd 12.0-STABLE FreeBSD 12.0-STABLE r350087 GENERIC  amd64

The failing test is:

    %  ns_addrbyhost this_should_not_resolve
    169.254.1.1

... meaning: the FreeBSD resolver seems to resolve unknown
addresses to 169.254.1.1 ... which is a link-local address [1].
This can be probably changed via some kernel configuration,
but i am not sure, we have to take care about that.

Greetings from Reunion
-g

[1] https://en.wikipedia.org/wiki/Link-local_address

Re: [naviserver-devel] Mercurial at Bitbucket is ending. What next?

[Roderick <hr...@gm...>]

On Fri, 30 Aug 2019, Jeff Rogers wrote:

> It's struck me as odd that the source was hosted on bitbucket ...

Perhaps the best would be to host it in a server running naviserver
with fossil as cgi (or scgi or http proxy) script. :)

https://fossil-scm.org/index.html/doc/trunk/www/server/

I find naviserver interesting because of its integration with tcl.
I really do not understand why tcl is getting forgotten and python
(that contains the entire tcl in its tkinter) is so popular.

In any case, putting it together with other things in some way associated
with tcl (like fossil) could be of advantage.

Rodrigo

Re: [naviserver-devel] Mercurial at Bitbucket is ending. What next?

[Jeff R. <dv...@di...>]

Sourceforge supports mercurial repos as well as git.   It's struck me as 
odd that the source was hosted on bitbucket while the "main page" and 
distributions are on SF;  is there history for why the source repo isn't 
on SF as well?

-J


On 08/29/2019 03:06 AM, Zoran Vasiljevic wrote:
> On Thu, 29 Aug 2019 09:13:17 +0000 (UTC)
> Roderick <hr...@gm...> wrote:
>
>> After reading something about mercurial only for cloning Naviservers
>> Repo, I read this:
>>
>> https://bitbucket.org/blog/sunsetting-mercurial-support-in-bitbucket
>>
>> It would be nice to have NaviServer as fossil repo. :)
> I believe most logical choice is git. I'm not fan of any of
> the systems, to be honest. Just, the chance that we will have
> to switch again (cvs -> mercurial -> ?) is less if the ? = git.
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] Mercurial at Bitbucket is ending. What next?

[Roderick <hr...@gm...>]

On Thu, 29 Aug 2019, Zoran Vasiljevic wrote:

> I believe most logical choice is git. I'm not fan of any of
> the systems, to be honest.

Yes, it seems. I am also not a fan of anyone, and much less of
this inflation of versioning systems. But I see advantages of
CVS and fossil, in quite very different issues: the first because
it saves everything as rcs files, the seccond because of its
simplicity, flexibility, everything (repo, wiki, tickets, etc)
integrated in a single sqlite file that can be moved in the file
system or to other computer as a normal file, simple commands
that also remind cvs.

I am not a professional programmer, do not have a lot of experience
with this systems, but I use fossil (not in its full power) for
my personall small programs. It is also the system used in tcl/tk
and many tcl projects.

Perhaps the following is interesting:

https://www.fossil-scm.org/xfer/doc/trunk/www/inout.wiki
https://www.fossil-scm.org/xfer/doc/trunk/www/mirrortogithub.md

Rodrigo

Re: [naviserver-devel] Mercurial at Bitbucket is ending. What next?

[Zoran V. <zv...@ar...>]

On Thu, 29 Aug 2019 09:13:17 +0000 (UTC)
Roderick <hr...@gm...> wrote:

> After reading something about mercurial only for cloning Naviservers
> Repo, I read this:
> 
> https://bitbucket.org/blog/sunsetting-mercurial-support-in-bitbucket
> 
> It would be nice to have NaviServer as fossil repo. :)

I believe most logical choice is git. I'm not fan of any of 
the systems, to be honest. Just, the chance that we will have
to switch again (cvs -> mercurial -> ?) is less if the ? = git.

[naviserver-devel] Mercurial at Bitbucket is ending. What next?

[Roderick <hr...@gm...>]

After reading something about mercurial only for cloning Naviservers
Repo, I read this:

https://bitbucket.org/blog/sunsetting-mercurial-support-in-bitbucket

It would be nice to have NaviServer as fossil repo. :)

Rodrigo

Re: [naviserver-devel] Problem compiling NaviServer 4.99.18 in FreeBSD 11.2

[Roderick <hr...@gm...>]

On Wed, 28 Aug 2019, Gustaf Neumann wrote:

> Can it be that you have to update some packages [3]?

I thank you again. The problem was not naviserver, but indeed my FreeBSD
installation. After upgrading to FreeBSD 11.3, configure runs and
it compiles quite cleanly.

Only "make test" made few errors (see below). I suppose nothing bad.

Regards
Rodrigo



----------------------------------------------

gmake test
[...]

s_adp_compress.test
ns_base64.test


==== ns_base64encode-3.0 UTF-8 data, from string or file,
     binary or not,
     convert everything via I/O to bytearray FAILED
---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
==== ns_base64encode-3.0 FAILED



==== ns_base64encode-3.1 euro-sign, in unicode-notation,
     literally and
     converted to a bytearray FAILED
---- Result was:
invalid command name "nsf::__db_get_obj"
---- Result should have been (exact matching):
{} {} bytearray {4oKs 4oKs 4oKs} {bytearray bytearray bytearray} {4oKs 4oKs 4oKs}
==== ns_base64encode-3.1 FAILED



==== ns_base64encode-3.2 mathematical symbols in UTF-8, from forums,
     literally and
     converted to a bytearray FAILED
---- Result was:
invalid command name "nsf::__db_get_obj"
---- Result should have been (exact matching):
{} bytearray {w6TDtsO8yoPJqmLJmWzJm864 w6TDtsO8yoPJqmLJmWzJm864} {bytearray bytearray} {w6TDtsO8yoPJqmLJmWzJm864 w6TDtsO8yoPJqmLJmWzJm864}
==== ns_base64encode-3.2 FAILED



==== ns_base64encode-3.3 binary data, from file or memory FAILED
---- Result was:
invalid command name "nsf::__db_get_obj"
---- Result should have been (exact matching):
{} bytearray bytearray string {AAECA+KYgA== AAECA+KYgA== AAECA+KYgA==} {bytearray bytearray bytearray} {AAECA+KYgA== AAECA+KYgA== AAECA+KYgA==}
==== ns_base64encode-3.3 FAILED



==== ns_base64decode-4.0 base64decoded from provided string + binary scan FAILED
---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
==== ns_base64decode-4.0 FAILED

[...]

==== md-utf-1.0 plain umlaut, 16 bits UTF-8 FAILED
---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
==== md-utf-1.0 FAILED



==== md-utf-1.1 character outside 16bit range, 24 bits UTF-8 FAILED
---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
==== md-utf-1.1 FAILED



==== hmac-md-1.0.1 test results of hmac passed to ns_crypto::md FAILED
---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
==== hmac-md-1.0.1 FAILED



==== hmac-md-1.1 test results of ns_crypto::hmac passed to ns_md FAILED
---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
==== hmac-md-1.1 FAILED



==== base64-md5.1 ns_base64encode and decode with 2 byte UTF-8 character "ü" FAILED
---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
==== base64-md5.1 FAILED



==== base64-md5.2 ns_base64encode and decode with 2 byte UTF-8 character "ü" FAILED
---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
==== base64-md5.2 FAILED

[...]

Tests ended at Wed Aug 28 19:27:40 UTC 2019
all.tcl:        Total   1438    Passed  1415    Skipped 12      Failed  11
Sourced 69 Test Files.
Files with failing tests: ns_base64.test ns_driver.test
Number of tests skipped for each constraint:
         2       Thread
         2       binaryMismatch
         2       knownBug
         1       notDarwin
         4       nsf
         1       stress
gmake: *** [Makefile:238: test] Terminated

Re: [naviserver-devel] Problem compiling NaviServer 4.99.18 in FreeBSD 11.2

[Roderick <hr...@gm...>]

Dear Gustaf, thanks for the answer!

> Is this a new installation or some upgrade?

New, first source from sourforge, later from a hg clone from github,
and then from an old version that compiled before. Configure fails
in the three with the error I mentioned.

For the hg clone I had to run "autogen.sh", it tried to run configure, but
failed because it did not find tcl, then I run "connfigure --with-tcl=XX".

> See in detail, what the error means [2].

I see. I never understood all these autotools and for what they are.
Seing them reminds me on inflation and hyperinflation. I would prefer
to read some docs and edit Makefiles as some years ago.

> Can it be that you have to update some packages [3]?

Or the opposite? I have autoconf-2.69

Regards
Rodrigo

Re: [naviserver-devel] Problem compiling NaviServer 4.99.18 in FreeBSD 11.2

[Gustaf N. <ne...@wu...>]

Dear Rodrigo,

i've recently used [1] to compile NaviServer for  FreeBSD-12.0-STABLE 
successfully, so there is no known problem in this regard.

Is this a new installation or some upgrade?
See in detail, what the error means [2].
Can it be that you have to update some packages [3]?

-g


[1] https://github.com/gustafn/install-ns
[2] 
https://www.gnu.org/software/autoconf/manual/autoconf-2.69/html_node/Present-But-Cannot-Be-Compiled.html
[3] 
https://forums.freebsd.org/threads/pkg-broken-after-ugrading-freebsd-from-10-1-to-10-2.52817/



On 28.08.19 12:43, Roderick wrote:
>
> Hello!
>
> Perhaps a litle hint helps. I get the following error:
>
> """"""
> """"""
> configure: WARNING: zlib.h: present but cannot be compiled
> configure: WARNING: zlib.h:     check for missing prerequisite headers?
> configure: WARNING: zlib.h: see the Autoconf documentation
> configure: WARNING: zlib.h:     section "Present But Cannot Be Compiled"
> configure: WARNING: zlib.h: proceeding with the compiler's result
> configure: WARNING:     ## 
> ----------------------------------------------------- ##
> configure: WARNING:     ## Report this to 
> nav...@li... ##
> configure: WARNING:     ## 
> ----------------------------------------------------- ##
> checking for zlib.h... no
> checking for compress2 in -lz... no
> configure: error: Zlib compression support requested but not available
> """"""
> """"""
>
> But /usr/include/zlib.h is there.
>
> I get similar warnings with
>  /usr/include/netinet/tcp.h
> sys/uio.h
> sys/time.h
> ...
>
> Any idea?
>
> Thanks
> Rodrigo 

[naviserver-devel] Problem compiling NaviServer 4.99.18 in FreeBSD 11.2

[Roderick <hr...@gm...>]

Hello!

Perhaps a litle hint helps. I get the following error:

""""""
""""""
configure: WARNING: zlib.h: present but cannot be compiled
configure: WARNING: zlib.h:     check for missing prerequisite headers?
configure: WARNING: zlib.h: see the Autoconf documentation
configure: WARNING: zlib.h:     section "Present But Cannot Be Compiled"
configure: WARNING: zlib.h: proceeding with the compiler's result
configure: WARNING:     ## 
----------------------------------------------------- ##
configure: WARNING:     ## Report this to 
nav...@li... ##
configure: WARNING:     ## 
----------------------------------------------------- ##
checking for zlib.h... no
checking for compress2 in -lz... no
configure: error: Zlib compression support requested but not available
""""""
""""""

But /usr/include/zlib.h is there.

I get similar warnings with
  /usr/include/netinet/tcp.h
sys/uio.h
sys/time.h
...

Any idea?

Thanks
Rodrigo

Re: [naviserver-devel] Naviserver Open Files

[Gustaf N. <ne...@wu...>]

Dear Wolfgang,

we do not see any growth in this area, below are statistics
from two different sites.

monthly graph


These files are from Ns_GetTemp() which maintains a pool of tmp files,
which are released via Ns_ReleaseTemp(). This pool is maintained like
this since many years.

One of the consumers of Ns_GetTemp() is nscgi. Are you using this module
on the site experiences the growing number?

-g

[naviserver-devel] Naviserver Open Files

[Wolfgang W. <Wol...@di...>]

Dear all,

out naviserver instance opens a lot of files. It used to be constant 200 
open files, now the number ist constantly growing, from 110 after a 
server restart to > 1000.

What we see with "lsof -n" are a lot of files in tmp, e.g. 
/tmp/nstmp.1563598239.865582 that obivously never get closed.

We updated from 4.99.17 to 4.99.18 but the problem persists.

Does anybody have an idea what could cause this behaviour? We checked 
all our changes to the system and could not find the cause.

Regards,

Wolfgang Winkler

-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

Re: [naviserver-devel] select/poll

[Gustaf N. <ne...@wu...>]

Dear jeff, no bad effects, i have this running locally on my machine.
-g

[naviserver-devel] select/poll (was: Re: Tcl nonblocking file IO in NaviServer?)

[Jeff R. <dv...@di...>]

Recent / upcoming releases of Tcl include support for poll and epoll to
get past the limits of select.  Is this going to conflict with anything
in naviserver?

https://core.tcl-lang.org/tips/doc/trunk/tip/458.md

-J


On 06/06/2019 12:23 PM, Gustaf Neumann wrote:
>
> I am rather planing to reduce the usage of the Tcl-based
> AsyncDiskWriter, since this depends on select() in current Tcl
> implementations, which will run into problems when there are
> more than 1024 fds are open. It is however quite easy to
> provide a Tcl Interface to NsAsyncWrite() in NaviServer,
> to implement this when needed, sine the base infrastructure is
> already there.

Re: [naviserver-devel] small bugs break Windows build

[Gustaf N. <ne...@wu...>]

Dear Andrew, many thanks for letting us know.
I've addressed these issues, and added a NS_INLINE that should as well
work for older Microsoft C compilers...

all the best
-g

On 16.07.19 21:07, Andrew Piskorski wrote:
> On Tue, Jul 16, 2019 at 02:51:06PM -0400, Andrew Piskorski wrote:
>> There are two small recent bugs that break compilation on Windows.
> One more:  In "naviserver/include/ns.h" on line 2874, the declaration
> of Ns_SockaddrParseIPMask() is missing NS_EXTERN, which breaks linking
> for nsperm, like so:
>
>    nsperm.o : error LNK2019: unresolved external symbol Ns_SockaddrParseIPMask referenced in function AddUserObjCmd
>    nsperm.dll : fatal error LNK1120: 1 unresolved externals
>
> Adding "NS_EXTERN" fixes the problem.
>

Re: [naviserver-devel] small bugs break Windows build

[Andrew P. <at...@pi...>]

On Tue, Jul 16, 2019 at 02:51:06PM -0400, Andrew Piskorski wrote:
> There are two small recent bugs that break compilation on Windows.

One more:  In "naviserver/include/ns.h" on line 2874, the declaration
of Ns_SockaddrParseIPMask() is missing NS_EXTERN, which breaks linking
for nsperm, like so:

  nsperm.o : error LNK2019: unresolved external symbol Ns_SockaddrParseIPMask referenced in function AddUserObjCmd
  nsperm.dll : fatal error LNK1120: 1 unresolved externals

Adding "NS_EXTERN" fixes the problem.

-- 
Andrew Piskorski <at...@pi...>

[naviserver-devel] small bugs break Windows build

[Andrew P. <at...@pi...>]

There are two small recent bugs that break compilation on Windows.

First, this 2019-07-16 commit has a simple typo in the Windows-only
code, "masedkBits" instead of "maskedBits":

  https://bitbucket.org/naviserver/naviserver/commits/65a47eea3818e515f235cebc59a385cea6e5cf06
  - New feature: context filter for urlspace

Second, the 2019-07-04 change to using "static inline bool" on the
Retry function causes the errors below.  Simply removing the "inline"
fixes the problem.  I don't know if there's a better way.

  https://bitbucket.org/naviserver/naviserver/commits/5a97670cb105330c645ff25ebe03c3eef8851799
  Define Retry as pure inline function

sock.c(76) : error C2054: expected '(' to follow 'inline'
sock.c(76) : error C2085: 'Retry' : not in formal parameter list
sock.c(78) : error C2085: 'CloseLater' : not in formal parameter list
sock.c(101) : error C2082: redefinition of formal parameter 'inline'
sock.c(101) : error C2146: syntax error : missing ',' before identifier ''
sock.c(101) : error C2146: syntax error : missing ',' before identifier ''
sock.c(101) : error C2143: syntax error : missing ';' before '('
sock.c(101) : error C2059: syntax error : ')'
sock.c(355) : warning C4013: 'Retry' undefined; assuming extern returning int
sock.c(665) : warning C4133: 'function' : incompatible types - from 'int *' to 'char *'
sock.c(1583) : warning C4133: 'function' : incompatible types - from 'int *' to 'char *'
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\Bin\amd64\cl.EXE"' : return code '0x2'
Stop.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] head http.test failures

[Gustaf N. <ne...@wu...>]

The noted test failures in http.test are not based on broken nsd 
behavior, but are false positives due to the move from low-level I/O to 
ns_http in the regression test (makes test with http und https uniform). 
the ns_http encoding issues still have to be addressed, but the 
following change make these FAILED results go away and explain the 
change programmatically.

https://bitbucket.org/naviserver/naviserver/commits/bc4a94384a6f2d74587a2eded54706489fd694f7

all the best
-g

On 15.07.19 10:46, Andrew Piskorski wrote:
> Using the latest NaviServer head (last commit 2019-07-09) on Linux
> (Ubuntu 16.04.2 LTS) and Tcl 8.6.9 (built from source), I'm getting
> 'make test' failures that I don't understand, primarily in http.test.
> Can anyone advice, please?  Brief failure messages are below, full
> test output is attached.
>
>
> Tests ended at Mon Jul 15 04:23:22 EDT 2019
> all.tcl:	Total	1391	Passed	1380	Skipped	8	Failed	3
> Sourced 69 Test Files.
> Files with failing tests: http.test ns_server.test
>
> ==== http-3.3new limits: too many headers FAILED
> ==== Contents of test case:
>
>      nstest::http -http 1.1 -getbody 1  -setheaders [split [string repeat xy 1024] ""]  GET /limits
>
> ---- Result was:
> ns_http failed: recv failed
> ---- Result should have been (exact matching):
> 414
> ==== http-3.3new FAILED
>
> ==== http-5.5 check encoding ns_conn content POST, binary FAILED
> ==== Contents of test case:
>
>      set string "Testing <äöüß>"
>      nstest::http -getbody 1 -setheaders {content-type text/html} POST /post $string
>
> ---- Result was:
> 200 {utf-8 <text/html> AÄATesting <äöüß>ZÜZ}
> ---- Result should have been (exact matching):
> 200 {utf-8 <text/html> AÄATesting <äöüß>ZÜZ}
> ==== http-5.5 FAILED
>
> ==== ns_server-2.5 basic operation FAILED
> ---- Result was:
> 10
> ---- Result should have been (exact matching):
> 9
> ==== ns_server-2.5 FAILED

Re: [naviserver-devel] head http.test failures

[Zoran V. <zv...@ar...>]

On Mon, 15 Jul 2019 04:46:16 -0400
Andrew Piskorski <at...@pi...> wrote:

> Using the latest NaviServer head (last commit 2019-07-09) on Linux
> (Ubuntu 16.04.2 LTS) and Tcl 8.6.9 (built from source), I'm getting
> 'make test' failures that I don't understand, primarily in http.test.
> Can anyone advice, please?  Brief failure messages are below, full
> test output is attached.

This is most probably related to my last rewrites. I am currently
on that place, plugging in the chunked encoding parser so I will
look into that. Unfortunately I have some issues with my eyes and 
am currently somehow limited in sight. This will take some time...

Cheers,
Zoran

[naviserver-devel] head http.test failures

[Andrew P. <at...@pi...>]

Using the latest NaviServer head (last commit 2019-07-09) on Linux
(Ubuntu 16.04.2 LTS) and Tcl 8.6.9 (built from source), I'm getting
'make test' failures that I don't understand, primarily in http.test.
Can anyone advice, please?  Brief failure messages are below, full
test output is attached.


Tests ended at Mon Jul 15 04:23:22 EDT 2019
all.tcl:	Total	1391	Passed	1380	Skipped	8	Failed	3
Sourced 69 Test Files.
Files with failing tests: http.test ns_server.test

==== http-3.3new limits: too many headers FAILED
==== Contents of test case:

    nstest::http -http 1.1 -getbody 1  -setheaders [split [string repeat xy 1024] ""]  GET /limits

---- Result was:
ns_http failed: recv failed
---- Result should have been (exact matching):
414
==== http-3.3new FAILED

==== http-5.5 check encoding ns_conn content POST, binary FAILED
==== Contents of test case:

    set string "Testing <äöüß>"
    nstest::http -getbody 1 -setheaders {content-type text/html} POST /post $string

---- Result was:
200 {utf-8 <text/html> AÄATesting <äöüß>ZÜZ}
---- Result should have been (exact matching):
200 {utf-8 <text/html> AÄATesting <äöüß>ZÜZ}
==== http-5.5 FAILED

==== ns_server-2.5 basic operation FAILED
---- Result was:
10
---- Result should have been (exact matching):
9
==== ns_server-2.5 FAILED

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] Tcl nonblocking file IO in NaviServer?

[Gustaf N. <ne...@wu...>]

i was a few days offline, so sorry for my
delayed reply.

> I am telling this because I just completed the ns_http
> rewrite that handles more/less similar issue w.r.t
> queued http reads/writes and I was using the same
> cut/splice technique to open a channel in one and
> read/write to it in the other thread.
>
> In order to keep this option open and not to clutter
> the Tcl API, the "ns_asyncwrite" should allow for
> 
>   ns_asyncwrite ?-channel chan | -fd fd? content
> 
> At some time, if needed, the writer can be expanded
> to understand Tcl channels as well.

> OK. Back-off. That would never work that way.

The problem with Tcl channels is that these (including
their buffer management) are per-tcl-interp structures,
and since different threads use different Tcl-interps
Tcl channels are not multi-thread friendly.

Therefore, the previously mentioned, Tcl-based background
writer used in OpenACS since many years is based on
a special Tcl thread (implemented via Tcl's libthread), where the
file is opened by sending the open command to this thread,
and the thread can do arbitrary Tcl file operations like
in single threaded applications. But, every "write" operation
from other threads triggers a send operation to this
Tcl-thread, which performs the operation. This works
very well within the previously mentioned limits.
But also in these cases, async write operations are
essentially fire-and-forget.

When looking on reusing NaviServer's AsyncWriter
thread, providing an interface on the numeric
file descriptors looks still the best option to me.
If we would  base the async-write operation
on a Tcl channel (as you mentioned earlier) this
would not work when multiple threads/Tcl
interps write to the same file, since the different
interps have different channels, fds, etc.. Furthermore
it make me shudder, when other Tcl operations
would happen on this Tcl file handle.

> As I see, this could only work for "fire-and-forget"
> cases were one is not interested in any result and
> where one is producing small amount of data to be
> written.
The whole thread was started with such an assumption.
I have implemented the (optional) AsyncWriterThread
in NaviServer exactly with these assumptions for disk-write
operations  back in 2013 - 4.99.5, we are using it on our
high traffic sites since this time without a single
problem (for the  access.log and error.log). Btw, the
AsyncWriterThread handles partial write operations.

But if ask me, if i would write life-critical information
this way, i would say that async disk writing would not
my first choice. OTOH, the C based implementation is
imho much more robust and simple than the Tcl base
approach I've implemented for OpenACS, so this is an
advantage. So, providing access from the Tcl level
makes sense.

But you are right, one should not raise expectations to
high on this, so using more specialized names is
probably better

    ns_asynclog_open
    ns_asynclog_write
    ns_asynclog_close

I think, it costed me less time to write this email
than to implement these wrapper calls, since the
base infrastructure is already in NaviServer.

Zoran, if you think, such naming measures are
not enough, i can make a few calls external and
implement these wrappers as a module (biggest
disadvantage: configuration and change management,
build and install scripts for projects using it....
such as OpenACS, etc.)

all the best

-g

Re: [naviserver-devel] Tcl nonblocking file IO in NaviServer?

[Zoran V. <zv...@ar...>]

On Fri, 7 Jun 2019 11:45:59 +0200
Zoran Vasiljevic <zv...@ar...> wrote:

> At some time, if needed, the writer can be expanded
> to understand Tcl channels as well.

OK. Back-off. That would never work that way.

Because the channel needs to be tossed back 
and forth, at lest 2 times. Once for the first
write and once for the last. The first case is
simple. The last isn't without user telling the
system that he will not write any more (which
requires additional "-last" option...)

But even with this above solved...

All this is also questionalble w.r.t feedback.
How do I know anything went into the file?
At some point one also has to introduce some
kind of limit as otherwise we can blow memory
by very fast producers and slow consumers...

As I see, this could only work for "fire-and-forget"
cases were one is not interested in any result and
where one is producing small amount of data to be
written. This is a highly specialized case, for
example, for feeding log files an similar. But not
for doing any kind of decisions based on the file
or content of it...
Now, being that specialized, one can also argue
if a more general-purpose solution is justifed.

Sorry for raining on the party, but it is good to
know what/when/why *in advance* before putting too
much work (or expectations) in the matter.

Re: [naviserver-devel] Tcl nonblocking file IO in NaviServer?

[Wolfgang W. <Wol...@di...>]

We could use this as well for special purpose logs, e.g. payment logs or 
for user deletions and pseudonymizations which conform to GDPR standards.

Am 07.06.19 um 11:25 schrieb Gustaf Neumann:
> On 06.06.19 23:09, Gustaf Neumann wrote:
>> Not sure, what your performance constraints are, but open/close
>> is pretty fast these days (measured under linux on openacs.org):
>>     % time {set F [open /tmp/foo a]; puts $F x; close $F} 1000
>>     55.333 microseconds per iteration
>>
>> if this is too slow for you, one can experiment with ramdisks,
>> and other already mentioned approaches.
>>
> i did some more experiments and implemented a "ns_asyncwrite"
> wrapper for NaviServer's internal function:
>      time {ns_asyncwrite 55 hello\n} 1000
>      0.957 microseconds per iteration
> This is pretty cool. Since the AsyncWriterThread is build around
> a queue, this is also protected against concurrent write() operations.
> this means, one can do ~1mio simple write operations per second
> in the thread protected way, independent from file system blocks
> (measured on the same machine with the same load as above).
>
> The only disadvantage is that one needs commands for opening/closing
> based on the numeric file descriptors (not on Tcl file handles), which
> means at the end small wrappers on top of NaviServer's 
> ns_write()/ns_close().
> Working with the numeric file descriptors has the advantage that it can be
> passed around between interpreters/threads, etc., so it is well suited
> for multi threaded applications, the FDs of open files can be kept
> in nsvs, etc.).
>
> I see also potentials for this in some of our projects (e.g. writing 
> application
> specific log-files e.g. for ingest in elasticsearch, etc.).
>
> If nobody objects, i'll try to bring this to a state for committing 
> this the next
> days.
>
> -g
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
-- 

*Wolfgang Winkler*
Geschäftsführung
wol...@di...
mobil +43.699.19971172

dc:*büro*
digital concepts Novak Winkler OG
Software & Design
Landstraße 68, 5. Stock, 4020 Linz
www.digital-concepts.com <http://www.digital-concepts.com>
tel +43.732.997117.72
tel +43.699.1997117.2

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz