naviserver-devel — Developer discussion

Re: [naviserver-devel] commit/naviserver: gustafn: - switch to the notation of Tcl_ObjCmdProc and Tcl_CmdProc as defined in tcl.h in Tcl 8.5.*

[Gustaf N. <ne...@wu...>]

> Is this going to break Tcl8.4 compatibility?

no, tcl 8.4 has exactly the same definitions.
-g

Re: [naviserver-devel] commit/naviserver: gustafn: - switch to the notation of Tcl_ObjCmdProc and Tcl_CmdProc as defined in tcl.h in Tcl 8.5.*

[Zoran V. <zv...@ar...>]

Hi!

Is this going to break Tcl8.4 compatibility?

Cheers,
Zoran

On 11 Oct 2014, at 16:20, Bitbucket <com...@bi...> wrote:

> 1 new commit in naviserver:
> 
> https://bitbucket.org/naviserver/naviserver/commits/cb29e6963cd5/
> Changeset:   cb29e6963cd5
> User:        gustafn
> Date:        2014-10-11 14:20:16+00:00
> Summary:     - switch to the notation of Tcl_ObjCmdProc and Tcl_CmdProc as defined in tcl.h in Tcl 8.5.*
> Affected #:  54 files
> 
> Repository URL: https://bitbucket.org/naviserver/naviserver/

Re: [naviserver-devel] Naviserver hangs on Windows

[Gustaf N. <ne...@wu...>]

On 09.10.14 02:16, Andrew Piskorski wrote:
> So clearly we do NOT need to make Ns_Time.sec a time_t.
that's what i've said :)
> Separately,
> I've asked tcl...@li... why it is a long in Tcl.
another practical reason in addition to Joe's answer is 
binary compatibility
for extensions.

-gn

Re: [naviserver-devel] Naviserver hangs on Windows

[Gustaf N. <ne...@wu...>]

On 08.10.14 20:25, Gustaf Neumann wrote:
> I've revived an old installation of win7 based on VirtualBox and Visual
> Studio 11,
> used the version of naviserver from bitbucket, compiled tcl, switched
> the line for the win-makefile, compiled nsd using nmake, and to my big
> surprise, "nsd,exe -c" worked!
did some more testing with nsd.exe under win7-64 (version 
from the main trunk).
Starting nsd with the standard nsd-config file works nicely 
(i just had to
replace .so by .dll for nscp, nssock, nslog, nscgi, and 
nsdb), the modules are
loading fine, browsing through the documentation via the 
windows binary
works nice. At least the basic functionality works without 
any problems.

The only quirks is currently the installation, i had to 
create the install directories
by hand and had to copy much of the content there (i hope 
for a better "make install"
from Andrew's version).

-g
PS: Btw, i am using now the windows-native version of 
NS_GetTime (using 2x long)
and mutex-timings turned on. Since Andrew uses this as well 
on 32bit-windows,
we should be able to use this in the main naviserver branch 
as well.

Re: [naviserver-devel] Naviserver hangs on Windows

[Andrew P. <at...@pi...>]

Gustaf, my fork should be suitable for merging back into
naviserver/naviserver now.

On Wed, Oct 08, 2014 at 09:32:56AM +0200, Gustaf Neumann wrote:

> So at least, when using Tcl_GetTime(),
> which returns the time as Tcl_Time, everything should be
> fine. Note, that in your native time implementation, you
> assign to timePtr->sec with (time_t), which might be
> part of the problem you are experiencing.

Your bugfix aecf84685138 is of course the the key change:

  https://bitbucket.org/naviserver/naviserver/commits/aecf8468513892cb6dc241f09a9e8ad2dcd57793

Now with that, the only remaining problem is using Ns_GetTimeFromTcl()
(the "platform-independent" approach) while mutex timings are turned
on.  Using Ns_Time.sec as long vs. time_t made no difference for that
problem, which makes sense given your explanation of the problems of
calling Tcl from inside DllMain().

All other combinations appear to work fine, regardless of whether
Ns_Time.sec is long or time_t.

To summarize, we've explored these four binary state changes on 64-bit
Windows 7:

1. LogTime() long/time_t arg passing bug (aecf846) Fixed vs. Broken.
2. Mutex timing ON vs. OFF.
3. Ns_Time.sec is type long vs. time_t.
4. Ns_GetTime() uses Ns_GetTimeFromTcl() vs. Ns_GetTimeFromWindowsFileTime().

Some of the combinations of those states or of interest for deciding
what we should do with the Naviserver codebase.  We want to know if
Naviserver seems (at least trivially) to run ok, vs. when its behavior
is clearly very bad.

Below, I will write each combination of states as a four-tuple, e.g.
"Broken,  On, long, FromTcl", with the result (Ok or Bad) to the left.

In earlier emails, I covered these combinations:

Bad:  Broken,  On, long, FromTcl
   Immediate hang on startup.
Bad:  Broken,  On, time_t, FromTcl
   Immediate hang on startup.
Bad:  Broken, Off, long, FromWindowsFileTime
   Slightly later failure during startup, EINVAL from localtime_s().
 Ok:  Broken, Off, time_t, FromTcl
 Ok:  Broken,  On, time_t, FromWindowsFileTime

Today I (manually) tested these combinations and found that:

 Ok:  Fixed, Off, time_t, FromWindowsFileTime
 Ok:  Fixed,  On, time_t, FromWindowsFileTime
 Ok:  Fixed, Off, time_t, FromTcl
 Ok:  Fixed,  On, time_t, FromTcl
 Ok:  Fixed, Off, long, FromWindowsFileTime
 Ok:  Fixed,  On, long, FromWindowsFileTime
 Ok:  Fixed, Off, long, FromTcl:
Bad:  Fixed,  On, long, FromTcl
  Immediate hang on startup.

So clearly we do NOT need to make Ns_Time.sec a time_t.  Separately,
I've asked tcl...@li... why it is a long in Tcl.

In Ns_GetTime(), if we use Ns_GetTimeFromWindowsFileTime(), we can
turn on mutex timing if we want.  If we instead use Ns_GetTimeFromTcl()
we must keep mutex timing turned off.  Since Gustaf seems to prefer
Ns_GetTimeFromTcl() over mutex timing, I've gone with that combination
in my fork as well.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] Naviserver hangs on Windows

[Gustaf N. <ne...@wu...>]

yes, it might be a good time. as wrote in the other thread,
i was able to compile an start nsd with very little windows
experience. Be aware, that runnning "nsd.exe -c" does
not mean that all of nsd is really running.

-g
PS: my own time budget will be reduced significantly
for the next weeks.

Am 08.10.14 18:50, schrieb Maurizio Martignano:
> Dear Andrew and Gustaf,
> 	In due time, when you both are confident on the status of Naviserver
> on Windows, I would try it inside my Windows-OpenACS distribution.
>
> First of all I would try it in few installations running on the machines I
> have in my offices, then once I see it they are running fine and everything
> I have now still behave the same, I would make the new version of the
> distribution available.
>
> Is that OK with you?
>
> Maurizio
>
>
> -----Original Message-----
> From: Andrew Piskorski [mailto:at...@pi...]
> Sent: 08 October 2014 18:39
> To: nav...@li...
> Subject: Re: [naviserver-devel] Naviserver hangs on Windows
>
> On Wed, Oct 08, 2014 at 09:32:56AM +0200, Gustaf Neumann wrote:
>
>> the advantage of approach 2 (disable mutex timings, use )  is that
>> this works for 32bit and 64 bit, while (1) works most likely only for
>> 64bit (if i look at the constant).
> No, I'm confident that the approach using EPOCH_BIAS works just fine on
> 32-bit Windows, because in AOLserver 4.0.7 and 4.0.8 that is the ONLY
> implementation of Ns_GetTime on Windows, and I ran those versions of
> AOLserver for years on Windows XP 32-bit.
>
> Calls to Ns_GetTime() are very common, so AFAICT, for those AOLservers to
> work at all, that EPOCH_BIAS code had to be working fine on 32-bit Windows
> XP.  The only other alternative is that AOLserver was somehow magically
> calling an entirely different function instead of Ns_GetTime(), which I
> judge as very unlikely.
>
>> The mutex timings is an additional feature in naviserver, so at least
>> for the time being, one can life without that.... and with the deeper
>> knowledge, we might be able to activate this later again.
> Ah, I didn't realize that.  I should turn them back off for Windows in my
> fork then?
>
>> The problem seems to be: DllMain() calls certain functions before the
>> normal entry point main() is called, which initializes tcl etc. The
>> Windows man page [1] says:
> Interesting.  And of course DllMain() only exists on Windows.  But, why do
> we need DllMain() at all?  I assume we DO need it of course, I just don't
> understand the design differences between Naviserver's pthread and winthread
> support.  Both Posix and Windows use Nsthreads_LibInit(), but DllMain()
> seems to be an extra layer not present for Posix.
>
>>   > changing the Ns_Time.sec (in nsthread.h) to be time_t rather than
>> long  > fixes nearly everything.
>>
>> The reason, naviserver uses 2x long in Ns_Time is probably due to Tcl,
>> using:
> AFAICT, using long is just wrong, and Tcl and Naviserver both should have
> been using time_t for many years now.  I'm not sure when time_t was invented
> and standardized, but since then it's been what Unix and Windows both use.
>
> So what am I missing?  Why did the Tcl maintainers stick with long for
> Tcl_Time.sec?  It could just be a holdover from before time_t existed or
> really mattered, but perhaps they had a better for keeping it as a long.
>
>> So at least, when using Tcl_GetTime(), which returns the time as
>> Tcl_Time, everything should be fine. Note, that in your native time
>> implementation, you assign to timePtr->sec with (time_t), which might
>> be part of the problem you are experiencing.
> Could be.  I will try the no-mutex-timings and Ns_GetTimeFromTcl() approach
> again, this time with Ns_Time.sec changed back to type long rather than
> time_t.
>
>> Probably, i should get a windows installation running, but the setup
>> costs are quite high in terms of time and lack on windows-knowlege on
>> my side.
> Yeah.  Maybe the Docker stuff Stephen showed us would make it easier?
>
> Definitely see the notes I sent a while back on exactly what Microsoft
> software to install.  All the main stuff you need is available for free but
> figuring out WHAT you're supposed to install is not easy.
>
> Btw, if you do go that route, be aware that I am using
>    nmake -f Makefile.win32
> for building and cleaning, but NOT for installing Naviserver.  I have not
> debugged the install commands of those two *.win32 makefiles at all, because
> I have my own old Tcl script ("win32-install-nsd.tcl") for intalling on
> Windows, which I was able to quickly adapt to Naviserver.
>
> If you want that install script let me know.  I haven't added it to
> Mercurial yet because it is in CVS, and I'd like to nicely retain its full
> history it, but haven't looked up how to best do that yet.
>
> --
> Andrew Piskorski <at...@pi...>
>

Re: [naviserver-devel] Naviserver hangs on Windows

[Gustaf N. <ne...@wu...>]

>> The problem seems to be: DllMain() calls certain functions
>> before the normal entry point main() is called, which initializes
>> tcl etc. The Windows man page [1] says:
> Interesting.  And of course DllMain() only exists on Windows.  But,
> why do we need DllMain() at all?  I assume we DO need it of course, I
> just don't understand the design differences between Naviserver's
> pthread and winthread support.  Both Posix and Windows use
> Nsthreads_LibInit(), but DllMain() seems to be an extra layer not
> present for Posix.
The porpose of DllMain() seems to be mostly for handling tls. To answer
why this was done this way requires a windows guru or a longer time of
investigation.
>> The reason, naviserver uses 2x long in Ns_Time is probably due to
>> Tcl, using:
> AFAICT, using long is just wrong, and Tcl and Naviserver both should
> have been using time_t for many years now.
we can't change tcl, it is important that tcl and naviserver have the same
understanding of time, unless we have a really good reason for
changing this.
>> Probably, i should get a windows installation running, but
>> the setup costs are quite high in terms of time and lack
>> on windows-knowlege on my side.
> Yeah.  Maybe the Docker stuff Stephen showed us would make it easier?
I've revived an old installation of win7 based on VirtualBox and Visual 
Studio 11,
used the version of naviserver from bitbucket, compiled tcl, switched
the line for the win-makefile, compiled nsd using nmake, and to my big
surprise, "nsd,exe -c" worked!

Well, it complained about a missing init.tcl, probably my installation 
needs more
tweaks, and i am sure, many more problems will show up.

-g
PS: btw, i've fixed the shift problem this moring.

Re: [naviserver-devel] Naviserver hangs on Windows

[Andrew P. <at...@pi...>]

On Wed, Oct 08, 2014 at 06:50:35PM +0200, Maurizio Martignano wrote:
> Dear Andrew and Gustaf,
> 	In due time, when you both are confident on the status of Naviserver
> on Windows, I would try it inside my Windows-OpenACS distribution.

That sounds excellent to me.  My own use of Naviserver on Windows will
be in a rather narrow application (not OpenACS; we use Linux for
that), so it would be great to see it tested on a wider range of
workloads.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] Naviserver hangs on Windows

[Maurizio M. <Mau...@sp...>]

Dear Andrew and Gustaf,
	In due time, when you both are confident on the status of Naviserver
on Windows, I would try it inside my Windows-OpenACS distribution.

First of all I would try it in few installations running on the machines I
have in my offices, then once I see it they are running fine and everything
I have now still behave the same, I would make the new version of the
distribution available.

Is that OK with you?

Maurizio


-----Original Message-----
From: Andrew Piskorski [mailto:at...@pi...] 
Sent: 08 October 2014 18:39
To: nav...@li...
Subject: Re: [naviserver-devel] Naviserver hangs on Windows

On Wed, Oct 08, 2014 at 09:32:56AM +0200, Gustaf Neumann wrote:

> the advantage of approach 2 (disable mutex timings, use )  is that 
> this works for 32bit and 64 bit, while (1) works most likely only for 
> 64bit (if i look at the constant).

No, I'm confident that the approach using EPOCH_BIAS works just fine on
32-bit Windows, because in AOLserver 4.0.7 and 4.0.8 that is the ONLY
implementation of Ns_GetTime on Windows, and I ran those versions of
AOLserver for years on Windows XP 32-bit.

Calls to Ns_GetTime() are very common, so AFAICT, for those AOLservers to
work at all, that EPOCH_BIAS code had to be working fine on 32-bit Windows
XP.  The only other alternative is that AOLserver was somehow magically
calling an entirely different function instead of Ns_GetTime(), which I
judge as very unlikely.

> The mutex timings is an additional feature in naviserver, so at least 
> for the time being, one can life without that.... and with the deeper 
> knowledge, we might be able to activate this later again.

Ah, I didn't realize that.  I should turn them back off for Windows in my
fork then?

> The problem seems to be: DllMain() calls certain functions before the 
> normal entry point main() is called, which initializes tcl etc. The 
> Windows man page [1] says:

Interesting.  And of course DllMain() only exists on Windows.  But, why do
we need DllMain() at all?  I assume we DO need it of course, I just don't
understand the design differences between Naviserver's pthread and winthread
support.  Both Posix and Windows use Nsthreads_LibInit(), but DllMain()
seems to be an extra layer not present for Posix.

>  > changing the Ns_Time.sec (in nsthread.h) to be time_t rather than 
> long  > fixes nearly everything.
> 
> The reason, naviserver uses 2x long in Ns_Time is probably due to Tcl, 
> using:

AFAICT, using long is just wrong, and Tcl and Naviserver both should have
been using time_t for many years now.  I'm not sure when time_t was invented
and standardized, but since then it's been what Unix and Windows both use.

So what am I missing?  Why did the Tcl maintainers stick with long for
Tcl_Time.sec?  It could just be a holdover from before time_t existed or
really mattered, but perhaps they had a better for keeping it as a long.

> So at least, when using Tcl_GetTime(), which returns the time as 
> Tcl_Time, everything should be fine. Note, that in your native time 
> implementation, you assign to timePtr->sec with (time_t), which might 
> be part of the problem you are experiencing.

Could be.  I will try the no-mutex-timings and Ns_GetTimeFromTcl() approach
again, this time with Ns_Time.sec changed back to type long rather than
time_t.

> Probably, i should get a windows installation running, but the setup 
> costs are quite high in terms of time and lack on windows-knowlege on 
> my side.

Yeah.  Maybe the Docker stuff Stephen showed us would make it easier?

Definitely see the notes I sent a while back on exactly what Microsoft
software to install.  All the main stuff you need is available for free but
figuring out WHAT you're supposed to install is not easy.

Btw, if you do go that route, be aware that I am using
  nmake -f Makefile.win32
for building and cleaning, but NOT for installing Naviserver.  I have not
debugged the install commands of those two *.win32 makefiles at all, because
I have my own old Tcl script ("win32-install-nsd.tcl") for intalling on
Windows, which I was able to quickly adapt to Naviserver.

If you want that install script let me know.  I haven't added it to
Mercurial yet because it is in CVS, and I'd like to nicely retain its full
history it, but haven't looked up how to best do that yet.

--
Andrew Piskorski <at...@pi...>

----------------------------------------------------------------------------
--
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI
DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you
Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI
DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Naviserver hangs on Windows

[Andrew P. <at...@pi...>]

On Wed, Oct 08, 2014 at 09:32:56AM +0200, Gustaf Neumann wrote:

> the advantage of approach 2 (disable mutex timings, use )  is that this 
> works for 32bit and 64 bit, while (1)
> works most likely only for 64bit (if i look at the constant).

No, I'm confident that the approach using EPOCH_BIAS works just fine
on 32-bit Windows, because in AOLserver 4.0.7 and 4.0.8 that is the
ONLY implementation of Ns_GetTime on Windows, and I ran those versions
of AOLserver for years on Windows XP 32-bit.

Calls to Ns_GetTime() are very common, so AFAICT, for those AOLservers
to work at all, that EPOCH_BIAS code had to be working fine on 32-bit
Windows XP.  The only other alternative is that AOLserver was somehow
magically calling an entirely different function instead of
Ns_GetTime(), which I judge as very unlikely.

> The mutex timings is an additional feature in naviserver, so at least
> for the time being, one can life without that.... and with the
> deeper knowledge, we might be able to activate this later again.

Ah, I didn't realize that.  I should turn them back off for Windows in
my fork then?

> The problem seems to be: DllMain() calls certain functions
> before the normal entry point main() is called, which initializes
> tcl etc. The Windows man page [1] says:

Interesting.  And of course DllMain() only exists on Windows.  But,
why do we need DllMain() at all?  I assume we DO need it of course, I
just don't understand the design differences between Naviserver's
pthread and winthread support.  Both Posix and Windows use
Nsthreads_LibInit(), but DllMain() seems to be an extra layer not
present for Posix.

>  > changing the Ns_Time.sec (in nsthread.h) to be time_t rather than long
>  > fixes nearly everything.
> 
> The reason, naviserver uses 2x long in Ns_Time is probably due to
> Tcl, using:

AFAICT, using long is just wrong, and Tcl and Naviserver both should
have been using time_t for many years now.  I'm not sure when time_t
was invented and standardized, but since then it's been what Unix and
Windows both use.

So what am I missing?  Why did the Tcl maintainers stick with long for
Tcl_Time.sec?  It could just be a holdover from before time_t existed
or really mattered, but perhaps they had a better for keeping it as a
long.

> So at least, when using Tcl_GetTime(),
> which returns the time as Tcl_Time, everything should be
> fine. Note, that in your native time implementation, you
> assign to timePtr->sec with (time_t), which might be
> part of the problem you are experiencing.

Could be.  I will try the no-mutex-timings and Ns_GetTimeFromTcl()
approach again, this time with Ns_Time.sec changed back to type long
rather than time_t.

> Probably, i should get a windows installation running, but
> the setup costs are quite high in terms of time and lack
> on windows-knowlege on my side.

Yeah.  Maybe the Docker stuff Stephen showed us would make it easier?

Definitely see the notes I sent a while back on exactly what Microsoft
software to install.  All the main stuff you need is available for
free but figuring out WHAT you're supposed to install is not easy.

Btw, if you do go that route, be aware that I am using
  nmake -f Makefile.win32
for building and cleaning, but NOT for installing Naviserver.  I have
not debugged the install commands of those two *.win32 makefiles at
all, because I have my own old Tcl script ("win32-install-nsd.tcl")
for intalling on Windows, which I was able to quickly adapt to
Naviserver.

If you want that install script let me know.  I haven't added it to
Mercurial yet because it is in CVS, and I'd like to nicely retain its
full history it, but haven't looked up how to best do that yet.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] warning, shift count undefined behavior

[Maurizio M. <Mau...@sp...>]

SonarQube (http://www.sonarqube.org) is "just" a web application displaying
the results of some analyses performed by other tools.

I am currently working on an Ada plugin and C/C++ plugin that I use for some
Verification and Validation activities.

For each language there might be a plugin. For each plugin there might be
the so called sensors (i.e. interfaces to external tools).

In the installation you see the C plugin is using two sensors: one for
CppCheck and one for PClint/FLEXELint (configured to check compliance with
MISRA C 2004).

Sometimes the warnings may be kind of misleading and "false positive", most
of the times they are common sense and it is worthwhile to check them.

I will keep the data in sync with Naviserver evolution.

Maurizio


-----Original Message-----
From: Andrew Piskorski [mailto:at...@pi...] 
Sent: 08 October 2014 17:34
To: nav...@li...
Subject: Re: [naviserver-devel] warning, shift count undefined behavior

On Wed, Oct 08, 2014 at 08:14:14AM +0200, Maurizio Martignano wrote:
> Dear Andrew,
> 	Yes, it is a potential issue.
> http://sonarsrv.spazioit.com:9000/dashboard/index?id=my%3Anaviserver%3
> Ansd%2

Neat!  Hm, it uses SonarQube, which I'd never heard of before.  What static
analyzer is ultimately being used to generate that info?

I couldn't seem to save a direct link to the shift/cast issue in
nsd/sockfile.c, but it is pretty easy to browse to it.

--
Andrew Piskorski <at...@pi...>

----------------------------------------------------------------------------
--
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI
DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you
Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI
DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] warning, shift count undefined behavior

[Andrew P. <at...@pi...>]

On Wed, Oct 08, 2014 at 08:14:14AM +0200, Maurizio Martignano wrote:
> Dear Andrew,
> 	Yes, it is a potential issue.
> http://sonarsrv.spazioit.com:9000/dashboard/index?id=my%3Anaviserver%3Ansd%2

Neat!  Hm, it uses SonarQube, which I'd never heard of before.  What
static analyzer is ultimately being used to generate that info?

I couldn't seem to save a direct link to the shift/cast issue in
nsd/sockfile.c, but it is pretty easy to browse to it.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] Naviserver hangs on Windows

[Andrew P. <at...@pi...>]

On Wed, Oct 08, 2014 at 08:05:42AM +0200, Maurizio Martignano wrote:

> I do agree with your remarks in your mail, I had a look at the Visual Studio
> Project Files you use, the only relevant differences I can see are the

I do not use any Visual Studio Project Files for Naviserver.  The ones
in Mercurial are old and I have no idea if they work or not.

The Naviserver Windows build I've been working on is controlled by the
makefiles, a combination of the two Windows-only Makefile.win32 for
nmake, and the individual module-specific makefiles which work in both
Gnu make and Microsoft nmake.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] Naviserver hangs on Windows

[Gustaf N. <ne...@wu...>]

> I have to look at this closer before i have a better seeing
> of consequences of changing the types in Ns_Time, i would
> certainly prefer to stick to the Tcl conventions, since we 
> depend
> strongly on tcl behavior all over the place.
There was a bug in log.c passing the argument to ns_localtime,
which shows up only on machines where sizeof(long) != 
sizeof(time_t).
This is obviously the case for the 64bit windows build.

With that fix, i would assume that the currently committed 
version
works now on windows (with mutex timings and 
win-platform-specific
Ns_GetTime deactivated) as far as the version with the modified
Ns_Time structure.

-g

Re: [naviserver-devel] Naviserver hangs on Windows

[Gustaf N. <ne...@wu...>]

Andrew, these are good news!

the advantage of approach 2 (disable mutex timings, use )  is that this 
works for 32bit and 64 bit, while (1)
works most likely only for 64bit (if i look at the constant). The
mutex timings is an additional feature in naviserver, so at least
for the time being, one can life without that.... and with the
deeper knowledge, we might be able to activate this later again.

The problem seems to be: DllMain() calls certain functions
before the normal entry point main() is called, which initializes
tcl etc. The Windows man page [1] says:

*   Warning*  There are serious limits on what you can do in a DLL entry 
point.

... not without a reason. It seems, that nn the windows side, one
has to be very careful, what is called inside these routines, since
all calls to Tcl* are potentially unsafe.

 > changing the Ns_Time.sec (in nsthread.h) to be time_t rather than long
 > fixes nearly everything.

The reason, naviserver uses 2x long in Ns_Time is probably due to
Tcl, using:

typedef struct Tcl_Time {
     long sec;                   /* Seconds. */
     long usec;                  /* Microseconds. */
} Tcl_Time;

So at least, when using Tcl_GetTime(),
which returns the time as Tcl_Time, everything should be
fine. Note, that in your native time implementation, you
assign to timePtr->sec with (time_t), which might be
part of the problem you are experiencing.

I have to look at this closer before i have a better seeing
of consequences of changing the types in Ns_Time, i would
certainly prefer to stick to the Tcl conventions, since we depend
strongly on tcl behavior all over the place.

Probably, i should get a windows installation running, but
the setup costs are quite high in terms of time and lack
on windows-knowlege on my side.

All together, things look already much better.
-g

[1] 
http://msdn.microsoft.com/en-us/library/windows/desktop/ms682583%28v=vs.85%29.aspx

Am 08.10.14 07:14, schrieb Andrew Piskorski:
> On Tue, Oct 07, 2014 at 12:09:41AM +0200, Gustaf Neumann wrote:
>
>> Please something more to try: to provide some potential evidence for
>> my "too early for tcl calls" hypothesis, i've deactivated for the
>> time being the mutex time monitoring for windows, since the earliest
>> calls are from mutex calls. Can you check when possible, whether
>> Tcl_GetTime() can be used now inside Ns_GetTime()?
> Gustaf, I think you were on to something there.
>
> In my fork I turned the Windows mutex timings back on, because they
> work now.  However, they ONLY work correctly when Ns_GetTime() is
> using the old-style AOLserver code with EPOCH_BIAS, etc.
>
> If I change Ns_GetTime() back to using the Ns_GetTimeFromTcl()
> "platform-independent" code, then the server immediately hangs on
> startup in TclpGetDate(), just like in my original email on 10/3.
>
> To summarize, there seem to be two combinations that work ok (so far):
>
> 1. Ns_GetTime() uses EPOCH_BIAS calculation, normal mutex timings in
>     nsthread/mutex.c turned on.
>
> 2. Ns_GetTime() uses Ns_GetTimeFromTcl(), mutex timings in
>     nsthread/mutex.c DISABLED.
>
> With either of those combinations, Naviserver will start up and then
> shut down, which is really all I've tested.
>


-- 
Univ.Prof. Dr. Gustaf Neumann
WU Vienna
Institute of Information Systems and New Media
Welthandelsplatz 1, A-1020 Vienna, Austria

Re: [naviserver-devel] warning, shift count undefined behavior

[Maurizio M. <Mau...@sp...>]

Dear Andrew,
	Yes, it is a potential issue.
http://sonarsrv.spazioit.com:9000/dashboard/index?id=my%3Anaviserver%3Ansd%2
Fsockfile.c
Maurizio



-----Original Message-----
From: Andrew Piskorski [mailto:at...@pi...] 
Sent: 08 October 2014 07:48
To: nav...@li...
Subject: [naviserver-devel] warning, shift count undefined behavior

Hm, this warning sounds kind of scary:

  sockfile.c(351) : warning C4293: '>>' : shift count negative or too big,
undefined behavior

That's in the Windows-only implementation of pread(), in nsd/sockfile.c:

  overlapped.Offset = (DWORD)offset;
  overlapped.OffsetHigh = ((DWORD)offset >> 32); 

I don't know if it's a real problem or how to fix it.

Some related docs:
http://msdn.microsoft.com/en-us/library/cx0bb1cy.aspx
http://msdn.microsoft.com/en-us/library/windows/desktop/ms684342%28v=vs.85%2
9.aspx

--
Andrew Piskorski <at...@pi...>

----------------------------------------------------------------------------
--
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI
DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you
Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI
DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Naviserver hangs on Windows

[Maurizio M. <Mau...@sp...>]

Dear Andrew,
	Sorry for the late reply, but have been away of my computer.

1. _USE_32BIT_TIME_T
Indeed, you are absolutely write. My Windows distribution had since 2011 to
2013 both Windows 32 and Windows 64 binaries.
I was using more or less the same make and configuration files to build it.
The define _USE_32BIT_TIME_T was used for the Windows 32 build but not for
the Windows 64. The current version of my distribution only contains Windows
64 binaries and in fact the define is commented out in the makefile.
Sorry for the time I made you waste on this point.

2. About the configuration
I am using the tools made available by Visual Studio 2013 Professional
Edition in the "VS2013 X64 Native Tools Command Prompt". That is:
CL: Microsoft (R) C/C++ Optimizing Compiler Version 18.00.21005.1 for x64
LINK: Microsoft (R) Incremental Linker Version 12.00.21005.1
This is what I use and your questions below:
COPTS	= /EHsc /W3 /nologo /c
DEFS	= $(DEFS) /D "_WINDOWS" /D "TCL_THREADS=1" /D "WIN32" /D "_WIN32" \
	  /D "FD_SETSIZE=128" /D "NO_CONST=1" /D "_MBCS" # /D
"_USE_32BIT_TIME_T"
LOPTS   = /NOLOGO /SUBSYSTEM:CONSOLE /OPT:NOREF /OPT:NOICF

Did you comment out your _USE_32BIT_TIME_T there?
[MM] Absolutely, I already explained what I did in point 1 and once again
apologies.
What does "/D _WIN32" do?
[MM] This should be set automatically by the compiler, so in theory it is
redundant... but one never knows...
What C run-time library are you using?
[MM] Visual C++ Redistributable Packages for Visual Studio 2013 (Release, No
Debug)
http://www.microsoft.com/en-us/download/details.aspx?id=40784

I do agree with your remarks in your mail, I had a look at the Visual Studio
Project Files you use, the only relevant differences I can see are the
following:
1. you are targeting as Machine_X64, I am targeting AMD64
2. you use Active TCL, I did compile from scratch also TCL, using the same
compiler options (I believe this is very important as we have no control on
how Active TCL is built)

Hope this helps,
Maurizio




-----Original Message-----
From: Andrew Piskorski [mailto:at...@pi...] 
Sent: 07 October 2014 22:36
To: nav...@li...
Subject: Re: [naviserver-devel] Naviserver hangs on Windows

On Mon, Oct 06, 2014 at 05:18:08PM -0400, Andrew Piskorski wrote:
> On Sun, Oct 05, 2014 at 09:37:39PM +0200, Maurizio Martignano wrote:
> > > Did you use the define _USE_32BIT_TIME_T yes or not?

> I haven't tried using _USE_32BIT_TIME_T yet, but I think using it 
> would be INCORRECT on Windows-64.

If I do pass the "/D _USE_32BIT_TIME_T" flag, I get this:

C:\Program Files (x86)\Microsoft Visual Studio
10.0\VC\INCLUDE\crtdefs.h(463) : fatal error C1189: #error :  You cannot use
32-bit time_t (_USE_32BIT_TIME_T) with _WIN64 

So that seems pretty clear, I must NOT set that flag.

--
Andrew Piskorski <at...@pi...>

----------------------------------------------------------------------------
--
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI
DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you
Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI
DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
naviserver-devel mailing list
nav...@li...
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] warning, shift count undefined behavior

[Andrew P. <at...@pi...>]

Hm, this warning sounds kind of scary:

  sockfile.c(351) : warning C4293: '>>' : shift count negative or too big, undefined behavior

That's in the Windows-only implementation of pread(), in nsd/sockfile.c:

  overlapped.Offset = (DWORD)offset; 
  overlapped.OffsetHigh = ((DWORD)offset >> 32); 

I don't know if it's a real problem or how to fix it.

Some related docs:
http://msdn.microsoft.com/en-us/library/cx0bb1cy.aspx
http://msdn.microsoft.com/en-us/library/windows/desktop/ms684342%28v=vs.85%29.aspx

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] Naviserver hangs on Windows

[Andrew P. <at...@pi...>]

On Tue, Oct 07, 2014 at 12:09:41AM +0200, Gustaf Neumann wrote:

> Please something more to try: to provide some potential evidence for
> my "too early for tcl calls" hypothesis, i've deactivated for the
> time being the mutex time monitoring for windows, since the earliest
> calls are from mutex calls. Can you check when possible, whether
> Tcl_GetTime() can be used now inside Ns_GetTime()?

Gustaf, I think you were on to something there.

In my fork I turned the Windows mutex timings back on, because they
work now.  However, they ONLY work correctly when Ns_GetTime() is
using the old-style AOLserver code with EPOCH_BIAS, etc.

If I change Ns_GetTime() back to using the Ns_GetTimeFromTcl()
"platform-independent" code, then the server immediately hangs on
startup in TclpGetDate(), just like in my original email on 10/3.

To summarize, there seem to be two combinations that work ok (so far):

1. Ns_GetTime() uses EPOCH_BIAS calculation, normal mutex timings in
   nsthread/mutex.c turned on.

2. Ns_GetTime() uses Ns_GetTimeFromTcl(), mutex timings in
   nsthread/mutex.c DISABLED.

With either of those combinations, Naviserver will start up and then
shut down, which is really all I've tested.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] Naviserver hangs on Windows

[Andrew P. <at...@pi...>]

On Tue, Oct 07, 2014 at 05:02:41PM -0400, Andrew Piskorski wrote:

> And yet it still fails in exactly the same way, with EINVAL = 22.
> This makes me think there is nothing wrong with the tlsPtr code or
> data, rather something is seriously broken in my build.

I think my analysis above was entirely wrong.  I now see evidence that
the time_t value going into ns_localtime() is incorrect after all,
likely due to some sort of type mismatch.

And, after a very small amount of testing, it looks like simply
changing the Ns_Time.sec (in nsthread.h) to be time_t rather than long
fixes nearly everything.  Those changs are here:

  https://bitbucket.org/apiskors/naviserver/commits/all

Since I'd already written it earlier when I was more confused, here's
the slow, convoluted path I took to reach that conclusion:

I was stuck, so in a stand-alone C program, I confirmed that with a
64-bit time_t (which is the default), by casting to long long I can
correctly display the integer Unix time regardless of its particular
value, like so:

  time_t clock;
  //time(&clock);  /* Now */
  //clock = 1412722229; /* Oct 2014 */
  clock = 2174774400;  /* Nov 2038*/
  printf("clock lld:  %lld\n", (long long)clock);

Note that Nov. 2038 requires a 64-bit time_t, it is too big for
32-bit.

With both those integers above, printf() displays the same value I
hard-coded into my code.  If I use other format specifiers with
printf, the displayed integer can change completely depending on the
itns value, but by casting to long long and using %lld I seem to
always get the same output as my input.

Next, I changed ns_localtime() to use the same constant, hard-coded
time_t *clock value of 2174774400.  That worked!  The EINVAL failure
went away, and I was able to start up and shut down Naviserver normally!

Or if Inside ns_localtime() I completely ignore the incoming clock
pointer and instead just call time() to get the current time myself,
that works too!

Finally, I restored ns_localtime to its normal behavior, but left in a
debugging printf, like this:

  errNum = localtime_s(&tlsPtr->ltbuf, clock);
  printf("\nDebug:atp: ns_localtime: errNum from localtime_s:  %d", errNum);
  printf("\n  clock lld:  %lld", (long long)*clock);
  printf("\n  clock  ld:  %ld" , (long)*clock);
  printf("\n  clock   d:  %d\n", (int)*clock);

And here's the corresponding output from starting up the server twice,
which immediately aborts:

Debug:atp: ns_localtime: errNum from localtime_s:  22
  clock lld:  2996724649205424
  clock  ld:  1412734640
  clock   d:  1412734640
[3280.d64][-main-] Notice: nsmain: NaviServer/4.99 starting
[3280.d64][-main-] Fatal: nsthreads: localtime_s failed in ns_localtime: win32 err: 22

Debug:atp: ns_localtime: errNum from localtime_s:  22
  clock lld:  435292053216956
  clock  ld:  1412734652
  clock   d:  1412734652
[2892.58c][-main-] Notice: nsmain: NaviServer/4.99 starting
[2892.58c][-main-] Fatal: nsthreads: localtime_s failed in ns_localtime: win32 err: 22

$ tclsh8.5
% clock format 435292053216956
Sat Dec 29 13:55:56 EST 2036604
% clock format 1412734640 
Tue Oct 07 22:17:20 EDT 2014

Those huge 15-digits integers are VERY far in the future, and they
change seemingly arbitrarily each time I run nsd.  But their shorter
integer representation is the correct time!  Thus my earlier
confusion, because I was calling printf with %i or %d, and only seeing
the seemingly-correct short value.

Now, Tcl can process the year 2,036,604 just fine, so I'm not sure why
the Windows localtime_s() is throwing EINVAL on those inputs.  But
those Unix time values sure seem suspect...  And that lead me to the
definition of Ns_Time.  (Which I now recall wondering about days or
weeks ago, but then forgot about.  Ugh.)

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] Naviserver hangs on Windows

[Andrew P. <at...@pi...>]

On Tue, Oct 07, 2014 at 04:09:02PM -0400, Andrew Piskorski wrote:

> C:\> C:\web\nsd-atp\bin\nsd -c
> [384.e14][-main-] Notice: nsmain: NaviServer/4.99 starting
> [384.e14][-main-] Fatal: nsthreads: localtime_s failed in ns_localtime: win32 err: 22

In ns_localtime(), I changed this call:
    errNum = localtime_s(&tlsPtr->ltbuf, clock); 
to instead do this:
    struct tm foo; 
    errNum = localtime_s(&foo, clock); 

And yet it still fails in exactly the same way, with EINVAL = 22.
This makes me think there is nothing wrong with the tlsPtr code or
data, rather something is seriously broken in my build.  (But I have
no idea what.)

Possibly of interest, is that _localtime64_s() fails in exactly the
same way.  _localtime32_s() apparently does not trigger EINVAL, but
instead fails on the same debug assertion, where the tm_mday value is
out of range (and most likely -1 just like before, although I did not
confirm that).

I increased my compiler warning level from /W1 to /W3, and now I see
LOTS of type warnings, e.g.:

  log.c(689) : warning C4133: 'function' : incompatible types - from 'long *' to 'const time_t *'
  nssock.c(377) : warning C4133: 'function' : incompatible types - from 'int *' to 'const char *'
  binder.c(324) : warning C4244: 'function' : conversion from 'SOCKET' to 'int', possible loss of data
  driver.c(3888) : warning C4244: '=' : conversion from 'SOCKET' to 'int', possible loss of data
  driver.c(2356) : warning C4244: 'function' : conversion from 'ssize_t' to 'unsigned int', possible loss of data
  driver.c(3475) : warning C4244: '=' : conversion from 'ssize_t' to 'off_t', possible loss of data
  driver.c(3351) : warning C4244: 'function' : conversion from 'Tcl_WideInt' to 'long', possible loss of data
  driver.c(2346) : warning C4267: 'function' : conversion from 'size_t' to 'int', possible loss of data
  driver.c(3354) : warning C4267: 'function' : conversion from 'size_t' to 'unsigned int', possible loss of data
  tclhttp.c(371) : warning C4244: '=' : conversion from '__int64' to 'int', possible loss of data

That first one in log.c looks mildly suspicious.  But, on a 64-bit
machine like this "long" and "time_t" are both 64-bit integers, so I
don't see the problem.  Why are they "incompatible"?

  http://msdn.microsoft.com/en-us/library/323b6b3k%28v=vs.100%29.aspx

However, in include/nsthread.h, why is Ns_Time defined like this?:

  typedef struct Ns_Time { 
      long    sec; 
      long    usec; 
  } Ns_Time; 

Shouldn't the type of sec be time_t rather than long?

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] Naviserver hangs on Windows

[Andrew P. <at...@pi...>]

On Mon, Oct 06, 2014 at 05:18:08PM -0400, Andrew Piskorski wrote:
> On Sun, Oct 05, 2014 at 09:37:39PM +0200, Maurizio Martignano wrote:
> > > Did you use the define _USE_32BIT_TIME_T yes or not?

> I haven't tried using _USE_32BIT_TIME_T yet, but I think using it
> would be INCORRECT on Windows-64.

If I do pass the "/D _USE_32BIT_TIME_T" flag, I get this:

C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\INCLUDE\crtdefs.h(463) : fatal error C1189: #error :  You cannot use 32-bit time_t (_USE_32BIT_TIME_T) with _WIN64 

So that seems pretty clear, I must NOT set that flag.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] Naviserver hangs on Windows

[Andrew P. <at...@pi...>]

On Tue, Oct 07, 2014 at 12:09:41AM +0200, Gustaf Neumann wrote:

> Please something more to try: to provide some potential evidence for
> my "too early for tcl calls" hypothesis, i've deactivated for the
> time being the mutex time monitoring for windows, since the earliest
> calls are from mutex calls. Can you check when possible, whether
> Tcl_GetTime() can be used now inside Ns_GetTime()?

I merged in all your changes to my sandbox.  Your change deactivating
the mutex timings, here:

  https://bitbucket.org/naviserver/naviserver/commits/6ba02284c80242b3046fc7e8cddc63cc0677b2b5

definitely does make a difference - the server now gets slightly
farther and then exits:

  C:\> C:\web\nsd-atp\bin\nsd -c
  [384.e14][-main-] Notice: nsmain: NaviServer/4.99 starting
  [384.e14][-main-] Fatal: nsthreads: localtime_s failed in ns_localtime: win32 err: 22

  C:\> C:\web\nsd-atp\bin\nsd -f -t C:\web\nsd-atp\conf\nsd-config.tcl
  [3768.9b4][-main-] Notice: nsmain: enable progess statistics for uploads >= 1048576 bytes
  [3768.9b4][-main-] Fatal: nsthreads: localtime_s failed in ns_localtime: win32 err: 22

The (partial) backtrace from WinDbg looks like this:

  tcl85!Tcl_Panic
  nsthread!NsThreadFatal
  nsthread!ns_localtime
  nsd_7fef0060000!Ns_InfoNameOfExecutable
  nsd_7fef0060000!Ns_InfoNameOfExecutable
  nsd!main

So this looks like the same problem as before, just reached from a
later point in the Naviserver startup.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] Naviserver hangs on Windows

[Andrew P. <at...@pi...>]

On Fri, Oct 03, 2014 at 04:16:41PM +0200, Maurizio Martignano wrote:
> Dear Andrew,
> 	These are the basic options I would use for a Windows 64 bit, AMD64:
> 
> COPTS	= /EHsc /W3 /nologo /c
> DEFS	= $(DEFS) /D "_WINDOWS" /D "TCL_THREADS=1" /D "WIN32" /D "_WIN32" \
> 	  /D "FD_SETSIZE=128" /D "NO_CONST=1" /D "_MBCS" # /D "_USE_32BIT_TIME_T"
> LOPTS   = /NOLOGO /SUBSYSTEM:CONSOLE /OPT:NOREF /OPT:NOICF

Did you comment out your _USE_32BIT_TIME_T there?
What does "/D _WIN32" do?
What C run-time library are you using?

I've been able to find Microsoft docs on the various compiler and
linker flags, but NOT anything clear on what all the magic /D
preprocessor definitions actually do.

The nmake files I got from Ibrahim (at Archiware) use the "/MTd"
compiler flag, which should give me the debug version of the
"libcmtd.lib" CRT, as explained here:

  http://msdn.microsoft.com/en-us/library/abx4dbyh%28v=vs.100%29.aspx
  http://msdn.microsoft.com/en-us/library/2kzt1wy3%28v=vs.100%29.aspx

The other main option is /MDd for "msvcrtd.lib", but I don't know
which is the default when you pass no flags at all.

Most of our other build options are the same.  However, besides the
ones I mentioned above, I am also using these extra flags that you do
not have:

In COPTS:  /Zi /RTC1 
In  DEFS:  /D _CRT_SECURE_NO_WARNINGS /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D "_DEBUG" 
In LOPTS:  /DEBUG

AFAICT those all just enable additional debugging and/or run-time
checks, and so should be pretty safe.

-- 
Andrew Piskorski <at...@pi...>

Re: [naviserver-devel] Naviserver hangs on Windows

[Stephen <yo...@gr...>]

On Tue, Oct 7, 2014 at 12:11 PM, Gustaf Neumann <ne...@wu...> wrote:
>
> Stephen,
>
> the Docker image is cool stuff! Many thanks!
> This eases mingw builds significantly.
>
> It compiles now fine the first bunch of files in nsthread
> fine, but stops then with
>
> x86_64-w64-mingw32-gcc -shared -L../nsthread -L../nsd
> -L../nsdb -o libnsthread.dll error.o master.o memory.o
> mutex.o cslock.o rwlock.o reentrant.o sema.o thread.o tls.o
> time.o pthread.o fork.o signal.o winthread.o
> -L/usr/local/tcl8.6.2-mingw/lib -ltcl86 -lnetapi32
> -lkernel32 -luser32 -ladvapi32 -lws2_32
> -L/usr/local/tcl8.6.2-mingw/lib
> /usr/lib64/gcc/x86_64-w64-mingw32/4.8.3/../../../../x86_64-w64-mingw32/bin/ld:
> cannot find -ltcl86
> collect2: error: ld returned 1 exit status
> make[1]: *** [libnsthread.dll] Error 1
>
> Looks like the installation of tcl is missing...

When you compile tcl --with-symbols it adds a 'g' to the lib and
executabe names so I guess it needs to be something like: -ltcl86g.

If you scroll up to the end of the previously successful docker step
it should say something like:

    Now type `make' to compile.
    ---> 5c66713ca361

You can jump into the container with the file system as it was after
that step by typing something like:

    $ docker run -t -i 5c66713ca361 /bin/bash

...and poke around from there.