From: Gustaf N. <ne...@wu...> - 2014-10-08 18:29:35
|
yes, it might be a good time. as wrote in the other thread, i was able to compile an start nsd with very little windows experience. Be aware, that runnning "nsd.exe -c" does not mean that all of nsd is really running. -g PS: my own time budget will be reduced significantly for the next weeks. Am 08.10.14 18:50, schrieb Maurizio Martignano: > Dear Andrew and Gustaf, > In due time, when you both are confident on the status of Naviserver > on Windows, I would try it inside my Windows-OpenACS distribution. > > First of all I would try it in few installations running on the machines I > have in my offices, then once I see it they are running fine and everything > I have now still behave the same, I would make the new version of the > distribution available. > > Is that OK with you? > > Maurizio > > > -----Original Message----- > From: Andrew Piskorski [mailto:at...@pi...] > Sent: 08 October 2014 18:39 > To: nav...@li... > Subject: Re: [naviserver-devel] Naviserver hangs on Windows > > On Wed, Oct 08, 2014 at 09:32:56AM +0200, Gustaf Neumann wrote: > >> the advantage of approach 2 (disable mutex timings, use ) is that >> this works for 32bit and 64 bit, while (1) works most likely only for >> 64bit (if i look at the constant). > No, I'm confident that the approach using EPOCH_BIAS works just fine on > 32-bit Windows, because in AOLserver 4.0.7 and 4.0.8 that is the ONLY > implementation of Ns_GetTime on Windows, and I ran those versions of > AOLserver for years on Windows XP 32-bit. > > Calls to Ns_GetTime() are very common, so AFAICT, for those AOLservers to > work at all, that EPOCH_BIAS code had to be working fine on 32-bit Windows > XP. The only other alternative is that AOLserver was somehow magically > calling an entirely different function instead of Ns_GetTime(), which I > judge as very unlikely. > >> The mutex timings is an additional feature in naviserver, so at least >> for the time being, one can life without that.... and with the deeper >> knowledge, we might be able to activate this later again. > Ah, I didn't realize that. I should turn them back off for Windows in my > fork then? > >> The problem seems to be: DllMain() calls certain functions before the >> normal entry point main() is called, which initializes tcl etc. The >> Windows man page [1] says: > Interesting. And of course DllMain() only exists on Windows. But, why do > we need DllMain() at all? I assume we DO need it of course, I just don't > understand the design differences between Naviserver's pthread and winthread > support. Both Posix and Windows use Nsthreads_LibInit(), but DllMain() > seems to be an extra layer not present for Posix. > >> > changing the Ns_Time.sec (in nsthread.h) to be time_t rather than >> long > fixes nearly everything. >> >> The reason, naviserver uses 2x long in Ns_Time is probably due to Tcl, >> using: > AFAICT, using long is just wrong, and Tcl and Naviserver both should have > been using time_t for many years now. I'm not sure when time_t was invented > and standardized, but since then it's been what Unix and Windows both use. > > So what am I missing? Why did the Tcl maintainers stick with long for > Tcl_Time.sec? It could just be a holdover from before time_t existed or > really mattered, but perhaps they had a better for keeping it as a long. > >> So at least, when using Tcl_GetTime(), which returns the time as >> Tcl_Time, everything should be fine. Note, that in your native time >> implementation, you assign to timePtr->sec with (time_t), which might >> be part of the problem you are experiencing. > Could be. I will try the no-mutex-timings and Ns_GetTimeFromTcl() approach > again, this time with Ns_Time.sec changed back to type long rather than > time_t. > >> Probably, i should get a windows installation running, but the setup >> costs are quite high in terms of time and lack on windows-knowlege on >> my side. > Yeah. Maybe the Docker stuff Stephen showed us would make it easier? > > Definitely see the notes I sent a while back on exactly what Microsoft > software to install. All the main stuff you need is available for free but > figuring out WHAT you're supposed to install is not easy. > > Btw, if you do go that route, be aware that I am using > nmake -f Makefile.win32 > for building and cleaning, but NOT for installing Naviserver. I have not > debugged the install commands of those two *.win32 makefiles at all, because > I have my own old Tcl script ("win32-install-nsd.tcl") for intalling on > Windows, which I was able to quickly adapt to Naviserver. > > If you want that install script let me know. I haven't added it to > Mercurial yet because it is in CVS, and I'd like to nicely retain its full > history it, but haven't looked up how to best do that yet. > > -- > Andrew Piskorski <at...@pi...> > |
From: Gustaf N. <ne...@wu...> - 2014-10-08 18:25:47
|
>> The problem seems to be: DllMain() calls certain functions >> before the normal entry point main() is called, which initializes >> tcl etc. The Windows man page [1] says: > Interesting. And of course DllMain() only exists on Windows. But, > why do we need DllMain() at all? I assume we DO need it of course, I > just don't understand the design differences between Naviserver's > pthread and winthread support. Both Posix and Windows use > Nsthreads_LibInit(), but DllMain() seems to be an extra layer not > present for Posix. The porpose of DllMain() seems to be mostly for handling tls. To answer why this was done this way requires a windows guru or a longer time of investigation. >> The reason, naviserver uses 2x long in Ns_Time is probably due to >> Tcl, using: > AFAICT, using long is just wrong, and Tcl and Naviserver both should > have been using time_t for many years now. we can't change tcl, it is important that tcl and naviserver have the same understanding of time, unless we have a really good reason for changing this. >> Probably, i should get a windows installation running, but >> the setup costs are quite high in terms of time and lack >> on windows-knowlege on my side. > Yeah. Maybe the Docker stuff Stephen showed us would make it easier? I've revived an old installation of win7 based on VirtualBox and Visual Studio 11, used the version of naviserver from bitbucket, compiled tcl, switched the line for the win-makefile, compiled nsd using nmake, and to my big surprise, "nsd,exe -c" worked! Well, it complained about a missing init.tcl, probably my installation needs more tweaks, and i am sure, many more problems will show up. -g PS: btw, i've fixed the shift problem this moring. |
From: Gustaf N. <ne...@wu...> - 2014-10-09 11:46:30
|
On 08.10.14 20:25, Gustaf Neumann wrote: > I've revived an old installation of win7 based on VirtualBox and Visual > Studio 11, > used the version of naviserver from bitbucket, compiled tcl, switched > the line for the win-makefile, compiled nsd using nmake, and to my big > surprise, "nsd,exe -c" worked! did some more testing with nsd.exe under win7-64 (version from the main trunk). Starting nsd with the standard nsd-config file works nicely (i just had to replace .so by .dll for nscp, nssock, nslog, nscgi, and nsdb), the modules are loading fine, browsing through the documentation via the windows binary works nice. At least the basic functionality works without any problems. The only quirks is currently the installation, i had to create the install directories by hand and had to copy much of the content there (i hope for a better "make install" from Andrew's version). -g PS: Btw, i am using now the windows-native version of NS_GetTime (using 2x long) and mutex-timings turned on. Since Andrew uses this as well on 32bit-windows, we should be able to use this in the main naviserver branch as well. |
From: Stephen <yo...@gr...> - 2014-10-06 23:04:01
|
On Mon, Oct 6, 2014 at 10:31 PM, Gustaf Neumann <ne...@wu...> wrote: > > I've merged in your changes to the main branch. Here are some build errors while cross-compiling with mingw using this docker image: https://bitbucket.org/groks/build-naviserver-mingw/src/tip/naviserver/ ---> 0eaa9093aff9 Removing intermediate container 4d4d06c9d5b4 Step 8 : RUN PATH=/usr/x86_64-w64-mingw32/sys-root/mingw/bin:$PATH make all build-doc \ INSTALL_ROOT="/" STRINGS=/usr/bin/x86_64-w64-mingw32-strings OBJCOPY=/usr/bin/x86_64-w64-mingw32-objcopy NM=/usr/bin/x86_64-w64-mingw32-nm DLLWRAP=/usr/bin/x86_64-w64-mingw32-dllwrap GCC=/usr/bin/x86_64-w64-mingw32-gcc WINDRES=/usr/bin/x86_64-w64-mingw32-windres ADDR2LINE=/usr/bin/x86_64-w64-mingw32-addr2line AS=/usr/bin/x86_64-w64-mingw32-as PKG_CONFIG=/usr/bin/x86_64-w64-mingw32-pkg-config CPP=/usr/bin/x86_64-w64-mingw32-cpp ELFEDIT=/usr/bin/x86_64-w64-mingw32-elfedit HOST_CC=gcc LD=/usr/bin/x86_64-w64-mingw32-ld DLLTOOL=/usr/bin/x86_64-w64-mingw32-dlltool AR=/usr/bin/x86_64-w64-mingw32-ar SIZE=/usr/bin/x86_64-w64-mingw32-size WINDMC=/usr/bin/x86_64-w64-mingw32-windmc GPROF=/usr/bin/x86_64-w64-mingw32-gprof RANLIB=/usr/bin/x86_64-w64-mingw32-ranlib CXXFILT=/usr/bin/x86_64-w64-mingw32-c++filt GCOV=/usr/bin/x86_64-w64-mingw32-gcov CC=x86_64-w64-mingw32-gcc READELF=/usr/bin/x86_64-w64-mingw32-readelf OBJDUMP=/usr/bin/x86_64-w64-mingw32-objdump STRIP=/usr/bin/x86_64-w64-mingw32-strip LD_BFD=/usr/bin/x86_64-w64-mingw32-ld.bfd ---> Running in ed1e545df7f9 make[1]: Entering directory `/usr/local/src/naviserver-tip/nsthread' x86_64-w64-mingw32-gcc -g -O2 -fomit-frame-pointer -Wall -pipe -I../include -I"/usr/local/tcl8.6.2-mingw/include" -DHAVE_CONFIG_H -c -o error.o error.c In file included from thread.h:40:0, from error.c:37: ../include/nsthread.h:168:0: warning: "lseek" redefined [enabled by default] # define lseek _lseek ^ In file included from ../include/nsthread.h:107:0, from thread.h:40, from error.c:37: /usr/x86_64-w64-mingw32/sys-root/mingw/include/io.h:335:0: note: this is the location of the previous definition #define lseek lseek64 ^ x86_64-w64-mingw32-gcc -g -O2 -fomit-frame-pointer -Wall -pipe -I../include -I"/usr/local/tcl8.6.2-mingw/include" -DHAVE_CONFIG_H -c -o master.o master.c In file included from thread.h:40:0, from master.c:36: ../include/nsthread.h:168:0: warning: "lseek" redefined [enabled by default] # define lseek _lseek ^ In file included from ../include/nsthread.h:107:0, from thread.h:40, from master.c:36: /usr/x86_64-w64-mingw32/sys-root/mingw/include/io.h:335:0: note: this is the location of the previous definition #define lseek lseek64 ^ x86_64-w64-mingw32-gcc -g -O2 -fomit-frame-pointer -Wall -pipe -I../include -I"/usr/local/tcl8.6.2-mingw/include" -DHAVE_CONFIG_H -c -o memory.o memory.c In file included from thread.h:40:0, from memory.c:36: ../include/nsthread.h:168:0: warning: "lseek" redefined [enabled by default] # define lseek _lseek ^ In file included from ../include/nsthread.h:107:0, from thread.h:40, from memory.c:36: /usr/x86_64-w64-mingw32/sys-root/mingw/include/io.h:335:0: note: this is the location of the previous definition #define lseek lseek64 ^ x86_64-w64-mingw32-gcc -g -O2 -fomit-frame-pointer -Wall -pipe -I../include -I"/usr/local/tcl8.6.2-mingw/include" -DHAVE_CONFIG_H -c -o mutex.o mutex.c In file included from thread.h:40:0, from mutex.c:36: ../include/nsthread.h:168:0: warning: "lseek" redefined [enabled by default] # define lseek _lseek ^ In file included from ../include/nsthread.h:107:0, from thread.h:40, from mutex.c:36: /usr/x86_64-w64-mingw32/sys-root/mingw/include/io.h:335:0: note: this is the location of the previous definition #define lseek lseek64 ^ x86_64-w64-mingw32-gcc -g -O2 -fomit-frame-pointer -Wall -pipe -I../include -I"/usr/local/tcl8.6.2-mingw/include" -DHAVE_CONFIG_H -c -o cslock.o cslock.c In file included from thread.h:40:0, from cslock.c:47: ../include/nsthread.h:168:0: warning: "lseek" redefined [enabled by default] # define lseek _lseek ^ In file included from ../include/nsthread.h:107:0, from thread.h:40, from cslock.c:47: /usr/x86_64-w64-mingw32/sys-root/mingw/include/io.h:335:0: note: this is the location of the previous definition #define lseek lseek64 ^ x86_64-w64-mingw32-gcc -g -O2 -fomit-frame-pointer -Wall -pipe -I../include -I"/usr/local/tcl8.6.2-mingw/include" -DHAVE_CONFIG_H -c -o rwlock.o rwlock.c In file included from thread.h:40:0, from rwlock.c:48: ../include/nsthread.h:168:0: warning: "lseek" redefined [enabled by default] # define lseek _lseek ^ In file included from ../include/nsthread.h:107:0, from thread.h:40, from rwlock.c:48: /usr/x86_64-w64-mingw32/sys-root/mingw/include/io.h:335:0: note: this is the location of the previous definition #define lseek lseek64 ^ x86_64-w64-mingw32-gcc -g -O2 -fomit-frame-pointer -Wall -pipe -I../include -I"/usr/local/tcl8.6.2-mingw/include" -DHAVE_CONFIG_H -c -o reentrant.o reentrant.c In file included from thread.h:40:0, from reentrant.c:38: ../include/nsthread.h:168:0: warning: "lseek" redefined [enabled by default] # define lseek _lseek ^ In file included from ../include/nsthread.h:107:0, from thread.h:40, from reentrant.c:38: /usr/x86_64-w64-mingw32/sys-root/mingw/include/io.h:335:0: note: this is the location of the previous definition #define lseek lseek64 ^ reentrant.c: In function 'ns_ctime': reentrant.c:116:5: warning: implicit declaration of function 'ctime_s' [-Wimplicit-function-declaration] ctime_s(tlsPtr->ctbuf, sizeof(tlsPtr->ctbuf), clock); ^ x86_64-w64-mingw32-gcc -g -O2 -fomit-frame-pointer -Wall -pipe -I../include -I"/usr/local/tcl8.6.2-mingw/include" -DHAVE_CONFIG_H -c -o sema.o sema.c In file included from thread.h:40:0, from sema.c:40: ../include/nsthread.h:168:0: warning: "lseek" redefined [enabled by default] # define lseek _lseek ^ In file included from ../include/nsthread.h:107:0, from thread.h:40, from sema.c:40: /usr/x86_64-w64-mingw32/sys-root/mingw/include/io.h:335:0: note: this is the location of the previous definition #define lseek lseek64 ^ x86_64-w64-mingw32-gcc -g -O2 -fomit-frame-pointer -Wall -pipe -I../include -I"/usr/local/tcl8.6.2-mingw/include" -DHAVE_CONFIG_H -c -o thread.o thread.c In file included from thread.h:40:0, from thread.c:37: ../include/nsthread.h:168:0: warning: "lseek" redefined [enabled by default] # define lseek _lseek ^ In file included from ../include/nsthread.h:107:0, from thread.h:40, from thread.c:37: /usr/x86_64-w64-mingw32/sys-root/mingw/include/io.h:335:0: note: this is the location of the previous definition #define lseek lseek64 ^ x86_64-w64-mingw32-gcc -g -O2 -fomit-frame-pointer -Wall -pipe -I../include -I"/usr/local/tcl8.6.2-mingw/include" -DHAVE_CONFIG_H -c -o tls.o tls.c In file included from thread.h:40:0, from tls.c:38: ../include/nsthread.h:168:0: warning: "lseek" redefined [enabled by default] # define lseek _lseek ^ In file included from ../include/nsthread.h:107:0, from thread.h:40, from tls.c:38: /usr/x86_64-w64-mingw32/sys-root/mingw/include/io.h:335:0: note: this is the location of the previous definition #define lseek lseek64 ^ x86_64-w64-mingw32-gcc -g -O2 -fomit-frame-pointer -Wall -pipe -I../include -I"/usr/local/tcl8.6.2-mingw/include" -DHAVE_CONFIG_H -c -o time.o time.c In file included from thread.h:40:0, from time.c:37: ../include/nsthread.h:168:0: warning: "lseek" redefined [enabled by default] # define lseek _lseek ^ In file included from ../include/nsthread.h:107:0, from thread.h:40, from time.c:37: /usr/x86_64-w64-mingw32/sys-root/mingw/include/io.h:335:0: note: this is the location of the previous definition #define lseek lseek64 ^ time.c: In function 'Ns_GetTime': time.c:65:21: error: invalid suffix "i64" on integer constant #define EPOCH_BIAS 116444736000000000i64 ^ time.c:72:37: note: in expansion of macro 'EPOCH_BIAS' timePtr->sec = (time_t)((ft.i - EPOCH_BIAS) / 10000000i64); ^ time.c:72:51: error: invalid suffix "i64" on integer constant timePtr->sec = (time_t)((ft.i - EPOCH_BIAS) / 10000000i64); ^ time.c:73:35: error: invalid suffix "i64" on integer constant timePtr->usec =(long)((ft.i / 10i64) % 1000000i64); ^ time.c:73:44: error: invalid suffix "i64" on integer constant timePtr->usec =(long)((ft.i / 10i64) % 1000000i64); ^ make[1]: *** [time.o] Error 1 make[1]: Leaving directory `/usr/local/src/naviserver-tip/nsthread' make: *** [all] Error 1 |
From: Andrew P. <at...@pi...> - 2014-10-06 23:42:44
|
On Mon, Oct 06, 2014 at 11:33:25PM +0100, Stephen wrote: > Here are some build errors while cross-compiling with mingw using this > docker image: > > https://bitbucket.org/groks/build-naviserver-mingw/src/tip/naviserver/ The docker image sounds useful. However, building with Mingw on Windows would use the Unix makefiles. I've never tested that build combination and doubt that anyone else has done so recently either. Right now I'm focused on just getting Windows builds to work with nmake and MSVC. Mingw can come later, especially since it (at least currently) and MSVC are tied to different build systems. It would be nice to eventually adopt a single build system that would work well with all combinations of platforms and compilers, but discussing that seems somewhat premature. Btw, the pages below on Tcl-based replacements for make and autoconf are interesting, especially in that it should be feasible to have **Tcl itself** self-host on such tools, while avoiding the chicken-and-egg problem: http://wiki.tcl.tk/27561 http://wiki.tcl.tk/27197 http://wiki.tcl.tk/12593 But I'd want get the opinion of the Tcl core team and other experts before going very far down such a path, either for Naviserver or in my own code. -- Andrew Piskorski <at...@pi...> |
From: Andrew P. <at...@pi...> - 2014-10-08 16:39:17
|
On Wed, Oct 08, 2014 at 09:32:56AM +0200, Gustaf Neumann wrote: > the advantage of approach 2 (disable mutex timings, use ) is that this > works for 32bit and 64 bit, while (1) > works most likely only for 64bit (if i look at the constant). No, I'm confident that the approach using EPOCH_BIAS works just fine on 32-bit Windows, because in AOLserver 4.0.7 and 4.0.8 that is the ONLY implementation of Ns_GetTime on Windows, and I ran those versions of AOLserver for years on Windows XP 32-bit. Calls to Ns_GetTime() are very common, so AFAICT, for those AOLservers to work at all, that EPOCH_BIAS code had to be working fine on 32-bit Windows XP. The only other alternative is that AOLserver was somehow magically calling an entirely different function instead of Ns_GetTime(), which I judge as very unlikely. > The mutex timings is an additional feature in naviserver, so at least > for the time being, one can life without that.... and with the > deeper knowledge, we might be able to activate this later again. Ah, I didn't realize that. I should turn them back off for Windows in my fork then? > The problem seems to be: DllMain() calls certain functions > before the normal entry point main() is called, which initializes > tcl etc. The Windows man page [1] says: Interesting. And of course DllMain() only exists on Windows. But, why do we need DllMain() at all? I assume we DO need it of course, I just don't understand the design differences between Naviserver's pthread and winthread support. Both Posix and Windows use Nsthreads_LibInit(), but DllMain() seems to be an extra layer not present for Posix. > > changing the Ns_Time.sec (in nsthread.h) to be time_t rather than long > > fixes nearly everything. > > The reason, naviserver uses 2x long in Ns_Time is probably due to > Tcl, using: AFAICT, using long is just wrong, and Tcl and Naviserver both should have been using time_t for many years now. I'm not sure when time_t was invented and standardized, but since then it's been what Unix and Windows both use. So what am I missing? Why did the Tcl maintainers stick with long for Tcl_Time.sec? It could just be a holdover from before time_t existed or really mattered, but perhaps they had a better for keeping it as a long. > So at least, when using Tcl_GetTime(), > which returns the time as Tcl_Time, everything should be > fine. Note, that in your native time implementation, you > assign to timePtr->sec with (time_t), which might be > part of the problem you are experiencing. Could be. I will try the no-mutex-timings and Ns_GetTimeFromTcl() approach again, this time with Ns_Time.sec changed back to type long rather than time_t. > Probably, i should get a windows installation running, but > the setup costs are quite high in terms of time and lack > on windows-knowlege on my side. Yeah. Maybe the Docker stuff Stephen showed us would make it easier? Definitely see the notes I sent a while back on exactly what Microsoft software to install. All the main stuff you need is available for free but figuring out WHAT you're supposed to install is not easy. Btw, if you do go that route, be aware that I am using nmake -f Makefile.win32 for building and cleaning, but NOT for installing Naviserver. I have not debugged the install commands of those two *.win32 makefiles at all, because I have my own old Tcl script ("win32-install-nsd.tcl") for intalling on Windows, which I was able to quickly adapt to Naviserver. If you want that install script let me know. I haven't added it to Mercurial yet because it is in CVS, and I'd like to nicely retain its full history it, but haven't looked up how to best do that yet. -- Andrew Piskorski <at...@pi...> |
From: Andrew P. <at...@pi...> - 2014-10-09 00:16:10
|
Gustaf, my fork should be suitable for merging back into naviserver/naviserver now. On Wed, Oct 08, 2014 at 09:32:56AM +0200, Gustaf Neumann wrote: > So at least, when using Tcl_GetTime(), > which returns the time as Tcl_Time, everything should be > fine. Note, that in your native time implementation, you > assign to timePtr->sec with (time_t), which might be > part of the problem you are experiencing. Your bugfix aecf84685138 is of course the the key change: https://bitbucket.org/naviserver/naviserver/commits/aecf8468513892cb6dc241f09a9e8ad2dcd57793 Now with that, the only remaining problem is using Ns_GetTimeFromTcl() (the "platform-independent" approach) while mutex timings are turned on. Using Ns_Time.sec as long vs. time_t made no difference for that problem, which makes sense given your explanation of the problems of calling Tcl from inside DllMain(). All other combinations appear to work fine, regardless of whether Ns_Time.sec is long or time_t. To summarize, we've explored these four binary state changes on 64-bit Windows 7: 1. LogTime() long/time_t arg passing bug (aecf846) Fixed vs. Broken. 2. Mutex timing ON vs. OFF. 3. Ns_Time.sec is type long vs. time_t. 4. Ns_GetTime() uses Ns_GetTimeFromTcl() vs. Ns_GetTimeFromWindowsFileTime(). Some of the combinations of those states or of interest for deciding what we should do with the Naviserver codebase. We want to know if Naviserver seems (at least trivially) to run ok, vs. when its behavior is clearly very bad. Below, I will write each combination of states as a four-tuple, e.g. "Broken, On, long, FromTcl", with the result (Ok or Bad) to the left. In earlier emails, I covered these combinations: Bad: Broken, On, long, FromTcl Immediate hang on startup. Bad: Broken, On, time_t, FromTcl Immediate hang on startup. Bad: Broken, Off, long, FromWindowsFileTime Slightly later failure during startup, EINVAL from localtime_s(). Ok: Broken, Off, time_t, FromTcl Ok: Broken, On, time_t, FromWindowsFileTime Today I (manually) tested these combinations and found that: Ok: Fixed, Off, time_t, FromWindowsFileTime Ok: Fixed, On, time_t, FromWindowsFileTime Ok: Fixed, Off, time_t, FromTcl Ok: Fixed, On, time_t, FromTcl Ok: Fixed, Off, long, FromWindowsFileTime Ok: Fixed, On, long, FromWindowsFileTime Ok: Fixed, Off, long, FromTcl: Bad: Fixed, On, long, FromTcl Immediate hang on startup. So clearly we do NOT need to make Ns_Time.sec a time_t. Separately, I've asked tcl...@li... why it is a long in Tcl. In Ns_GetTime(), if we use Ns_GetTimeFromWindowsFileTime(), we can turn on mutex timing if we want. If we instead use Ns_GetTimeFromTcl() we must keep mutex timing turned off. Since Gustaf seems to prefer Ns_GetTimeFromTcl() over mutex timing, I've gone with that combination in my fork as well. -- Andrew Piskorski <at...@pi...> |
From: Gustaf N. <ne...@wu...> - 2014-10-09 11:54:03
|
On 09.10.14 02:16, Andrew Piskorski wrote: > So clearly we do NOT need to make Ns_Time.sec a time_t. that's what i've said :) > Separately, > I've asked tcl...@li... why it is a long in Tcl. another practical reason in addition to Joe's answer is binary compatibility for extensions. -gn |
From: Maurizio M. <Mau...@sp...> - 2014-10-03 15:47:43
|
The Windows-OpenACS distribution which I make available here (http://www.spazioit.com/pages_en/sol_inf_en/windows-openacs_en/) is based on AOLServer 4.5.2, contains the sources, is compiled with Visual Studio 2013 and runs on Windows 64. So if I where you I would give a look at that distribution and see how all those time related functions have been handled. That could save you some time. Maurizio -----Original Message----- From: Andrew Piskorski [mailto:at...@pi...] Sent: 03 October 2014 16:49 To: nav...@li... Subject: Re: [naviserver-devel] Naviserver hangs on Windows Hm, the old AOLserver 4.0.7 version of Ns_GetTime() had a Windows ifdef which is entirely missing from Naviserver! (See below.) I found two relevant commits in Mercurial, from 2009 and 2012, below. >From those logs, it looks like what happened is that in 2009, Zoran removed the platform specific C code and instead made Naviserver call Tcl_GetTime (somehow). Then in 2012, Gustaf added back the old Unix-specific gettimeofday() implementation for performance reasons. So should we just add back the old Windows-specific C implementation too? Or is it better to use Zoran's way of invoking Tcl_GetTime? Do we even know for sure if that method really is "platform-independent" as Zoran thought it was? E.g., did Zoran or someone else at Archiware test it on Windows? Part of Ns_GetTime() from ancient AOLserver 4.0.7 sources: ------------------------------------------------------------ #ifdef _WIN32 /* * Number of 100 nanosecond units from 1/1/1601 to 1/1/1970 */ #define EPOCH_BIAS 116444736000000000i64 union { unsigned __int64 i; FILETIME s; } ft; GetSystemTimeAsFileTime(&ft.s); timePtr->sec = (time_t)((ft.i - EPOCH_BIAS) / 10000000i64); timePtr->usec =(long)((ft.i / 10i64) % 1000000i64); #else #endif changeset: 2435:16508ab4b3de user: Gustaf Neumann <ne...@wu...> date: Fri Nov 02 01:54:38 2012 +0100 files: configure.in include/nsconfig.h.in nsthread/time.c description: use gettimeofday() if available instead of the rount trip to tcl changeset: 2009:4bf1ee6ebf84 user: Zoran Vasiljevic <zv...@ar...> date: Sat Sep 29 14:14:03 2007 +0100 files: ChangeLog nsproxy/nsproxylib.c nsthread/time.c description: Ns_GetTime now relies on Tcl_GetTime to be platform-independent -- Andrew Piskorski <at...@pi...> ---------------------------------------------------------------------------- -- Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk _______________________________________________ naviserver-devel mailing list nav...@li... https://lists.sourceforge.net/lists/listinfo/naviserver-devel |
From: Andrew P. <at...@pi...> - 2014-10-04 07:10:58
|
On Fri, Oct 03, 2014 at 05:20:48PM +0200, Maurizio Martignano wrote: > Subject: Re: Naviserver hangs on Windows > The Windows-OpenACS distribution which I make available here > (http://www.spazioit.com/pages_en/sol_inf_en/windows-openacs_en/) is based > on AOLServer 4.5.2, contains the sources, is compiled with Visual Studio > 2013 and runs on Windows 64. Indeed, the first thing I noticed is that Maurizio's Windows build seems to use nsconfig.tcl in place of the Unix configure/autoconf, which Jim Davidson added back in 2005. Naviserver folks, can you comment on what you think of that approach? Does Naviserver not include it solely because it wasn't included in the AOLserver 4.0.10 branch that Naviserver forked from? https://bitbucket.org/aolserver/aolserver/src/2aa0f24395ae0a42fd590f3635e4fdf2c7eefdfd/nsconfig.tcl?at=default andy@milo:/home/nobackup/co/nsd-aol/aolserver-head-hg$ hg log nsconfig.tcl configure.tcl changeset: 1370:76895d8bf843 user: Jim Davidson <jgd...@ao...> date: Thu Aug 18 21:48:21 2005 +0100 summary: Renamed configure script configure.tcl changeset: 1367:9c88ea73dcad user: Jim Davidson <jgd...@ao...> date: Wed Aug 17 23:55:57 2005 +0100 summary: Updates to new build tools to support Unix changeset: 1360:697679717350 user: Jim Davidson <jgd...@ao...> date: Wed Aug 17 22:18:46 2005 +0100 summary: New platform-indepedent build support -- Andrew Piskorski <at...@pi...> |
From: Jeff R. <dv...@di...> - 2014-10-06 17:16:43
|
Gustaf Neumann wrote: > Am 06.10.14 06:46, schrieb Jeff Rogers: >> >> This struck me as an interesting optimization question, so I wrote a >> quick program to test it (attached). > This is not an interesting optimization question, since Tcl itself uses > gettimeofday() > (plus jumping around which might have a bad cache and locality > influence, and some > more copying). I think my underlying point was missed in the measurements. If there is not significant performance advantage to the platform-specific optimized version of Ns_GetTime over the platform-independent Tcl_GetTime, is it worth it to keep the optimized version? The penalty to be paid is in problems like the current one, where the windows build is for some reason having problems. On windows the function is delegating to Tcl_GetTime, but the system-specific bits confuse the issue. Your measurements are consistent and show the relative results I expected. Such is the benefit of real hardware :) They show a small advantage to Ns_GetTime over Tcl_GetTime, about 1.2%. This *is* a very frequently called function, but it is also pretty fast in any case. That's 1.2% of perhaps 5% (?? totally random guess) of overall request processing time. To me, that adds up to not a lot. A different takeaway from this benchmark is that if you are trying to wring every last bit of performance out of this call, it might be advantageous to define it as a macro (or use gcc/c99 inline declarations so it can be inlined and avoid the function call overhead and get to within a few instructions of the raw gettimeofday call (presumably you'll still need to copy the elements to the Ns_Time struct). -J > > One cannot expect a big difference. For better reliability, > i've increased the count. Below are 4 runs on openacs.org. > The native call is from 8% to 20% faster (the latter one most > probably due to external influences). The machine is a bare metal. > Numbers will vary depending on the OS. The faster the > system function gettimeofday() is, the bigger is the percentage > difference. > > -g > > gustafn@openacs:~$ ./tt > count: 100000000 > Tcl_GetTime: 3858686 usec 0.04 per > Ns_GetTime: 3811593 usec 0.04 per > gettimeofday: 3561367 usec 0.04 per > gustafn@openacs:~$ ./tt > count: 100000000 > Tcl_GetTime: 3858657 usec 0.04 per > Ns_GetTime: 3824398 usec 0.04 per > gettimeofday: 3563398 usec 0.04 per > gustafn@openacs:~$ ./tt > count: 100000000 > Tcl_GetTime: 4351765 usec 0.04 per > Ns_GetTime: 4024717 usec 0.04 per > gettimeofday: 3594736 usec 0.04 per > gustafn@openacs:~$ ./tt > count: 100000000 > Tcl_GetTime: 3864511 usec 0.04 per > Ns_GetTime: 3831866 usec 0.04 per > gettimeofday: 3541932 usec 0.04 per > > > > ------------------------------------------------------------------------------ > Slashdot TV. Videos for Nerds. Stuff that Matters. > http://pubads.g.doubleclick.net/gampad/clk?id=160591471&iu=/4140/ostg.clktrk > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel > |
From: Andrew P. <at...@pi...> - 2014-10-06 18:18:45
|
On Mon, Oct 06, 2014 at 10:16:31AM -0700, Jeff Rogers wrote: > If there is not significant performance advantage to the > platform-specific optimized version of Ns_GetTime over the > platform-independent Tcl_GetTime, is it worth it to keep the optimized > version? It is very much worth it to me, at least for now, because: One, I don't understand how the invocation of the "platform-independent Tcl_GetTime" even works. There is never any explicit call to Tcl_GetTime nor any other Tcl function, and Tcl_GetTime does not appear on the call stack. The so-callled "optimized" code is easier to understand. Two, the current "platform-independent" approach locks up in my Windows build, while the old Windows-specific code that AOLserver used for many years does not. I do not know why this is, but I certainly will use the old Windows-specific code until I figure it out. -- Andrew Piskorski <at...@pi...> |
From: Gustaf N. <ne...@wu...> - 2014-10-07 11:11:15
|
Stephen, the Docker image is cool stuff! Many thanks! This eases mingw builds significantly. It compiles now fine the first bunch of files in nsthread fine, but stops then with x86_64-w64-mingw32-gcc -shared -L../nsthread -L../nsd -L../nsdb -o libnsthread.dll error.o master.o memory.o mutex.o cslock.o rwlock.o reentrant.o sema.o thread.o tls.o time.o pthread.o fork.o signal.o winthread.o -L/usr/local/tcl8.6.2-mingw/lib -ltcl86 -lnetapi32 -lkernel32 -luser32 -ladvapi32 -lws2_32 -L/usr/local/tcl8.6.2-mingw/lib /usr/lib64/gcc/x86_64-w64-mingw32/4.8.3/../../../../x86_64-w64-mingw32/bin/ld: cannot find -ltcl86 collect2: error: ld returned 1 exit status make[1]: *** [libnsthread.dll] Error 1 Looks like the installation of tcl is missing... -g On 07.10.14 00:33, Stephen wrote: > Here are some build errors while cross-compiling with > mingw using this docker image: > > https://bitbucket.org/groks/build-naviserver-mingw/src/tip/naviserver/ > > |
From: Stephen <yo...@gr...> - 2014-10-07 12:00:36
|
On Tue, Oct 7, 2014 at 12:11 PM, Gustaf Neumann <ne...@wu...> wrote: > > Stephen, > > the Docker image is cool stuff! Many thanks! > This eases mingw builds significantly. > > It compiles now fine the first bunch of files in nsthread > fine, but stops then with > > x86_64-w64-mingw32-gcc -shared -L../nsthread -L../nsd > -L../nsdb -o libnsthread.dll error.o master.o memory.o > mutex.o cslock.o rwlock.o reentrant.o sema.o thread.o tls.o > time.o pthread.o fork.o signal.o winthread.o > -L/usr/local/tcl8.6.2-mingw/lib -ltcl86 -lnetapi32 > -lkernel32 -luser32 -ladvapi32 -lws2_32 > -L/usr/local/tcl8.6.2-mingw/lib > /usr/lib64/gcc/x86_64-w64-mingw32/4.8.3/../../../../x86_64-w64-mingw32/bin/ld: > cannot find -ltcl86 > collect2: error: ld returned 1 exit status > make[1]: *** [libnsthread.dll] Error 1 > > Looks like the installation of tcl is missing... When you compile tcl --with-symbols it adds a 'g' to the lib and executabe names so I guess it needs to be something like: -ltcl86g. If you scroll up to the end of the previously successful docker step it should say something like: Now type `make' to compile. ---> 5c66713ca361 You can jump into the container with the file system as it was after that step by typing something like: $ docker run -t -i 5c66713ca361 /bin/bash ...and poke around from there. |
From: Andrew P. <at...@pi...> - 2014-10-07 20:09:12
|
On Tue, Oct 07, 2014 at 12:09:41AM +0200, Gustaf Neumann wrote: > Please something more to try: to provide some potential evidence for > my "too early for tcl calls" hypothesis, i've deactivated for the > time being the mutex time monitoring for windows, since the earliest > calls are from mutex calls. Can you check when possible, whether > Tcl_GetTime() can be used now inside Ns_GetTime()? I merged in all your changes to my sandbox. Your change deactivating the mutex timings, here: https://bitbucket.org/naviserver/naviserver/commits/6ba02284c80242b3046fc7e8cddc63cc0677b2b5 definitely does make a difference - the server now gets slightly farther and then exits: C:\> C:\web\nsd-atp\bin\nsd -c [384.e14][-main-] Notice: nsmain: NaviServer/4.99 starting [384.e14][-main-] Fatal: nsthreads: localtime_s failed in ns_localtime: win32 err: 22 C:\> C:\web\nsd-atp\bin\nsd -f -t C:\web\nsd-atp\conf\nsd-config.tcl [3768.9b4][-main-] Notice: nsmain: enable progess statistics for uploads >= 1048576 bytes [3768.9b4][-main-] Fatal: nsthreads: localtime_s failed in ns_localtime: win32 err: 22 The (partial) backtrace from WinDbg looks like this: tcl85!Tcl_Panic nsthread!NsThreadFatal nsthread!ns_localtime nsd_7fef0060000!Ns_InfoNameOfExecutable nsd_7fef0060000!Ns_InfoNameOfExecutable nsd!main So this looks like the same problem as before, just reached from a later point in the Naviserver startup. -- Andrew Piskorski <at...@pi...> |
From: Andrew P. <at...@pi...> - 2014-10-07 21:02:48
|
On Tue, Oct 07, 2014 at 04:09:02PM -0400, Andrew Piskorski wrote: > C:\> C:\web\nsd-atp\bin\nsd -c > [384.e14][-main-] Notice: nsmain: NaviServer/4.99 starting > [384.e14][-main-] Fatal: nsthreads: localtime_s failed in ns_localtime: win32 err: 22 In ns_localtime(), I changed this call: errNum = localtime_s(&tlsPtr->ltbuf, clock); to instead do this: struct tm foo; errNum = localtime_s(&foo, clock); And yet it still fails in exactly the same way, with EINVAL = 22. This makes me think there is nothing wrong with the tlsPtr code or data, rather something is seriously broken in my build. (But I have no idea what.) Possibly of interest, is that _localtime64_s() fails in exactly the same way. _localtime32_s() apparently does not trigger EINVAL, but instead fails on the same debug assertion, where the tm_mday value is out of range (and most likely -1 just like before, although I did not confirm that). I increased my compiler warning level from /W1 to /W3, and now I see LOTS of type warnings, e.g.: log.c(689) : warning C4133: 'function' : incompatible types - from 'long *' to 'const time_t *' nssock.c(377) : warning C4133: 'function' : incompatible types - from 'int *' to 'const char *' binder.c(324) : warning C4244: 'function' : conversion from 'SOCKET' to 'int', possible loss of data driver.c(3888) : warning C4244: '=' : conversion from 'SOCKET' to 'int', possible loss of data driver.c(2356) : warning C4244: 'function' : conversion from 'ssize_t' to 'unsigned int', possible loss of data driver.c(3475) : warning C4244: '=' : conversion from 'ssize_t' to 'off_t', possible loss of data driver.c(3351) : warning C4244: 'function' : conversion from 'Tcl_WideInt' to 'long', possible loss of data driver.c(2346) : warning C4267: 'function' : conversion from 'size_t' to 'int', possible loss of data driver.c(3354) : warning C4267: 'function' : conversion from 'size_t' to 'unsigned int', possible loss of data tclhttp.c(371) : warning C4244: '=' : conversion from '__int64' to 'int', possible loss of data That first one in log.c looks mildly suspicious. But, on a 64-bit machine like this "long" and "time_t" are both 64-bit integers, so I don't see the problem. Why are they "incompatible"? http://msdn.microsoft.com/en-us/library/323b6b3k%28v=vs.100%29.aspx However, in include/nsthread.h, why is Ns_Time defined like this?: typedef struct Ns_Time { long sec; long usec; } Ns_Time; Shouldn't the type of sec be time_t rather than long? -- Andrew Piskorski <at...@pi...> |
From: Andrew P. <at...@pi...> - 2014-10-08 05:02:44
|
On Tue, Oct 07, 2014 at 05:02:41PM -0400, Andrew Piskorski wrote: > And yet it still fails in exactly the same way, with EINVAL = 22. > This makes me think there is nothing wrong with the tlsPtr code or > data, rather something is seriously broken in my build. I think my analysis above was entirely wrong. I now see evidence that the time_t value going into ns_localtime() is incorrect after all, likely due to some sort of type mismatch. And, after a very small amount of testing, it looks like simply changing the Ns_Time.sec (in nsthread.h) to be time_t rather than long fixes nearly everything. Those changs are here: https://bitbucket.org/apiskors/naviserver/commits/all Since I'd already written it earlier when I was more confused, here's the slow, convoluted path I took to reach that conclusion: I was stuck, so in a stand-alone C program, I confirmed that with a 64-bit time_t (which is the default), by casting to long long I can correctly display the integer Unix time regardless of its particular value, like so: time_t clock; //time(&clock); /* Now */ //clock = 1412722229; /* Oct 2014 */ clock = 2174774400; /* Nov 2038*/ printf("clock lld: %lld\n", (long long)clock); Note that Nov. 2038 requires a 64-bit time_t, it is too big for 32-bit. With both those integers above, printf() displays the same value I hard-coded into my code. If I use other format specifiers with printf, the displayed integer can change completely depending on the itns value, but by casting to long long and using %lld I seem to always get the same output as my input. Next, I changed ns_localtime() to use the same constant, hard-coded time_t *clock value of 2174774400. That worked! The EINVAL failure went away, and I was able to start up and shut down Naviserver normally! Or if Inside ns_localtime() I completely ignore the incoming clock pointer and instead just call time() to get the current time myself, that works too! Finally, I restored ns_localtime to its normal behavior, but left in a debugging printf, like this: errNum = localtime_s(&tlsPtr->ltbuf, clock); printf("\nDebug:atp: ns_localtime: errNum from localtime_s: %d", errNum); printf("\n clock lld: %lld", (long long)*clock); printf("\n clock ld: %ld" , (long)*clock); printf("\n clock d: %d\n", (int)*clock); And here's the corresponding output from starting up the server twice, which immediately aborts: Debug:atp: ns_localtime: errNum from localtime_s: 22 clock lld: 2996724649205424 clock ld: 1412734640 clock d: 1412734640 [3280.d64][-main-] Notice: nsmain: NaviServer/4.99 starting [3280.d64][-main-] Fatal: nsthreads: localtime_s failed in ns_localtime: win32 err: 22 Debug:atp: ns_localtime: errNum from localtime_s: 22 clock lld: 435292053216956 clock ld: 1412734652 clock d: 1412734652 [2892.58c][-main-] Notice: nsmain: NaviServer/4.99 starting [2892.58c][-main-] Fatal: nsthreads: localtime_s failed in ns_localtime: win32 err: 22 $ tclsh8.5 % clock format 435292053216956 Sat Dec 29 13:55:56 EST 2036604 % clock format 1412734640 Tue Oct 07 22:17:20 EDT 2014 Those huge 15-digits integers are VERY far in the future, and they change seemingly arbitrarily each time I run nsd. But their shorter integer representation is the correct time! Thus my earlier confusion, because I was calling printf with %i or %d, and only seeing the seemingly-correct short value. Now, Tcl can process the year 2,036,604 just fine, so I'm not sure why the Windows localtime_s() is throwing EINVAL on those inputs. But those Unix time values sure seem suspect... And that lead me to the definition of Ns_Time. (Which I now recall wondering about days or weeks ago, but then forgot about. Ugh.) -- Andrew Piskorski <at...@pi...> |
From: Maurizio M. <Mau...@sp...> - 2014-10-08 06:05:54
|
Dear Andrew, Sorry for the late reply, but have been away of my computer. 1. _USE_32BIT_TIME_T Indeed, you are absolutely write. My Windows distribution had since 2011 to 2013 both Windows 32 and Windows 64 binaries. I was using more or less the same make and configuration files to build it. The define _USE_32BIT_TIME_T was used for the Windows 32 build but not for the Windows 64. The current version of my distribution only contains Windows 64 binaries and in fact the define is commented out in the makefile. Sorry for the time I made you waste on this point. 2. About the configuration I am using the tools made available by Visual Studio 2013 Professional Edition in the "VS2013 X64 Native Tools Command Prompt". That is: CL: Microsoft (R) C/C++ Optimizing Compiler Version 18.00.21005.1 for x64 LINK: Microsoft (R) Incremental Linker Version 12.00.21005.1 This is what I use and your questions below: COPTS = /EHsc /W3 /nologo /c DEFS = $(DEFS) /D "_WINDOWS" /D "TCL_THREADS=1" /D "WIN32" /D "_WIN32" \ /D "FD_SETSIZE=128" /D "NO_CONST=1" /D "_MBCS" # /D "_USE_32BIT_TIME_T" LOPTS = /NOLOGO /SUBSYSTEM:CONSOLE /OPT:NOREF /OPT:NOICF Did you comment out your _USE_32BIT_TIME_T there? [MM] Absolutely, I already explained what I did in point 1 and once again apologies. What does "/D _WIN32" do? [MM] This should be set automatically by the compiler, so in theory it is redundant... but one never knows... What C run-time library are you using? [MM] Visual C++ Redistributable Packages for Visual Studio 2013 (Release, No Debug) http://www.microsoft.com/en-us/download/details.aspx?id=40784 I do agree with your remarks in your mail, I had a look at the Visual Studio Project Files you use, the only relevant differences I can see are the following: 1. you are targeting as Machine_X64, I am targeting AMD64 2. you use Active TCL, I did compile from scratch also TCL, using the same compiler options (I believe this is very important as we have no control on how Active TCL is built) Hope this helps, Maurizio -----Original Message----- From: Andrew Piskorski [mailto:at...@pi...] Sent: 07 October 2014 22:36 To: nav...@li... Subject: Re: [naviserver-devel] Naviserver hangs on Windows On Mon, Oct 06, 2014 at 05:18:08PM -0400, Andrew Piskorski wrote: > On Sun, Oct 05, 2014 at 09:37:39PM +0200, Maurizio Martignano wrote: > > > Did you use the define _USE_32BIT_TIME_T yes or not? > I haven't tried using _USE_32BIT_TIME_T yet, but I think using it > would be INCORRECT on Windows-64. If I do pass the "/D _USE_32BIT_TIME_T" flag, I get this: C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\INCLUDE\crtdefs.h(463) : fatal error C1189: #error : You cannot use 32-bit time_t (_USE_32BIT_TIME_T) with _WIN64 So that seems pretty clear, I must NOT set that flag. -- Andrew Piskorski <at...@pi...> ---------------------------------------------------------------------------- -- Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk _______________________________________________ naviserver-devel mailing list nav...@li... https://lists.sourceforge.net/lists/listinfo/naviserver-devel |
From: Andrew P. <at...@pi...> - 2014-10-08 15:22:24
|
On Wed, Oct 08, 2014 at 08:05:42AM +0200, Maurizio Martignano wrote: > I do agree with your remarks in your mail, I had a look at the Visual Studio > Project Files you use, the only relevant differences I can see are the I do not use any Visual Studio Project Files for Naviserver. The ones in Mercurial are old and I have no idea if they work or not. The Naviserver Windows build I've been working on is controlled by the makefiles, a combination of the two Windows-only Makefile.win32 for nmake, and the individual module-specific makefiles which work in both Gnu make and Microsoft nmake. -- Andrew Piskorski <at...@pi...> |