I have succeeded compiling the NumPy computational package for Python using mingw-w64. However, when running the test suite, I consistently get a crash. Here it is the backtrace using gdb:
C:\Users\francesc\Desktop\NumPy>gdbpythonGNUgdb(GDB)7.1.50.20100318-cvsCopyright(C)2010FreeSoftwareFoundation,Inc.LicenseGPLv3+:GNUGPLversion3orlater<http://gnu.org/licenses/gpl.html>Thisisfreesoftware:youarefreetochangeandredistributeit.ThereisNOWARRANTY,totheextentpermittedbylaw.Type"show copying"and"show warranty"fordetails.ThisGDBwasconfiguredas"x86_64-w64-mingw32".Forbugreportinginstructions,pleasesee:<http://www.gnu.org/software/gdb/bugs/>...Readingsymbolsfrom \Python26_64/python.exe...(nodebuggingsymbolsfound)...done.(gdb)rStartingprogram: \Python26_64/python.exe[NewThread2880.0x410]Python2.6.5(r265:79096,Mar192010,18:02:59)[MSCv.150064bit(AMD64)]onwin32Type"help","copyright","credits"or"license"formoreinformation.>>>importnumpyC:\Python26_64\lib\site-packages\numpy\core\__init__.py:5:Warning:NumpybuiltwithMINGW-W64onWindows64bitsisexperimental,andonlyavailablefortesting.Youareadvisednottouseitforproduction.CRASHESARETOBEEXPECTED-PLEASEREPORTTHEMTONUMPYDEVELOPERSimportmultiarray>>>numpy.test()RunningunittestsfornumpyNumPyversion1.4.0NumPyisinstalledinC:\Python26_64\lib\site-packages\numpyPythonversion2.6.5(r265:79096,Mar192010,18:02:59)[MSCv.150064bit(AMD64)]noseversion0.11.3ForcingDISTUTILS_USE_SDK=1.....warning:HEAP[python.exe]:warning:InvalidaddressspecifiedtoRtlFreeHeap(0000000002240000,0000000000C25110)ProgramreceivedsignalSIGTRAP,Trace/breakpointtrap.0x0000000076fa6061inntdll!DbgBreakPoint()fromC:\Windows\system32\ntdll.dll(gdb)bt#0 0x0000000076fa6061 in ntdll!DbgBreakPoint ()fromC:\Windows\system32\ntdll.dll#1 0x0000000076ffe17a in ntdll!EtwEventProviderEnabled ()fromC:\Windows\system32\ntdll.dll#2 0x00000000022af0d8 in ?? ()#3 0x000000005104095c in ?? ()#4 0x0000000000219a08 in ?? ()#5 0x000000000e040001 in ?? ()#6 0x0000000002240000 in ?? ()#7 0x0000000076fe27a1 in ntdll!MD4Final () from C:\Windows\system32\ntdll.dll#8 0x0000000076fb9630 in ntdll!LdrGetProcedureAddress ()fromC:\Windows\system32\ntdll.dll#9 0x0000000076fb9500 in ntdll!LdrGetProcedureAddress ()fromC:\Windows\system32\ntdll.dll#10 0x0000000002240000 in ?? ()#11 0x0000000000c25110 in ?? ()#12 0x0000000002240000 in ?? ()#13 0x000000000623f197 in ?? ()#14 0x0000000002240000 in ?? ()#15 0x00000000770151a9 in ntdll!RtlTraceDatabaseCreate ()fromC:\Windows\system32\ntdll.dll#16 0x0000000000000000 in ?? ()(gdb)
Any hint on what is going on?
Francesc
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
have you built with debugging information (-g)?
Those kind of failures are mainly reasoned by pointer-truncation. This mainly happens on cast from pointer to an integer scalar. As it is pretty common on linux to assume that sizeof (long) == sizeof (void *) for x86_64 (this isn't true for x64 windows), check your build-logs for such truncation messages. For fixing this use uintptr_t/intptr_t types instead of 'int'/'long'/or'long long' types for pointer cast to scalar type.
Hopes this helps,
Kai
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Many thanks for your response. I'd say that numpy developers have been very careful on this, and that uintptr_t/intptr_t types are used as they should. In fact, numpy can be compiled with MSVC 2008 64-bit without problem. What I'm trying to see now is whether one can use a free compiler for doing this.
I'm attaching the log of the compilation with mingw-w64 on my machine. As you see, the -g flag has been used as well as "-Wall -Wstrict-prototypes". Believe it or not, it compiles without a single warning (for a project that generates around 50K LOC in C this is a big achiement indeed!). There is there some info about configuration, if that helps to see what could be happening.
logs are looking fine and I don't see anything failing here. As you get a crash in RtlFree, I assume that there is a memory corruption or a double free of heap, which shows this failure. It could be also reasoned by strict-aliasing (this more a guess). You can try to use the option '-Wstrict-aliasing=2' to check. Does this problem appears for an unoptimized build, too, or just by using optimization?
Just for you notice. We are at the moment about to do a rewrite of our complex-math routines. The current version (the same as for mingw.org) has some issues in respect to ISO-C99 conformancy. You can give our new version (to be found in tree /exprimental/new_complex/src) a try. As VC doesn't support 'long double' as 80-bit float (for VC 'long double' remains 64-bit precission), it possibly could be reasoned by these different sizes, too. Also MS print can't display 80-bit floats, therefore your application should use the mingw-printf routines instead (you can compile source by defining _POSIX and we switch printf functionality in background automatically).
I hope some of my thoughts can help you solve your issue.
Regards,
Kai
T
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
No, I don't think strict-aliasing would be a problem as I remember this was discusses on the numpy list and I think numpy is free of strict-aliasing problem (in addition -Wall already includes '-Wstrict-aliasing=3' ).
My previous tries were made with -g -O0, but replacing this by -O3 gives the same crash, so I don't think it is a problem with optimization neither.
I've tried with giving more verbosity to tests, and I've got more hints:
test_list (test_multiarray.TestFancyIndexing) ... ok
test_tuple (test_multiarray.TestFancyIndexing) ... ok
test_otherflags (test_multiarray.TestFlags) ... ok
test_writeable (test_multiarray.TestFlags) ... ok
test_ip_basic (test_multiarray.TestFromBuffer) ... ok
test_multiarray.TestIO.test_ascii ...
Program received signal SIGSEGV, Segmentation fault.
0x000000007700592a in ntdll!RtlNtStatusToDosErrorNoTeb ()
from C:\Windows\system32\ntdll.dll
(gdb) bt
#0 0x000000007700592a in ntdll!RtlNtStatusToDosErrorNoTeb ()
from C:\Windows\system32\ntdll.dll
#1 0x0000000000000000 in ?? ()
(gdb)
So, it seems that it is a problem with char I/O. Also, I've run into problems in many Unicode tests. So I think the problem is more related with char/unicode code. Oh god, I think this would be very hairy to solve.
Thanks anyway for your help!
Francesc
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
I have succeeded compiling the NumPy computational package for Python using mingw-w64. However, when running the test suite, I consistently get a crash. Here it is the backtrace using gdb:
Any hint on what is going on?
Francesc
Hello,
have you built with debugging information (-g)?
Those kind of failures are mainly reasoned by pointer-truncation. This mainly happens on cast from pointer to an integer scalar. As it is pretty common on linux to assume that sizeof (long) == sizeof (void *) for x86_64 (this isn't true for x64 windows), check your build-logs for such truncation messages. For fixing this use uintptr_t/intptr_t types instead of 'int'/'long'/or'long long' types for pointer cast to scalar type.
Hopes this helps,
Kai
Hi Kai,
Many thanks for your response. I'd say that numpy developers have been very careful on this, and that uintptr_t/intptr_t types are used as they should. In fact, numpy can be compiled with MSVC 2008 64-bit without problem. What I'm trying to see now is whether one can use a free compiler for doing this.
I'm attaching the log of the compilation with mingw-w64 on my machine. As you see, the -g flag has been used as well as "-Wall -Wstrict-prototypes". Believe it or not, it compiles without a single warning (for a project that generates around 50K LOC in C this is a big achiement indeed!). There is there some info about configuration, if that helps to see what could be happening.
Hi Falted,
logs are looking fine and I don't see anything failing here. As you get a crash in RtlFree, I assume that there is a memory corruption or a double free of heap, which shows this failure. It could be also reasoned by strict-aliasing (this more a guess). You can try to use the option '-Wstrict-aliasing=2' to check. Does this problem appears for an unoptimized build, too, or just by using optimization?
Just for you notice. We are at the moment about to do a rewrite of our complex-math routines. The current version (the same as for mingw.org) has some issues in respect to ISO-C99 conformancy. You can give our new version (to be found in tree /exprimental/new_complex/src) a try. As VC doesn't support 'long double' as 80-bit float (for VC 'long double' remains 64-bit precission), it possibly could be reasoned by these different sizes, too. Also MS print can't display 80-bit floats, therefore your application should use the mingw-printf routines instead (you can compile source by defining _POSIX and we switch printf functionality in background automatically).
I hope some of my thoughts can help you solve your issue.
Regards,
Kai
T
Kai,
No, I don't think strict-aliasing would be a problem as I remember this was discusses on the numpy list and I think numpy is free of strict-aliasing problem (in addition -Wall already includes '-Wstrict-aliasing=3' ).
My previous tries were made with -g -O0, but replacing this by -O3 gives the same crash, so I don't think it is a problem with optimization neither.
I've tried with giving more verbosity to tests, and I've got more hints:
So, it seems that it is a problem with char I/O. Also, I've run into problems in many Unicode tests. So I think the problem is more related with char/unicode code. Oh god, I think this would be very hairy to solve.
Thanks anyway for your help!
Francesc