From: Ryan H. <hu...@wf...> - 2005-07-26 20:00:25
|
Any help or comments concerning the following problem will be greatly appreciated. Basically, I am developing a scientific application which uses Fortran 90, C, and C++ code, and does some heavy operations on matrices. The details are unimportant (I think), as the program runs to completion without issues using 3D matrices of smallish sizes. When the matrix dimensions are increased to say, 128 x 128 x 128, it appears as if I am running out of memory...malloc() is failing and responding with errno 12 which maps to not enough memory. The problem is that according to every way I have been monitoring the memory of my system, I have plenty left! If anyone can come up with any explanation, or even just something to investigate, I would very much appreciate it...I am running on Windows XP with MinGW and Msys, using G95 to compile and link, and using g++/gcc to compile the c++/c code. Thanks so much for everyones time. - Ryan |
From: Michael G. <mg...@te...> - 2005-07-27 09:28:52
|
> Any help or comments concerning the following problem will be greatly > appreciated. Basically, I am developing a scientific application which u= ses > Fortran 90, C, and C++ code, and does some heavy operations on matrices. = The > details are unimportant (I think), as the program runs to completion with= out > issues using 3D matrices of smallish sizes. When the matrix dimensions a= re > increased to say, 128 x 128 x 128, it appears as if I am running out of > memory...malloc() is failing and responding with errno 12 which maps to n= ot > enough memory. The problem is that according to every way I have been > monitoring the memory of my system, I have plenty left! If anyone can co= me up > with any explanation, or even just something to investigate, I would very= much > appreciate it...I am running on Windows XP with MinGW and Msys, using G95= to > compile and link, and using g++/gcc to compile the c++/c code. Thanks so= much > for everyones time. Could you provide a small testprogram that shows this problem ? The following code works without problems on WinXP when compiled with mingw-gcc-3.4.4. #include <stdlib.h> #include <stdio.h> #define MATRIX_COUNT 30 #define MATRIX_DIM_1 128 #define MATRIX_DIM_2 128 #define MATRIX_DIM_3 128 int main(int argc, char** argv) { double**** aMatrix =3D (double****)malloc(MATRIX_COUNT*sizeof(double***= )); if (!aMatrix) { printf("*** Could not malloc %d double***\n", MATRIX_COUNT); exit(1); } for (int i=3D0; i<MATRIX_COUNT; ++i) { aMatrix[i] =3D (double***)malloc(MATRIX_DIM_1*sizeof(double**)); if (!aMatrix[i]) { printf("*** i=3D%d: Could not malloc %d double**\n", i, MATRIX_DIM_1); exit(1); } for (int j=3D0; j<MATRIX_DIM_1; ++j) { aMatrix[i][j] =3D (double**)malloc(MATRIX_DIM_2*sizeof(double*)); if (!aMatrix[i][j]) { printf("*** i=3D%d, j=3D%d: Could not malloc %d double*\n", i, j, MATRIX_= DIM_2); exit(1); } for (int k=3D0; k<MATRIX_DIM_2; ++k) { aMatrix[i][j][k] =3D (double*)malloc(MATRIX_DIM_3*sizeof(double)); if (!aMatrix[i][j][k]) { printf("*** i=3D%d, j=3D%d, k=3D%d: Could not malloc %d double\n", i,= j, k, MATRIX_DIM_3); exit(1); } } } } printf("Successfully allocated %d matrixes of %d x %d x %d double\n", M= ATRIX_COUNT, MATRIX_DIM_1, MATRIX_DIM_2, MATRIX_DIM_3); exit(0); } Best, Michael =2D-=20 Vote against SPAM - see http://www.politik-digital.de/spam/ Michael Gerdau email: mg...@te... GPG-keys available on request or at public keyserver |
From: yGREK H. <he...@ya...> - 2005-07-27 15:52:35
|
Greetings, >> Any help or comments concerning the following problem will be greatly >> appreciated. Basically, I am developing a scientific application which = uses >> Fortran 90, C, and C++ code, and does some heavy operations on matrices.= The >> details are unimportant (I think), as the program runs to completion wit= hout >> issues using 3D matrices of smallish sizes. When the matrix dimensions = are >> increased to say, 128 x 128 x 128, it appears as if I am running out of >> memory...malloc() is failing and responding with errno 12 which maps to = not >> enough memory. The problem is that according to every way I have been >> monitoring the memory of my system, I have plenty left! If anyone can c= ome up >> with any explanation, or even just something to investigate, I would ver= y much >> appreciate it...I am running on Windows XP with MinGW and Msys, using G9= 5 to >> compile and link, and using g++/gcc to compile the c++/c code. Thanks s= o much >> for everyones time. Maybe you are allocating too much memory at a time? Malloc is suitable for relatively small pieces of memory only. Use calloc and see whether the pro= blem disappears. ---------=20 With best regards, yGREK heretix <he...@ya...> http://www.i.com.ua/~richy=20 ICQ UIN: 292696772 Jabber ID : yG...@ja... |
From: Ryan H. <hu...@wf...> - 2005-07-27 16:45:04
|
Thank you for the reply...your test program does indeed allow me to grab as much memory as is available (setting MATRIX_COUNT to 120 works). I modified your test program slightly, to attempt to mimic what I am doing. Basically, I call your main function by renaming it to "test_", and placing an "extern "C"" in front of the function. I then compile it (g++ 3.4.2), and then link it with an F90 program (which is shown below, linking/compiling done with g95 4.0.1): PROGRAM MEMTEST external test call test() END PROGRAM g95 -o test.x MatrixTest.f90 MatrixTestFromF90.o -lstdc++ If I run test.x, compiled and linked as shown above, the mallocs fail fairly quickly. If I compile and link as shown below though, I can malloc more, although it still fails with plenty of memory left available: g95 -o test.x MatrixTest.f90 MatrixTestFromF90.o -lstdc++ -Wl,--heap,0x20000000 If I compile and link with -Wl,--heap,0x10000000, it gets even farther (why??). So it appears as if when I give the heap more memory, I can get over this hurdle. The strange part about this is, when I increase the heap with my program (my real one, not this test example) it fails during a dynamic memory allocation even earlier on...any ideas? Thanks again. - Ryan Quoting Michael Gerdau <mg...@te...>: > > Any help or comments concerning the following problem will be greatly > > appreciated. Basically, I am developing a scientific application which > uses > > Fortran 90, C, and C++ code, and does some heavy operations on matrices. > The > > details are unimportant (I think), as the program runs to completion > without > > issues using 3D matrices of smallish sizes. When the matrix dimensions are > > increased to say, 128 x 128 x 128, it appears as if I am running out of > > memory...malloc() is failing and responding with errno 12 which maps to not > > enough memory. The problem is that according to every way I have been > > monitoring the memory of my system, I have plenty left! If anyone can come > up > > with any explanation, or even just something to investigate, I would very > much > > appreciate it...I am running on Windows XP with MinGW and Msys, using G95 > to > > compile and link, and using g++/gcc to compile the c++/c code. Thanks so > much > > for everyones time. > > Could you provide a small testprogram that shows this problem ? > > The following code works without problems on WinXP when compiled > with mingw-gcc-3.4.4. > > #include <stdlib.h> > #include <stdio.h> > > #define MATRIX_COUNT 30 > #define MATRIX_DIM_1 128 > #define MATRIX_DIM_2 128 > #define MATRIX_DIM_3 128 > > > int main(int argc, char** argv) > { > double**** aMatrix = (double****)malloc(MATRIX_COUNT*sizeof(double***)); > if (!aMatrix) > { > printf("*** Could not malloc %d double***\n", MATRIX_COUNT); > exit(1); > } > for (int i=0; i<MATRIX_COUNT; ++i) > { > aMatrix[i] = (double***)malloc(MATRIX_DIM_1*sizeof(double**)); > if (!aMatrix[i]) > { > printf("*** i=%d: Could not malloc %d double**\n", i, MATRIX_DIM_1); > exit(1); > } > for (int j=0; j<MATRIX_DIM_1; ++j) > { > aMatrix[i][j] = (double**)malloc(MATRIX_DIM_2*sizeof(double*)); > if (!aMatrix[i][j]) > { > printf("*** i=%d, j=%d: Could not malloc %d double*\n", i, j, > MATRIX_DIM_2); > exit(1); > } > for (int k=0; k<MATRIX_DIM_2; ++k) > { > aMatrix[i][j][k] = (double*)malloc(MATRIX_DIM_3*sizeof(double)); > if (!aMatrix[i][j][k]) > { > printf("*** i=%d, j=%d, k=%d: Could not malloc %d double\n", i, j, k, > MATRIX_DIM_3); > exit(1); > } > } > } > } > printf("Successfully allocated %d matrixes of %d x %d x %d double\n", > MATRIX_COUNT, MATRIX_DIM_1, MATRIX_DIM_2, MATRIX_DIM_3); > exit(0); > } > > Best, > Michael > -- > Vote against SPAM - see http://www.politik-digital.de/spam/ > Michael Gerdau email: mg...@te... > GPG-keys available on request or at public keyserver > |
From: Michael G. <mg...@te...> - 2005-07-27 18:08:55
|
[description of testcase snipped] > If I run test.x, compiled and linked as shown above, the mallocs fail fai= rly > quickly. If I compile and link as shown below though, I can malloc more, > although it still fails with plenty of memory left available: >=20 > g95 -o test.x MatrixTest.f90 MatrixTestFromF90.o -lstdc++ -Wl,--heap,0x20= 000000 >=20 > If I compile and link with -Wl,--heap,0x10000000, it gets even farther (w= hy??).=20 > So it appears as if when I give the heap more memory, I can get over this > hurdle. The strange part about this is, when I increase the heap with my > program (my real one, not this test example) it fails during a dynamic me= mory > allocation even earlier on...any ideas? Thanks again. malloc allocs from the heap so having a large heap should allow you to malloc large amounts of memory... Contrary to what another poster wrote malloc is perfectly capable of allocating memory in large chunks (apart from that the testprogram mallocs memory in chunks of 1kB -- nothing we should have to worry even if said poster were correct). I have no real practical experience with g77 or g95 (last Fortran ** program I wrote is ages ago). However (wild guess): Could it be that the g75 frontend adds some options regarding stack/heap/static memory which effectively results in using up one of these (e.g. heap) "before time" ? (I seem to recall that Fortran uses lots of static and stack memory). While googling for 'heap+fortran' I came across some notes hinting towardsindicating possible problems here. Proposed were things like adding '-fno-automatic'. Also there was a remark regarding automatically putting stack and heap into the same memory area (i.e. trying to _NOT_ do that). Since you use malloc you probably don't have large static areas... Anyway: I suppose this is more a code generation/linker problem specific to your environment since I just tried you modified (as you described) testcase under Linux using g77 3.3.5 as well as gfortran-4.0.2 and both work nicely. Best, Michael =2D-=20 Vote against SPAM - see http://www.politik-digital.de/spam/ Michael Gerdau email: mg...@te... GPG-keys available on request or at public keyserver |