From: David Xu <dxu@my...>  20060723 00:15:52

On 7/22/06, Roy Stogner <roystgnr@...> wrote: > > On Sat, 22 Jul 2006, David Xu wrote: > > > I was trying to assemble the stiffness and mass matrices on a dense mesh > > with 531441 nodes (40x40x40, HEX27, 3rd, HERMITE) > > Can I suggest trying HEX8? The HERMITE elements are unique among our > higher order elements in that all their degrees of freedom are > topologically associated with mesh vertices, so unless you need > quadratic mapping functions you don't need HEX27 elements. Those > nodes aren't reponsible for most of your memory use (the system matrix > is), but every megabyte helps. Are you saying HEX27 and HEX8 will give same level of accuracy in the solutions if I don't need quadratic mapping functions? > and I ran into "Out of memory" problem at the line of: > > > > equation_systems.init(); > > > > > > Here's the error message: > > > > [0]PETSC ERROR: PetscMallocAlign() line 62 in src/sys/memory/mal.c > > [0]PETSC ERROR: Out of memory. This could be due to allocating > > [0]PETSC ERROR: too large an object or bleeding by not properly > > [0]PETSC ERROR: destroying unneeded objects. > > [0]PETSC ERROR: Memory allocated 1380430780 Memory used by process > 886095872 > > [0]PETSC ERROR: Try running with malloc_dump or malloc_log for info. > > [0]PETSC ERROR: Memory requested 907039528! > > [0]PETSC ERROR: PetscTrMallocDefault() line 191 in src/sys/memory/mtr.c > > [0]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 2735 in > > src/mat/impls/aij/seq/aij.c > > [0]PETSC ERROR: MatCreateSeqAIJ() line 2621 in > src/mat/impls/aij/seq/aij.c > > [0]PETSC ERROR: User provided function() line 137 in > > unknowndirectory/src/numerics/petsc_matrix.C > > [unset]: aborting job: > > application called MPI_Abort(comm=0x84000000, 1)  process 0 > > > > My question is: is it possible to assemble the matrices without having > to > > initilize the equation system? > > I'm afraid not. You could initialize a finite element object and > evaluate the element matrices, but putting them into the system matrix > requires that matrix and the degree of freedom structures to be > initialized, and those two things are probably what's sucking up all > your memory. I see. > My goal is just to output the assembled system matrices to files and > > i don't have to solve them inside libmesh. > > The system matrix should have 551368 degrees of freedom, most of which > couple to 216 others. With 8 byte coefficients that's a hundred megs > of RAM and with sparsity pattern overhead it's probably two hundred > megs... but nine hundred MB seems excessive. Are you solving for more > than one scalar, using a system like a generalized EigenSystem that > builds more than one matrix, using complexvalued variables, or > anything else that might bump up the RAM requirements? It's for solving 2 system matrices in a realvalued generalized eigenvalue problem. I went back and tried (30x30x30, HEX27, 3rd order HERMITE). This time it didn't blew out the memory, but did take significantly longer time to assemble the matrices than (30x30x30, HEX27, 2nd order LANGRANGE). Maybe HERMITE is the problem? I'd appreciate it if you've got debugging tools that can give you a > memory breakdown by object type and you could give use such output. > It sounds like either we or PETSc might need to do a little more > optimization. I don't have any debugging tools and I have to admit that I'm a belowaverage C++ user. To work around your immediate problem, however: can you output the > element matrices instead of the system matrix, and assemble them > outside of libMesh? It sounds like you're using a structured mesh, > which can require much less overhead than the unstructured mesh class > in libMesh. How to assemble the element matrices outside of libmesh? Do you know any existing code/program that can do that? So, that would be, output each element matrix to a file and a problem should be able to read in all the element matrices from the file and assemble them to system matrices. I'm definitely interested if this is doable. Thanks, David 