From: Roy Stogner <roystgnr@ic...>  20060722 23:53:53

On Sat, 22 Jul 2006, David Xu wrote: > I was trying to assemble the stiffness and mass matrices on a dense mesh > with 531441 nodes (40x40x40, HEX27, 3rd, HERMITE) Can I suggest trying HEX8? The HERMITE elements are unique among our higher order elements in that all their degrees of freedom are topologically associated with mesh vertices, so unless you need quadratic mapping functions you don't need HEX27 elements. Those nodes aren't reponsible for most of your memory use (the system matrix is), but every megabyte helps. > and I ran into "Out of memory" problem at the line of: > > equation_systems.init(); > > > Here's the error message: > > [0]PETSC ERROR: PetscMallocAlign() line 62 in src/sys/memory/mal.c > [0]PETSC ERROR: Out of memory. This could be due to allocating > [0]PETSC ERROR: too large an object or bleeding by not properly > [0]PETSC ERROR: destroying unneeded objects. > [0]PETSC ERROR: Memory allocated 1380430780 Memory used by process 886095872 > [0]PETSC ERROR: Try running with malloc_dump or malloc_log for info. > [0]PETSC ERROR: Memory requested 907039528! > [0]PETSC ERROR: PetscTrMallocDefault() line 191 in src/sys/memory/mtr.c > [0]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 2735 in > src/mat/impls/aij/seq/aij.c > [0]PETSC ERROR: MatCreateSeqAIJ() line 2621 in src/mat/impls/aij/seq/aij.c > [0]PETSC ERROR: User provided function() line 137 in > unknowndirectory/src/numerics/petsc_matrix.C > [unset]: aborting job: > application called MPI_Abort(comm=0x84000000, 1)  process 0 > > My question is: is it possible to assemble the matrices without having to > initilize the equation system? I'm afraid not. You could initialize a finite element object and evaluate the element matrices, but putting them into the system matrix requires that matrix and the degree of freedom structures to be initialized, and those two things are probably what's sucking up all your memory. > My goal is just to output the assembled system matrices to files and > i don't have to solve them inside libmesh. The system matrix should have 551368 degrees of freedom, most of which couple to 216 others. With 8 byte coefficients that's a hundred megs of RAM and with sparsity pattern overhead it's probably two hundred megs... but nine hundred MB seems excessive. Are you solving for more than one scalar, using a system like a generalized EigenSystem that builds more than one matrix, using complexvalued variables, or anything else that might bump up the RAM requirements? I'd appreciate it if you've got debugging tools that can give you a memory breakdown by object type and you could give use such output. It sounds like either we or PETSc might need to do a little more optimization. To work around your immediate problem, however: can you output the element matrices instead of the system matrix, and assemble them outside of libMesh? It sounds like you're using a structured mesh, which can require much less overhead than the unstructured mesh class in libMesh.  Roy 