On 7/22/06, Roy Stogner <roystgnr@ices.utexas.edu> wrote:
On Sat, 22 Jul 2006, David Xu wrote:

> I was trying to assemble the stiffness and mass matrices on a dense mesh
> with 531441 nodes (40x40x40, HEX27, 3rd, HERMITE)

Can I suggest trying HEX8?  The HERMITE elements are unique among our
higher order elements in that all their degrees of freedom are
topologically associated with mesh vertices, so unless you need
quadratic mapping functions you don't need HEX27 elements.  Those
nodes aren't reponsible for most of your memory use (the system matrix
is), but every megabyte helps.

Are you saying HEX27 and HEX8 will give same level of accuracy in the solutions if I don't need quadratic mapping functions?

> and I ran into "Out of memory" problem at the line of:
> equation_systems.init();
> Here's the error message:
> [0]PETSC ERROR: PetscMallocAlign() line 62 in src/sys/memory/mal.c
> [0]PETSC ERROR: Out of memory. This could be due to allocating
> [0]PETSC ERROR: too large an object or bleeding by not properly
> [0]PETSC ERROR: destroying unneeded objects.
> [0]PETSC ERROR: Memory allocated 1380430780 Memory used by process 886095872
> [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
> [0]PETSC ERROR: Memory requested 907039528!
> [0]PETSC ERROR: PetscTrMallocDefault() line 191 in src/sys/memory/mtr.c
> [0]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 2735 in
> src/mat/impls/aij/seq/aij.c
> [0]PETSC ERROR: MatCreateSeqAIJ() line 2621 in src/mat/impls/aij/seq/aij.c
> [0]PETSC ERROR: User provided function() line 137 in
> unknowndirectory/src/numerics/petsc_matrix.C
> [unset]: aborting job:
> application called MPI_Abort(comm=0x84000000, 1) - process 0
> My question is: is it possible to assemble the matrices without having to
> initilize the equation system?

I'm afraid not.  You could initialize a finite element object and
evaluate the element matrices, but putting them into the system matrix
requires that matrix and the degree of freedom structures to be
initialized, and those two things are probably what's sucking up all
your memory.

I see.

> My goal is just to output the assembled system matrices to files and
> i don't have to solve them inside libmesh.

The system matrix should have 551368 degrees of freedom, most of which
couple to 216 others.  With 8 byte coefficients that's a hundred megs
of RAM and with sparsity pattern overhead it's probably two hundred
megs... but nine hundred MB seems excessive.  Are you solving for more
than one scalar, using a system like a generalized EigenSystem that
builds more than one matrix, using complex-valued variables, or
anything else that might bump up the RAM requirements?

It's for solving 2 system matrices in a real-valued generalized eigenvalue problem. I went back and tried (30x30x30, HEX27, 3rd order HERMITE). This time it didn't blew out the memory, but did take significantly longer time to assemble the matrices than (30x30x30, HEX27, 2nd order LANGRANGE). Maybe HERMITE is the problem?

I'd appreciate it if you've got debugging tools that can give you a
memory breakdown by object type and you could give use such output.
It sounds like either we or PETSc might need to do a little more

I don't have any debugging tools and I have to  admit that I'm a below-average  C++ user.

To work around your immediate problem, however: can you output the
element matrices instead of the system matrix, and assemble them
outside of libMesh?  It sounds like you're using a structured mesh,
which can require much less overhead than the unstructured mesh class
in libMesh.

How to assemble the element matrices outside  of libmesh? Do you know any existing code/program that can do that?  So, that would be, output each element matrix to a file and a problem should be able to read in all the element matrices from the file and assemble them to system matrices. I'm definitely interested if this is doable.