Re: [Libmesh-users] mesh refinement in parallel

[John P. <jwp...@gm...>]

On Tue, Aug 30, 2011 at 9:56 AM, robert <rob...@un...> wrote:
>
> The mesh.uniformly_refine(1) takes some minutes when I do it in serial
> on my PC. However, on the BlueGene/P it doesn't come to an end (I
> aborted after 15 min). Do I have to prepare my mesh in a special way for
> use in parallel? One thing I have tried was to call the
> prepare_for_use() before the refinement. I thought this makes sense
> because the mesh is partitioned before. But, the problem remains.

Your mesh is serial unless you specifically compiled libmesh with
--enable-parmesh.

What this means is that the mesh data structure (and any refinements,
etc.) is duplicated on all processors -- convenient for writing code,
not great for conserving memory.

And it's particularly bad on multicore shared memory architectures:
you will have one copy of the mesh per MPI process, typically that
means one per core.

> Mesh Information:
>  Mesh Information:
>  mesh_dimension()=3
>  spatial_dimension()=3
>  n_nodes()=447273
>    n_local_nodes()=447273
>  n_elem()=2994336
>    n_local_elem()=2994336
>    n_active_elem()=2661632
>  n_subdomains()=2
>  n_processors()=1
>  processor_id()=0

You may be going into swap depending on how little memory is available
per core on this machine, but 3M elements isn't particularly huge....
so

.) How much memory do you have per core?
.) What is going on in 'top' or the equivalent on BGL when your code is running?

-- 
John