libmesh-users — General information for libmesh users

Re: [Libmesh-users] Libmesh sticks in prepare_for_use() when using 65536 MPIs

[Li L. <li...@ka...>]

Thank you for your reply!

I changed from "typename IdxType=unsigned int" to "typename
IdxType=dof_id_type" in parallel_sort.h (libmesh_dir/include/parallel/),
and then reconfigure and make. Errors occur as follows:
  ...
  CXXLD    libmesh_opt.la
  CXX      src/apps/getpot_parse_opt-getpot_parse.o
  CXXLD    getpot_parse-opt
/usr/bin/ld: ./.libs/libmesh_opt.so: undefined reference to
`libMesh::Parallel::Sort<Hilbert::HilbertIndices, unsigned long>::bin()'
/usr/bin/ld: ./.libs/libmesh_opt.so: undefined reference to
`libMesh::Parallel::Sort<Hilbert::HilbertIndices, unsigned
long>::Sort(libMesh::Parallel::Communicator const&,
std::vector<Hilbert::HilbertIndices,
std::allocator<Hilbert::HilbertIndices> >&)'
/usr/bin/ld: ./.libs/libmesh_opt.so: undefined reference to
`libMesh::Parallel::Sort<Hilbert::HilbertIndices, unsigned long>::sort()'
/usr/bin/ld: link errors found, deleting executable `.libs/getpot_parse-opt'
/usr/bin/sha1sum: .libs/getpot_parse-opt: No such file or directory
collect2: error: ld returned 1 exit status
Makefile:10169: recipe for target 'getpot_parse-opt' failed
make[1]: *** [getpot_parse-opt] Error 1
make[1]: Leaving directory '/lustre/project/k1305/liluo/lib/libmesh-0.9.3'
Makefile:27210: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

To John:
I am an old fan of libMesh and I am now still using libmesh0.9.3, yes, it
is very old, but due to the large amount of early codes, I hope to fix this
issue on the current version.
The config.log shows
#define DOF_ID_BYTES 8
#define PROCESSOR_ID_BYTES 4
#define SUBDOMAIN_ID_BYTES 4
#define UNIQUE_ID_BYTES 8

Best,
Li Luo

On Thu, Jan 31, 2019 at 5:49 PM John Peterson <jwp...@gm...> wrote:

>
>
> On Wed, Jan 30, 2019 at 11:08 AM Stogner, Roy H <roy...@ic...>
> wrote:
>
>>
>> On Wed, 30 Jan 2019, Li Luo wrote:
>>
>> > I am using libMesh for large scale parallelization. To enable the usage
>> of
>> > 65536 processor cores, the options
>> > --with-dof-id-bytes=8 --with-processor-id-bytes=4
>> > --with-subdomain-id-bytes=4
>> > are already used for configuration.
>>
>
> Would you mind looking in your
> $LIBMESH_DIR/include/libmesh/libmesh_config.h file and verifying that
>
> #define LIBMESH_PROCESSOR_ID_BYTES 4
>
> is indeed set?  65536 is right at the uint16_t limit, so I just want to
> rule out any funny business with the configure test that sets that value.
> Also, 4 bytes is now the default for processor_id_type so you should be
> able to safely omit that option if you are using a relatively recent
> version of libmesh. Speaking on which, can you let us know what version of
> libmesh you are using? A number of improvements have been made to the
> parallel communications routines since the last tagged release, so you'd be
> best off using libmesh master for any huge runs you are attempting.
>
> --
> John
>


-- 

Postdoctoral Fellow
Extreme Computing Research Center
King Abdullah University of Science & Technology
https://sites.google.com/site/rolyliluo/

-- 

This message and its contents, including attachments are intended solely 
for the original recipient. If you are not the intended recipient or have 
received this message in error, please notify me immediately and delete 
this message from your computer system. Any unauthorized use or 
distribution is prohibited. Please consider the environment before printing 
this email.

Re: [Libmesh-users] !libmesh_assert_pass' failed during an AMR operation

[Stogner, R. H <roy...@ic...>]

On Thu, 31 Jan 2019, Salazar De Troya, Miguel wrote:

> Would it be possible for me to send you the mesh in CPR format just
> right before calling test_unflagged() in
> MeshRefinement::refine_and_coarsen_elements()? Would that help to
> debug this issue? Not sure if the mesh in CPR actually stores the
> flags for refinement.

It doesn't, I'm afraid.  I'd love it if you could send me something
that reproduces the problem, but it wouldn't be that easy.  Maybe save
a .cpa (for ease of examination) with unique_ids and manually write
out a separate file with refinement flags?
---
Roy

Re: [Libmesh-users] !libmesh_assert_pass' failed during an AMR operation

[Salazar De T. M. <sal...@ll...>]

Would it be possible for me to send you the mesh in CPR format just right before calling test_unflagged() in MeshRefinement::refine_and_coarsen_elements()? Would that help to debug this issue? Not sure if the mesh in CPR actually stores the flags for refinement.

Thanks
Miguel

On 1/7/19, 11:51 AM, "Salazar De Troya, Miguel via Libmesh-users" <lib...@li...> wrote:

    
    
    On 1/4/19, 3:37 PM, "Stogner, Roy H" <roy...@ic...> wrote:
    
        
        On Thu, 3 Jan 2019, Salazar De Troya, Miguel via Libmesh-users wrote:
        
        >> Running an example with 10 processors (works in serial) fails with the error:
        
        >One of our examples?
    
        No. My own example so to speak. Sorry for the confusion.
        
        >> !libmesh_assert_pass' failed
        >>
        >> The program stops at libmesh_assert(test_unflagged(true)) in
        >> MeshRefinement::refine_and_coarsen_elements(). The remaining stack
        >> trace when TotalView stops is as follows (sorry for the rudimentary
        >> copy and paste, it’s all I can do now):
        >>
        >> MeshRefinement::test_unflagged (bool libmesh_dbg_var(libmesh_assert_pass))
        >> std::ostream & operator << (std::ostream & os, const Elem & e)
        >> Elem::print_info (std::ostream & os)
        >> std::string Elem::get_info ()
        >> bool DofObject::valid_id () <-- Stops here.
        
        >That's... baffling.  At that point _coarsen_elements should have
        >coarsened (and cleared flags of) anything flagged to coarsen, then
        >_refine_elements should have refined (and cleared flags of) anything
        >flagged to refine, then there should obviously be nothing left to
        >coarsen or to refine.
        
        >Hmm... the flag clearing in coarsening is actually done when the
        >parents get coarsen() called.  Is it possible that we're screwing up
        >make_coarsening_compatible somewhere and leaving a child marked
        >COARSEN without properly marking its parent COARSEN_INACTIVE?
    
       I noticed there are two test_unflagged() in MeshRefinement::refine_and_coarsen_elements(). It is the first one where the problem is (where the condition (coarsening_changed_mesh || refining_changed_mesh) is met).
        
        >> Please let me know if you need more details. I will keep investigating.
        
        >There's no output from that print_info?
    
       There is no more output. 
    
       I should apologize because the "stack trace" that I sent is not the one for the processor that failed (I'm a newbie at TotalView). This is the one for the processor that failed:
    
     libmesh_assert(!libmesh_assert_pass); in MeshRefinement::test_unflagged (bool libmesh_dbg_var(libmesh_assert_pass)) (mesh_refinement.C 467)
     libMesh::write_traceout() in MacroFunctions::report_error()
    libMesh::print_trace(traceout);
    gdb_worked = gdb_backtrace(out_stream);
    int fd = mkstemp(temp_file);
    
    and then I see two frames with what looks assembly language from the file libc.so.6
    
           
        >I don't know what I can do about this unless I can reproduce it.  I
        >could suggest some asserts that might catch the hypothesis above.
    
    Could you please suggest those assertions? Sending the code could be a bit cumbersome.
    
    Thanks
        ---
        Roy
    
    
    _______________________________________________
    Libmesh-users mailing list
    Lib...@li...
    https://lists.sourceforge.net/lists/listinfo/libmesh-users
    

Re: [Libmesh-users] Libmesh sticks in prepare_for_use() when using 65536 MPIs

[John P. <jwp...@gm...>]

On Wed, Jan 30, 2019 at 11:08 AM Stogner, Roy H <roy...@ic...>
wrote:

>
> On Wed, 30 Jan 2019, Li Luo wrote:
>
> > I am using libMesh for large scale parallelization. To enable the usage
> of
> > 65536 processor cores, the options
> > --with-dof-id-bytes=8 --with-processor-id-bytes=4
> > --with-subdomain-id-bytes=4
> > are already used for configuration.
>

Would you mind looking in your
$LIBMESH_DIR/include/libmesh/libmesh_config.h file and verifying that

#define LIBMESH_PROCESSOR_ID_BYTES 4

is indeed set?  65536 is right at the uint16_t limit, so I just want to
rule out any funny business with the configure test that sets that value.
Also, 4 bytes is now the default for processor_id_type so you should be
able to safely omit that option if you are using a relatively recent
version of libmesh. Speaking on which, can you let us know what version of
libmesh you are using? A number of improvements have been made to the
parallel communications routines since the last tagged release, so you'd be
best off using libmesh master for any huge runs you are attempting.

-- 
John

Re: [Libmesh-users] Libmesh sticks in prepare_for_use() when using 65536 MPIs

[Stogner, R. H <roy...@ic...>]

On Wed, 30 Jan 2019, Li Luo wrote:

> I am using libMesh for large scale parallelization. To enable the usage of
> 65536 processor cores, the options
> --with-dof-id-bytes=8 --with-processor-id-bytes=4
> --with-subdomain-id-bytes=4
> are already used for configuration.
>
> However, the code 'sticks' in the following fuction:
> Parallel::Sort<Hilbert::HilbertIndices> sorter (communicator,
> sorted_hilbert_keys);
> sorter.sort();

So, this looks horribly suspicious.  In parallel_sort.h:52, where it
defaults "IdxType=unsigned int", would you try "IdxType=dof_id_type"
instead?  That might be a red herring (the problem here would be
sorting at least 2^32 objects, not sorting them on at least 2^16
processors) but it sure looks like a bug to me and there's at least a
chance it's the bug affecting you.

> in the routing MeshCommunication::find_global_indices (in
> file mesh_communication_global_indices.C), which is called from routine
> Partitioner::partition_unpartitioned_elem (in file partitioner.C).

> Since libMesh calls libHilbert for this sort function, is there anything
> should be noticed for the configuration of libHilbert when using large
> scale parallelization?

Quite possibly.  We currently have libHilbert set to use 32-bit
integers internally.  That should be fine in theory (coordinates get
identified by triples of integers, and if you're using unique_id then
that disambiguates any contiguous nodes).  But cranking that up to 64
in contrib/libHilbert/include/Hilbert/FixBitVec.hpp would be what I'd
suggest as Plan B.

> Is that possible not to use libHilbert? If so, any efficiency
> degenerates?

You lose the ability to do N->M restarts with xdr/xda EquationSystems
output, and you lose compatibility of xdr/xda EquationSystems output
between with- and without- libHilbert libMesh compiles...  No
efficiency loss, though, and I don't think either of those features
can scale up to your processor count anyway, so trying with libHilbert
disabled (it's a configure option) should be your plan C.

There might also be an inadvertent libHilbert dependency somewhere
when using "slit meshes" or anything else that gives multiple
topologically distinct nodes the exact same geometric coordinates - I
don't think this is the case but it's a possible bug to watch out for.

Thanks for the bug report, and please keep us up to speed with what
works or fails to fix it!
---
Roy

[Libmesh-users] Libmesh sticks in prepare_for_use() when using 65536 MPIs

[Li L. <li...@ka...>]

Dear developers,

I am using libMesh for large scale parallelization. To enable the usage of
65536 processor cores, the options
--with-dof-id-bytes=8 --with-processor-id-bytes=4
--with-subdomain-id-bytes=4
are already used for configuration.

However, the code 'sticks' in the following fuction:
Parallel::Sort<Hilbert::HilbertIndices> sorter (communicator,
sorted_hilbert_keys);
sorter.sort();
in the routing MeshCommunication::find_global_indices (in
file mesh_communication_global_indices.C), which is called from routine
Partitioner::partition_unpartitioned_elem (in file partitioner.C).

Since libMesh calls libHilbert for this sort function, is there anything
should be noticed for the configuration of libHilbert when using large
scale parallelization?
Is that possible not to use libHilbert? If so, any efficiency degenerates?

Best,
Li Luo

-- 

This message and its contents, including attachments are intended solely 
for the original recipient. If you are not the intended recipient or have 
received this message in error, please notify me immediately and delete 
this message from your computer system. Any unauthorized use or 
distribution is prohibited. Please consider the environment before printing 
this email.

Re: [Libmesh-users] output results of very huge mesh

[Bin L. <ken...@ho...>]

Hi John,

Thanks. The code is working now. However, I still have some problem to write solutions in different time step into a single file. Therefore, I write them into separate files.

The following code is the one I used. Hopefully it can help the others.
//============ export results ================
Nemesis_IO(mesh).write(file_name.str());
Nemesis_IO(mesh).write_timestep(file_name.str(), equation_systems,
                                                         iter+1, cntT);
//========================================

I also noticed I have to write mesh into each separate file at different time step as well. On the other hand, the the version of paraview above 4.4.0 report the same error, "cannot find any mesh file". Whereas, the versions lower than that are fine to open those nemesis format files. Is there a compatibility issue?

Ken

________________________________
From: John Peterson <jwp...@gm...>
Sent: Monday, January 28, 2019 10:05 PM
To: Bin Liu
Cc: lib...@li...
Subject: Re: [Libmesh-users] output results of very huge mesh



On Mon, Jan 28, 2019 at 12:15 AM Bin Liu <ken...@ho...<mailto:ken...@ho...>> wrote:
Hi John,
The following codes are used for import and export mesh and solution.

//============= import mesh ===============
DistributedMesh mesh(init.comm());
mesh.read("BOX.cpr");
//=======================================

//============ export solution ==============
MeshSerializer serialize(mesh);
Nemesis_IO(mesh).write_equation_systems (file_name.str(),equation_systems);
//=======================================

In fact, I was trying to avoid using "MeshSerializer", since it took too much memory. Did I miss anything to ouptut solution in nemesis format? Should I write out mesh as well? Could you suggest a successful example to write out solution and mesh in parallel through each processor?

You should not use a MeshSerializer when using the Nemesis_IO format... Can you try replacing those two lines of code with the following, and let us know what happens?

Nemesis_IO(mesh).write_timestep("filename.e", equation_systems, /*timestep=*/1, /*time=*/0.);

--
John

Re: [Libmesh-users] output results of very huge mesh

[John P. <jwp...@gm...>]

On Mon, Jan 28, 2019 at 12:15 AM Bin Liu <ken...@ho...> wrote:

> Hi John,
> The following codes are used for import and export mesh and solution.
>
> //============= import mesh ===============
> DistributedMesh mesh(init.comm());
> mesh.read("BOX.cpr");
> //=======================================
>
> //============ export solution ==============
> MeshSerializer serialize(mesh);
> Nemesis_IO(mesh).write_equation_systems (file_name.str(),equation_systems);
> //=======================================
>
> In fact, I was trying to avoid using "MeshSerializer", since it took too
> much memory. Did I miss anything to ouptut solution in nemesis format?
> Should I write out mesh as well? Could you suggest a successful example to
> write out solution and mesh in parallel through each processor?
>

You should not use a MeshSerializer when using the Nemesis_IO format... Can
you try replacing those two lines of code with the following, and let us
know what happens?

Nemesis_IO(mesh).write_timestep("filename.e", equation_systems,
/*timestep=*/1, /*time=*/0.);

--
John

Re: [Libmesh-users] output results of very huge mesh

[Bin L. <ken...@ho...>]

Hi John,
The following codes are used for import and export mesh and solution.

//============= import mesh ===============
DistributedMesh mesh(init.comm());
mesh.read("BOX.cpr");
//=======================================

//============ export solution ==============
MeshSerializer serialize(mesh);
Nemesis_IO(mesh).write_equation_systems (file_name.str(),equation_systems);
//=======================================

In fact, I was trying to avoid using "MeshSerializer", since it took too much memory. Did I miss anything to ouptut solution in nemesis format? Should I write out mesh as well? Could you suggest a successful example to write out solution and mesh in parallel through each processor?

Thanks a lot
Ken





________________________________
From: John Peterson <jwp...@gm...>
Sent: Monday, January 28, 2019 6:16 AM
To: Bin Liu
Cc: lib...@li...
Subject: Re: [Libmesh-users] output results of very huge mesh



On Sat, Jan 26, 2019 at 8:47 PM Bin Liu <ken...@ho...<mailto:ken...@ho...>> wrote:
Hi John,
Indeed, I tried NemesisIO. It writes solution into multiple files in parallel. However, when I tried to open these files with paraview, the paraview reported an error, saying "Cannot find any mesh". What's could be my mistake here?

Well, the API is admittedly a bit difficult to use correctly, so it's possible you need to call it differently. It's also possible there is a bug... if you could post the code you used to write the Mesh and let us know what version of libMesh you are using, that might help us track down the issue.

--
John

Re: [Libmesh-users] output results of very huge mesh

[John P. <jwp...@gm...>]

On Sat, Jan 26, 2019 at 8:47 PM Bin Liu <ken...@ho...> wrote:

> Hi John,
> Indeed, I tried NemesisIO. It writes solution into multiple files in
> parallel. However, when I tried to open these files with paraview, the
> paraview reported an error, saying "Cannot find any mesh". What's could be
> my mistake here?
>

Well, the API is admittedly a bit difficult to use correctly, so it's
possible you need to call it differently. It's also possible there is a
bug... if you could post the code you used to write the Mesh and let us
know what version of libMesh you are using, that might help us track down
the issue.

-- 
John

Re: [Libmesh-users] output results of very huge mesh

[Bin L. <ken...@ho...>]

Hi Simone,
I will try VTK format. I have tried NemesisIO few days ago. It can output multiple files from each processor in parallel. However, when I tried to open these files with paraview, the paraview reports an error, saying "cannot find any mesh". Does it mean I need to write out mesh files in parallel as well? Do I need to serialize the mesh in this case?

Thanks
Ken

________________________________
From: Simone Rossi <sim...@gm...>
Sent: Saturday, January 26, 2019 12:17 AM
To: Bin Liu
Cc: lib...@li...
Subject: Re: [Libmesh-users] output results of very huge mesh

Hey Ken,
have you tried using VTK to export your solution? It exports a file per processor with only local information.
There may be other solutions available, but I’m using VTK to export the solution in parallel.
I hope it helps,
Simone

On Jan 25, 2019, at 2:09, Bin Liu <ken...@ho...> wrote:

> Hi,
> I have written a mpi finite element program to solve a very large 3D problem, around 50 millions nodes. I have been able to split the mesh and read these decomposed mesh from each processors. However, I have a big problem to output the results for visualization.
>
> From my understanding, I should Serialize the mesh temporarily and output the mesh and solutions together. However, this serialization of mesh is very inefficient on one processor and takes a huge amount of memory and time to generate it.
>
> Is there any way to write out the mesh and solutions together from each processor in parallel? It is not a problem for me to post-process them, as long as they are mesh and solution are exported.
>
> Thanks
> Ken
>
> _______________________________________________
> Libmesh-users mailing list
> Lib...@li...
> https://lists.sourceforge.net/lists/listinfo/libmesh-users

Re: [Libmesh-users] output results of very huge mesh

[Bin L. <ken...@ho...>]

Hi John,
Indeed, I tried NemesisIO. It writes solution into multiple files in parallel. However, when I tried to open these files with paraview, the paraview reported an error, saying "Cannot find any mesh". What's could be my mistake here?

Thanks
Ken

________________________________
From: John Peterson <jwp...@gm...>
Sent: Saturday, January 26, 2019 12:16 AM
To: Bin Liu
Cc: lib...@li...
Subject: Re: [Libmesh-users] output results of very huge mesh



On Fri, Jan 25, 2019 at 1:10 AM Bin Liu <ken...@ho...<mailto:ken...@ho...>> wrote:
Hi,
I have written a mpi finite element program to solve a very large 3D problem, around 50 millions nodes. I have been able to split the mesh and read these decomposed mesh from each processors. However, I have a big problem to output the results for visualization.

>From my understanding, I should Serialize the mesh temporarily and output the mesh and solutions together. However, this serialization of mesh is very inefficient on one processor and takes a huge amount of memory and time to generate it.

Is there any way to write out the mesh and solutions together from each processor in parallel? It is not a problem for me to post-process them, as long as they are mesh and solution are exported.

You can try using the NemesisIO object instead, which is basically an Exodus file per MPI rank and does not require serialization of the mesh.

--
John

Re: [Libmesh-users] output results of very huge mesh

[John P. <jwp...@gm...>]

On Fri, Jan 25, 2019 at 1:10 AM Bin Liu <ken...@ho...> wrote:

> Hi,
> I have written a mpi finite element program to solve a very large 3D
> problem, around 50 millions nodes. I have been able to split the mesh and
> read these decomposed mesh from each processors. However, I have a big
> problem to output the results for visualization.
>
> From my understanding, I should Serialize the mesh temporarily and output
> the mesh and solutions together. However, this serialization of mesh is
> very inefficient on one processor and takes a huge amount of memory and
> time to generate it.
>
> Is there any way to write out the mesh and solutions together from each
> processor in parallel? It is not a problem for me to post-process them, as
> long as they are mesh and solution are exported.
>

You can try using the NemesisIO object instead, which is basically an
Exodus file per MPI rank and does not require serialization of the mesh.

-- 
John

Re: [Libmesh-users] output results of very huge mesh

[Simone R. <sim...@gm...>]

Hey Ken,
have you tried using VTK to export your solution? It exports a file per processor with only local information.
There may be other solutions available, but I’m using VTK to export the solution in parallel.
I hope it helps,
Simone

On Jan 25, 2019, at 2:09, Bin Liu <ken...@ho...> wrote:

> Hi,
> I have written a mpi finite element program to solve a very large 3D problem, around 50 millions nodes. I have been able to split the mesh and read these decomposed mesh from each processors. However, I have a big problem to output the results for visualization.
> 
> From my understanding, I should Serialize the mesh temporarily and output the mesh and solutions together. However, this serialization of mesh is very inefficient on one processor and takes a huge amount of memory and time to generate it.
> 
> Is there any way to write out the mesh and solutions together from each processor in parallel? It is not a problem for me to post-process them, as long as they are mesh and solution are exported.
> 
> Thanks
> Ken
> 
> _______________________________________________
> Libmesh-users mailing list
> Lib...@li...
> https://lists.sourceforge.net/lists/listinfo/libmesh-users

[Libmesh-users] output results of very huge mesh

[Bin L. <ken...@ho...>]

Hi,
I have written a mpi finite element program to solve a very large 3D problem, around 50 millions nodes. I have been able to split the mesh and read these decomposed mesh from each processors. However, I have a big problem to output the results for visualization.

>From my understanding, I should Serialize the mesh temporarily and output the mesh and solutions together. However, this serialization of mesh is very inefficient on one processor and takes a huge amount of memory and time to generate it.

Is there any way to write out the mesh and solutions together from each processor in parallel? It is not a problem for me to post-process them, as long as they are mesh and solution are exported.

Thanks
Ken

Re: [Libmesh-users] 答复: Load-balance for unstructured mesh

[Stogner, R. H <roy...@ic...>]

On Thu, 24 Jan 2019, Xiang LI wrote:

> However, I tried your suggestion of adding the three options --load-balanced-nodes-linear 
> --load-balanced-nodes-bfs or --load_balanced_nodes_petscpartitioner to the command line,
> no response. I checked for the manual on your homepage and find that the code related to
> these three options, originally in partitioner.C line 764, had been removed in the new versions
> of libMesh. I also cannot find the related functions in the whole package.

Just the opposite, actually.  That code has been *added* in newer
libMesh, and hasn't made it into an official release yet.  You may
have to switch to using the git HEAD unless you want to wait for us to
get 1.4.0 out.

> 2. Is the metis algorithm in libMesh package serial or parallel?

Serial.  I'm afraid Parmetis is our only currently available
partitioner for a distributed mesh.  Some of the others might be able
to work on a seralized DistributedMesh.
---
Roy

Re: [Libmesh-users] Load-balance for unstructured mesh

[Stogner, R. H <roy...@ic...>]

On Wed, 23 Jan 2019, Xiang LI wrote:

> Currently I am using libMesh to solve problems with large parallel computers. I met some
> problems with the load-balance of unstructured mesh. I was wondering that if you have any
> suggestions for it.
>
> I am solving a fluid system in a cube. The mesh I use is unstructured, in which the element
> size decays exponentially as it approaches to the boundary of the cube. I generate the mesh
> by directly modifying mesh_generation.C, giving the coordinates of the dofs directly in the
> code. The code works well and I can get correct results.
>
> In order to solve it in parallel, I use --enable-parmesh in the configuration of libMesh and
> use asm preconditioners with PETSc GMRES solver. The problem is that the degrees of
> freedom seem not to be distributed "evenly"  over each processor. For example, in a test
> problem,  I tried to solve with 960 processors. In this case, some processors are given
> about 10,000 dofs, while some other processors are given about 20,000 dofs.
>
> In this case, seems that the load-balance is not correctly done for the unstructered mesh.
> I was wondering that if libMesh can handle it, and how to modify the code to make my code
> well-balanced in parallel?

I'm afraid that the default load balancing works well on elements but
uses a simplistic rule on nodes, which (except for unlikely cases such
as very high order elements, or physics whose assembly time well
outweighs solve time) typically leaves too many DoFs on lower numbered
ranks.

There are currently three alternatives, accessible with command line
options --load-balanced-nodes-linear --load-balanced-nodes-bfs or
--load_balanced_nodes_petscpartitioner, that you might experiment
with.
---
Roy

[Libmesh-users] Load-balance for unstructured mesh

[Xiang LI <ma...@us...>]

Dear Libmesh developers and users,



Currently I am using libMesh to solve problems with large parallel computers. I met some

problems with the load-balance of unstructured mesh. I was wondering that if you have any

suggestions for it.



I am solving a fluid system in a cube. The mesh I use is unstructured, in which the element

size decays exponentially as it approaches to the boundary of the cube. I generate the mesh

by directly modifying mesh_generation.C, giving the coordinates of the dofs directly in the

code. The code works well and I can get correct results.



In order to solve it in parallel, I use --enable-parmesh in the configuration of libMesh and

use asm preconditioners with PETSc GMRES solver. The problem is that the degrees of

freedom seem not to be distributed "evenly"  over each processor. For example, in a test

problem,  I tried to solve with 960 processors. In this case, some processors are given

about 10,000 dofs, while some other processors are given about 20,000 dofs.



In this case, seems that the load-balance is not correctly done for the unstructered mesh.

I was wondering that if libMesh can handle it, and how to modify the code to make my code

well-balanced in parallel?



Best, Leo

Re: [Libmesh-users] Examples in topology optimization?

[Yuxiang W. <yw...@vi...>]

Hi Bailey,

Thank you for your response! Yes - I totally agree with you. I have been
using tosca with ABAQUS and getting great results. I am sure that
developing a code to do that will take way longer time.

I am still interested in learning the math behind the theory, for the sake
of learning as well as potential (very) application-specific optimizations
once I learn well enough. I was thinking that implementing SIMP on my own
is a great starting point, but figured that if someone did it with libmesh
already it'd be nicer to know. :)

Best,
Shawn

On Tue, Jan 22, 2019 at 10:58 AM Bailey Curzadd <bcu...@gm...> wrote:

> Hi Shawn,
>
> Minimum compliance optimizations and, to some extent, stress-based
> optimizations of static structures are so well studied that developing
> libmesh code to solve those problems is a bit excessive. These
> optimizations are already implemented in most of the common commercial FE
> software packages; ANSYS, for example, has a great topology optimizer that
> can do both compliance-based and stress-based optimizations and takes
> minutes to setup. I can tell you from experience that developing your own
> code will require way more time than using a commercial solution.
>
> Bailey
>
> On Tue, Jan 22, 2019 at 5:34 PM Yuxiang Wang <yw...@vi...> wrote:
>
>> Thank you Bailey!
>>
>> I am looking for structural mechanics application. For example, we can
>> tune the distribution of isotropic materials (by tuning the element
>> density) and get minimum compliance, or minimum of maximum stress.
>>
>> Best,
>> Shawn
>>
>> On Sun, Jan 20, 2019 at 8:12 AM Bailey Curzadd <bcu...@gm...>
>> wrote:
>>
>>> I used libmesh for a topology optimization code I developed. What
>>> exactly are you looking for? What kind of optimization are you trying to
>>> implement?
>>>
>>> Bailey
>>>
>>> On Sat, Jan 19, 2019 at 8:14 PM Yuxiang Wang <yw...@vi...> wrote:
>>>
>>>> Dear all,
>>>>
>>>> Sorry for the spam - I am curious in whether anyone knows about if there
>>>> are existing examples running topology optimization with libmesh. I
>>>> googled
>>>> and found a few papers, but the source code seems to be unavailable.
>>>>
>>>> Any hints would be appreciated!
>>>>
>>>> Best,
>>>> Shawn
>>>>
>>>> --
>>>> Yuxiang "Shawn" Wang, PhD
>>>> yw...@vi...
>>>> +1 (434) 284-0836
>>>>
>>>> _______________________________________________
>>>> Libmesh-users mailing list
>>>> Lib...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/libmesh-users
>>>>
>>>
>>
>> --
>> Yuxiang "Shawn" Wang, PhD
>> yw...@vi...
>> +1 (434) 284-0836
>>
>

-- 
Yuxiang "Shawn" Wang, PhD
yw...@vi...
+1 (434) 284-0836

Re: [Libmesh-users] Examples in topology optimization?

[Bailey C. <bcu...@gm...>]

Hi Shawn,

Minimum compliance optimizations and, to some extent, stress-based
optimizations of static structures are so well studied that developing
libmesh code to solve those problems is a bit excessive. These
optimizations are already implemented in most of the common commercial FE
software packages; ANSYS, for example, has a great topology optimizer that
can do both compliance-based and stress-based optimizations and takes
minutes to setup. I can tell you from experience that developing your own
code will require way more time than using a commercial solution.

Bailey

On Tue, Jan 22, 2019 at 5:34 PM Yuxiang Wang <yw...@vi...> wrote:

> Thank you Bailey!
>
> I am looking for structural mechanics application. For example, we can
> tune the distribution of isotropic materials (by tuning the element
> density) and get minimum compliance, or minimum of maximum stress.
>
> Best,
> Shawn
>
> On Sun, Jan 20, 2019 at 8:12 AM Bailey Curzadd <bcu...@gm...>
> wrote:
>
>> I used libmesh for a topology optimization code I developed. What exactly
>> are you looking for? What kind of optimization are you trying to implement?
>>
>> Bailey
>>
>> On Sat, Jan 19, 2019 at 8:14 PM Yuxiang Wang <yw...@vi...> wrote:
>>
>>> Dear all,
>>>
>>> Sorry for the spam - I am curious in whether anyone knows about if there
>>> are existing examples running topology optimization with libmesh. I
>>> googled
>>> and found a few papers, but the source code seems to be unavailable.
>>>
>>> Any hints would be appreciated!
>>>
>>> Best,
>>> Shawn
>>>
>>> --
>>> Yuxiang "Shawn" Wang, PhD
>>> yw...@vi...
>>> +1 (434) 284-0836
>>>
>>> _______________________________________________
>>> Libmesh-users mailing list
>>> Lib...@li...
>>> https://lists.sourceforge.net/lists/listinfo/libmesh-users
>>>
>>
>
> --
> Yuxiang "Shawn" Wang, PhD
> yw...@vi...
> +1 (434) 284-0836
>

Re: [Libmesh-users] Examples in topology optimization?

[Yuxiang W. <yw...@vi...>]

Thank you Bailey!

I am looking for structural mechanics application. For example, we can tune
the distribution of isotropic materials (by tuning the element density) and
get minimum compliance, or minimum of maximum stress.

Best,
Shawn

On Sun, Jan 20, 2019 at 8:12 AM Bailey Curzadd <bcu...@gm...> wrote:

> I used libmesh for a topology optimization code I developed. What exactly
> are you looking for? What kind of optimization are you trying to implement?
>
> Bailey
>
> On Sat, Jan 19, 2019 at 8:14 PM Yuxiang Wang <yw...@vi...> wrote:
>
>> Dear all,
>>
>> Sorry for the spam - I am curious in whether anyone knows about if there
>> are existing examples running topology optimization with libmesh. I
>> googled
>> and found a few papers, but the source code seems to be unavailable.
>>
>> Any hints would be appreciated!
>>
>> Best,
>> Shawn
>>
>> --
>> Yuxiang "Shawn" Wang, PhD
>> yw...@vi...
>> +1 (434) 284-0836
>>
>> _______________________________________________
>> Libmesh-users mailing list
>> Lib...@li...
>> https://lists.sourceforge.net/lists/listinfo/libmesh-users
>>
>

-- 
Yuxiang "Shawn" Wang, PhD
yw...@vi...
+1 (434) 284-0836

Re: [Libmesh-users] [RB] How to print output error bounds in the offline stage (RB training)

[<ss...@pu...>]

Thank you, David.

 

I also thought about changing the return value of the “Real RBEvaluation::rb_solve(unsigned int N),” but I did not edit it because of the fear of an error.

Since you said there is no other way, I will try to add the code in “rb_construction.C” or “rb_evaluation.C” instead of changing the return value.

 

I always appreciate your help.

 

Best regards,

SKang

 

 

 

 

From: David Knezevic <dav...@ak...> 
Sent: Monday, January 21, 2019 10:05 PM
To: 강신성 <ss...@pu...>
Cc: libmesh-users <lib...@li...>
Subject: Re: [Libmesh-users] [RB] How to print output error bounds in the offline stage (RB training)

 

On Sun, Jan 20, 2019 at 10:46 PM <ss...@pu... <mailto:ss...@pu...> > wrote:

Thank you for a reply, David.

 

I know the code, "libMesh::out << "Maximum error bound is " << training_greedy_error << std::endl" in rb_construction.C in order to print error bounds during the RB training.

However, the "training_greedy_error" is a "Maximum solution error bound," not a "Maximum output error bound."

 

If you look at the "Real RBEvaluation::rb_solve(unsigned int N)," variables associated with "Maximum solution error bound" and "Maximum output error bound" are as follows.

========================================================================================

* "abs_error_bound" at line 272

>> This is the "solution error bound," which is already in print at every step of RB training.

 

* "RB_output_error_bounds" at line 277

>> This is the "output error bound," the error I want to print at every step of RB training.

=====================================================================================

 

In other words, my question is as follows:

Can I print a "Maximum output solution error bound" in each basis dimension?

 

I look forward to hearing from you.

 

Oh OK, I see, you want to print the output error bounds.

 

This should be possible, but the code isn't set up to do that at the moment. I think you'd have to edit rb_construction.C in order to achieve this. You should be able to call the same code that generates the output error indicator during an online solve, and use that to print out the output indicator during training, but I'm afraid that I don't have time to look into the details myself at the moment.

 

Best regards,

David

 

 

From: David Knezevic <dav...@ak... <mailto:dav...@ak...> > 

Sent: Friday, January 18, 2019 11:57 PM
To: 강신성 <ss...@pu... <mailto:ss...@pu...> >
Cc: libmesh-users <lib...@li... <mailto:lib...@li...> >
Subject: Re: [Libmesh-users] [RB] How to print output error bounds in the offline stage (RB training)

 

On Fri, Jan 18, 2019 at 1:37 AM <ss...@pu... <mailto:ss...@pu...> > wrote:

Hello, all.



I have a question about printing output error bounds in the offline stage.



We can see that maximum solution error bounds of each step in the RB
training process, such as

==================================================================

.

---- Basis dimension: 1 ----

Performing RB solves on training set

Maximum error bound is 1.07082



Performing truth solve at parameter:

.

==================================================================



Besides, using the code "rb_eval.RB_output_error_bounds[0]" in the main
program, I printed the output error bound after the RB training.

However, I cannot find any way to print all output error bounds during an RB
training.

In other words, I want to see output error bound corresponding the basis
dimension similar to the above code.



Could you please tell me how to print them?



I always thank you for your help.

 

I'm not really sure what you want to print... but the print statement you referred to above is in rb_construction.C, where it does "libMesh::out << "Maximum error bound is " << training_greedy_error << std::endl << std::endl;". I guess you could do something like that in your code, or add libMesh::out calls in your local libMesh in order to print more info?

 

Best,

David

Re: [Libmesh-users] [RB] How to print output error bounds in the offline stage (RB training)

[David K. <dav...@ak...>]

On Sun, Jan 20, 2019 at 10:46 PM <ss...@pu...> wrote:

> Thank you for a reply, David.
>
>
>
> I know the code, "libMesh::out << "Maximum error bound is " <<
> training_greedy_error << std::endl" in rb_construction.C in order to print
> error bounds during the RB training.
>
> However, the "training_greedy_error" is a "Maximum solution error bound,"
> not a "Maximum output error bound."
>
>
>
> If you look at the "Real RBEvaluation::rb_solve(unsigned int N),"
> variables associated with "Maximum solution error bound" and "Maximum
> output error bound" are as follows.
>
>
> ========================================================================================
>
> * "abs_error_bound" at line 272
>
> >> This is the "solution error bound," which is already in print at every
> step of RB training.
>
>
>
> * "RB_output_error_bounds" at line 277
>
> >> This is the "output error bound," the error I want to print at every
> step of RB training.
>
>
> =====================================================================================
>
>
>
> In other words, my question is as follows:
>
> Can I print a "Maximum output solution error bound" in each basis
> dimension?
>
>
>
> I look forward to hearing from you.
>

Oh OK, I see, you want to print the output error bounds.

This should be possible, but the code isn't set up to do that at the
moment. I think you'd have to edit rb_construction.C in order to achieve
this. You should be able to call the same code that generates the output
error indicator during an online solve, and use that to print out the
output indicator during training, but I'm afraid that I don't have time to
look into the details myself at the moment.

Best regards,
David


*From:* David Knezevic <dav...@ak...>
>
> *Sent:* Friday, January 18, 2019 11:57 PM
> *To:* 강신성 <ss...@pu...>
> *Cc:* libmesh-users <lib...@li...>
> *Subject:* Re: [Libmesh-users] [RB] How to print output error bounds in
> the offline stage (RB training)
>
>
>
> On Fri, Jan 18, 2019 at 1:37 AM <ss...@pu...> wrote:
>
> Hello, all.
>
>
>
> I have a question about printing output error bounds in the offline stage.
>
>
>
> We can see that maximum solution error bounds of each step in the RB
> training process, such as
>
> ==================================================================
>
> .
>
> ---- Basis dimension: 1 ----
>
> Performing RB solves on training set
>
> Maximum error bound is 1.07082
>
>
>
> Performing truth solve at parameter:
>
> .
>
> ==================================================================
>
>
>
> Besides, using the code "rb_eval.RB_output_error_bounds[0]" in the main
> program, I printed the output error bound after the RB training.
>
> However, I cannot find any way to print all output error bounds during an
> RB
> training.
>
> In other words, I want to see output error bound corresponding the basis
> dimension similar to the above code.
>
>
>
> Could you please tell me how to print them?
>
>
>
> I always thank you for your help.
>
>
>
> I'm not really sure what you want to print... but the print statement you
> referred to above is in rb_construction.C, where it does "libMesh::out <<
> "Maximum error bound is " << training_greedy_error << std::endl <<
> std::endl;". I guess you could do something like that in your code, or add
> libMesh::out calls in your local libMesh in order to print more info?
>
>
>
> Best,
>
> David
>

Re: [Libmesh-users] [RB] How to print output error bounds in the offline stage (RB training)

[<ss...@pu...>]

Thank you for a reply, David.

 

I know the code, "libMesh::out << "Maximum error bound is " << training_greedy_error << std::endl" in rb_construction.C in order to print error bounds during the RB training.

However, the "training_greedy_error" is a "Maximum solution error bound," not a "Maximum output error bound."

 

If you look at the "Real RBEvaluation::rb_solve(unsigned int N)," variables associated with "Maximum solution error bound" and "Maximum output error bound" are as follows.

========================================================================================

* "abs_error_bound" at line 272

>> This is the "solution error bound," which is already in print at every step of RB training.

 

* "RB_output_error_bounds" at line 277

>> This is the "output error bound," the error I want to print at every step of RB training.

=====================================================================================

 

In other words, my question is as follows:

Can I print a "Maximum output solution error bound" in each basis dimension?

 

I look forward to hearing from you.

 

Best regards,

 

SKang

 

 

From: David Knezevic <dav...@ak...> 
Sent: Friday, January 18, 2019 11:57 PM
To: 강신성 <ss...@pu...>
Cc: libmesh-users <lib...@li...>
Subject: Re: [Libmesh-users] [RB] How to print output error bounds in the offline stage (RB training)

 

On Fri, Jan 18, 2019 at 1:37 AM <ss...@pu... <mailto:ss...@pu...> > wrote:

Hello, all.



I have a question about printing output error bounds in the offline stage.



We can see that maximum solution error bounds of each step in the RB
training process, such as

==================================================================

.

---- Basis dimension: 1 ----

Performing RB solves on training set

Maximum error bound is 1.07082



Performing truth solve at parameter:

.

==================================================================



Besides, using the code "rb_eval.RB_output_error_bounds[0]" in the main
program, I printed the output error bound after the RB training.

However, I cannot find any way to print all output error bounds during an RB
training.

In other words, I want to see output error bound corresponding the basis
dimension similar to the above code.



Could you please tell me how to print them?



I always thank you for your help.

 

I'm not really sure what you want to print... but the print statement you referred to above is in rb_construction.C, where it does "libMesh::out << "Maximum error bound is " << training_greedy_error << std::endl << std::endl;". I guess you could do something like that in your code, or add libMesh::out calls in your local libMesh in order to print more info?

 

Best,

David

Re: [Libmesh-users] Examples in topology optimization?

[Bailey C. <bcu...@gm...>]

I used libmesh for a topology optimization code I developed. What exactly
are you looking for? What kind of optimization are you trying to implement?

Bailey

On Sat, Jan 19, 2019 at 8:14 PM Yuxiang Wang <yw...@vi...> wrote:

> Dear all,
>
> Sorry for the spam - I am curious in whether anyone knows about if there
> are existing examples running topology optimization with libmesh. I googled
> and found a few papers, but the source code seems to be unavailable.
>
> Any hints would be appreciated!
>
> Best,
> Shawn
>
> --
> Yuxiang "Shawn" Wang, PhD
> yw...@vi...
> +1 (434) 284-0836
>
> _______________________________________________
> Libmesh-users mailing list
> Lib...@li...
> https://lists.sourceforge.net/lists/listinfo/libmesh-users
>