From: <Rob...@un...> - 2011-10-29 08:56:10
|
In serial every processor has a copy of the mesh, right? Is it possible to have such a copy only on one processor and not on the others - this would already free some space. > You'd either need to borrow a large memory machine for > long enough to read in the Tetgen file and rewrite it as > Nemesis, or you'd need to modify the TetGenIO to support > reading a pre-parallelized ParallelMesh. I had a look at the class Nemesis_IO. There seem to be some differences between the online documentation and the class provided in version 0.7.1. The documentation has the following constructor: Nemesis_IO (MeshBase &mesh) (see http://libmesh.sourceforge.net/doxygen/classlibMesh_1_1Nemesis__IO.php) but in the file libmesh/src/mesh/nemesis_io.C one can find the following constructor: Nemesis_IO::Nemesis_IO (ParallelMesh& mesh) Thus, I tried the following: Mesh mesh; TetGenIO TETGEN(mesh); { // read TETGEN file OStringStream inmesh; inmesh << mesh_dir << ReadMesh[actual_pulse]<<".ele"; TETGEN.read(inmesh.str().c_str()); } mesh.prepare_for_use(); ParallelMesh parallelmesh(mesh); Nemesis_IO NEM (parallelmesh); NEM.write("out/nemmesh"); The function 'write' is listed in the documentation. Nevertheless, I get the following error when compiling: trySplitNemesis.C:109:6: error: class libMesh::Nemesis_IO has no member named write Could you give me some lines of example code how you would do it? Thank you, Robert |
From: <Rob...@un...> - 2011-10-30 11:19:47
|
> >> You'd either need to borrow a large memory machine for > >> long enough to read in the Tetgen file and rewrite it > as >> Nemesis, or you'd need to modify the TetGenIO to > support >> reading a pre-parallelized ParallelMesh. > > > > I had a look at the class Nemesis_IO. There seem to be > > some differences between the online documentation and > > the class provided in version 0.7.1. > > You may need to use the SVN head; I forget how recent the > latest ParallelMesh development was, but it's quite > possible that it wasn't working yet in 0.7.1. I have tried with the SVN version. The following code for writing the nemesis file worked fine: Mesh mesh; TetGenIO TETGEN(mesh); { // read TETGEN file OStringStream inmesh; inmesh << mesh_dir << ReadMesh[actual_pulse]<<".ele"; TETGEN.read(inmesh.str().c_str()); } mesh.prepare_for_use(); Nemesis_IO NEM (mesh); NEM.write("out/nemmesh"); Afterwards, when I want to read it on one processor: ./tryRead-opt Mesh mesh; Nemesis_IO NEM (mesh); NEM.read("nemmesh.1.0"); reading seems to work but, at the end of the program I get the following error: -------------------------------------------------------------------------------- Error flushing buffers to file. [0] src/mesh/exodusII_io_helper.C, line 151, compiled Oct 30 2011 at 11:09:28 terminate called after throwing an instance of 'libMesh::LogicError' what(): Error in libMesh internal logic [tomo-PaineCalc:28510] *** Process received signal *** [tomo-PaineCalc:28510] Signal: Aborted (6) [tomo-PaineCalc:28510] Signal code: (-6) [tomo-PaineCalc:28510] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) [0x7f0606345c60] [tomo-PaineCalc:28510] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f0605115d05] [tomo-PaineCalc:28510] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x186) [0x7f0605119ab6] [tomo-PaineCalc:28510] [ 3] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x11d) [0x7f06055316dd] [tomo-PaineCalc:28510] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9926) [0x7f060552f926] [tomo-PaineCalc:28510] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9953) [0x7f060552f953] [tomo-PaineCalc:28510] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9a5e) [0x7f060552fa5e] [tomo-PaineCalc:28510] [ 7] /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu_opt/libmesh.so(_ZN7libMesh18ExodusII_IO_Helper9check_errEiSs+0x1b3) [0x7f060adbaa73] [tomo-PaineCalc:28510] [ 8] /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu_opt/libmesh.so(_ZN7libMesh17Nemesis_IO_HelperD1Ev+0x48) [0x7f060ae88978] [tomo-PaineCalc:28510] [ 9] /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu_opt/libmesh.so(_ZN7libMesh17Nemesis_IO_HelperD0Ev+0x9) [0x7f060ae89599] [tomo-PaineCalc:28510] [10] /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu_opt/libmesh.so(_ZN7libMesh10Nemesis_IOD1Ev+0x2d) [0x7f060ae7c41d] [tomo-PaineCalc:28510] [11] ./tryRead-opt(main+0x202) [0x4589e2] [tomo-PaineCalc:28510] [12] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xff) [0x7f0605100eff] [tomo-PaineCalc:28510] [13] ./tryRead-opt() [0x458719] [tomo-PaineCalc:28510] *** End of error message *** Aborted -------------------------------------------------------------------------------- However, if I want to read it when running with 'mpiexec -np 8 ./tryRead-opt' I get a long error message - see at the end of the mail. For configuring of libmesh I used the following options: ./configure --enable-slepc --enable-mpi --enable-parmesh --enable-petsc Do I maybe have to change something there? thanks, Robert -------------------------------------------------------------------------------- what(): Error in libMesh internal logic [tomo-PaineCalc:28499] *** Process received signal *** [tomo-PaineCalc:28499] Signal: Aborted (6) [tomo-PaineCalc:28499] Signal code: (-6) [tomo-PaineCalc:28500] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) [0x7f9eac762c60] [tomo-PaineCalc:28500] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f9eab532d05] [tomo-PaineCalc:28500] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x186) [0x7f9eab536ab6] [tomo-PaineCalc:28500] [ 3] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x11d) [0x7f9eab94e6dd] [tomo-PaineCalc:28500] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9926) [0x7f9eab94c926] [tomo-PaineCalc:28500] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9953) [0x7f9eab94c953] [tomo-PaineCalc:28500] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9a5e) [0x7f9eab94ca5e] [tomo-PaineCalc:28500] [ 7] /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu_opt/libmesh.so(_ZN7libMesh18ExodusII_IO_Helper9check_errEiSs+0x1b3) [0x7f9eb11d7a73] [tomo-PaineCalc:28500] [ 8] /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu_opt/libmesh.so(_ZN7libMesh18ExodusII_IO_Helper4openEPKc+0x99) [0x7f9eb11da299] [tomo-PaineCalc:28500] [ 9] /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu_opt/libmesh.so(_ZN7libMesh10Nemesis_IO4readERKSs+0xe5) [0x7f9eb129a955] [tomo-PaineCalc:28500] [10] ./tryRead-opt(main+0xe6) [0x4588c6] [tomo-PaineCalc:28500] [11] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xff) [0x7f9eab51deff] [tomo-PaineCalc:28500] [12] ./tryRead-opt() [0x458719] [tomo-PaineCalc:28503] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) [0x7f497c5fbc60] [tomo-PaineCalc:28503] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f497b3cbd05] [tomo-PaineCalc:28503] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x186) [0x7f497b3cfab6] [tomo-PaineCalc:28503] [ 3] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x11d) [0x7f497b7e76dd] [tomo-PaineCalc:28503] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9926) [0x7f497b7e5926] [tomo-PaineCalc:28503] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9953) [0x7f497b7e5953] [tomo-PaineCalc:28503] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9a5e) [0x7f497b7e5a5e] [tomo-PaineCalc:28503] [ 7] /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu_opt/libmesh.so(_ZN7libMesh18ExodusII_IO_Helper9check_errEiSs+0x1b3) [0x7f4981070a73] [tomo-PaineCalc:28503] [ 8] /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu_opt/libmesh.so(_ZN7libMesh18ExodusII_IO_Helper4openEPKc+0x99) [0x7f4981073299] [tomo-PaineCalc:28503] [ 9] /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu_opt/libmesh.so(_ZN7libMesh10Nemesis_IO4readERKSs+0xe5) [0x7f4981133955] [tomo-PaineCalc:28503] [10] ./tryRead-opt(main+0xe6) [0x4588c6] [tomo-PaineCalc:28503] [11] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xff) [0x7f497b3b6eff] [tomo-PaineCalc:28503] [12] ./tryRead-opt() [0x458719] [tomo-PaineCalc:28503] *** End of error message *** [tomo-PaineCalc:28502] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) [0x7fb12c0c9c60] [tomo-PaineCalc:28502] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7fb12ae99d05] [tomo-PaineCalc:28502] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x186) [0x7fb12ae9dab6]-------------------------------------------------------------------------- mpiexec noticed that process rank 3 with PID 28501 on node tomo-PaineCalc exited on signal 6 (Aborted). -------------------------------------------------------------------------- 5 total processes killed (some possibly by mpiexec during cleanup) ... and so on :-| -------------------------------------------------------------------------------- |
From: <Rob...@un...> - 2011-10-31 15:31:39
|
I have spent more than 20 hours now with the simple problem to read a tetgen file and write it with Nemesis_IO i such a way that I can re-read it in parallel. My mesh fits on one processor - but I can only use one because a copy of it on a second processor wouldn´t fit anymore. I read in the mesh and tried every method I found to partition it - but I never succeeded to write the nemesis files for the different processors. I only get one file which, I can't read in i parallel because the program searches the files mesh.8.0, mesh.8.1, mesh.8.2, ... . How can I get these files when reading in the serial tetgen mesh on one processor? I am sure it is quite easy - unfortunately I don't progress any more without help. thank you, Robert > > >> You'd either need to borrow a large memory machine > > for >> long enough to read in the Tetgen file and > > rewrite it as >> Nemesis, or you'd need to modify the > > TetGenIO to support >> reading a pre-parallelized > > ParallelMesh. > > > > I had a look at the class Nemesis_IO. There seem to be > > > some differences between the online documentation and > > > the class provided in version 0.7.1. > > > > You may need to use the SVN head; I forget how recent > > the latest ParallelMesh development was, but it's quite > > possible that it wasn't working yet in 0.7.1. > > I have tried with the SVN version. The following code for > writing the nemesis file worked fine: > > Mesh mesh; > > TetGenIO TETGEN(mesh); > > { // read TETGEN file > OStringStream inmesh; > inmesh << mesh_dir << ReadMesh[actual_pulse]<<".ele"; > TETGEN.read(inmesh.str().c_str()); > } > > mesh.prepare_for_use(); > > > Nemesis_IO NEM (mesh); > > NEM.write("out/nemmesh"); > > > Afterwards, when I want to read it on one processor: > /tryRead-opt > > Mesh mesh; > Nemesis_IO NEM (mesh); > NEM.read("nemmesh.1.0"); > > reading seems to work but, at the end of the program I get > the following error: > > ---------------------------------------------------------- > ---------------------- Error flushing buffers to file. > [0] src/mesh/exodusII_io_helper.C, line 151, compiled Oct > 30 2011 at 11:09:28 > terminate called after throwing an instance of > 'libMesh::LogicError' > what(): Error in libMesh internal logic > [tomo-PaineCalc:28510] *** Process received signal *** > [tomo-PaineCalc:28510] Signal: Aborted (6) > [tomo-PaineCalc:28510] Signal code: (-6) > [tomo-PaineCalc:28510] [ 0] > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) > [0x7f0606345c60] > [tomo-PaineCalc:28510] [ 1] > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) > [0x7f0605115d05] > [tomo-PaineCalc:28510] [ 2] > /lib/x86_64-linux-gnu/libc.so.6(abort+0x186) > [0x7f0605119ab6] > [tomo-PaineCalc:28510] [ 3] > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__ > verbose_terminate_handlerEv+0x11d) [0x7f06055316dd] > [tomo-PaineCalc:28510] [ 4] > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9926) > [0x7f060552f926] > [tomo-PaineCalc:28510] [ 5] > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9953) > [0x7f060552f953] > [tomo-PaineCalc:28510] [ 6] > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9a5e) > [0x7f060552fa5e] > [tomo-PaineCalc:28510] [ 7] > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > _opt/libmesh.so(_ZN7libMesh18ExodusII_IO_Helper9check_errE > iSs+0x1b3) [0x7f060adbaa73] > [tomo-PaineCalc:28510] [ 8] > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > _opt/libmesh.so(_ZN7libMesh17Nemesis_IO_HelperD1Ev+0x48) > [0x7f060ae88978] [tomo-PaineCalc:28510] [ 9] > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > _opt/libmesh.so(_ZN7libMesh17Nemesis_IO_HelperD0Ev+0x9) > [0x7f060ae89599] [tomo-PaineCalc:28510] [10] > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > _opt/libmesh.so(_ZN7libMesh10Nemesis_IOD1Ev+0x2d) > [0x7f060ae7c41d] [tomo-PaineCalc:28510] [11] > /tryRead-opt(main+0x202) [0x4589e2] > [tomo-PaineCalc:28510] [12] > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xff) > [0x7f0605100eff] > [tomo-PaineCalc:28510] [13] ./tryRead-opt() [0x458719] > [tomo-PaineCalc:28510] *** End of error message *** > Aborted > > ---------------------------------------------------------- > ---------------------- > > > > > > However, if I want to read it when running with 'mpiexec > -np 8 ./tryRead-opt' > I get a long error message - see at the end of the mail. > > For configuring of libmesh I used the following options: > /configure --enable-slepc --enable-mpi --enable-parmesh > --enable-petsc > > Do I maybe have to change something there? > thanks, Robert > > > ---------------------------------------------------------- > ---------------------- > what(): Error in libMesh internal logic > [tomo-PaineCalc:28499] *** Process received signal *** > [tomo-PaineCalc:28499] Signal: Aborted (6) > [tomo-PaineCalc:28499] Signal code: (-6) > [tomo-PaineCalc:28500] [ 0] > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) > [0x7f9eac762c60] > [tomo-PaineCalc:28500] [ 1] > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) > [0x7f9eab532d05] > [tomo-PaineCalc:28500] [ 2] > /lib/x86_64-linux-gnu/libc.so.6(abort+0x186) > [0x7f9eab536ab6] > [tomo-PaineCalc:28500] [ 3] > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__ > verbose_terminate_handlerEv+0x11d) [0x7f9eab94e6dd] > [tomo-PaineCalc:28500] [ 4] > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9926) > [0x7f9eab94c926] > [tomo-PaineCalc:28500] [ 5] > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9953) > [0x7f9eab94c953] > [tomo-PaineCalc:28500] [ 6] > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9a5e) > [0x7f9eab94ca5e] > [tomo-PaineCalc:28500] [ 7] > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > _opt/libmesh.so(_ZN7libMesh18ExodusII_IO_Helper9check_errE > iSs+0x1b3) [0x7f9eb11d7a73] > [tomo-PaineCalc:28500] [ 8] > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > _opt/libmesh.so(_ZN7libMesh18ExodusII_IO_Helper4openEPKc+0 > x99) [0x7f9eb11da299] > [tomo-PaineCalc:28500] [ 9] > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > _opt/libmesh.so(_ZN7libMesh10Nemesis_IO4readERKSs+0xe5) > [0x7f9eb129a955] [tomo-PaineCalc:28500] [10] > /tryRead-opt(main+0xe6) [0x4588c6] > [tomo-PaineCalc:28500] [11] > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xff) > [0x7f9eab51deff] > [tomo-PaineCalc:28500] [12] ./tryRead-opt() [0x458719] > [tomo-PaineCalc:28503] [ 0] > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) > [0x7f497c5fbc60] > [tomo-PaineCalc:28503] [ 1] > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) > [0x7f497b3cbd05] > [tomo-PaineCalc:28503] [ 2] > /lib/x86_64-linux-gnu/libc.so.6(abort+0x186) > [0x7f497b3cfab6] > [tomo-PaineCalc:28503] [ 3] > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__ > verbose_terminate_handlerEv+0x11d) [0x7f497b7e76dd] > [tomo-PaineCalc:28503] [ 4] > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9926) > [0x7f497b7e5926] > [tomo-PaineCalc:28503] [ 5] > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9953) > [0x7f497b7e5953] > [tomo-PaineCalc:28503] [ 6] > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9a5e) > [0x7f497b7e5a5e] > [tomo-PaineCalc:28503] [ 7] > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > _opt/libmesh.so(_ZN7libMesh18ExodusII_IO_Helper9check_errE > iSs+0x1b3) [0x7f4981070a73] > [tomo-PaineCalc:28503] [ 8] > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > _opt/libmesh.so(_ZN7libMesh18ExodusII_IO_Helper4openEPKc+0 > x99) [0x7f4981073299] > [tomo-PaineCalc:28503] [ 9] > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > _opt/libmesh.so(_ZN7libMesh10Nemesis_IO4readERKSs+0xe5) > [0x7f4981133955] [tomo-PaineCalc:28503] [10] > /tryRead-opt(main+0xe6) [0x4588c6] > [tomo-PaineCalc:28503] [11] > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xff) > [0x7f497b3b6eff] > [tomo-PaineCalc:28503] [12] ./tryRead-opt() [0x458719] > [tomo-PaineCalc:28503] *** End of error message *** > [tomo-PaineCalc:28502] [ 0] > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) > [0x7fb12c0c9c60] > [tomo-PaineCalc:28502] [ 1] > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) > [0x7fb12ae99d05] > [tomo-PaineCalc:28502] [ 2] > /lib/x86_64-linux-gnu/libc.so.6(abort+0x186) > [0x7fb12ae9dab6]------------------------------------------ > -------------------------------- mpiexec noticed that > process rank 3 with PID 28501 on node tomo-PaineCalc > exited on signal 6 (Aborted). > ---------------------------------------------------------- > ---------------- 5 total processes killed (some possibly > by mpiexec during cleanup) > > > > .. and so on :-| > > > ---------------------------------------------------------- > ---------------------- > > ---------------------------------------------------------- > -------------------- Get your Android app more play: Bring > it to the BlackBerry PlayBook in minutes. BlackBerry App > World now supports Android Apps for the > BlackBerry® PlayBook. Discover just how easy and > simple it is! http://p.sf.net/sfu/android-dev2dev > _______________________________________________ > Libmesh-users mailing list > Lib...@li... > https://lists.sourceforge.net/lists/listinfo/libmesh-users |
From: Roy S. <roy...@ic...> - 2011-11-01 19:40:06
|
On Mon, 31 Oct 2011, Rob...@un... wrote: > I never succeeded to write the nemesis files for the different > processors. I only get one file which, I can't read in i parallel > because the program searches the files mesh.8.0, mesh.8.1, mesh.8.2, > ... . How can I get these files when reading in the serial tetgen > mesh on one processor? Yeah; right now libMesh assumes that you're reading in a Nemesis file that's already partitioned on n_processors(), and libMesh only lets you write out a Nemesis file that's using your current partitioning on n_processors(). Kind of a catch-22 if you want to write with n_processors()==1 then read with n_processors()==8. For N-to-M restarts, I think our only option is a third-party tool: http://sourceforge.net/projects/seacas/files/ There's a utility called nem_spread that you can build from the seacas.tar.bz2 sources there; I've never used it myself but IIRC the INL folks have been using it successfully with libMesh Nemesis I/O. --- Roy |
From: Derek G. <fri...@gm...> - 2011-11-01 19:54:44
|
I apologize for not responding to these emails earlier…. Here is a basic algorithm for using NEMESIS when starting with a file format that _isn't_ already EXODUS _and_ your mesh is so large that you can't read it on "n" number of processors. 1. Read file into libMesh 2. Write out Exodus file 3. Run nem_spread on the Exodus file to generate Nemesis files (one for each processor) 4. With a version of your code compiled against a libMesh that is using parallel mesh… use the nemesis reader to read the nemesis files. nem_spread is a _serial_ process that produces "n" number of nemesis files (it is doing an "offline" partitioning). You will need a machine with quite a bit of RAM. Derek On Nov 1, 2011, at 1:39 PM, Roy Stogner wrote: > > On Mon, 31 Oct 2011, Rob...@un... wrote: > >> I never succeeded to write the nemesis files for the different >> processors. I only get one file which, I can't read in i parallel >> because the program searches the files mesh.8.0, mesh.8.1, mesh.8.2, >> ... . How can I get these files when reading in the serial tetgen >> mesh on one processor? > > Yeah; right now libMesh assumes that you're reading in a Nemesis file > that's already partitioned on n_processors(), and libMesh only lets > you write out a Nemesis file that's using your current partitioning on > n_processors(). Kind of a catch-22 if you want to write with > n_processors()==1 then read with n_processors()==8. > > For N-to-M restarts, I think our only option is a third-party tool: > > http://sourceforge.net/projects/seacas/files/ > > There's a utility called nem_spread that you can build from the > seacas.tar.bz2 sources there; I've never used it myself but IIRC the > INL folks have been using it successfully with libMesh Nemesis I/O. > --- > Roy > > ------------------------------------------------------------------------------ > RSA® Conference 2012 > Save $700 by Nov 18 > Register now! > http://p.sf.net/sfu/rsa-sfdev2dev1 > _______________________________________________ > Libmesh-users mailing list > Lib...@li... > https://lists.sourceforge.net/lists/listinfo/libmesh-users |
From: <Rob...@un...> - 2011-10-31 22:31:57
|
> I have spent more than 20 hours now with the simple > problem to read a tetgen file and write it with Nemesis_IO > i such a way that I can re-read it in parallel. My mesh > fits on one processor - but I can only use one because a > copy of it on a second processor wouldn´t fit anymore. I > read in the mesh and tried every method I found to > partition it - but I never succeeded to write the nemesis > files for the different processors. I only get one file > which, I can't read in i parallel because the program > searches the files mesh.8.0, mesh.8.1, mesh.8.2, ... . How > can I get these files when reading in the serial tetgen > mesh on one processor? > > I am sure it is quite easy - unfortunately I don't > progress any more without help. I have tried to partition the mesh using the Metis Partitioner. Mesh mesh; TetGenIO TETGEN(mesh); TETGEN.read("meshfile"); mesh.find_neighbors(); > thank you, > Robert > > > > > > >> You'd either need to borrow a large memory machine > > > for >> long enough to read in the Tetgen file and > > > rewrite it as >> Nemesis, or you'd need to modify the > > > TetGenIO to support >> reading a pre-parallelized > > > ParallelMesh. > > > > > I had a look at the class Nemesis_IO. There seem to > > > > be some differences between the online documentation > > > > and the class provided in version 0.7.1. > > > > > > You may need to use the SVN head; I forget how recent > > > the latest ParallelMesh development was, but it's > > > quite possible that it wasn't working yet in 0.7.1. > > > > I have tried with the SVN version. The following code > > for writing the nemesis file worked fine: > > > > Mesh mesh; > > > > TetGenIO TETGEN(mesh); > > > > { // read TETGEN file > > OStringStream inmesh; > > inmesh << mesh_dir << ReadMesh[actual_pulse]<<".ele" > > ; TETGEN.read(inmesh.str().c_str()); > > } > > > > mesh.prepare_for_use(); > > > > > > Nemesis_IO NEM (mesh); > > > > NEM.write("out/nemmesh"); > > > > > > Afterwards, when I want to read it on one processor: > > /tryRead-opt > > > > Mesh mesh; > > Nemesis_IO NEM (mesh); > > NEM.read("nemmesh.1.0"); > > > > reading seems to work but, at the end of the program I > > get the following error: > > > > > > > ---------------------------------------------------------- > > ---------------------- Error flushing buffers to file. > > [0] src/mesh/exodusII_io_helper.C, line 151, compiled > > Oct 30 2011 at 11:09:28 terminate called after throwing > > an instance of 'libMesh::LogicError' > > what(): Error in libMesh internal logic > > [tomo-PaineCalc:28510] *** Process received signal *** > > [tomo-PaineCalc:28510] Signal: Aborted (6) > > [tomo-PaineCalc:28510] Signal code: (-6) > > [tomo-PaineCalc:28510] [ 0] > > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) > > [0x7f0606345c60] > > [tomo-PaineCalc:28510] [ 1] > > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) > > [0x7f0605115d05] > > [tomo-PaineCalc:28510] [ 2] > > /lib/x86_64-linux-gnu/libc.so.6(abort+0x186) > > [0x7f0605119ab6] > > [tomo-PaineCalc:28510] [ 3] > > > > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__ > > verbose_terminate_handlerEv+0x11d) [0x7f06055316dd] > > [tomo-PaineCalc:28510] [ 4] > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9926) > > [0x7f060552f926] [tomo-PaineCalc:28510] [ 5] > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9953) > > [0x7f060552f953] > > [tomo-PaineCalc:28510] [ 6] > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9a5e) > > [0x7f060552fa5e] > > [tomo-PaineCalc:28510] [ 7] > > > > > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > > > _opt/libmesh.so(_ZN7libMesh18ExodusII_IO_Helper9check_errE > > iSs+0x1b3) [0x7f060adbaa73] [tomo-PaineCalc:28510] [ 8] > > > > > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > > _opt/libmesh.so(_ZN7libMesh17Nemesis_IO_HelperD1Ev+0x48) > > [0x7f060ae88978] [tomo-PaineCalc:28510] [ 9] > > > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > > _opt/libmesh.so(_ZN7libMesh17Nemesis_IO_HelperD0Ev+0x9) > > [0x7f060ae89599] [tomo-PaineCalc:28510] [10] > > > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > > _opt/libmesh.so(_ZN7libMesh10Nemesis_IOD1Ev+0x2d) > > [0x7f060ae7c41d] [tomo-PaineCalc:28510] [11] > > /tryRead-opt(main+0x202) [0x4589e2] > > [tomo-PaineCalc:28510] [12] > > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xff) > > [0x7f0605100eff] [tomo-PaineCalc:28510] [13] > > ./tryRead-opt() [0x458719] [tomo-PaineCalc:28510] *** > > End of error message *** Aborted > > > > > > > ---------------------------------------------------------- > > ---------------------- > > > > > > > > > > However, if I want to read it when running with 'mpiexec > > -np 8 ./tryRead-opt' > > I get a long error message - see at the end of the mail. > > > > For configuring of libmesh I used the following options: > > /configure --enable-slepc --enable-mpi --enable-parmesh > > --enable-petsc > > > > Do I maybe have to change something there? > > thanks, Robert > > > > > > > > > ---------------------------------------------------------- > > ---------------------- what(): Error in libMesh > > internal logic [tomo-PaineCalc:28499] *** Process > > received signal *** [tomo-PaineCalc:28499] Signal: > > Aborted (6) [tomo-PaineCalc:28499] Signal code: (-6) > > [tomo-PaineCalc:28500] [ 0] > > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) > > [0x7f9eac762c60] > > [tomo-PaineCalc:28500] [ 1] > > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) > > [0x7f9eab532d05] > > [tomo-PaineCalc:28500] [ 2] > > /lib/x86_64-linux-gnu/libc.so.6(abort+0x186) > > [0x7f9eab536ab6] > > [tomo-PaineCalc:28500] [ 3] > > > > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__ > > verbose_terminate_handlerEv+0x11d) [0x7f9eab94e6dd] > > [tomo-PaineCalc:28500] [ 4] > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9926) > > [0x7f9eab94c926] [tomo-PaineCalc:28500] [ 5] > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9953) > > [0x7f9eab94c953] > > [tomo-PaineCalc:28500] [ 6] > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9a5e) > > [0x7f9eab94ca5e] > > [tomo-PaineCalc:28500] [ 7] > > > > > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > > > _opt/libmesh.so(_ZN7libMesh18ExodusII_IO_Helper9check_errE > > iSs+0x1b3) [0x7f9eb11d7a73] [tomo-PaineCalc:28500] [ 8] > > > > > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > > > _opt/libmesh.so(_ZN7libMesh18ExodusII_IO_Helper4openEPKc+0 > > x99) [0x7f9eb11da299] [tomo-PaineCalc:28500] [ 9] > > > > > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > > _opt/libmesh.so(_ZN7libMesh10Nemesis_IO4readERKSs+0xe5) > > [0x7f9eb129a955] [tomo-PaineCalc:28500] [10] > > /tryRead-opt(main+0xe6) [0x4588c6] > > [tomo-PaineCalc:28500] [11] > > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xff) > > [0x7f9eab51deff] [tomo-PaineCalc:28500] [12] > > ./tryRead-opt() [0x458719] [tomo-PaineCalc:28503] [ 0] > > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) > > [0x7f497c5fbc60] > > [tomo-PaineCalc:28503] [ 1] > > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) > > [0x7f497b3cbd05] > > [tomo-PaineCalc:28503] [ 2] > > /lib/x86_64-linux-gnu/libc.so.6(abort+0x186) > > [0x7f497b3cfab6] > > [tomo-PaineCalc:28503] [ 3] > > > > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__ > > verbose_terminate_handlerEv+0x11d) [0x7f497b7e76dd] > > [tomo-PaineCalc:28503] [ 4] > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9926) > > [0x7f497b7e5926] [tomo-PaineCalc:28503] [ 5] > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9953) > > [0x7f497b7e5953] > > [tomo-PaineCalc:28503] [ 6] > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9a5e) > > [0x7f497b7e5a5e] > > [tomo-PaineCalc:28503] [ 7] > > > > > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > > > _opt/libmesh.so(_ZN7libMesh18ExodusII_IO_Helper9check_errE > > iSs+0x1b3) [0x7f4981070a73] [tomo-PaineCalc:28503] [ 8] > > > > > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > > > _opt/libmesh.so(_ZN7libMesh18ExodusII_IO_Helper4openEPKc+0 > > x99) [0x7f4981073299] [tomo-PaineCalc:28503] [ 9] > > > > > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > > _opt/libmesh.so(_ZN7libMesh10Nemesis_IO4readERKSs+0xe5) > > [0x7f4981133955] [tomo-PaineCalc:28503] [10] > > /tryRead-opt(main+0xe6) [0x4588c6] > > [tomo-PaineCalc:28503] [11] > > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xff) > > [0x7f497b3b6eff] [tomo-PaineCalc:28503] [12] > > ./tryRead-opt() [0x458719] [tomo-PaineCalc:28503] *** > > End of error message *** [tomo-PaineCalc:28502] [ 0] > > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) > > [0x7fb12c0c9c60] > > [tomo-PaineCalc:28502] [ 1] > > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) > > [0x7fb12ae99d05] > > [tomo-PaineCalc:28502] [ 2] > > /lib/x86_64-linux-gnu/libc.so.6(abort+0x186) > > > > > [0x7fb12ae9dab6]------------------------------------------ > > -------------------------------- mpiexec noticed that > > process rank 3 with PID 28501 on node tomo-PaineCalc > > exited on signal 6 (Aborted). > > > ---------------------------------------------------------- > > ---------------- 5 total processes killed (some possibly > > by mpiexec during cleanup) > > > > > > .. and so on :-| > > > > > > > > > ---------------------------------------------------------- > > ---------------------- > > > > > ---------------------------------------------------------- > > -------------------- Get your Android app more play: > Bring it to the BlackBerry PlayBook in minutes. > > BlackBerry App World now supports Android Apps for > > the BlackBerry® PlayBook. Discover just how easy and > > simple it is! http://p.sf.net/sfu/android-dev2dev > > _______________________________________________ > > Libmesh-users mailing list > > Lib...@li... > > > https://lists.sourceforge.net/lists/listinfo/libmesh-users > > ---------------------------------------------------------- > -------------------- Get your Android app more play: Bring > it to the BlackBerry PlayBook in minutes. BlackBerry App > World now supports Android Apps for the > BlackBerry® PlayBook. Discover just how easy and > simple it is! http://p.sf.net/sfu/android-dev2dev > _______________________________________________ > Libmesh-users mailing list > Lib...@li... > https://lists.sourceforge.net/lists/listinfo/libmesh-users |
From: <Rob...@un...> - 2011-10-31 22:37:56
|
> I have spent more than 20 hours now with the simple > problem to read a tetgen file and write it with Nemesis_IO > i such a way that I can re-read it in parallel. My mesh > fits on one processor - but I can only use one because a > copy of it on a second processor wouldn´t fit anymore. I > read in the mesh and tried every method I found to > partition it - but I never succeeded to write the nemesis > files for the different processors. I only get one file > which, I can't read in i parallel because the program > searches the files mesh.8.0, mesh.8.1, mesh.8.2, ... . How > can I get these files when reading in the serial tetgen > mesh on one processor? > > I am sure it is quite easy - unfortunately I don't > progress any more without help. I have tried to use the Metis Partitioner. I am loading the mesh on one CPU and trying to split it to more parts: Mesh mesh; TetGenIO TETGEN(mesh); TETGEN.read("meshfile"); mesh.find_neighbors(); MetisPartitioner().partition(mesh,8); With the last line I try to make 8 partitions for 8 processors. Of course, it doesn't work this way - I think I can't make more partitions than actually used processors - which is 1. Here is the point where I don't see how to write nemesis-files for the processors. How can I split my big mesh??? I am sorry for writing again - I have already started with a similar problem more than a month ago but still, I could't solve the problem. Thanks very much for your help in advance, Robert > thank you, > Robert > > > > > > >> You'd either need to borrow a large memory machine > > > for >> long enough to read in the Tetgen file and > > > rewrite it as >> Nemesis, or you'd need to modify the > > > TetGenIO to support >> reading a pre-parallelized > > > ParallelMesh. > > > > > I had a look at the class Nemesis_IO. There seem to > > > > be some differences between the online documentation > > > > and the class provided in version 0.7.1. > > > > > > You may need to use the SVN head; I forget how recent > > > the latest ParallelMesh development was, but it's > > > quite possible that it wasn't working yet in 0.7.1. > > > > I have tried with the SVN version. The following code > > for writing the nemesis file worked fine: > > > > Mesh mesh; > > > > TetGenIO TETGEN(mesh); > > > > { // read TETGEN file > > OStringStream inmesh; > > inmesh << mesh_dir << ReadMesh[actual_pulse]<<".ele" > > ; TETGEN.read(inmesh.str().c_str()); > > } > > > > mesh.prepare_for_use(); > > > > > > Nemesis_IO NEM (mesh); > > > > NEM.write("out/nemmesh"); > > > > > > Afterwards, when I want to read it on one processor: > > /tryRead-opt > > > > Mesh mesh; > > Nemesis_IO NEM (mesh); > > NEM.read("nemmesh.1.0"); > > > > reading seems to work but, at the end of the program I > > get the following error: > > > > > > > ---------------------------------------------------------- > > ---------------------- Error flushing buffers to file. > > [0] src/mesh/exodusII_io_helper.C, line 151, compiled > > Oct 30 2011 at 11:09:28 terminate called after throwing > > an instance of 'libMesh::LogicError' > > what(): Error in libMesh internal logic > > [tomo-PaineCalc:28510] *** Process received signal *** > > [tomo-PaineCalc:28510] Signal: Aborted (6) > > [tomo-PaineCalc:28510] Signal code: (-6) > > [tomo-PaineCalc:28510] [ 0] > > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) > > [0x7f0606345c60] > > [tomo-PaineCalc:28510] [ 1] > > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) > > [0x7f0605115d05] > > [tomo-PaineCalc:28510] [ 2] > > /lib/x86_64-linux-gnu/libc.so.6(abort+0x186) > > [0x7f0605119ab6] > > [tomo-PaineCalc:28510] [ 3] > > > > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__ > > verbose_terminate_handlerEv+0x11d) [0x7f06055316dd] > > [tomo-PaineCalc:28510] [ 4] > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9926) > > [0x7f060552f926] [tomo-PaineCalc:28510] [ 5] > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9953) > > [0x7f060552f953] > > [tomo-PaineCalc:28510] [ 6] > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9a5e) > > [0x7f060552fa5e] > > [tomo-PaineCalc:28510] [ 7] > > > > > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > > > _opt/libmesh.so(_ZN7libMesh18ExodusII_IO_Helper9check_errE > > iSs+0x1b3) [0x7f060adbaa73] [tomo-PaineCalc:28510] [ 8] > > > > > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > > _opt/libmesh.so(_ZN7libMesh17Nemesis_IO_HelperD1Ev+0x48) > > [0x7f060ae88978] [tomo-PaineCalc:28510] [ 9] > > > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > > _opt/libmesh.so(_ZN7libMesh17Nemesis_IO_HelperD0Ev+0x9) > > [0x7f060ae89599] [tomo-PaineCalc:28510] [10] > > > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > > _opt/libmesh.so(_ZN7libMesh10Nemesis_IOD1Ev+0x2d) > > [0x7f060ae7c41d] [tomo-PaineCalc:28510] [11] > > /tryRead-opt(main+0x202) [0x4589e2] > > [tomo-PaineCalc:28510] [12] > > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xff) > > [0x7f0605100eff] [tomo-PaineCalc:28510] [13] > > ./tryRead-opt() [0x458719] [tomo-PaineCalc:28510] *** > > End of error message *** Aborted > > > > > > > ---------------------------------------------------------- > > ---------------------- > > > > > > > > > > However, if I want to read it when running with 'mpiexec > > -np 8 ./tryRead-opt' > > I get a long error message - see at the end of the mail. > > > > For configuring of libmesh I used the following options: > > /configure --enable-slepc --enable-mpi --enable-parmesh > > --enable-petsc > > > > Do I maybe have to change something there? > > thanks, Robert > > > > > > > > > ---------------------------------------------------------- > > ---------------------- what(): Error in libMesh > > internal logic [tomo-PaineCalc:28499] *** Process > > received signal *** [tomo-PaineCalc:28499] Signal: > > Aborted (6) [tomo-PaineCalc:28499] Signal code: (-6) > > [tomo-PaineCalc:28500] [ 0] > > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) > > [0x7f9eac762c60] > > [tomo-PaineCalc:28500] [ 1] > > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) > > [0x7f9eab532d05] > > [tomo-PaineCalc:28500] [ 2] > > /lib/x86_64-linux-gnu/libc.so.6(abort+0x186) > > [0x7f9eab536ab6] > > [tomo-PaineCalc:28500] [ 3] > > > > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__ > > verbose_terminate_handlerEv+0x11d) [0x7f9eab94e6dd] > > [tomo-PaineCalc:28500] [ 4] > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9926) > > [0x7f9eab94c926] [tomo-PaineCalc:28500] [ 5] > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9953) > > [0x7f9eab94c953] > > [tomo-PaineCalc:28500] [ 6] > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9a5e) > > [0x7f9eab94ca5e] > > [tomo-PaineCalc:28500] [ 7] > > > > > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > > > _opt/libmesh.so(_ZN7libMesh18ExodusII_IO_Helper9check_errE > > iSs+0x1b3) [0x7f9eb11d7a73] [tomo-PaineCalc:28500] [ 8] > > > > > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > > > _opt/libmesh.so(_ZN7libMesh18ExodusII_IO_Helper4openEPKc+0 > > x99) [0x7f9eb11da299] [tomo-PaineCalc:28500] [ 9] > > > > > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > > _opt/libmesh.so(_ZN7libMesh10Nemesis_IO4readERKSs+0xe5) > > [0x7f9eb129a955] [tomo-PaineCalc:28500] [10] > > /tryRead-opt(main+0xe6) [0x4588c6] > > [tomo-PaineCalc:28500] [11] > > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xff) > > [0x7f9eab51deff] [tomo-PaineCalc:28500] [12] > > ./tryRead-opt() [0x458719] [tomo-PaineCalc:28503] [ 0] > > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) > > [0x7f497c5fbc60] > > [tomo-PaineCalc:28503] [ 1] > > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) > > [0x7f497b3cbd05] > > [tomo-PaineCalc:28503] [ 2] > > /lib/x86_64-linux-gnu/libc.so.6(abort+0x186) > > [0x7f497b3cfab6] > > [tomo-PaineCalc:28503] [ 3] > > > > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__ > > verbose_terminate_handlerEv+0x11d) [0x7f497b7e76dd] > > [tomo-PaineCalc:28503] [ 4] > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9926) > > [0x7f497b7e5926] [tomo-PaineCalc:28503] [ 5] > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9953) > > [0x7f497b7e5953] > > [tomo-PaineCalc:28503] [ 6] > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9a5e) > > [0x7f497b7e5a5e] > > [tomo-PaineCalc:28503] [ 7] > > > > > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > > > _opt/libmesh.so(_ZN7libMesh18ExodusII_IO_Helper9check_errE > > iSs+0x1b3) [0x7f4981070a73] [tomo-PaineCalc:28503] [ 8] > > > > > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > > > _opt/libmesh.so(_ZN7libMesh18ExodusII_IO_Helper4openEPKc+0 > > x99) [0x7f4981073299] [tomo-PaineCalc:28503] [ 9] > > > > > /home/ThermoPaine/libmesh_svn/lib/x86_64-unknown-linux-gnu > > _opt/libmesh.so(_ZN7libMesh10Nemesis_IO4readERKSs+0xe5) > > [0x7f4981133955] [tomo-PaineCalc:28503] [10] > > /tryRead-opt(main+0xe6) [0x4588c6] > > [tomo-PaineCalc:28503] [11] > > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xff) > > [0x7f497b3b6eff] [tomo-PaineCalc:28503] [12] > > ./tryRead-opt() [0x458719] [tomo-PaineCalc:28503] *** > > End of error message *** [tomo-PaineCalc:28502] [ 0] > > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) > > [0x7fb12c0c9c60] > > [tomo-PaineCalc:28502] [ 1] > > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) > > [0x7fb12ae99d05] > > [tomo-PaineCalc:28502] [ 2] > > /lib/x86_64-linux-gnu/libc.so.6(abort+0x186) > > > > > [0x7fb12ae9dab6]------------------------------------------ > > -------------------------------- mpiexec noticed that > > process rank 3 with PID 28501 on node tomo-PaineCalc > > exited on signal 6 (Aborted). > > > ---------------------------------------------------------- > > ---------------- 5 total processes killed (some possibly > > by mpiexec during cleanup) > > > > > > .. and so on :-| > > > > > > > > > ---------------------------------------------------------- > > ---------------------- > > > > > ---------------------------------------------------------- > > -------------------- Get your Android app more play: > Bring it to the BlackBerry PlayBook in minutes. > > BlackBerry App World now supports Android Apps for > > the BlackBerry® PlayBook. Discover just how easy and > > simple it is! http://p.sf.net/sfu/android-dev2dev > > _______________________________________________ > > Libmesh-users mailing list > > Lib...@li... > > > https://lists.sourceforge.net/lists/listinfo/libmesh-users > > ---------------------------------------------------------- > -------------------- Get your Android app more play: Bring > it to the BlackBerry PlayBook in minutes. BlackBerry App > World now supports Android Apps for the > BlackBerry® PlayBook. Discover just how easy and > simple it is! http://p.sf.net/sfu/android-dev2dev > _______________________________________________ > Libmesh-users mailing list > Lib...@li... > https://lists.sourceforge.net/lists/listinfo/libmesh-users |
From: <Rob...@un...> - 2011-11-02 08:12:14
|
> I apologize for not responding to these emails earlier . > > Here is a basic algorithm for using NEMESIS when starting > with a file format that _isn't_ already EXODUS _and_ your > mesh is so large that you can't read it on "n" number of > processors. > > 1. Read file into libMesh > 2. Write out Exodus file > 3. Run nem_spread on the Exodus file to generate Nemesis > files (one for each processor) 4. With a version of your > code compiled against a libMesh that is using parallel > mesh use the nemesis reader to read the nemesis files. > > nem_spread is a _serial_ process that produces "n" number > of nemesis files (it is doing an "offline" partitioning). > You will need a machine with quite a bit of RAM. > > Derek Hi Derek, thanks for your help. I have to do some labwork for the rest of the week but I'll let you know how it worked as soon as I have tried. Robert |
From: <Rob...@un...> - 2011-11-02 11:19:57
|
Hello, I haven't tried the suggestion of Derek yet. Instead, I wanted to try to split a smaller mesh with nemesis and see if my code runs in parallel (configure with --enable-parmesh) I split the mesh with the following code: Mesh mesh; TetGenIO TETGEN(mesh); TETGEN.read("meshfile.ele"); set_subdomains(&TETGEN, mesh); set_boundary_ids(mesh); std::cout<<"tetgen finished reading"<<std::endl; mesh.find_neighbors(); std::cout<<"neighbors found"<<std::endl; MetisPartitioner().partition(mesh,4); mesh.print_info(); std::cout<<"partition done"<<std::endl; Nemesis_IO NEM ( mesh ); std::cout<<"write"<<std::endl; { OStringStream outmesh; outmesh << "mesh1/" << WriteMesh[actual_pulse]<<".nemesis"; NEM.write(outmesh.str().c_str()); } |
From: <Rob...@un...> - 2011-11-02 11:52:58
|
Sorry for the last message, it was not finished yet - i unintentionally pressed the send button. Once again: Hello, I haven't tried the suggestion of Derek yet. Instead, I wanted to try to split a smaller mesh with nemesis and see if my code runs in parallel (libmesh configure was done with --enable-parmesh) I split a smaller (about 1 000 000 nodes) mesh with the following code: --------------------------------------- //CODE 1 Mesh mesh; TetGenIO TETGEN(mesh); TETGEN.read("meshfile.ele"); set_subdomains(&TETGEN, mesh); // I manually set subdomain_id set_boundary_ids(mesh); // I manually set boundary_id mesh.find_neighbors(); MetisPartitioner().partition(mesh,4); mesh.print_info(); Nemesis_IO NEM ( mesh ); WriteMesh[actual_pulse]<<"main_alm.nemesis"; //END CODE 1 --------------------------------------- After this I get 4 files (running on 4 processors) xxx.4.0, xxx.4.1, xxx.4.2, xxx.4.3 where xxx stands for the file name. Now, in a new program I want to carry out my calculations: --------------------------------------- // CODE 2 Mesh new_mesh; Nemesis_IO NEMESIS (new_mesh); NEMESIS.read("xxx"); if (!new_mesh. is_prepared ()) new_mesh.prepare_for_use(); new_mesh.print_info(); // MeshRefinement meshrefine (new_mesh); // meshrefine.uniformly_refine(1); // new_mesh.print_info(); EquationSystems new_equation_systems (new_mesh); TransientLinearImplicitSystem & T_system = new_equation_systems.add_system<TransientLinearImplicitSystem> ("Diffusion"); T_system.add_variable ("T", FIRST); T_system.attach_assemble_function (assemble_cd); T_system.attach_init_function (init_geothermal_gradient); new_equation_systems.init (); new_equation_systems.print_info(); // ********** TIME LOOP ******* int t_write = 0; std::cout << "t_step "<<t_step<<" t_next_pulse[actual_pulse] " << t_next_pulse[actual_pulse]<<" dt "<<dt<<std::endl; for (; t_step < t_next_pulse[actual_pulse]/dt; t_step++) { time += dt; new_equation_systems.parameters.set<Real> ("time") = time; TransientLinearImplicitSystem& T_system = new_equation_systems.get_system<TransientLinearImplicitSystem>("Diffusion"); *T_system.old_local_solution = *T_system.current_local_solution; new_equation_systems.get_system("Diffusion").solve(); } // END CODE 2 --------------------------------------- The assemble function is similar to the one in example 9. At every time step I get the following warning: Warning: This MeshInput subclass only support meshes which have been serialized! [0] /home/ThermoPaine/libmesh_svn/include/mesh/mesh_input.h, line 148, compiled Oct 30 2011 at 11:09:26 Warning: This MeshOutput subclass only support meshes which have been serialized! [0] /home/ThermoPaine/libmesh_svn/include/mesh/mesh_output.h, line 181, compiled Oct 30 2011 at 11:09:26 ----------- QUESTION 1: ----------- What does this warning mean? Can I simply ignore it? I would have said that there is still a problem with my configuration for parallel??? ----------- QUESTION 2: ----------- There is another thing which disturbs me: When I partition the mesh with the first code, I get n_subdomains()=4 n_partitions()=4 n_processors()=4 in mesh.print_info(). But, after reading the mesh with the second code, I get n_subdomains()=4 n_partitions()=1 n_processors()=4 ----------- QUESTION 3 ----------- In several discussions I read that there are still serious bugs in ParallelMesh. Do these bugs also concern the code I have above? AFAIK there are bugs with adaptive coarsening and refinement. In my code I don't use the class ParallelMesh - so I was wondering if the data-structure, which I get with --enable-parmesh, allows refinement and coarsening. So, although I read in 1 file per processor the n_partitions() has changed. Thank you, Robert |
From: Roy S. <roy...@ic...> - 2011-10-29 21:03:40
|
On Sat, 29 Oct 2011, Rob...@un... wrote: > In serial every processor has a copy of the mesh, right? Right. > Is it possible to have such a copy only on one processor and not on > the others - this would already free some space. You could run with only one MPI rank but with threading to make use of the other processors. That would free a lot of space, but might currently be less efficient CPU-wise depending on your application and algebraic solver software. >> You'd either need to borrow a large memory machine for >> long enough to read in the Tetgen file and rewrite it as >> Nemesis, or you'd need to modify the TetGenIO to support >> reading a pre-parallelized ParallelMesh. > > I had a look at the class Nemesis_IO. There seem to be some > differences between the online documentation and the class > provided in version 0.7.1. You may need to use the SVN head; I forget how recent the latest ParallelMesh development was, but it's quite possible that it wasn't working yet in 0.7.1. --- Roy |