#41 IO of snapshots in HDF5 (monolithic files)

[Anonymous]

Originally created by: Bourdin.KIS (code.google.com)
Originally owned by: Bourdin.KIS (code.google.com)

Beeing able to load and write data from/to monolithic HDF5 snapshot files
would make the distribution of data to pencils generic and there would
nomore be the need to setup proc* directories and distribute an initial
snapshot to them. Also restarting with different amount of processors would
work without any modification of the data files.

Importing and exporting data from/to HDF5 is error-proof regarding physical
units, endianess etc.

IDL, Matlab and many other programs/visualisation tools already support
HDF5 or their developers should be asked support HDF5 in future.

What has to be done is:
- find a good HDF file structure for The Pencil Code
- adapt the distributed-/MPI-IO module to collect all information and write it

[Anonymous]

Originally posted by: Bourdin.KIS (code.google.com)

Since the module 'io_mpio' is working again, we would only need a data conversion script that can be used as a post-processing tool, to convert the monolithic snapshots to HDF5.

Labels: -Priority-Medium Priority-Low

[Anonymous]

Originally posted by: Bourdin.KIS (code.google.com)

There will soon be a new IO module collecting all data on the root processor and write it from there. This module can later easily be adapted to write HFD5 files.

[Anonymous]

Originally posted by: mppiyali

I am not sure if this is the correct place to ask but still excuse me. Does io_mpi2 work correctly for everyone? My pencil code set-up on NASA Pleiades crashes with IO=io_mpi2 giving an error message for fatal error in MPI_Allreduce. However, I have used MPI2 with all processors writing to a single file on Pleiades using a modified version of Zeus code so I suppose MPI2 should work on NAS systems.