You can subscribe to this list here.
2013 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
(19) |
Jul
(6) |
Aug
|
Sep
(68) |
Oct
(38) |
Nov
(63) |
Dec
(1) |
---|---|---|---|---|---|---|---|---|---|---|---|---|
2014 |
Jan
(36) |
Feb
(42) |
Mar
(5) |
Apr
(23) |
May
(26) |
Jun
(22) |
Jul
(27) |
Aug
(4) |
Sep
|
Oct
(62) |
Nov
(7) |
Dec
(13) |
2015 |
Jan
(9) |
Feb
(52) |
Mar
|
Apr
|
May
|
Jun
|
Jul
(30) |
Aug
(94) |
Sep
(8) |
Oct
(24) |
Nov
(8) |
Dec
(4) |
2016 |
Jan
|
Feb
|
Mar
(5) |
Apr
(1) |
May
(4) |
Jun
|
Jul
|
Aug
(4) |
Sep
(3) |
Oct
(7) |
Nov
|
Dec
(2) |
2017 |
Jan
(14) |
Feb
(14) |
Mar
(4) |
Apr
|
May
|
Jun
(8) |
Jul
(4) |
Aug
(2) |
Sep
(16) |
Oct
(4) |
Nov
(9) |
Dec
|
2018 |
Jan
(3) |
Feb
(16) |
Mar
(2) |
Apr
(2) |
May
(2) |
Jun
|
Jul
(13) |
Aug
(10) |
Sep
(6) |
Oct
(21) |
Nov
(7) |
Dec
(19) |
2019 |
Jan
(6) |
Feb
(1) |
Mar
(15) |
Apr
(30) |
May
(16) |
Jun
(32) |
Jul
(10) |
Aug
(6) |
Sep
(8) |
Oct
(2) |
Nov
(15) |
Dec
(25) |
2020 |
Jan
|
Feb
|
Mar
(2) |
Apr
|
May
|
Jun
|
Jul
|
Aug
(11) |
Sep
(18) |
Oct
(7) |
Nov
(1) |
Dec
(1) |
2021 |
Jan
(1) |
Feb
|
Mar
|
Apr
(2) |
May
|
Jun
|
Jul
|
Aug
|
Sep
(4) |
Oct
|
Nov
|
Dec
|
2022 |
Jan
|
Feb
(6) |
Mar
(3) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(9) |
Dec
(8) |
2023 |
Jan
(7) |
Feb
(1) |
Mar
(1) |
Apr
|
May
|
Jun
(5) |
Jul
(1) |
Aug
|
Sep
|
Oct
(10) |
Nov
(10) |
Dec
(6) |
2024 |
Jan
|
Feb
(9) |
Mar
(4) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
From: Blondel, S. <sbl...@ut...> - 2024-03-04 19:49:18
|
Thank you Junchao for looking into it. I managed to build a previous version of Xolotl that uses Kokkos (but PETSc is default PETSc without Kokkos) so that we have at least partial GPU support. Best, Sophie ________________________________ From: Junchao Zhang <jun...@gm...> Sent: Monday, March 4, 2024 12:10 To: Blondel, Sophie <sbl...@ut...> Cc: xol...@li... <xol...@li...>; pet...@mc... <pet...@mc...> Subject: Re: [petsc-users] PAMI error on Summit You don't often get email from jun...@gm.... Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Hi, Sophie, I tried various modules and compilers on Summit and failed to find one that works with gpu aware mpi. The one that could build petsc and kokkos was "module load cuda/11.7.1 gcc/9.3.0-compiler_only spectrum-mpi essl netlib-lapack". But it only worked with "-use_gpu_aware_mpi 0". Without it, I saw code crashes. From what I can see, the gpu-aware mpi on Summit is an unusable and unmaintained state. --Junchao Zhang On Fri, Mar 1, 2024 at 3:58 PM Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> wrote: It is weird, with jsrun --smpiargs "-gpu" -n 6 -a 1 -c 1 -g 1 /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos/src/ksp/ksp/tutorials/bench_kspsolve -mat_type aijkokkos -use_gpu_aware_mpi 1 petsc tried to test if the MPI is gpu aware (by doing an MPI_Allreduce on device buffers). It tried and found it was not, so it threw out the complaint in the error message. From https://docs.olcf.ornl.gov/systems/summit_user_guide.html#cuda-aware-mpi, I think your flags were right. I just got my Summit account reactivated today. I will give it a try. --Junchao Zhang On Fri, Mar 1, 2024 at 3:32 PM Blondel, Sophie <sbl...@ut...<mailto:sbl...@ut...>> wrote: I have been using --smpiargs "-gpu". I tried the benchmark with "jsrun --smpiargs "-gpu" -n 6 -a 1 -c 1 -g 1 /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos/src/ksp/ksp/tutorials/bench_kspsolve -mat_type aijkokkos -use_gpu_aware_mpi 0" and it seems to work: Fri Mar 1 16:27:14 EST 2024 =========================================== Test: KSP performance - Poisson Input matrix: 27-pt finite difference stencil -n 100 DoFs = 1000000 Number of nonzeros = 26463592 Step1 - creating Vecs and Mat... Step2 - running KSPSolve()... Step3 - calculating error norm... Error norm: 5.591e-02 KSP iters: 63 KSPSolve: 3.16646 seconds FOM: 3.158e+05 DoFs/sec =========================================== ------------------------------------------------------------ Sender: LSF System <lsfadmin@batch3> Subject: Job 3322694: <xolotlTest> in cluster <summit> Done Job <xolotlTest> was submitted from host <login2> by user <bqo> in cluster <summit> at Fri Mar 1 16:26:58 2024 Job was executed on host(s) <1*batch3>, in queue <debug>, as user <bqo> in cluster <summit> at Fri Mar 1 16:27:00 2024 <42*a35n05> </ccs/home/bqo> was used as the home directory. </gpfs/alpine2/mat267/scratch/bqo/test> was used as the working directory. Started at Fri Mar 1 16:27:00 2024 Terminated at Fri Mar 1 16:27:26 2024 Results reported at Fri Mar 1 16:27:26 2024 The output (if any) is above this job summary. If I switch to "jsrun --smpiargs "-gpu" -n 6 -a 1 -c 1 -g 1 /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos/src/ksp/ksp/tutorials/bench_kspsolve -mat_type aijkokkos -use_gpu_aware_mpi 1" it complains: Fri Mar 1 16:25:02 EST 2024 =========================================== Test: KSP performance - Poisson Input matrix: 27-pt finite difference stencil -n 100 DoFs = 1000000 Number of nonzeros = 26463592 Step1 - creating Vecs and Mat... [5]PETSC ERROR: PETSc is configured with GPU support, but your MPI is not GPU-aware. For better performance, please use a GPU-aware MPI. [5]PETSC ERROR: If you do not care, add option -use_gpu_aware_mpi 0. To not see the message again, add the option to your .petscrc, OR add it to the env var PETSC_OPTIONS. [5]PETSC ERROR: If you do care, for IBM Spectrum MPI on OLCF Summit, you may need jsrun --smpiargs=-gpu. [5]PETSC ERROR: For Open MPI, you need to configure it --with-cuda (https://www.open-mpi.org/faq/?category=buildcuda) [5]PETSC ERROR: For MVAPICH2-GDR, you need to set MV2_USE_CUDA=1 (http://mvapich.cse.ohio-state.edu/userguide/gdr/) [5]PETSC ERROR: For Cray-MPICH, you need to set MPICH_GPU_SUPPORT_ENABLED=1 (man mpi to see manual of cray-mpich) -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF with errorcode 76. Best, Sophie ________________________________ From: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> Sent: Thursday, February 29, 2024 17:09 To: Blondel, Sophie <sbl...@ut...<mailto:sbl...@ut...>> Cc: xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>>; pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>> Subject: Re: [petsc-users] PAMI error on Summit You don't often get email from jun...@gm...<mailto:jun...@gm...>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Could you try a petsc example to see if the environment is good? For example, cd src/ksp/ksp/tutorials make bench_kspsolve mpirun -n 6 ./bench_kspsolve -mat_type aijkokkos -use_gpu_aware_mpi {0 or 1} BTW, I remember to use gpu-aware mpi on Summit, one needs to pass --smpiargs "-gpu" to jsrun --Junchao Zhang On Thu, Feb 29, 2024 at 3:22 PM Blondel, Sophie via petsc-users <pet...@mc...<mailto:pet...@mc...>> wrote: I still get the same error when deactivating GPU-aware MPI. I also tried unloading spectrum MPI and using openMPI instead (recompiling everything) and I get a segfault in PETSc in that case (still using GPU-aware MPI I think, at least not explicitly ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I still get the same error when deactivating GPU-aware MPI. I also tried unloading spectrum MPI and using openMPI instead (recompiling everything) and I get a segfault in PETSc in that case (still using GPU-aware MPI I think, at least not explicitly turning it off): 0 TS dt 1e-12 time 0. [ERROR] [0]PETSC ERROR: [ERROR] ------------------------------------------------------------------------ [ERROR] [0]PETSC ERROR: [ERROR] Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [ERROR] [0]PETSC ERROR: [ERROR] Try option -start_in_debugger or -on_error_attach_debugger [ERROR] [0]PETSC ERROR: [ERROR] or see https://petsc.org/release/faq/#valgrind<https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__petsc.org_release_faq_-23valgrind&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=Loebf9sk4dgXGOOKPK3IHxp-C5SjGtr7Svr49LwaM4E&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI0jsXekj$> and https://petsc.org/release/faq/<https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__petsc.org_release_faq_&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=7e9oLVYLacda_1-8rSkzDEHL4Zy1BFnO4pnrfMNlgO4&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI74qqyaL$> [ERROR] [0]PETSC ERROR: [ERROR] or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html<https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__docs.nvidia.com_cuda_cuda-2Dmemcheck_index.html&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=2gHentsiEM2njpPim4k40mYA96k7v_ivjI3erSECebM&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI3YGCBJ5$> on NVIDIA CUDA systems to find memory corruption errors [ERROR] [0]PETSC ERROR: [ERROR] configure using --with-debugging=yes, recompile, link, and run [ERROR] [0]PETSC ERROR: [ERROR] to get more information on the crash. [ERROR] [0]PETSC ERROR: [ERROR] Run with -malloc_debug to check if memory corruption is causing the crash. -------------------------------------------------------------------------- Best, Sophie ________________________________ From: Blondel, Sophie via Xolotl-psi-development <xol...@li...<mailto:xol...@li...>> Sent: Thursday, February 29, 2024 10:17 To: xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>>; pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>> Subject: [Xolotl-psi-development] PAMI error on Summit Hi, I am using PETSc build with the Kokkos CUDA backend on Summit but when I run my code with multiple MPI tasks I get the following error: 0 TS dt 1e-12 time 0. errno 14 pid 864558 xolotl: /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: size_t PAMI::Dev ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. errno 14 pid 864557 xolotl: /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: size_t PAMI::Dev ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. [e28n07:864557] *** Process received signal *** [e28n07:864557] Signal: Aborted (6) [e28n07:864557] Signal code: (-6) [e28n07:864557] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] [e28n07:864557] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(gsignal+0xd8)[0x200005d796f8] [e28n07:864557] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(abort+0x164)[0x200005d53ff4] [e28n07:864557] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(+0x3d280)[0x200005d6d280] [e28n07:864557] [ 4] [e28n07:864558] *** Process received signal *** [e28n07:864558] Signal: Aborted (6) [e28n07:864558] Signal code: (-6) [e28n07:864558] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] [e28n07:864558] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(gsignal+0xd8)[0x200005d796f8] [e28n07:864558] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(abort+0x164)[0x200005d53ff4] [e28n07:864558] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(+0x3d280)[0x200005d6d280] [e28n07:864558] [ 4] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(__assert_fail+0x64)[0x200005d6d324] [e28n07:864557] [ 5] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] [e28n07:864557] [ 6] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] [e28n07:864557] [ 7] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] [e28n07:864557] [ 8] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (PAMI_Rget+0x18)[0x20007f2d94a8] [e28n07:864557] [ 9] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] [e28n07:864557] [10] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] [e28n07:864557] [11] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(__assert_fail+0x64)[0x200005d6d324] [e28n07:864558] [ 5] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] [e28n07:864558] [ 6] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] [e28n07:864558] [ 7] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] [e28n07:864558] [ 8] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (PAMI_Rget+0x18)[0x20007f2d94a8] [e28n07:864558] [ 9] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] [e28n07:864558] [10] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] [e28n07:864558] [11] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] [e28n07:864557] [12] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (PAMI_Context_advancev+0x6b0)[0x20007f2da540] [e28n07:864557] [13] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] [e28n07:864557] [14] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ progress+0x6c)[0x20003d60640c] [e28n07:864557] [15] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r equest_default_wait_all+0x144)[0x2000034c4b04] [e28n07:864557] [16] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W aitall+0x10c)[0x20000352790c] [e28n07:864557] [17] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] [e28n07:864558] [12] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (PAMI_Context_advancev+0x6b0)[0x20007f2da540] [e28n07:864558] [13] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] [e28n07:864558] [14] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ progress+0x6c)[0x20003d60640c] [e28n07:864558] [15] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r equest_default_wait_all+0x144)[0x2000034c4b04] [e28n07:864558] [16] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W aitall+0x10c)[0x20000352790c] [e28n07:864558] [17] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] [e28n07:864557] [18] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] [e28n07:864558] [18] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] [e28n07:864557] [19] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] [e28n07:864558] [19] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] [e28n07:864557] [20] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] [e28n07:864558] [20] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] [e28n07:864557] [21] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] [e28n07:864558] [21] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] [e28n07:864558] [22] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] [e28n07:864557] [22] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] [e28n07:864558] [23] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] [e28n07:864557] [23] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] [e28n07:864557] [24] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 _+0xc4)[0x200005f710d4] [e28n07:864557] [25] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 00005f7130c] [e28n07:864557] [26] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] [e28n07:864558] [24] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 _+0xc4)[0x200005f710d4] [e28n07:864558] [25] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 00005f7130c] [e28n07:864558] [26] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] [e28n07:864557] [27] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] [e28n07:864558] [27] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] [e28n07:864557] [28] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] [e28n07:864558] [28] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] [e28n07:864557] [29] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] [e28n07:864558] [29] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] [e28n07:864557] *** End of error message *** /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] [e28n07:864558] *** End of error message *** It seems to be pointing to https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/<https://urldefense.us/v3/__https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI30Ylvr6$> so I wanted to check if you had seen this type of error before and if it could be related to how the code is compiled or run. Let me know if I can provide any additional information. Best, Sophie |
From: Junchao Z. <jun...@gm...> - 2024-03-04 17:11:12
|
Hi, Sophie, I tried various modules and compilers on Summit and failed to find one that works with gpu aware mpi. The one that could build petsc and kokkos was "module load cuda/11.7.1 gcc/9.3.0-compiler_only spectrum-mpi essl netlib-lapack". But it only worked with "-use_gpu_aware_mpi 0". Without it, I saw code crashes. From what I can see, the gpu-aware mpi on Summit is an unusable and unmaintained state. --Junchao Zhang On Fri, Mar 1, 2024 at 3:58 PM Junchao Zhang <jun...@gm...> wrote: > It is weird, with > jsrun --smpiargs "-gpu" -n 6 -a 1 -c 1 -g 1 /gpfs/alpine2/mat267/proj- > shared/dependencies/petsc-kokkos/src/ksp/ksp/tutorials/bench_kspsolve > -mat_type aijkokkos -use_gpu_aware_mpi 1 > > petsc tried to test if the MPI is gpu aware (by doing an MPI_Allreduce on > device buffers). It tried and found it was not, so it threw out the > complaint in the error message. > > From > https://docs.olcf.ornl.gov/systems/summit_user_guide.html#cuda-aware-mpi, > I think your flags were right. > > I just got my Summit account reactivated today. I will give it a try. > > --Junchao Zhang > > > On Fri, Mar 1, 2024 at 3:32 PM Blondel, Sophie <sbl...@ut...> wrote: > >> I have been using --smpiargs "-gpu". >> >> I tried the benchmark with "jsrun --smpiargs "-gpu" -n 6 -a 1 -c 1 -g 1 >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos/src/ksp/ksp/tutorials/bench_kspsolve >> -mat_type aijkokkos -use_gpu_aware_mpi 0" and it seems to work: >> Fri Mar 1 16:27:14 EST 2024 >> =========================================== >> Test: KSP performance - Poisson >> Input matrix: 27-pt finite difference stencil >> -n 100 >> DoFs = 1000000 >> Number of nonzeros = 26463592 >> >> Step1 - creating Vecs and Mat... >> Step2 - running KSPSolve()... >> Step3 - calculating error norm... >> >> Error norm: 5.591e-02 >> KSP iters: 63 >> KSPSolve: 3.16646 seconds >> FOM: 3.158e+05 DoFs/sec >> =========================================== >> >> ------------------------------------------------------------ >> Sender: LSF System <lsfadmin@batch3> >> Subject: Job 3322694: <xolotlTest> in cluster <summit> Done >> >> Job <xolotlTest> was submitted from host <login2> by user <bqo> in >> cluster <summit> at Fri Mar 1 16:26:58 2024 >> Job was executed on host(s) <1*batch3>, in queue <debug>, as user <bqo> >> in cluster <summit> at Fri Mar 1 16:27:00 2024 >> <42*a35n05> >> </ccs/home/bqo> was used as the home directory. >> </gpfs/alpine2/mat267/scratch/bqo/test> was used as the working directory. >> Started at Fri Mar 1 16:27:00 2024 >> Terminated at Fri Mar 1 16:27:26 2024 >> Results reported at Fri Mar 1 16:27:26 2024 >> >> The output (if any) is above this job summary. >> >> >> If I switch to "jsrun --smpiargs "-gpu" -n 6 -a 1 -c 1 -g 1 >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos/src/ksp/ksp/tutorials/bench_kspsolve >> -mat_type aijkokkos -use_gpu_aware_mpi 1" it complains: >> Fri Mar 1 16:25:02 EST 2024 >> =========================================== >> Test: KSP performance - Poisson >> Input matrix: 27-pt finite difference stencil >> -n 100 >> DoFs = 1000000 >> Number of nonzeros = 26463592 >> >> Step1 - creating Vecs and Mat... >> [5]PETSC ERROR: PETSc is configured with GPU support, but your MPI is not >> GPU-aware. For better performance, please use a GPU-aware MPI. >> [5]PETSC ERROR: If you do not care, add option -use_gpu_aware_mpi 0. To >> not see the message again, add the option to your .petscrc, OR add it to >> the env var PETSC_OPTIONS. >> [5]PETSC ERROR: If you do care, for IBM Spectrum MPI on OLCF Summit, you >> may need jsrun --smpiargs=-gpu. >> [5]PETSC ERROR: For Open MPI, you need to configure it --with-cuda ( >> https://www.open-mpi.org/faq/?category=buildcuda) >> [5]PETSC ERROR: For MVAPICH2-GDR, you need to set MV2_USE_CUDA=1 ( >> http://mvapich.cse.ohio-state.edu/userguide/gdr/) >> [5]PETSC ERROR: For Cray-MPICH, you need to set >> MPICH_GPU_SUPPORT_ENABLED=1 (man mpi to see manual of cray-mpich) >> -------------------------------------------------------------------------- >> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF >> with errorcode 76. >> >> Best, >> >> Sophie >> ------------------------------ >> *From:* Junchao Zhang <jun...@gm...> >> *Sent:* Thursday, February 29, 2024 17:09 >> *To:* Blondel, Sophie <sbl...@ut...> >> *Cc:* xol...@li... < >> xol...@li...>; pet...@mc... < >> pet...@mc...> >> *Subject:* Re: [petsc-users] PAMI error on Summit >> >> You don't often get email from jun...@gm.... Learn why this >> is important <https://aka.ms/LearnAboutSenderIdentification> >> Could you try a petsc example to see if the environment is good? >> For example, >> >> cd src/ksp/ksp/tutorials >> make bench_kspsolve >> mpirun -n 6 ./bench_kspsolve -mat_type aijkokkos -use_gpu_aware_mpi {0 >> or 1} >> >> BTW, I remember to use gpu-aware mpi on Summit, one needs to pass >> --smpiargs "-gpu" to jsrun >> >> --Junchao Zhang >> >> >> On Thu, Feb 29, 2024 at 3:22 PM Blondel, Sophie via petsc-users < >> pet...@mc...> wrote: >> >> I still get the same error when deactivating GPU-aware MPI. I also tried >> unloading spectrum MPI and using openMPI instead (recompiling everything) >> and I get a segfault in PETSc in that case (still using GPU-aware MPI I >> think, at least not explicitly >> ZjQcmQRYFpfptBannerStart >> This Message Is From an External Sender >> This message came from outside your organization. >> >> ZjQcmQRYFpfptBannerEnd >> I still get the same error when deactivating GPU-aware MPI. >> >> I also tried unloading spectrum MPI and using openMPI instead >> (recompiling everything) and I get a segfault in PETSc in that case (still >> using GPU-aware MPI I think, at least not explicitly turning it off): >> >> 0 TS dt 1e-12 time 0. >> >> [ERROR] [0]PETSC ERROR: >> >> [ERROR] >> ------------------------------------------------------------------------ >> >> [ERROR] [0]PETSC ERROR: >> >> [ERROR] Caught signal number 11 SEGV: Segmentation Violation, probably >> memory access out of range >> >> [ERROR] [0]PETSC ERROR: >> >> [ERROR] Try option -start_in_debugger or -on_error_attach_debugger >> >> [ERROR] [0]PETSC ERROR: >> >> [ERROR] or see https://petsc.org/release/faq/#valgrind >> <https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__petsc.org_release_faq_-23valgrind&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=Loebf9sk4dgXGOOKPK3IHxp-C5SjGtr7Svr49LwaM4E&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI0jsXekj$> and >> https://petsc.org/release/faq/ >> <https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__petsc.org_release_faq_&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=7e9oLVYLacda_1-8rSkzDEHL4Zy1BFnO4pnrfMNlgO4&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI74qqyaL$> >> >> [ERROR] [0]PETSC ERROR: >> >> [ERROR] or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html >> <https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__docs.nvidia.com_cuda_cuda-2Dmemcheck_index.html&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=2gHentsiEM2njpPim4k40mYA96k7v_ivjI3erSECebM&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI3YGCBJ5$> on >> NVIDIA CUDA systems to find memory corruption errors >> >> [ERROR] [0]PETSC ERROR: >> >> [ERROR] configure using --with-debugging=yes, recompile, link, and run >> >> [ERROR] [0]PETSC ERROR: >> >> [ERROR] to get more information on the crash. >> >> [ERROR] [0]PETSC ERROR: >> >> [ERROR] Run with -malloc_debug to check if memory corruption is causing >> the crash. >> >> -------------------------------------------------------------------------- >> >> Best, >> >> Sophie >> ------------------------------ >> *From:* Blondel, Sophie via Xolotl-psi-development < >> xol...@li...> >> *Sent:* Thursday, February 29, 2024 10:17 >> *To:* xol...@li... < >> xol...@li...>; pet...@mc... < >> pet...@mc...> >> *Subject:* [Xolotl-psi-development] PAMI error on Summit >> >> Hi, >> >> I am using PETSc build with the Kokkos CUDA backend on Summit but when I >> run my code with multiple MPI tasks I get the following error: >> 0 TS dt 1e-12 time 0. >> errno 14 pid 864558 >> xolotl: >> /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: >> size_t PAMI::Dev >> ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, >> PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. >> errno 14 pid 864557 >> xolotl: >> /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: >> size_t PAMI::Dev >> ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, >> PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. >> [e28n07:864557] *** Process received signal *** >> [e28n07:864557] Signal: Aborted (6) >> [e28n07:864557] Signal code: (-6) >> [e28n07:864557] [ 0] >> linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] >> [e28n07:864557] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so >> (gsignal+0xd8)[0x200005d796f8] >> [e28n07:864557] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so >> (abort+0x164)[0x200005d53ff4] >> [e28n07:864557] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so >> (+0x3d280)[0x200005d6d280] >> [e28n07:864557] [ 4] [e28n07:864558] *** Process received signal *** >> [e28n07:864558] Signal: Aborted (6) >> [e28n07:864558] Signal code: (-6) >> [e28n07:864558] [ 0] >> linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] >> [e28n07:864558] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so >> (gsignal+0xd8)[0x200005d796f8] >> [e28n07:864558] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so >> (abort+0x164)[0x200005d53ff4] >> [e28n07:864558] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so >> (+0x3d280)[0x200005d6d280] >> [e28n07:864558] [ 4] /lib64/glibc-hwcaps/power9/libc-2.28.so >> (__assert_fail+0x64)[0x200005d6d324] >> [e28n07:864557] [ 5] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> >> (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt >> >> omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] >> [e28n07:864557] [ 6] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> >> (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] >> [e28n07:864557] [ 7] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] >> [e28n07:864557] [ 8] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (PAMI_Rget+0x18)[0x20007f2d94a8] >> [e28n07:864557] [ 9] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p >> ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] >> [e28n07:864557] [10] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p >> ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] >> [e28n07:864557] [11] /lib64/glibc-hwcaps/power9/libc-2.28.so >> (__assert_fail+0x64)[0x200005d6d324] >> [e28n07:864558] [ 5] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> >> (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt >> >> omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] >> [e28n07:864558] [ 6] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> >> (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] >> [e28n07:864558] [ 7] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] >> [e28n07:864558] [ 8] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (PAMI_Rget+0x18)[0x20007f2d94a8] >> [e28n07:864558] [ 9] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p >> ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] >> [e28n07:864558] [10] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p >> ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] >> [e28n07:864558] [11] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> >> (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic >> >> 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] >> [e28n07:864557] [12] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (PAMI_Context_advancev+0x6b0)[0x20007f2da540] >> [e28n07:864557] [13] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p >> ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] >> [e28n07:864557] [14] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ >> progress+0x6c)[0x20003d60640c] >> [e28n07:864557] [15] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r >> equest_default_wait_all+0x144)[0x2000034c4b04] >> [e28n07:864557] [16] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W >> aitall+0x10c)[0x20000352790c] >> [e28n07:864557] [17] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> >> (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic >> >> 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] >> [e28n07:864558] [12] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (PAMI_Context_advancev+0x6b0)[0x20007f2da540] >> [e28n07:864558] [13] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p >> ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] >> [e28n07:864558] [14] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ >> progress+0x6c)[0x20003d60640c] >> [e28n07:864558] [15] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r >> equest_default_wait_all+0x144)[0x2000034c4b04] >> [e28n07:864558] [16] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W >> aitall+0x10c)[0x20000352790c] >> [e28n07:864558] [17] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] >> [e28n07:864557] [18] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] >> [e28n07:864558] [18] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] >> [e28n07:864557] [19] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] >> [e28n07:864558] [19] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] >> [e28n07:864557] [20] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] >> [e28n07:864558] [20] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] >> [e28n07:864557] [21] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] >> [e28n07:864558] [21] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] >> [e28n07:864558] [22] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] >> [e28n07:864557] [22] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] >> [e28n07:864558] [23] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] >> [e28n07:864557] [23] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] >> [e28n07:864557] [24] >> /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 >> _+0xc4)[0x200005f710d4] >> [e28n07:864557] [25] >> /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 >> 00005f7130c] >> [e28n07:864557] [26] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] >> [e28n07:864558] [24] >> /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 >> _+0xc4)[0x200005f710d4] >> [e28n07:864558] [25] >> /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 >> 00005f7130c] >> [e28n07:864558] [26] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] >> [e28n07:864557] [27] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] >> [e28n07:864558] [27] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] >> [e28n07:864557] [28] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] >> [e28n07:864558] [28] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] >> [e28n07:864557] [29] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] >> [e28n07:864558] [29] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] >> [e28n07:864557] *** End of error message *** >> >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] >> [e28n07:864558] *** End of error message *** >> >> It seems to be pointing to >> https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/ >> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI30Ylvr6$> >> so I wanted to check if you had seen this type of error before and if it >> could be related to how the code is compiled or run. Let me know if I can >> provide any additional information. >> >> Best, >> >> Sophie >> >> |
From: Blondel, S. <sbl...@ut...> - 2024-03-01 23:07:39
|
I have been using --smpiargs "-gpu". I tried the benchmark with "jsrun --smpiargs "-gpu" -n 6 -a 1 -c 1 -g 1 /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos/src/ksp/ksp/tutorials/bench_kspsolve -mat_type aijkokkos -use_gpu_aware_mpi 0" and it seems to work: Fri Mar 1 16:27:14 EST 2024 =========================================== Test: KSP performance - Poisson Input matrix: 27-pt finite difference stencil -n 100 DoFs = 1000000 Number of nonzeros = 26463592 Step1 - creating Vecs and Mat... Step2 - running KSPSolve()... Step3 - calculating error norm... Error norm: 5.591e-02 KSP iters: 63 KSPSolve: 3.16646 seconds FOM: 3.158e+05 DoFs/sec =========================================== ------------------------------------------------------------ Sender: LSF System <lsfadmin@batch3> Subject: Job 3322694: <xolotlTest> in cluster <summit> Done Job <xolotlTest> was submitted from host <login2> by user <bqo> in cluster <summit> at Fri Mar 1 16:26:58 2024 Job was executed on host(s) <1*batch3>, in queue <debug>, as user <bqo> in cluster <summit> at Fri Mar 1 16:27:00 2024 <42*a35n05> </ccs/home/bqo> was used as the home directory. </gpfs/alpine2/mat267/scratch/bqo/test> was used as the working directory. Started at Fri Mar 1 16:27:00 2024 Terminated at Fri Mar 1 16:27:26 2024 Results reported at Fri Mar 1 16:27:26 2024 The output (if any) is above this job summary. If I switch to "jsrun --smpiargs "-gpu" -n 6 -a 1 -c 1 -g 1 /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos/src/ksp/ksp/tutorials/bench_kspsolve -mat_type aijkokkos -use_gpu_aware_mpi 1" it complains: Fri Mar 1 16:25:02 EST 2024 =========================================== Test: KSP performance - Poisson Input matrix: 27-pt finite difference stencil -n 100 DoFs = 1000000 Number of nonzeros = 26463592 Step1 - creating Vecs and Mat... [5]PETSC ERROR: PETSc is configured with GPU support, but your MPI is not GPU-aware. For better performance, please use a GPU-aware MPI. [5]PETSC ERROR: If you do not care, add option -use_gpu_aware_mpi 0. To not see the message again, add the option to your .petscrc, OR add it to the env var PETSC_OPTIONS. [5]PETSC ERROR: If you do care, for IBM Spectrum MPI on OLCF Summit, you may need jsrun --smpiargs=-gpu. [5]PETSC ERROR: For Open MPI, you need to configure it --with-cuda (https://www.open-mpi.org/faq/?category=buildcuda) [5]PETSC ERROR: For MVAPICH2-GDR, you need to set MV2_USE_CUDA=1 (http://mvapich.cse.ohio-state.edu/userguide/gdr/) [5]PETSC ERROR: For Cray-MPICH, you need to set MPICH_GPU_SUPPORT_ENABLED=1 (man mpi to see manual of cray-mpich) -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF with errorcode 76. Best, Sophie ________________________________ From: Junchao Zhang <jun...@gm...> Sent: Thursday, February 29, 2024 17:09 To: Blondel, Sophie <sbl...@ut...> Cc: xol...@li... <xol...@li...>; pet...@mc... <pet...@mc...> Subject: Re: [petsc-users] PAMI error on Summit You don't often get email from jun...@gm.... Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Could you try a petsc example to see if the environment is good? For example, cd src/ksp/ksp/tutorials make bench_kspsolve mpirun -n 6 ./bench_kspsolve -mat_type aijkokkos -use_gpu_aware_mpi {0 or 1} BTW, I remember to use gpu-aware mpi on Summit, one needs to pass --smpiargs "-gpu" to jsrun --Junchao Zhang On Thu, Feb 29, 2024 at 3:22 PM Blondel, Sophie via petsc-users <pet...@mc...<mailto:pet...@mc...>> wrote: I still get the same error when deactivating GPU-aware MPI. I also tried unloading spectrum MPI and using openMPI instead (recompiling everything) and I get a segfault in PETSc in that case (still using GPU-aware MPI I think, at least not explicitly ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I still get the same error when deactivating GPU-aware MPI. I also tried unloading spectrum MPI and using openMPI instead (recompiling everything) and I get a segfault in PETSc in that case (still using GPU-aware MPI I think, at least not explicitly turning it off): 0 TS dt 1e-12 time 0. [ERROR] [0]PETSC ERROR: [ERROR] ------------------------------------------------------------------------ [ERROR] [0]PETSC ERROR: [ERROR] Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [ERROR] [0]PETSC ERROR: [ERROR] Try option -start_in_debugger or -on_error_attach_debugger [ERROR] [0]PETSC ERROR: [ERROR] or see https://petsc.org/release/faq/#valgrind<https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__petsc.org_release_faq_-23valgrind&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=Loebf9sk4dgXGOOKPK3IHxp-C5SjGtr7Svr49LwaM4E&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI0jsXekj$> and https://petsc.org/release/faq/<https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__petsc.org_release_faq_&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=7e9oLVYLacda_1-8rSkzDEHL4Zy1BFnO4pnrfMNlgO4&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI74qqyaL$> [ERROR] [0]PETSC ERROR: [ERROR] or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html<https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__docs.nvidia.com_cuda_cuda-2Dmemcheck_index.html&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=2gHentsiEM2njpPim4k40mYA96k7v_ivjI3erSECebM&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI3YGCBJ5$> on NVIDIA CUDA systems to find memory corruption errors [ERROR] [0]PETSC ERROR: [ERROR] configure using --with-debugging=yes, recompile, link, and run [ERROR] [0]PETSC ERROR: [ERROR] to get more information on the crash. [ERROR] [0]PETSC ERROR: [ERROR] Run with -malloc_debug to check if memory corruption is causing the crash. -------------------------------------------------------------------------- Best, Sophie ________________________________ From: Blondel, Sophie via Xolotl-psi-development <xol...@li...<mailto:xol...@li...>> Sent: Thursday, February 29, 2024 10:17 To: xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>>; pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>> Subject: [Xolotl-psi-development] PAMI error on Summit Hi, I am using PETSc build with the Kokkos CUDA backend on Summit but when I run my code with multiple MPI tasks I get the following error: 0 TS dt 1e-12 time 0. errno 14 pid 864558 xolotl: /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: size_t PAMI::Dev ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. errno 14 pid 864557 xolotl: /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: size_t PAMI::Dev ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. [e28n07:864557] *** Process received signal *** [e28n07:864557] Signal: Aborted (6) [e28n07:864557] Signal code: (-6) [e28n07:864557] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] [e28n07:864557] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(gsignal+0xd8)[0x200005d796f8] [e28n07:864557] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(abort+0x164)[0x200005d53ff4] [e28n07:864557] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(+0x3d280)[0x200005d6d280] [e28n07:864557] [ 4] [e28n07:864558] *** Process received signal *** [e28n07:864558] Signal: Aborted (6) [e28n07:864558] Signal code: (-6) [e28n07:864558] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] [e28n07:864558] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(gsignal+0xd8)[0x200005d796f8] [e28n07:864558] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(abort+0x164)[0x200005d53ff4] [e28n07:864558] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(+0x3d280)[0x200005d6d280] [e28n07:864558] [ 4] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(__assert_fail+0x64)[0x200005d6d324] [e28n07:864557] [ 5] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] [e28n07:864557] [ 6] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] [e28n07:864557] [ 7] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] [e28n07:864557] [ 8] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (PAMI_Rget+0x18)[0x20007f2d94a8] [e28n07:864557] [ 9] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] [e28n07:864557] [10] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] [e28n07:864557] [11] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(__assert_fail+0x64)[0x200005d6d324] [e28n07:864558] [ 5] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] [e28n07:864558] [ 6] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] [e28n07:864558] [ 7] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] [e28n07:864558] [ 8] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (PAMI_Rget+0x18)[0x20007f2d94a8] [e28n07:864558] [ 9] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] [e28n07:864558] [10] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] [e28n07:864558] [11] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] [e28n07:864557] [12] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (PAMI_Context_advancev+0x6b0)[0x20007f2da540] [e28n07:864557] [13] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] [e28n07:864557] [14] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ progress+0x6c)[0x20003d60640c] [e28n07:864557] [15] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r equest_default_wait_all+0x144)[0x2000034c4b04] [e28n07:864557] [16] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W aitall+0x10c)[0x20000352790c] [e28n07:864557] [17] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] [e28n07:864558] [12] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (PAMI_Context_advancev+0x6b0)[0x20007f2da540] [e28n07:864558] [13] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] [e28n07:864558] [14] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ progress+0x6c)[0x20003d60640c] [e28n07:864558] [15] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r equest_default_wait_all+0x144)[0x2000034c4b04] [e28n07:864558] [16] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W aitall+0x10c)[0x20000352790c] [e28n07:864558] [17] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] [e28n07:864557] [18] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] [e28n07:864558] [18] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] [e28n07:864557] [19] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] [e28n07:864558] [19] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] [e28n07:864557] [20] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] [e28n07:864558] [20] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] [e28n07:864557] [21] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] [e28n07:864558] [21] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] [e28n07:864558] [22] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] [e28n07:864557] [22] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] [e28n07:864558] [23] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] [e28n07:864557] [23] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] [e28n07:864557] [24] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 _+0xc4)[0x200005f710d4] [e28n07:864557] [25] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 00005f7130c] [e28n07:864557] [26] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] [e28n07:864558] [24] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 _+0xc4)[0x200005f710d4] [e28n07:864558] [25] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 00005f7130c] [e28n07:864558] [26] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] [e28n07:864557] [27] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] [e28n07:864558] [27] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] [e28n07:864557] [28] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] [e28n07:864558] [28] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] [e28n07:864557] [29] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] [e28n07:864558] [29] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] [e28n07:864557] *** End of error message *** /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] [e28n07:864558] *** End of error message *** It seems to be pointing to https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/<https://urldefense.us/v3/__https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI30Ylvr6$> so I wanted to check if you had seen this type of error before and if it could be related to how the code is compiled or run. Let me know if I can provide any additional information. Best, Sophie |
From: Junchao Z. <jun...@gm...> - 2024-03-01 21:58:27
|
It is weird, with jsrun --smpiargs "-gpu" -n 6 -a 1 -c 1 -g 1 /gpfs/alpine2/mat267/proj- shared/dependencies/petsc-kokkos/src/ksp/ksp/tutorials/bench_kspsolve -mat_type aijkokkos -use_gpu_aware_mpi 1 petsc tried to test if the MPI is gpu aware (by doing an MPI_Allreduce on device buffers). It tried and found it was not, so it threw out the complaint in the error message. From https://docs.olcf.ornl.gov/systems/summit_user_guide.html#cuda-aware-mpi, I think your flags were right. I just got my Summit account reactivated today. I will give it a try. --Junchao Zhang On Fri, Mar 1, 2024 at 3:32 PM Blondel, Sophie <sbl...@ut...> wrote: > I have been using --smpiargs "-gpu". > > I tried the benchmark with "jsrun --smpiargs "-gpu" -n 6 -a 1 -c 1 -g 1 > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos/src/ksp/ksp/tutorials/bench_kspsolve > -mat_type aijkokkos -use_gpu_aware_mpi 0" and it seems to work: > Fri Mar 1 16:27:14 EST 2024 > =========================================== > Test: KSP performance - Poisson > Input matrix: 27-pt finite difference stencil > -n 100 > DoFs = 1000000 > Number of nonzeros = 26463592 > > Step1 - creating Vecs and Mat... > Step2 - running KSPSolve()... > Step3 - calculating error norm... > > Error norm: 5.591e-02 > KSP iters: 63 > KSPSolve: 3.16646 seconds > FOM: 3.158e+05 DoFs/sec > =========================================== > > ------------------------------------------------------------ > Sender: LSF System <lsfadmin@batch3> > Subject: Job 3322694: <xolotlTest> in cluster <summit> Done > > Job <xolotlTest> was submitted from host <login2> by user <bqo> in cluster > <summit> at Fri Mar 1 16:26:58 2024 > Job was executed on host(s) <1*batch3>, in queue <debug>, as user <bqo> in > cluster <summit> at Fri Mar 1 16:27:00 2024 > <42*a35n05> > </ccs/home/bqo> was used as the home directory. > </gpfs/alpine2/mat267/scratch/bqo/test> was used as the working directory. > Started at Fri Mar 1 16:27:00 2024 > Terminated at Fri Mar 1 16:27:26 2024 > Results reported at Fri Mar 1 16:27:26 2024 > > The output (if any) is above this job summary. > > > If I switch to "jsrun --smpiargs "-gpu" -n 6 -a 1 -c 1 -g 1 > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos/src/ksp/ksp/tutorials/bench_kspsolve > -mat_type aijkokkos -use_gpu_aware_mpi 1" it complains: > Fri Mar 1 16:25:02 EST 2024 > =========================================== > Test: KSP performance - Poisson > Input matrix: 27-pt finite difference stencil > -n 100 > DoFs = 1000000 > Number of nonzeros = 26463592 > > Step1 - creating Vecs and Mat... > [5]PETSC ERROR: PETSc is configured with GPU support, but your MPI is not > GPU-aware. For better performance, please use a GPU-aware MPI. > [5]PETSC ERROR: If you do not care, add option -use_gpu_aware_mpi 0. To > not see the message again, add the option to your .petscrc, OR add it to > the env var PETSC_OPTIONS. > [5]PETSC ERROR: If you do care, for IBM Spectrum MPI on OLCF Summit, you > may need jsrun --smpiargs=-gpu. > [5]PETSC ERROR: For Open MPI, you need to configure it --with-cuda ( > https://www.open-mpi.org/faq/?category=buildcuda) > [5]PETSC ERROR: For MVAPICH2-GDR, you need to set MV2_USE_CUDA=1 ( > http://mvapich.cse.ohio-state.edu/userguide/gdr/) > [5]PETSC ERROR: For Cray-MPICH, you need to set > MPICH_GPU_SUPPORT_ENABLED=1 (man mpi to see manual of cray-mpich) > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF > with errorcode 76. > > Best, > > Sophie > ------------------------------ > *From:* Junchao Zhang <jun...@gm...> > *Sent:* Thursday, February 29, 2024 17:09 > *To:* Blondel, Sophie <sbl...@ut...> > *Cc:* xol...@li... < > xol...@li...>; pet...@mc... < > pet...@mc...> > *Subject:* Re: [petsc-users] PAMI error on Summit > > You don't often get email from jun...@gm.... Learn why this is > important <https://aka.ms/LearnAboutSenderIdentification> > Could you try a petsc example to see if the environment is good? > For example, > > cd src/ksp/ksp/tutorials > make bench_kspsolve > mpirun -n 6 ./bench_kspsolve -mat_type aijkokkos -use_gpu_aware_mpi {0 or > 1} > > BTW, I remember to use gpu-aware mpi on Summit, one needs to pass > --smpiargs "-gpu" to jsrun > > --Junchao Zhang > > > On Thu, Feb 29, 2024 at 3:22 PM Blondel, Sophie via petsc-users < > pet...@mc...> wrote: > > I still get the same error when deactivating GPU-aware MPI. I also tried > unloading spectrum MPI and using openMPI instead (recompiling everything) > and I get a segfault in PETSc in that case (still using GPU-aware MPI I > think, at least not explicitly > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > I still get the same error when deactivating GPU-aware MPI. > > I also tried unloading spectrum MPI and using openMPI instead (recompiling > everything) and I get a segfault in PETSc in that case (still using > GPU-aware MPI I think, at least not explicitly turning it off): > > 0 TS dt 1e-12 time 0. > > [ERROR] [0]PETSC ERROR: > > [ERROR] > ------------------------------------------------------------------------ > > [ERROR] [0]PETSC ERROR: > > [ERROR] Caught signal number 11 SEGV: Segmentation Violation, probably > memory access out of range > > [ERROR] [0]PETSC ERROR: > > [ERROR] Try option -start_in_debugger or -on_error_attach_debugger > > [ERROR] [0]PETSC ERROR: > > [ERROR] or see https://petsc.org/release/faq/#valgrind > <https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__petsc.org_release_faq_-23valgrind&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=Loebf9sk4dgXGOOKPK3IHxp-C5SjGtr7Svr49LwaM4E&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI0jsXekj$> and > https://petsc.org/release/faq/ > <https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__petsc.org_release_faq_&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=7e9oLVYLacda_1-8rSkzDEHL4Zy1BFnO4pnrfMNlgO4&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI74qqyaL$> > > [ERROR] [0]PETSC ERROR: > > [ERROR] or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html > <https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__docs.nvidia.com_cuda_cuda-2Dmemcheck_index.html&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=2gHentsiEM2njpPim4k40mYA96k7v_ivjI3erSECebM&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI3YGCBJ5$> on > NVIDIA CUDA systems to find memory corruption errors > > [ERROR] [0]PETSC ERROR: > > [ERROR] configure using --with-debugging=yes, recompile, link, and run > > [ERROR] [0]PETSC ERROR: > > [ERROR] to get more information on the crash. > > [ERROR] [0]PETSC ERROR: > > [ERROR] Run with -malloc_debug to check if memory corruption is causing > the crash. > > -------------------------------------------------------------------------- > > Best, > > Sophie > ------------------------------ > *From:* Blondel, Sophie via Xolotl-psi-development < > xol...@li...> > *Sent:* Thursday, February 29, 2024 10:17 > *To:* xol...@li... < > xol...@li...>; pet...@mc... < > pet...@mc...> > *Subject:* [Xolotl-psi-development] PAMI error on Summit > > Hi, > > I am using PETSc build with the Kokkos CUDA backend on Summit but when I > run my code with multiple MPI tasks I get the following error: > 0 TS dt 1e-12 time 0. > errno 14 pid 864558 > xolotl: > /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: > size_t PAMI::Dev > ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, > PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. > errno 14 pid 864557 > xolotl: > /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: > size_t PAMI::Dev > ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, > PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. > [e28n07:864557] *** Process received signal *** > [e28n07:864557] Signal: Aborted (6) > [e28n07:864557] Signal code: (-6) > [e28n07:864557] [ 0] > linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] > [e28n07:864557] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so > (gsignal+0xd8)[0x200005d796f8] > [e28n07:864557] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so > (abort+0x164)[0x200005d53ff4] > [e28n07:864557] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so > (+0x3d280)[0x200005d6d280] > [e28n07:864557] [ 4] [e28n07:864558] *** Process received signal *** > [e28n07:864558] Signal: Aborted (6) > [e28n07:864558] Signal code: (-6) > [e28n07:864558] [ 0] > linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] > [e28n07:864558] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so > (gsignal+0xd8)[0x200005d796f8] > [e28n07:864558] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so > (abort+0x164)[0x200005d53ff4] > [e28n07:864558] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so > (+0x3d280)[0x200005d6d280] > [e28n07:864558] [ 4] /lib64/glibc-hwcaps/power9/libc-2.28.so > (__assert_fail+0x64)[0x200005d6d324] > [e28n07:864557] [ 5] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt > > omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] > [e28n07:864557] [ 6] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] > [e28n07:864557] [ 7] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] > [e28n07:864557] [ 8] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (PAMI_Rget+0x18)[0x20007f2d94a8] > [e28n07:864557] [ 9] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] > [e28n07:864557] [10] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] > [e28n07:864557] [11] /lib64/glibc-hwcaps/power9/libc-2.28.so > (__assert_fail+0x64)[0x200005d6d324] > [e28n07:864558] [ 5] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt > > omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] > [e28n07:864558] [ 6] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] > [e28n07:864558] [ 7] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] > [e28n07:864558] [ 8] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (PAMI_Rget+0x18)[0x20007f2d94a8] > [e28n07:864558] [ 9] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] > [e28n07:864558] [10] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] > [e28n07:864558] [11] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic > > 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] > [e28n07:864557] [12] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (PAMI_Context_advancev+0x6b0)[0x20007f2da540] > [e28n07:864557] [13] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] > [e28n07:864557] [14] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ > progress+0x6c)[0x20003d60640c] > [e28n07:864557] [15] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r > equest_default_wait_all+0x144)[0x2000034c4b04] > [e28n07:864557] [16] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W > aitall+0x10c)[0x20000352790c] > [e28n07:864557] [17] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic > > 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] > [e28n07:864558] [12] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (PAMI_Context_advancev+0x6b0)[0x20007f2da540] > [e28n07:864558] [13] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] > [e28n07:864558] [14] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ > progress+0x6c)[0x20003d60640c] > [e28n07:864558] [15] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r > equest_default_wait_all+0x144)[0x2000034c4b04] > [e28n07:864558] [16] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W > aitall+0x10c)[0x20000352790c] > [e28n07:864558] [17] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] > [e28n07:864557] [18] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] > [e28n07:864558] [18] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] > [e28n07:864557] [19] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] > [e28n07:864558] [19] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] > [e28n07:864557] [20] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] > [e28n07:864558] [20] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] > [e28n07:864557] [21] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] > [e28n07:864558] [21] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] > [e28n07:864558] [22] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] > [e28n07:864557] [22] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] > [e28n07:864558] [23] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] > [e28n07:864557] [23] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] > [e28n07:864557] [24] > /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 > _+0xc4)[0x200005f710d4] > [e28n07:864557] [25] > /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 > 00005f7130c] > [e28n07:864557] [26] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] > [e28n07:864558] [24] > /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 > _+0xc4)[0x200005f710d4] > [e28n07:864558] [25] > /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 > 00005f7130c] > [e28n07:864558] [26] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] > [e28n07:864557] [27] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] > [e28n07:864558] [27] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] > [e28n07:864557] [28] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] > [e28n07:864558] [28] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] > [e28n07:864557] [29] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] > [e28n07:864558] [29] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] > [e28n07:864557] *** End of error message *** > > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] > [e28n07:864558] *** End of error message *** > > It seems to be pointing to > https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/ > <https://urldefense.us/v3/__https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI30Ylvr6$> > so I wanted to check if you had seen this type of error before and if it > could be related to how the code is compiled or run. Let me know if I can > provide any additional information. > > Best, > > Sophie > > |
From: Junchao Z. <jun...@gm...> - 2024-02-29 22:10:00
|
Could you try a petsc example to see if the environment is good? For example, cd src/ksp/ksp/tutorials make bench_kspsolve mpirun -n 6 ./bench_kspsolve -mat_type aijkokkos -use_gpu_aware_mpi {0 or 1} BTW, I remember to use gpu-aware mpi on Summit, one needs to pass --smpiargs "-gpu" to jsrun --Junchao Zhang On Thu, Feb 29, 2024 at 3:22 PM Blondel, Sophie via petsc-users < pet...@mc...> wrote: > I still get the same error when deactivating GPU-aware MPI. I also tried > unloading spectrum MPI and using openMPI instead (recompiling everything) > and I get a segfault in PETSc in that case (still using GPU-aware MPI I > think, at least not explicitly > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > I still get the same error when deactivating GPU-aware MPI. > > I also tried unloading spectrum MPI and using openMPI instead (recompiling > everything) and I get a segfault in PETSc in that case (still using > GPU-aware MPI I think, at least not explicitly turning it off): > > 0 TS dt 1e-12 time 0. > > [ERROR] [0]PETSC ERROR: > > [ERROR] > ------------------------------------------------------------------------ > > [ERROR] [0]PETSC ERROR: > > [ERROR] Caught signal number 11 SEGV: Segmentation Violation, probably > memory access out of range > > [ERROR] [0]PETSC ERROR: > > [ERROR] Try option -start_in_debugger or -on_error_attach_debugger > > [ERROR] [0]PETSC ERROR: > > [ERROR] or see https://petsc.org/release/faq/#valgrind > <https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__petsc.org_release_faq_-23valgrind&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=Loebf9sk4dgXGOOKPK3IHxp-C5SjGtr7Svr49LwaM4E&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI0jsXekj$> and > https://petsc.org/release/faq/ > <https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__petsc.org_release_faq_&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=7e9oLVYLacda_1-8rSkzDEHL4Zy1BFnO4pnrfMNlgO4&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI74qqyaL$> > > [ERROR] [0]PETSC ERROR: > > [ERROR] or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html > <https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__docs.nvidia.com_cuda_cuda-2Dmemcheck_index.html&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=2gHentsiEM2njpPim4k40mYA96k7v_ivjI3erSECebM&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI3YGCBJ5$> on > NVIDIA CUDA systems to find memory corruption errors > > [ERROR] [0]PETSC ERROR: > > [ERROR] configure using --with-debugging=yes, recompile, link, and run > > [ERROR] [0]PETSC ERROR: > > [ERROR] to get more information on the crash. > > [ERROR] [0]PETSC ERROR: > > [ERROR] Run with -malloc_debug to check if memory corruption is causing > the crash. > > -------------------------------------------------------------------------- > > Best, > > Sophie > ------------------------------ > *From:* Blondel, Sophie via Xolotl-psi-development < > xol...@li...> > *Sent:* Thursday, February 29, 2024 10:17 > *To:* xol...@li... < > xol...@li...>; pet...@mc... < > pet...@mc...> > *Subject:* [Xolotl-psi-development] PAMI error on Summit > > Hi, > > I am using PETSc build with the Kokkos CUDA backend on Summit but when I > run my code with multiple MPI tasks I get the following error: > 0 TS dt 1e-12 time 0. > errno 14 pid 864558 > xolotl: > /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: > size_t PAMI::Dev > ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, > PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. > errno 14 pid 864557 > xolotl: > /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: > size_t PAMI::Dev > ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, > PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. > [e28n07:864557] *** Process received signal *** > [e28n07:864557] Signal: Aborted (6) > [e28n07:864557] Signal code: (-6) > [e28n07:864557] [ 0] > linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] > [e28n07:864557] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so > (gsignal+0xd8)[0x200005d796f8] > [e28n07:864557] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so > (abort+0x164)[0x200005d53ff4] > [e28n07:864557] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so > (+0x3d280)[0x200005d6d280] > [e28n07:864557] [ 4] [e28n07:864558] *** Process received signal *** > [e28n07:864558] Signal: Aborted (6) > [e28n07:864558] Signal code: (-6) > [e28n07:864558] [ 0] > linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] > [e28n07:864558] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so > (gsignal+0xd8)[0x200005d796f8] > [e28n07:864558] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so > (abort+0x164)[0x200005d53ff4] > [e28n07:864558] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so > (+0x3d280)[0x200005d6d280] > [e28n07:864558] [ 4] /lib64/glibc-hwcaps/power9/libc-2.28.so > (__assert_fail+0x64)[0x200005d6d324] > [e28n07:864557] [ 5] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt > > omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] > [e28n07:864557] [ 6] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] > [e28n07:864557] [ 7] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] > [e28n07:864557] [ 8] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (PAMI_Rget+0x18)[0x20007f2d94a8] > [e28n07:864557] [ 9] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] > [e28n07:864557] [10] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] > [e28n07:864557] [11] /lib64/glibc-hwcaps/power9/libc-2.28.so > (__assert_fail+0x64)[0x200005d6d324] > [e28n07:864558] [ 5] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt > > omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] > [e28n07:864558] [ 6] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] > [e28n07:864558] [ 7] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] > [e28n07:864558] [ 8] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (PAMI_Rget+0x18)[0x20007f2d94a8] > [e28n07:864558] [ 9] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] > [e28n07:864558] [10] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] > [e28n07:864558] [11] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic > > 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] > [e28n07:864557] [12] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (PAMI_Context_advancev+0x6b0)[0x20007f2da540] > [e28n07:864557] [13] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] > [e28n07:864557] [14] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ > progress+0x6c)[0x20003d60640c] > [e28n07:864557] [15] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r > equest_default_wait_all+0x144)[0x2000034c4b04] > [e28n07:864557] [16] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W > aitall+0x10c)[0x20000352790c] > [e28n07:864557] [17] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic > > 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] > [e28n07:864558] [12] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (PAMI_Context_advancev+0x6b0)[0x20007f2da540] > [e28n07:864558] [13] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] > [e28n07:864558] [14] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ > progress+0x6c)[0x20003d60640c] > [e28n07:864558] [15] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r > equest_default_wait_all+0x144)[0x2000034c4b04] > [e28n07:864558] [16] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W > aitall+0x10c)[0x20000352790c] > [e28n07:864558] [17] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] > [e28n07:864557] [18] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] > [e28n07:864558] [18] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] > [e28n07:864557] [19] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] > [e28n07:864558] [19] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] > [e28n07:864557] [20] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] > [e28n07:864558] [20] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] > [e28n07:864557] [21] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] > [e28n07:864558] [21] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] > [e28n07:864558] [22] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] > [e28n07:864557] [22] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] > [e28n07:864558] [23] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] > [e28n07:864557] [23] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] > [e28n07:864557] [24] > /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 > _+0xc4)[0x200005f710d4] > [e28n07:864557] [25] > /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 > 00005f7130c] > [e28n07:864557] [26] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] > [e28n07:864558] [24] > /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 > _+0xc4)[0x200005f710d4] > [e28n07:864558] [25] > /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 > 00005f7130c] > [e28n07:864558] [26] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] > [e28n07:864557] [27] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] > [e28n07:864558] [27] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] > [e28n07:864557] [28] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] > [e28n07:864558] [28] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] > [e28n07:864557] [29] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] > [e28n07:864558] [29] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] > [e28n07:864557] *** End of error message *** > > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] > [e28n07:864558] *** End of error message *** > > It seems to be pointing to > https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/ > <https://urldefense.us/v3/__https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI30Ylvr6$> > so I wanted to check if you had seen this type of error before and if it > could be related to how the code is compiled or run. Let me know if I can > provide any additional information. > > Best, > > Sophie > |
From: Matthew K. <kn...@gm...> - 2024-02-29 21:40:53
|
On Thu, Feb 29, 2024 at 4:22 PM Blondel, Sophie via petsc-users < pet...@mc...> wrote: > I still get the same error when deactivating GPU-aware MPI. I also tried > unloading spectrum MPI and using openMPI instead (recompiling everything) > and I get a segfault in PETSc in that case (still using GPU-aware MPI I > think, at least not explicitly > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > I still get the same error when deactivating GPU-aware MPI. > > I also tried unloading spectrum MPI and using openMPI instead (recompiling > everything) and I get a segfault in PETSc in that case (still using > GPU-aware MPI I think, at least not explicitly turning it off): > For this case, can you get a stack trace? Thanks, Matt > 0 TS dt 1e-12 time 0. > > [ERROR] [0]PETSC ERROR: > > [ERROR] > ------------------------------------------------------------------------ > > [ERROR] [0]PETSC ERROR: > > [ERROR] Caught signal number 11 SEGV: Segmentation Violation, probably > memory access out of range > > [ERROR] [0]PETSC ERROR: > > [ERROR] Try option -start_in_debugger or -on_error_attach_debugger > > [ERROR] [0]PETSC ERROR: > > [ERROR] or see https://petsc.org/release/faq/#valgrind > <https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__petsc.org_release_faq_-23valgrind&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=Loebf9sk4dgXGOOKPK3IHxp-C5SjGtr7Svr49LwaM4E&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI0jsXekj$> and > https://petsc.org/release/faq/ > <https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__petsc.org_release_faq_&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=7e9oLVYLacda_1-8rSkzDEHL4Zy1BFnO4pnrfMNlgO4&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI74qqyaL$> > > [ERROR] [0]PETSC ERROR: > > [ERROR] or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html > <https://urldefense.us/v3/__https://urldefense.us/v2/url?u=https-3A__docs.nvidia.com_cuda_cuda-2Dmemcheck_index.html&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=2gHentsiEM2njpPim4k40mYA96k7v_ivjI3erSECebM&e=__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI3YGCBJ5$> on > NVIDIA CUDA systems to find memory corruption errors > > [ERROR] [0]PETSC ERROR: > > [ERROR] configure using --with-debugging=yes, recompile, link, and run > > [ERROR] [0]PETSC ERROR: > > [ERROR] to get more information on the crash. > > [ERROR] [0]PETSC ERROR: > > [ERROR] Run with -malloc_debug to check if memory corruption is causing > the crash. > > -------------------------------------------------------------------------- > > Best, > > Sophie > ------------------------------ > *From:* Blondel, Sophie via Xolotl-psi-development < > xol...@li...> > *Sent:* Thursday, February 29, 2024 10:17 > *To:* xol...@li... < > xol...@li...>; pet...@mc... < > pet...@mc...> > *Subject:* [Xolotl-psi-development] PAMI error on Summit > > Hi, > > I am using PETSc build with the Kokkos CUDA backend on Summit but when I > run my code with multiple MPI tasks I get the following error: > 0 TS dt 1e-12 time 0. > errno 14 pid 864558 > xolotl: > /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: > size_t PAMI::Dev > ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, > PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. > errno 14 pid 864557 > xolotl: > /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: > size_t PAMI::Dev > ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, > PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. > [e28n07:864557] *** Process received signal *** > [e28n07:864557] Signal: Aborted (6) > [e28n07:864557] Signal code: (-6) > [e28n07:864557] [ 0] > linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] > [e28n07:864557] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so > (gsignal+0xd8)[0x200005d796f8] > [e28n07:864557] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so > (abort+0x164)[0x200005d53ff4] > [e28n07:864557] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so > (+0x3d280)[0x200005d6d280] > [e28n07:864557] [ 4] [e28n07:864558] *** Process received signal *** > [e28n07:864558] Signal: Aborted (6) > [e28n07:864558] Signal code: (-6) > [e28n07:864558] [ 0] > linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] > [e28n07:864558] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so > (gsignal+0xd8)[0x200005d796f8] > [e28n07:864558] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so > (abort+0x164)[0x200005d53ff4] > [e28n07:864558] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so > (+0x3d280)[0x200005d6d280] > [e28n07:864558] [ 4] /lib64/glibc-hwcaps/power9/libc-2.28.so > (__assert_fail+0x64)[0x200005d6d324] > [e28n07:864557] [ 5] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt > > omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] > [e28n07:864557] [ 6] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] > [e28n07:864557] [ 7] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] > [e28n07:864557] [ 8] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (PAMI_Rget+0x18)[0x20007f2d94a8] > [e28n07:864557] [ 9] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] > [e28n07:864557] [10] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] > [e28n07:864557] [11] /lib64/glibc-hwcaps/power9/libc-2.28.so > (__assert_fail+0x64)[0x200005d6d324] > [e28n07:864558] [ 5] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt > > omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] > [e28n07:864558] [ 6] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] > [e28n07:864558] [ 7] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] > [e28n07:864558] [ 8] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (PAMI_Rget+0x18)[0x20007f2d94a8] > [e28n07:864558] [ 9] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] > [e28n07:864558] [10] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] > [e28n07:864558] [11] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic > > 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] > [e28n07:864557] [12] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (PAMI_Context_advancev+0x6b0)[0x20007f2da540] > [e28n07:864557] [13] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] > [e28n07:864557] [14] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ > progress+0x6c)[0x20003d60640c] > [e28n07:864557] [15] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r > equest_default_wait_all+0x144)[0x2000034c4b04] > [e28n07:864557] [16] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W > aitall+0x10c)[0x20000352790c] > [e28n07:864557] [17] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic > > 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] > [e28n07:864558] [12] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (PAMI_Context_advancev+0x6b0)[0x20007f2da540] > [e28n07:864558] [13] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] > [e28n07:864558] [14] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ > progress+0x6c)[0x20003d60640c] > [e28n07:864558] [15] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r > equest_default_wait_all+0x144)[0x2000034c4b04] > [e28n07:864558] [16] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W > aitall+0x10c)[0x20000352790c] > [e28n07:864558] [17] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] > [e28n07:864557] [18] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] > [e28n07:864558] [18] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] > [e28n07:864557] [19] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] > [e28n07:864558] [19] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] > [e28n07:864557] [20] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] > [e28n07:864558] [20] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] > [e28n07:864557] [21] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] > [e28n07:864558] [21] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] > [e28n07:864558] [22] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] > [e28n07:864557] [22] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] > [e28n07:864558] [23] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] > [e28n07:864557] [23] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] > [e28n07:864557] [24] > /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 > _+0xc4)[0x200005f710d4] > [e28n07:864557] [25] > /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 > 00005f7130c] > [e28n07:864557] [26] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] > [e28n07:864558] [24] > /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 > _+0xc4)[0x200005f710d4] > [e28n07:864558] [25] > /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 > 00005f7130c] > [e28n07:864558] [26] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] > [e28n07:864557] [27] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] > [e28n07:864558] [27] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] > [e28n07:864557] [28] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] > [e28n07:864558] [28] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] > [e28n07:864557] [29] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] > [e28n07:864558] [29] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] > [e28n07:864557] *** End of error message *** > > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] > [e28n07:864558] *** End of error message *** > > It seems to be pointing to > https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/ > <https://urldefense.us/v3/__https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/__;!!G_uCfscf7eWS!bhpq7UF4Rq9PhMMRRb_zeSflUb9Cs5My48ggt02OxSWxoM4eIU_MDt3H6e2YnrxJizIsA21q76YdORVhI30Ylvr6$> > so I wanted to check if you had seen this type of error before and if it > could be related to how the code is compiled or run. Let me know if I can > provide any additional information. > > Best, > > Sophie > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/> |
From: Blondel, S. <sbl...@ut...> - 2024-02-29 21:37:44
|
I still get the same error when deactivating GPU-aware MPI. I also tried unloading spectrum MPI and using openMPI instead (recompiling everything) and I get a segfault in PETSc in that case (still using GPU-aware MPI I think, at least not explicitly turning it off): 0 TS dt 1e-12 time 0. [ERROR] [0]PETSC ERROR: [ERROR] ------------------------------------------------------------------------ [ERROR] [0]PETSC ERROR: [ERROR] Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [ERROR] [0]PETSC ERROR: [ERROR] Try option -start_in_debugger or -on_error_attach_debugger [ERROR] [0]PETSC ERROR: [ERROR] or see https://petsc.org/release/faq/#valgrind<https://urldefense.us/v2/url?u=https-3A__petsc.org_release_faq_-23valgrind&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=Loebf9sk4dgXGOOKPK3IHxp-C5SjGtr7Svr49LwaM4E&e=> and https://petsc.org/release/faq/<https://urldefense.us/v2/url?u=https-3A__petsc.org_release_faq_&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=7e9oLVYLacda_1-8rSkzDEHL4Zy1BFnO4pnrfMNlgO4&e=> [ERROR] [0]PETSC ERROR: [ERROR] or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html<https://urldefense.us/v2/url?u=https-3A__docs.nvidia.com_cuda_cuda-2Dmemcheck_index.html&d=DwQGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=SNsmM8pc4pmx4j-bqFq40w&m=1GLMwF9jewRd8MBil83VSwu-tVEn7Tkm_YfSAcgEMsZ9hDb2HvlnscmeqXsnzv5S&s=2gHentsiEM2njpPim4k40mYA96k7v_ivjI3erSECebM&e=> on NVIDIA CUDA systems to find memory corruption errors [ERROR] [0]PETSC ERROR: [ERROR] configure using --with-debugging=yes, recompile, link, and run [ERROR] [0]PETSC ERROR: [ERROR] to get more information on the crash. [ERROR] [0]PETSC ERROR: [ERROR] Run with -malloc_debug to check if memory corruption is causing the crash. -------------------------------------------------------------------------- Best, Sophie ________________________________ From: Blondel, Sophie via Xolotl-psi-development <xol...@li...> Sent: Thursday, February 29, 2024 10:17 To: xol...@li... <xol...@li...>; pet...@mc... <pet...@mc...> Subject: [Xolotl-psi-development] PAMI error on Summit Hi, I am using PETSc build with the Kokkos CUDA backend on Summit but when I run my code with multiple MPI tasks I get the following error: 0 TS dt 1e-12 time 0. errno 14 pid 864558 xolotl: /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: size_t PAMI::Dev ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. errno 14 pid 864557 xolotl: /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: size_t PAMI::Dev ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. [e28n07:864557] *** Process received signal *** [e28n07:864557] Signal: Aborted (6) [e28n07:864557] Signal code: (-6) [e28n07:864557] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] [e28n07:864557] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so(gsignal+0xd8)[0x200005d796f8] [e28n07:864557] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so(abort+0x164)[0x200005d53ff4] [e28n07:864557] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so(+0x3d280)[0x200005d6d280] [e28n07:864557] [ 4] [e28n07:864558] *** Process received signal *** [e28n07:864558] Signal: Aborted (6) [e28n07:864558] Signal code: (-6) [e28n07:864558] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] [e28n07:864558] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so(gsignal+0xd8)[0x200005d796f8] [e28n07:864558] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so(abort+0x164)[0x200005d53ff4] [e28n07:864558] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so(+0x3d280)[0x200005d6d280] [e28n07:864558] [ 4] /lib64/glibc-hwcaps/power9/libc-2.28.so(__assert_fail+0x64)[0x200005d6d324] [e28n07:864557] [ 5] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] [e28n07:864557] [ 6] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] [e28n07:864557] [ 7] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] [e28n07:864557] [ 8] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (PAMI_Rget+0x18)[0x20007f2d94a8] [e28n07:864557] [ 9] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] [e28n07:864557] [10] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] [e28n07:864557] [11] /lib64/glibc-hwcaps/power9/libc-2.28.so(__assert_fail+0x64)[0x200005d6d324] [e28n07:864558] [ 5] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] [e28n07:864558] [ 6] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] [e28n07:864558] [ 7] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] [e28n07:864558] [ 8] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (PAMI_Rget+0x18)[0x20007f2d94a8] [e28n07:864558] [ 9] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] [e28n07:864558] [10] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] [e28n07:864558] [11] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] [e28n07:864557] [12] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (PAMI_Context_advancev+0x6b0)[0x20007f2da540] [e28n07:864557] [13] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] [e28n07:864557] [14] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ progress+0x6c)[0x20003d60640c] [e28n07:864557] [15] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r equest_default_wait_all+0x144)[0x2000034c4b04] [e28n07:864557] [16] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W aitall+0x10c)[0x20000352790c] [e28n07:864557] [17] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] [e28n07:864558] [12] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (PAMI_Context_advancev+0x6b0)[0x20007f2da540] [e28n07:864558] [13] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] [e28n07:864558] [14] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ progress+0x6c)[0x20003d60640c] [e28n07:864558] [15] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r equest_default_wait_all+0x144)[0x2000034c4b04] [e28n07:864558] [16] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W aitall+0x10c)[0x20000352790c] [e28n07:864558] [17] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] [e28n07:864557] [18] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] [e28n07:864558] [18] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] [e28n07:864557] [19] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] [e28n07:864558] [19] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] [e28n07:864557] [20] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] [e28n07:864558] [20] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] [e28n07:864557] [21] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] [e28n07:864558] [21] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] [e28n07:864558] [22] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] [e28n07:864557] [22] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] [e28n07:864558] [23] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] [e28n07:864557] [23] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] [e28n07:864557] [24] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 _+0xc4)[0x200005f710d4] [e28n07:864557] [25] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 00005f7130c] [e28n07:864557] [26] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] [e28n07:864558] [24] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 _+0xc4)[0x200005f710d4] [e28n07:864558] [25] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 00005f7130c] [e28n07:864558] [26] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] [e28n07:864557] [27] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] [e28n07:864558] [27] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] [e28n07:864557] [28] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] [e28n07:864558] [28] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] [e28n07:864557] [29] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] [e28n07:864558] [29] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] [e28n07:864557] *** End of error message *** /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] [e28n07:864558] *** End of error message *** It seems to be pointing to https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/ so I wanted to check if you had seen this type of error before and if it could be related to how the code is compiled or run. Let me know if I can provide any additional information. Best, Sophie |
From: Blondel, S. <sbl...@ut...> - 2024-02-29 16:49:41
|
Hi, I am using PETSc build with the Kokkos CUDA backend on Summit but when I run my code with multiple MPI tasks I get the following error: 0 TS dt 1e-12 time 0. errno 14 pid 864558 xolotl: /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: size_t PAMI::Dev ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. errno 14 pid 864557 xolotl: /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: size_t PAMI::Dev ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. [e28n07:864557] *** Process received signal *** [e28n07:864557] Signal: Aborted (6) [e28n07:864557] Signal code: (-6) [e28n07:864557] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] [e28n07:864557] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so(gsignal+0xd8)[0x200005d796f8] [e28n07:864557] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so(abort+0x164)[0x200005d53ff4] [e28n07:864557] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so(+0x3d280)[0x200005d6d280] [e28n07:864557] [ 4] [e28n07:864558] *** Process received signal *** [e28n07:864558] Signal: Aborted (6) [e28n07:864558] Signal code: (-6) [e28n07:864558] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] [e28n07:864558] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so(gsignal+0xd8)[0x200005d796f8] [e28n07:864558] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so(abort+0x164)[0x200005d53ff4] [e28n07:864558] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so(+0x3d280)[0x200005d6d280] [e28n07:864558] [ 4] /lib64/glibc-hwcaps/power9/libc-2.28.so(__assert_fail+0x64)[0x200005d6d324] [e28n07:864557] [ 5] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] [e28n07:864557] [ 6] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] [e28n07:864557] [ 7] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] [e28n07:864557] [ 8] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (PAMI_Rget+0x18)[0x20007f2d94a8] [e28n07:864557] [ 9] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] [e28n07:864557] [10] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] [e28n07:864557] [11] /lib64/glibc-hwcaps/power9/libc-2.28.so(__assert_fail+0x64)[0x200005d6d324] [e28n07:864558] [ 5] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] [e28n07:864558] [ 6] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] [e28n07:864558] [ 7] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] [e28n07:864558] [ 8] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (PAMI_Rget+0x18)[0x20007f2d94a8] [e28n07:864558] [ 9] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] [e28n07:864558] [10] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] [e28n07:864558] [11] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] [e28n07:864557] [12] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (PAMI_Context_advancev+0x6b0)[0x20007f2da540] [e28n07:864557] [13] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] [e28n07:864557] [14] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ progress+0x6c)[0x20003d60640c] [e28n07:864557] [15] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r equest_default_wait_all+0x144)[0x2000034c4b04] [e28n07:864557] [16] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W aitall+0x10c)[0x20000352790c] [e28n07:864557] [17] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] [e28n07:864558] [12] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (PAMI_Context_advancev+0x6b0)[0x20007f2da540] [e28n07:864558] [13] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] [e28n07:864558] [14] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ progress+0x6c)[0x20003d60640c] [e28n07:864558] [15] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r equest_default_wait_all+0x144)[0x2000034c4b04] [e28n07:864558] [16] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W aitall+0x10c)[0x20000352790c] [e28n07:864558] [17] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] [e28n07:864557] [18] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] [e28n07:864558] [18] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] [e28n07:864557] [19] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] [e28n07:864558] [19] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] [e28n07:864557] [20] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] [e28n07:864558] [20] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] [e28n07:864557] [21] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] [e28n07:864558] [21] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] [e28n07:864558] [22] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] [e28n07:864557] [22] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] [e28n07:864558] [23] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] [e28n07:864557] [23] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] [e28n07:864557] [24] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 _+0xc4)[0x200005f710d4] [e28n07:864557] [25] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 00005f7130c] [e28n07:864557] [26] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] [e28n07:864558] [24] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 _+0xc4)[0x200005f710d4] [e28n07:864558] [25] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 00005f7130c] [e28n07:864558] [26] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] [e28n07:864557] [27] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] [e28n07:864558] [27] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] [e28n07:864557] [28] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] [e28n07:864558] [28] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] [e28n07:864557] [29] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] [e28n07:864558] [29] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] [e28n07:864557] *** End of error message *** /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] [e28n07:864558] *** End of error message *** It seems to be pointing to https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/ so I wanted to check if you had seen this type of error before and if it could be related to how the code is compiled or run. Let me know if I can provide any additional information. Best, Sophie |
From: Pierre J. <pi...@jo...> - 2024-02-29 16:30:48
|
> On 29 Feb 2024, at 5:06 PM, Matthew Knepley <kn...@gm...> wrote: > > This Message Is From an External Sender > This message came from outside your organization. > On Thu, Feb 29, 2024 at 11:03 AM Blondel, Sophie via petsc-users <pet...@mc... <mailto:pet...@mc...>> wrote: >> This Message Is From an External Sender >> This message came from outside your organization. >> >> Thank you Junchao, >> >> Yes, I am using gpu-aware MPI. >> >> Is "-use_gpu_aware_mpi 0" a runtime option or a compile option? > > That is a configure option, so No, it’s a runtime option, you don’t need to reconfigure, just add it to your command line arguments. Thanks, Pierre > cd $PETSC_DIR > ./${PETSC_ARCH}/lib./petsc/conf/reconfigure-${PETSC_ARCH}.py -use-gpu_aware_mpi 0 > make all > > Thanks, > > Matt > >> Best, >> >> Sophie >> From: Junchao Zhang <jun...@gm... <mailto:jun...@gm...>> >> Sent: Thursday, February 29, 2024 10:50 >> To: Blondel, Sophie <sbl...@ut... <mailto:sbl...@ut...>> >> Cc: xol...@li... <mailto:xol...@li...> <xol...@li... <mailto:xol...@li...>>; pet...@mc... <mailto:pet...@mc...> <pet...@mc... <mailto:pet...@mc...>> >> Subject: Re: [petsc-users] PAMI error on Summit >> >> You don't often get email from jun...@gm... <mailto:jun...@gm...>. Learn why this is important <https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YnlFkp83$> >> Hi Sophie, >> PetscSFBcastEnd() was calling MPI_Waitall() to finish the communication in DMGlobalToLocal. >> I guess you used gpu-aware MPI. The error you saw might be due to it. You can try without it with a petsc option -use_gpu_aware_mpi 0 >> But we generally recommend gpu-aware mpi. You can try on other GPU machines to see if it is just an IBM Spectrum MPI problem. >> >> Thanks. >> --Junchao Zhang >> >> >> On Thu, Feb 29, 2024 at 9:17 AM Blondel, Sophie via petsc-users <pet...@mc... <mailto:pet...@mc...>> wrote: >> This Message Is From an External Sender >> This message came from outside your organization. >> >> Hi, >> >> I am using PETSc build with the Kokkos CUDA backend on Summit but when I run my code with multiple MPI tasks I get the following error: >> 0 TS dt 1e-12 time 0. >> errno 14 pid 864558 >> xolotl: /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: size_t PAMI::Dev >> ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. >> errno 14 pid 864557 >> xolotl: /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: size_t PAMI::Dev >> ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. >> [e28n07:864557] *** Process received signal *** >> [e28n07:864557] Signal: Aborted (6) >> [e28n07:864557] Signal code: (-6) >> [e28n07:864557] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] >> [e28n07:864557] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$>(gsignal+0xd8)[0x200005d796f8] >> [e28n07:864557] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$>(abort+0x164)[0x200005d53ff4] >> [e28n07:864557] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$>(+0x3d280)[0x200005d6d280] >> [e28n07:864557] [ 4] [e28n07:864558] *** Process received signal *** >> [e28n07:864558] Signal: Aborted (6) >> [e28n07:864558] Signal code: (-6) >> [e28n07:864558] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] >> [e28n07:864558] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$>(gsignal+0xd8)[0x200005d796f8] >> [e28n07:864558] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$>(abort+0x164)[0x200005d53ff4] >> [e28n07:864558] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$>(+0x3d280)[0x200005d6d280] >> [e28n07:864558] [ 4] /lib64/glibc-hwcaps/power9/libc-2.28.so <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$>(__assert_fail+0x64)[0x200005d6d324] >> [e28n07:864557] [ 5] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt >> omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] >> [e28n07:864557] [ 6] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] >> [e28n07:864557] [ 7] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] >> [e28n07:864557] [ 8] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (PAMI_Rget+0x18)[0x20007f2d94a8] >> [e28n07:864557] [ 9] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p >> ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] >> [e28n07:864557] [10] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p >> ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] >> [e28n07:864557] [11] /lib64/glibc-hwcaps/power9/libc-2.28.so <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$>(__assert_fail+0x64)[0x200005d6d324] >> [e28n07:864558] [ 5] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt >> omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] >> [e28n07:864558] [ 6] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] >> [e28n07:864558] [ 7] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] >> [e28n07:864558] [ 8] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (PAMI_Rget+0x18)[0x20007f2d94a8] >> [e28n07:864558] [ 9] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p >> ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] >> [e28n07:864558] [10] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p >> ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] >> [e28n07:864558] [11] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic >> 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] >> [e28n07:864557] [12] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (PAMI_Context_advancev+0x6b0)[0x20007f2da540] >> [e28n07:864557] [13] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p >> ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] >> [e28n07:864557] [14] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ >> progress+0x6c)[0x20003d60640c] >> [e28n07:864557] [15] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r >> equest_default_wait_all+0x144)[0x2000034c4b04] >> [e28n07:864557] [16] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W >> aitall+0x10c)[0x20000352790c] >> [e28n07:864557] [17] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic >> 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] >> [e28n07:864558] [12] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (PAMI_Context_advancev+0x6b0)[0x20007f2da540] >> [e28n07:864558] [13] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p >> ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] >> [e28n07:864558] [14] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ >> progress+0x6c)[0x20003d60640c] >> [e28n07:864558] [15] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r >> equest_default_wait_all+0x144)[0x2000034c4b04] >> [e28n07:864558] [16] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W >> aitall+0x10c)[0x20000352790c] >> [e28n07:864558] [17] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] >> [e28n07:864557] [18] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] >> [e28n07:864558] [18] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] >> [e28n07:864557] [19] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] >> [e28n07:864558] [19] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] >> [e28n07:864557] [20] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] >> [e28n07:864558] [20] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] >> [e28n07:864557] [21] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] >> [e28n07:864558] [21] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] >> [e28n07:864558] [22] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] >> [e28n07:864557] [22] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] >> [e28n07:864558] [23] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] >> [e28n07:864557] [23] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] >> [e28n07:864557] [24] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 >> _+0xc4)[0x200005f710d4] >> [e28n07:864557] [25] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 >> 00005f7130c] >> [e28n07:864557] [26] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] >> [e28n07:864558] [24] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 >> _+0xc4)[0x200005f710d4] >> [e28n07:864558] [25] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 >> 00005f7130c] >> [e28n07:864558] [26] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] >> [e28n07:864557] [27] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] >> [e28n07:864558] [27] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] >> [e28n07:864557] [28] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] >> [e28n07:864558] [28] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] >> [e28n07:864557] [29] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] >> [e28n07:864558] [29] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] >> [e28n07:864557] *** End of error message *** >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] >> [e28n07:864558] *** End of error message *** >> >> It seems to be pointing to https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/ <https://urldefense.us/v3/__https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/__;!!G_uCfscf7eWS!Y8h5WLnoArdfhK2_UDmISOEiqxAN9gBUzvWniKOoMwtA9cGjg8w9sYX6V8aIgfzL8Uhea5ppiRbuTGr1jZ_R2DOV$> so I wanted to check if you had seen this type of error before and if it could be related to how the code is compiled or run. Let me know if I can provide any additional information. >> >> Best, >> >> Sophie > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!alnN9lGj6QLPXpVJ1V1bDi40lqd3DTGEpANSS9orjCElucgd0fAjX5VcHJ7J8afDWcWiUr8Fg2QwHxAwswTs$> |
From: Blondel, S. <sbl...@ut...> - 2024-02-29 16:17:05
|
Thank you Junchao, Yes, I am using gpu-aware MPI. Is "-use_gpu_aware_mpi 0" a runtime option or a compile option? Best, Sophie ________________________________ From: Junchao Zhang <jun...@gm...> Sent: Thursday, February 29, 2024 10:50 To: Blondel, Sophie <sbl...@ut...> Cc: xol...@li... <xol...@li...>; pet...@mc... <pet...@mc...> Subject: Re: [petsc-users] PAMI error on Summit You don't often get email from jun...@gm.... Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Hi Sophie, PetscSFBcastEnd() was calling MPI_Waitall() to finish the communication in DMGlobalToLocal. I guess you used gpu-aware MPI. The error you saw might be due to it. You can try without it with a petsc option -use_gpu_aware_mpi 0 But we generally recommend gpu-aware mpi. You can try on other GPU machines to see if it is just an IBM Spectrum MPI problem. Thanks. --Junchao Zhang On Thu, Feb 29, 2024 at 9:17 AM Blondel, Sophie via petsc-users <pet...@mc...<mailto:pet...@mc...>> wrote: Hi, I am using PETSc build with the Kokkos CUDA backend on Summit but when I run my code with multiple MPI tasks I get the following error: 0 TS dt 1e-12 time 0. errno 14 pid 864558 xolotl: /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/. . /pami/components/devices/shmem/shaddr/CMAShaddr. h: 164: ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, I am using PETSc build with the Kokkos CUDA backend on Summit but when I run my code with multiple MPI tasks I get the following error: 0 TS dt 1e-12 time 0. errno 14 pid 864558 xolotl: /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: size_t PAMI::Dev ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. errno 14 pid 864557 xolotl: /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: size_t PAMI::Dev ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. [e28n07:864557] *** Process received signal *** [e28n07:864557] Signal: Aborted (6) [e28n07:864557] Signal code: (-6) [e28n07:864557] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] [e28n07:864557] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(gsignal+0xd8)[0x200005d796f8] [e28n07:864557] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(abort+0x164)[0x200005d53ff4] [e28n07:864557] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(+0x3d280)[0x200005d6d280] [e28n07:864557] [ 4] [e28n07:864558] *** Process received signal *** [e28n07:864558] Signal: Aborted (6) [e28n07:864558] Signal code: (-6) [e28n07:864558] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] [e28n07:864558] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(gsignal+0xd8)[0x200005d796f8] [e28n07:864558] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(abort+0x164)[0x200005d53ff4] [e28n07:864558] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(+0x3d280)[0x200005d6d280] [e28n07:864558] [ 4] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(__assert_fail+0x64)[0x200005d6d324] [e28n07:864557] [ 5] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] [e28n07:864557] [ 6] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] [e28n07:864557] [ 7] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] [e28n07:864557] [ 8] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (PAMI_Rget+0x18)[0x20007f2d94a8] [e28n07:864557] [ 9] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] [e28n07:864557] [10] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] [e28n07:864557] [11] /lib64/glibc-hwcaps/power9/libc-2.28.so<http://libc-2.28.so>(__assert_fail+0x64)[0x200005d6d324] [e28n07:864558] [ 5] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] [e28n07:864558] [ 6] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] [e28n07:864558] [ 7] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] [e28n07:864558] [ 8] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (PAMI_Rget+0x18)[0x20007f2d94a8] [e28n07:864558] [ 9] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] [e28n07:864558] [10] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] [e28n07:864558] [11] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] [e28n07:864557] [12] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (PAMI_Context_advancev+0x6b0)[0x20007f2da540] [e28n07:864557] [13] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] [e28n07:864557] [14] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ progress+0x6c)[0x20003d60640c] [e28n07:864557] [15] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r equest_default_wait_all+0x144)[0x2000034c4b04] [e28n07:864557] [16] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W aitall+0x10c)[0x20000352790c] [e28n07:864557] [17] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] [e28n07:864558] [12] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 (PAMI_Context_advancev+0x6b0)[0x20007f2da540] [e28n07:864558] [13] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] [e28n07:864558] [14] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ progress+0x6c)[0x20003d60640c] [e28n07:864558] [15] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r equest_default_wait_all+0x144)[0x2000034c4b04] [e28n07:864558] [16] /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W aitall+0x10c)[0x20000352790c] [e28n07:864558] [17] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] [e28n07:864557] [18] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] [e28n07:864558] [18] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] [e28n07:864557] [19] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] [e28n07:864558] [19] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] [e28n07:864557] [20] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] [e28n07:864558] [20] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] [e28n07:864557] [21] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] [e28n07:864558] [21] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] [e28n07:864558] [22] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] [e28n07:864557] [22] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] [e28n07:864558] [23] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] [e28n07:864557] [23] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] [e28n07:864557] [24] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 _+0xc4)[0x200005f710d4] [e28n07:864557] [25] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 00005f7130c] [e28n07:864557] [26] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] [e28n07:864558] [24] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 _+0xc4)[0x200005f710d4] [e28n07:864558] [25] /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 00005f7130c] [e28n07:864558] [26] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] [e28n07:864557] [27] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] [e28n07:864558] [27] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] [e28n07:864557] [28] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] [e28n07:864558] [28] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] [e28n07:864557] [29] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] [e28n07:864558] [29] /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] [e28n07:864557] *** End of error message *** /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] [e28n07:864558] *** End of error message *** It seems to be pointing to https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/<https://urldefense.us/v3/__https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/__;!!G_uCfscf7eWS!Y8h5WLnoArdfhK2_UDmISOEiqxAN9gBUzvWniKOoMwtA9cGjg8w9sYX6V8aIgfzL8Uhea5ppiRbuTGr1jZ_R2DOV$> so I wanted to check if you had seen this type of error before and if it could be related to how the code is compiled or run. Let me know if I can provide any additional information. Best, Sophie |
From: Junchao Z. <jun...@gm...> - 2024-02-29 16:14:00
|
Yes, it is a runtime option. No need to reconfigure petsc. Just add "-use_gpu_aware_mpi 0" to your test's command line. --Junchao Zhang On Thu, Feb 29, 2024 at 10:07 AM Matthew Knepley <kn...@gm...> wrote: > On Thu, Feb 29, 2024 at 11:03 AM Blondel, Sophie via petsc-users < > pet...@mc...> wrote: > >> Thank you Junchao, Yes, I am using gpu-aware MPI. Is "-use_gpu_aware_mpi >> 0" a runtime option or a compile option? Best, Sophie From: Junchao Zhang >> <junchao. zhang@ gmail. com> Sent: Thursday, February 29, 2024 10: 50 >> To: Blondel, >> ZjQcmQRYFpfptBannerStart >> This Message Is From an External Sender >> This message came from outside your organization. >> >> ZjQcmQRYFpfptBannerEnd >> Thank you Junchao, >> >> Yes, I am using gpu-aware MPI. >> >> Is "-use_gpu_aware_mpi 0" a runtime option or a compile option? >> > > That is a configure option, so > > cd $PETSC_DIR > ./${PETSC_ARCH}/lib./petsc/conf/reconfigure-${PETSC_ARCH}.py > -use-gpu_aware_mpi 0 > make all > > Thanks, > > Matt > > >> Best, >> >> Sophie >> ------------------------------ >> *From:* Junchao Zhang <jun...@gm...> >> *Sent:* Thursday, February 29, 2024 10:50 >> *To:* Blondel, Sophie <sbl...@ut...> >> *Cc:* xol...@li... < >> xol...@li...>; pet...@mc... < >> pet...@mc...> >> *Subject:* Re: [petsc-users] PAMI error on Summit >> >> You don't often get email from jun...@gm.... Learn why this >> is important >> <https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YnlFkp83$> >> Hi Sophie, >> PetscSFBcastEnd() was calling MPI_Waitall() to finish the communication >> in DMGlobalToLocal. >> I guess you used gpu-aware MPI. The error you saw might be due to it. >> You can try without it with a petsc option -use_gpu_aware_mpi 0 >> But we generally recommend gpu-aware mpi. You can try on other GPU >> machines to see if it is just an IBM Spectrum MPI problem. >> >> Thanks. >> --Junchao Zhang >> >> >> On Thu, Feb 29, 2024 at 9:17 AM Blondel, Sophie via petsc-users < >> pet...@mc...> wrote: >> >> Hi, I am using PETSc build with the Kokkos CUDA backend on Summit but >> when I run my code with multiple MPI tasks I get the following error: 0 TS >> dt 1e-12 time 0. errno 14 pid 864558 xolotl: >> /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/. >> . /pami/components/devices/shmem/shaddr/CMAShaddr. h: 164: >> ZjQcmQRYFpfptBannerStart >> This Message Is From an External Sender >> This message came from outside your organization. >> >> ZjQcmQRYFpfptBannerEnd >> Hi, >> >> I am using PETSc build with the Kokkos CUDA backend on Summit but when I >> run my code with multiple MPI tasks I get the following error: >> 0 TS dt 1e-12 time 0. >> errno 14 pid 864558 >> xolotl: >> /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: >> size_t PAMI::Dev >> ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, >> PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. >> errno 14 pid 864557 >> xolotl: >> /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: >> size_t PAMI::Dev >> ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, >> PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. >> [e28n07:864557] *** Process received signal *** >> [e28n07:864557] Signal: Aborted (6) >> [e28n07:864557] Signal code: (-6) >> [e28n07:864557] [ 0] >> linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] >> [e28n07:864557] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so >> <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$> >> (gsignal+0xd8)[0x200005d796f8] >> [e28n07:864557] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so >> <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$> >> (abort+0x164)[0x200005d53ff4] >> [e28n07:864557] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so >> <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$> >> (+0x3d280)[0x200005d6d280] >> [e28n07:864557] [ 4] [e28n07:864558] *** Process received signal *** >> [e28n07:864558] Signal: Aborted (6) >> [e28n07:864558] Signal code: (-6) >> [e28n07:864558] [ 0] >> linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] >> [e28n07:864558] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so >> <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$> >> (gsignal+0xd8)[0x200005d796f8] >> [e28n07:864558] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so >> <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$> >> (abort+0x164)[0x200005d53ff4] >> [e28n07:864558] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so >> <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$> >> (+0x3d280)[0x200005d6d280] >> [e28n07:864558] [ 4] /lib64/glibc-hwcaps/power9/libc-2.28.so >> <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$> >> (__assert_fail+0x64)[0x200005d6d324] >> [e28n07:864557] [ 5] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> >> (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt >> >> omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] >> [e28n07:864557] [ 6] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> >> (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] >> [e28n07:864557] [ 7] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] >> [e28n07:864557] [ 8] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (PAMI_Rget+0x18)[0x20007f2d94a8] >> [e28n07:864557] [ 9] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p >> ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] >> [e28n07:864557] [10] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p >> ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] >> [e28n07:864557] [11] /lib64/glibc-hwcaps/power9/libc-2.28.so >> <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$> >> (__assert_fail+0x64)[0x200005d6d324] >> [e28n07:864558] [ 5] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> >> (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt >> >> omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] >> [e28n07:864558] [ 6] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> >> (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] >> [e28n07:864558] [ 7] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] >> [e28n07:864558] [ 8] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (PAMI_Rget+0x18)[0x20007f2d94a8] >> [e28n07:864558] [ 9] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p >> ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] >> [e28n07:864558] [10] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p >> ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] >> [e28n07:864558] [11] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> >> (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic >> >> 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] >> [e28n07:864557] [12] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (PAMI_Context_advancev+0x6b0)[0x20007f2da540] >> [e28n07:864557] [13] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p >> ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] >> [e28n07:864557] [14] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ >> progress+0x6c)[0x20003d60640c] >> [e28n07:864557] [15] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r >> equest_default_wait_all+0x144)[0x2000034c4b04] >> [e28n07:864557] [16] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W >> aitall+0x10c)[0x20000352790c] >> [e28n07:864557] [17] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> >> (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic >> >> 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] >> [e28n07:864558] [12] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 >> (PAMI_Context_advancev+0x6b0)[0x20007f2da540] >> [e28n07:864558] [13] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p >> ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] >> [e28n07:864558] [14] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ >> progress+0x6c)[0x20003d60640c] >> [e28n07:864558] [15] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r >> equest_default_wait_all+0x144)[0x2000034c4b04] >> [e28n07:864558] [16] >> /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W >> aitall+0x10c)[0x20000352790c] >> [e28n07:864558] [17] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] >> [e28n07:864557] [18] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] >> [e28n07:864558] [18] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] >> [e28n07:864557] [19] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] >> [e28n07:864558] [19] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] >> [e28n07:864557] [20] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] >> [e28n07:864558] [20] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] >> [e28n07:864557] [21] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] >> [e28n07:864558] [21] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] >> [e28n07:864558] [22] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] >> [e28n07:864557] [22] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] >> [e28n07:864558] [23] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] >> [e28n07:864557] [23] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] >> [e28n07:864557] [24] >> /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 >> _+0xc4)[0x200005f710d4] >> [e28n07:864557] [25] >> /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 >> 00005f7130c] >> [e28n07:864557] [26] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] >> [e28n07:864558] [24] >> /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 >> _+0xc4)[0x200005f710d4] >> [e28n07:864558] [25] >> /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 >> 00005f7130c] >> [e28n07:864558] [26] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] >> [e28n07:864557] [27] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] >> [e28n07:864558] [27] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] >> [e28n07:864557] [28] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] >> [e28n07:864558] [28] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] >> [e28n07:864557] [29] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] >> [e28n07:864558] [29] >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] >> [e28n07:864557] *** End of error message *** >> >> /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] >> [e28n07:864558] *** End of error message *** >> >> It seems to be pointing to >> https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/ >> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/__;!!G_uCfscf7eWS!Y8h5WLnoArdfhK2_UDmISOEiqxAN9gBUzvWniKOoMwtA9cGjg8w9sYX6V8aIgfzL8Uhea5ppiRbuTGr1jZ_R2DOV$> >> so I wanted to check if you had seen this type of error before and if it >> could be related to how the code is compiled or run. Let me know if I can >> provide any additional information. >> >> Best, >> >> Sophie >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> > |
From: Matthew K. <kn...@gm...> - 2024-02-29 16:07:22
|
On Thu, Feb 29, 2024 at 11:03 AM Blondel, Sophie via petsc-users < pet...@mc...> wrote: > Thank you Junchao, Yes, I am using gpu-aware MPI. Is "-use_gpu_aware_mpi > 0" a runtime option or a compile option? Best, Sophie From: Junchao Zhang > <junchao. zhang@ gmail. com> Sent: Thursday, February 29, 2024 10: 50 To: > Blondel, > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Thank you Junchao, > > Yes, I am using gpu-aware MPI. > > Is "-use_gpu_aware_mpi 0" a runtime option or a compile option? > That is a configure option, so cd $PETSC_DIR ./${PETSC_ARCH}/lib./petsc/conf/reconfigure-${PETSC_ARCH}.py -use-gpu_aware_mpi 0 make all Thanks, Matt > Best, > > Sophie > ------------------------------ > *From:* Junchao Zhang <jun...@gm...> > *Sent:* Thursday, February 29, 2024 10:50 > *To:* Blondel, Sophie <sbl...@ut...> > *Cc:* xol...@li... < > xol...@li...>; pet...@mc... < > pet...@mc...> > *Subject:* Re: [petsc-users] PAMI error on Summit > > You don't often get email from jun...@gm.... Learn why this is > important > <https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YnlFkp83$> > Hi Sophie, > PetscSFBcastEnd() was calling MPI_Waitall() to finish the communication > in DMGlobalToLocal. > I guess you used gpu-aware MPI. The error you saw might be due to it. > You can try without it with a petsc option -use_gpu_aware_mpi 0 > But we generally recommend gpu-aware mpi. You can try on other GPU > machines to see if it is just an IBM Spectrum MPI problem. > > Thanks. > --Junchao Zhang > > > On Thu, Feb 29, 2024 at 9:17 AM Blondel, Sophie via petsc-users < > pet...@mc...> wrote: > > Hi, I am using PETSc build with the Kokkos CUDA backend on Summit but when > I run my code with multiple MPI tasks I get the following error: 0 TS dt > 1e-12 time 0. errno 14 pid 864558 xolotl: > /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/. > . /pami/components/devices/shmem/shaddr/CMAShaddr. h: 164: > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Hi, > > I am using PETSc build with the Kokkos CUDA backend on Summit but when I > run my code with multiple MPI tasks I get the following error: > 0 TS dt 1e-12 time 0. > errno 14 pid 864558 > xolotl: > /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: > size_t PAMI::Dev > ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, > PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. > errno 14 pid 864557 > xolotl: > /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: > size_t PAMI::Dev > ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, > PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. > [e28n07:864557] *** Process received signal *** > [e28n07:864557] Signal: Aborted (6) > [e28n07:864557] Signal code: (-6) > [e28n07:864557] [ 0] > linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] > [e28n07:864557] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so > <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$> > (gsignal+0xd8)[0x200005d796f8] > [e28n07:864557] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so > <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$> > (abort+0x164)[0x200005d53ff4] > [e28n07:864557] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so > <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$> > (+0x3d280)[0x200005d6d280] > [e28n07:864557] [ 4] [e28n07:864558] *** Process received signal *** > [e28n07:864558] Signal: Aborted (6) > [e28n07:864558] Signal code: (-6) > [e28n07:864558] [ 0] > linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] > [e28n07:864558] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so > <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$> > (gsignal+0xd8)[0x200005d796f8] > [e28n07:864558] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so > <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$> > (abort+0x164)[0x200005d53ff4] > [e28n07:864558] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so > <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$> > (+0x3d280)[0x200005d6d280] > [e28n07:864558] [ 4] /lib64/glibc-hwcaps/power9/libc-2.28.so > <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$> > (__assert_fail+0x64)[0x200005d6d324] > [e28n07:864557] [ 5] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt > > omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] > [e28n07:864557] [ 6] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] > [e28n07:864557] [ 7] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] > [e28n07:864557] [ 8] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (PAMI_Rget+0x18)[0x20007f2d94a8] > [e28n07:864557] [ 9] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] > [e28n07:864557] [10] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] > [e28n07:864557] [11] /lib64/glibc-hwcaps/power9/libc-2.28.so > <https://urldefense.us/v3/__http://libc-2.28.so__;!!G_uCfscf7eWS!facODjKBT4NdUpQRG3bRO5aufX56SW84P_ciaPKeSHjRxArfCGndt8hUfkRcvfjlBIbRbEra5PwQNlA6YlO8YWjh$> > (__assert_fail+0x64)[0x200005d6d324] > [e28n07:864558] [ 5] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt > > omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] > [e28n07:864558] [ 6] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] > [e28n07:864558] [ 7] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] > [e28n07:864558] [ 8] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (PAMI_Rget+0x18)[0x20007f2d94a8] > [e28n07:864558] [ 9] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] > [e28n07:864558] [10] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] > [e28n07:864558] [11] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic > > 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] > [e28n07:864557] [12] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (PAMI_Context_advancev+0x6b0)[0x20007f2da540] > [e28n07:864557] [13] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] > [e28n07:864557] [14] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ > progress+0x6c)[0x20003d60640c] > [e28n07:864557] [15] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r > equest_default_wait_all+0x144)[0x2000034c4b04] > [e28n07:864557] [16] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W > aitall+0x10c)[0x20000352790c] > [e28n07:864557] [17] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic > > 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] > [e28n07:864558] [12] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (PAMI_Context_advancev+0x6b0)[0x20007f2da540] > [e28n07:864558] [13] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] > [e28n07:864558] [14] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ > progress+0x6c)[0x20003d60640c] > [e28n07:864558] [15] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r > equest_default_wait_all+0x144)[0x2000034c4b04] > [e28n07:864558] [16] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W > aitall+0x10c)[0x20000352790c] > [e28n07:864558] [17] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] > [e28n07:864557] [18] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] > [e28n07:864558] [18] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] > [e28n07:864557] [19] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] > [e28n07:864558] [19] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] > [e28n07:864557] [20] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] > [e28n07:864558] [20] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] > [e28n07:864557] [21] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] > [e28n07:864558] [21] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] > [e28n07:864558] [22] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] > [e28n07:864557] [22] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] > [e28n07:864558] [23] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] > [e28n07:864557] [23] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] > [e28n07:864557] [24] > /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 > _+0xc4)[0x200005f710d4] > [e28n07:864557] [25] > /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 > 00005f7130c] > [e28n07:864557] [26] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] > [e28n07:864558] [24] > /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 > _+0xc4)[0x200005f710d4] > [e28n07:864558] [25] > /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 > 00005f7130c] > [e28n07:864558] [26] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] > [e28n07:864557] [27] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] > [e28n07:864558] [27] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] > [e28n07:864557] [28] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] > [e28n07:864558] [28] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] > [e28n07:864557] [29] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] > [e28n07:864558] [29] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] > [e28n07:864557] *** End of error message *** > > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] > [e28n07:864558] *** End of error message *** > > It seems to be pointing to > https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/ > <https://urldefense.us/v3/__https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/__;!!G_uCfscf7eWS!Y8h5WLnoArdfhK2_UDmISOEiqxAN9gBUzvWniKOoMwtA9cGjg8w9sYX6V8aIgfzL8Uhea5ppiRbuTGr1jZ_R2DOV$> > so I wanted to check if you had seen this type of error before and if it > could be related to how the code is compiled or run. Let me know if I can > provide any additional information. > > Best, > > Sophie > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/> |
From: Junchao Z. <jun...@gm...> - 2024-02-29 15:50:54
|
Hi Sophie, PetscSFBcastEnd() was calling MPI_Waitall() to finish the communication in DMGlobalToLocal. I guess you used gpu-aware MPI. The error you saw might be due to it. You can try without it with a petsc option -use_gpu_aware_mpi 0 But we generally recommend gpu-aware mpi. You can try on other GPU machines to see if it is just an IBM Spectrum MPI problem. Thanks. --Junchao Zhang On Thu, Feb 29, 2024 at 9:17 AM Blondel, Sophie via petsc-users < pet...@mc...> wrote: > Hi, I am using PETSc build with the Kokkos CUDA backend on Summit but when > I run my code with multiple MPI tasks I get the following error: 0 TS dt > 1e-12 time 0. errno 14 pid 864558 xolotl: > /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/. > . /pami/components/devices/shmem/shaddr/CMAShaddr. h: 164: > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Hi, > > I am using PETSc build with the Kokkos CUDA backend on Summit but when I > run my code with multiple MPI tasks I get the following error: > 0 TS dt 1e-12 time 0. > errno 14 pid 864558 > xolotl: > /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: > size_t PAMI::Dev > ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, > PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. > errno 14 pid 864557 > xolotl: > /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: > size_t PAMI::Dev > ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, > PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed. > [e28n07:864557] *** Process received signal *** > [e28n07:864557] Signal: Aborted (6) > [e28n07:864557] Signal code: (-6) > [e28n07:864557] [ 0] > linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] > [e28n07:864557] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so > (gsignal+0xd8)[0x200005d796f8] > [e28n07:864557] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so > (abort+0x164)[0x200005d53ff4] > [e28n07:864557] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so > (+0x3d280)[0x200005d6d280] > [e28n07:864557] [ 4] [e28n07:864558] *** Process received signal *** > [e28n07:864558] Signal: Aborted (6) > [e28n07:864558] Signal code: (-6) > [e28n07:864558] [ 0] > linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8] > [e28n07:864558] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so > (gsignal+0xd8)[0x200005d796f8] > [e28n07:864558] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so > (abort+0x164)[0x200005d53ff4] > [e28n07:864558] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so > (+0x3d280)[0x200005d6d280] > [e28n07:864558] [ 4] /lib64/glibc-hwcaps/power9/libc-2.28.so > (__assert_fail+0x64)[0x200005d6d324] > [e28n07:864557] [ 5] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt > > omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] > [e28n07:864557] [ 6] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] > [e28n07:864557] [ 7] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] > [e28n07:864557] [ 8] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (PAMI_Rget+0x18)[0x20007f2d94a8] > [e28n07:864557] [ 9] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] > [e28n07:864557] [10] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] > [e28n07:864557] [11] /lib64/glibc-hwcaps/power9/libc-2.28.so > (__assert_fail+0x64)[0x200005d6d324] > [e28n07:864558] [ 5] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol3Get7GetRdmaINS_6Device5Shmem8DmaModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAt > > omicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEELb0EEESL_E6simpleEP18pami_rget_simple_t+0x1d8)[0x20007f3971d8] > [e28n07:864558] [ 6] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol3Get13CompositeRGetINS1_4RGetES3_E6simpleEP18pami_rget_simple_t+0x40)[0x20007f2ecc10] > [e28n07:864558] [ 7] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (_ZN4PAMI7Context9rget_implEP18pami_rget_simple_t+0x28c)[0x20007f31a78c] > [e28n07:864558] [ 8] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (PAMI_Rget+0x18)[0x20007f2d94a8] > [e28n07:864558] [ 9] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(process_rndv_msg+0x46c)[0x2000a80159ac] > [e28n07:864558] [10] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(pml_pami_recv_rndv_cb+0x2bc)[0x2000a801670c] > [e28n07:864558] [11] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic > > 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] > [e28n07:864557] [12] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (PAMI_Context_advancev+0x6b0)[0x20007f2da540] > [e28n07:864557] [13] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] > [e28n07:864557] [14] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ > progress+0x6c)[0x20003d60640c] > [e28n07:864557] [15] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r > equest_default_wait_all+0x144)[0x2000034c4b04] > [e28n07:864557] [16] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W > aitall+0x10c)[0x20000352790c] > [e28n07:864557] [17] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > > (_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic > > 12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE5EE15dispatch_packedEPvSP_mSP_SP_+0x4c)[0x20007f2e30ac] > [e28n07:864558] [12] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/pami_port/libpami.so.3 > (PAMI_Context_advancev+0x6b0)[0x20007f2da540] > [e28n07:864558] [13] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/spectrum_mpi/mca_pml_p > ami.so(mca_pml_pami_progress+0x34)[0x2000a80073e4] > [e28n07:864558] [14] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libopen-pal.so.3(opal_ > progress+0x6c)[0x20003d60640c] > [e28n07:864558] [15] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(ompi_r > equest_default_wait_all+0x144)[0x2000034c4b04] > [e28n07:864558] [16] > /sw/summit/spack-envs/summit-plus/opt/gcc-12.1.0/spectrum-mpi-10.4.0.6-20230210-db5xakaaqowbhp3nqwebpxrdbwtm4knu/container/../lib/libmpi_ibm.so.3(PMPI_W > aitall+0x10c)[0x20000352790c] > [e28n07:864558] [17] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] > [e28n07:864557] [18] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3ca7b0)[0x2000004ea7b0] > [e28n07:864558] [18] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] > [e28n07:864557] [19] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3c5e68)[0x2000004e5e68] > [e28n07:864558] [19] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] > [e28n07:864557] [20] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(PetscSFBcastEnd+0x74)[0x2000004c9214] > [e28n07:864558] [20] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] > [e28n07:864557] [21] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x3b4cb0)[0x2000004d4cb0] > [e28n07:864558] [21] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] > [e28n07:864558] [22] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(VecScatterEnd+0x178)[0x2000004dd038] > [e28n07:864557] [22] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] > [e28n07:864558] [23] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x1112be0)[0x200001232be0] > [e28n07:864557] [23] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] > [e28n07:864557] [24] > /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 > _+0xc4)[0x200005f710d4] > [e28n07:864557] [25] > /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 > 00005f7130c] > [e28n07:864557] [26] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(DMGlobalToLocalEnd+0x470)[0x200000e9b0f0] > [e28n07:864558] [24] > /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11PetscSolver11rhsFunctionEP5_p_TSdP6_p_VecS5 > _+0xc4)[0x200005f710d4] > [e28n07:864558] [25] > /gpfs/alpine2/mat267/proj-shared/code/xolotl-stable-cuda/xolotl/solver/libxolotlSolver.so(_ZN6xolotl6solver11RHSFunctionEP5_p_TSdP6_p_VecS4_Pv+0x2c)[0x2 > 00005f7130c] > [e28n07:864558] [26] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] > [e28n07:864557] [27] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeRHSFunction+0x1bc)[0x2000017621dc] > [e28n07:864558] [27] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] > [e28n07:864557] [28] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSComputeIFunction+0x418)[0x200001763ad8] > [e28n07:864558] [28] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] > [e28n07:864557] [29] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(+0x16f2ef0)[0x200001812ef0] > [e28n07:864558] [29] > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] > [e28n07:864557] *** End of error message *** > > /gpfs/alpine2/mat267/proj-shared/dependencies/petsc-kokkos-cuda/lib/libpetsc.so.3.020(TSStep+0x228)[0x200001768088] > [e28n07:864558] *** End of error message *** > > It seems to be pointing to > https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/ > <https://urldefense.us/v3/__https://petsc.org/release/manualpages/PetscSF/PetscSFBcastEnd/__;!!G_uCfscf7eWS!Y8h5WLnoArdfhK2_UDmISOEiqxAN9gBUzvWniKOoMwtA9cGjg8w9sYX6V8aIgfzL8Uhea5ppiRbuTGr1jZ_R2DOV$> > so I wanted to check if you had seen this type of error before and if it > could be related to how the code is compiled or run. Let me know if I can > provide any additional information. > > Best, > > Sophie > |
From: Jed B. <je...@je...> - 2023-12-18 19:47:25
|
Great, thanks for letting us know. It'll merge to release shortly and thus be in petsc >= 3.20.3. "Fackler, Philip" <fac...@or...> writes: > Jed, > > That seems to have worked (ridiculously well). It's now 55MB, and it's happening in the call to MatSetPreallocationCOO. > > Thank you, > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory > ________________________________ > From: Jed Brown <je...@je...> > Sent: Thursday, December 14, 2023 16:27 > To: Fackler, Philip <fac...@or...>; pet...@mc... <pet...@mc...>; xol...@li... <xol...@li...> > Subject: [EXTERNAL] Re: [petsc-users] Call to DMSetMatrixPreallocateSkip not changing allocation behavior > > I had a one-character typo in the diff above. This MR to release should work now. > > https://urldefense.us/v2/url?u=https-3A__gitlab.com_petsc_petsc_-2D_merge-5Frequests_7120&d=DwIBAg&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=v9sHqomCGBRWotign4NcwYwOpszOJehUGs_EO3eGn4SSZqxnfK7Iv15-X8nO1lii&s=h_jIP-6WcIjR6LssfGrV6Z2DojlN_w7Me4-a4rBE074&e= > > Jed Brown <je...@je...> writes: > >> 17 GB for a 1D DMDA, wow. :-) >> >> Could you try applying this diff to make it work for DMDA (it's currently handled by DMPlex)? >> >> diff --git i/src/dm/impls/da/fdda.c w/src/dm/impls/da/fdda.c >> index cad4d926504..bd2a3bda635 100644 >> --- i/src/dm/impls/da/fdda.c >> +++ w/src/dm/impls/da/fdda.c >> @@ -675,19 +675,21 @@ PetscErrorCode DMCreateMatrix_DA(DM da, Mat *J) >> specialized setting routines depend only on the particular preallocation >> details of the matrix, not the type itself. >> */ >> - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIAIJSetPreallocation_C", &aij)); >> - if (!aij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqAIJSetPreallocation_C", &aij)); >> - if (!aij) { >> - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIBAIJSetPreallocation_C", &baij)); >> - if (!baij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqBAIJSetPreallocation_C", &baij)); >> - if (!baij) { >> - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISBAIJSetPreallocation_C", &sbaij)); >> - if (!sbaij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSBAIJSetPreallocation_C", &sbaij)); >> - if (!sbaij) { >> - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISELLSetPreallocation_C", &sell)); >> - if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSELLSetPreallocation_C", &sell)); >> + if (!dm->prealloc_skip) { // Flag is likely set when user intends to use MatSetPreallocationCOO() >> + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIAIJSetPreallocation_C", &aij)); >> + if (!aij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqAIJSetPreallocation_C", &aij)); >> + if (!aij) { >> + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIBAIJSetPreallocation_C", &baij)); >> + if (!baij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqBAIJSetPreallocation_C", &baij)); >> + if (!baij) { >> + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISBAIJSetPreallocation_C", &sbaij)); >> + if (!sbaij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSBAIJSetPreallocation_C", &sbaij)); >> + if (!sbaij) { >> + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISELLSetPreallocation_C", &sell)); >> + if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSELLSetPreallocation_C", &sell)); >> + } >> + if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatISSetPreallocation_C", &is)); >> } >> - if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatISSetPreallocation_C", &is)); >> } >> } >> if (aij) { >> >> >> "Fackler, Philip via petsc-users" <pet...@mc...> writes: >> >>> I'm using the following sequence of functions related to the Jacobian matrix: >>> >>> DMDACreate1d(..., &da); >>> DMSetFromOptions(da); >>> DMSetUp(da); >>> DMSetMatType(da, MATAIJKOKKOS); >>> DMSetMatrixPreallocateSkip(da, PETSC_TRUE); >>> Mat J; >>> DMCreateMatrix(da, &J); >>> MatSetPreallocationCOO(J, ...); >>> >>> I recently added the call to DMSetMatrixPreallocateSkip, hoping the allocation would be delayed to MatSetPreallocationCOO, and that it would require less memory. The documentation<https://urldefense.us/v2/url?u=https-3A__petsc.org_release_manualpages_DM_DMSetMatrixPreallocateSkip_&d=DwIBAg&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=v9sHqomCGBRWotign4NcwYwOpszOJehUGs_EO3eGn4SSZqxnfK7Iv15-X8nO1lii&s=IMLBs0ydxDPvuXeD6jmsq1BN_8oieHVSyG6VA9c0DyM&e= > says that the data structures will not be preallocated. The following data from heaptrack shows that the allocation is still happening in the call to DMCreateMatrix. >>> >>> [cid:bda9ef12-a46f-47b2-9b9b-a4b2808b6b13] >>> >>> Can someone help me understand this? >>> >>> Thanks, >>> >>> Philip Fackler >>> Research Software Engineer, Application Engineering Group >>> Advanced Computing Systems Research Section >>> Computer Science and Mathematics Division >>> Oak Ridge National Laboratory |
From: Fackler, P. <fac...@or...> - 2023-12-18 18:54:43
|
Jed, That seems to have worked (ridiculously well). It's now 55MB, and it's happening in the call to MatSetPreallocationCOO. Thank you, Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory ________________________________ From: Jed Brown <je...@je...> Sent: Thursday, December 14, 2023 16:27 To: Fackler, Philip <fac...@or...>; pet...@mc... <pet...@mc...>; xol...@li... <xol...@li...> Subject: [EXTERNAL] Re: [petsc-users] Call to DMSetMatrixPreallocateSkip not changing allocation behavior I had a one-character typo in the diff above. This MR to release should work now. https://urldefense.us/v2/url?u=https-3A__gitlab.com_petsc_petsc_-2D_merge-5Frequests_7120&d=DwIBAg&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=v9sHqomCGBRWotign4NcwYwOpszOJehUGs_EO3eGn4SSZqxnfK7Iv15-X8nO1lii&s=h_jIP-6WcIjR6LssfGrV6Z2DojlN_w7Me4-a4rBE074&e= Jed Brown <je...@je...> writes: > 17 GB for a 1D DMDA, wow. :-) > > Could you try applying this diff to make it work for DMDA (it's currently handled by DMPlex)? > > diff --git i/src/dm/impls/da/fdda.c w/src/dm/impls/da/fdda.c > index cad4d926504..bd2a3bda635 100644 > --- i/src/dm/impls/da/fdda.c > +++ w/src/dm/impls/da/fdda.c > @@ -675,19 +675,21 @@ PetscErrorCode DMCreateMatrix_DA(DM da, Mat *J) > specialized setting routines depend only on the particular preallocation > details of the matrix, not the type itself. > */ > - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIAIJSetPreallocation_C", &aij)); > - if (!aij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqAIJSetPreallocation_C", &aij)); > - if (!aij) { > - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIBAIJSetPreallocation_C", &baij)); > - if (!baij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqBAIJSetPreallocation_C", &baij)); > - if (!baij) { > - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISBAIJSetPreallocation_C", &sbaij)); > - if (!sbaij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSBAIJSetPreallocation_C", &sbaij)); > - if (!sbaij) { > - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISELLSetPreallocation_C", &sell)); > - if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSELLSetPreallocation_C", &sell)); > + if (!dm->prealloc_skip) { // Flag is likely set when user intends to use MatSetPreallocationCOO() > + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIAIJSetPreallocation_C", &aij)); > + if (!aij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqAIJSetPreallocation_C", &aij)); > + if (!aij) { > + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIBAIJSetPreallocation_C", &baij)); > + if (!baij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqBAIJSetPreallocation_C", &baij)); > + if (!baij) { > + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISBAIJSetPreallocation_C", &sbaij)); > + if (!sbaij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSBAIJSetPreallocation_C", &sbaij)); > + if (!sbaij) { > + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISELLSetPreallocation_C", &sell)); > + if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSELLSetPreallocation_C", &sell)); > + } > + if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatISSetPreallocation_C", &is)); > } > - if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatISSetPreallocation_C", &is)); > } > } > if (aij) { > > > "Fackler, Philip via petsc-users" <pet...@mc...> writes: > >> I'm using the following sequence of functions related to the Jacobian matrix: >> >> DMDACreate1d(..., &da); >> DMSetFromOptions(da); >> DMSetUp(da); >> DMSetMatType(da, MATAIJKOKKOS); >> DMSetMatrixPreallocateSkip(da, PETSC_TRUE); >> Mat J; >> DMCreateMatrix(da, &J); >> MatSetPreallocationCOO(J, ...); >> >> I recently added the call to DMSetMatrixPreallocateSkip, hoping the allocation would be delayed to MatSetPreallocationCOO, and that it would require less memory. The documentation<https://urldefense.us/v2/url?u=https-3A__petsc.org_release_manualpages_DM_DMSetMatrixPreallocateSkip_&d=DwIBAg&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=v9sHqomCGBRWotign4NcwYwOpszOJehUGs_EO3eGn4SSZqxnfK7Iv15-X8nO1lii&s=IMLBs0ydxDPvuXeD6jmsq1BN_8oieHVSyG6VA9c0DyM&e= > says that the data structures will not be preallocated. The following data from heaptrack shows that the allocation is still happening in the call to DMCreateMatrix. >> >> [cid:bda9ef12-a46f-47b2-9b9b-a4b2808b6b13] >> >> Can someone help me understand this? >> >> Thanks, >> >> Philip Fackler >> Research Software Engineer, Application Engineering Group >> Advanced Computing Systems Research Section >> Computer Science and Mathematics Division >> Oak Ridge National Laboratory |
From: Jed B. <je...@je...> - 2023-12-14 21:28:07
|
I had a one-character typo in the diff above. This MR to release should work now. https://gitlab.com/petsc/petsc/-/merge_requests/7120 Jed Brown <je...@je...> writes: > 17 GB for a 1D DMDA, wow. :-) > > Could you try applying this diff to make it work for DMDA (it's currently handled by DMPlex)? > > diff --git i/src/dm/impls/da/fdda.c w/src/dm/impls/da/fdda.c > index cad4d926504..bd2a3bda635 100644 > --- i/src/dm/impls/da/fdda.c > +++ w/src/dm/impls/da/fdda.c > @@ -675,19 +675,21 @@ PetscErrorCode DMCreateMatrix_DA(DM da, Mat *J) > specialized setting routines depend only on the particular preallocation > details of the matrix, not the type itself. > */ > - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIAIJSetPreallocation_C", &aij)); > - if (!aij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqAIJSetPreallocation_C", &aij)); > - if (!aij) { > - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIBAIJSetPreallocation_C", &baij)); > - if (!baij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqBAIJSetPreallocation_C", &baij)); > - if (!baij) { > - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISBAIJSetPreallocation_C", &sbaij)); > - if (!sbaij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSBAIJSetPreallocation_C", &sbaij)); > - if (!sbaij) { > - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISELLSetPreallocation_C", &sell)); > - if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSELLSetPreallocation_C", &sell)); > + if (!dm->prealloc_skip) { // Flag is likely set when user intends to use MatSetPreallocationCOO() > + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIAIJSetPreallocation_C", &aij)); > + if (!aij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqAIJSetPreallocation_C", &aij)); > + if (!aij) { > + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIBAIJSetPreallocation_C", &baij)); > + if (!baij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqBAIJSetPreallocation_C", &baij)); > + if (!baij) { > + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISBAIJSetPreallocation_C", &sbaij)); > + if (!sbaij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSBAIJSetPreallocation_C", &sbaij)); > + if (!sbaij) { > + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISELLSetPreallocation_C", &sell)); > + if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSELLSetPreallocation_C", &sell)); > + } > + if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatISSetPreallocation_C", &is)); > } > - if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatISSetPreallocation_C", &is)); > } > } > if (aij) { > > > "Fackler, Philip via petsc-users" <pet...@mc...> writes: > >> I'm using the following sequence of functions related to the Jacobian matrix: >> >> DMDACreate1d(..., &da); >> DMSetFromOptions(da); >> DMSetUp(da); >> DMSetMatType(da, MATAIJKOKKOS); >> DMSetMatrixPreallocateSkip(da, PETSC_TRUE); >> Mat J; >> DMCreateMatrix(da, &J); >> MatSetPreallocationCOO(J, ...); >> >> I recently added the call to DMSetMatrixPreallocateSkip, hoping the allocation would be delayed to MatSetPreallocationCOO, and that it would require less memory. The documentation<https://petsc.org/release/manualpages/DM/DMSetMatrixPreallocateSkip/> says that the data structures will not be preallocated. The following data from heaptrack shows that the allocation is still happening in the call to DMCreateMatrix. >> >> [cid:bda9ef12-a46f-47b2-9b9b-a4b2808b6b13] >> >> Can someone help me understand this? >> >> Thanks, >> >> Philip Fackler >> Research Software Engineer, Application Engineering Group >> Advanced Computing Systems Research Section >> Computer Science and Mathematics Division >> Oak Ridge National Laboratory |
From: Matthew K. <kn...@gm...> - 2023-12-14 21:19:27
|
On Thu, Dec 14, 2023 at 2:06 PM Fackler, Philip via petsc-users < pet...@mc...> wrote: > I'm using the following sequence of functions related to the Jacobian > matrix: > > DMDACreate1d(..., &da); > DMSetFromOptions(da); > DMSetUp(da); > DMSetMatType(da, MATAIJKOKKOS); > DMSetMatrixPreallocateSkip(da, PETSC_TRUE); > Mat J; > DMCreateMatrix(da, &J); > MatSetPreallocationCOO(J, ...); > > I recently added the call to DMSetMatrixPreallocateSkip, hoping the > allocation would be delayed to MatSetPreallocationCOO, and that it would > require less memory. The documentation > <https://petsc.org/release/manualpages/DM/DMSetMatrixPreallocateSkip/> says > that the data structures will not be preallocated. > You are completely correct. DMDA is just ignoring this flag. We will fix it. Thanks for catching this. Matt > The following data from heaptrack shows that the allocation is still > happening in the call to DMCreateMatrix. > > > Can someone help me understand this? > > Thanks, > > > *Philip Fackler * > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > *Oak Ridge National Laboratory* > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/> |
From: Jed B. <je...@je...> - 2023-12-14 20:49:40
|
17 GB for a 1D DMDA, wow. :-) Could you try applying this diff to make it work for DMDA (it's currently handled by DMPlex)? diff --git i/src/dm/impls/da/fdda.c w/src/dm/impls/da/fdda.c index cad4d926504..bd2a3bda635 100644 --- i/src/dm/impls/da/fdda.c +++ w/src/dm/impls/da/fdda.c @@ -675,19 +675,21 @@ PetscErrorCode DMCreateMatrix_DA(DM da, Mat *J) specialized setting routines depend only on the particular preallocation details of the matrix, not the type itself. */ - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIAIJSetPreallocation_C", &aij)); - if (!aij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqAIJSetPreallocation_C", &aij)); - if (!aij) { - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIBAIJSetPreallocation_C", &baij)); - if (!baij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqBAIJSetPreallocation_C", &baij)); - if (!baij) { - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISBAIJSetPreallocation_C", &sbaij)); - if (!sbaij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSBAIJSetPreallocation_C", &sbaij)); - if (!sbaij) { - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISELLSetPreallocation_C", &sell)); - if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSELLSetPreallocation_C", &sell)); + if (!dm->prealloc_skip) { // Flag is likely set when user intends to use MatSetPreallocationCOO() + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIAIJSetPreallocation_C", &aij)); + if (!aij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqAIJSetPreallocation_C", &aij)); + if (!aij) { + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIBAIJSetPreallocation_C", &baij)); + if (!baij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqBAIJSetPreallocation_C", &baij)); + if (!baij) { + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISBAIJSetPreallocation_C", &sbaij)); + if (!sbaij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSBAIJSetPreallocation_C", &sbaij)); + if (!sbaij) { + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISELLSetPreallocation_C", &sell)); + if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSELLSetPreallocation_C", &sell)); + } + if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatISSetPreallocation_C", &is)); } - if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatISSetPreallocation_C", &is)); } } if (aij) { "Fackler, Philip via petsc-users" <pet...@mc...> writes: > I'm using the following sequence of functions related to the Jacobian matrix: > > DMDACreate1d(..., &da); > DMSetFromOptions(da); > DMSetUp(da); > DMSetMatType(da, MATAIJKOKKOS); > DMSetMatrixPreallocateSkip(da, PETSC_TRUE); > Mat J; > DMCreateMatrix(da, &J); > MatSetPreallocationCOO(J, ...); > > I recently added the call to DMSetMatrixPreallocateSkip, hoping the allocation would be delayed to MatSetPreallocationCOO, and that it would require less memory. The documentation<https://petsc.org/release/manualpages/DM/DMSetMatrixPreallocateSkip/> says that the data structures will not be preallocated. The following data from heaptrack shows that the allocation is still happening in the call to DMCreateMatrix. > > [cid:bda9ef12-a46f-47b2-9b9b-a4b2808b6b13] > > Can someone help me understand this? > > Thanks, > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory |
From: Fackler, P. <fac...@or...> - 2023-12-14 19:06:10
|
I'm using the following sequence of functions related to the Jacobian matrix: DMDACreate1d(..., &da); DMSetFromOptions(da); DMSetUp(da); DMSetMatType(da, MATAIJKOKKOS); DMSetMatrixPreallocateSkip(da, PETSC_TRUE); Mat J; DMCreateMatrix(da, &J); MatSetPreallocationCOO(J, ...); I recently added the call to DMSetMatrixPreallocateSkip, hoping the allocation would be delayed to MatSetPreallocationCOO, and that it would require less memory. The documentation<https://petsc.org/release/manualpages/DM/DMSetMatrixPreallocateSkip/> says that the data structures will not be preallocated. The following data from heaptrack shows that the allocation is still happening in the call to DMCreateMatrix. [cid:bda9ef12-a46f-47b2-9b9b-a4b2808b6b13] Can someone help me understand this? Thanks, Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory |
From: Jed B. <je...@je...> - 2023-11-30 16:52:04
|
Looks like both splits are of the same size and same number of nonzeros. GAMG is probably a good choice there, and likely doesn't need an inner Krylov (i.e., you could use -fieldsplit_1_ksp_type preonly). "Blondel, Sophie" <sbl...@ut...> writes: > Attached is the output with -ksp_view, it is not exactly what Philip has been running because this was done on my laptop instead of Ascent. > > Looking at the options we've been using, "redundant" is in charge of the diffusive part here (finite difference cell centered), in 1D simulation. When we do 2D or 3D simulations we switch it to "-fieldsplit_1_pc_type gamg -fieldsplit_1_ksp_type gmres -ksp_type fgmres". > > Cheers, > > Sophie > ________________________________ > From: Jed Brown <je...@je...> > Sent: Thursday, November 30, 2023 00:59 > To: Blondel, Sophie <sbl...@ut...>; Fackler, Philip <fac...@or...>; Junchao Zhang <jun...@gm...> > Cc: pet...@mc... <pet...@mc...>; xol...@li... <xol...@li...> > Subject: Re: [Xolotl-psi-development] [petsc-users] [EXTERNAL] Re: Unexpected performance losses switching to COO interface > > [You don't often get email from je...@je.... Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] > > "Blondel, Sophie" <sbl...@ut...> writes: > >> Hi Jed, >> >> I'm not sure I'm going to reply to your question correctly because I don't really understand how the split is done. Is it related to on diagonal and off diagonal? If so, the off-diagonal part is usually pretty small (less than 20 DOFs) and related to diffusion, the diagonal part involves thousands of DOFs for the reaction term. > > From the run-time option, it'll be a default (additive) split and we're interested in the two diagonal blocks. One currently has a cheap solver that would only be efficient with a well-conditioned positive definite matrix and the other is using a direct solver ('redundant'). If you were to run with -ksp_view and share the output, it would be informative. > > Either way, I'd like to understand what physics are beind the equation currently being solved with 'redundant'. If it's diffusive, then algebraic multigrid would be a good place to start. > >> Let us know what we can do to answer this question more accurately. >> >> Cheers, >> >> Sophie >> ________________________________ >> From: Jed Brown <je...@je...> >> Sent: Tuesday, November 28, 2023 19:07 >> To: Fackler, Philip <fac...@or...>; Junchao Zhang <jun...@gm...> >> Cc: pet...@mc... <pet...@mc...>; xol...@li... <xol...@li...> >> Subject: Re: [Xolotl-psi-development] [petsc-users] [EXTERNAL] Re: Unexpected performance losses switching to COO interface >> >> [Some people who received this message don't often get email from je...@je.... Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] >> >> "Fackler, Philip via petsc-users" <pet...@mc...> writes: >> >>> That makes sense. Here are the arguments that I think are relevant: >>> >>> -fieldsplit_1_pc_type redundant -fieldsplit_0_pc_type sor -pc_type fieldsplit -pc_fieldsplit_detect_coupling >> >> What sort of physics are in splits 0 and 1? >> >> SOR is not a good GPU algorithm, so we'll want to change that one way or another. Are the splits of similar size or very different? >> >>> What would you suggest to make this better? >>> >>> Also, note that the cases marked "serial" are running on CPU only, that is, using only the SERIAL backend for kokkos. >>> >>> Philip Fackler >>> Research Software Engineer, Application Engineering Group >>> Advanced Computing Systems Research Section >>> Computer Science and Mathematics Division >>> Oak Ridge National Laboratory >>> ________________________________ >>> From: Junchao Zhang <jun...@gm...> >>> Sent: Tuesday, November 28, 2023 15:51 >>> To: Fackler, Philip <fac...@or...> >>> Cc: pet...@mc... <pet...@mc...>; xol...@li... <xol...@li...> >>> Subject: Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface >>> >>> Hi, Philip, >>> I opened hpcdb-PSI_9-serial and it seems you used PCLU. Since Kokkos does not have a GPU LU implementation, we do it on CPU via MatLUFactorNumeric_SeqAIJ(). Perhaps you can try other PC types? >>> >>> [Screenshot 2023-11-28 at 2.43.03 PM.png] >>> --Junchao Zhang >>> >>> >>> On Wed, Nov 22, 2023 at 10:43 AM Fackler, Philip <fac...@or...<mailto:fac...@or...>> wrote: >>> I definitely dropped the ball on this. I'm sorry for that. I have new profiling data using the latest (as of yesterday) of petsc/main. I've put them in a single google drive folder linked here: >>> >>> https://drive.google.com/drive/folders/14ScvyfxOzc4OzXs9HZVeQDO-g6FdIVAI?usp=drive_link<https://urldefense.us/v2/url?u=https-3A__drive.google.com_drive_folders_14ScvyfxOzc4OzXs9HZVeQDO-2Dg6FdIVAI-3Fusp-3Ddrive-5Flink&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=Qn5D9xuzFcMdyuL0I2ruKmU6yeez0NrOx69oUjRaAXTeKD6etHt4USuZgnbqF4v6&s=_Lqg9v8aa4KXUdud3zqSp55FiYkZ12Pp5ZY54_9OvJI&e=> >>> >>> Have a happy holiday weekend! >>> >>> Thanks, >>> >>> Philip Fackler >>> Research Software Engineer, Application Engineering Group >>> Advanced Computing Systems Research Section >>> Computer Science and Mathematics Division >>> Oak Ridge National Laboratory >>> ________________________________ >>> From: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> >>> Sent: Monday, October 16, 2023 15:24 >>> To: Fackler, Philip <fac...@or...<mailto:fac...@or...>> >>> Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>> >>> Subject: Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface >>> >>> Hi, Philip, >>> That branch was merged to petsc/main today. Let me know once you have new profiling results. >>> >>> Thanks. >>> --Junchao Zhang >>> >>> >>> On Mon, Oct 16, 2023 at 9:33 AM Fackler, Philip <fac...@or...<mailto:fac...@or...>> wrote: >>> Junchao, >>> >>> I've attached updated timing plots (red and blue are swapped from before; yellow is the new one). There is an improvement for the NE_3 case only with CUDA. Serial stays the same, and the PSI cases stay the same. In the PSI cases, MatShift doesn't show up (I assume because we're using different preconditioner arguments). So, there must be some other primary culprit. I'll try to get updated profiling data to you soon. >>> >>> Thanks, >>> >>> Philip Fackler >>> Research Software Engineer, Application Engineering Group >>> Advanced Computing Systems Research Section >>> Computer Science and Mathematics Division >>> Oak Ridge National Laboratory >>> ________________________________ >>> From: Fackler, Philip via Xolotl-psi-development <xol...@li...<mailto:xol...@li...>> >>> Sent: Wednesday, October 11, 2023 11:31 >>> To: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> >>> Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>> >>> Subject: Re: [Xolotl-psi-development] [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface >>> >>> I'm on it. >>> >>> Philip Fackler >>> Research Software Engineer, Application Engineering Group >>> Advanced Computing Systems Research Section >>> Computer Science and Mathematics Division >>> Oak Ridge National Laboratory >>> ________________________________ >>> From: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> >>> Sent: Wednesday, October 11, 2023 10:14 >>> To: Fackler, Philip <fac...@or...<mailto:fac...@or...>> >>> Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>>; Blondel, Sophie <sbl...@ut...<mailto:sbl...@ut...>> >>> Subject: Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface >>> >>> Hi, Philip, >>> Could you try this branch jczhang/2023-10-05/feature-support-matshift-aijkokkos ? >>> >>> Thanks. >>> --Junchao Zhang >>> >>> >>> On Thu, Oct 5, 2023 at 4:52 PM Fackler, Philip <fac...@or...<mailto:fac...@or...>> wrote: >>> Aha! That makes sense. Thank you. >>> >>> Philip Fackler >>> Research Software Engineer, Application Engineering Group >>> Advanced Computing Systems Research Section >>> Computer Science and Mathematics Division >>> Oak Ridge National Laboratory >>> ________________________________ >>> From: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> >>> Sent: Thursday, October 5, 2023 17:29 >>> To: Fackler, Philip <fac...@or...<mailto:fac...@or...>> >>> Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>>; Blondel, Sophie <sbl...@ut...<mailto:sbl...@ut...>> >>> Subject: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface >>> >>> Wait a moment, it seems it was because we do not have a GPU implementation of MatShift... >>> Let me see how to add it. >>> --Junchao Zhang >>> >>> >>> On Thu, Oct 5, 2023 at 10:58 AM Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> wrote: >>> Hi, Philip, >>> I looked at the hpcdb-NE_3-cuda file. It seems you used MatSetValues() instead of the COO interface? MatSetValues() needs to copy the data from device to host and thus is expensive. >>> Do you have profiling results with COO enabled? >>> >>> [Screenshot 2023-10-05 at 10.55.29 AM.png] >>> >>> >>> --Junchao Zhang >>> >>> >>> On Mon, Oct 2, 2023 at 9:52 AM Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> wrote: >>> Hi, Philip, >>> I will look into the tarballs and get back to you. >>> Thanks. >>> --Junchao Zhang >>> >>> >>> On Mon, Oct 2, 2023 at 9:41 AM Fackler, Philip via petsc-users <pet...@mc...<mailto:pet...@mc...>> wrote: >>> We finally have xolotl ported to use the new COO interface and the aijkokkos implementation for Mat (and kokkos for Vec). Comparing this port to our previous version (using MatSetValuesStencil and the default Mat and Vec implementations), we expected to see an improvement in performance for both the "serial" and "cuda" builds (here I'm referring to the kokkos configuration). >>> >>> Attached are two plots that show timings for three different cases. All of these were run on Ascent (the Summit-like training system) with 6 MPI tasks (on a single node). The CUDA cases were given one GPU per task (and used CUDA-aware MPI). The labels on the blue bars indicate speedup. In all cases we used "-fieldsplit_0_pc_type jacobi" to keep the comparison as consistent as possible. >>> >>> The performance of RHSJacobian (where the bulk of computation happens in xolotl) behaved basically as expected (better than expected in the serial build). NE_3 case in CUDA was the only one that performed worse, but not surprisingly, since its workload for the GPUs is much smaller. We've still got more optimization to do on this. >>> >>> The real surprise was how much worse the overall solve times were. This seems to be due simply to switching to the kokkos-based implementation. I'm wondering if there are any changes we can make in configuration or runtime arguments to help with PETSc's performance here. Any help looking into this would be appreciated. >>> >>> The tarballs linked here<https://urldefense.us/v2/url?u=https-3A__drive.google.com_file_d_19X-5FL3SVkGBM9YUzXnRR-5FkVWFG0JFwqZ3_view-3Fusp-3Ddrive-5Flink&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=GTpC2k9hIdMhUg_aJkeAqd-1CP5M8bwJMJjTriVE1k-j36ZnEHerQkZOzszxWoG2&s=GW0ImGWhWr4rR5AoSULCnaP1CN1QWxTSeMDhdOuhTEA&e=> and here<https://urldefense.us/v2/url?u=https-3A__drive.google.com_file_d_15yDBN7-2DYlO1g6RJNPYNImzr611i1Ffhv_view-3Fusp-3Ddrive-5Flink&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=GTpC2k9hIdMhUg_aJkeAqd-1CP5M8bwJMJjTriVE1k-j36ZnEHerQkZOzszxWoG2&s=tO-BnNY2myA-pIsRnBjQNoaOSjn-B3-lWGiQp7XXJwk&e=> are profiling databases which, once extracted, can be viewed with hpcviewer. I don't know how helpful that will be, but hopefully it can give you some direction. >>> >>> Thanks for your help, >>> >>> Philip Fackler >>> Research Software Engineer, Application Engineering Group >>> Advanced Computing Systems Research Section >>> Computer Science and Mathematics Division >>> Oak Ridge National Laboratory >> >> >> _______________________________________________ >> Xolotl-psi-development mailing list >> Xol...@li... >> https://lists.sourceforge.net/lists/listinfo/xolotl-psi-development > Starting Xolotl (2.4.0-83-e84edee7) > Thu Nov 30 09:48:13 2023 > NetworkHandler: Loaded network of 312 DOF with: Helium Vacancy Interstitial > MaterialHandler: The selected material is: W111 with the following processes: advec attenuation diff modifiedTM movingSurface reaction ; a custom fit flux handler is used reading: tridyn_benchmark_PSI_9.dat > TemperatureHandler: Using the time profile defined in: temp_benchmark_PSI_9.dat > SolverHandler: 1D simulation with surface BC: free surface and bulk BC: free surface, initial concentration for Id: 0 of: 1e-18 nm-3, grid (nm): 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.7999999999999999 0.8999999999999999 0.9999999999999999 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2 2.100000000000001 2.200000000000001 2.300000000000001 2.400000000000001 2.500000000000001 2.750000000000001 3.000000000000001 3.250000000000001 3.500000000000001 3.750000000000001 4.000000000000002 4.250000000000002 4.500000000000002 4.750000000000002 5.000000000000002 > 0 TS dt 1e-10 time 0. > > Time: 0 > Helium content = 0 > Vacancy content = 4.750000000000002e-18 > Interstitial content = 0 > Fluence = 0 > > KSP Object: 4 MPI processes > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 4 MPI processes > type: fieldsplit > > FieldSplit with MULTIPLICATIVE composition: total splits = 2, blocksize = 313 > > Solver info for each split is in the following KSP objects: > > Split number 0 Defined by IS > > KSP Object: (fieldsplit_0_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_0_) 4 MPI processes > > type: jacobi > > > type DIAGONAL > > linear system matrix = precond matrix: > > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268, bs=313 > > > > total: nonzeros=399318, allocated nonzeros=0 > > > > total number of mallocs used during MatSetValues calls=0 > > Split number 1 Defined by IS > > KSP Object: (fieldsplit_1_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_1_) 4 MPI processes > > type: redundant > > > First (color=0) of 4 PCs follows > KSP Object: (fieldsplit_1_redundant_) 1 MPI process > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_1_redundant_) 1 MPI process > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 3.17245 > Factored matrix follows: > Mat Object: (fieldsplit_1_redundant_) 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > package used to perform factorization: kokkos > total: nonzeros=1266817, allocated nonzeros=1266817 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > total: nonzeros=399318, allocated nonzeros=399318 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > > linear system matrix = precond matrix: > > > Mat Object: (fieldsplit_1_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268 > > > > total: nonzeros=399318, allocated nonzeros=399318 > > > > total number of mallocs used during MatSetValues calls=0 > > > > > not using I-node (on process 0) routines > linear system matrix = precond matrix: > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > type: mpiaijkokkos > > > rows=11268, cols=11268, bs=313 > > > total: nonzeros=399318, allocated nonzeros=0 > > > total number of mallocs used during MatSetValues calls=0 > KSP Object: 4 MPI processes > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 4 MPI processes > type: fieldsplit > > FieldSplit with MULTIPLICATIVE composition: total splits = 2, blocksize = 313 > > Solver info for each split is in the following KSP objects: > > Split number 0 Defined by IS > > KSP Object: (fieldsplit_0_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_0_) 4 MPI processes > > type: jacobi > > > type DIAGONAL > > linear system matrix = precond matrix: > > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268, bs=313 > > > > total: nonzeros=399318, allocated nonzeros=0 > > > > total number of mallocs used during MatSetValues calls=0 > > Split number 1 Defined by IS > > KSP Object: (fieldsplit_1_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_1_) 4 MPI processes > > type: redundant > > > First (color=0) of 4 PCs follows > KSP Object: (fieldsplit_1_redundant_) 1 MPI process > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_1_redundant_) 1 MPI process > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 3.17245 > Factored matrix follows: > Mat Object: (fieldsplit_1_redundant_) 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > package used to perform factorization: kokkos > total: nonzeros=1266817, allocated nonzeros=1266817 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > total: nonzeros=399318, allocated nonzeros=399318 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > > linear system matrix = precond matrix: > > > Mat Object: (fieldsplit_1_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268 > > > > total: nonzeros=399318, allocated nonzeros=399318 > > > > total number of mallocs used during MatSetValues calls=0 > > > > > not using I-node (on process 0) routines > linear system matrix = precond matrix: > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > type: mpiaijkokkos > > > rows=11268, cols=11268, bs=313 > > > total: nonzeros=399318, allocated nonzeros=0 > > > total number of mallocs used during MatSetValues calls=0 > KSP Object: 4 MPI processes > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 4 MPI processes > type: fieldsplit > > FieldSplit with MULTIPLICATIVE composition: total splits = 2, blocksize = 313 > > Solver info for each split is in the following KSP objects: > > Split number 0 Defined by IS > > KSP Object: (fieldsplit_0_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_0_) 4 MPI processes > > type: jacobi > > > type DIAGONAL > > linear system matrix = precond matrix: > > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268, bs=313 > > > > total: nonzeros=399318, allocated nonzeros=0 > > > > total number of mallocs used during MatSetValues calls=0 > > Split number 1 Defined by IS > > KSP Object: (fieldsplit_1_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_1_) 4 MPI processes > > type: redundant > > > First (color=0) of 4 PCs follows > KSP Object: (fieldsplit_1_redundant_) 1 MPI process > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_1_redundant_) 1 MPI process > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 3.17245 > Factored matrix follows: > Mat Object: (fieldsplit_1_redundant_) 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > package used to perform factorization: kokkos > total: nonzeros=1266817, allocated nonzeros=1266817 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > total: nonzeros=399318, allocated nonzeros=399318 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > > linear system matrix = precond matrix: > > > Mat Object: (fieldsplit_1_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268 > > > > total: nonzeros=399318, allocated nonzeros=399318 > > > > total number of mallocs used during MatSetValues calls=0 > > > > > not using I-node (on process 0) routines > linear system matrix = precond matrix: > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > type: mpiaijkokkos > > > rows=11268, cols=11268, bs=313 > > > total: nonzeros=399318, allocated nonzeros=0 > > > total number of mallocs used during MatSetValues calls=0 > 1 TS dt 1e-09 time 1e-10 > > Time: 1e-10 > Helium content = 2.424952199503062e-06 > Vacancy content = 6.71808169858663e-15 > Interstitial content = 3.133080177694656e-09 > Fluence = 5.4e-06 > > KSP Object: 4 MPI processes > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 4 MPI processes > type: fieldsplit > > FieldSplit with MULTIPLICATIVE composition: total splits = 2, blocksize = 313 > > Solver info for each split is in the following KSP objects: > > Split number 0 Defined by IS > > KSP Object: (fieldsplit_0_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_0_) 4 MPI processes > > type: jacobi > > > type DIAGONAL > > linear system matrix = precond matrix: > > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268, bs=313 > > > > total: nonzeros=399318, allocated nonzeros=0 > > > > total number of mallocs used during MatSetValues calls=0 > > Split number 1 Defined by IS > > KSP Object: (fieldsplit_1_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_1_) 4 MPI processes > > type: redundant > > > First (color=0) of 4 PCs follows > KSP Object: (fieldsplit_1_redundant_) 1 MPI process > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_1_redundant_) 1 MPI process > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 3.17245 > Factored matrix follows: > Mat Object: (fieldsplit_1_redundant_) 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > package used to perform factorization: kokkos > total: nonzeros=1266817, allocated nonzeros=1266817 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > total: nonzeros=399318, allocated nonzeros=399318 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > > linear system matrix = precond matrix: > > > Mat Object: (fieldsplit_1_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268 > > > > total: nonzeros=399318, allocated nonzeros=399318 > > > > total number of mallocs used during MatSetValues calls=0 > > > > > not using I-node (on process 0) routines > linear system matrix = precond matrix: > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > type: mpiaijkokkos > > > rows=11268, cols=11268, bs=313 > > > total: nonzeros=399318, allocated nonzeros=0 > > > total number of mallocs used during MatSetValues calls=0 > KSP Object: 4 MPI processes > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 4 MPI processes > type: fieldsplit > > FieldSplit with MULTIPLICATIVE composition: total splits = 2, blocksize = 313 > > Solver info for each split is in the following KSP objects: > > Split number 0 Defined by IS > > KSP Object: (fieldsplit_0_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_0_) 4 MPI processes > > type: jacobi > > > type DIAGONAL > > linear system matrix = precond matrix: > > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268, bs=313 > > > > total: nonzeros=399318, allocated nonzeros=0 > > > > total number of mallocs used during MatSetValues calls=0 > > Split number 1 Defined by IS > > KSP Object: (fieldsplit_1_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_1_) 4 MPI processes > > type: redundant > > > First (color=0) of 4 PCs follows > KSP Object: (fieldsplit_1_redundant_) 1 MPI process > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_1_redundant_) 1 MPI process > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 3.17245 > Factored matrix follows: > Mat Object: (fieldsplit_1_redundant_) 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > package used to perform factorization: kokkos > total: nonzeros=1266817, allocated nonzeros=1266817 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > total: nonzeros=399318, allocated nonzeros=399318 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > > linear system matrix = precond matrix: > > > Mat Object: (fieldsplit_1_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268 > > > > total: nonzeros=399318, allocated nonzeros=399318 > > > > total number of mallocs used during MatSetValues calls=0 > > > > > not using I-node (on process 0) routines > linear system matrix = precond matrix: > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > type: mpiaijkokkos > > > rows=11268, cols=11268, bs=313 > > > total: nonzeros=399318, allocated nonzeros=0 > > > total number of mallocs used during MatSetValues calls=0 > KSP Object: 4 MPI processes > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 4 MPI processes > type: fieldsplit > > FieldSplit with MULTIPLICATIVE composition: total splits = 2, blocksize = 313 > > Solver info for each split is in the following KSP objects: > > Split number 0 Defined by IS > > KSP Object: (fieldsplit_0_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_0_) 4 MPI processes > > type: jacobi > > > type DIAGONAL > > linear system matrix = precond matrix: > > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268, bs=313 > > > > total: nonzeros=399318, allocated nonzeros=0 > > > > total number of mallocs used during MatSetValues calls=0 > > Split number 1 Defined by IS > > KSP Object: (fieldsplit_1_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_1_) 4 MPI processes > > type: redundant > > > First (color=0) of 4 PCs follows > KSP Object: (fieldsplit_1_redundant_) 1 MPI process > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_1_redundant_) 1 MPI process > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 3.17245 > Factored matrix follows: > Mat Object: (fieldsplit_1_redundant_) 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > package used to perform factorization: kokkos > total: nonzeros=1266817, allocated nonzeros=1266817 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > total: nonzeros=399318, allocated nonzeros=399318 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > > linear system matrix = precond matrix: > > > Mat Object: (fieldsplit_1_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268 > > > > total: nonzeros=399318, allocated nonzeros=399318 > > > > total number of mallocs used during MatSetValues calls=0 > > > > > not using I-node (on process 0) routines > linear system matrix = precond matrix: > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > type: mpiaijkokkos > > > rows=11268, cols=11268, bs=313 > > > total: nonzeros=399318, allocated nonzeros=0 > > > total number of mallocs used during MatSetValues calls=0 > KSP Object: 4 MPI processes > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 4 MPI processes > type: fieldsplit > > FieldSplit with MULTIPLICATIVE composition: total splits = 2, blocksize = 313 > > Solver info for each split is in the following KSP objects: > > Split number 0 Defined by IS > > KSP Object: (fieldsplit_0_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_0_) 4 MPI processes > > type: jacobi > > > type DIAGONAL > > linear system matrix = precond matrix: > > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268, bs=313 > > > > total: nonzeros=399318, allocated nonzeros=0 > > > > total number of mallocs used during MatSetValues calls=0 > > Split number 1 Defined by IS > > KSP Object: (fieldsplit_1_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_1_) 4 MPI processes > > type: redundant > > > First (color=0) of 4 PCs follows > KSP Object: (fieldsplit_1_redundant_) 1 MPI process > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_1_redundant_) 1 MPI process > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 3.17245 > Factored matrix follows: > Mat Object: (fieldsplit_1_redundant_) 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > package used to perform factorization: kokkos > total: nonzeros=1266817, allocated nonzeros=1266817 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > total: nonzeros=399318, allocated nonzeros=399318 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > > linear system matrix = precond matrix: > > > Mat Object: (fieldsplit_1_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268 > > > > total: nonzeros=399318, allocated nonzeros=399318 > > > > total number of mallocs used during MatSetValues calls=0 > > > > > not using I-node (on process 0) routines > linear system matrix = precond matrix: > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > type: mpiaijkokkos > > > rows=11268, cols=11268, bs=313 > > > total: nonzeros=399318, allocated nonzeros=0 > > > total number of mallocs used during MatSetValues calls=0 > 2 TS dt 1e-08 time 1.1e-09 > > Time: 1.1e-09 > Helium content = 2.296916600451812e-05 > Vacancy content = 2.31356670859338e-12 > Interstitial content = 3.19440449167048e-09 > Fluence = 5.940000000000001e-05 > > KSP Object: 4 MPI processes > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 4 MPI processes > type: fieldsplit > > FieldSplit with MULTIPLICATIVE composition: total splits = 2, blocksize = 313 > > Solver info for each split is in the following KSP objects: > > Split number 0 Defined by IS > > KSP Object: (fieldsplit_0_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_0_) 4 MPI processes > > type: jacobi > > > type DIAGONAL > > linear system matrix = precond matrix: > > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268, bs=313 > > > > total: nonzeros=399318, allocated nonzeros=0 > > > > total number of mallocs used during MatSetValues calls=0 > > Split number 1 Defined by IS > > KSP Object: (fieldsplit_1_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_1_) 4 MPI processes > > type: redundant > > > First (color=0) of 4 PCs follows > KSP Object: (fieldsplit_1_redundant_) 1 MPI process > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_1_redundant_) 1 MPI process > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 3.17245 > Factored matrix follows: > Mat Object: (fieldsplit_1_redundant_) 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > package used to perform factorization: kokkos > total: nonzeros=1266817, allocated nonzeros=1266817 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > total: nonzeros=399318, allocated nonzeros=399318 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > > linear system matrix = precond matrix: > > > Mat Object: (fieldsplit_1_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268 > > > > total: nonzeros=399318, allocated nonzeros=399318 > > > > total number of mallocs used during MatSetValues calls=0 > > > > > not using I-node (on process 0) routines > linear system matrix = precond matrix: > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > type: mpiaijkokkos > > > rows=11268, cols=11268, bs=313 > > > total: nonzeros=399318, allocated nonzeros=0 > > > total number of mallocs used during MatSetValues calls=0 > KSP Object: 4 MPI processes > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 4 MPI processes > type: fieldsplit > > FieldSplit with MULTIPLICATIVE composition: total splits = 2, blocksize = 313 > > Solver info for each split is in the following KSP objects: > > Split number 0 Defined by IS > > KSP Object: (fieldsplit_0_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_0_) 4 MPI processes > > type: jacobi > > > type DIAGONAL > > linear system matrix = precond matrix: > > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268, bs=313 > > > > total: nonzeros=399318, allocated nonzeros=0 > > > > total number of mallocs used during MatSetValues calls=0 > > Split number 1 Defined by IS > > KSP Object: (fieldsplit_1_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_1_) 4 MPI processes > > type: redundant > > > First (color=0) of 4 PCs follows > KSP Object: (fieldsplit_1_redundant_) 1 MPI process > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_1_redundant_) 1 MPI process > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 3.17245 > Factored matrix follows: > Mat Object: (fieldsplit_1_redundant_) 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > package used to perform factorization: kokkos > total: nonzeros=1266817, allocated nonzeros=1266817 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > total: nonzeros=399318, allocated nonzeros=399318 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > > linear system matrix = precond matrix: > > > Mat Object: (fieldsplit_1_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268 > > > > total: nonzeros=399318, allocated nonzeros=399318 > > > > total number of mallocs used during MatSetValues calls=0 > > > > > not using I-node (on process 0) routines > linear system matrix = precond matrix: > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > type: mpiaijkokkos > > > rows=11268, cols=11268, bs=313 > > > total: nonzeros=399318, allocated nonzeros=0 > > > total number of mallocs used during MatSetValues calls=0 > KSP Object: 4 MPI processes > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 4 MPI processes > type: fieldsplit > > FieldSplit with MULTIPLICATIVE composition: total splits = 2, blocksize = 313 > > Solver info for each split is in the following KSP objects: > > Split number 0 Defined by IS > > KSP Object: (fieldsplit_0_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_0_) 4 MPI processes > > type: jacobi > > > type DIAGONAL > > linear system matrix = precond matrix: > > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268, bs=313 > > > > total: nonzeros=399318, allocated nonzeros=0 > > > > total number of mallocs used during MatSetValues calls=0 > > Split number 1 Defined by IS > > KSP Object: (fieldsplit_1_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_1_) 4 MPI processes > > type: redundant > > > First (color=0) of 4 PCs follows > KSP Object: (fieldsplit_1_redundant_) 1 MPI process > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_1_redundant_) 1 MPI process > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 3.17245 > Factored matrix follows: > Mat Object: (fieldsplit_1_redundant_) 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > package used to perform factorization: kokkos > total: nonzeros=1266817, allocated nonzeros=1266817 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > total: nonzeros=399318, allocated nonzeros=399318 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > > linear system matrix = precond matrix: > > > Mat Object: (fieldsplit_1_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268 > > > > total: nonzeros=399318, allocated nonzeros=399318 > > > > total number of mallocs used during MatSetValues calls=0 > > > > > not using I-node (on process 0) routines > linear system matrix = precond matrix: > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > type: mpiaijkokkos > > > rows=11268, cols=11268, bs=313 > > > total: nonzeros=399318, allocated nonzeros=0 > > > total number of mallocs used during MatSetValues calls=0 > KSP Object: 4 MPI processes > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 4 MPI processes > type: fieldsplit > > FieldSplit with MULTIPLICATIVE composition: total splits = 2, blocksize = 313 > > Solver info for each split is in the following KSP objects: > > Split number 0 Defined by IS > > KSP Object: (fieldsplit_0_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_0_) 4 MPI processes > > type: jacobi > > > type DIAGONAL > > linear system matrix = precond matrix: > > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268, bs=313 > > > > total: nonzeros=399318, allocated nonzeros=0 > > > > total number of mallocs used during MatSetValues calls=0 > > Split number 1 Defined by IS > > KSP Object: (fieldsplit_1_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_1_) 4 MPI processes > > type: redundant > > > First (color=0) of 4 PCs follows > KSP Object: (fieldsplit_1_redundant_) 1 MPI process > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_1_redundant_) 1 MPI process > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 3.17245 > Factored matrix follows: > Mat Object: (fieldsplit_1_redundant_) 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > package used to perform factorization: kokkos > total: nonzeros=1266817, allocated nonzeros=1266817 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > total: nonzeros=399318, allocated nonzeros=399318 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > > linear system matrix = precond matrix: > > > Mat Object: (fieldsplit_1_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268 > > > > total: nonzeros=399318, allocated nonzeros=399318 > > > > total number of mallocs used during MatSetValues calls=0 > > > > > not using I-node (on process 0) routines > linear system matrix = precond matrix: > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > type: mpiaijkokkos > > > rows=11268, cols=11268, bs=313 > > > total: nonzeros=399318, allocated nonzeros=0 > > > total number of mallocs used during MatSetValues calls=0 > KSP Object: 4 MPI processes > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 4 MPI processes > type: fieldsplit > > FieldSplit with MULTIPLICATIVE composition: total splits = 2, blocksize = 313 > > Solver info for each split is in the following KSP objects: > > Split number 0 Defined by IS > > KSP Object: (fieldsplit_0_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_0_) 4 MPI processes > > type: jacobi > > > type DIAGONAL > > linear system matrix = precond matrix: > > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268, bs=313 > > > > total: nonzeros=399318, allocated nonzeros=0 > > > > total number of mallocs used during MatSetValues calls=0 > > Split number 1 Defined by IS > > KSP Object: (fieldsplit_1_) 4 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (fieldsplit_1_) 4 MPI processes > > type: redundant > > > First (color=0) of 4 PCs follows > KSP Object: (fieldsplit_1_redundant_) 1 MPI process > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_1_redundant_) 1 MPI process > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 3.17245 > Factored matrix follows: > Mat Object: (fieldsplit_1_redundant_) 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > package used to perform factorization: kokkos > total: nonzeros=1266817, allocated nonzeros=1266817 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI process > type: seqaijkokkos > rows=11268, cols=11268 > total: nonzeros=399318, allocated nonzeros=399318 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > > linear system matrix = precond matrix: > > > Mat Object: (fieldsplit_1_) 4 MPI processes > > > type: mpiaijkokkos > > > > rows=11268, cols=11268 > > > > total: nonzeros=399318, allocated nonzeros=399318 > > > > total number of mallocs used during MatSetValues calls=0 > > > > > not using I-node (on process 0) routines > linear system matrix = precond matrix: > > Mat Object: Mat_0x556cd9a44ae0_0 (fieldsplit_0_) 4 MPI processes > > type: mpiaijkokkos > > > rows=11268, cols=11268, bs=313 > > > total: nonzeros=399318, allocated nonzeros=0 > > > total number of mallocs used during MatSetValues calls=0 > KSP Object: 4 MPI processes > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test ... [truncated message content] |
From: Blondel, S. <sbl...@ut...> - 2023-11-30 15:15:41
|
Attached is the output with -ksp_view, it is not exactly what Philip has been running because this was done on my laptop instead of Ascent. Looking at the options we've been using, "redundant" is in charge of the diffusive part here (finite difference cell centered), in 1D simulation. When we do 2D or 3D simulations we switch it to "-fieldsplit_1_pc_type gamg -fieldsplit_1_ksp_type gmres -ksp_type fgmres". Cheers, Sophie ________________________________ From: Jed Brown <je...@je...> Sent: Thursday, November 30, 2023 00:59 To: Blondel, Sophie <sbl...@ut...>; Fackler, Philip <fac...@or...>; Junchao Zhang <jun...@gm...> Cc: pet...@mc... <pet...@mc...>; xol...@li... <xol...@li...> Subject: Re: [Xolotl-psi-development] [petsc-users] [EXTERNAL] Re: Unexpected performance losses switching to COO interface [You don't often get email from je...@je.... Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] "Blondel, Sophie" <sbl...@ut...> writes: > Hi Jed, > > I'm not sure I'm going to reply to your question correctly because I don't really understand how the split is done. Is it related to on diagonal and off diagonal? If so, the off-diagonal part is usually pretty small (less than 20 DOFs) and related to diffusion, the diagonal part involves thousands of DOFs for the reaction term. From the run-time option, it'll be a default (additive) split and we're interested in the two diagonal blocks. One currently has a cheap solver that would only be efficient with a well-conditioned positive definite matrix and the other is using a direct solver ('redundant'). If you were to run with -ksp_view and share the output, it would be informative. Either way, I'd like to understand what physics are beind the equation currently being solved with 'redundant'. If it's diffusive, then algebraic multigrid would be a good place to start. > Let us know what we can do to answer this question more accurately. > > Cheers, > > Sophie > ________________________________ > From: Jed Brown <je...@je...> > Sent: Tuesday, November 28, 2023 19:07 > To: Fackler, Philip <fac...@or...>; Junchao Zhang <jun...@gm...> > Cc: pet...@mc... <pet...@mc...>; xol...@li... <xol...@li...> > Subject: Re: [Xolotl-psi-development] [petsc-users] [EXTERNAL] Re: Unexpected performance losses switching to COO interface > > [Some people who received this message don't often get email from je...@je.... Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] > > "Fackler, Philip via petsc-users" <pet...@mc...> writes: > >> That makes sense. Here are the arguments that I think are relevant: >> >> -fieldsplit_1_pc_type redundant -fieldsplit_0_pc_type sor -pc_type fieldsplit -pc_fieldsplit_detect_coupling > > What sort of physics are in splits 0 and 1? > > SOR is not a good GPU algorithm, so we'll want to change that one way or another. Are the splits of similar size or very different? > >> What would you suggest to make this better? >> >> Also, note that the cases marked "serial" are running on CPU only, that is, using only the SERIAL backend for kokkos. >> >> Philip Fackler >> Research Software Engineer, Application Engineering Group >> Advanced Computing Systems Research Section >> Computer Science and Mathematics Division >> Oak Ridge National Laboratory >> ________________________________ >> From: Junchao Zhang <jun...@gm...> >> Sent: Tuesday, November 28, 2023 15:51 >> To: Fackler, Philip <fac...@or...> >> Cc: pet...@mc... <pet...@mc...>; xol...@li... <xol...@li...> >> Subject: Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface >> >> Hi, Philip, >> I opened hpcdb-PSI_9-serial and it seems you used PCLU. Since Kokkos does not have a GPU LU implementation, we do it on CPU via MatLUFactorNumeric_SeqAIJ(). Perhaps you can try other PC types? >> >> [Screenshot 2023-11-28 at 2.43.03 PM.png] >> --Junchao Zhang >> >> >> On Wed, Nov 22, 2023 at 10:43 AM Fackler, Philip <fac...@or...<mailto:fac...@or...>> wrote: >> I definitely dropped the ball on this. I'm sorry for that. I have new profiling data using the latest (as of yesterday) of petsc/main. I've put them in a single google drive folder linked here: >> >> https://drive.google.com/drive/folders/14ScvyfxOzc4OzXs9HZVeQDO-g6FdIVAI?usp=drive_link<https://urldefense.us/v2/url?u=https-3A__drive.google.com_drive_folders_14ScvyfxOzc4OzXs9HZVeQDO-2Dg6FdIVAI-3Fusp-3Ddrive-5Flink&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=Qn5D9xuzFcMdyuL0I2ruKmU6yeez0NrOx69oUjRaAXTeKD6etHt4USuZgnbqF4v6&s=_Lqg9v8aa4KXUdud3zqSp55FiYkZ12Pp5ZY54_9OvJI&e=> >> >> Have a happy holiday weekend! >> >> Thanks, >> >> Philip Fackler >> Research Software Engineer, Application Engineering Group >> Advanced Computing Systems Research Section >> Computer Science and Mathematics Division >> Oak Ridge National Laboratory >> ________________________________ >> From: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> >> Sent: Monday, October 16, 2023 15:24 >> To: Fackler, Philip <fac...@or...<mailto:fac...@or...>> >> Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>> >> Subject: Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface >> >> Hi, Philip, >> That branch was merged to petsc/main today. Let me know once you have new profiling results. >> >> Thanks. >> --Junchao Zhang >> >> >> On Mon, Oct 16, 2023 at 9:33 AM Fackler, Philip <fac...@or...<mailto:fac...@or...>> wrote: >> Junchao, >> >> I've attached updated timing plots (red and blue are swapped from before; yellow is the new one). There is an improvement for the NE_3 case only with CUDA. Serial stays the same, and the PSI cases stay the same. In the PSI cases, MatShift doesn't show up (I assume because we're using different preconditioner arguments). So, there must be some other primary culprit. I'll try to get updated profiling data to you soon. >> >> Thanks, >> >> Philip Fackler >> Research Software Engineer, Application Engineering Group >> Advanced Computing Systems Research Section >> Computer Science and Mathematics Division >> Oak Ridge National Laboratory >> ________________________________ >> From: Fackler, Philip via Xolotl-psi-development <xol...@li...<mailto:xol...@li...>> >> Sent: Wednesday, October 11, 2023 11:31 >> To: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> >> Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>> >> Subject: Re: [Xolotl-psi-development] [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface >> >> I'm on it. >> >> Philip Fackler >> Research Software Engineer, Application Engineering Group >> Advanced Computing Systems Research Section >> Computer Science and Mathematics Division >> Oak Ridge National Laboratory >> ________________________________ >> From: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> >> Sent: Wednesday, October 11, 2023 10:14 >> To: Fackler, Philip <fac...@or...<mailto:fac...@or...>> >> Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>>; Blondel, Sophie <sbl...@ut...<mailto:sbl...@ut...>> >> Subject: Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface >> >> Hi, Philip, >> Could you try this branch jczhang/2023-10-05/feature-support-matshift-aijkokkos ? >> >> Thanks. >> --Junchao Zhang >> >> >> On Thu, Oct 5, 2023 at 4:52 PM Fackler, Philip <fac...@or...<mailto:fac...@or...>> wrote: >> Aha! That makes sense. Thank you. >> >> Philip Fackler >> Research Software Engineer, Application Engineering Group >> Advanced Computing Systems Research Section >> Computer Science and Mathematics Division >> Oak Ridge National Laboratory >> ________________________________ >> From: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> >> Sent: Thursday, October 5, 2023 17:29 >> To: Fackler, Philip <fac...@or...<mailto:fac...@or...>> >> Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>>; Blondel, Sophie <sbl...@ut...<mailto:sbl...@ut...>> >> Subject: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface >> >> Wait a moment, it seems it was because we do not have a GPU implementation of MatShift... >> Let me see how to add it. >> --Junchao Zhang >> >> >> On Thu, Oct 5, 2023 at 10:58 AM Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> wrote: >> Hi, Philip, >> I looked at the hpcdb-NE_3-cuda file. It seems you used MatSetValues() instead of the COO interface? MatSetValues() needs to copy the data from device to host and thus is expensive. >> Do you have profiling results with COO enabled? >> >> [Screenshot 2023-10-05 at 10.55.29 AM.png] >> >> >> --Junchao Zhang >> >> >> On Mon, Oct 2, 2023 at 9:52 AM Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> wrote: >> Hi, Philip, >> I will look into the tarballs and get back to you. >> Thanks. >> --Junchao Zhang >> >> >> On Mon, Oct 2, 2023 at 9:41 AM Fackler, Philip via petsc-users <pet...@mc...<mailto:pet...@mc...>> wrote: >> We finally have xolotl ported to use the new COO interface and the aijkokkos implementation for Mat (and kokkos for Vec). Comparing this port to our previous version (using MatSetValuesStencil and the default Mat and Vec implementations), we expected to see an improvement in performance for both the "serial" and "cuda" builds (here I'm referring to the kokkos configuration). >> >> Attached are two plots that show timings for three different cases. All of these were run on Ascent (the Summit-like training system) with 6 MPI tasks (on a single node). The CUDA cases were given one GPU per task (and used CUDA-aware MPI). The labels on the blue bars indicate speedup. In all cases we used "-fieldsplit_0_pc_type jacobi" to keep the comparison as consistent as possible. >> >> The performance of RHSJacobian (where the bulk of computation happens in xolotl) behaved basically as expected (better than expected in the serial build). NE_3 case in CUDA was the only one that performed worse, but not surprisingly, since its workload for the GPUs is much smaller. We've still got more optimization to do on this. >> >> The real surprise was how much worse the overall solve times were. This seems to be due simply to switching to the kokkos-based implementation. I'm wondering if there are any changes we can make in configuration or runtime arguments to help with PETSc's performance here. Any help looking into this would be appreciated. >> >> The tarballs linked here<https://urldefense.us/v2/url?u=https-3A__drive.google.com_file_d_19X-5FL3SVkGBM9YUzXnRR-5FkVWFG0JFwqZ3_view-3Fusp-3Ddrive-5Flink&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=GTpC2k9hIdMhUg_aJkeAqd-1CP5M8bwJMJjTriVE1k-j36ZnEHerQkZOzszxWoG2&s=GW0ImGWhWr4rR5AoSULCnaP1CN1QWxTSeMDhdOuhTEA&e=> and here<https://urldefense.us/v2/url?u=https-3A__drive.google.com_file_d_15yDBN7-2DYlO1g6RJNPYNImzr611i1Ffhv_view-3Fusp-3Ddrive-5Flink&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=GTpC2k9hIdMhUg_aJkeAqd-1CP5M8bwJMJjTriVE1k-j36ZnEHerQkZOzszxWoG2&s=tO-BnNY2myA-pIsRnBjQNoaOSjn-B3-lWGiQp7XXJwk&e=> are profiling databases which, once extracted, can be viewed with hpcviewer. I don't know how helpful that will be, but hopefully it can give you some direction. >> >> Thanks for your help, >> >> Philip Fackler >> Research Software Engineer, Application Engineering Group >> Advanced Computing Systems Research Section >> Computer Science and Mathematics Division >> Oak Ridge National Laboratory > > > _______________________________________________ > Xolotl-psi-development mailing list > Xol...@li... > https://lists.sourceforge.net/lists/listinfo/xolotl-psi-development |
From: Jed B. <je...@je...> - 2023-11-30 06:16:03
|
"Blondel, Sophie" <sbl...@ut...> writes: > Hi Jed, > > I'm not sure I'm going to reply to your question correctly because I don't really understand how the split is done. Is it related to on diagonal and off diagonal? If so, the off-diagonal part is usually pretty small (less than 20 DOFs) and related to diffusion, the diagonal part involves thousands of DOFs for the reaction term. >From the run-time option, it'll be a default (additive) split and we're interested in the two diagonal blocks. One currently has a cheap solver that would only be efficient with a well-conditioned positive definite matrix and the other is using a direct solver ('redundant'). If you were to run with -ksp_view and share the output, it would be informative. Either way, I'd like to understand what physics are beind the equation currently being solved with 'redundant'. If it's diffusive, then algebraic multigrid would be a good place to start. > Let us know what we can do to answer this question more accurately. > > Cheers, > > Sophie > ________________________________ > From: Jed Brown <je...@je...> > Sent: Tuesday, November 28, 2023 19:07 > To: Fackler, Philip <fac...@or...>; Junchao Zhang <jun...@gm...> > Cc: pet...@mc... <pet...@mc...>; xol...@li... <xol...@li...> > Subject: Re: [Xolotl-psi-development] [petsc-users] [EXTERNAL] Re: Unexpected performance losses switching to COO interface > > [Some people who received this message don't often get email from je...@je.... Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] > > "Fackler, Philip via petsc-users" <pet...@mc...> writes: > >> That makes sense. Here are the arguments that I think are relevant: >> >> -fieldsplit_1_pc_type redundant -fieldsplit_0_pc_type sor -pc_type fieldsplit -pc_fieldsplit_detect_coupling > > What sort of physics are in splits 0 and 1? > > SOR is not a good GPU algorithm, so we'll want to change that one way or another. Are the splits of similar size or very different? > >> What would you suggest to make this better? >> >> Also, note that the cases marked "serial" are running on CPU only, that is, using only the SERIAL backend for kokkos. >> >> Philip Fackler >> Research Software Engineer, Application Engineering Group >> Advanced Computing Systems Research Section >> Computer Science and Mathematics Division >> Oak Ridge National Laboratory >> ________________________________ >> From: Junchao Zhang <jun...@gm...> >> Sent: Tuesday, November 28, 2023 15:51 >> To: Fackler, Philip <fac...@or...> >> Cc: pet...@mc... <pet...@mc...>; xol...@li... <xol...@li...> >> Subject: Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface >> >> Hi, Philip, >> I opened hpcdb-PSI_9-serial and it seems you used PCLU. Since Kokkos does not have a GPU LU implementation, we do it on CPU via MatLUFactorNumeric_SeqAIJ(). Perhaps you can try other PC types? >> >> [Screenshot 2023-11-28 at 2.43.03 PM.png] >> --Junchao Zhang >> >> >> On Wed, Nov 22, 2023 at 10:43 AM Fackler, Philip <fac...@or...<mailto:fac...@or...>> wrote: >> I definitely dropped the ball on this. I'm sorry for that. I have new profiling data using the latest (as of yesterday) of petsc/main. I've put them in a single google drive folder linked here: >> >> https://drive.google.com/drive/folders/14ScvyfxOzc4OzXs9HZVeQDO-g6FdIVAI?usp=drive_link<https://urldefense.us/v2/url?u=https-3A__drive.google.com_drive_folders_14ScvyfxOzc4OzXs9HZVeQDO-2Dg6FdIVAI-3Fusp-3Ddrive-5Flink&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=Qn5D9xuzFcMdyuL0I2ruKmU6yeez0NrOx69oUjRaAXTeKD6etHt4USuZgnbqF4v6&s=_Lqg9v8aa4KXUdud3zqSp55FiYkZ12Pp5ZY54_9OvJI&e=> >> >> Have a happy holiday weekend! >> >> Thanks, >> >> Philip Fackler >> Research Software Engineer, Application Engineering Group >> Advanced Computing Systems Research Section >> Computer Science and Mathematics Division >> Oak Ridge National Laboratory >> ________________________________ >> From: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> >> Sent: Monday, October 16, 2023 15:24 >> To: Fackler, Philip <fac...@or...<mailto:fac...@or...>> >> Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>> >> Subject: Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface >> >> Hi, Philip, >> That branch was merged to petsc/main today. Let me know once you have new profiling results. >> >> Thanks. >> --Junchao Zhang >> >> >> On Mon, Oct 16, 2023 at 9:33 AM Fackler, Philip <fac...@or...<mailto:fac...@or...>> wrote: >> Junchao, >> >> I've attached updated timing plots (red and blue are swapped from before; yellow is the new one). There is an improvement for the NE_3 case only with CUDA. Serial stays the same, and the PSI cases stay the same. In the PSI cases, MatShift doesn't show up (I assume because we're using different preconditioner arguments). So, there must be some other primary culprit. I'll try to get updated profiling data to you soon. >> >> Thanks, >> >> Philip Fackler >> Research Software Engineer, Application Engineering Group >> Advanced Computing Systems Research Section >> Computer Science and Mathematics Division >> Oak Ridge National Laboratory >> ________________________________ >> From: Fackler, Philip via Xolotl-psi-development <xol...@li...<mailto:xol...@li...>> >> Sent: Wednesday, October 11, 2023 11:31 >> To: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> >> Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>> >> Subject: Re: [Xolotl-psi-development] [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface >> >> I'm on it. >> >> Philip Fackler >> Research Software Engineer, Application Engineering Group >> Advanced Computing Systems Research Section >> Computer Science and Mathematics Division >> Oak Ridge National Laboratory >> ________________________________ >> From: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> >> Sent: Wednesday, October 11, 2023 10:14 >> To: Fackler, Philip <fac...@or...<mailto:fac...@or...>> >> Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>>; Blondel, Sophie <sbl...@ut...<mailto:sbl...@ut...>> >> Subject: Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface >> >> Hi, Philip, >> Could you try this branch jczhang/2023-10-05/feature-support-matshift-aijkokkos ? >> >> Thanks. >> --Junchao Zhang >> >> >> On Thu, Oct 5, 2023 at 4:52 PM Fackler, Philip <fac...@or...<mailto:fac...@or...>> wrote: >> Aha! That makes sense. Thank you. >> >> Philip Fackler >> Research Software Engineer, Application Engineering Group >> Advanced Computing Systems Research Section >> Computer Science and Mathematics Division >> Oak Ridge National Laboratory >> ________________________________ >> From: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> >> Sent: Thursday, October 5, 2023 17:29 >> To: Fackler, Philip <fac...@or...<mailto:fac...@or...>> >> Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>>; Blondel, Sophie <sbl...@ut...<mailto:sbl...@ut...>> >> Subject: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface >> >> Wait a moment, it seems it was because we do not have a GPU implementation of MatShift... >> Let me see how to add it. >> --Junchao Zhang >> >> >> On Thu, Oct 5, 2023 at 10:58 AM Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> wrote: >> Hi, Philip, >> I looked at the hpcdb-NE_3-cuda file. It seems you used MatSetValues() instead of the COO interface? MatSetValues() needs to copy the data from device to host and thus is expensive. >> Do you have profiling results with COO enabled? >> >> [Screenshot 2023-10-05 at 10.55.29 AM.png] >> >> >> --Junchao Zhang >> >> >> On Mon, Oct 2, 2023 at 9:52 AM Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> wrote: >> Hi, Philip, >> I will look into the tarballs and get back to you. >> Thanks. >> --Junchao Zhang >> >> >> On Mon, Oct 2, 2023 at 9:41 AM Fackler, Philip via petsc-users <pet...@mc...<mailto:pet...@mc...>> wrote: >> We finally have xolotl ported to use the new COO interface and the aijkokkos implementation for Mat (and kokkos for Vec). Comparing this port to our previous version (using MatSetValuesStencil and the default Mat and Vec implementations), we expected to see an improvement in performance for both the "serial" and "cuda" builds (here I'm referring to the kokkos configuration). >> >> Attached are two plots that show timings for three different cases. All of these were run on Ascent (the Summit-like training system) with 6 MPI tasks (on a single node). The CUDA cases were given one GPU per task (and used CUDA-aware MPI). The labels on the blue bars indicate speedup. In all cases we used "-fieldsplit_0_pc_type jacobi" to keep the comparison as consistent as possible. >> >> The performance of RHSJacobian (where the bulk of computation happens in xolotl) behaved basically as expected (better than expected in the serial build). NE_3 case in CUDA was the only one that performed worse, but not surprisingly, since its workload for the GPUs is much smaller. We've still got more optimization to do on this. >> >> The real surprise was how much worse the overall solve times were. This seems to be due simply to switching to the kokkos-based implementation. I'm wondering if there are any changes we can make in configuration or runtime arguments to help with PETSc's performance here. Any help looking into this would be appreciated. >> >> The tarballs linked here<https://urldefense.us/v2/url?u=https-3A__drive.google.com_file_d_19X-5FL3SVkGBM9YUzXnRR-5FkVWFG0JFwqZ3_view-3Fusp-3Ddrive-5Flink&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=GTpC2k9hIdMhUg_aJkeAqd-1CP5M8bwJMJjTriVE1k-j36ZnEHerQkZOzszxWoG2&s=GW0ImGWhWr4rR5AoSULCnaP1CN1QWxTSeMDhdOuhTEA&e=> and here<https://urldefense.us/v2/url?u=https-3A__drive.google.com_file_d_15yDBN7-2DYlO1g6RJNPYNImzr611i1Ffhv_view-3Fusp-3Ddrive-5Flink&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=GTpC2k9hIdMhUg_aJkeAqd-1CP5M8bwJMJjTriVE1k-j36ZnEHerQkZOzszxWoG2&s=tO-BnNY2myA-pIsRnBjQNoaOSjn-B3-lWGiQp7XXJwk&e=> are profiling databases which, once extracted, can be viewed with hpcviewer. I don't know how helpful that will be, but hopefully it can give you some direction. >> >> Thanks for your help, >> >> Philip Fackler >> Research Software Engineer, Application Engineering Group >> Advanced Computing Systems Research Section >> Computer Science and Mathematics Division >> Oak Ridge National Laboratory > > > _______________________________________________ > Xolotl-psi-development mailing list > Xol...@li... > https://lists.sourceforge.net/lists/listinfo/xolotl-psi-development |
From: Blondel, S. <sbl...@ut...> - 2023-11-29 20:35:17
|
Hi Jed, I'm not sure I'm going to reply to your question correctly because I don't really understand how the split is done. Is it related to on diagonal and off diagonal? If so, the off-diagonal part is usually pretty small (less than 20 DOFs) and related to diffusion, the diagonal part involves thousands of DOFs for the reaction term. Let us know what we can do to answer this question more accurately. Cheers, Sophie ________________________________ From: Jed Brown <je...@je...> Sent: Tuesday, November 28, 2023 19:07 To: Fackler, Philip <fac...@or...>; Junchao Zhang <jun...@gm...> Cc: pet...@mc... <pet...@mc...>; xol...@li... <xol...@li...> Subject: Re: [Xolotl-psi-development] [petsc-users] [EXTERNAL] Re: Unexpected performance losses switching to COO interface [Some people who received this message don't often get email from je...@je.... Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] "Fackler, Philip via petsc-users" <pet...@mc...> writes: > That makes sense. Here are the arguments that I think are relevant: > > -fieldsplit_1_pc_type redundant -fieldsplit_0_pc_type sor -pc_type fieldsplit -pc_fieldsplit_detect_coupling What sort of physics are in splits 0 and 1? SOR is not a good GPU algorithm, so we'll want to change that one way or another. Are the splits of similar size or very different? > What would you suggest to make this better? > > Also, note that the cases marked "serial" are running on CPU only, that is, using only the SERIAL backend for kokkos. > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory > ________________________________ > From: Junchao Zhang <jun...@gm...> > Sent: Tuesday, November 28, 2023 15:51 > To: Fackler, Philip <fac...@or...> > Cc: pet...@mc... <pet...@mc...>; xol...@li... <xol...@li...> > Subject: Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface > > Hi, Philip, > I opened hpcdb-PSI_9-serial and it seems you used PCLU. Since Kokkos does not have a GPU LU implementation, we do it on CPU via MatLUFactorNumeric_SeqAIJ(). Perhaps you can try other PC types? > > [Screenshot 2023-11-28 at 2.43.03 PM.png] > --Junchao Zhang > > > On Wed, Nov 22, 2023 at 10:43 AM Fackler, Philip <fac...@or...<mailto:fac...@or...>> wrote: > I definitely dropped the ball on this. I'm sorry for that. I have new profiling data using the latest (as of yesterday) of petsc/main. I've put them in a single google drive folder linked here: > > https://drive.google.com/drive/folders/14ScvyfxOzc4OzXs9HZVeQDO-g6FdIVAI?usp=drive_link<https://urldefense.us/v2/url?u=https-3A__drive.google.com_drive_folders_14ScvyfxOzc4OzXs9HZVeQDO-2Dg6FdIVAI-3Fusp-3Ddrive-5Flink&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=Qn5D9xuzFcMdyuL0I2ruKmU6yeez0NrOx69oUjRaAXTeKD6etHt4USuZgnbqF4v6&s=_Lqg9v8aa4KXUdud3zqSp55FiYkZ12Pp5ZY54_9OvJI&e=> > > Have a happy holiday weekend! > > Thanks, > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory > ________________________________ > From: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> > Sent: Monday, October 16, 2023 15:24 > To: Fackler, Philip <fac...@or...<mailto:fac...@or...>> > Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>> > Subject: Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface > > Hi, Philip, > That branch was merged to petsc/main today. Let me know once you have new profiling results. > > Thanks. > --Junchao Zhang > > > On Mon, Oct 16, 2023 at 9:33 AM Fackler, Philip <fac...@or...<mailto:fac...@or...>> wrote: > Junchao, > > I've attached updated timing plots (red and blue are swapped from before; yellow is the new one). There is an improvement for the NE_3 case only with CUDA. Serial stays the same, and the PSI cases stay the same. In the PSI cases, MatShift doesn't show up (I assume because we're using different preconditioner arguments). So, there must be some other primary culprit. I'll try to get updated profiling data to you soon. > > Thanks, > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory > ________________________________ > From: Fackler, Philip via Xolotl-psi-development <xol...@li...<mailto:xol...@li...>> > Sent: Wednesday, October 11, 2023 11:31 > To: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> > Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>> > Subject: Re: [Xolotl-psi-development] [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface > > I'm on it. > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory > ________________________________ > From: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> > Sent: Wednesday, October 11, 2023 10:14 > To: Fackler, Philip <fac...@or...<mailto:fac...@or...>> > Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>>; Blondel, Sophie <sbl...@ut...<mailto:sbl...@ut...>> > Subject: Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface > > Hi, Philip, > Could you try this branch jczhang/2023-10-05/feature-support-matshift-aijkokkos ? > > Thanks. > --Junchao Zhang > > > On Thu, Oct 5, 2023 at 4:52 PM Fackler, Philip <fac...@or...<mailto:fac...@or...>> wrote: > Aha! That makes sense. Thank you. > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory > ________________________________ > From: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> > Sent: Thursday, October 5, 2023 17:29 > To: Fackler, Philip <fac...@or...<mailto:fac...@or...>> > Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>>; Blondel, Sophie <sbl...@ut...<mailto:sbl...@ut...>> > Subject: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface > > Wait a moment, it seems it was because we do not have a GPU implementation of MatShift... > Let me see how to add it. > --Junchao Zhang > > > On Thu, Oct 5, 2023 at 10:58 AM Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> wrote: > Hi, Philip, > I looked at the hpcdb-NE_3-cuda file. It seems you used MatSetValues() instead of the COO interface? MatSetValues() needs to copy the data from device to host and thus is expensive. > Do you have profiling results with COO enabled? > > [Screenshot 2023-10-05 at 10.55.29 AM.png] > > > --Junchao Zhang > > > On Mon, Oct 2, 2023 at 9:52 AM Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> wrote: > Hi, Philip, > I will look into the tarballs and get back to you. > Thanks. > --Junchao Zhang > > > On Mon, Oct 2, 2023 at 9:41 AM Fackler, Philip via petsc-users <pet...@mc...<mailto:pet...@mc...>> wrote: > We finally have xolotl ported to use the new COO interface and the aijkokkos implementation for Mat (and kokkos for Vec). Comparing this port to our previous version (using MatSetValuesStencil and the default Mat and Vec implementations), we expected to see an improvement in performance for both the "serial" and "cuda" builds (here I'm referring to the kokkos configuration). > > Attached are two plots that show timings for three different cases. All of these were run on Ascent (the Summit-like training system) with 6 MPI tasks (on a single node). The CUDA cases were given one GPU per task (and used CUDA-aware MPI). The labels on the blue bars indicate speedup. In all cases we used "-fieldsplit_0_pc_type jacobi" to keep the comparison as consistent as possible. > > The performance of RHSJacobian (where the bulk of computation happens in xolotl) behaved basically as expected (better than expected in the serial build). NE_3 case in CUDA was the only one that performed worse, but not surprisingly, since its workload for the GPUs is much smaller. We've still got more optimization to do on this. > > The real surprise was how much worse the overall solve times were. This seems to be due simply to switching to the kokkos-based implementation. I'm wondering if there are any changes we can make in configuration or runtime arguments to help with PETSc's performance here. Any help looking into this would be appreciated. > > The tarballs linked here<https://urldefense.us/v2/url?u=https-3A__drive.google.com_file_d_19X-5FL3SVkGBM9YUzXnRR-5FkVWFG0JFwqZ3_view-3Fusp-3Ddrive-5Flink&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=GTpC2k9hIdMhUg_aJkeAqd-1CP5M8bwJMJjTriVE1k-j36ZnEHerQkZOzszxWoG2&s=GW0ImGWhWr4rR5AoSULCnaP1CN1QWxTSeMDhdOuhTEA&e=> and here<https://urldefense.us/v2/url?u=https-3A__drive.google.com_file_d_15yDBN7-2DYlO1g6RJNPYNImzr611i1Ffhv_view-3Fusp-3Ddrive-5Flink&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=GTpC2k9hIdMhUg_aJkeAqd-1CP5M8bwJMJjTriVE1k-j36ZnEHerQkZOzszxWoG2&s=tO-BnNY2myA-pIsRnBjQNoaOSjn-B3-lWGiQp7XXJwk&e=> are profiling databases which, once extracted, can be viewed with hpcviewer. I don't know how helpful that will be, but hopefully it can give you some direction. > > Thanks for your help, > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory _______________________________________________ Xolotl-psi-development mailing list Xol...@li... https://lists.sourceforge.net/lists/listinfo/xolotl-psi-development |
From: Fackler, P. <fac...@or...> - 2023-11-29 16:07:15
|
I'm sorry for the extra confusion. I copied those arguments from the wrong place. We're actually using jacobi instead of sor for fieldsplit 0. Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory ________________________________ From: Blondel, Sophie <sbl...@ut...> Sent: Wednesday, November 29, 2023 11:03 To: Brown, Jed <je...@je...>; Fackler, Philip <fac...@or...>; Junchao Zhang <jun...@gm...> Cc: pet...@mc... <pet...@mc...>; xol...@li... <xol...@li...> Subject: Re: [Xolotl-psi-development] [petsc-users] [EXTERNAL] Re: Unexpected performance losses switching to COO interface Hi Jed, I'm not sure I'm going to reply to your question correctly because I don't really understand how the split is done. Is it related to on diagonal and off diagonal? If so, the off-diagonal part is usually pretty small (less than 20 DOFs) and related to diffusion, the diagonal part involves thousands of DOFs for the reaction term. Let us know what we can do to answer this question more accurately. Cheers, Sophie ________________________________ From: Jed Brown <je...@je...> Sent: Tuesday, November 28, 2023 19:07 To: Fackler, Philip <fac...@or...>; Junchao Zhang <jun...@gm...> Cc: pet...@mc... <pet...@mc...>; xol...@li... <xol...@li...> Subject: Re: [Xolotl-psi-development] [petsc-users] [EXTERNAL] Re: Unexpected performance losses switching to COO interface [Some people who received this message don't often get email from je...@je.... Learn why this is important at https://aka.ms/LearnAboutSenderIdentification<https://urldefense.us/v2/url?u=https-3A__aka.ms_LearnAboutSenderIdentification&d=DwMGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=owjSIigcMR9I7l2CwJBbVn5A6D7SU6RaEKwSNLafLr2msXHmjhMhZWSPCqdwWRAI&s=dxTWlHKB4itRnwMh05b4rPf4V4axP7XpXAJIUNJYWoQ&e=> ] "Fackler, Philip via petsc-users" <pet...@mc...> writes: > That makes sense. Here are the arguments that I think are relevant: > > -fieldsplit_1_pc_type redundant -fieldsplit_0_pc_type sor -pc_type fieldsplit -pc_fieldsplit_detect_coupling What sort of physics are in splits 0 and 1? SOR is not a good GPU algorithm, so we'll want to change that one way or another. Are the splits of similar size or very different? > What would you suggest to make this better? > > Also, note that the cases marked "serial" are running on CPU only, that is, using only the SERIAL backend for kokkos. > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory > ________________________________ > From: Junchao Zhang <jun...@gm...> > Sent: Tuesday, November 28, 2023 15:51 > To: Fackler, Philip <fac...@or...> > Cc: pet...@mc... <pet...@mc...>; xol...@li... <xol...@li...> > Subject: Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface > > Hi, Philip, > I opened hpcdb-PSI_9-serial and it seems you used PCLU. Since Kokkos does not have a GPU LU implementation, we do it on CPU via MatLUFactorNumeric_SeqAIJ(). Perhaps you can try other PC types? > > [Screenshot 2023-11-28 at 2.43.03 PM.png] > --Junchao Zhang > > > On Wed, Nov 22, 2023 at 10:43 AM Fackler, Philip <fac...@or...<mailto:fac...@or...>> wrote: > I definitely dropped the ball on this. I'm sorry for that. I have new profiling data using the latest (as of yesterday) of petsc/main. I've put them in a single google drive folder linked here: > > https://drive.google.com/drive/folders/14ScvyfxOzc4OzXs9HZVeQDO-g6FdIVAI?usp=drive_link<https://urldefense.us/v2/url?u=https-3A__drive.google.com_drive_folders_14ScvyfxOzc4OzXs9HZVeQDO-2Dg6FdIVAI-3Fusp-3Ddrive-5Flink&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=Qn5D9xuzFcMdyuL0I2ruKmU6yeez0NrOx69oUjRaAXTeKD6etHt4USuZgnbqF4v6&s=_Lqg9v8aa4KXUdud3zqSp55FiYkZ12Pp5ZY54_9OvJI&e=><https://urldefense.us/v2/url?u=https-3A__drive.google.com_drive_folders_14ScvyfxOzc4OzXs9HZVeQDO-2Dg6FdIVAI-3Fusp-3Ddrive-5Flink-253Chttps-3A__urldefense.us_v2_url-3Fu-3Dhttps-2D3A-5F-5Fdrive.google.com-5Fdrive-5Ffolders-5F14ScvyfxOzc4OzXs9HZVeQDO-2D2Dg6FdIVAI-2D3Fusp-2D3Ddrive-2D5Flink-26d-3DDwMFaQ-26c-3Dv4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-2DO7C4ViYc-26r-3DDAkLCjn8leYU-2DuJ-2DkfNEQMhPZWx9lzc4d5KgIR-2DRZWQ-26m-3DQn5D9xuzFcMdyuL0I2ruKmU6yeez0NrOx69oUjRaAXTeKD6etHt4USuZgnbqF4v6-26s-3D-5FLqg9v8aa4KXUdud3zqSp55FiYkZ12Pp5ZY54-5F9OvJI-26e-3D-253E&d=DwMGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=owjSIigcMR9I7l2CwJBbVn5A6D7SU6RaEKwSNLafLr2msXHmjhMhZWSPCqdwWRAI&s=oGtVi32Tn6u94h6NHstgb1mNruDdymyS686_qBYnxTs&e=> > > Have a happy holiday weekend! > > Thanks, > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory > ________________________________ > From: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> > Sent: Monday, October 16, 2023 15:24 > To: Fackler, Philip <fac...@or...<mailto:fac...@or...>> > Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>> > Subject: Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface > > Hi, Philip, > That branch was merged to petsc/main today. Let me know once you have new profiling results. > > Thanks. > --Junchao Zhang > > > On Mon, Oct 16, 2023 at 9:33 AM Fackler, Philip <fac...@or...<mailto:fac...@or...>> wrote: > Junchao, > > I've attached updated timing plots (red and blue are swapped from before; yellow is the new one). There is an improvement for the NE_3 case only with CUDA. Serial stays the same, and the PSI cases stay the same. In the PSI cases, MatShift doesn't show up (I assume because we're using different preconditioner arguments). So, there must be some other primary culprit. I'll try to get updated profiling data to you soon. > > Thanks, > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory > ________________________________ > From: Fackler, Philip via Xolotl-psi-development <xol...@li...<mailto:xol...@li...>> > Sent: Wednesday, October 11, 2023 11:31 > To: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> > Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>> > Subject: Re: [Xolotl-psi-development] [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface > > I'm on it. > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory > ________________________________ > From: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> > Sent: Wednesday, October 11, 2023 10:14 > To: Fackler, Philip <fac...@or...<mailto:fac...@or...>> > Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>>; Blondel, Sophie <sbl...@ut...<mailto:sbl...@ut...>> > Subject: Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface > > Hi, Philip, > Could you try this branch jczhang/2023-10-05/feature-support-matshift-aijkokkos ? > > Thanks. > --Junchao Zhang > > > On Thu, Oct 5, 2023 at 4:52 PM Fackler, Philip <fac...@or...<mailto:fac...@or...>> wrote: > Aha! That makes sense. Thank you. > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory > ________________________________ > From: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> > Sent: Thursday, October 5, 2023 17:29 > To: Fackler, Philip <fac...@or...<mailto:fac...@or...>> > Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>>; Blondel, Sophie <sbl...@ut...<mailto:sbl...@ut...>> > Subject: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface > > Wait a moment, it seems it was because we do not have a GPU implementation of MatShift... > Let me see how to add it. > --Junchao Zhang > > > On Thu, Oct 5, 2023 at 10:58 AM Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> wrote: > Hi, Philip, > I looked at the hpcdb-NE_3-cuda file. It seems you used MatSetValues() instead of the COO interface? MatSetValues() needs to copy the data from device to host and thus is expensive. > Do you have profiling results with COO enabled? > > [Screenshot 2023-10-05 at 10.55.29 AM.png] > > > --Junchao Zhang > > > On Mon, Oct 2, 2023 at 9:52 AM Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> wrote: > Hi, Philip, > I will look into the tarballs and get back to you. > Thanks. > --Junchao Zhang > > > On Mon, Oct 2, 2023 at 9:41 AM Fackler, Philip via petsc-users <pet...@mc...<mailto:pet...@mc...>> wrote: > We finally have xolotl ported to use the new COO interface and the aijkokkos implementation for Mat (and kokkos for Vec). Comparing this port to our previous version (using MatSetValuesStencil and the default Mat and Vec implementations), we expected to see an improvement in performance for both the "serial" and "cuda" builds (here I'm referring to the kokkos configuration). > > Attached are two plots that show timings for three different cases. All of these were run on Ascent (the Summit-like training system) with 6 MPI tasks (on a single node). The CUDA cases were given one GPU per task (and used CUDA-aware MPI). The labels on the blue bars indicate speedup. In all cases we used "-fieldsplit_0_pc_type jacobi" to keep the comparison as consistent as possible. > > The performance of RHSJacobian (where the bulk of computation happens in xolotl) behaved basically as expected (better than expected in the serial build). NE_3 case in CUDA was the only one that performed worse, but not surprisingly, since its workload for the GPUs is much smaller. We've still got more optimization to do on this. > > The real surprise was how much worse the overall solve times were. This seems to be due simply to switching to the kokkos-based implementation. I'm wondering if there are any changes we can make in configuration or runtime arguments to help with PETSc's performance here. Any help looking into this would be appreciated. > > The tarballs linked here<https://urldefense.us/v2/url?u=https-3A__drive.google.com_file_d_19X-5FL3SVkGBM9YUzXnRR-5FkVWFG0JFwqZ3_view-3Fusp-3Ddrive-5Flink&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=GTpC2k9hIdMhUg_aJkeAqd-1CP5M8bwJMJjTriVE1k-j36ZnEHerQkZOzszxWoG2&s=GW0ImGWhWr4rR5AoSULCnaP1CN1QWxTSeMDhdOuhTEA&e=> and here<https://urldefense.us/v2/url?u=https-3A__drive.google.com_file_d_15yDBN7-2DYlO1g6RJNPYNImzr611i1Ffhv_view-3Fusp-3Ddrive-5Flink&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=GTpC2k9hIdMhUg_aJkeAqd-1CP5M8bwJMJjTriVE1k-j36ZnEHerQkZOzszxWoG2&s=tO-BnNY2myA-pIsRnBjQNoaOSjn-B3-lWGiQp7XXJwk&e=> are profiling databases which, once extracted, can be viewed with hpcviewer. I don't know how helpful that will be, but hopefully it can give you some direction. > > Thanks for your help, > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory _______________________________________________ Xolotl-psi-development mailing list Xol...@li... https://lists.sourceforge.net/lists/listinfo/xolotl-psi-development<https://urldefense.us/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_xolotl-2Dpsi-2Ddevelopment&d=DwMGaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=owjSIigcMR9I7l2CwJBbVn5A6D7SU6RaEKwSNLafLr2msXHmjhMhZWSPCqdwWRAI&s=zn0HH-IEm_zIRwx6-ygXmOaATMISyU9y9ykDhR8w9O8&e=> |
From: Jed B. <je...@je...> - 2023-11-29 01:50:55
|
"Fackler, Philip via petsc-users" <pet...@mc...> writes: > That makes sense. Here are the arguments that I think are relevant: > > -fieldsplit_1_pc_type redundant -fieldsplit_0_pc_type sor -pc_type fieldsplit -pc_fieldsplit_detect_coupling What sort of physics are in splits 0 and 1? SOR is not a good GPU algorithm, so we'll want to change that one way or another. Are the splits of similar size or very different? > What would you suggest to make this better? > > Also, note that the cases marked "serial" are running on CPU only, that is, using only the SERIAL backend for kokkos. > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory > ________________________________ > From: Junchao Zhang <jun...@gm...> > Sent: Tuesday, November 28, 2023 15:51 > To: Fackler, Philip <fac...@or...> > Cc: pet...@mc... <pet...@mc...>; xol...@li... <xol...@li...> > Subject: Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface > > Hi, Philip, > I opened hpcdb-PSI_9-serial and it seems you used PCLU. Since Kokkos does not have a GPU LU implementation, we do it on CPU via MatLUFactorNumeric_SeqAIJ(). Perhaps you can try other PC types? > > [Screenshot 2023-11-28 at 2.43.03 PM.png] > --Junchao Zhang > > > On Wed, Nov 22, 2023 at 10:43 AM Fackler, Philip <fac...@or...<mailto:fac...@or...>> wrote: > I definitely dropped the ball on this. I'm sorry for that. I have new profiling data using the latest (as of yesterday) of petsc/main. I've put them in a single google drive folder linked here: > > https://drive.google.com/drive/folders/14ScvyfxOzc4OzXs9HZVeQDO-g6FdIVAI?usp=drive_link<https://urldefense.us/v2/url?u=https-3A__drive.google.com_drive_folders_14ScvyfxOzc4OzXs9HZVeQDO-2Dg6FdIVAI-3Fusp-3Ddrive-5Flink&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=Qn5D9xuzFcMdyuL0I2ruKmU6yeez0NrOx69oUjRaAXTeKD6etHt4USuZgnbqF4v6&s=_Lqg9v8aa4KXUdud3zqSp55FiYkZ12Pp5ZY54_9OvJI&e=> > > Have a happy holiday weekend! > > Thanks, > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory > ________________________________ > From: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> > Sent: Monday, October 16, 2023 15:24 > To: Fackler, Philip <fac...@or...<mailto:fac...@or...>> > Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>> > Subject: Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface > > Hi, Philip, > That branch was merged to petsc/main today. Let me know once you have new profiling results. > > Thanks. > --Junchao Zhang > > > On Mon, Oct 16, 2023 at 9:33 AM Fackler, Philip <fac...@or...<mailto:fac...@or...>> wrote: > Junchao, > > I've attached updated timing plots (red and blue are swapped from before; yellow is the new one). There is an improvement for the NE_3 case only with CUDA. Serial stays the same, and the PSI cases stay the same. In the PSI cases, MatShift doesn't show up (I assume because we're using different preconditioner arguments). So, there must be some other primary culprit. I'll try to get updated profiling data to you soon. > > Thanks, > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory > ________________________________ > From: Fackler, Philip via Xolotl-psi-development <xol...@li...<mailto:xol...@li...>> > Sent: Wednesday, October 11, 2023 11:31 > To: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> > Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>> > Subject: Re: [Xolotl-psi-development] [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface > > I'm on it. > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory > ________________________________ > From: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> > Sent: Wednesday, October 11, 2023 10:14 > To: Fackler, Philip <fac...@or...<mailto:fac...@or...>> > Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>>; Blondel, Sophie <sbl...@ut...<mailto:sbl...@ut...>> > Subject: Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface > > Hi, Philip, > Could you try this branch jczhang/2023-10-05/feature-support-matshift-aijkokkos ? > > Thanks. > --Junchao Zhang > > > On Thu, Oct 5, 2023 at 4:52 PM Fackler, Philip <fac...@or...<mailto:fac...@or...>> wrote: > Aha! That makes sense. Thank you. > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory > ________________________________ > From: Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> > Sent: Thursday, October 5, 2023 17:29 > To: Fackler, Philip <fac...@or...<mailto:fac...@or...>> > Cc: pet...@mc...<mailto:pet...@mc...> <pet...@mc...<mailto:pet...@mc...>>; xol...@li...<mailto:xol...@li...> <xol...@li...<mailto:xol...@li...>>; Blondel, Sophie <sbl...@ut...<mailto:sbl...@ut...>> > Subject: [EXTERNAL] Re: [petsc-users] Unexpected performance losses switching to COO interface > > Wait a moment, it seems it was because we do not have a GPU implementation of MatShift... > Let me see how to add it. > --Junchao Zhang > > > On Thu, Oct 5, 2023 at 10:58 AM Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> wrote: > Hi, Philip, > I looked at the hpcdb-NE_3-cuda file. It seems you used MatSetValues() instead of the COO interface? MatSetValues() needs to copy the data from device to host and thus is expensive. > Do you have profiling results with COO enabled? > > [Screenshot 2023-10-05 at 10.55.29 AM.png] > > > --Junchao Zhang > > > On Mon, Oct 2, 2023 at 9:52 AM Junchao Zhang <jun...@gm...<mailto:jun...@gm...>> wrote: > Hi, Philip, > I will look into the tarballs and get back to you. > Thanks. > --Junchao Zhang > > > On Mon, Oct 2, 2023 at 9:41 AM Fackler, Philip via petsc-users <pet...@mc...<mailto:pet...@mc...>> wrote: > We finally have xolotl ported to use the new COO interface and the aijkokkos implementation for Mat (and kokkos for Vec). Comparing this port to our previous version (using MatSetValuesStencil and the default Mat and Vec implementations), we expected to see an improvement in performance for both the "serial" and "cuda" builds (here I'm referring to the kokkos configuration). > > Attached are two plots that show timings for three different cases. All of these were run on Ascent (the Summit-like training system) with 6 MPI tasks (on a single node). The CUDA cases were given one GPU per task (and used CUDA-aware MPI). The labels on the blue bars indicate speedup. In all cases we used "-fieldsplit_0_pc_type jacobi" to keep the comparison as consistent as possible. > > The performance of RHSJacobian (where the bulk of computation happens in xolotl) behaved basically as expected (better than expected in the serial build). NE_3 case in CUDA was the only one that performed worse, but not surprisingly, since its workload for the GPUs is much smaller. We've still got more optimization to do on this. > > The real surprise was how much worse the overall solve times were. This seems to be due simply to switching to the kokkos-based implementation. I'm wondering if there are any changes we can make in configuration or runtime arguments to help with PETSc's performance here. Any help looking into this would be appreciated. > > The tarballs linked here<https://urldefense.us/v2/url?u=https-3A__drive.google.com_file_d_19X-5FL3SVkGBM9YUzXnRR-5FkVWFG0JFwqZ3_view-3Fusp-3Ddrive-5Flink&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=GTpC2k9hIdMhUg_aJkeAqd-1CP5M8bwJMJjTriVE1k-j36ZnEHerQkZOzszxWoG2&s=GW0ImGWhWr4rR5AoSULCnaP1CN1QWxTSeMDhdOuhTEA&e=> and here<https://urldefense.us/v2/url?u=https-3A__drive.google.com_file_d_15yDBN7-2DYlO1g6RJNPYNImzr611i1Ffhv_view-3Fusp-3Ddrive-5Flink&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=GTpC2k9hIdMhUg_aJkeAqd-1CP5M8bwJMJjTriVE1k-j36ZnEHerQkZOzszxWoG2&s=tO-BnNY2myA-pIsRnBjQNoaOSjn-B3-lWGiQp7XXJwk&e=> are profiling databases which, once extracted, can be viewed with hpcviewer. I don't know how helpful that will be, but hopefully it can give you some direction. > > Thanks for your help, > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory |