From: Jed B. <je...@je...> - 2023-12-18 19:47:25
|
Great, thanks for letting us know. It'll merge to release shortly and thus be in petsc >= 3.20.3. "Fackler, Philip" <fac...@or...> writes: > Jed, > > That seems to have worked (ridiculously well). It's now 55MB, and it's happening in the call to MatSetPreallocationCOO. > > Thank you, > > Philip Fackler > Research Software Engineer, Application Engineering Group > Advanced Computing Systems Research Section > Computer Science and Mathematics Division > Oak Ridge National Laboratory > ________________________________ > From: Jed Brown <je...@je...> > Sent: Thursday, December 14, 2023 16:27 > To: Fackler, Philip <fac...@or...>; pet...@mc... <pet...@mc...>; xol...@li... <xol...@li...> > Subject: [EXTERNAL] Re: [petsc-users] Call to DMSetMatrixPreallocateSkip not changing allocation behavior > > I had a one-character typo in the diff above. This MR to release should work now. > > https://urldefense.us/v2/url?u=https-3A__gitlab.com_petsc_petsc_-2D_merge-5Frequests_7120&d=DwIBAg&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=v9sHqomCGBRWotign4NcwYwOpszOJehUGs_EO3eGn4SSZqxnfK7Iv15-X8nO1lii&s=h_jIP-6WcIjR6LssfGrV6Z2DojlN_w7Me4-a4rBE074&e= > > Jed Brown <je...@je...> writes: > >> 17 GB for a 1D DMDA, wow. :-) >> >> Could you try applying this diff to make it work for DMDA (it's currently handled by DMPlex)? >> >> diff --git i/src/dm/impls/da/fdda.c w/src/dm/impls/da/fdda.c >> index cad4d926504..bd2a3bda635 100644 >> --- i/src/dm/impls/da/fdda.c >> +++ w/src/dm/impls/da/fdda.c >> @@ -675,19 +675,21 @@ PetscErrorCode DMCreateMatrix_DA(DM da, Mat *J) >> specialized setting routines depend only on the particular preallocation >> details of the matrix, not the type itself. >> */ >> - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIAIJSetPreallocation_C", &aij)); >> - if (!aij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqAIJSetPreallocation_C", &aij)); >> - if (!aij) { >> - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIBAIJSetPreallocation_C", &baij)); >> - if (!baij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqBAIJSetPreallocation_C", &baij)); >> - if (!baij) { >> - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISBAIJSetPreallocation_C", &sbaij)); >> - if (!sbaij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSBAIJSetPreallocation_C", &sbaij)); >> - if (!sbaij) { >> - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISELLSetPreallocation_C", &sell)); >> - if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSELLSetPreallocation_C", &sell)); >> + if (!dm->prealloc_skip) { // Flag is likely set when user intends to use MatSetPreallocationCOO() >> + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIAIJSetPreallocation_C", &aij)); >> + if (!aij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqAIJSetPreallocation_C", &aij)); >> + if (!aij) { >> + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIBAIJSetPreallocation_C", &baij)); >> + if (!baij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqBAIJSetPreallocation_C", &baij)); >> + if (!baij) { >> + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISBAIJSetPreallocation_C", &sbaij)); >> + if (!sbaij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSBAIJSetPreallocation_C", &sbaij)); >> + if (!sbaij) { >> + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISELLSetPreallocation_C", &sell)); >> + if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSELLSetPreallocation_C", &sell)); >> + } >> + if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatISSetPreallocation_C", &is)); >> } >> - if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatISSetPreallocation_C", &is)); >> } >> } >> if (aij) { >> >> >> "Fackler, Philip via petsc-users" <pet...@mc...> writes: >> >>> I'm using the following sequence of functions related to the Jacobian matrix: >>> >>> DMDACreate1d(..., &da); >>> DMSetFromOptions(da); >>> DMSetUp(da); >>> DMSetMatType(da, MATAIJKOKKOS); >>> DMSetMatrixPreallocateSkip(da, PETSC_TRUE); >>> Mat J; >>> DMCreateMatrix(da, &J); >>> MatSetPreallocationCOO(J, ...); >>> >>> I recently added the call to DMSetMatrixPreallocateSkip, hoping the allocation would be delayed to MatSetPreallocationCOO, and that it would require less memory. The documentation<https://urldefense.us/v2/url?u=https-3A__petsc.org_release_manualpages_DM_DMSetMatrixPreallocateSkip_&d=DwIBAg&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=v9sHqomCGBRWotign4NcwYwOpszOJehUGs_EO3eGn4SSZqxnfK7Iv15-X8nO1lii&s=IMLBs0ydxDPvuXeD6jmsq1BN_8oieHVSyG6VA9c0DyM&e= > says that the data structures will not be preallocated. The following data from heaptrack shows that the allocation is still happening in the call to DMCreateMatrix. >>> >>> [cid:bda9ef12-a46f-47b2-9b9b-a4b2808b6b13] >>> >>> Can someone help me understand this? >>> >>> Thanks, >>> >>> Philip Fackler >>> Research Software Engineer, Application Engineering Group >>> Advanced Computing Systems Research Section >>> Computer Science and Mathematics Division >>> Oak Ridge National Laboratory |