From: Jed B. <je...@59...> - 2010-08-31 20:02:22
|
On Tue, 31 Aug 2010 13:24:44 -0600, Derek Gaston <der...@in...> wrote: > On Aug 31, 2010, at 1:11 PM, Jed Brown wrote: > > > If a VecGhost was passed in to SNES, then all the Krylov vectors > > will also be VecGhost. If not, create a new VecGhost > > (VecCreateGhost() or VecDuplicate() an existing VecGhost) and > > VecCopy from the plain Vec to the ghost Vec. > > Well, that is odd then. Because I'm sure that libMesh is passing in > ghosted vectors (hmmm... maybe the residual (rhs) isn't ghosted by > default?). Maybe libMesh is failing at detecting that they are > ghosted? It looks like libMesh is looking for a local to global > mapping to decide if it's ghosted or not. Is that the proper way? Not really, you can set a LocalToGlobalMapping even for unghosted, in order to use VecSetValuesLocal, though libmesh itself may never use this functionality. Looking at the source, there isn't a public way to check if a Vec has a local form. This could be added, but I'm not convinced it's necessary (or would be good design to use). If you really need it, I can think of two solutions: /* Nasty because this is only available when the source tree is around. */ #include <../src/vec/vec/impls/mpi/pvecimpl.h> if (vec->localrep) { /* is ghosted */ } OR PetscPushErrorHandler(PetscReturnErrorHandler,0); ierr = VecGhostGetLocalForm(X,&loc); PetscPopErrorHandler(); if (!err) { /* is ghosted */ } Note that all serial Vecs are trivially ghosted (the updates are no-ops). > Jed: how much overhead will be incurred by having all vectors in my > SNES system be ghosted? Will it be a noticeable slowdown or > communication? No extra communication, the only price is the extra memory for the ghosted part that isn't being used. That could be significant for very small or poorly shaped subdomains. PETSc could and maybe should deal with this by creating the Krylov space with non-ghosted vectors, but we currently always use vanilla VecDuplicate() so that the vector operations will be done with the appropriate types (e.g. VecCUDA). > Yeah; the real trouble is that ghosted vectors have to fall back to > serial when ghosting isn't available, e.g. on Trilinos. I'm going to > assume that you don't want to implement ghosted support there? I assume you mean working with a separate vector for the local form. > Otherwise the only thing to do is add a non-default code path, where > instead of letting PETSc work on solution while we work to keep > current_local_solution sync'ed up we do the opposite. PETSc is always going to work with a parallel vector (VecGhost is parallel). The different code path is that with VecGhost, you would normally VecGhostUpdate then VecGhostGetLocalForm instead of VecScatter to an independent local vector. Are you talking about this or something else? Jed |