On Fri, 8 Feb 2008, Yujie wrote:
> "get" the global matrix  compute it: such as, I output the matrices
> at individual computers and directly rearrange them to a whole matrix, which
> is same with the global matrix assembled in a single processor?
> I think it is not same, because the matrices at individual computers should
> be processed regarding the boundary (should be some discretizated points in
> the mesh) between computers.
Okay. The answer is "yes" and "no". The global matrix is the same
whether assembled in parallel or in serial, in the sense that the
entry corresponding to any two specified degrees of freedom will be
the same. In other words, if you've got node A at (0.5,0.5) and node
B at (0.5,0.625), then the matrix entry Mij, at row i corresponding to
variable V on node A and row j corresponding to variable V on node B,
will be the same no matter whether you compute it in serial or in
parallel. (except for some minor differences in floating point error
from computations done in different orders)
However, the matrix will be different in parallel, because "i" and "j"
will be different. libMesh assigns degree of freedom numberings
differently depending on how your mesh is partitioned, so the matrix
you get in parallel is actually a permutation of the matrix you get in
serial (or of the matrix you get in parallel on a different number of
processors).
> In Ben's thesis (page 38), he mentioned that there are three steps to do for
> parallelly assembling matrix,
> 1. Synchronize data with remote processors. This is required so that any
> remote data
> needed in the element matrix assembly (required in nonlinear applications,
> for example)
> is available on the local processor.
> 2. Perform a loop over the active elements on the local processor. Compute
> the element
> matrix and distribute it into the global matrix.
> 3. Communicate local element contributions to degrees of freedom owned by
> remote
> processors.
> Second step is done by user code. I can't find how to realize the first and
> third steps in libmesh. Is third step done by PETScMatrix::add_matrix()?
The first step is done by System::update(), which uses
NumericVector::localize() to put the relevant parts of the remote
processors' solution data into the local procesor's
current_local_solution vector.
The third step is started by SparseMatrix::add_matrix() (called by
user code) and finished by SparseMatrix::close() (called by the
library).
> However, how to deal with the element or points in the mesh is on boundary
> between processors if this function is used?
The current_local_solution vector keeps copies of the "ghost" degrees
of freedom which are owned by remote processors but which have support
on elements touching the boundary of the local processor.

Roy
