From: STEPHANE TCHOUANMO <tchouanm@ms...>  20090303 20:15:28

Oups! Seems like you've got it Ben. Here is my Konsole output with info for a small case (9261 dofs). [0] PetscInitialize(): PETSc successfully started: number of processors = 1 [0] PetscGetHostName(): Rejecting domainname, likely is NIS linuxstchouan. [0] PetscInitialize(): Running on machine: linuxstchouan Mesh Information: mesh_dimension()=3 spatial_dimension()=3 n_nodes()=9261 n_elem()=8000 n_local_elem()=8000 n_active_elem()=8000 n_subdomains()=1 n_processors()=1 processor_id()=0 ... [0] VecScatterCreate(): Special case: sequential vector general scatter EquationSystems n_systems()=1 System "dc" Type "TransientNonlinearImplicit" Variables="P" Finite Element Types="LAGRANGE" Approximation Orders="FIRST" n_dofs()=9261 n_local_dofs()=9261 n_constrained_dofs()=0 n_vectors()=3 ==> Solving time step 0, time = 0.01 ... [0] VecScatterCreate(): Special case: sequential vector general scatter NL step 0, residual_2 = 4.119734e05 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 2080374783 [0] PetscCommDuplicate(): returning tag 2147483632 [0] PetscCommDuplicate(): Using internal PETSc communicator 2080374784 2080374782 [0] PetscCommDuplicate(): returning tag 2147483641 [0] PetscCommDuplicate(): returning tag 2147483631 [0] VecScatterCreate(): Special case: sequential vector general scatter [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 9261 X 9261; storage space: 56570 unneeded,226981 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 18286 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 27 [0] Mat_CheckInode(): Found 9261 nodes out of 9261 rows. Not using Inode routines [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 9261 X 9261; storage space: 0 unneeded,226981 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 27 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 9261 X 9261; storage space: 0 unneeded,226981 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 27 So I have: Number of mallocs during MatSetValues() is 18286 Its only non zero at this point. The rest of the time its zero. The value probably increases exponentially with the number of unknowns in the system. It also seems to depend on the type of elements I use. Here I have Hexes but I will further work with tetras. What should I do to fix this Ben? Thanks a lot. Stephane > From: benjamin.kirk1@... > To: tchouanm@... > CC: libmeshusers@... > Date: Tue, 3 Mar 2009 13:42:24 0600 > Subject: Re: [Libmeshusers] Petsc slowness on a single processor machine > > > Thanks Ben. > > > > I use mpich21.0.7. > > > > After a discussion with PETSc developpers, the problem might come from lots of > > allocation made by LibMesh within the call of PETSc. In fact if you look at > > the PETSc log summary of the problem I solve, you can clearly see that most of > > the time (more than 90%) is spent in the SNESSolve stage. The KSPSolve stage > > for solving the linear system in Newton takes at most 5% of the time. > > Actually, my problem is really at the very first Newton iteration which can > > last a hour out of a 3 hours total time resolution. Here is the behavior i > > have: > > > > ==> Solving time step 0, time = 0.01 > > NL step 0, residual_2 = 5.346581e05 > > .. 1 hour .. > > NL step 1, residual_2 = 8.790777e10 > > > > ==> Solving time step 1, time = 2.000000e02 > > NL step 0, residual_2 = 6.043076e05 > > NL step 1, residual_2 = 9.936468e10 > > ... > > ... > > etc. until the end for a total CPU of 3 hours. > > Finally I always get the right solution but i dont understand the sudden stop > > at the beginning. > > It might not be only VecScatterCreate but i think its a whole bunch of memory > > allocation that happens. > > > > What do you think? > > > I think the problem is most definitely in the sparse matrix allocation. > libMesh builds the graph of (what it thinks is) your sparse matrix so that > the underlying PETSc data structures can be allocated perfectly. If for > some reason the linear system you are assembling has a different structure > than what we thought it would, insertions into the sparse matrix can be > horrifically slow the first time you assemble the linear system. > > what you should look for is something like 'number of mallocs called during > MatSetValues' when you run with info. we want that to be 0. what is it on > the first linear solve? what type of elements are you using?? > > Ben > _________________________________________________________________ More than messages–check out the rest of the Windows Live™. http://www.microsoft.com/windows/windowslive/ 