Screenshot instructions:
Windows
Mac
Red Hat Linux
Ubuntu
Click URL instructions:
Rightclick on ad, choose "Copy Link", then paste here →
(This may not be possible with some types of ads)
From: Ataollah Mesgarnejad <amesga1@ti...>  20130610 14:42:15
Attachments:
text/plain

Dear all, I've been doing some scaling tests on my code. When I look at time (or % of time) spent at each stage in libMesh log I see that the enforce_constraints_exactly stage in DofMap is scaling very bad. I was wondering if anyone can comment. Here is my EquationSystems.print_info(): EquationSystems n_systems()=2 System #0, "elasticity_system" Type "TransientLinearImplicit" Variables={ "u" "v" } Finite Element Types="LAGRANGE", "JACOBI_20_00" Infinite Element Mapping="CARTESIAN" Approximation Orders="FIRST", "THIRD" n_dofs()=48660 n_local_dofs()=930 n_constrained_dofs()=1048 n_local_constrained_dofs()=56 n_vectors()=3 n_matrices()=1 DofMap Sparsity Average OnProcessor Bandwidth <= 13.6478 Average OffProcessor Bandwidth <= 0.904233 Maximum OnProcessor Bandwidth <= 20 Maximum OffProcessor Bandwidth <= 16 DofMap Constraints Number of DoF Constraints = 1048 Average DoF Constraint Length= 0 Number of Node Constraints = 0 System #1, "fracture_system" Type "TransientNonlinearImplicit" Variables="psi" Finite Element Types="LAGRANGE", "JACOBI_20_00" Infinite Element Mapping="CARTESIAN" Approximation Orders="FIRST", "THIRD" n_dofs()=24330 n_local_dofs()=465 n_constrained_dofs()=167 n_local_constrained_dofs()=0 n_vectors()=3 n_matrices()=1 DofMap Sparsity Average OnProcessor Bandwidth <= 6.82388 Average OffProcessor Bandwidth <= 0.452117 Maximum OnProcessor Bandwidth <= 10 Maximum OffProcessor Bandwidth <= 8 DofMap Constraints Number of DoF Constraints = 167 Average DoF Constraint Length= 0 Number of Node Constraints = 0 and here is the how scaling looks for every stage that took > 1% of time: 
From: Cody Permann <codypermann@gm...>  20130610 14:48:45

Ata, You might be scaling past the reasonable limit for libMesh. I don't know what solver you are using, but for a strong scaling study we generally don't go below 10,000 local DOFs. This is the recommended floor for PETSc too: http://www.mcs.anl.gov/petsc/documentation/faq.html#slowerparallel Before you start drawing conclusions about scaling, you might start with a bigger problem and see if it scales well to the ~20,000 local DOF range. Cody On Mon, Jun 10, 2013 at 8:42 AM, Ataollah Mesgarnejad < amesga1@...> wrote: > Dear all, > > I've been doing some scaling tests on my code. When I look at time (or % > of time) spent at each stage in libMesh log I see that the > enforce_constraints_exactly stage in DofMap is scaling very bad. I was > wondering if anyone can comment. > > Here is my EquationSystems.print_info(): > > EquationSystems > n_systems()=2 > System #0, "elasticity_system" > Type "TransientLinearImplicit" > Variables={ "u" "v" } > Finite Element Types="LAGRANGE", "JACOBI_20_00" > Infinite Element Mapping="CARTESIAN" > Approximation Orders="FIRST", "THIRD" > n_dofs()=48660 > n_local_dofs()=930 > n_constrained_dofs()=1048 > n_local_constrained_dofs()=56 > n_vectors()=3 > n_matrices()=1 > DofMap Sparsity > Average OnProcessor Bandwidth <= 13.6478 > Average OffProcessor Bandwidth <= 0.904233 > Maximum OnProcessor Bandwidth <= 20 > Maximum OffProcessor Bandwidth <= 16 > DofMap Constraints > Number of DoF Constraints = 1048 > Average DoF Constraint Length= 0 > Number of Node Constraints = 0 > System #1, "fracture_system" > Type "TransientNonlinearImplicit" > Variables="psi" > Finite Element Types="LAGRANGE", "JACOBI_20_00" > Infinite Element Mapping="CARTESIAN" > Approximation Orders="FIRST", "THIRD" > n_dofs()=24330 > n_local_dofs()=465 > n_constrained_dofs()=167 > n_local_constrained_dofs()=0 > n_vectors()=3 > n_matrices()=1 > DofMap Sparsity > Average OnProcessor Bandwidth <= 6.82388 > Average OffProcessor Bandwidth <= 0.452117 > Maximum OnProcessor Bandwidth <= 10 > Maximum OffProcessor Bandwidth <= 8 > DofMap Constraints > Number of DoF Constraints = 167 > Average DoF Constraint Length= 0 > Number of Node Constraints = 0 > > > and here is the how scaling looks for every stage that took > 1% of time: > > > > Best, > Ata > >  > How ServiceNow helps IT people transform IT departments: > 1. A cloud service to automate IT design, transition and operations > 2. Dashboards that offer highlevel views of enterprise services > 3. A single system of record for all IT processes > http://p.sf.net/sfu/servicenowd2dj > _______________________________________________ > Libmeshusers mailing list > Libmeshusers@... > https://lists.sourceforge.net/lists/listinfo/libmeshusers > > 
From: Ataollah Mesgarnejad <amesga1@ti...>  20130610 15:16:21

Cody, I'm not sure if you saw the graph I uploaded it again here: https://dl.dropboxusercontent.com/u/19391830/scaling.jpg. In all these runs the NDOFs/Processor is less than 10000. What is bothering me is that the enforce_constraints_exactly is taking up more and more time as number of processor grows for the same problem. Now I can think of an explanation that the NDOFs/Processor is so low that the communication time is becoming a problem! That said my main concern is that I'm using the DirichletBCs API badly and that results in bad scaling. PS: Sorry for multiple copies I forgot to CC libMesh user list. Best, Ata On Jun 10, 2013, at 9:48 AM, Cody Permann <codypermann@...> wrote: > Ata, > > You might be scaling past the reasonable limit for libMesh. I don't know what solver you are using, but for a strong scaling study we generally don't go below 10,000 local DOFs. This is the recommended floor for PETSc too: > http://www.mcs.anl.gov/petsc/documentation/faq.html#slowerparallel > > Before you start drawing conclusions about scaling, you might start with a bigger problem and see if it scales well to the ~20,000 local DOF range. > > Cody > > > > On Mon, Jun 10, 2013 at 8:42 AM, Ataollah Mesgarnejad <amesga1@...> wrote: > Dear all, > > I've been doing some scaling tests on my code. When I look at time (or % of time) spent at each stage in libMesh log I see that the enforce_constraints_exactly stage in DofMap is scaling very bad. I was wondering if anyone can comment. > > Here is my EquationSystems.print_info(): > > EquationSystems > n_systems()=2 > System #0, "elasticity_system" > Type "TransientLinearImplicit" > Variables={ "u" "v" } > Finite Element Types="LAGRANGE", "JACOBI_20_00" > Infinite Element Mapping="CARTESIAN" > Approximation Orders="FIRST", "THIRD" > n_dofs()=48660 > n_local_dofs()=930 > n_constrained_dofs()=1048 > n_local_constrained_dofs()=56 > n_vectors()=3 > n_matrices()=1 > DofMap Sparsity > Average OnProcessor Bandwidth <= 13.6478 > Average OffProcessor Bandwidth <= 0.904233 > Maximum OnProcessor Bandwidth <= 20 > Maximum OffProcessor Bandwidth <= 16 > DofMap Constraints > Number of DoF Constraints = 1048 > Average DoF Constraint Length= 0 > Number of Node Constraints = 0 > System #1, "fracture_system" > Type "TransientNonlinearImplicit" > Variables="psi" > Finite Element Types="LAGRANGE", "JACOBI_20_00" > Infinite Element Mapping="CARTESIAN" > Approximation Orders="FIRST", "THIRD" > n_dofs()=24330 > n_local_dofs()=465 > n_constrained_dofs()=167 > n_local_constrained_dofs()=0 > n_vectors()=3 > n_matrices()=1 > DofMap Sparsity > Average OnProcessor Bandwidth <= 6.82388 > Average OffProcessor Bandwidth <= 0.452117 > Maximum OnProcessor Bandwidth <= 10 > Maximum OffProcessor Bandwidth <= 8 > DofMap Constraints > Number of DoF Constraints = 167 > Average DoF Constraint Length= 0 > Number of Node Constraints = 0 > > > and here is the how scaling looks for every stage that took > 1% of time: > > > > Best, > Ata >  > How ServiceNow helps IT people transform IT departments: > 1. A cloud service to automate IT design, transition and operations > 2. Dashboards that offer highlevel views of enterprise services > 3. A single system of record for all IT processes > http://p.sf.net/sfu/servicenowd2dj > _______________________________________________ > Libmeshusers mailing list > Libmeshusers@... > https://lists.sourceforge.net/lists/listinfo/libmeshusers > > 
From: Dmitry Karpeyev <karpeev@mc...>  20130610 15:51:35

It may be a good idea to combine this study with looking at the output of log_summary (assuming PETSc is your solver engine) to see if there is a correlation with the growth in communication or other components of matrix assembly. Dmitry. On Mon, Jun 10, 2013 at 10:16 AM, Ataollah Mesgarnejad <amesga1@...> wrote: > Cody, > > I'm not sure if you saw the graph I uploaded it again here: https://dl.dropboxusercontent.com/u/19391830/scaling.jpg. > > In all these runs the NDOFs/Processor is less than 10000. What is bothering me is that the enforce_constraints_exactly is taking up more and more time as number of processor grows for the same problem. > > Now I can think of an explanation that the NDOFs/Processor is so low that the communication time is becoming a problem! That said my main concern is that I'm using the DirichletBCs API badly and that results in bad scaling. > > PS: Sorry for multiple copies I forgot to CC libMesh user list. > > Best, > Ata > On Jun 10, 2013, at 9:48 AM, Cody Permann <codypermann@...> wrote: > >> Ata, >> >> You might be scaling past the reasonable limit for libMesh. I don't know what solver you are using, but for a strong scaling study we generally don't go below 10,000 local DOFs. This is the recommended floor for PETSc too: >> http://www.mcs.anl.gov/petsc/documentation/faq.html#slowerparallel >> >> Before you start drawing conclusions about scaling, you might start with a bigger problem and see if it scales well to the ~20,000 local DOF range. >> >> Cody >> >> >> >> On Mon, Jun 10, 2013 at 8:42 AM, Ataollah Mesgarnejad <amesga1@...> wrote: >> Dear all, >> >> I've been doing some scaling tests on my code. When I look at time (or % of time) spent at each stage in libMesh log I see that the enforce_constraints_exactly stage in DofMap is scaling very bad. I was wondering if anyone can comment. >> >> Here is my EquationSystems.print_info(): >> >> EquationSystems >> n_systems()=2 >> System #0, "elasticity_system" >> Type "TransientLinearImplicit" >> Variables={ "u" "v" } >> Finite Element Types="LAGRANGE", "JACOBI_20_00" >> Infinite Element Mapping="CARTESIAN" >> Approximation Orders="FIRST", "THIRD" >> n_dofs()=48660 >> n_local_dofs()=930 >> n_constrained_dofs()=1048 >> n_local_constrained_dofs()=56 >> n_vectors()=3 >> n_matrices()=1 >> DofMap Sparsity >> Average OnProcessor Bandwidth <= 13.6478 >> Average OffProcessor Bandwidth <= 0.904233 >> Maximum OnProcessor Bandwidth <= 20 >> Maximum OffProcessor Bandwidth <= 16 >> DofMap Constraints >> Number of DoF Constraints = 1048 >> Average DoF Constraint Length= 0 >> Number of Node Constraints = 0 >> System #1, "fracture_system" >> Type "TransientNonlinearImplicit" >> Variables="psi" >> Finite Element Types="LAGRANGE", "JACOBI_20_00" >> Infinite Element Mapping="CARTESIAN" >> Approximation Orders="FIRST", "THIRD" >> n_dofs()=24330 >> n_local_dofs()=465 >> n_constrained_dofs()=167 >> n_local_constrained_dofs()=0 >> n_vectors()=3 >> n_matrices()=1 >> DofMap Sparsity >> Average OnProcessor Bandwidth <= 6.82388 >> Average OffProcessor Bandwidth <= 0.452117 >> Maximum OnProcessor Bandwidth <= 10 >> Maximum OffProcessor Bandwidth <= 8 >> DofMap Constraints >> Number of DoF Constraints = 167 >> Average DoF Constraint Length= 0 >> Number of Node Constraints = 0 >> >> >> and here is the how scaling looks for every stage that took > 1% of time: >> >> >> >> Best, >> Ata >>  >> How ServiceNow helps IT people transform IT departments: >> 1. A cloud service to automate IT design, transition and operations >> 2. Dashboards that offer highlevel views of enterprise services >> 3. A single system of record for all IT processes >> http://p.sf.net/sfu/servicenowd2dj >> _______________________________________________ >> Libmeshusers mailing list >> Libmeshusers@... >> https://lists.sourceforge.net/lists/listinfo/libmeshusers >> >> > >  > How ServiceNow helps IT people transform IT departments: > 1. A cloud service to automate IT design, transition and operations > 2. Dashboards that offer highlevel views of enterprise services > 3. A single system of record for all IT processes > http://p.sf.net/sfu/servicenowd2dj > _______________________________________________ > Libmeshusers mailing list > Libmeshusers@... > https://lists.sourceforge.net/lists/listinfo/libmeshusers 