You can subscribe to this list here.
2003 
_{Jan}

_{Feb}

_{Mar}

_{Apr}

_{May}

_{Jun}

_{Jul}

_{Aug}

_{Sep}
(2) 
_{Oct}
(2) 
_{Nov}
(27) 
_{Dec}
(31) 

2004 
_{Jan}
(6) 
_{Feb}
(15) 
_{Mar}
(33) 
_{Apr}
(10) 
_{May}
(46) 
_{Jun}
(11) 
_{Jul}
(21) 
_{Aug}
(15) 
_{Sep}
(13) 
_{Oct}
(23) 
_{Nov}
(1) 
_{Dec}
(8) 
2005 
_{Jan}
(27) 
_{Feb}
(57) 
_{Mar}
(86) 
_{Apr}
(23) 
_{May}
(37) 
_{Jun}
(34) 
_{Jul}
(24) 
_{Aug}
(17) 
_{Sep}
(50) 
_{Oct}
(24) 
_{Nov}
(10) 
_{Dec}
(60) 
2006 
_{Jan}
(47) 
_{Feb}
(46) 
_{Mar}
(127) 
_{Apr}
(19) 
_{May}
(26) 
_{Jun}
(62) 
_{Jul}
(47) 
_{Aug}
(51) 
_{Sep}
(61) 
_{Oct}
(42) 
_{Nov}
(50) 
_{Dec}
(33) 
2007 
_{Jan}
(60) 
_{Feb}
(55) 
_{Mar}
(77) 
_{Apr}
(102) 
_{May}
(82) 
_{Jun}
(102) 
_{Jul}
(169) 
_{Aug}
(117) 
_{Sep}
(80) 
_{Oct}
(37) 
_{Nov}
(51) 
_{Dec}
(43) 
2008 
_{Jan}
(71) 
_{Feb}
(94) 
_{Mar}
(98) 
_{Apr}
(125) 
_{May}
(54) 
_{Jun}
(119) 
_{Jul}
(60) 
_{Aug}
(111) 
_{Sep}
(118) 
_{Oct}
(125) 
_{Nov}
(119) 
_{Dec}
(94) 
2009 
_{Jan}
(109) 
_{Feb}
(38) 
_{Mar}
(93) 
_{Apr}
(88) 
_{May}
(29) 
_{Jun}
(57) 
_{Jul}
(53) 
_{Aug}
(48) 
_{Sep}
(68) 
_{Oct}
(151) 
_{Nov}
(23) 
_{Dec}
(35) 
2010 
_{Jan}
(84) 
_{Feb}
(60) 
_{Mar}
(184) 
_{Apr}
(112) 
_{May}
(60) 
_{Jun}
(90) 
_{Jul}
(23) 
_{Aug}
(70) 
_{Sep}
(119) 
_{Oct}
(27) 
_{Nov}
(47) 
_{Dec}
(54) 
2011 
_{Jan}
(22) 
_{Feb}
(19) 
_{Mar}
(92) 
_{Apr}
(93) 
_{May}
(35) 
_{Jun}
(91) 
_{Jul}
(32) 
_{Aug}
(61) 
_{Sep}
(7) 
_{Oct}
(69) 
_{Nov}
(81) 
_{Dec}
(23) 
2012 
_{Jan}
(64) 
_{Feb}
(95) 
_{Mar}
(35) 
_{Apr}
(36) 
_{May}
(63) 
_{Jun}
(98) 
_{Jul}
(70) 
_{Aug}
(171) 
_{Sep}
(149) 
_{Oct}
(64) 
_{Nov}
(67) 
_{Dec}
(126) 
2013 
_{Jan}
(108) 
_{Feb}
(104) 
_{Mar}
(171) 
_{Apr}
(133) 
_{May}
(108) 
_{Jun}
(100) 
_{Jul}
(93) 
_{Aug}
(126) 
_{Sep}
(74) 
_{Oct}
(59) 
_{Nov}
(145) 
_{Dec}
(93) 
2014 
_{Jan}
(38) 
_{Feb}
(45) 
_{Mar}
(26) 
_{Apr}
(41) 
_{May}
(125) 
_{Jun}
(70) 
_{Jul}
(61) 
_{Aug}
(66) 
_{Sep}
(60) 
_{Oct}
(110) 
_{Nov}
(27) 
_{Dec}
(30) 
2015 
_{Jan}
(43) 
_{Feb}
(67) 
_{Mar}
(71) 
_{Apr}
(92) 
_{May}
(39) 
_{Jun}
(15) 
_{Jul}
(46) 
_{Aug}
(63) 
_{Sep}
(84) 
_{Oct}
(82) 
_{Nov}
(69) 
_{Dec}
(45) 
2016 
_{Jan}
(92) 
_{Feb}
(91) 
_{Mar}
(148) 
_{Apr}
(43) 
_{May}
(58) 
_{Jun}
(117) 
_{Jul}
(92) 
_{Aug}
(140) 
_{Sep}
(49) 
_{Oct}
(33) 
_{Nov}
(85) 
_{Dec}
(40) 
2017 
_{Jan}
(41) 
_{Feb}
(36) 
_{Mar}
(49) 
_{Apr}
(41) 
_{May}
(73) 
_{Jun}
(51) 
_{Jul}
(12) 
_{Aug}
(69) 
_{Sep}
(26) 
_{Oct}
(35) 
_{Nov}

_{Dec}

S  M  T  W  T  F  S 




1

2

3
(4) 
4

5

6

7

8
(8) 
9
(3) 
10

11
(2) 
12

13

14
(1) 
15
(2) 
16
(3) 
17
(4) 
18

19

20
(2) 
21

22
(2) 
23

24

25
(1) 
26

27
(2) 
28

29
(1) 
30

31
(3) 

From: Dafang Wang <dafang.wang@jh...>  20140109 04:29:52

Hi folks, Here are my replies to some of your questions. I used the libraries in the following way (via CMake): Debug mode: my code (g), libmesh (dbg mode), PetsC (optimized mode). RelWithDebInfo mode: my code (O2 g), libmesh (devel mode), PetsC (optimized mode) To Jed: my use of NDEBUG concerns my code only, not LibMesh. To Benjamin: my user code made multiple calls to standard c++ assert(), but no calls to libmesh_assert(). To Roy: can you please tell me how to apply DNDEBUG to libMesh alone (without affecting my user code)? Thanks for all of your comments. Cheers, Dafang On 01/08/2014 05:28 PM, Roy Stogner wrote: > > On Wed, 8 Jan 2014, Dafang Wang wrote: > >> I figured out the cause of the problem: DNDEBUG in CXXFLAGS. With >> this flag on, the program returned bad convergence results both on my >> desktop and on the cluster. Without this flag, both machines yielded >> good results. So this problem is not to the intel compiler on the >> cluster. > > This is pretty astonishing. Can you get the bad convergence results > with DNDEBUG applied to libMesh alone? If so then we've probably got > a bug worth looking into somewhere. > > I can't imagine where, though. We just don't have a lot of places > where there's a significant difference between normal and NDEBUG code > paths. >  > Roy  Dafang Wang, Ph.D Postdoctoral Fellow Institute of Computational Medicine Department of Biomedical Engineering Johns Hopkins University Hackerman Hall Room 218 Baltimore, MD, 21218 
From: Roy Stogner <roystgnr@ic...>  20140108 23:24:59

On Wed, 8 Jan 2014, John Peterson wrote: > Yep... someone will correct me but I think the *only* thing DNDEBUG does is compile out asserts? Effectively that's all it does, however: Grep for "NDEBUG" and you'll see a bunch of preprocessor tests. Some of our asserts require extra precomputation before the asserted statement, and we generally wrap that in "#ifndef NDEBUG" to avoid unnecessary computations in optimized code and overzealous warnings from compilers and static checkers.  Roy 
From: John Peterson <jwpeterson@gm...>  20140108 22:58:06

On Wed, Jan 8, 2014 at 3:42 PM, Kirk, Benjamin (JSCEG311) < benjamin.kirk@...> wrote: > > > On Jan 8, 2014, at 4:28 PM, "Roy Stogner" <roystgnr@...> > wrote: > > > > > > I can't imagine where, though. We just don't have a lot of places > > where there's a significant difference between normal and NDEBUG code > > paths. > > Does your user code make any calls to e.g. libmesh_assert(...); ? > > That's the only thing I could think of, a predicate inside an assert that > is incorrectly constructed and is perhaps an assignment or something > instead of just a check. > Yep... someone will correct me but I think the *only* thing DNDEBUG does is compile out asserts?  John 
From: Kirk, Benjamin (JSCEG311) <benjamin.kirk@na...>  20140108 22:43:03

> On Jan 8, 2014, at 4:28 PM, "Roy Stogner" <roystgnr@...> wrote: > > > I can't imagine where, though. We just don't have a lot of places > where there's a significant difference between normal and NDEBUG code > paths. Does your user code make any calls to e.g. libmesh_assert(...); ? That's the only thing I could think of, a predicate inside an assert that is incorrectly constructed and is perhaps an assignment or something instead of just a check. Ben 
From: Roy Stogner <roystgnr@ic...>  20140108 22:28:12

On Wed, 8 Jan 2014, Dafang Wang wrote: > I figured out the cause of the problem: DNDEBUG in CXXFLAGS. With this flag > on, the program returned bad convergence results both on my desktop and on > the cluster. Without this flag, both machines yielded good results. So this > problem is not to the intel compiler on the cluster. This is pretty astonishing. Can you get the bad convergence results with DNDEBUG applied to libMesh alone? If so then we've probably got a bug worth looking into somewhere. I can't imagine where, though. We just don't have a lot of places where there's a significant difference between normal and NDEBUG code paths.  Roy 
From: Dafang Wang <dafang.wang@jh...>  20140108 22:20:29

Hi Roy (and Jed), I figured out the cause of the problem: DNDEBUG in CXXFLAGS. With this flag on, the program returned bad convergence results both on my desktop and on the cluster. Without this flag, both machines yielded good results. So this problem is not to the intel compiler on the cluster. Adding "O2 fpmodel precise" and turning on "snes_linesearchmonitor", as you suggested, changed the convergence behavior slightly but not fundamentally. The intel compiler version I am using is "Compiler/11.0/083/". It is a quite old version. Many thanks to both of you for your helpful advice. Cheers, Dafang On 01/08/2014 03:57 PM, Roy Stogner wrote: > > On Wed, 8 Jan 2014, Dafang Wang wrote: > >> Thanks for your prompt reply. Did you mean turning O2 "on" or >> "off"? I got bad results when turning it on. > > I mean using "O2 fpmodel precise". IIRC that often gets icpc to > turn off its inexactinfloatingpoint optimizations without forcing > it to turn off any optimizations which don't change the final code > output. > > No guarantees, though. I dug into the compiler configuration files > > https://github.com/manufacturedsolutions/MASA/blob/master/m4/common/ax_cxx_minopt.m4 > > > from the manufactured solutions code where we encountered these > problems, to see which options we'd ended up needing, and it looks > like we eventually gave up on any optimizations at all when we > demanded maximum accuracy there. >  > Roy  Dafang Wang, Ph.D Postdoctoral Fellow Institute of Computational Medicine Department of Biomedical Engineering Johns Hopkins University Hackerman Hall Room 218 Baltimore, MD, 21218 
From: Roy Stogner <roystgnr@ic...>  20140108 20:57:34

On Wed, 8 Jan 2014, Dafang Wang wrote: > Thanks for your prompt reply. Did you mean turning O2 "on" or "off"? I got > bad results when turning it on. I mean using "O2 fpmodel precise". IIRC that often gets icpc to turn off its inexactinfloatingpoint optimizations without forcing it to turn off any optimizations which don't change the final code output. No guarantees, though. I dug into the compiler configuration files https://github.com/manufacturedsolutions/MASA/blob/master/m4/common/ax_cxx_minopt.m4 from the manufactured solutions code where we encountered these problems, to see which options we'd ended up needing, and it looks like we eventually gave up on any optimizations at all when we demanded maximum accuracy there.  Roy 
From: Roy Stogner <roystgnr@ic...>  20140108 20:49:40

On Wed, 8 Jan 2014, Dafang Wang wrote: > When running the libmesh (and Petsc) in the debug mode (flag= g), > both the desktop and the cluster returned the same convergence > results. However, when running libmesh in the RelWithDebugInfo mode > (flag= O2), the desktop yielded good convergence whereas the > cluster yielded bad convergence. > > The cluster uses a specific Intel compiler which is different from the > desktop. Turning off O2 can really hurt, but you might be able to get away with enabling most of Intel's optimizations while still not ruining your results: try adding "fpmodel precise" ?  Roy 
From: Dafang Wang <dafang.wang@jh...>  20140108 20:33:10

Hi Jed (and others), Here is an update of my previous problem about SNES convergence difference on different machines: The convergence difference seems to be due to the different compilation modes of Petsc. When running the libmesh (and Petsc) in the debug mode (flag= g), both the desktop and the cluster returned the same convergence results. However, when running libmesh in the RelWithDebugInfo mode (flag= O2), the desktop yielded good convergence whereas the cluster yielded bad convergence. The cluster uses a specific Intel compiler which is different from the desktop. In response to Jed's previous questions: 1. No worry for the seg fault problem, which has been resolved. 2. The initial state for the nonlinear system was an allzero solution vector. This problem seems to be relevant to Petsc rather than Libmesh. Any comments would be appreciated. I can provide further details about my simulation if needed. Below are the convergence results obtained with the flags "pc_type lu ksp_monitor ksp_converged_reason snes_monitor snes_converged_reason ".  Debug mode (same results on both machines): Desktop machine: NL step 0, residual_2 = 1.440000e+00 0 SNES Function norm 1.440000000000e+00 0 KSP Residual norm 4.021760102139e+00 1 KSP Residual norm 1.040484538968e14 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 1, residual_2 = 5.960576e01 1 SNES Function norm 5.960575795099e01 0 KSP Residual norm 4.727634711341e01 1 KSP Residual norm 1.169178369623e14 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 2, residual_2 = 3.745537e02 2 SNES Function norm 3.745536863104e02 0 KSP Residual norm 5.571738493521e02 1 KSP Residual norm 4.222606354657e16 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 3, residual_2 = 7.370172e04 3 SNES Function norm 7.370171737019e04 0 KSP Residual norm 6.418014965935e04 1 KSP Residual norm 2.132391208383e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 4, residual_2 = 1.433233e07 4 SNES Function norm 1.433233447799e07 0 KSP Residual norm 5.038509414466e07 1 KSP Residual norm 1.948639405146e21 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 5, residual_2 = 3.440465e13 5 SNES Function norm 3.440465235277e13 Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 5 Mechanics system solved at nonlinear iteration 5 , final nonlinear residual norm: 3.440465e13, converge reason= 3 Cluster: NL step 0, residual_2 = 1.440000e+00 0 SNES Function norm 1.440000000000e+00 0 KSP Residual norm 4.021760102139e+00 1 KSP Residual norm 2.379314305795e14 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 1, residual_2 = 5.960576e01 1 SNES Function norm 5.960575795099e01 0 KSP Residual norm 4.727634711341e01 1 KSP Residual norm 7.976234556294e15 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 2, residual_2 = 3.745537e02 2 SNES Function norm 3.745536863104e02 0 KSP Residual norm 5.571738493521e02 1 KSP Residual norm 2.844838895371e16 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 3, residual_2 = 7.370172e04 3 SNES Function norm 7.370171737018e04 0 KSP Residual norm 6.418014965934e04 1 KSP Residual norm 4.308781214538e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 4, residual_2 = 1.433233e07 4 SNES Function norm 1.433233452409e07 0 KSP Residual norm 5.038509410375e07 1 KSP Residual norm 2.281704035369e21 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 5, residual_2 = 3.436775e13 5 SNES Function norm 3.436774565070e13 Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 5 Mechanics system solved at nonlinear iteration 5 , final nonlinear residual norm: 3.436775e13, converge reason= 3  RelWIthDebugInfo mode (O2), different convergence behavior Desktop (good convergence, also same as the debug mode): NL step 0, residual_2 = 1.440000e+00 0 SNES Function norm 1.440000000000e+00 0 KSP Residual norm 4.021760102139e+00 1 KSP Residual norm 1.040484538968e14 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 1, residual_2 = 5.960576e01 1 SNES Function norm 5.960575795099e01 0 KSP Residual norm 4.727634711341e01 1 KSP Residual norm 1.169178369623e14 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 2, residual_2 = 3.745537e02 2 SNES Function norm 3.745536863104e02 0 KSP Residual norm 5.571738493521e02 1 KSP Residual norm 4.222606354657e16 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 3, residual_2 = 7.370172e04 3 SNES Function norm 7.370171737019e04 0 KSP Residual norm 6.418014965935e04 1 KSP Residual norm 2.132391208383e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 4, residual_2 = 1.433233e07 4 SNES Function norm 1.433233447799e07 0 KSP Residual norm 5.038509414466e07 1 KSP Residual norm 1.948639405146e21 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 5, residual_2 = 3.440465e13 5 SNES Function norm 3.440465235277e13 Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 5 Mechanics system solved at nonlinear iteration 5 , final nonlinear residual norm: 3.440465e13, converge reason= 3 Cluster (poor convergence, different from the debug mode): NL step 0, residual_2 = 1.440000e+00 0 SNES Function norm 1.440000000000e+00 0 KSP Residual norm 4.021760102139e+00 1 KSP Residual norm 2.222981076063e14 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 1, residual_2 = 5.960576e01 1 SNES Function norm 5.960575795099e01 0 KSP Residual norm 5.724516760789e01 1 KSP Residual norm 4.044030898145e14 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 2, residual_2 = 1.482067e01 2 SNES Function norm 1.482067440081e01 0 KSP Residual norm 1.899198487299e01 1 KSP Residual norm 2.132006073251e15 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 3, residual_2 = 7.270703e02 3 SNES Function norm 7.270702856431e02 0 KSP Residual norm 3.483657684658e02 1 KSP Residual norm 3.286054408774e16 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 4, residual_2 = 3.049751e03 4 SNES Function norm 3.049750785313e03 0 KSP Residual norm 4.812920494029e03 1 KSP Residual norm 5.700508014744e16 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 5, residual_2 = 4.513461e04 5 SNES Function norm 4.513461421076e04 0 KSP Residual norm 1.209655693065e03 1 KSP Residual norm 4.945249451450e16 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 6, residual_2 = 1.793522e04 6 SNES Function norm 1.793521679461e04 0 KSP Residual norm 7.011178005420e04 1 KSP Residual norm 8.758630689120e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 7, residual_2 = 9.131320e05 7 SNES Function norm 9.131320113186e05 0 KSP Residual norm 3.544244645037e04 1 KSP Residual norm 1.540261136238e16 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 8, residual_2 = 4.648251e05 8 SNES Function norm 4.648251104473e05 0 KSP Residual norm 2.790752923777e04 1 KSP Residual norm 1.079811215541e16 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 9, residual_2 = 2.515710e05 9 SNES Function norm 2.515709920202e05 0 KSP Residual norm 1.355995795589e02 1 KSP Residual norm 1.111094748172e16 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 10, residual_2 = 2.351012e05 10 SNES Function norm 2.351012455292e05 0 KSP Residual norm 4.715035865139e03 1 KSP Residual norm 5.621155808935e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 11, residual_2 = 2.248736e05 11 SNES Function norm 2.248736326877e05 0 KSP Residual norm 3.557797254429e03 1 KSP Residual norm 3.341418332138e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 12, residual_2 = 2.128736e05 12 SNES Function norm 2.128735852036e05 0 KSP Residual norm 7.703099954589e03 1 KSP Residual norm 1.154597763933e16 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 13, residual_2 = 2.048972e05 13 SNES Function norm 2.048971947471e05 0 KSP Residual norm 2.679644378167e03 1 KSP Residual norm 8.909261104450e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 14, residual_2 = 1.965486e05 14 SNES Function norm 1.965485620055e05 0 KSP Residual norm 1.479045966367e02 1 KSP Residual norm 5.645787321175e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 15, residual_2 = 1.737326e05 15 SNES Function norm 1.737325739134e05 0 KSP Residual norm 5.144625209331e03 1 KSP Residual norm 8.043824552860e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 16, residual_2 = 1.680721e05 16 SNES Function norm 1.680721071217e05 0 KSP Residual norm 1.409004856850e03 1 KSP Residual norm 1.055045286656e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 17, residual_2 = 1.505455e05 17 SNES Function norm 1.505455093030e05 0 KSP Residual norm 7.771120407446e03 1 KSP Residual norm 5.978364642141e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 18, residual_2 = 1.413325e05 18 SNES Function norm 1.413324859093e05 0 KSP Residual norm 2.703906031576e03 1 KSP Residual norm 3.156540186874e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 19, residual_2 = 1.349265e05 19 SNES Function norm 1.349264959991e05 0 KSP Residual norm 2.604619849008e03 1 KSP Residual norm 4.556754801716e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 20, residual_2 = 1.288406e05 20 SNES Function norm 1.288406277123e05 0 KSP Residual norm 2.408639307512e03 1 KSP Residual norm 1.574105435806e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 21, residual_2 = 1.228811e05 21 SNES Function norm 1.228810775661e05 0 KSP Residual norm 2.592267975097e03 1 KSP Residual norm 4.275624728417e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 22, residual_2 = 1.177481e05 22 SNES Function norm 1.177481128032e05 0 KSP Residual norm 1.447675648634e03 1 KSP Residual norm 1.769626897387e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 23, residual_2 = 1.115206e05 23 SNES Function norm 1.115206302877e05 0 KSP Residual norm 7.987558615343e03 1 KSP Residual norm 3.471776844754e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 24, residual_2 = 9.968437e06 24 SNES Function norm 9.968436832047e06 0 KSP Residual norm 2.779729958997e03 1 KSP Residual norm 2.539893189750e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 25, residual_2 = 9.631119e06 25 SNES Function norm 9.631119115197e06 0 KSP Residual norm 4.793844577865e04 1 KSP Residual norm 1.642814961535e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 26, residual_2 = 7.658449e06 26 SNES Function norm 7.658449202595e06 0 KSP Residual norm 5.463625248925e03 1 KSP Residual norm 4.389074959683e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 27, residual_2 = 6.852486e06 27 SNES Function norm 6.852485650682e06 0 KSP Residual norm 1.901774052606e03 1 KSP Residual norm 1.166130675804e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 28, residual_2 = 6.620653e06 28 SNES Function norm 6.620653139738e06 0 KSP Residual norm 3.141182445840e04 1 KSP Residual norm 6.065443405558e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 29, residual_2 = 5.171843e06 29 SNES Function norm 5.171843144827e06 0 KSP Residual norm 3.871258196172e03 1 KSP Residual norm 1.951195061396e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 30, residual_2 = 4.576905e06 30 SNES Function norm 4.576905178809e06 0 KSP Residual norm 1.347692935990e03 1 KSP Residual norm 1.446770176793e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 31, residual_2 = 4.427166e06 31 SNES Function norm 4.427166013787e06 0 KSP Residual norm 3.574992580610e04 1 KSP Residual norm 7.908091467650e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 32, residual_2 = 3.950633e06 32 SNES Function norm 3.950633118913e06 0 KSP Residual norm 1.970680556721e03 1 KSP Residual norm 1.088685998265e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 33, residual_2 = 3.721235e06 33 SNES Function norm 3.721234938692e06 0 KSP Residual norm 6.861687491055e04 1 KSP Residual norm 1.953135470491e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 34, residual_2 = 3.547135e06 34 SNES Function norm 3.547134826276e06 0 KSP Residual norm 7.856677852322e04 1 KSP Residual norm 1.330652439994e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 35, residual_2 = 3.404737e06 35 SNES Function norm 3.404737346554e06 0 KSP Residual norm 3.073815582292e04 1 KSP Residual norm 1.603901159353e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 36, residual_2 = 3.075132e06 36 SNES Function norm 3.075131676655e06 0 KSP Residual norm 1.694745606692e03 1 KSP Residual norm 1.791266224670e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 37, residual_2 = 2.866068e06 37 SNES Function norm 2.866068373248e06 0 KSP Residual norm 5.901013004031e04 1 KSP Residual norm 3.927275386298e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 38, residual_2 = 2.743693e06 38 SNES Function norm 2.743693466503e06 0 KSP Residual norm 3.838214494089e04 1 KSP Residual norm 9.894588963925e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 39, residual_2 = 2.673055e06 39 SNES Function norm 2.673055265273e06 0 KSP Residual norm 2.117297982903e03 1 KSP Residual norm 9.292131679050e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 40, residual_2 = 2.331440e06 40 SNES Function norm 2.331439782776e06 0 KSP Residual norm 7.371907291326e04 1 KSP Residual norm 8.470282280236e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 41, residual_2 = 2.256311e06 41 SNES Function norm 2.256310976814e06 0 KSP Residual norm 2.567895446571e04 1 KSP Residual norm 6.852266839368e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 42, residual_2 = 2.105991e06 42 SNES Function norm 2.105991050251e06 0 KSP Residual norm 1.416238836146e03 1 KSP Residual norm 1.550853158673e17 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 43, residual_2 = 1.906527e06 43 SNES Function norm 1.906526750558e06 0 KSP Residual norm 4.931289500004e04 1 KSP Residual norm 5.290867863556e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 44, residual_2 = 1.838924e06 44 SNES Function norm 1.838923996912e06 0 KSP Residual norm 1.405762049962e05 1 KSP Residual norm 4.585529421262e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 45, residual_2 = 1.058048e06 45 SNES Function norm 1.058047908625e06 0 KSP Residual norm 7.729636904681e04 1 KSP Residual norm 4.133047612862e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 46, residual_2 = 9.417074e07 46 SNES Function norm 9.417073985418e07 0 KSP Residual norm 2.691875835493e04 1 KSP Residual norm 1.023461541301e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 47, residual_2 = 9.104019e07 47 SNES Function norm 9.104018687209e07 0 KSP Residual norm 5.848764695778e05 1 KSP Residual norm 3.731574398649e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 48, residual_2 = 7.821364e07 48 SNES Function norm 7.821364289332e07 0 KSP Residual norm 4.227277266633e04 1 KSP Residual norm 4.822287553611e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 49, residual_2 = 7.306530e07 49 SNES Function norm 7.306529623165e07 0 KSP Residual norm 1.472269887921e04 1 KSP Residual norm 3.360940671559e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 50, residual_2 = 6.989741e07 50 SNES Function norm 6.989741368342e07 0 KSP Residual norm 1.079610402578e04 1 KSP Residual norm 6.085658602505e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 51, residual_2 = 6.986050e07 51 SNES Function norm 6.986049538262e07 0 KSP Residual norm 5.955220857101e04 1 KSP Residual norm 5.588488888499e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 52, residual_2 = 5.960243e07 52 SNES Function norm 5.960242923830e07 0 KSP Residual norm 2.073824147391e04 1 KSP Residual norm 5.058556742009e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 53, residual_2 = 5.746872e07 53 SNES Function norm 5.746872331107e07 0 KSP Residual norm 7.224510981546e05 1 KSP Residual norm 9.534294596381e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 54, residual_2 = 5.467005e07 54 SNES Function norm 5.467005265595e07 0 KSP Residual norm 3.984467805183e04 1 KSP Residual norm 1.600986090410e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 55, residual_2 = 4.868459e07 55 SNES Function norm 4.868459415833e07 0 KSP Residual norm 1.387564478005e04 1 KSP Residual norm 9.383394994032e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 56, residual_2 = 4.706315e07 56 SNES Function norm 4.706315384417e07 0 KSP Residual norm 2.944560147317e05 1 KSP Residual norm 6.638072600237e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 57, residual_2 = 4.016471e07 57 SNES Function norm 4.016470859394e07 0 KSP Residual norm 2.238135265392e04 1 KSP Residual norm 1.576103980007e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 58, residual_2 = 3.738898e07 58 SNES Function norm 3.738897675506e07 0 KSP Residual norm 7.794670768988e05 1 KSP Residual norm 5.237500072288e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 59, residual_2 = 3.581483e07 59 SNES Function norm 3.581482680037e07 0 KSP Residual norm 4.670929752150e05 1 KSP Residual norm 1.147265035210e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 60, residual_2 = 3.433438e07 60 SNES Function norm 3.433437845239e07 0 KSP Residual norm 2.576227607006e04 1 KSP Residual norm 2.498690560261e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 61, residual_2 = 3.037047e07 61 SNES Function norm 3.037046887097e07 0 KSP Residual norm 8.971382982151e05 1 KSP Residual norm 3.992428337482e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 62, residual_2 = 2.938123e07 62 SNES Function norm 2.938123021372e07 0 KSP Residual norm 2.434251616706e05 1 KSP Residual norm 6.288305083577e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 63, residual_2 = 2.629026e07 63 SNES Function norm 2.629025543286e07 0 KSP Residual norm 1.341713156961e04 1 KSP Residual norm 9.005362267699e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 64, residual_2 = 2.470770e07 64 SNES Function norm 2.470769796053e07 0 KSP Residual norm 4.672848087816e05 1 KSP Residual norm 5.806073812076e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 65, residual_2 = 2.357535e07 65 SNES Function norm 2.357534534358e07 0 KSP Residual norm 4.758611162337e05 1 KSP Residual norm 2.479409167491e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 66, residual_2 = 2.255490e07 66 SNES Function norm 2.255489784177e07 0 KSP Residual norm 3.455253521731e05 1 KSP Residual norm 1.848686476485e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 67, residual_2 = 2.249190e07 67 SNES Function norm 2.249189779176e07 0 KSP Residual norm 1.905971621337e04 1 KSP Residual norm 1.693538835814e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 68, residual_2 = 1.922701e07 68 SNES Function norm 1.922701107315e07 0 KSP Residual norm 6.637113796734e05 1 KSP Residual norm 5.714211105785e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 69, residual_2 = 1.854509e07 69 SNES Function norm 1.854508680080e07 0 KSP Residual norm 2.312103836137e05 1 KSP Residual norm 9.855648245687e20 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 70, residual_2 = 1.761273e07 70 SNES Function norm 1.761273273016e07 0 KSP Residual norm 1.275181139415e04 1 KSP Residual norm 4.506070339430e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 71, residual_2 = 1.570743e07 71 SNES Function norm 1.570743326797e07 0 KSP Residual norm 4.440640194736e05 1 KSP Residual norm 5.422068035420e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 72, residual_2 = 1.518161e07 72 SNES Function norm 1.518161153491e07 0 KSP Residual norm 8.797907000209e06 1 KSP Residual norm 2.075176146890e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 73, residual_2 = 1.268831e07 73 SNES Function norm 1.268830968047e07 0 KSP Residual norm 7.705514125563e05 1 KSP Residual norm 1.017793788404e18 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 74, residual_2 = 1.167839e07 74 SNES Function norm 1.167838585279e07 0 KSP Residual norm 2.683450054484e05 1 KSP Residual norm 5.132689633111e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 75, residual_2 = 1.122477e07 75 SNES Function norm 1.122477442296e07 0 KSP Residual norm 7.409290455259e06 1 KSP Residual norm 3.645143742976e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 76, residual_2 = 9.706797e08 76 SNES Function norm 9.706796620796e08 0 KSP Residual norm 5.082292041698e05 1 KSP Residual norm 3.544425465304e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 77, residual_2 = 9.099011e08 77 SNES Function norm 9.099010557425e08 0 KSP Residual norm 1.769998454096e05 1 KSP Residual norm 3.363150042480e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 78, residual_2 = 8.692366e08 78 SNES Function norm 8.692366224026e08 0 KSP Residual norm 1.566736777879e05 1 KSP Residual norm 2.819452990843e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 79, residual_2 = 8.276834e08 79 SNES Function norm 8.276834291448e08 0 KSP Residual norm 1.990319801652e05 1 KSP Residual norm 1.068580372799e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 80, residual_2 = 7.966571e08 80 SNES Function norm 7.966570700260e08 0 KSP Residual norm 3.003444562499e06 1 KSP Residual norm 5.206872525061e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 81, residual_2 = 5.608802e08 81 SNES Function norm 5.608802450435e08 0 KSP Residual norm 5.211286640517e05 1 KSP Residual norm 1.336954688010e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 82, residual_2 = 4.633277e08 82 SNES Function norm 4.633277025854e08 0 KSP Residual norm 1.814671884668e05 1 KSP Residual norm 1.945186873110e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 83, residual_2 = 4.441497e08 83 SNES Function norm 4.441496884398e08 0 KSP Residual norm 6.320882672742e06 1 KSP Residual norm 6.522712340090e20 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 84, residual_2 = 4.345014e08 84 SNES Function norm 4.345013744524e08 0 KSP Residual norm 3.486532415243e05 1 KSP Residual norm 2.820388278092e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 85, residual_2 = 3.776562e08 85 SNES Function norm 3.776561517098e08 0 KSP Residual norm 1.214102392542e05 1 KSP Residual norm 4.637953640500e20 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 86, residual_2 = 3.652905e08 86 SNES Function norm 3.652904667916e08 0 KSP Residual norm 4.229676380623e06 1 KSP Residual norm 2.985543935183e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 87, residual_2 = 3.421015e08 87 SNES Function norm 3.421015322140e08 0 KSP Residual norm 2.332617444021e05 1 KSP Residual norm 1.899596182975e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 88, residual_2 = 3.088663e08 88 SNES Function norm 3.088662974125e08 0 KSP Residual norm 8.123061571912e06 1 KSP Residual norm 9.730785565672e20 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 89, residual_2 = 2.980237e08 89 SNES Function norm 2.980236808814e08 0 KSP Residual norm 4.955116260996e07 1 KSP Residual norm 3.777642296031e20 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 90, residual_2 = 2.461180e08 90 SNES Function norm 2.461180426859e08 0 KSP Residual norm 3.089926846929e05 1 KSP Residual norm 1.152580814246e19 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 91, residual_2 = 1.641444e08 91 SNES Function norm 1.641444000238e08 0 KSP Residual norm 1.075937296980e05 1 KSP Residual norm 9.644931034537e20 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 92, residual_2 = 1.492579e08 92 SNES Function norm 1.492578941113e08 0 KSP Residual norm 3.746844578568e06 1 KSP Residual norm 2.240329961008e20 Linear solve converged due to CONVERGED_RTOL iterations 1 NL step 93, residual_2 = 1.438360e08 93 SNES Function norm 1.438360232831e08 Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 93 Mechanics system solved at nonlinear iteration 93 , final nonlinear residual norm: 1.438360e08, converge reason= 3 Thanks, Dafang On 01/03/2014 08:51 AM, Jed Brown wrote: > Dafang Wang <dafang.wang@...> writes: > >> Hi Jed, >> >> Below are the output details. (Results were obtained with snes_converged_reason, >> snes_monitor. The ksp_converged_reason ended up with seg fault, so was not included. It >> should not be the key problem though. ) > It should not SEGV. Can you get a stack trace? > >> Output from the desktop (good convergence rate): >> total constitutive equations = ngauss = 1000 >> EquationSystems >> n_systems()=1 >> System #0, "Elasticity" >> Type "NonlinearImplicit" >> Variables={ "ux" "uy" "uz" } >> Finite Element Types="LAGRANGE" >> Approximation Orders="FIRST" >> n_dofs()=648 >> n_local_dofs()=648 >> n_constrained_dofs()=108 >> n_local_constrained_dofs()=108 >> n_vectors()=1 >> n_matrices()=1 >> DofMap Sparsity >> Average OnProcessor Bandwidth <= 56.8889 >> Average OffProcessor Bandwidth <= 0 >> Maximum OnProcessor Bandwidth <= 81 >> Maximum OffProcessor Bandwidth <= 0 >> DofMap Constraints >> Number of DoF Constraints = 108 >> Average DoF Constraint Length= 0 >> >> *** Warning, This code is deprecated, and likely to be removed in future library versions! >> src/mesh/boundary_info.C, line 752, compiled Oct 29 2013 at 08:06:16 *** > What is this about? > >> NL step 0, residual_2 = 1.440000e+00 >> 0 SNES Function norm 1.440000000000e+00 >> NL step 1, residual_2 = 5.960479e01 >> 1 SNES Function norm 5.960478586457e01 >> NL step 2, residual_2 = 3.745465e02 >> 2 SNES Function norm 3.745464932748e02 >> NL step 3, residual_2 = 7.372288e04 >> 3 SNES Function norm 7.372287735879e04 >> NL step 4, residual_2 = 1.426607e07 >> 4 SNES Function norm 1.426607123296e07 >> NL step 5, residual_2 = 4.472564e12 >> 5 SNES Function norm 4.472564229796e12 >> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 5 >> Mechanics system solved at nonlinear iteration 5 , final nonlinear residual norm: >> 4.472564e12, converge reason= 3 >>  >> Output from the cluster (poor convergence rate): >> total constitutive equations = ngauss = 1000 >> EquationSystems >> n_systems()=1 >> System #0, "Elasticity" >> Type "NonlinearImplicit" >> Variables={ "ux" "uy" "uz" } >> Finite Element Types="LAGRANGE" >> Approximation Orders="FIRST" >> n_dofs()=648 >> n_local_dofs()=648 >> n_constrained_dofs()=108 >> n_local_constrained_dofs()=108 >> n_vectors()=1 >> n_matrices()=1 >> DofMap Sparsity >> Average OnProcessor Bandwidth <= 56.8889 >> Average OffProcessor Bandwidth <= 0 >> Maximum OnProcessor Bandwidth <= 81 >> Maximum OffProcessor Bandwidth <= 0 >> DofMap Constraints >> Number of DoF Constraints = 108 >> Average DoF Constraint Length= 0 >> >> *** Warning, This code is deprecated, and likely to be removed in future library versions! >> src/mesh/boundary_info.C, line 752, compiled Dec 19 2013 at 15:59:31 *** >> NL step 0, residual_2 = 1.440000e+00 >> 0 SNES Function norm 1.440000000000e+00 >> NL step 1, residual_2 = 5.960479e01 >> 1 SNES Function norm 5.960478586457e01 >> NL step 2, residual_2 = 1.481775e01 >> 2 SNES Function norm 1.481775334529e01 > Can you try with low optimization? This has the look of an incorrect > Jacobian (perhaps due to bad optimization or a bad dependent library). > > What is the initial state? Can you try running with snes_mf_operator? > (And get the ksp_monitor ksp_converged_reason working; we need to know > what's happening with that solve.)  Dafang Wang, Ph.D Postdoctoral Fellow Institute of Computational Medicine Department of Biomedical Engineering Johns Hopkins University Hackerman Hall Room 218 Baltimore, MD, 21218 
From: Robert Blake <rob.c.blake.3@gm...>  20140103 12:22:32

As someone who is also working on this code, the problem is not uninitialized memory. Valgrind reports a clean bill of health. On 01/02/2014 09:45 PM, Derek Gaston wrote: > This is almost definitely not a libMesh problem. It sounds like an > uninitialized variable. I recommend running your application using > valgrind ( http://valgrind.org ) . Use a debug (dbg) build of your > application and valgrind will point you right to the problem. > > Derek > > > On Thu, Jan 2, 2014 at 7:30 PM, Dafang Wang <dafang.wang@...> wrote: > >> Hi, >> >> My libmesh progam, which uses SNES to solve a nonlinear equation, had different >> convergence behavior when running on a linux desktop and on a cluster machine. >> >> All other running conditions were double checked to be identical: It was the same program >> running as a single thread on both machines. Both machines were using Libmesh 0.9.2.2. >> >> My program was solving a simple elastic model with good numerical/physical behavior. (The >> model consists of a 5*5*5 cubic mesh with a total of ~600 degress of freedom.) For the >> nonlinear solve, I was using the Newton method and line search (the >> default settings in Petsc SNES). >> >> The Newton method took 5 iterations to converge on my desktop (Ubutun 12.04), whereas it >> took 114 iterations to converge on the cluster. Moreover, both machines generated >> identical results in the first 2 Newton iterations, then they diverged >> from the 3rd iteration. >> >> I am wondering if such difference is common for Libmesh/Petsc programs running on >> different machines. Does anyone have similar experience of running programs on different >> clusters? >> >> Thanks, >> Dafang >>  >> Dafang Wang, Ph.D. >> Postdoctoral Fellow >> Institute of Computational Medicine >> Hackerman Hall, Room 218 >> Johns Hopkins University, Baltimore, 21218 >> http://lagniappe.icm.jhu.edu/~dwang/ >> >> >>  >> Rapidly troubleshoot problems before they affect your business. Most IT >> organizations don't have a clear picture of how application performance >> affects their revenue. With AppDynamics, you get 100% visibility into your >> Java,.NET, & PHP application. Start your 15day FREE TRIAL of AppDynamics >> Pro! >> http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk >> _______________________________________________ >> Libmeshusers mailing list >> Libmeshusers@... >> https://lists.sourceforge.net/lists/listinfo/libmeshusers >> >  > Rapidly troubleshoot problems before they affect your business. Most IT > organizations don't have a clear picture of how application performance > affects their revenue. With AppDynamics, you get 100% visibility into your > Java,.NET, & PHP application. Start your 15day FREE TRIAL of AppDynamics Pro! > http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk > _______________________________________________ > Libmeshusers mailing list > Libmeshusers@... > https://lists.sourceforge.net/lists/listinfo/libmeshusers 
From: Dafang Wang <dafang.wang@jh...>  20140103 06:41:29

Hi Jed, Below are the output details. (Results were obtained with snes_converged_reason, snes_monitor. The ksp_converged_reason ended up with seg fault, so was not included. It should not be the key problem though. ) Output from the desktop (good convergence rate): total constitutive equations = ngauss = 1000 EquationSystems n_systems()=1 System #0, "Elasticity" Type "NonlinearImplicit" Variables={ "ux" "uy" "uz" } Finite Element Types="LAGRANGE" Approximation Orders="FIRST" n_dofs()=648 n_local_dofs()=648 n_constrained_dofs()=108 n_local_constrained_dofs()=108 n_vectors()=1 n_matrices()=1 DofMap Sparsity Average OnProcessor Bandwidth <= 56.8889 Average OffProcessor Bandwidth <= 0 Maximum OnProcessor Bandwidth <= 81 Maximum OffProcessor Bandwidth <= 0 DofMap Constraints Number of DoF Constraints = 108 Average DoF Constraint Length= 0 *** Warning, This code is deprecated, and likely to be removed in future library versions! src/mesh/boundary_info.C, line 752, compiled Oct 29 2013 at 08:06:16 *** NL step 0, residual_2 = 1.440000e+00 0 SNES Function norm 1.440000000000e+00 NL step 1, residual_2 = 5.960479e01 1 SNES Function norm 5.960478586457e01 NL step 2, residual_2 = 3.745465e02 2 SNES Function norm 3.745464932748e02 NL step 3, residual_2 = 7.372288e04 3 SNES Function norm 7.372287735879e04 NL step 4, residual_2 = 1.426607e07 4 SNES Function norm 1.426607123296e07 NL step 5, residual_2 = 4.472564e12 5 SNES Function norm 4.472564229796e12 Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 5 Mechanics system solved at nonlinear iteration 5 , final nonlinear residual norm: 4.472564e12, converge reason= 3  Output from the cluster (poor convergence rate): total constitutive equations = ngauss = 1000 EquationSystems n_systems()=1 System #0, "Elasticity" Type "NonlinearImplicit" Variables={ "ux" "uy" "uz" } Finite Element Types="LAGRANGE" Approximation Orders="FIRST" n_dofs()=648 n_local_dofs()=648 n_constrained_dofs()=108 n_local_constrained_dofs()=108 n_vectors()=1 n_matrices()=1 DofMap Sparsity Average OnProcessor Bandwidth <= 56.8889 Average OffProcessor Bandwidth <= 0 Maximum OnProcessor Bandwidth <= 81 Maximum OffProcessor Bandwidth <= 0 DofMap Constraints Number of DoF Constraints = 108 Average DoF Constraint Length= 0 *** Warning, This code is deprecated, and likely to be removed in future library versions! src/mesh/boundary_info.C, line 752, compiled Dec 19 2013 at 15:59:31 *** NL step 0, residual_2 = 1.440000e+00 0 SNES Function norm 1.440000000000e+00 NL step 1, residual_2 = 5.960479e01 1 SNES Function norm 5.960478586457e01 NL step 2, residual_2 = 1.481775e01 2 SNES Function norm 1.481775334529e01 NL step 3, residual_2 = 7.269557e02 3 SNES Function norm 7.269556782845e02 NL step 4, residual_2 = 3.051185e03 4 SNES Function norm 3.051184607506e03 NL step 5, residual_2 = 2.572549e03 5 SNES Function norm 2.572548835065e03 NL step 6, residual_2 = 2.207251e03 6 SNES Function norm 2.207251245116e03 NL step 7, residual_2 = 2.131013e03 7 SNES Function norm 2.131012652111e03 NL step 8, residual_2 = 2.054042e03 8 SNES Function norm 2.054041562027e03 NL step 9, residual_2 = 1.981749e03 9 SNES Function norm 1.981749134719e03 NL step 10, residual_2 = 1.829931e03 10 SNES Function norm 1.829930870093e03 NL step 11, residual_2 = 1.732368e03 11 SNES Function norm 1.732367614052e03 NL step 12, residual_2 = 1.661435e03 12 SNES Function norm 1.661435082740e03 NL step 13, residual_2 = 1.550510e03 13 SNES Function norm 1.550509928763e03 NL step 14, residual_2 = 1.498905e03 14 SNES Function norm 1.498904724330e03 NL step 15, residual_2 = 1.451520e03 15 SNES Function norm 1.451520028630e03 NL step 16, residual_2 = 1.415582e03 16 SNES Function norm 1.415581852666e03 NL step 17, residual_2 = 1.314334e03 17 SNES Function norm 1.314334358891e03 NL step 18, residual_2 = 1.274014e03 18 SNES Function norm 1.274013709112e03 NL step 19, residual_2 = 1.774630e04 19 SNES Function norm 1.774630063354e04 NL step 20, residual_2 = 1.525944e04 20 SNES Function norm 1.525944094760e04 NL step 21, residual_2 = 1.477031e04 21 SNES Function norm 1.477031246669e04 NL step 22, residual_2 = 1.453195e04 22 SNES Function norm 1.453195226456e04 NL step 23, residual_2 = 1.302904e04 23 SNES Function norm 1.302904239436e04 NL step 24, residual_2 = 1.264238e04 24 SNES Function norm 1.264238361159e04 NL step 25, residual_2 = 1.135667e04 25 SNES Function norm 1.135667030231e04 NL step 26, residual_2 = 1.070810e04 26 SNES Function norm 1.070810206898e04 NL step 27, residual_2 = 1.036418e04 27 SNES Function norm 1.036417750251e04 NL step 28, residual_2 = 1.004803e04 28 SNES Function norm 1.004803143137e04 NL step 29, residual_2 = 9.154985e05 29 SNES Function norm 9.154985324734e05 NL step 30, residual_2 = 8.900270e05 30 SNES Function norm 8.900270443665e05 NL step 31, residual_2 = 6.523513e05 31 SNES Function norm 6.523513427832e05 NL step 32, residual_2 = 5.196890e05 32 SNES Function norm 5.196889874832e05 NL step 33, residual_2 = 4.962914e05 33 SNES Function norm 4.962914434970e05 NL step 34, residual_2 = 4.802445e05 34 SNES Function norm 4.802445335167e05 NL step 35, residual_2 = 4.660265e05 35 SNES Function norm 4.660265300997e05 NL step 36, residual_2 = 4.560508e05 36 SNES Function norm 4.560508074305e05 NL step 37, residual_2 = 4.176890e05 37 SNES Function norm 4.176889934246e05 NL step 38, residual_2 = 4.065922e05 38 SNES Function norm 4.065922035297e05 NL step 39, residual_2 = 2.819062e05 39 SNES Function norm 2.819062187692e05 NL step 40, residual_2 = 1.229634e05 40 SNES Function norm 1.229633591270e05 NL step 41, residual_2 = 8.466334e06 41 SNES Function norm 8.466333505983e06 NL step 42, residual_2 = 7.785196e06 42 SNES Function norm 7.785196405201e06 NL step 43, residual_2 = 7.517116e06 43 SNES Function norm 7.517115760778e06 NL step 44, residual_2 = 5.517172e06 44 SNES Function norm 5.517171865105e06 NL step 45, residual_2 = 4.627790e06 45 SNES Function norm 4.627789977237e06 NL step 46, residual_2 = 4.457517e06 46 SNES Function norm 4.457517058356e06 NL step 47, residual_2 = 4.393732e06 47 SNES Function norm 4.393732353601e06 NL step 48, residual_2 = 3.879515e06 48 SNES Function norm 3.879515135247e06 NL step 49, residual_2 = 3.763677e06 49 SNES Function norm 3.763677330091e06 NL step 50, residual_2 = 3.500036e06 50 SNES Function norm 3.500036191831e06 NL step 51, residual_2 = 3.256444e06 51 SNES Function norm 3.256444300502e06 NL step 52, residual_2 = 3.147962e06 52 SNES Function norm 3.147962249732e06 NL step 53, residual_2 = 2.860652e06 53 SNES Function norm 2.860652455813e06 NL step 54, residual_2 = 2.724702e06 54 SNES Function norm 2.724702170315e06 NL step 55, residual_2 = 2.619289e06 55 SNES Function norm 2.619289101215e06 NL step 56, residual_2 = 2.531273e06 56 SNES Function norm 2.531273013757e06 NL step 57, residual_2 = 2.229636e06 57 SNES Function norm 2.229636056679e06 NL step 58, residual_2 = 2.073304e06 58 SNES Function norm 2.073303801469e06 NL step 59, residual_2 = 2.003962e06 59 SNES Function norm 2.003961608610e06 NL step 60, residual_2 = 1.846291e06 60 SNES Function norm 1.846291365509e06 NL step 61, residual_2 = 1.752788e06 61 SNES Function norm 1.752788290496e06 NL step 62, residual_2 = 1.687328e06 62 SNES Function norm 1.687327986194e06 NL step 63, residual_2 = 1.633939e06 63 SNES Function norm 1.633938540202e06 NL step 64, residual_2 = 1.546997e06 64 SNES Function norm 1.546997405760e06 NL step 65, residual_2 = 1.436955e06 65 SNES Function norm 1.436954500901e06 NL step 66, residual_2 = 1.394765e06 66 SNES Function norm 1.394765119799e06 NL step 67, residual_2 = 1.177653e06 67 SNES Function norm 1.177652615109e06 NL step 68, residual_2 = 1.056466e06 68 SNES Function norm 1.056465738701e06 NL step 69, residual_2 = 1.029155e06 69 SNES Function norm 1.029154936896e06 NL step 70, residual_2 = 9.585998e07 70 SNES Function norm 9.585998143404e07 NL step 71, residual_2 = 9.124501e07 71 SNES Function norm 9.124500896542e07 NL step 72, residual_2 = 8.813674e07 72 SNES Function norm 8.813674410684e07 NL step 73, residual_2 = 8.557197e07 73 SNES Function norm 8.557196934474e07 NL step 74, residual_2 = 8.266958e07 74 SNES Function norm 8.266957904914e07 NL step 75, residual_2 = 7.628019e07 75 SNES Function norm 7.628018815310e07 NL step 76, residual_2 = 7.427199e07 76 SNES Function norm 7.427198948047e07 NL step 77, residual_2 = 2.892657e07 77 SNES Function norm 2.892656631445e07 NL step 78, residual_2 = 2.110709e07 78 SNES Function norm 2.110709381321e07 NL step 79, residual_2 = 1.971659e07 79 SNES Function norm 1.971659429523e07 NL step 80, residual_2 = 1.894502e07 80 SNES Function norm 1.894501578413e07 NL step 81, residual_2 = 1.811535e07 81 SNES Function norm 1.811534729574e07 NL step 82, residual_2 = 1.619090e07 82 SNES Function norm 1.619090109708e07 NL step 83, residual_2 = 1.568687e07 83 SNES Function norm 1.568686804349e07 NL step 84, residual_2 = 1.366121e07 84 SNES Function norm 1.366120654786e07 NL step 85, residual_2 = 1.276916e07 85 SNES Function norm 1.276915915061e07 NL step 86, residual_2 = 1.223932e07 86 SNES Function norm 1.223931884127e07 NL step 87, residual_2 = 1.163863e07 87 SNES Function norm 1.163863154650e07 NL step 88, residual_2 = 1.121774e07 88 SNES Function norm 1.121774061403e07 NL step 89, residual_2 = 1.079065e07 89 SNES Function norm 1.079065066402e07 NL step 90, residual_2 = 9.627717e08 90 SNES Function norm 9.627717396832e08 NL step 91, residual_2 = 9.338040e08 91 SNES Function norm 9.338040321284e08 NL step 92, residual_2 = 8.395781e08 92 SNES Function norm 8.395780687987e08 NL step 93, residual_2 = 7.972458e08 93 SNES Function norm 7.972458335224e08 NL step 94, residual_2 = 7.599603e08 94 SNES Function norm 7.599603469718e08 NL step 95, residual_2 = 7.371015e08 95 SNES Function norm 7.371015086782e08 NL step 96, residual_2 = 6.365861e08 96 SNES Function norm 6.365860767062e08 NL step 97, residual_2 = 5.893407e08 97 SNES Function norm 5.893407486421e08 NL step 98, residual_2 = 5.673266e08 98 SNES Function norm 5.673265571740e08 NL step 99, residual_2 = 5.195474e08 99 SNES Function norm 5.195473569965e08 NL step 100, residual_2 = 4.885457e08 100 SNES Function norm 4.885457271896e08 NL step 101, residual_2 = 4.685175e08 101 SNES Function norm 4.685175202156e08 NL step 102, residual_2 = 4.505639e08 102 SNES Function norm 4.505639250964e08 NL step 103, residual_2 = 4.324372e08 103 SNES Function norm 4.324372186040e08 NL step 104, residual_2 = 4.150867e08 104 SNES Function norm 4.150867469968e08 NL step 105, residual_2 = 4.006847e08 105 SNES Function norm 4.006846993522e08 NL step 106, residual_2 = 2.916694e08 106 SNES Function norm 2.916693756766e08 NL step 107, residual_2 = 2.374078e08 107 SNES Function norm 2.374078268250e08 NL step 108, residual_2 = 2.272917e08 108 SNES Function norm 2.272917460537e08 NL step 109, residual_2 = 2.173385e08 109 SNES Function norm 2.173385018336e08 NL step 110, residual_2 = 2.114292e08 110 SNES Function norm 2.114291507081e08 NL step 111, residual_2 = 2.040838e08 111 SNES Function norm 2.040838239152e08 NL step 112, residual_2 = 1.864519e08 112 SNES Function norm 1.864518544055e08 NL step 113, residual_2 = 1.805016e08 113 SNES Function norm 1.805015962441e08 NL step 114, residual_2 = 1.024459e08 114 SNES Function norm 1.024458891160e08 Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 114 Mechanics system solved at nonlinear iteration 114 , final nonlinear residual norm: 1.024459e08, converge reason= 3 Thanks, Dafang On 1/2/2014 9:33 PM, Jed Brown wrote: > Dafang Wang <dafang.wang@...> writes: > >> Hi, >> >> My libmesh progam, which uses SNES to solve a nonlinear equation, had different >> convergence behavior when running on a linux desktop and on a cluster machine. >> >> All other running conditions were double checked to be identical: It was the same program >> running as a single thread on both machines. Both machines were using Libmesh 0.9.2.2. >> >> My program was solving a simple elastic model with good numerical/physical behavior. (The >> model consists of a 5*5*5 cubic mesh with a total of ~600 degress of freedom.) For the >> nonlinear solve, I was using the Newton method and line search (the default settings in >> Petsc SNES). >> >> The Newton method took 5 iterations to converge on my desktop (Ubutun 12.04), whereas it >> took 114 iterations to converge on the cluster. Moreover, both machines generated >> identical results in the first 2 Newton iterations, then they diverged from the 3rd >> iteration. > Please send the output of ksp_converged_reason > ksp_monitor_true_residual snes_monitor snes_converged_reason > snes_view for both cases.  Dafang Wang, Ph.D. Postdoctoral Fellow Institute of Computational Medicine Hackerman Hall, Room 218 Johns Hopkins University, Baltimore, 21218 http://lagniappe.icm.jhu.edu/~dwang/ 
From: Derek Gaston <friedmud@gm...>  20140103 02:45:59

This is almost definitely not a libMesh problem. It sounds like an uninitialized variable. I recommend running your application using valgrind ( http://valgrind.org ) . Use a debug (dbg) build of your application and valgrind will point you right to the problem. Derek On Thu, Jan 2, 2014 at 7:30 PM, Dafang Wang <dafang.wang@...> wrote: > Hi, > > My libmesh progam, which uses SNES to solve a nonlinear equation, had > different > convergence behavior when running on a linux desktop and on a cluster > machine. > > All other running conditions were double checked to be identical: It was > the same program > running as a single thread on both machines. Both machines were using > Libmesh 0.9.2.2. > > My program was solving a simple elastic model with good numerical/physical > behavior. (The > model consists of a 5*5*5 cubic mesh with a total of ~600 degress of > freedom.) For the > nonlinear solve, I was using the Newton method and line search (the > default settings in > Petsc SNES). > > The Newton method took 5 iterations to converge on my desktop (Ubutun > 12.04), whereas it > took 114 iterations to converge on the cluster. Moreover, both machines > generated > identical results in the first 2 Newton iterations, then they diverged > from the 3rd > iteration. > > I am wondering if such difference is common for Libmesh/Petsc programs > running on > different machines. Does anyone have similar experience of running > programs on different > clusters? > > Thanks, > Dafang >  > Dafang Wang, Ph.D. > Postdoctoral Fellow > Institute of Computational Medicine > Hackerman Hall, Room 218 > Johns Hopkins University, Baltimore, 21218 > http://lagniappe.icm.jhu.edu/~dwang/ > > >  > Rapidly troubleshoot problems before they affect your business. Most IT > organizations don't have a clear picture of how application performance > affects their revenue. With AppDynamics, you get 100% visibility into your > Java,.NET, & PHP application. Start your 15day FREE TRIAL of AppDynamics > Pro! > http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk > _______________________________________________ > Libmeshusers mailing list > Libmeshusers@... > https://lists.sourceforge.net/lists/listinfo/libmeshusers > 
From: Dafang Wang <dafang.wang@jh...>  20140103 02:30:19

Hi, My libmesh progam, which uses SNES to solve a nonlinear equation, had different convergence behavior when running on a linux desktop and on a cluster machine. All other running conditions were double checked to be identical: It was the same program running as a single thread on both machines. Both machines were using Libmesh 0.9.2.2. My program was solving a simple elastic model with good numerical/physical behavior. (The model consists of a 5*5*5 cubic mesh with a total of ~600 degress of freedom.) For the nonlinear solve, I was using the Newton method and line search (the default settings in Petsc SNES). The Newton method took 5 iterations to converge on my desktop (Ubutun 12.04), whereas it took 114 iterations to converge on the cluster. Moreover, both machines generated identical results in the first 2 Newton iterations, then they diverged from the 3rd iteration. I am wondering if such difference is common for Libmesh/Petsc programs running on different machines. Does anyone have similar experience of running programs on different clusters? Thanks, Dafang  Dafang Wang, Ph.D. Postdoctoral Fellow Institute of Computational Medicine Hackerman Hall, Room 218 Johns Hopkins University, Baltimore, 21218 http://lagniappe.icm.jhu.edu/~dwang/ 