[Libmesh-devel] Weak scaling updates for GMG post NBX

[Boris B. <bor...@bu...>]

Hello all,

Out of some curiosity I recently rebased my GMG implementation on to the 
upcoming NBX changes in PR #1965 to do some weak scaling analysis.

I ran an np 256 Poisson problem using GMG with ~10k dofs/proc, and in 
short it, seems like the NBX changes provide some solid improvement 
bringing my total runtime from something like ~19s to ~16s, so a nice 
step on the road to  weak GMG scaling.  Pre-NBX I had a good chunk (30% 
total w/sub) of my time being spent in the alltoall() which post-NBX is 
down to 2%! This came with a fairly large amount of calls to 
possible_receive() and now 15% of my total time being spent in there, 
but the overall timing seems to be a win so thanks much for this work!

Despite these improvements, the weak scaling for the GMG implementation 
is still a bit lacking unfortunately as np1=~1s. I ran these tests 
through gperf in order to gain some more insight and it looks to me that 
major components slowing down the setup time are still 
refining/coarsening/distributing_dofs which in turn do a lot of nodal 
parallel consistency adjusting and setting nonlocal_dof_objects and am 
wondering if there are maybe some low hanging fruit to improve on around 
those calls.

Logs for these runs and the corresponding gperf outputs can be found at  
[1], and come with the minor note that I had some trouble combining all 
of the profile_pid logs into a single gperf output file. Basically all 
the function names get garbled if I pass in all the profile_pid outputs 
to pprof ( hoping maybe Derek Gaston can provide some tips on how to do 
this properly? ), but as such the linked 'web' plots are taken from a 
single representative gprof_pid output. I cross compared a number of 
such outputs and the results seemed fairly consistent between samples, 
but obviously this isnt perfect.

Anyways, figured I'd post this as an update, please let me know if I can 
provide anything further surrounding these tests that would be helpful.

Thanks as always,

- Boris

[1]: https://drive.google.com/open?id=1dgHKiC8oStpTR0yJFv5uVb8wyJRIFTkk