Well... it has to do with the craziness of me swapping and unswapping MPI communicators to get libMesh to work on sub-communicators ;-)
When I swap on one processor it sets both COMM_WORLD and CommWorld to the sub-communicator. When I swap back it sets them both to what COMM_WORLD was before the swap... so on any processor that swapped and swapped back COMM_WORLD would match CommWorld.
If any processor didn't do any sub-solve then it won't swap at all... and so it has a mismatched COMM_WORLD and CommWorld.... so now the CommWorlds on processors that swapped won't match the CommWorlds on processors that don't swap..... and the next thing done using CommWorld will hang.
Thoroughly confused? This is exactly why Ben's branch is so damn important ;-)