In case there's anyone that hasn't muted this conversation yet...

I think I've tracked the INL's issue down to mpich2.  (I happen to be using mpich2-1.4.1p1, but the problem is probably present in mpich2-1.5 as well.)

(Correct me if I'm wrong, but everyone at Texas on a Mac is using OpenMPI, right?)

Here's the behavior I'm observing, and a possible explanation:

The automake branch works fine for me using hand-built GCC 4.5.4, 4.6.3, and 4.7.2 compilers *without MPI or PETSc* on an otherwise stock Mountain Lion install (no MacPorts).  Yes, I really built all those GCCs by hand on a laptop with 4 cores :-P

Using the same compilers, if I load my Mpich2 module (which sets CC=mpicc and CXX=mpicxx) the automake branch suddenly exhibits the mysterious segfaults we've discussed here ad-nauseum in the adjoints_ex1 example.

So, what the hell, mpich?

Let me note first that we use a very vanilla configuration of mpich2, effectively just

./configure --prefix=... --enable-shared --enable-debuginfo
make install

What does this produce?

$ mpicxx -show
g++ -Wl,-flat_namespace \
  -I/opt/packages/mpich2/mpich2-1.4.1p1/include \
  -L/opt/packages/mpich2/mpich2-1.4.1p1/lib \
  -lmpichcxx -lpmpich -lmpich -lopa -lmpl -lpthread

Ugh... the prime suspect here has to be -Wl,-flat_namespace.  The mpich people must be adding this flag if they detect Darwin... but I think that may be the wrong thing for the Lion and Mountain Lion operating systems.

So I could try to figure out how to make mpich not use that flag, but I think my next step is going to be trying out OpenMPI.  Anybody have special tips for building that by hand?