[Chromium-dev] IB support changes

[Mike H. <mho...@gr...>]

I see you've updated the IB stuff, at least the makefiles, to use the 
TopSpin stack.  This is a really bad idea.  You'll want to run the 
OpenIB gen1 stack or better yet the Mellanox HPC Gold stack which is 
public and "easy" to use.  The later is now seemingly the standard stack 
until the OpenIB gen2 stack cleans up and goes for Linux kernel inclusion.

The IB (VAPI) code was fragile already.  SDP was better off, but is only 
known to work, albeit hackily, with the OpenIB gen1 stack.  It "should" 
work with the Mellanox HPC Gold stack since it uses the OpenIB SDP and 
OpenSM code.  The main performance issue there is getting AIO up and 
running so we can do non-blocking sends.  But this quickly leads to a 
network layer rewrite so all layers can safely handle non-blocking sends 
and possibly non-blocking receives.  This code has also not been touched 
since the last round of network changes, as can be seen from the compile 
issues, so other things may have changed that might cause issues.

The VAPI code will have to be re-written as the VAPI interface in 
changing in the OpenIB gen2 stack.  When this happens, a decision will 
have to be made if we rewrite the code to use the new VAPI interface, or 
we move to one of the DAPLs, or just settle on an MPI-2 layer for 
Chromium.  SDP has some interesting IP/legal issues and might not be 
supported by all IB vendors in the future.

I have exams for the next 2 weeks, but if all IB Chromium users can move 
to a standard stack, like the Mellanox stack, than we can try to work 
together to sort out the remaining issues towards the end of the month.

-Mike