Linux InfiniBand Project Files

InfiniBand Code Drop                                                01/20/2003
------------------------------------------------------------------------------

This is a more complete implementation that supports Profile B HCA and switch.

*** Please read the file before building to understand changes to build ***

Changes Since Last Code Drop
----------------------------
- Code is stable for 48 hrs with IPOIB running IOMeter
- Tested for overnight runs with IPOIB in a 6 node cluster.
- Works for Multicast across more than 1 switch (Tested only with 2 so far)

Unsupported API's IN AL
----------------------
ib_get_spl_qp() - Only QP0_ALIAS and QP1_ALIAS are supported at this time.
ib_reg_shmid()	- Coded, but not tested
ib_reg_pnp()	- Does not support IOC events.
ib_subscribe()
ib_unsubscribe()- Register/De-register with class manager for event notification
  				  This feature is not supported.
ib_reject_ioc()	- Not supported.

CM does not support RD and related EE management at this time.

The following CM API's are coded, but not tested in both kernel and user 
mode.

ib_cm_mra, ib_cm_lap, ib_cm_apr, and ib_force_apm.

Key features of this release include

- Component library in kernel mode has memory tracking and proc file.
  - If you need to track memory allocations, then load complib with 
  module param mem_track=1. You can now track currently allocated memory
  blocks from cat /proc/complib/meminfo. In addition when all drivers are
  unloaded, unloading complib will print any memory blocks that were not freed.
  Helps in identifying memory leaks early.

- Full feature opensm 
  - Configure both Profile A and Profile B switches and HCA's
  - Tested with Mellanox and InfiniCon Switches.
  - Configures multiple vendor HCA's in a single system. 
    (Currently tested with Mellanox and Intel GEN1 HCA).
  - Supports Standard SA queries, Path, Node, portinfo etc.
  - Supports Multicast join and leave functions across more than one
    switch dynamically.
  - Supports Service Registration
  - osmtest utility to stress test SA functions, also can retrieve 
    node and path info records from any node by running standard SA queries 
	from AL.
  - Generates /tmp/subnet.lst file to view fabric using Mellanox GUI.
	 **** NEW COMMAND LINE SWITCHES ***

	 -V - More Verbose output
	 -s 0 - Stop Subnet sweep after one pass, but sleep so SA is functional.
	 -o - Run once and quit option.

- Access Layer
  - Supports CM in both kernel and user mode
  - Complete SMI/GSI implementation.
  - GSI now includes complete Reliable Multipacket Protocol (RMPP) 
    version 2.0 that matches the IBTA Specification release 1.1.
  - ib_query
      - Now supports synchronous support.
      - API changed to now provide a list of MAD's vs list of structures. 
        Consumer must perform a put of the MADs via ib_put_mad. 
	    ib_get_query_result can be used to obtain the exact data portion from 
		the list of ib_mad_element. If the query is for path records, one 
		could also use ib_get_query_path_rec.

- Functional Connection Manager
  - Includes full support for CM version 2.0 update from IBTA release 1.1 for 
    Reliable Connected QPs.
  - Includes SIDR support for UD QP's.
    - NOTE: SIDR api's have changed. Instead of having separate SIDR api's 
	  They are not merged with the CM API's. Please see the ib_al.h for 
	  relevant changes.
	   - ib_sidr_reg is now ib_cm_listen
	   - ib_sidr_req is now ib_cm_req
	   - ib_sidr_dereg is now ib_cm_cancel
	   - ib_sidr_rep is now ib_cm_rep
  - New Version Field added to cm_listen, or cm_req to permit connection with
    older targets that run 1.0a version of CM protocol.

  - No support yet for RD, LAP support yet.

- AL Test Suite (alts)
  - Facilitates basic verification tests that support both kernel and
    user mode.

- IPoIB 
  - Tested with upto 6 nodes connected via switch running stress for > 48+ hrs.
  - First ping may be long due to path resolution process.
  - IPOIB patches are now in official linus 2.5 kernel tree.
  - Please make the changes to kernel sources to ensure
    proper functioning of the ipoib driver. Follow directions in
	Readme file under ipoib directory. Also follow instructions for
	bringing up the ipoib interface. 
  - Not much stress testing has been performed, so be gentle.

This drop contains the following:

*  Component Library (include/iba/complib and drivers/iba/complib)
      See below for more explanation on this component.
*  Kernel verbs header file (include/iba/ib_ci.h)
      More on the kernel verbs can be found on the project website
      at http://infiniband.sourceforge.net, sub-project "HCA DDK".  
*  Sample HCA driver scaffold (drivers/iba/hca/model)
*  Usermode verbs header file (include/iba/ib_uvp.h)
      More on the usermode verbs can be found on the project website
      at http://infiniband.sourceforge.net, sub-project "HCA DDK".  
*  Access Layer (drivers/iba/al)
      More on the Access Layer can be found on the project website
      at http://infiniband.sourceforge.net, sub-project "Access Layer".  
*  IPoIB (drivers/iba/net/ipoib)
      More on IPoIB can be found on the project website 
      at http://infiniband.sourceforge.net, sub-project "IPoIB".  
*  API documentation (iba_docs)
      Note that the Access Layer and the Verbs API documentation 
      (both kernel and usermode) can be found on the project website 
      (http://infiniband.sourceforge.net) at the Documents page off of the 
      "Access Layer" and "HCA DDK" sub-projects. 
*  Subnet Manager (linuxuser/iba/OpenSM)
	Initial implementation of Open Source SM using subset of transport
	functionality. 


Be sure to consult the Readme, HOWTO-iba and FAQ-IB-BUILD documents for more 
details concerning the specifics of this drop.

Browse at http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/infiniband.

How do i build this tarball distribution?
-----------------------------------------
Make sure you fix kernel sources as indicated in the ipoib/Readme 
file for proper functioning of the drivers.

You will need to build the kernel and user mode distributions.
# cd linux

now edit Makeconf.inc to modify the following.

LINUX_VERSION - Determines the default source and install directories
                for which the release is to be built.
set REDHAT=0 - to build with stock kernel.org sources.

if your sources do not exist in /usr/src/linux-{VERSION} then set the 
default location of your sources in LINUX_SRC variable in Makeconf.inc

Run make in linux directory to build all kernel sources

# cd linuxuser/iba

Now run make to build the user mode utilities


What is the Component Library?
------------------------------

The Component Library (complib) is a set of useful utilities that provide 
basic abstractions for providing OS variations. They are pretty lightweight 
and should not affect any performance, mostly macros translating to base types. 
In addition there are list management, and resource pool management routines 
that otherwise will be re-invented every time new code is written. Having 
complib and all drivers using same set of code will lead to fewer lines of 
code in kernel and also provide code that is tested to all other modules.

How do i generate documentation for the API's?
----------------------------------------------

API documentation in the form of html browsable files is shipped with each 
tarball posted. One could also regenerate the html files using the script
'mkdocs' which will place all generated docs under iba_docs directory.

How can i use Intel GEN1 hardware and drivers with GEN2 software?
-----------------------------------------------------------------
Shim and Intel Gen1 driver (linux/drivers/iba/hca/shim)
 How to build and use:

* Build the shim driver under linux/drivers/iba/hca/shim
* Build the gen1 access driver under linux/drivers/iba/hca/shim/public
* Apply the latest patch for the Intel Gen1 driver 82808XA 
	if you have not already done so. Its named Early_Ref_071602.patch.

	This change was necessary to provide additional context necessary for
	GEN2 drivers.

* Build 82808XA.o driver following the instructions for building the Early
  Reference code.

* Now load the drivers in the following sequence.
	- complib.o 
	- ibal.o
	- shim_82808xx.o
	- public.o
	- 82808XA.o

This completes the build and load process for the Intel HCA driver.

How to build for Mellanox hardware?
------------------------------------

1. Obtain SDK from mellanox
2. Populate directory mellanox under linux/drivers/iba/hca/mellanox
   under the build tree
3. Edit the top level Makefile in linux to add

   MLNX=y

4. Obtain documentation on how to start drivers for Mellanox from
   the SDK area or from Mellanox support.

5. Build and load.

How to run the loopback tests 
-----------------------------

NOTE: The AL test suite tests run only in a single system configuration.
      Simply connect two ports directly or via a switch, and run opensm 
	  to configure.


1. Connect any 2 ports of HCA's in same system.
2. Run opensm, run with options to not sweep to reduce SM activity.

   opensm -V -s0

 This will assign fixed lids 1 and 2 and move the ports to active state.

3. Load the altsdrv.o which is the test driver.
4. run alts --tc=MultiSend [--km or --um]

   km - kernel mode test
   um - user mode test