Menu
▾
▴
RE: [Infiniband-general] udapl:al:query_req_cb() !ERROR!: query failed:
|
From: moral m. <mor...@ho...> - 2004-04-27 03:05:20
|
Hello
According to the installation:
---------------------------------------------------------------------------------------
Source Forge Notes
1) Log onto Source Force site
a. http://infiniband.sourceforge.net
2) Download "Bitkeeper" software
a. Click on Bitkeeper Repository link (under Source Forge
Services)
b. Click on download link
c. Click on Download Bitkeeper
d. Determine proper platform and complete download
e. Note: You must fill in form and provide email address.
SW will email you login info to "bitmover site"
f. http://www.bitmover.com/download
* login: bitkeeper
* password: get bitkeeper
3) After you have downloaded bitkeeper software, change the mode of
the
file to make it executable (i.e. chmod a+x *.bin)
a. chmod a+x 3.0.1-x86-glibc22-linux.bin
b. ./ 3.0.1-x86-glibc22-linux.bin
4) The above process creates an executable called "bk". It also
creates
a symbolic link from bk to /usr/bin/bk
5) Create a directory where you would like to place the SF project
a. Ex. mkdir /usr/sf-iba
6) Use the bk program to pick up a clone of the latest Source Force
release
for iba
a. cd /usr/sf-iba
b. bk clone http://project.bkbits.net/repository
where: project = infiniband, repository = iba; producing:
c. bk clone http:/infiniband.bkbits.net/iba
7) Running "bk" will create a sub_folder in your current working
directory
a. => /usr/sf-iba/iba
8) Add Mellanox source to newly downloaded source tree
a. cd /usr/sf-iba/iba
b. Download and copy mellanox_xxx.tgz file to this directory and
unzip/untar file. Note: archive will place into base and
create
linux folders. It is very important that you place the
mellanox
files into the correct directory structure
c. tar xzvf mellanox_xxx.tgz. This will create the following:
* /usr/sf-iba/iba/base/linux/hca/mellanox
* /usr/sf-iba/iba/linux/run
9) Create a build tree for the project using the "mktree" command.
This will pull out all sources from the base directory to a
directory
of your choosing
a. cd /usr/sf-iba/iba/base/linux/utils
b. ./mktree -source /usr/sf-iba/iba -target /usr/sf-iba/iba
* mktree create 2 new directories linux and linuxuser in
/usr/sf-iba/iba
10) Rebuild the kernel and boot with new kernel (example using
linux-2.4.18-10 kernel tree)
a. cd /usr/src/linux-2.4.18-10
b. make xconfig
* make any changes to fit your hardware configuration:
o SMP/UP option (YES for Process type and
features/Symmetric
multi-processing support buttons)
o network driver (YES for Network device
support/Ethernet
(10 or 100Mbit)/Ethernet (10 or 100Mbit)... buttons
o scsi driver (YES for SCSI support/ SCSI low-level
drivers/...
buttons)
o local file system (YES for File systems/Ext3
journaling
file system support - Second extended fs support
buttons)
* hit "save and exit" button
c. Patch the kernel to support IPoIB with larger MAX_ADDR_LEN
* edit /usr/src/linux-2.4.18-10/include/linux/netdevice.h to
change MAX_ADDR_LEN = 8 to = 32
* edit /usr/src/linux-2.4.18-10/net/core/dev.c to change:
The SIOCGIFHWADDR case from:
memcpy(ifr->ifr_hwaddr.sa_data, dev->dev_addr,
MAX_ADDR_LEN);
to
memcpy(ifr->ifr_hwaddr.sa_data, dev->dev_addr,
min((unsigned char)
sizeof(ifr->ifr_hwaddr.sa_data),
dev->addr_len));
The SIOCSIFHWBROADCAST case from:
memcpy(dev->broadcast, ifr->ifr_hwaddr.sa_data,
MAX_ADDR_LEN);
to
memcpy(dev->broadcast, ifr->ifr_hwaddr.sa_data,
min((unsigned char)
sizeof(ifr->ifr_hwaddr.sa_data),
dev->addr_len));
d. make dep
e. make bzImage
f. make install
g. verify /etc/lilo.conf (or /etc/grub.conf) to reboot the
system
with new kernel
11) Install Mellanox MST and MSDK
a. install mst driver
b. install MSDK
c. verify the installation (at least run vapi start)
12) The next step is to build all the driver for the stack
(including
Mellanox module)
a. cd /usr/sf-iba/iba/linux
b. edit Makeconf.inc
* edit LINUX_VERSION = 2.4.18-10
* edit REDHAT=1 if you run original kernel from RH
installation.
In our case, we are running costumed kernel, we leave it
as default
* edit LINUX_SRC. In our case, we edit LINUX_VERSION;
therefore,
we leave this as default
c. edit Makefile.config
* Change BUILD_SMP line to: BUILD_SMP =0 (if kernel is
UP)
* Change BUILD_SMP line to: BUILD_SMP =1 (if kernel is
SMP)
d. Edit Makefile; Change "MLNX=n" to "MLNX=y, MODEL,SHIM=y
to
MODEL,SHIM=n)
13) Build/Compile kernel driver components by running make from top
level
directory
a. cd /usr/sf-iba/iba/linux
b. make debug (or free)
c. All the object files are in
* ~bin/$(ARCH)/$(LINUX_VERSION)/$(MP_MODE)/$BLD
* in our case ARCH=x86, LINUX_VERSION=2.4.18-10, MP_MODE=up,
BLD=debug
14) Build/Compile user mode driver components.
a. cd /usr/sf-iba/iba/linuxuser/iba
b. make debug (or free)
c. All the executable files (opensm, alts...) are in
/usr/sf-iba/iba/linuxuser/bin/$(ARCH)/$(LINUX_VERSION)/bin/$(BLD)
d. All the libraries are in
/usr/sf-iba/iba/linuxuser/bin/$(ARCH)/$(LINUX_VERSION)/lib/$(BLD)
e. In our case ARCH=x86, LINUX_VERSION=2.4.18-10, BLD=debug
f. create the symbolic link for liballib, libcomplib and
libmlnx_uvp
* cd /usr/sf-iba/iba/linuxuser/bin/x86/2.4.18-10/lib/debug
* ln -s liballib.so.0.0 liballib.so
* ln -s libcomplib.so.0.0 libcomplib.so
15) Run the stack with "ibal" script which is the equivalent to
"vapi" script
a. cd /usr/sf-iba/iba/linux/run
b. edit ibal
* edit line 35,36.
* Assuming that we are running UP kernel and debug mode then
o line 35 should be
MODPATH_I=$IBAL_PATH/bin/x86/$KERVER/up/debug
o line 36 should be
MODPATH_TV=$IBAL_PATH/drivers/iba/hca/mellanox/thh/obj_linux_x86_up
* Assuming that we are running SMP kernel and free mode then
o line 35 should be
MODPATH_I=$IBAL_PATH/bin/x86/$KERVER/smp/free
o line 36 should be
MODPATH_TV=$IBAL_PATH/drivers/iba/hca/mellanox/thh/obj_linux_x86_smp
c. cd /usr/sf-iba/iba/linux and issue command "run/ibal start"
16) Set up the linking library path. There are two ways.
a. Edit the /root/.bashrc and add in the following line
* export LD_LIBRARY_PATH=
/usr/sf-iba/iba/linuxuser/bin/x86/2.4.18-10/lib/debug
b. Or
* edit /etc/ld.so.conf and add in the following line
/usr/sf-iba/iba/linuxuser/bin/x86/2.4.18-10/lib/debug
* run ldconfig
17) Run the OPENSM
a. opensm executable file locates at
/usr/sf-iba/iba/linuxuser/bin/x86/2.4.18-10/bin/debug or
/usr/sf-iba/iba/linuxuser/iba/opensm directories
b. cd /usr/sf-iba/iba/linuxuser/bin/x86/2.4.18-10/bin/debug
c. ./opensm
d. Whew... finally
e. leave the opensm running still (sweeping mode)
18) Run IPoIB
a. cd /usr/sf-iba/iba/linux/bin/x86/2.4.18-10/up/debug
b. insmod ipoib.o
c. ifconfig -a ( you will see ib01 and ib02 pop up)
d. ifconfig ib01 11.0.0.1
e. ping 11.0.0.1
19) Run uDAPL
a. edit /etc/hosts to add in IPaddress for all DAPL nodes in
cluster.
Distribute the file for every node in the cluster. IPaddress
can be the same as IPoIB IP address as long as node name has
to end
with "xx-ibX". Even though you are not running IPoIB, you
have to add
in Interface Adapter Address (IP address) for DAPL IA. For
now,
one IP address for each DAPL IA (for each HCA)
Example: local node is mtilab10
10.2.2.10 mtilab10 ==> this is ethernet 0
10.2.2.11 mtilab11
192.0.0.10 mtilab10-ib0 ==> this is IPoIB or Interface
Adapter
address for local DAPL IA
1
192.0.0.14 mtilab10-ib0a ==> this is IPoIB or Interface
Adapter
address for local DAPL IA
1 port 1
192.0.0.12 mtilab10-ib1 ==> this is IPoIB or Interface
Adapter
address for local DAPL IA
2
192.0.0.11 mtilab11-ib0 ==> this is IPoIB or Interface
Adapter
address for remote DAPL IA
b. Copy mv_dapl/doc/dat.conf to /etc/dat.conf. Please read
mv_dapl/doc/dapl_registry_design.txt for more infomation how
to
edit /etc/dat.conf correctly
NOTES: If you compile dapltest with EXPLICIT_LINK=1 then
dapltest
is explicitly linked to libdapl.so library; therefore,
you
have to uncomment the entry in dat.conf or do not have
dat.conf in /etc.
c. dapltest executable file locates at
/usr/sf-iba/iba/linuxuser/bin/x86/2.4.18-10/up(smp)/debug(free) or
/usr/sf-iba/iba/linuxuser/iba/mv_dapl/test/udapl/dapltest
c. Run dapltest testsuite for 2 nodes (mtilab10-ib0 and
mtilab11-ib0)
* At both nodes
o start the driver (ibal start)
o cd
/usr/sf-iba/iba/linuxuser/iba/mv_dapl/test/udapl/dapltest
* At mtilab10-ib0
o ./srv.sh ==> start server
* At mtilab11-ib0
o ./bw.sh mtilab10-ib0 ==> run bw test
o ./lat_poll.sh mtilab10-ib0 ==> run latency polling
mode test
o ./lat_block.sh mtilab10-ib0 ==> run latency blocking
mode test
o ./cl.sh mtilab10-ib0 ==> run transaction tests
o ./regress.sh mtilab10-ib0 ==> run transaction
regression test
o ./quit.sh mtilab10-ib0 ==> instruct the server to
quit
* both can run limit test ./lim.sh (don't need to run server)
d. Please reference to dapltest.txt for more test options
-----------------------------------------------------------------------------------------
I have finished step 18),but I meet the problem in step 19).I think the
opensm is well in
step 18).
Thank you
WangJue
_________________________________________________________________
与联机的朋友进行交流，请使用 MSN Messenger: http://messenger.msn.com/cn
|
