Qlogic Infiniband RDS高速互联驱动程序安装及配置

在前几天写了配置基于RDS协议的Infiniband高速内部互联的RAC数据库方案时，第一步必须是安装好驱动程序并配置好内部互联的IP地址，保证内部互联配置完成，才能进行下一步的RAC数据库安装和创建。下面简要介绍一下Infiniband安装配置的过程：
一、首先删除操作系统自带的OFED模块，否则在日后的使用中，可能会和后续安装的Infiniband驱动程序发生冲突：

#rm -rf /lib/modules/2.6.9-55.ELlargesmp/kernel/drivers/infiniband
#rm -rf /usr/src/kernels/2.6.9-55.EL-largesmp-x86_64/drivers/infiniband

这里面的2.6.9-55.Ellargesmp是LINUX的内核版本号，根据安装机器的OS内核而定。重启服务器，执行lsmod 去检查OFED是否还在运行。

二、编译Infiniband安装程序：
将InfiniServ安装文件COPY至相应目录，解压并编译：

#cd /opt
#tar -zxvf InfiniServ. 4.1.0.2.2.tgz
#cd InfiniServ. 4.1.0.2.2/ALL_HOST
#./do_build

编译完成后，该OS上的Ininiband安装文件即可生成，对应安装文件在/opt/InfiniServ.4.1.0.2.2/ALL_HOST/release/redhat/X86_64/InfiniServ.4.1.0.2.2G里面。
三、 安装Infiniband安装程序：

#cd /opt/InfiniServ.4.1.0.2.2/ALL_HOST/release/redhat/X86_64/InfiniServ.4.1.0.2.2G
#.INSTALL

屏幕显示如下画面：

SilverStorm Technologies Inc. InfiniBand 4.1.0.2.2 G Software

1) Install/Uninstall Software
2) Reconfigure IP over IB
3) Reconfigure Driver Autostart
4) Update HCA Firmware
5) Generate Supporting Information for Problem Report
6) Host Setup via Fast Fabric
7) Host Admin via Fast Fabric
8 ) Chassis Admin via Fast Fabric
9) Externally Managed Switch Admin via Fast Fabric

X) Exit

选择"1"，屏幕显示如下画面：

SilverStorm Technologies Inc. IB Install (4.1.0.2.2G release) Menu

Please Select Install Action:

0) IB Network Stack [ Install ] [Available]
1) IB Development [ Install ] [Available]
2) IB Boot [ Install ] [Available]
3) Fast Fabric [ Install ] [Available]
4) Virtual HBA (SRP) [ Install ] [Available]
5) Virtual NIC [ Install ] [Available]
6) IP over IB [ Install ] [Available]
7) MPI Runtime [ Install ] [Available]
8 ) MPI Development [ Install ] [Available]
9) MPI Source [ Install ] [Available]
a) uDAPL [ Install ] [Available]
b) SDP [ Install ] [Available]
c) RDS [ Install ] [Available]

P) Perform the selected actions
I) Install All
U) Uninstall All

X) Return to Previous Menu (or ESC)

上图中显示，P：执行选择的项
I：全部安装以上11项
U：全部卸载以上11项
选择安装某项的方法如下：
例如要安装IB Network Stack项，我们可以通过在键盘上敲击IB Network Stack前面的数字0，来切换到Install提示（此处提示共有Install，Do Not Install或Up To Date三种选择）。
如果HPC应用，只需选择0,1,3,6,7,8,9,b项进行安装即可。
如果使用RDS for Oracle应用， 选择全部进行安装即可
如果使用IPOIB for Oracle应用，选择0,1,3,6 进行安装就可以。
选择各项为Install提示后，
输入"P"，执行，系统出现如下画面：

-----------------------------------------------------
Installing IB Network Stack...
Adding module dependencies...
Adding memory locking limits...
Copying ibt.ko...
Copying ics_dsc.ko...
Copying 82808XA.ko...
Copying mt23108vpd.ko...
Copying mt25218vpd.ko...
Creating IB Network Stack (iba) system startup files...
Creating IB Port Monitor (iba_mon) system startup files...
-----------------------------------------------------
Installing IB Development...
-----------------------------------------------------
Installing Fast Fabric...
-----------------------------------------------------
Installing IP over IB Driver...
Copying ipoib.ko...
Assign IP over IB static IPV4 addresses now? [n]: y
提示是否配置IPOIB，输入"y",出现如下画面：
You may configure an IP over IB interface for each IB port
Or you may select to have IP over IB only run on some IB ports
Or you may select to configure redundant IB ports with a
pair of IB ports running a single IP over IB interface
How many IP over IB interfaces would you like to configure? [1]:

提示需要配置多少个ip，默认为1个，如果我们只需设置1个IP地址，直接回车或输入"1"回车，屏幕出现如下画面：

Preparing to create IP over IB ifcfg files in /etc/sysconfig/network-scripts
Assign interface names sequentially starting with ib1? [y]:
Assign Internet Addresses sequentially from a base IP address? [y]:
Enter IPV4 address in dot notation for ib1: 192.168.100.145
Is IPV4 address '192.168.100.145′ correct? (y/n): y
Enter IPV4 netmask in dot notation for ib1 192.168.100.145 [255.255.255.0]:
Creating ifcfg-ib1 for 192.168.100.145 mask 255.255.255.0

IP over IB requires /etc/sysconfig/ipoib.cfg specify parameters
for each IP over IB device.
The default configuration file provides for a 2 port redundant configuration.
If you desire a different configuration for IP over IB, Manually edit the file.
Hit any key to continue...

根据屏幕提示输入：
Assign interface names sequentially starting with ib1? [y]:
提示是否要将IB接口名称设定为ib1, 直接回车或键入 y 回车.
Assign Internet Addresses sequentially from a base IP address? [y]:
提示是否要分配一个IP地址，直接回车或键入 y 回车.
在提示Enter IPV4 address in dot notation for ib1: 键入 IP 地址xxx.xxx.xxx.x并回车
在提示 Is IPV4 address 'xxx.xxx.xxx.x' correct? (y/n): 键入y并回车.
在提示 Enter IPV4 netmask in dot notation for ib1 xxx.xxx.xxx.x [255.255.255.0]: 回车 以设置默认值 (255.255.255.0).

敲入任何键，继续：

Creating IP over IB (ipoib) system startup files...
-----------------------------------------------------
Installing MPI Runtime...
Make links for mpd in ///opt/iba/bin
Make links for mpd in ///usr/local/bin
-----------------------------------------------------
Installing MPI Development...
-----------------------------------------------------
Installing MPI Source...
-----------------------------------------------------
Installing uDAPL...
Copying udapl_module.ko...
Creating uDAPL (udapl) system startup files...
-----------------------------------------------------
Installing SDP Driver...
Copying ics_offload.ko...
Copying ics_sdp.ko...
Creating SDP (ics_sdp) system startup files...

Enable IB Network Stack (iba) to autostart? [y]:
Enable IB Port Monitor (iba_mon) to autostart? [y]:
Enable IP over IB (ipoib) to autostart? [y]:
Enable uDAPL (udapl) to autostart? [y]: n
Enable SDP (ics_sdp) to autostart? [y]: n
Hit any key to continue...

在提示 Enable xxx to autostart? [y]: 键入 Y或N 回车,屏幕显示如下图：

Generating module dependencies...
Updating HCA Firmware ...
Select HCAs to Update:
1) HCA 1 (23108 Rev a1 psid "MT_0030000001″ Node GUID: 0×00066a0098004c15)
Selection (a for all, n for none) [a]:

提示是否更新HCA firmware 版本，输入 n , 回车，屏幕出现最初画面：

SilverStorm Technologies Inc. InfiniBand 4.1.0.2.2G Software

1) Install/Uninstall Software
2) Reconfigure IP over IB
3) Reconfigure Driver Autostart
4) Update HCA Firmware
5) Generate Supporting Information for Problem Report
6) Host Setup via Fast Fabric
7) Host Admin via Fast Fabric
8 ) Chassis Admin via Fast Fabric
9) Externally Managed Switch Admin via Fast Fabric

X) Exit

敲入"x"退出安装。

5) 重新启动系统
#reboot
6) 重新启动系统后，检查HCA驱动是否正常启动
#lsmod
显示如下：

o ipoib
o mt23108vpd
o ics_dsc
o ics_offload
o ibt
o scsi_mod
o ics_sdp

至此，此节点的Infiniserv软件安装完成。