Adding iSCSI Storage to a RHEL VM

Prerequisites

Be sure the following pre0requisites are met and information gathered before proceeding. Values selected and used for this example are shown in brackets.

iSCSI - TCP port 2500

The vFiler from which the data is to be served is identified - see "vFiler Contexts and Networks". (ARTS_NASpool 10.90.32.8 on kitsilano)
An IP address in the same subnet as the vFiler has been assigned for the vm and registered in the DNS - see "vFiler Contexts and Networks". (10.90.32.20 arts-dbprod.nas.usg.it.ubc.ca)
The required volumes have been created and assigned to the vFiler - see "Add Volume to vFiler". (ARTS_DBprod_LOG, ARTS_DBprod_DATA)
The required LUNs have been created and the filer configured to make them available via iSCSI - see "Configure vFiler iSCSI Access". (/vol/ARTS_DBprod_LOG/VGdb_LOG/PVlog1.lun, /vol/ARTS_DBprod_DATA/VGdb_DATA/pvdata1.lun)
The mountpoint for the filesystem is identified. (/dblogs, /dbdata)

Configure Eth0 Interface

The configuration for eth0 must be altered to ensure eth0 correctly binds to "Network Adapter 1":

Edit /etc/sysconfig/network-scripts/ifcfg-eth0
Add the line: HWADDR=<Recorded mac address for Network Adapter 1>

# Gigabit Ethernet Controller (Copper)
[root@kratos iscsi]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
NM_CONTROLLED="yes"
ONBOOT=yes
TYPE=Ethernet
BOOTPROTO=none
IPADDR=142.103.236.15
PREFIX=24
GATEWAY=142.103.236.254
DNS1=142.103.236.1
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth0"
UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03
HWADDR=E4:1F:13:B9:DD:C8

Configure Eth1 Interface

Eth1 is configured to communicate with the vFiler:

Edit /etc/sysconfig/network-scripts/ifcfg-eth1. You may have to manually create this file on hardened images (not running kudzu).
Set the BROADCAST, IPADDR, NETMASK and NETWORK for the NAS network the adpater is attached to (ARTS_NASpool in this example).
Set the HWADDR value to the recorded mac address of "Virtual Ethernet Adapter" (the adapter attached to the ARTS_NASpool VLAN in this example).

# Ethernet Controller
[root@kratos iscsi]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE="eth1"
NM_CONTROLLED="yes"
ONBOOT=yes
TYPE=Ethernet
BOOTPROTO=none
IPADDR=10.20.14.15
PREFIX=24
GATEWAY=10.20.14.254
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth1"
UUID=9c92fad9-6ecb-3e6c-eb4d-8a47c6f50c04
HWADDR=E4:1F:13:B9:DD:CA

Enter vFiler IP in /etc/hosts

Add an entry for the vFiler to /etc/hosts:

[root@kratos iscsi]# cat /etc/hosts
127.0.0.1	localhost.localdomain	localhost
::1	kratos.phas.ubc.ca	kratos	localhost6.localdomain6	localhost6
10.91.134.136	PHAS_NASpool

[root@kratos ~]# ifconfig -a
eth0      Link encap:Ethernet  HWaddr E4:1F:13:B9:DD:C8
          inet addr:142.103.236.15  Bcast:142.103.236.255  Mask:255.255.255.0
          inet6 addr: fe80::e61f:13ff:feb9:ddc8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2484 errors:0 dropped:0 overruns:0 frame:0
          TX packets:263 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:422959 (413.0 KiB)  TX bytes:31491 (30.7 KiB)
          Interrupt:28 Memory:96000000-96012800

eth1      Link encap:Ethernet  HWaddr E4:1F:13:B9:DD:CA
          inet addr:10.20.14.15  Bcast:10.20.14.255  Mask:255.255.255.0
          inet6 addr: fe80::e61f:13ff:feb9:ddca/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:26 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:6164 (6.0 KiB)
          Interrupt:40 Memory:98000000-98012800

eth2      Link encap:Ethernet  HWaddr 5C:F3:FC:39:EC:3C
          inet addr:192.168.1.15  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::5ef3:fcff:fe39:ec3c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1243 errors:0 dropped:0 overruns:0 frame:0
          TX packets:33 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:207182 (202.3 KiB)  TX bytes:6618 (6.4 KiB)
          Interrupt:29 Memory:92000000-92012800

[root@kratos ~]# route add -net 10.91.134.128 netmask 255.255.255.128 gw 10.20.14.254

[root@kratos ~]# netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.91.134.128   10.20.14.254    255.255.255.128 UG        0 0          0 eth1
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth2
142.103.236.0   0.0.0.0         255.255.255.0   U         0 0          0 eth0
192.168.122.0   0.0.0.0         255.255.255.0   U         0 0          0 virbr0
10.20.14.0      0.0.0.0         255.255.255.0   U         0 0          0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth2
0.0.0.0         142.103.236.254 0.0.0.0         UG        0 0          0 eth0

[root@kratos ~]# ping 10.91.134.136
PING 10.91.134.136 (10.91.134.136) 56(84) bytes of data.
64 bytes from 10.91.134.136: icmp_seq=1 ttl=254 time=1001 ms

Install iSCSI Drivers

# yum install iscsi-initiator-utils

Update iSCSI Configuration

Update initiator name to "iqn.1994-05.com.redhat:<fqdn>" where <fqdn> is replaced with the fully qualified domain name of the host ( kratos.phas.ubc.ca in this example).

# cd /etc/iscsi
# cp -p initiatorname.iscsi initiatorname.iscsi.ORIG
# echo "InitiatorName=iqn.1994-05.com.redhat:kratos.phas.ubc.ca" > initiatorname.iscsi

[root@kratos iscsi]# cat initiatorname.iscsi
InitiatorName=iqn.1994-05.com.redhat:kratos.phas.ubc.ca

Configure CHAP authentication, CHAP Credentials, and login timeouts:

# cp -p iscsid.conf iscsid.conf.ORIG
# vi iscsid.conf    <-- Uncomment following lines and set to shown values

node.session.auth.authmethod = CHAP
node.session.auth.username = phasiscsi
node.session.auth.password = pHas_iscsi01
discovery.sendtargets.auth.authmethod = CHAP
discovery.sendtargets.auth.username = phasiscsi
discovery.sendtargets.auth.password = pHas_iscsi01
node.session.timeo.replacement_timeout = 5

Start iSCSI Service, Discover Targets, Verify

Enable the iSCSI and netfs services at boot. Start the iSCSI service.

# chkconfig iscsi on
# chkconfig iscsid on
# chkconfig netfs on
# service iscsid start
Turning off network shutdown.                              [  OK  ]
Starting iSCSI daemon:                                     [  OK  ]

Discover targets:

[root@kratos iscsi]# iscsiadm --mode discovery --type sendtargets --portal 10.91.134.136
10.91.134.136:3260,2019 iqn.1992-08.com.netapp:sn.151709189:vf.893008c2-8c80-11e0-b2b4-00a098113b7a

If more than one target is returned, you must delete the unwanted targets (all except the 10.91.134.136 interface in this example).
Use the "delete" operation to delete all the unwanted targets. Verify only the desired target(s) remain.

# iscsiadm -m node --portal 172.20.24.227 --op delete

[root@kratos iscsi]# iscsiadm --m node
10.91.134.136:3260,2019 iqn.1992-08.com.netapp:sn.151709189:vf.893008c2-8c80-11e0-b2b4-00a098113b7a

Now log into the target:

[root@kratos iscsi]# iscsiadm --m node -l
Logging in to [iface: default, target: iqn.1992-08.com.netapp:sn.151709189:vf.893008c2-8c80-11e0-b2b4-00a098113b7a, portal: 10.91.134.136,3260]
Login to [iface: default, target: iqn.1992-08.com.netapp:sn.151709189:vf.893008c2-8c80-11e0-b2b4-00a098113b7a, portal: 10.91.134.136,3260] successful.

Verify that the LUNs on the target were successfully connected.

[root@kratos iscsi]# iscsiadm -m session -P 3
iSCSI Transport Class version 2.0-870
version 2.0-872
Target: iqn.1992-08.com.netapp:sn.151709189:vf.893008c2-8c80-11e0-b2b4-00a098113b7a
	Current Portal: 10.91.134.136:3260,2019
	Persistent Portal: 10.91.134.136:3260,2019
		**********
		Interface:
		**********
		Iface Name: default
		Iface Transport: tcp
		Iface Initiatorname: iqn.1994-05.com.redhat:kratos.phas.ubc.ca
		Iface IPaddress: 10.20.14.15
		Iface HWaddress: 
		Iface Netdev: 
		SID: 1
		iSCSI Connection State: LOGGED IN
		iSCSI Session State: LOGGED_IN
		Internal iscsid Session State: NO CHANGE
		************************
		Negotiated iSCSI params:
		************************
		HeaderDigest: None
		DataDigest: None
		MaxRecvDataSegmentLength: 262144
		MaxXmitDataSegmentLength: 65536
		FirstBurstLength: 65536
		MaxBurstLength: 65536
		ImmediateData: Yes
		InitialR2T: No
		MaxOutstandingR2T: 1
		************************
		Attached SCSI devices:
		************************
		Host Number: 11	State: running
		scsi11 Channel 00 Id 0 Lun: 0
			Attached scsi disk sdb		State: running
		scsi11 Channel 00 Id 0 Lun: 1
			Attached scsi disk sdc		State: running

The above output shows that the LUN mapped to ID 0 is connected as /dev/sdb, and the LUN mapped to ID 1 is connected as /dev/sdc

Configure Device Mapper

Configure device mapper so a lun is always given a consistent and meaningful device name based on it's wwid. Once this step is complete the lun will always be assigned a consistent name under /dev/mapper, regardless of which LUN ID it is mapped to or which "/dev/sd?" device it is assgigned to.

/vol/ARTS_DBPROD_LOG/VGdb_LOG/PVlog1.lun mapped to LUN ID 0 is assigned /dev/mapper/VGdb_LOG-PV01 (currently assigned /dev/sdb)
/vol/ARTS_DBPROD_DATA/VGdb_DATA/PVdata1.lun mapped to LUN ID 10 is assigned /dev/mapper/VGdb_DATA-PV01 (currently assigned /dev/sdc)

Ensure "device-mapper-multipath" package is installed:

# yum install device-mapper-multipath

Retrieve the "wwid" of each lun:

[root@kratos iscsi]# /lib/udev/scsi_id --page=0x83 --whitelisted --device=/dev/sdb
360a98000572d4534694a646149766c41
[root@kratos iscsi]# /lib/udev/scsi_id --page=0x83 --whitelisted --device=/dev/sdc
360a98000572d4534694a646149767970

Save a copy of /etc/multipath.conf, then update it and assign a device mapper alias to each of the above wwid's

# cd /etc

# cp -p multipath.conf multipath.conf.ORIG
OR
# cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults multipath.conf

# vi multipath.conf

# Replace the entire contents with the following configuration:
defaults {
        user_friendly_names     yes
}
blacklist {
        devnode "^sda$"
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^hd[a-z]"
        devnode "^dcssblk[0-9]*"
}
##
## name  : devices,  list of per storage controller settings
##
devices {
        device {
                vendor                  "NETAPP"
                product                 "LUN.*"
                getuid_callout          "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
                features                "1 queue_if_no_path"
                hardware_handler        "0"
                path_selector           "round-robin 0"
                path_grouping_policy    multibus
                failback                immediate
                rr_weight               uniform
                rr_min_io               128
                path_checker            directio
                prio                    ontap
        }
}
multipaths {
        #
        # name  : multipath
        # scope : multipath & multipathd
        # desc  : container for settings that apply to one specific multipath
        #
        # map /vol/...  to /dev/mapper/VG01-PV01
        multipath {
                wwid    360a98000572d4534694a646149766c41
                alias   VG01-PV01
        }
        # map /vol/...  to /dev/mapper/VG01-PV02
        multipath {
                wwid    360a98000572d4534694a646149767970
                alias   VG01-PV02
        }
}

Start and check multipathd

[root@kratos iscsi]# chkconfig multipathd on
[root@kratos iscsi]# service multipathd start
[root@kratos iscsi]# multipath -ll
VG01-PV02 (360a98000572d4534694a646149767970) dm-3 NETAPP,LUN
size=64G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=2 status=active
  `- 11:0:0:1 sdc 8:32 active ready running
VG01-PV01 (360a98000572d4534694a646149766c41) dm-4 NETAPP,LUN
size=64G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=2 status=active
  `- 11:0:0:0 sdb 8:16 active ready running

Configure Logical Volume Manager (LVM)

Initialize the LUNs as LVM physical volumes:

[root@kratos etc]# pvcreate /dev/mapper/VG01-PV01
  Physical volume "/dev/mapper/VG01-PV01" successfully created
[root@kratos etc]# pvcreate /dev/mapper/VG01-PV02
  Physical volume "/dev/mapper/VG01-PV02" successfully created

Create volume groups and add physical volumes

[root@kratos etc]# vgcreate VG01 /dev/mapper/VG01-PV01
  Volume group "VG01" successfully created

Create logical volumes that use all space in the volume group

[root@kratos etc]# lvcreate -l 100%VG -n LVhome VG01
  Logical volume "LVhome" created

[root@kratos etc]# /sbin/mkfs -t ext4 /dev/VG01/LVhome
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
4194304 inodes, 16777216 blocks
838860 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
512 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424

Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 22 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

[root@kratos etc]# vgextend VG01 /dev/mapper/VG01-PV02
  Volume group "VG01" successfully extended
[root@kratos etc]# tune2fs -c0 -i0 /dev/VG01/LVhome
tune2fs 1.41.12 (17-May-2010)
Setting maximal mount count to -1
Setting interval between checks to 0 seconds

Create the Mountpoints:

# mkdir /homes

Add the following lines to /etc/fstab. Note that the _netdev mount option defers the mount of the filesystem until the network is up and running:

/dev/VG01/LVhome	/homes			ext4	_netdev	0 0

Mount and check filesystem(s):

[root@kratos ~]# mount /homes
[root@kratos ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_kratos-lv_root
                       50G  3.0G   44G   7% /
tmpfs                  12G   88K   12G   1% /dev/shm
/dev/sda3             485M   56M  405M  13% /boot
/dev/sda2             200M  256K  200M   1% /boot/efi
/dev/mapper/vg_kratos-lv_opt
                       59G  180M   56G   1% /opt
/dev/mapper/VG01-LVhome
                      126G  188M  120G   1% /homes

Reboot and check mounts...

Reboot the server - be sure all filesystems are available after the reboot....

11-09-22 Adding more iSCSI storage