Information Technology: May 2014

Tuesday, May 20, 2014

HP-UX networking related tools and commands

HP-UX networking related tools and commands,

Recently I had to mount six instances on a Linux host. Everything was fine until I started the fourth instance:

ORA-00845: MEMORY_TARGET not supported on this system

The alert.log file had the following information:

WARNING: You are trying to use the MEMORY_TARGET feature. This feature requires the /dev/shm file system to be mounted for at least 1073741824 bytes. /dev/shm is either not mounted or is mounted with available space less than this size. Please fix this so that MEMORY_TARGET can work as expected. Current available is 899670016 and used is 2236674048 bytes. Ensure that the mount point is /dev/shm for this directory.

Obliviously the solution is mount a temporary file system with a larger size:

[root@nova Desktop]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_nova-lv_root
                       50G  3.7G   44G   8% /
tmpfs                 2.0G  1.4G  646M  68% /dev/shm
/dev/sda1             485M   55M  405M  12% /boot
/dev/mapper/vg_nova-lv_u01
                      439G  208G  210G  50% /u01
[root@nova Desktop]# mount -t tmpfs shmfs -o size=12g /dev/shm
[root@nova Desktop]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_nova-lv_root
                       50G  3.7G   44G   8% /
tmpfs                  12G     0   12G   0% /dev/shm
/dev/sda1             485M   55M  405M  12% /boot
/dev/mapper/vg_nova-lv_u01
                      439G  208G  210G  50% /u01
shmfs                  12G     0   12G   0% /dev/shm
[root@nova Desktop]#

The shared memory file system should be big enough to accommodate the MEMORY_TARGET and MEMORY_MAX_TARGET values, or Oracle will throw the ORA-00845 error. By default, a tmpfs partition has its maximum size set to half of the total RAM. Note that the actual memory/swap consumption depends on how much is it filled up, as tmpfs partitions do not consume any memory until it is actually needed.

When changing something with the mount command, the changes are not permanent. To make the change persistent, the /etc/fstab file must be edited to include the option:

[root@oracle-em ~]# cat /etc/fstab
[..]
tmpfs            /dev/shm         tmpfs   defaults,size=12g        0 0

Note 1: There is a registered bug on Redhat and OEL that prevents the new size to be loaded from the fstab file (https://bugzilla.redhat.com/show_bug.cgi?id=669700), its solved installing the package mentioned on that link. The Update detail for the bug 669700 is:

“Prior to this update, the /dev/shm file system was mounted by the dracut utility without attributes from the /etc/fstab file. To fix this bug, /dev/shm is now remounted by the rc.sysinit script. As a result, /dev/shm now contains the attributes from /etc/fstab.”

One workaround suggested that could be used instead of the update package is add the following line in the script /etc/rc.d/rc.local:

mount -o remount /dev/shm

Workaround Note 2: The mount command should not have the size option added or it will show an error:

[root@nova rc.d]# mount -o remount,size=6GB /dev/shm
mount: /dev/shm not mounted already, or bad option

Thursday, May 8, 2014

Linux ate my memory

Linux ate my memory,

What's going on?

Linux is borrowing unused memory for disk caching. This makes it looks like you are low on memory, but you are not! Everything is fine!

Why is it doing this?

Disk caching makes the system much faster! There are no downsides, except for confusing newbies. It does not take memory away from applications in any way, ever!

What if I want to run more applications?

If your applications want more memory, they just take back a chunk that the disk cache borrowed. Disk cache can always be given back to applications immediately! You are not low on ram!

Do I need more swap?

No, disk caching only borrows the ram that applications don't currently want. It will not use swap. If applications want more memory, they just take it back from the disk cache. They will not start swapping.

How do I stop Linux from doing this?

You can't disable disk caching. The only reason anyone ever wants to disable disk caching is because they think it takes memory away from their applications, which it doesn't! Disk cache makes applications load faster and run smoother, but it NEVER EVER takes memory away from them! Therefore, there's absolutely no reason to disable it!

Why does top and free say all my ram is used if it isn't?

This is just a misunderstanding of terms. Both you and Linux agree that memory taken by applications is "used", while memory that isn't used for anything is "free".
But what do you call memory that is both used for something and available for applications?
You would call that "free", but Linux calls it "used".

Memory that is	You'd call it	Linux calls it
taken by applications	Used	Used
available for applications, and used for something	Free	Used
not used for anything	Free	Free

This "something" is what top and free calls "buffers" and "cached". Since your and Linux's terminology differs, you think you are low on ram when you're not.

How do I see how much free ram I really have?

To see how much ram is free to use for your applications, run free -m and look at the row that says "-/+ buffers/cache" in the column that says "free". That is your answer in megabytes:

$ free -m
             total       used       free     shared    buffers     cached
Mem:          1504       1491         13          0         91        764
-/+ buffers/cache:        635        869
Swap:         2047          6       2041
$

If you don't know how to read the numbers, you'll think the ram is 99% full when it's really just 42%.

How can I verify these things?

See this page for more details and how you can experiment with disk cache.

Wednesday, May 7, 2014

How do I configure device-mapper-multipath?

Solution:

1. Verify that your device-mapper and multipath driver are at least the version shown below or higher by running the command: rpm -qa | grep device-mapperdevice-mapper-1.02.28-2.el5

device-mapper-multipath-0.4.7-23.el5.4.0.1
device-mapper-event-1.02.28-2.el5

2. Identify your local disks i.e. /dev/sda. Once your local disk is determined run the following command to obtain its scsi_id: scsi_id –gus /block/sda

360024e805314c20011faf391033bdece

3. Open the /etc/multipath.conf file and locate and comment out the section below:

#blacklist {
# devnode “*”
#}
4. Once the scsi_id of your local disk has been retrieved, you must blacklist this scsi_id from being used as a multipath device. Within /etc/multipath.conf file and locate, uncomment, and modify the section below:

blacklist {
    wwid <enter your local disk scsi_id here>
    devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
    devnode "^hd[a-z]"
}

5. Uncomment your defaults section within your /etc/multipath.conf

defaults {
    udev_dir /dev
    polling_interval 10
    selector "round-robin 0"
    path_grouping_policy multibus
    getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
    prio_callout /bin/true
    path_checker readsector0
    rr_min_io 100
    max_fds 8192
    rr_weight priorities
    failback immediate
    no_path_retry fail
    user_friendly_names yes
}

6. Locate the multipaths section within your /etc/multipath.conf file. In this section you will provide the scsi_id of each iSCSI volume and provide an alias in order to keep a consistent naming convention across all of your OVM Server nodes. An example is shown below:multipaths {
    multipath {
        wwid <scsi_id of volume1>
        alias alias_of_volume1
    }
    multipath {
        wwid <scsi_id of volume2>
        alias alias_of_volume2
    }
}

NOTE:

Additional multipath subsections can be added to meet the number of volumes you need.

7. Restart your multipath daemon service using the following command:

service multipathd restart

8. Verify that your multipath volumes alias are displayed properly by running the following command:multipath –ll

Alias_of_volume1 (scsi_id of volume1) dm-2 EQLOGIC,100E-00
[size=220G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
\_ 6:0:0:0 sdaj 66:48 [active][ready]
\_ 4:0:0:0 sdag 66:0 [active][ready]
\_ 3:0:0:0 sdd 8:48 [active][ready]
\_ 5:0:0:0 sdn 8:208 [active][ready]

9. Make sure the iSCSI service starts upon boot using the command:
chkconfig multipathd on

Installing ZFS on a CentOS 6 Linux server

Installing ZFS on a CentOS 6 Linux server,

As most of my long term readers know I am a huge Solaris fan. How can’t you love an Operating System that comes with ZFS, DTrace, Zones, FMA and Network Virtualization amongst other things? I use Linux during my day job, and I’ve been hoping for quite some time that Oracle would port one or more of these technologies to Linux. Well the first salvo has been fired, though it wasn’t from Oracle. It comes by way of the ZFS on Linux project, which is an in-kernel implementation of ZFS (this project is different from the FUSE ZFS port).

I had some free time this weekend to play around with ZFS on Linux, and my initial impressions are quite positive. The port on Linux is based on the latest version of ZFS that is part of OpenSolaris (version 28), so things like snapshots, de-duplication, improved performance and ZFS send and recv are available out of the box. There are a few missing items, but from what I can tell from the documentation there is plenty more coming.

The ZFS file system for Linux comes as source code, which you build into loadable kernel modules (this is how they get around the license incompatibilities). The implementation also contains the userland utilities (zfs, zpool, etc.) most Solaris admins are used to, and they act just like their Solaris counterparts! Nice!

My testing occurred on a CentOS 6 machine, specifically 6.2:

$ cat /etc/redhat-release
CentOS release 6.2 (Final)

The build process is quite easy. Prior to compiling source code you will need to install a few dependencies:

$ yum install kernel-devel zlib-devel libuuid-devel libblkid-devel libselinux-devel parted lsscsi

Once these are installed you can retrieve and build spl and zfs packages:

$ wget http://github.com/downloads/zfsonlinux/spl/spl-0.6.0-rc6.tar.gz

$ tar xfvz spl-0.6.0-rc6.tar.gz && cd spl*6

$ ./configure && make rpm

$ rpm -Uvh *.x86_64.rpm

Preparing...                ########################################### [100%]
   1:spl-modules-devel      ########################################### [ 33%]
   2:spl-modules            ########################################### [ 67%]
   3:spl                    ########################################### [100%]

$ wget http://github.com/downloads/zfsonlinux/zfs/zfs-0.6.0-rc6.tar.gz

$ tar xfvz zfs-0.6.0-rc6.tar.gz && cd zfs*6

$ ./configure && make rpm

$ rpm -Uvh *.x86_64.rpm

Preparing...                ########################################### [100%]
   1:zfs-test               ########################################### [ 17%]
   2:zfs-modules-devel      ########################################### [ 33%]
   3:zfs-modules            ########################################### [ 50%]
   4:zfs-dracut             ########################################### [ 67%]
   5:zfs-devel              ########################################### [ 83%]
   6:zfs                    ########################################### [100%]

If everything went as planned you now have the ZFS kernel modules and userland utilities installed! To begin using ZFS you will first need to load the kernel modules with modprobe:

$ modprobe zfs

To verify the module loaded you can tail /var/log/messages:

Feb 12 17:54:27 centos6 kernel: SPL: Loaded module v0.6.0, using hostid 0x00000000
Feb 12 17:54:27 centos6 kernel: zunicode: module license 'CDDL' taints kernel.
Feb 12 17:54:27 centos6 kernel: Disabling lock debugging due to kernel taint
Feb 12 17:54:27 centos6 kernel: ZFS: Loaded module v0.6.0, ZFS pool version 28, ZFS filesystem version 5

And run lsmod to verify they are there:

$ lsmod | grep -i zfs

zfs                  1038053  0 
zcommon                42478  1 zfs
znvpair                47487  2 zfs,zcommon
zavl                    6925  1 zfs
zunicode              323120  1 zfs
spl                   210887  5 zfs,zcommon,znvpair,zavl,zunicode

To create our first pool we can use the zpool utilities create option:

$ zpool create mysqlpool mirror sdb sdc

The example above created a mirrored pool out of the sdb and sdc block devices. We can see this layout in the output of `zpool status`:

$ zpool status -v

  pool: mysqlpool
 state: ONLINE
 scan: none requested
config:

 NAME        STATE     READ WRITE CKSUM
 mysqlpool   ONLINE       0     0     0
   mirror-0  ONLINE       0     0     0
     sdb     ONLINE       0     0     0
     sdc     ONLINE       0     0     0

errors: No known data errors

Awesome! Since we are at pool version 28 lets disable atime updates and enable compression and deduplication:

$ zfs set compression=on mysqlpool

$ zfs set dedup=on mysqlpool

$ zfs set atime=off mysqlpool

For a somewhat real world test, I stopped one of my MySQL slaves, mounted the pool on /var/lib/mysql, synchronized the previous data over to the ZFS file system and then started MySQL. No errors to report, and MySQL is working just fine. Next up, I trash one side of the mirror and verified that resilvering works:

$ dd if=/dev/zero of=/dev/sdb

$ zpool scrub mysqlpool

I let this run for a few minutes then ran `zpool status` to verify the scrub fixed everything:

$ zpool status -v

  pool: mysqlpool
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
 attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
 using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-9P
 scan: scrub repaired 966K in 0h0m with 0 errors on Sun Feb 12 18:54:51 2012
config:

 NAME        STATE     READ WRITE CKSUM
 mysqlpool   ONLINE       0     0     0
   mirror-0  ONLINE       0     0     0
     sdb     ONLINE       0     0   175
     sdc     ONLINE       0     0     0

I beat on the pool pretty good and didn’t encounter any hangs or kernel oopses. The file systems port is still in its infancy, so I won’t be trusting it with production data quite yet. Hopefully it will mature in the coming months, and if we’re lucky maybe one of the major distributions will begin including it! That would be killer!!

Getting E-mail notifications when MD devices fail

I use the MD (multiple device) logical volume manager to mirror the boot devices on the Linux servers I support. When I first started using MD, the mdadm utility was not available to manage and monitor MD devices. Since disk failures are relatively common in large shops, I used the shell script from my SysAdmin article Monitoring and Managing Linux Software RAID to send E-mail when a device entered the failed state. While reading through the mdadm(8) manual page, I came across the “–monitor” and “–mail” options. These options can be used to monitor the operational state of the MD devices in a server, and generate E-mail notifications if a problem is detected. E-mail notification support can be enabled by running mdadm with the “–monitor” option to monitor devices, the “–daemonise” option to create a daemon process, and the “–mail” option to generate E-mail:

$ /sbin/mdadm –monitor –scan –daemonise –mail=root@localhost

Once mdadm is daemonized, an E-mail similar to the following will be sent each time a failure is detected:

From: mdadm monitoring 
To: root@localhost.localdomain
Subject: Fail event on /dev/md1:biscuit

This is an automatically generated mail message from mdadm
running on biscuit

A Fail event had been detected on md device /dev/md1.

Faithfully yours, etc.

I digs me some mdadm!

Locating Linux LVM (Logical Volume Manager) free space

The Linux Logical Volume Manager (LVM) provides a relatively easy way to combine block devices into a pool of storage that you can allocate storage out of. In LVM terminology, there are three main concepts:

Physical Volumes – A sequence of sectors on a physical device.
Volume Groups – A group of physical volumes.
Logical Volumes – A logical device that is allocated from a volume group.

When you use LVM to manage your storage, you will typically do something similar to this when new storage requests are made:

1. Create a physical volume on a block device or partition on a block device.

2. Add one or more physical volumes to a volume group

3. Allocate logical volumes from the volume group.

4. Create a file system on the logical volume.

With this approach you can end up with free space in one or more physical volumes or one or more volume groups depending on how you provisioned the storage. To see how much free space your physical volumes have you can run the pvs utility without any arguments:

$ pvs

  PV         VG       Fmt  Attr PSize  PFree  
  /dev/sda2  VolGroup lvm2 a--   8.51g      0 
  /dev/sdb   DataVG   lvm2 a--  18.00g  18.00g
  /dev/sdc   DataVG   lvm2 a--  18.00g 184.00m

The “PFree” column shows the free space for each physical volume in the system. To see how much free space your volume groups have you can run the vgs utility without any arguments:

$ vgs

  VG       #PV #LV #SN Attr   VSize  VFree 
  DataVG     2   1   0 wz--n- 35.99g 18.18g
  VolGroup   1   2   0 wz--n-  8.51g     0

In the vgs output the “VFree” column shows the amount of free space in each volume group. LVM is nice, but I’m definitely a ZFS fan when it comes to storage management. I’m hopeful that Oracle will come around and port ZFS to Linux, since it would benefit a lot of users and hopefully help to repair some of the broken relations between Oracle and the opensource community. I may be too much of an optimist though.

Speeding up Oracle disk I/O on RHEL4 systems

While poking around the web last wek, I came across a good paper from Redhat that describes how to utilize asynchronous and direct I/O with Oracle. I have been using the Oracle filesystemio_options=”SetAll” initialization parameter on a few RHEL 4 database servers to efficiently use memory, and had no idea idea that it provided the throughput numbers listed in Figure 2. Nice!

Monitoring Linux server performance with procallator

Monitoring Linux server performance with procallator,

I manage a fair number of Linux hosts, and like to keep tabs on how my systems are performing. One way I accomplish this is with procallator, which is a Perl script that collects performance data that can be graphed by orca. The graphs that orca produces are great awesome for trening server performance over time, and can be extremely valuable when debugging performance problems.

To setup procallator to collect performance data, you first need to retrieve the latest orca CVS snapshot from the orcaware snapshots directory (the procallator script is included with the orca snapshot, and the latest version contains a number of fixes). Once orca is downloaded, you will need to extract the tarball and run configure to modify the variables in the header of the procallator script:

$ tar xfj orca-snapshot-r529.tar.bz2

$ cd orca-snapshot-r529

$ ./configure –prefix=/opt/orca-r529 –with-html-dir=/opt/html

After the configure operation completes, you can install the procallator scripts with the Makefile’s install option:

$ make install

This will place the procallator perl script in $PREFIX/bin. To make sure the script starts at system boot, you can copy the $PREFIX/data_gathers/procallator/S99procallator script to /etc/rc3.d (or /etc/init.d depending on how you install your init scripts):

$ cp S99procallator /etc/rc3.d

Once these files are in place, you can start procallator by invoking the init script with the start option:

$ /etc/rc3.d/S99procallator start

This will start the procallator script as a daemon process, and the script will write performance data to the directory defined in the procallator script’s DEST_DIR variable every 5 minutes (this is tunable). The performance files will contain the name proccol-YYYY-MM-DD-INDEX, and one file will be produced each day. To graph the data in the procallator files, you can use orca and the procallator.cfg file that is in the $PREFIX/data_gathers/procallator directory. I placed a sample set of performance graphs on my website, and you can reference the article monitoring LDAP performance article for details on setting up orca to graph data. I digs me some procallator!

Why isn’t Oracle using huge pages on my Redhat Linux server?

I am currently working on upgrading a number of Oracle RAC nodes from RHEL4 to RHEL5. After I upgraded the first node in the cluster, my DBA contacted me because the RHEL5 node was extremely sluggish. When I looked at top, I saw that a number of kswapd processes were consuming CPU:

$ top

top - 18:04:20 up 6 days,  3:22,  7 users,  load average: 14.25, 12.61, 14.41
Tasks: 536 total,   2 running, 533 sleeping,   0 stopped,   1 zombie
Cpu(s): 12.9%us, 19.2%sy,  0.0%ni, 20.9%id, 45.0%wa,  0.1%hi,  1.9%si,  0.0%st
Mem:  16373544k total, 16334112k used,    39432k free,     4916k buffers
Swap: 16777208k total,  2970156k used, 13807052k free,  5492216k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                        
  491 root      10  -5     0    0    0 D 55.6  0.0  67:22.85 kswapd0                         
  492 root      10  -5     0    0    0 S 25.8  0.0  37:01.75 kswapd1                         
  494 root      11  -5     0    0    0 S 24.8  0.0  42:15.31 kswapd3                         
 8730 oracle    -2   0 8352m 3.5g 3.5g S  9.9 22.4 139:36.18 oracle                          
 8726 oracle    -2   0 8352m 3.5g 3.5g S  9.6 22.5 138:13.54 oracle                          
32643 oracle    15   0 8339m  97m  92m S  9.6  0.6   0:01.31 oracle                          
  493 root      11  -5     0    0    0 S  9.3  0.0  43:11.31 kswapd2                         
 8714 oracle    -2   0 8352m 3.5g 3.5g S  9.3 22.4 137:14.96 oracle                          
 8718 oracle    -2   0 8352m 3.5g 3.5g S  8.9 22.3 137:01.91 oracle                          
19398 oracle    15   0 8340m 547m 545m R  7.9  3.4   0:05.26 oracle                          
 8722 oracle    -2   0 8352m 3.5g 3.5g S  7.6 22.5 139:18.33 oracle

The kswapd process is responsible for scanning memory to locate free pages, and scheduling dirty pages to be written to disk. Periodic kswapd invocations are fine, but seeing kswapd continuosly appearing in the top output is a really really bad thing. Since this host should have had plenty of free memory, I was perplexed by the following output (the free output didn’t match up with the values on the other nodes):

$ free

             total       used       free     shared    buffers     cached
Mem:      16373544   16268540     105004          0       1520    5465680
-/+ buffers/cache:   10801340    5572204
Swap:     16777208    2948684   13828524

To start debugging the issue, I first looked at ipcs to see how much shared memory the database allocated. In the output below, we can see that there is a 128MB and a 8GB shared memory segment allocated:

$ ipcs -a

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x62e08f78 0          oracle    640        132120576  16                      
0xdd188948 32769      oracle    660        8592031744 87

The first segment is dedicated to the Oracle ASM instance, and the second to the actual database. When I checked the number of huge pages allocated to the machine, I saw something a bit odd:

$ grep Huge cat /proc/meminfo

HugePages_Total:  4106
HugePages_Free:   4051
HugePages_Rsvd:      8
Hugepagesize:     2048 kB

While our DBA had set vm.nr_hugepages to a sufficiently large value in /etc/syscl.conf, the database was utilizing a very small portion of them. This meant that the database was being allocated out of non huge page memory (Linux dedicates memory to the huge page area, and it is wasted if nothing utilizes it), and inactive pages were being paged out to disk since the database wasn’t utilizing the huge page area we reserved for it . After a bit of bc’ing (I love doing my calculations with bc), I noticed that the total amount of memory allocated to huge pages was 8610906112 bytes:

$ grep vm.nr_hugepages /etc/sysctl.conf
vm.nr_hugepages=4106

$ bc
4106*(1024*1024*2)
8610906112

If we add the totals from the two shared memory segments above:

$ bc
8592031744+132120576
8724152320

We can see that we don’t have enough huge page memory to support both shared memory segments. Yikes! After adjusting vm.nr_hugepages to account for both databases, the system no longer swapped and database performance increased. This debugging adventure taught me a couple of things:

1. Double check system values people send you

2. Solaris does a MUCH better job of handling large page sizes (huge pages are used transparently)

3. The Linux tools for investigating huge page allocations are severely lacking

4. Oracle is able to allocate a continuos 8GB chunk of shared memory on RHEL5, but not RHEL4 (I need to do some research to find out why)

Hopefully more work will go into the Linux huge page implementation, and allow me to scratch the second and third items off of my list. Viva la problem resolution!

Getting an accurate view of process memory usage on Linux hosts

Having debugged a number of memory-related issues on Linux, one thing I’ve always wanted was a tool to display proportional memory usage. Specifically, I wanted to be able to see how much memory was unique to a process, and have an equal portion of shared memory (libraries, SMS, etc.) added to this value. My wish came true a while back when I discovered the smem utility. When run without any arguments, smem will give you the resident set size (RSS), the unique set size (USS) and the proportional set size (PSS) which is the unique set size plus a portion of the shared memory that is being used by this process. This results in output similar to the following:

$ smem -r

  PID User     Command                         Swap      USS      PSS      RSS 
 3636 root     /usr/lib/virtualbox/Virtual        0  1151596  1153670  1165568 
 3678 matty    /usr/lib64/firefox-3.5.9/fi        0   189628   191483   203028 
 5779 root     /usr/bin/python /usr/bin/sm        0    38584    39114    40368 
 1847 root     /usr/bin/Xorg :0 -nr -verbo        0    34024    35874    92504 
 4103 matty    pidgin                             0    19364    21072    32412 
 3825 matty    gnome-terminal                     0    12388    13242    21992 
 3404 matty    python /usr/share/system-co        0    11596    12622    19216 
 3710 matty    gnome-screensaver                  0     9872    10287    14640 
 3283 matty    nautilus                           0     7104     8373    18484 
 3263 matty    gnome-panel                        0     5828     6731    15780

To calculate the portion of shared memory that is being used by each process, you can add up the shared memory per process (you would probably index this by the type of shared resource), the number of processes using these pages, and then divide the two values to get a proportional value of shared memory per process. This is a very cool utility, and one that gets installed on all of my systems now!

Installing QLogic drivers on CentOS Linux hosts

I have a couple of hosts in my lab with QLA2342 HBAs, and use the drivers from QLogic instead of the drivers that come with the kernel. There are a number of reasons for this, but I’ll save that explanation for a future post. To install the QLogic drivers on a CentOS 5.3 host, you will first need to download the qlafc package from QLogic’s website. Once you retrieve the bits, you can run the qlinstall utility to build and install a qlaXxxx driver that works with the kernel you are running:

$ tar xfvz qlafc-linux-8.02.14.01-1-install.tgz

$ cd qlafc-linux-8.02.14.01-1-install

$ ls -la

total 11772
drwxr-xr-x 4 root  root     4096 Jul 10  2008 .
drwx------ 3 matty matty    4096 Apr 30 07:45 ..
drwxr-xr-x 4 root  root     4096 Jul 10  2008 agents
drwxr-xr-x 7 root  root     4096 Apr 30 07:47 LinuxTools
-rw-r--r-- 1 root  root  2376930 Jul 10  2008 qla2xxx-v8.02.14.01-1.noarch.rpm
-rwxr-xr-x 1 root  root   146155 Jul 10  2008 qlinstall
-rw-r--r-- 1 root  root     7321 Jul 10  2008 ql-pci.ids
-rw-r--r-- 1 root  root    14119 Jul 10  2008 README.qlinstall.txt
-rw-r--r-- 1 root  root  2905583 Jul 10  2008 scli-1.7.1-23.i386.rpm
-rw-r--r-- 1 root  root  3736548 Jul 10  2008 scli-1.7.1-23.ia64.rpm
-rw-r--r-- 1 root  root  2792038 Jul 10  2008 scli-1.7.1-23.ppc64.rpm
-rwxr-xr-x 1 root  root    18465 Jul 10  2008 set_driver_param

$ ./qlinstall

#*********************************************************#
#           SANsurfer Driver Installer for Linux          #
#             Installer Version:  1.01.00pre21        #
#*********************************************************#

Kernel version: 2.6.18-128.1.6.el5
Distribution: CentOS release 5.3 (Final)

Found following QLogic Adapter in the system
    1. QLA2342
Installation will begin for following driver
    1. qla2xxx version: v8.02.14.01


Unloading any loaded drivers
Unloaded module qla2xxx

Installing Driver...
Preparing...                ##################################################
qla2xxx                     ##################################################

QLA2XXX -- Building the qla2xxx driver, please wait...
Installing intermodule.ko in /lib/modules/2.6.18-128.1.6.el5/kernel/kernel/
 


QLA2XXX -- Installing the qla2xxx modules to 
/lib/modules/2.6.18-128.1.6.el5/kernel/drivers/scsi/qla2xxx/...

Setting up QLogic HBA API library...
Please make sure the /usr/lib/libqlsdm.so file is not in use.
Installing 32bit api binary for x86_64.
Installing 64bit api binary for x86_64.
Library 4.00 build12 already installed at /usr/lib/libqlsdm.so.
Done.

Loading module qla2xxx_conf version: v8.02.14.01....
Loaded module qla2xxx_conf
Loading module qla2xxx version: v8.02.14.01....
Loaded module qla2xxx


Building default persistent binding using SCLI
 
Saved copy of /etc/modprobe.conf as
/usr/src/qlogic/v8.02.14.01-1/backup/modprobe.conf-2.6.18-128.1.6.el5-043009-082949.bak

Saved copy of /boot/initrd-2.6.18-128.1.6.el5.img as
/boot/initrd-2.6.18-128.1.6.el5_QLI.org
QLA2XXX -- Rebuilding ramdisk image...
Ramdisk created.

Reloading the QLogic FC HBA drivers....
Unloaded module qla2xxx
Loading module qla2xxx_conf version: v8.02.14.01....
Loaded module qla2xxx_conf
Loading module qla2xxx version: v8.02.14.01....
Loaded module qla2xxx


Target Information on all HBAs:
==============================
 --------------------------------------------------------------------------------
HBA Instance 0: QLA2342 Port 1 WWPN 21-00-00-1B-32-04-86-C3 PortID 01-12-00
--------------------------------------------------------------------------------
No device connected to selected HBA (Instance 0)!
--------------------------------------------------------------------------------
HBA Instance 1: QLA2342 Port 2 WWPN 21-01-00-1B-32-24-86-C3 PortID 01-13-00
--------------------------------------------------------------------------------
No device connected to selected HBA (Instance 1)!
Installing the qlinstall-autoload script in /etc/init.d/ 

#*********************************************************#
#               INSTALLATION SUCCESSFUL!!                 #
#    SANsurfer Driver installation for Linux completed    #
#*********************************************************#

Once the driver is installed and loaded, you can verify the version by running the modinfo utility:

$ modinfo qla2xxx | head -10

filename:       /lib/modules/2.6.18-128.1.6.el5/kernel/drivers/scsi/qla2xxx/qla2xxx.ko
version:        8.02.14.01
license:        GPL
description:    QLogic Fibre Channel HBA Driver
author:         QLogic Corporation
srcversion:     90F727BC62E8CB20433AB36
alias:          pci:v00001077d00002532sv*sd*bc*sc*i*
alias:          pci:v00001077d00005432sv*sd*bc*sc*i*
alias:          pci:v00001077d00005422sv*sd*bc*sc*i*
alias:          pci:v00001077d00008432sv*sd*bc*sc*i*

QLogic includes a number of nifty utilities with their driver bundle, all of which I plan to blog about in future posts.

QLogic ISP2532-based 8Gb Fibre Channel

You shouldn't need to do anything as the driver is already available as part of the CentOS supplied kernel and modules. To check this, I looked up ISP2532 in /usr/share/hwdata/pci.ids and found that it has PCI Ventodr:Device ID 1077:2532. Then I ran

CODE: SELECT ALL: $ grep 1077 /lib/modules/2.6.18-348.1.1.el5xen/modules.* | grep 2532 /lib/modules/2.6.18-348.1.1.el5xen/modules.alias:alias pci:v00001077d00002532sv*sd*bc*sc*i* qla2xxx

This shows that the device is supported by the qla2xxx module. If running `lsmod | grep qla2xxx` on the running system returns no output, then try running `modprobe qla2xxx` and then look in /var/log/messages to see what appears at the end of the file. You might want to check the output from `lspci -nn` to make sure that your adapter has the same PCI Vendor:Device ID that I looked for.

Linux Bass-files Fiber Channel Multipath configuration

INSTALL Red Hat 5.3                     Update March 31, 2009
-------------------

Make sure the fibre hba cables are UNPLUGGED. The Red Hat installer gets
confused with the multiple paths to the 6140 StorageTek disk array.
There are 4 paths to each disk drive LUN.  We think this is because
the existing LUNS that are presented to the server are lvm logical
volumes. The server see's multiple paths to the same lvm physical
volume and this causes problems. Once the system sees the lvm physical
disks as one device things work properly. That is the qlogic and rdac
drivers need to be configured before the hba fibre cables are plugged
in. The RDAC drivers make the multiple paths to the LUNS's look like
one device. Note you can also disable the port in the Brocade switch if
you like but that is much more complicated. You can also use the Brocade
java gui to disable the ports.

Do a normal Red Hat 5 install. Do not plug the hba cables in yet! Do not
use the Red Hat multipath software. By default the /etc/multipath.conf
file has all disk devices disabled for use as multipath devices.

Install the Qlogic drivers first then Sun RDAC/MPP drivers.

NOTE: The Red Hat supplied qlogic drivers do not work,
      you need to use the Qlogic supplied drivers!

Get the latest Qlogic drivers at:

http://support.qlogic.com/

Click the downloads tab at the top of the page. Click on Sun under the
OEM MODELS heading.

Look for model SG-XPCIE2FC-QF4, click on "Software for:" Linux URL just
below the table!

Download the latest "Linux FC driver for 2.6 kernel (x86/x64/IA64)"
tarball file, for example:

        qla2xxx-v8.02.14_01-dist.tgz


There is also a "SANsurfer Linux Driver Installer (x86/x64/IA64)" download
that contains the SANsurfer command line utility and the driver. Do not
use this method for installing. Download the "SANsurfer
CLI (x86/x64)" rpm by itself and install:

        scli-1.7.1-23.i386.rpm.gz

gunzip scli-1.7.1-23.i386.rpm.gz
rpm -vih scli-1.7.1-23.i386.rpm

The scli command gets install in:

        /opt/QLogic_Corporation/SANsurferCLI/scli

Use the scli command in this directory to probe information about the
qlogic card WWPN World Wide Port Nodes. Use this utility after
you install the Qlogic driver.

Install the Qlogic provided driver. We will be using the Linux FC
driver for 2.6 download which is a compressed tar file. Do not use the
"qlinstall" command that comes with the "SANsurfer Linux Driver Installer
(x86/x64/IA64)" tarball! Here are the steps to instll the Qlogic driver
using the version as of this writing:

copy the downloaded driver software to /var/tmp and unpack:

# tar zxvf qla2xxx-v8.02.14_01-dist.tgz

The Qlogic installation instructions are in the file:

        README.qla2xxx


# cd qlogic
# ./drvrsetup
# cd qla2xxx-8.02.14
# ./extras/build.sh install

--- output example ---

bash-3.2# ./extras/build.sh install

QLA2XXX -- Building the qla2xxx driver, please wait...
Installing intermodule.ko in /lib/modules/2.6.18-92.1.10.el5/kernel/kernel/
QLA2XXX -- Build done.

QLA2XXX -- Installing the qla2xxx modules to 
/lib/modules/2.6.18-92.1.10.el5/kernel/drivers/scsi/qla2xxx/...

--- end output example ---

Place the following option in /etc/modprob.conf The RDAC driver cannot
co-exist with an HBA-level failover driver. Use echo or vi to update
/etc/modprobe.conf:

# echo "options qla2xxx ql2xfailover=0"  >> /etc/modprobe.conf

Optionaly make a new boot/intrd-*.img ram disk image.  You only need
to do this if you are troubleshooting before you install the Sun RDAC
drivers! The Sun RDAC build makes a new initrd-*.img file also.

# cd /boot

Save old image:

# cp initrd-`uname -r`.img initrd-`uname -r`.img.bak

Make new image to load qlogic driver on boot:

# mkinitrd -f initrd-`uname -r`.img `uname -r`

Reboot system, do not plug fibre cable in yet! Skip to the next step,
installing the RDAC drivers and do not reboot if you do not need to
troubleshoot the LUNS.

# init 6

Use the Qlogic scli command above to troubleshoot the
LUNS/StorageTek 6140.


INSTALL SUN RDAC/MPP
--------------------
Now get and install the Sun RDAC/MPP driver, Redundant Disk Array
Controller Driver (a.k.a. Multi-Path Proxy Driver or MPP)

AS of this writing you need to contact sun to get the latest linux driver.
Below is the output of the make install command.

# tar zxvf rdac-LINUX-09.02.C2.13-source.tar.gz
# cd linuxrdac-09.02.C2.13
# make
# make uninstall
# make install

If you do not do a "make uninstall" you will get this error if upgrading
to the kernel, the builds needs to remove all old files:

The system has old MPP driver package installed.
Please do "make uninstall" in the current directory before installing the new one.
make: *** [hbacheck] Error 1

Do as it says "make uninstall" then do "make install".
Also answer yes to the following question:

Host Adapters from different supported vendors co-exists on your system.
Please make sure that only one supported model of HBA is connected to Storage Array.
Do you want to continue (yes or no) ? yes

The installer will create a new mpp-kernel_release.img ram disk. Update
grub.conf to use this ramd disk, for example:

# vi /etc/grub.conf or /boot/grub/grub.conf

default=0
timeout=5
serial --unit=0 --speed=9600
terminal --timeout=5 serial console
title Red Hat Enterprise Linux Server SUN-RDAC (2.6.18-128.1.1.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-128.1.1.el5 ro root=/dev/sdasysvg/root
        console=ttyS0,9600n1 rhgb quiet
        initrd /mpp-2.6.18-128.1.1.el5.img

You can save the old kernel entries, make this entry the default boot
entry. This way you can boot the non RDAC kernel if you have problems.

Halt the system, "init 0". You can now attach the hba fibre cables.
Bring up the system. The lsmod command should show the qla drivers get
loaded and should look like this:

# lsmod|grep ql
qla2xxx              1009580  1 
qla2xxx_conf          335368  1 
intermodule            37508  2 qla2xxx,qla2xxx_conf
scsi_mod              188665  10 qla2xxx,sr_mod,usb_storage,mptsas,mptscsih,scsi_transport_sas,,sg,sd_mod


The disk LUNS will get presented as a single /dev/sd? device. The RDAC
driver coalesces the multiple paths to the disk array as one device.
You can use "fdisk -l" to list your disk devices that were created from
the LUNS. You can also use the RDAC command /opt/mpp/lsvdev to see
the Lun ot disk  mappings, very handy.

If you were using lvm logical volumes on the LUNS run the following
commands so the os will see the lvm physical volumes:

Scan for lvm physical volumes
# /sbin/pvscan

Verify the list of lvm physical volumes:
# /usr/sbin/pvs
# /usr/sbin/lvs -o +devices

You probably will have to turn the "available" attribute on for the
logical volumes after a fresh install. You can see if the lvm logic
device exists in /dev, for example "ls /dev/volume_group_name/*", If he
volume group device does not exist enable the available attribute for
all the volume groups with:

# lvm vgchange -ay volume_group volume_group1 ...

This should create the logical volume groups and volume device:

        /dev/volume_groupx/logical_volume_name

Again, use "ls /dev/volume_group_name/*" to see if the volume groups
got created

You can then mount the logical volume:
        mount /dev/volume_groupx/logical_volume_name /mount_point

Update the /etc/fstab for all your mounts and mount them.

Reboot the system and verify the boot process and mounts.


UPGRADING TO A NEW KERNEL
-------------------------

You can leave the fibre cables plugged in at this point. The Qlogic/RDAC
drivers only install to the kernel that is currently running.

Run yum to update your system and its kernel.

Comment out all StorageTek 6140 file systems that are mounted!  If you
forget to do this and boot the new kernel fsck will complain and drop
you into single user mode first asking you to enter the root password. If
you accidentally do this here is how to fix:

Enter root password and remount / as a read/write file system with:

        /bin/mount -o remount,rw /

edit /etc/fstab with vi and comment out the StorageTek file systems,
or execut "touch /fastboot" which creates and empty file that fsck
checks for and will bypass the fsck check.

Halt the system and remove the fibre cables from the system. Or user
the Brocade guid to disable the ports for the host.

Boot to the new kernel and install the Qlogic and RDAC drivers as
described above.

After checking the LUNS and volumes are present as described above
uncomment the /etc/fstab SotragTek mounts and re-mount the file systems.


QLOGIC SANSURFER UTILITY
------------------------

Get the qlafc-linux-8.02.08-1-install.tgz as described above and
unpack it:

tar ztvf qlafc-linux-8.02.08-1-install.tgz
cd qlafc-linux-8.02.08-1-install


Install the rpm for your platform, for example:

rpm -ivh scli-1.7.1-18.i386.rpm

The utility gets install at:

/opt/QLogic_Corporation/SANsurferCLI/scli



Some helful commands and file info:
----------------------------------

man RDAC, explains RDAC driver 

/opt/mpp/lsvdev command that displays lun to sd device mappings.

/proc/scsi/mpp/3, (or maybe some other number),  file that contains lun
and controller info.

mppUtil -S command, this will print all the multipaths, read mppUtil man page.

mppBusRescan command  run after mapping new lun so host registers it
on a live sytem if you do not want to reboot.

vgdisplay: Volume group not activated

Here is the full message being shown to me by the OS after invoking the vgdisplay command:

vgdisplay: Volume group not activated.
vgdisplay: Cannot display volume group "/dev/vg10".

If the server is NOT part of a cluster, use

# vgchange -a y vg10

# strings /etc/lvmtab

# more /etc/fstab

# mount <info from fstab>