Getting an accurate view of process memory usage on Linux hosts

Getting an accurate view of process memory usage on Linux hosts

Having debugged a number of memory-related issues on Linux, one thing I’ve always wanted was a tool to display proportional memory usage. Specifically, I wanted to be able to see how much memory was unique to a process, and have an equal portion of shared memory (libraries, SMS, etc.) added to this value. My wish came true a while back when I discovered the smem utility. When run without any arguments, smem will give you the resident set size (RSS), the unique set size (USS) and the proportional set size (PSS) which is the unique set size plus a portion of the shared memory that is being used by this process. This results in output similar to the following:

$ smem -r

  PID User     Command                         Swap      USS      PSS      RSS 
 3636 root     /usr/lib/virtualbox/Virtual        0  1151596  1153670  1165568 
 3678 matty    /usr/lib64/firefox-3.5.9/fi        0   189628   191483   203028 
 5779 root     /usr/bin/python /usr/bin/sm        0    38584    39114    40368 
 1847 root     /usr/bin/Xorg :0 -nr -verbo        0    34024    35874    92504 
 4103 matty    pidgin                             0    19364    21072    32412 
 3825 matty    gnome-terminal                     0    12388    13242    21992 
 3404 matty    python /usr/share/system-co        0    11596    12622    19216 
 3710 matty    gnome-screensaver                  0     9872    10287    14640 
 3283 matty    nautilus                           0     7104     8373    18484 
 3263 matty    gnome-panel                        0     5828     6731    15780 

To calculate the portion of shared memory that is being used by each process, you can add up the shared memory per process (you would probably index this by the type of shared resource), the number of processes using these pages, and then divide the two values to get a proportional value of shared memory per process. This is a very cool utility, and one that gets installed on all of my systems now!

Installing QLogic drivers on CentOS Linux hosts

Installing QLogic drivers on CentOS Linux hosts

I have a couple of hosts in my lab with QLA2342 HBAs, and use the drivers from QLogic instead of the drivers that come with the kernel. There are a number of reasons for this, but I’ll save that explanation for a future post. To install the QLogic drivers on a CentOS 5.3 host, you will first need to download the qlafc package from QLogic’s website. Once you retrieve the bits, you can run the qlinstall utility to build and install a qlaXxxx driver that works with the kernel you are running:

$ tar xfvz qlafc-linux-8.02.14.01-1-install.tgz

$ cd qlafc-linux-8.02.14.01-1-install

$ ls -la

total 11772
drwxr-xr-x 4 root  root     4096 Jul 10  2008 .
drwx------ 3 matty matty    4096 Apr 30 07:45 ..
drwxr-xr-x 4 root  root     4096 Jul 10  2008 agents
drwxr-xr-x 7 root  root     4096 Apr 30 07:47 LinuxTools
-rw-r--r-- 1 root  root  2376930 Jul 10  2008 qla2xxx-v8.02.14.01-1.noarch.rpm
-rwxr-xr-x 1 root  root   146155 Jul 10  2008 qlinstall
-rw-r--r-- 1 root  root     7321 Jul 10  2008 ql-pci.ids
-rw-r--r-- 1 root  root    14119 Jul 10  2008 README.qlinstall.txt
-rw-r--r-- 1 root  root  2905583 Jul 10  2008 scli-1.7.1-23.i386.rpm
-rw-r--r-- 1 root  root  3736548 Jul 10  2008 scli-1.7.1-23.ia64.rpm
-rw-r--r-- 1 root  root  2792038 Jul 10  2008 scli-1.7.1-23.ppc64.rpm
-rwxr-xr-x 1 root  root    18465 Jul 10  2008 set_driver_param

$ ./qlinstall

#*********************************************************#
#           SANsurfer Driver Installer for Linux          #
#             Installer Version:  1.01.00pre21        #
#*********************************************************#

Kernel version: 2.6.18-128.1.6.el5
Distribution: CentOS release 5.3 (Final)

Found following QLogic Adapter in the system
    1. QLA2342
Installation will begin for following driver
    1. qla2xxx version: v8.02.14.01


Unloading any loaded drivers
Unloaded module qla2xxx

Installing Driver...
Preparing...                ##################################################
qla2xxx                     ##################################################

QLA2XXX -- Building the qla2xxx driver, please wait...
Installing intermodule.ko in /lib/modules/2.6.18-128.1.6.el5/kernel/kernel/
 


QLA2XXX -- Installing the qla2xxx modules to 
/lib/modules/2.6.18-128.1.6.el5/kernel/drivers/scsi/qla2xxx/...

Setting up QLogic HBA API library...
Please make sure the /usr/lib/libqlsdm.so file is not in use.
Installing 32bit api binary for x86_64.
Installing 64bit api binary for x86_64.
Library 4.00 build12 already installed at /usr/lib/libqlsdm.so.
Done.

Loading module qla2xxx_conf version: v8.02.14.01....
Loaded module qla2xxx_conf
Loading module qla2xxx version: v8.02.14.01....
Loaded module qla2xxx


Building default persistent binding using SCLI
 
Saved copy of /etc/modprobe.conf as
/usr/src/qlogic/v8.02.14.01-1/backup/modprobe.conf-2.6.18-128.1.6.el5-043009-082949.bak

Saved copy of /boot/initrd-2.6.18-128.1.6.el5.img as
/boot/initrd-2.6.18-128.1.6.el5_QLI.org
QLA2XXX -- Rebuilding ramdisk image...
Ramdisk created.

Reloading the QLogic FC HBA drivers....
Unloaded module qla2xxx
Loading module qla2xxx_conf version: v8.02.14.01....
Loaded module qla2xxx_conf
Loading module qla2xxx version: v8.02.14.01....
Loaded module qla2xxx


Target Information on all HBAs:
==============================
 --------------------------------------------------------------------------------
HBA Instance 0: QLA2342 Port 1 WWPN 21-00-00-1B-32-04-86-C3 PortID 01-12-00
--------------------------------------------------------------------------------
No device connected to selected HBA (Instance 0)!
--------------------------------------------------------------------------------
HBA Instance 1: QLA2342 Port 2 WWPN 21-01-00-1B-32-24-86-C3 PortID 01-13-00
--------------------------------------------------------------------------------
No device connected to selected HBA (Instance 1)!
Installing the qlinstall-autoload script in /etc/init.d/ 

#*********************************************************#
#               INSTALLATION SUCCESSFUL!!                 #
#    SANsurfer Driver installation for Linux completed    #
#*********************************************************#

Once the driver is installed and loaded, you can verify the version by running the modinfo utility:

$ modinfo qla2xxx | head -10

filename:       /lib/modules/2.6.18-128.1.6.el5/kernel/drivers/scsi/qla2xxx/qla2xxx.ko
version:        8.02.14.01
license:        GPL
description:    QLogic Fibre Channel HBA Driver
author:         QLogic Corporation
srcversion:     90F727BC62E8CB20433AB36
alias:          pci:v00001077d00002532sv*sd*bc*sc*i*
alias:          pci:v00001077d00005432sv*sd*bc*sc*i*
alias:          pci:v00001077d00005422sv*sd*bc*sc*i*
alias:          pci:v00001077d00008432sv*sd*bc*sc*i*

QLogic includes a number of nifty utilities with their driver bundle, all of which I plan to blog about in future posts.

QLogic ISP2532-based 8Gb Fibre Channel

QLogic ISP2532-based 8Gb Fibre Channel

You shouldn't need to do anything as the driver is already available as part of the CentOS supplied kernel and modules. To check this, I looked up ISP2532 in /usr/share/hwdata/pci.ids and found that it has PCI Ventodr:Device ID 1077:2532. Then I ran

CODE: SELECT ALL

$ grep 1077 /lib/modules/2.6.18-348.1.1.el5xen/modules.* | grep 2532
/lib/modules/2.6.18-348.1.1.el5xen/modules.alias:alias pci:v00001077d00002532sv*sd*bc*sc*i* qla2xxx

This shows that the device is supported by the qla2xxx module. If running `lsmod | grep qla2xxx` on the running system returns no output, then try running `modprobe qla2xxx` and then look in /var/log/messages to see what appears at the end of the file. You might want to check the output from `lspci -nn` to make sure that your adapter has the same PCI Vendor:Device ID that I looked for.

Linux Bass-files Fiber Channel Multipath configuration

Linux Bass-files Fiber Channel Multipath configuration

INSTALL Red Hat 5.3                     Update March 31, 2009
-------------------

Make sure the fibre hba cables are UNPLUGGED. The Red Hat installer gets
confused with the multiple paths to the 6140 StorageTek disk array.
There are 4 paths to each disk drive LUN.  We think this is because
the existing LUNS that are presented to the server are lvm logical
volumes. The server see's multiple paths to the same lvm physical
volume and this causes problems. Once the system sees the lvm physical
disks as one device things work properly. That is the qlogic and rdac
drivers need to be configured before the hba fibre cables are plugged
in. The RDAC drivers make the multiple paths to the LUNS's look like
one device. Note you can also disable the port in the Brocade switch if
you like but that is much more complicated. You can also use the Brocade
java gui to disable the ports.

Do a normal Red Hat 5 install. Do not plug the hba cables in yet! Do not
use the Red Hat multipath software. By default the /etc/multipath.conf
file has all disk devices disabled for use as multipath devices.

Install the Qlogic drivers first then Sun RDAC/MPP drivers.

NOTE: The Red Hat supplied qlogic drivers do not work,
      you need to use the Qlogic supplied drivers!

Get the latest Qlogic drivers at:

http://support.qlogic.com/

Click the downloads tab at the top of the page. Click on Sun under the
OEM MODELS heading.

Look for model SG-XPCIE2FC-QF4, click on "Software for:" Linux URL just
below the table!

Download the latest "Linux FC driver for 2.6 kernel (x86/x64/IA64)"
tarball file, for example:

        qla2xxx-v8.02.14_01-dist.tgz


There is also a "SANsurfer Linux Driver Installer (x86/x64/IA64)" download
that contains the SANsurfer command line utility and the driver. Do not
use this method for installing. Download the "SANsurfer
CLI (x86/x64)" rpm by itself and install:

        scli-1.7.1-23.i386.rpm.gz

gunzip scli-1.7.1-23.i386.rpm.gz
rpm -vih scli-1.7.1-23.i386.rpm

The scli command gets install in:

        /opt/QLogic_Corporation/SANsurferCLI/scli

Use the scli command in this directory to probe information about the
qlogic card WWPN World Wide Port Nodes. Use this utility after
you install the Qlogic driver.

Install the Qlogic provided driver. We will be using the Linux FC
driver for 2.6 download which is a compressed tar file. Do not use the
"qlinstall" command that comes with the "SANsurfer Linux Driver Installer
(x86/x64/IA64)" tarball! Here are the steps to instll the Qlogic driver
using the version as of this writing:

copy the downloaded driver software to /var/tmp and unpack:

# tar zxvf qla2xxx-v8.02.14_01-dist.tgz

The Qlogic installation instructions are in the file:

        README.qla2xxx


# cd qlogic
# ./drvrsetup
# cd qla2xxx-8.02.14
# ./extras/build.sh install

--- output example ---

bash-3.2# ./extras/build.sh install

QLA2XXX -- Building the qla2xxx driver, please wait...
Installing intermodule.ko in /lib/modules/2.6.18-92.1.10.el5/kernel/kernel/
QLA2XXX -- Build done.

QLA2XXX -- Installing the qla2xxx modules to 
/lib/modules/2.6.18-92.1.10.el5/kernel/drivers/scsi/qla2xxx/...

--- end output example ---

Place the following option in /etc/modprob.conf The RDAC driver cannot
co-exist with an HBA-level failover driver. Use echo or vi to update
/etc/modprobe.conf:

# echo "options qla2xxx ql2xfailover=0"  >> /etc/modprobe.conf

Optionaly make a new boot/intrd-*.img ram disk image.  You only need
to do this if you are troubleshooting before you install the Sun RDAC
drivers! The Sun RDAC build makes a new initrd-*.img file also.

# cd /boot

Save old image:

# cp initrd-`uname -r`.img initrd-`uname -r`.img.bak

Make new image to load qlogic driver on boot:

# mkinitrd -f initrd-`uname -r`.img `uname -r`

Reboot system, do not plug fibre cable in yet! Skip to the next step,
installing the RDAC drivers and do not reboot if you do not need to
troubleshoot the LUNS.

# init 6

Use the Qlogic scli command above to troubleshoot the
LUNS/StorageTek 6140.


INSTALL SUN RDAC/MPP
--------------------
Now get and install the Sun RDAC/MPP driver, Redundant Disk Array
Controller Driver (a.k.a. Multi-Path Proxy Driver or MPP)

AS of this writing you need to contact sun to get the latest linux driver.
Below is the output of the make install command.

# tar zxvf rdac-LINUX-09.02.C2.13-source.tar.gz
# cd linuxrdac-09.02.C2.13
# make
# make uninstall
# make install

If you do not do a "make uninstall" you will get this error if upgrading
to the kernel, the builds needs to remove all old files:

The system has old MPP driver package installed.
Please do "make uninstall" in the current directory before installing the new one.
make: *** [hbacheck] Error 1

Do as it says "make uninstall" then do "make install".
Also answer yes to the following question:

Host Adapters from different supported vendors co-exists on your system.
Please make sure that only one supported model of HBA is connected to Storage Array.
Do you want to continue (yes or no) ? yes

The installer will create a new mpp-kernel_release.img ram disk. Update
grub.conf to use this ramd disk, for example:

# vi /etc/grub.conf or /boot/grub/grub.conf

default=0
timeout=5
serial --unit=0 --speed=9600
terminal --timeout=5 serial console
title Red Hat Enterprise Linux Server SUN-RDAC (2.6.18-128.1.1.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-128.1.1.el5 ro root=/dev/sdasysvg/root
        console=ttyS0,9600n1 rhgb quiet
        initrd /mpp-2.6.18-128.1.1.el5.img

You can save the old kernel entries, make this entry the default boot
entry. This way you can boot the non RDAC kernel if you have problems.

Halt the system, "init 0". You can now attach the hba fibre cables.
Bring up the system. The lsmod command should show the qla drivers get
loaded and should look like this:

# lsmod|grep ql
qla2xxx              1009580  1 
qla2xxx_conf          335368  1 
intermodule            37508  2 qla2xxx,qla2xxx_conf
scsi_mod              188665  10 qla2xxx,sr_mod,usb_storage,mptsas,mptscsih,scsi_transport_sas,,sg,sd_mod


The disk LUNS will get presented as a single /dev/sd? device. The RDAC
driver coalesces the multiple paths to the disk array as one device.
You can use "fdisk -l" to list your disk devices that were created from
the LUNS. You can also use the RDAC command /opt/mpp/lsvdev to see
the Lun ot disk  mappings, very handy.

If you were using lvm logical volumes on the LUNS run the following
commands so the os will see the lvm physical volumes:

Scan for lvm physical volumes
# /sbin/pvscan

Verify the list of lvm physical volumes:
# /usr/sbin/pvs
# /usr/sbin/lvs -o +devices

You probably will have to turn the "available" attribute on for the
logical volumes after a fresh install. You can see if the lvm logic
device exists in /dev, for example "ls /dev/volume_group_name/*", If he
volume group device does not exist enable the available attribute for
all the volume groups with:

# lvm vgchange -ay volume_group volume_group1 ...

This should create the logical volume groups and volume device:

        /dev/volume_groupx/logical_volume_name

Again, use "ls /dev/volume_group_name/*" to see if the volume groups
got created

You can then mount the logical volume:
        mount /dev/volume_groupx/logical_volume_name /mount_point

Update the /etc/fstab for all your mounts and mount them.

Reboot the system and verify the boot process and mounts.


UPGRADING TO A NEW KERNEL
-------------------------

You can leave the fibre cables plugged in at this point. The Qlogic/RDAC
drivers only install to the kernel that is currently running.

Run yum to update your system and its kernel.

Comment out all StorageTek 6140 file systems that are mounted!  If you
forget to do this and boot the new kernel fsck will complain and drop
you into single user mode first asking you to enter the root password. If
you accidentally do this here is how to fix:

Enter root password and remount / as a read/write file system with:

        /bin/mount -o remount,rw /

edit /etc/fstab with vi and comment out the StorageTek file systems,
or execut "touch /fastboot" which creates and empty file that fsck
checks for and will bypass the fsck check.

Halt the system and remove the fibre cables from the system. Or user
the Brocade guid to disable the ports for the host.

Boot to the new kernel and install the Qlogic and RDAC drivers as
described above.

After checking the LUNS and volumes are present as described above
uncomment the /etc/fstab SotragTek mounts and re-mount the file systems.


QLOGIC SANSURFER UTILITY
------------------------

Get the qlafc-linux-8.02.08-1-install.tgz as described above and
unpack it:

tar ztvf qlafc-linux-8.02.08-1-install.tgz
cd qlafc-linux-8.02.08-1-install


Install the rpm for your platform, for example:

rpm -ivh scli-1.7.1-18.i386.rpm

The utility gets install at:

/opt/QLogic_Corporation/SANsurferCLI/scli



Some helful commands and file info:
----------------------------------

man RDAC, explains RDAC driver 

/opt/mpp/lsvdev command that displays lun to sd device mappings.

/proc/scsi/mpp/3, (or maybe some other number),  file that contains lun
and controller info.

mppUtil -S command, this will print all the multipaths, read mppUtil man page.

mppBusRescan command  run after mapping new lun so host registers it
on a live sytem if you do not want to reboot.

vgdisplay: Volume group not activated

vgdisplay: Volume group not activated

Here is the full message being shown to me by the OS after invoking the vgdisplay command:

vgdisplay: Volume group not activated.
vgdisplay: Cannot display volume group "/dev/vg10".

If the server is NOT part of a cluster, use

# vgchange -a y vg10

# strings /etc/lvmtab

# more /etc/fstab

# mount <info from fstab>



  

How to Scan new LUNs on Linux with QLogic driver

Q: I am using QLogic driver and I would like to know how do I scan new LUNs on Linux operating system?

A: You need to find out driver proc file /proc/scsi/qlaXXX. For example on my system it is /proc/scsi/qla2300/0

Once file is identified you need to type following command (login as the root user):

# echo "scsi-qlascan" > /proc/scsi/qla2300/0
# cat /proc/scsi/qla2300/0

Now use the script rescan-scsi-bus.sh new LUN as a device. Run script as follows:

# ./rescan-scsi-bus.sh -l -w –c

Hostname change at Linux with Oracle DB

Hostname change at Linux with Oracle DB

24OCT

If you have to change a host name then following steps should have done for Oracle DB and OEM to work properly.

1. Change the host name properly:
   /etc/hosts
/etc/sysconfig/network
2. Change the hostname in tnsnames.ora and listener.ora:
   $ORACLE_HOME/network/admin/tnsnames.ora
$ORACLE_HOME/network/admin/listener.ora
3. Change the hostname in emd.properties:
   $ORACLE_HOME/sysman/config/emd.properties
4. Reboot
5. Restart the listener manually:
   lsnrctl stop
lsnrctl start
lsnrctl status
6. Recreate EM repository:
   emca -deconfig dbcontrol db -repos drop
emca -config dbcontrol db -repos create