Extending an existing LUN w/ Oracle ASM (on NetApp)

Customer asked me yesterday for the steps to resize a LUN on NetApp, and extend an existing ASM disk/diskgroup.  The pre-grow the asm disk first method was throwing an error.  Time to get my hands dirty.

Below is a step by step of how I was able to resize an existing LUN and ASM disk to a new size.  (I will be illustrating a simple 2GB -> 4GB increase in size)

Oracle recommends for ONLINE disk group space increase to actually create a NEW LUN, and add it to the existing disk group. There have been a few bugs w/ ASM recognizing. If you do wan’t to attempt the resize method, you can attempt the instructions below on a test system. Things like this are why I typically recommend NFS. No partitions, no filesystems, none of the issues.

Use NFS whenever possible!  You will be so much happier you did!!

— My steps to build my work area to test on, I did this within Vms and on a fresh netapp.

filer1>
filer1> vol create iscsivol1 -s volume aggr0 4g
Creation of volume ‘iscsivol1’ with size 4g on containing aggregate
‘aggr0’ has completed.
— creating an igroup to share with my test VM
filer1> igroup create -i -t linux oracle iqn.1994-05.com.redhat:773eee9e5ed7
filer1> igroup show
oracle (iSCSI) (ostype: linux):
iqn.1994-05.com.redhat:773eee9e5ed7 (not logged in)
filer1> df -h
Filesystem total used avail capacity Mounted on
/vol/iscsivol1/ 3891MB 148KB 3891MB 0% /vol/iscsivol1/
/vol/iscsivol1/.snapshot 204MB 60KB 204MB 0% /vol/iscsivol1/.snapshot
— Making a 2g lun to test on
filer1> lun create -s 2g -t linux /vol/iscsivol1/lun1.lun
filer1> lun map /vol/iscsivol1/lun1.lun oracle

— On OS as root I checked to see it was there.
Disk /dev/sdd: 2147 MB, 2147483648 bytes
67 heads, 62 sectors/track, 1009 cylinders
Units = cylinders of 4154 * 512 = 2126848 bytes

[root@oracledb iscsi]# sanlun lun show
controller(7mode)/ device host lun
vserver(Cmode) lun-pathname filename adapter protocol size mode
———————————————————————————————–
filer1 /vol/iscsivol1/lun1.lun /dev/sdd host1 iSCSI 2g 7

— making the volume bigger for the lun on the san
filer1> vol size iscsivol1 5g
vol size: Flexible volume ‘iscsivol1’ size set to 5g.
filer1> lun show
/vol/iscsivol1/lun1.lun 2g (2147483648) (r/w, online, mapped)

— making the lun 4g
filer1> lun resize /vol/iscsivol1/lun1.lun 4g
filer1> lun show
/vol/iscsivol1/lun1.lun 4g (4294967296) (r/w, online, mapped)

— OS sees the new lun size.

[root@oracledb tmp]# sanlun lun show
controller(7mode)/ device host lun
vserver(Cmode) lun-pathname filename adapter protocol size mode
———————————————————————————————–
filer1 /vol/iscsivol1/lun1.lun /dev/sdd host1 iSCSI 4g 7

— You go into the scsi_host directory and rescan all scsi buses.

[root@oracledb scsi_host]# cd /sys/class/scsi_host
[root@oracledb scsi_host]# ls
host0 host1
— For each hostX you issue a command to echo 3 dashes w/ spaces in between to the scan object in that directory.
[root@oracledb scsi_host]# echo “- – -” > host1/scan

— Then you go into the specific block device you want, and issue a rescan. In my case the device is /dev/sdd. In case of multipathing, you will use the device mapper multipath entry, not the sdXX entries.

[root@oracledb device]# cd /sys/block/sdd/device/
[root@oracledb device]# echo 1 > rescan

— We can see by looking at dmesg output that the kernel saw the size change

[root@oracledb device]# dmesg | tail -5
sdd: detected capacity change from 2147483648 to 4294967296
SCSI device sdd: 8388608 512-byte hdwr sectors (4295 MB)
sdd: Write Protect is off
sdd: Mode Sense: bd 00 00 08
SCSI device sdd: drive cache: write through

— I verify with fdisk -l that it saw the change.

[root@oracledb device]# fdisk -l /dev/sdd

Disk /dev/sdd: 4294 MB, 4294967296 bytes
67 heads, 62 sectors/track, 2019 cylinders
Units = cylinders of 4154 * 512 = 2126848 bytes

Device Boot Start End Blocks Id System
/dev/sdd1 1 1009 2095662 83 Linux <— notice its ending on 1009 not 2019

— Delete old partition and create a new one.

[root@oracledb ~]# fdisk /dev/sdd

The number of cylinders for this disk is set to 2019.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): d

Selected partition 1

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-2019, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-2019, default 2019): Using default value 2019

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.

— Check with oracle…

/

GROUP_NUMBER DISK_NUMBER NAME STATE MODE_ST MOUNT_S
———— ———– —————————— ——– ——- ——-
TOTAL_MB FREE_MB
———- ———-
1 0 ARCH1 NORMAL ONLINE CACHED
10236 9862

2 0 ASMTEST NORMAL ONLINE CACHED
2046 1996 <— nope still sees old size… awww balls.

—  Oracle ASM actually tags the size of the disk obnoxiously onto the headers of the ASM

—  Use kfed to get the output of the disk headers

[oracle@oracledb disks]$ kfed read ASMTEST  > /tmp/asmtest.kfed

— Edit the /tmp/asmtest.kfed, VI/nano whatever…
— Change the line for  kfdhdb.dsksize to the new disk size, and the HEX equivalent of the size.

kfdhdb.dsksize: 2048 ; 0x0c4: 0x00000800
to
kfdhdb.dsksize: 4096 ; 0x0c4: 0x00001000

— save the file

— update the ASM disk header with the new size.  Remember, this is manually dorking with the ASM headers… DO AT OWN RISK!!!

[oracle@oracledb disks]$ kfed merge /dev/oracleasm/disks/ASMTEST text=/tmp/asmtest.kfed

— Verify ASM sees the new size.

SQL> /

GROUP_NUMBER DISK_NUMBER NAME STATE MODE_ST MOUNT_S
———— ———– —————————— ——– ——- ——-
TOTAL_MB FREE_MB
———- ———-
1 0 ARCH1 NORMAL ONLINE CACHED
10236 9862

2 0 ASMTEST NORMAL ONLINE CACHED
4096 4046

3 0 DATA1 NORMAL ONLINE CACHED
20473 18874

— Resize diskgroup disk

SQL> alter diskgroup asmt1 resize all size 4g;

Diskgroup altered.  <— Woot it worked!!!!

It’s a pretty involved process. Adding a new LUN to the existing disk group would be a much easier process. (Or just use NFS)

 

 

[asa]0071496076[/asa]

Be Sociable, Share!

, , ,