Customer asked me yesterday for the steps to resize a LUN on NetApp, and extend an existing ASM disk/diskgroup. The pre-grow the asm disk first method was throwing an error. Time to get my hands dirty.
Below is a step by step of how I was able to resize an existing LUN and ASM disk to a new size. (I will be illustrating a simple 2GB -> 4GB increase in size)
Oracle recommends for ONLINE disk group space increase to actually create a NEW LUN, and add it to the existing disk group. There have been a few bugs w/ ASM recognizing. If you do wan’t to attempt the resize method, you can attempt the instructions below on a test system. Things like this are why I typically recommend NFS. No partitions, no filesystems, none of the issues.
Use NFS whenever possible! You will be so much happier you did!!
— My steps to build my work area to test on, I did this within Vms and on a fresh netapp.
filer1>
filer1> vol create iscsivol1 -s volume aggr0 4g
Creation of volume ‘iscsivol1’ with size 4g on containing aggregate
‘aggr0’ has completed.
— creating an igroup to share with my test VM
filer1> igroup create -i -t linux oracle iqn.1994-05.com.redhat:773eee9e5ed7
filer1> igroup show
oracle (iSCSI) (ostype: linux):
iqn.1994-05.com.redhat:773eee9e5ed7 (not logged in)
filer1> df -h
Filesystem total used avail capacity Mounted on
/vol/iscsivol1/ 3891MB 148KB 3891MB 0% /vol/iscsivol1/
/vol/iscsivol1/.snapshot 204MB 60KB 204MB 0% /vol/iscsivol1/.snapshot
— Making a 2g lun to test on
filer1> lun create -s 2g -t linux /vol/iscsivol1/lun1.lun
filer1> lun map /vol/iscsivol1/lun1.lun oracle
— On OS as root I checked to see it was there.
Disk /dev/sdd: 2147 MB, 2147483648 bytes
67 heads, 62 sectors/track, 1009 cylinders
Units = cylinders of 4154 * 512 = 2126848 bytes
|
1 |
<em>Disk /dev/sdd doesn't contain a valid partition table</em> |
|
1 |
<em>----</em> |
|
1 |
<span style="color: #008000;"><em>-- Sanlun was also able to see it as 2G.</em></span> |
[[email protected] iscsi]# sanlun lun show
controller(7mode)/ device host lun
vserver(Cmode) lun-pathname filename adapter protocol size mode
———————————————————————————————–
filer1 /vol/iscsivol1/lun1.lun /dev/sdd host1 iSCSI 2g 7
|
1 2 |
<span style="color: #008000;"><em> -- use fdisk to make 1 partition of type linux for full size --- (required when using ASM SnapManager for Oracle on RHEL) -- print out the size and partition table</em></span> <em> [root@oracledb iscsi]#<span style="color: #ff0000;"> fdisk -l /dev/sdd</span></em> |
|
1 2 3 |
<em>Disk /dev/sdd: <strong>2147</strong> MB, 2147483648 bytes</em> <em> 67 heads, 62 sectors/track, 1009 cylinders</em> <em> Units = cylinders of 4154 * 512 = 2126848 bytes</em> |
|
1 2 |
<em>Device Boot Start End Blocks Id System</em> <em> /dev/sdd1 1 1009 2095662 83 Linux</em> |
|
1 2 |
<em>[root@oracledb iscsi]# <span style="color: #ff0000;">/etc/init.d/oracleasm createdisk ASMTEST /dev/sdd1</span></em> <em> Marking disk "ASMTEST" as an ASM disk: [ OK ]</em> |
|
1 2 3 |
<em>[root@oracledb iscsi]# su - oracle</em> <em> [oracle@oracledb ~]$ . .asm</em> <em> [oracle@oracledb ~]$ sqlplus / as sysasm</em> |
|
1 |
<em>SQL*Plus: Release 11.2.0.1.0 Production on Mon Aug 13 21:41:30 2012</em> |
|
1 |
<em>Copyright (c) 1982, 2009, Oracle. All rights reserved.</em> |
|
1 2 3 |
<em>Connected to:</em> <em> Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production</em> <em> With the Automatic Storage Management option</em> |
|
1 |
<em>SQL><span style="color: #ff0000;"> select group_number,disk_number,name,state,mode_status,mount_status,total_mb,free_mb from v$asm_disk;</span></em> |
|
1 2 3 4 5 |
<em>GROUP_NUMBER DISK_NUMBER NAME STATE MODE_ST MOUNT_S TOTAL_MB FREE_MB</em> <em> ------------ ----------- ------------------------------ -------- ------- ------- ---------- ----------</em> <em> 0 0 NORMAL ONLINE CLOSED 0 0</em> <em> 1 0 ARCH1 NORMAL ONLINE CACHED 10236 9862</em> <em> 2 0 DATA1 NORMAL ONLINE CACHED 20473 18874</em> |
|
1 2 3 |
<em>SQL><span style="color: #ff0000;"> create diskgroup asmt1 external redundancy disk 'ORCL:ASMTEST';</span></em> <em> Diskgroup created.</em> <em> SQL> <span style="color: #ff0000;">select group_number,disk_number,name,state,mode_status,mount_status,total_mb,free_mb from v$asm_disk;</span></em> |
|
1 2 3 4 5 |
<em>GROUP_NUMBER DISK_NUMBER NAME STATE MODE_ST MOUNT_S TOTAL_MB FREE_MB</em> <em> ------------ ----------- ------------------------------ -------- ------- ------- ---------- ----------</em> <em> 1 0 ARCH1 NORMAL ONLINE CACHED 10236 9862</em> <em> 2 0 DATA1 NORMAL ONLINE CACHED 20473 18874</em> <em> <span style="color: #ff0000;"><strong>3 0 ASMTEST NORMAL ONLINE CACHED 2046 1996</strong></span></em> |
— making the volume bigger for the lun on the san
filer1> vol size iscsivol1 5g
vol size: Flexible volume ‘iscsivol1’ size set to 5g.
filer1> lun show
/vol/iscsivol1/lun1.lun 2g (2147483648) (r/w, online, mapped)
— making the lun 4g
filer1> lun resize /vol/iscsivol1/lun1.lun 4g
filer1> lun show
/vol/iscsivol1/lun1.lun 4g (4294967296) (r/w, online, mapped)
— OS sees the new lun size.
[[email protected] tmp]# sanlun lun show
controller(7mode)/ device host lun
vserver(Cmode) lun-pathname filename adapter protocol size mode
———————————————————————————————–
filer1 /vol/iscsivol1/lun1.lun /dev/sdd host1 iSCSI 4g 7
— You go into the scsi_host directory and rescan all scsi buses.
[[email protected] scsi_host]# cd /sys/class/scsi_host
[[email protected] scsi_host]# ls
host0 host1
— For each hostX you issue a command to echo 3 dashes w/ spaces in between to the scan object in that directory.
[[email protected] scsi_host]# echo “- – -” > host1/scan
— Then you go into the specific block device you want, and issue a rescan. In my case the device is /dev/sdd. In case of multipathing, you will use the device mapper multipath entry, not the sdXX entries.
[[email protected] device]# cd /sys/block/sdd/device/
[[email protected] device]# echo 1 > rescan
— We can see by looking at dmesg output that the kernel saw the size change
[[email protected] device]# dmesg | tail -5
sdd: detected capacity change from 2147483648 to 4294967296
SCSI device sdd: 8388608 512-byte hdwr sectors (4295 MB)
sdd: Write Protect is off
sdd: Mode Sense: bd 00 00 08
SCSI device sdd: drive cache: write through
— I verify with fdisk -l that it saw the change.
[[email protected] device]# fdisk -l /dev/sdd
Disk /dev/sdd: 4294 MB, 4294967296 bytes
67 heads, 62 sectors/track, 2019 cylinders
Units = cylinders of 4154 * 512 = 2126848 bytes
Device Boot Start End Blocks Id System
/dev/sdd1 1 1009 2095662 83 Linux <— notice its ending on 1009 not 2019
— Delete old partition and create a new one.
[[email protected] ~]# fdisk /dev/sdd
The number of cylinders for this disk is set to 2019.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help): d
Selected partition 1
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-2019, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-2019, default 2019): Using default value 2019
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
— Check with oracle…
/
GROUP_NUMBER DISK_NUMBER NAME STATE MODE_ST MOUNT_S
———— ———– —————————— ——– ——- ——-
TOTAL_MB FREE_MB
———- ———-
1 0 ARCH1 NORMAL ONLINE CACHED
10236 9862
2 0 ASMTEST NORMAL ONLINE CACHED
2046 1996 <— nope still sees old size… awww balls.
— Oracle ASM actually tags the size of the disk obnoxiously onto the headers of the ASM
— Use kfed to get the output of the disk headers
[[email protected] disks]$ kfed read ASMTEST > /tmp/asmtest.kfed
— Edit the /tmp/asmtest.kfed, VI/nano whatever…
— Change the line for kfdhdb.dsksize to the new disk size, and the HEX equivalent of the size.
kfdhdb.dsksize: 2048 ; 0x0c4: 0x00000800
to
kfdhdb.dsksize: 4096 ; 0x0c4: 0x00001000
— save the file
— update the ASM disk header with the new size. Remember, this is manually dorking with the ASM headers… DO AT OWN RISK!!!
[[email protected] disks]$ kfed merge /dev/oracleasm/disks/ASMTEST text=/tmp/asmtest.kfed
— Verify ASM sees the new size.
SQL> /
GROUP_NUMBER DISK_NUMBER NAME STATE MODE_ST MOUNT_S
———— ———– —————————— ——– ——- ——-
TOTAL_MB FREE_MB
———- ———-
1 0 ARCH1 NORMAL ONLINE CACHED
10236 9862
2 0 ASMTEST NORMAL ONLINE CACHED
4096 4046
3 0 DATA1 NORMAL ONLINE CACHED
20473 18874
— Resize diskgroup disk
SQL> alter diskgroup asmt1 resize all size 4g;
Diskgroup altered. <— Woot it worked!!!!
It’s a pretty involved process. Adding a new LUN to the existing disk group would be a much easier process. (Or just use NFS)
[asa]0071496076[/asa]
I do know this was a very interesting post thanks for writing it!