4.11.2005

VXFS: Re-online a diskgroup with a missing disk

We lost a disk out of our Oracle disks, that was under Veritas control, due to a problem on the SAN. To get the volumes back off the missing disk we had to do the following:

1. Re-scan the system to pull back the missing disk:

devfsadm

2. See if the disk still contains valid information in the veritas private region. First grab the information, then pipe it through to see if the information is still valid:

vxprivutil dumpconfig /dev/rdsk/c6t1d0s3 > /tmp/out
cat /tmp/outvxprint -d - -ht

3. Restart the disk group to add back in the missing disk, then check to make sure the old disk is back in the group.

vxdg deport oracle
vxdg import oracle
vxdisk list

4. See if the volumes came back auto-magically, if so you are done otherwise continue:

vxprint -ht -g oracle

5. Re-initialize the private region of the missing disk, which we will write over later.

vxdisk -f init c6t1d0

6. Using the disk layout from the missing disk, collected in step 2, generate the information required to rebuild the missing volume(s). repeat for all missing volumes:

cat /tmp/outvxprint -D - -mpvsqQr u03 > /tmp/u03

7. Re-create the volume with the script(s) generated in step six:

vxmake -g oracle -d /tmp/u03

8. Verify that the volume(s) was actually made, if so start it:

vxprint -ht
vxvol -g oracle start u03

5. Fsck the disk(s) and then mount it up:

fsck -F vxfs /dev/vx/dsk/oracle/u03
mount -F vxfs /dev/vx/dsk/oracle/u03