This post is also available in:
Português
Prior Oracle 11gR2, it was only possible to the OCR and Voting Disk files in multiple raw devices, shared between the nodes of a cluster.
From this version on, it's very interesting to take the advantages and capabilities of ASM (striping and mirror) to store those files.
To store the OCR and Voting Disk files on a diskgroup, you must obey the following rules:
- If your diskgroup has "external" redundancy, it must have at least one disk associated.
- If your diskgroup has "normal" redundancy, it must have at least 3 disks associated, even though it can be create diskgroup with only 2 disks.
- If your diskgroup has "high" redundancy, it must have at least 5 disks associated, even though it can be create diskgroup with only 3 disks.
In this article, I will use the second option, normal redundancy. As in my server there are only 2 disks available for the diskgroup, I asked to the storage team to add another 1 GB disk:
[root@orarac01 ~]# /etc/init.d/oracleasm listdisks DATA01 OCRVOTE01 OCRVOTE02 [root@orarac01 ~]#
The disk has already been available:
[root@orarac01 ~]# multipath -ll DATA01 (36000c29ab0accd959da8f1f78dd2de2f) dm-4 VMware,Virtual disk size=300G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active `- 3:0:2:0 sde 8:64 active ready running OCRVOTE03 (36000c299550e110b041c9ab26acd33ac) dm-5 VMware,Virtual disk size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active `- 3:0:3:0 sdf 8:80 active ready running OCRVOTE02 (36000c29b67329567f6a6547d6fd3c769) dm-2 VMware,Virtual disk size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active `- 3:0:0:0 sdc 8:32 active ready running OCRVOTE01 (36000c29c1a57ee047d443991ef48894a) dm-3 VMware,Virtual disk size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active `- 3:0:1:0 sdd 8:48 active ready running [root@orarac01 ~]#
Adding a new disk to oracleasm:
[root@orarac01 ~]# /etc/init.d/oracleasm createdisk OCRVOTE03 /dev/dm-5 Marking disk "OCRVOTE03" as an ASM disk: [ OK ] [root@orarac01 ~]# /etc/init.d/oracleasm listdisks DATA01 OCRVOTE01 OCRVOTE02 OCRVOTE03 [root@orarac01 ~]#
Enabling disk on another RAC node:
[root@orarac02 ~]# /etc/init.d/oracleasm scandisks Scanning the system for Oracle ASMLib disks: [ OK ] [root@orarac02 ~]# /etc/init.d/oracleasm listdisks DATA01 OCRVOTE01 OCRVOTE02 OCRVOTE03 [root@orarac02 ~]#
The next step is to create the new ASM diskgroup
First, let's try to put them as "High Redundancy" and see what happens.
CREATE DISKGROUP OCRVOTE HIGH REDUNDANCY FAILGROUP OCRVOTE_FG01 DISK '/dev/oracleasm/disks/OCRVOTE01' NAME OCRVOTE01 SIZE 1024M FAILGROUP OCRVOTE_FG02 DISK '/dev/oracleasm/disks/OCRVOTE02' NAME OCRVOTE02 SIZE 1024M FAILGROUP OCRVOTE_FG03 DISK '/dev/oracleasm/disks/OCRVOTE03' NAME OCRVOTE03 SIZE 1024M ATTRIBUTE 'compatible.asm'='11.2.0.0.0','au_size'='1M';
[root@orarac01 ~]# crsctl replace votedisk +OCRVOTE Failed to create voting files on disk group OCRVOTE. Change to configuration failed, but was successfully rolled back. CRS-4000: Command Replace failed, or completed with errors. [root@orarac01 ~]#
[root@orarac01 ~]# tail -n 200 $GRID_HOME/log/orarac01/client/crsctl_root.log ... 2013-06-18 11:46:13.220: [ CSSCLNT][278546160]clsssConfigLockTrace: === clsssConfigLock === 2013-06-18 11:46:13.220: [ CSSCLNT][278546160]clsssConfigLockTrace: lock(0x2838a50), version(1), size(1088) 2013-06-18 11:46:13.220: [ CSSCLNT][278546160]clsssConfigLockTrace: offsets(24), activever(186647296) 2013-06-18 11:46:13.220: [ CSSCLNT][278546160]clsssConfigLockTrace: id(15), instantiation(5), incarn(1) 2013-06-18 11:46:13.220: [ CSSCLNT][278546160]clsssConfigLockTrace: mapoff(28), configoff(548), mapsize(512) 2013-06-18 11:46:13.220: [ CSSCLNT][278546160]clsssConfigLockTrace: state(10), holders(1), waiters(0) 2013-06-18 11:46:13.220: [ CSSCLNT][278546160]clsssConfigLockTrace: committimestamp(0), commitstate(0) 2013-06-18 11:46:13.220: [ CSSCLNT][278546160]clsssConfigLockTrace: === Map (of first 7 entries) === 2013-06-18 11:46:13.220: [ CSSCLNT][278546160]clsssConfigLockTrace: 010 000 000 000 000 000 000 2013-06-18 11:46:13.221: [ CRSCTL][278546160]crsctl_replace_votedisk: calling kgfdvfPushDiskString 2013-06-18 11:46:13.397: [ CRSCTL][278546160]function crsctl_replace_votedisk, error 1 [clsuSlosFormatDiag called with non-error slos.] . 2013-06-18 11:46:13.397: [ CRSCTL][278546160]crsctl_replace_votedisk: pushed the ASM DS to the profile 1 [ CLWAL][278546160]clsw_Initialize: OLR initlevel [30000] 2013-06-18 11:46:13.432: [ CRSCTL][278546160]crsctl_format_diskgroup: creating diskgroup OCRVOTE 2013-06-18 11:46:16.555: [ CRSCTL][278546160]function crsctl_format_diskgroup, error 1 SLOS: cat=-1, opn=(none), dep=(none), loc=ORA-01405: f . 2013-06-18 11:46:16.555: [ CRSCTL][278546160]crsctl_format_diskgroup: diskgroup OCRVOTE creation with status 1. Please check the alert log file for ASM 2013-06-18 11:46:16.568: [ CSSCLNT][278546160]clsssConfigLockTrace: === clsssConfigUnlock === 2013-06-18 11:46:16.568: [ CSSCLNT][278546160]clsssConfigLockTrace: lock(0x2837b50), version(1), size(1088) 2013-06-18 11:46:16.568: [ CSSCLNT][278546160]clsssConfigLockTrace: offsets(24), activever(186647296) 2013-06-18 11:46:16.568: [ CSSCLNT][278546160]clsssConfigLockTrace: id(15), instantiation(5), incarn(3) 2013-06-18 11:46:16.568: [ CSSCLNT][278546160]clsssConfigLockTrace: mapoff(28), configoff(548), mapsize(512) 2013-06-18 11:46:16.568: [ CSSCLNT][278546160]clsssConfigLockTrace: state(0), holders(0), waiters(0) 2013-06-18 11:46:16.568: [ CSSCLNT][278546160]clsssConfigLockTrace: committimestamp(0), commitstate(0) 2013-06-18 11:46:16.568: [ CSSCLNT][278546160]clsssConfigLockTrace: === Map (of first 7 entries) === 2013-06-18 11:46:16.568: [ CSSCLNT][278546160]clsssConfigLockTrace: 000 000 000 000 000 000 000 ...
[root@orarac01 ~]# tail -n 500 /u01/app/grid/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log ... Tue Jun 18 11:46:13 2013 NOTE: Creating voting files in diskgroup OCRVOTE Tue Jun 18 11:46:13 2013 NOTE: Voting File refresh pending for group 2/0x7f386577 (OCRVOTE) NOTE: Attempting voting file creation in diskgroup OCRVOTE NOTE: voting file allocation on grp 2 disk OCRVOTE_0000 NOTE: voting file allocation on grp 2 disk OCRVOTE_0001 NOTE: voting file allocation on grp 2 disk OCRVOTE_0002 ERROR: Voting file allocation failed for group OCRVOTE Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_6301.trc: ORA-15274: Not enough failgroups (5) to create voting files NOTE: Attempting voting file refresh on diskgroup OCRVOTE NOTE: Voting file relocation is required in diskgroup OCRVOTE NOTE: Attempting voting file relocation on diskgroup OCRVOTE NOTE: voting file deletion on grp 2 disk OCRVOTE_0000 NOTE: voting file deletion on grp 2 disk OCRVOTE_0001 NOTE: voting file deletion on grp 2 disk OCRVOTE_0002 ...
The error is that it is necessary at least 5 failgroups to create the Voting Disk.
We will then recreate them with the correct redundancy:
CREATE DISKGROUP OCRVOTE NORMAL REDUNDANCY FAILGROUP OCRVOTE_FG01 DISK '/dev/oracleasm/disks/OCRVOTE01' NAME OCRVOTE01 SIZE 1024M FAILGROUP OCRVOTE_FG02 DISK '/dev/oracleasm/disks/OCRVOTE02' NAME OCRVOTE02 SIZE 1024M FAILGROUP OCRVOTE_FG03 DISK '/dev/oracleasm/disks/OCRVOTE03' NAME OCRVOTE03 SIZE 1024M ATTRIBUTE 'compatible.asm'='11.2.0.0.0','au_size'='1M';
[root@orarac01 ~]# crsctl replace votedisk +OCRVOTE Successful addition of voting disk 05ca1320e3244f70bf45d1fc37d325d5. Successful addition of voting disk 1affc0b6da314fe8bfd8dec18af4923b. Successful addition of voting disk 3d87bf2793e04fe4bfd6764f24ffafb1. Successful deletion of voting disk 4c9ddcb1acd14f27bf77652370a1a5b1. Successfully replaced voting disk group with +OCRVOTE. CRS-4266: Voting file(s) successfully replaced [root@orarac01 ~]# crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 05ca1320e3244f70bf45d1fc37d325d5 (/dev/oracleasm/disks/OCRVOTE01) [OCRVOTE] 2. ONLINE 1affc0b6da314fe8bfd8dec18af4923b (/dev/oracleasm/disks/OCRVOTE02) [OCRVOTE] 3. ONLINE 3d87bf2793e04fe4bfd6764f24ffafb1 (/dev/oracleasm/disks/OCRVOTE03) [OCRVOTE] Located 3 voting disk(s). [root@orarac01 ~]#
The Voting Disk step is done.
The next step is the CRS. Note that we are transferring him from another diskgroup (+DATA), which is used to store datafiles, to this another diskgroup recently created:
[root@orarac01 ~]# ocrconfig -replace +DATA -replacement +OCRVOTE PROT-28: Cannot delete or replace the only configured Oracle Cluster Registry location [root@orarac01 ~]#
As we have only one OCR location configured, the "replace" command doesn't work. We'll need to add another one and delete the old:
[root@orarac01 ~]# ocrconfig -add +OCRVOTE
[root@orarac01 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2836
Available space (kbytes) : 259284
ID : 1089167638
Device/File Name : +DATA
Device/File integrity check succeeded
Device/File Name : +OCRVOTE
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@orarac01 ~]#
Now erase the old:
[root@orarac01 ~]# ocrconfig -delete +DATA
[root@orarac01 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2836
Available space (kbytes) : 259284
ID : 1089167638
Device/File Name : +OCRVOTE
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@orarac01 ~]# cat /etc/oracle/ocr.loc
#Device/file +DATA getting replaced by device +OCRVOTE
ocrconfig_loc=+OCRVOTE
local_only=false
[root@orarac01 ~]#
Done!
More information can be obtained at the Metalink article below:
OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE) [ID 428681.1]
