Oracle 11gR2 was first released on Linux in September 2009, a year later Oracle 11.2.0.2 became available. As you are probably aware by now, Oracle Clusterware (CRS) 11gR2 has a very different architecture to the 11gR1 version, the main difference being the tight integration with Oracle Automatic Storage Management (ASM). Here the OCR and Voting disks are located in a dedicated ASM management (+MGMT) diskgroup, as shown in the example below:
Voting Disk
[oracle@dbserver01a ~]$ crsctl query css votedisk
##
STATE File Universal
Id File Name Disk group
-- ----- ----------------- --------- ---------
1.
ONLINE 04b2a8c3c11b4f46bf1d37a65940b47e
(/dev/emcpowerf) [MGMT]
Located 1 voting disk(s).
OCR Disk
[oracle@dbserver01 ~]$ cat
/etc/oracle/ocr.loc
ocrconfig_loc=+MGMT
local_only=FALSE
When I first learnt of this approach, I did think this to be a “chicken and egg” situation, where CRS cannot start if there was a problem with ASM and vice-versa. So how do you fix this if you cannot access your ASM instance?.
This short article reveals just how you can “fix” ASM should it not start and get your Oracle cluster up and running.
Firstly, for this case study we need to break ASM.
To stop the ASM instance from starting up when CRS starts, I introduced a memory problem by increasing the maximum number of processes to some large value. ASM by default operates in a relatively small SGA, in fact the default memory_target is only 272MB. By setting the processes parameter to 1000 should exhaust the allocated memory.
[oracle@dbserver1a ~]$ . oraenv
ORACLE_SID = [ORCL1] ?
+ASM1
The Oracle base remains unchanged with value
/u01/app/oracle
[oracle@dbserver1a ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.2.0 Production on Mon Apr 2
13:47:44 2012
SQL> show parameter memory_target
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
memory_target big integer 272M
SQL> show parameter spfile
NAME TYPE VALUE
------------------------------------
----------- ------------------------------
spfile string +MGMT/dbserver-orcl/asmparameterfi
le/registry.253.756994263
SQL> alter system set processes=1000 scope=spfile sid=’*’;
System altered.
SQL> exit
Stopping and then attempting to restart CRS resulted in the following errors seen in their respective log files:
CRS Alert Log
[/u01/app/product/11.2.0/grid/bin/oraagent.bin(29649)]CRS-5019:All OCR locations are on ASM disk groups [MGMT], and none
of these disk groups are mounted. Details are at "(:CLSN00100:)"
in "/u01/app/product/11.2.0/grid/log/dbserver01a/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
Oracle Agent Log
2012-03-30 14:58:43.924: [ AGFW][1119709504]
{0:0:2} sending status msg [CRS-5017: The resource action
"ora.asm start" encountered the following error:
ORA-04031: unable to allocate 32 bytes of shared
memory ("shared pool","ALTER DISKGROUP
ALL MOUNT /*...","SQLA","tmp")]
for start for resource: ora.asm 1 1
Obviously this error will affect all RAC instances as the cluster is down. The next step is therefore to get at least 1 ASM instance running so we can start CRS on that node. In order to start the failed ASM instance we must first obtain the initialization parameters from a good instance. This can be done by executing the following command on a working environment (11.2.0.2 ASM instance).
SQL> create pfile=’/home/oracle/pfile.ora’
from spfile;
File created.
SQL> exit
$ cat
/home/oracle/pfile.ora
+ASM1.__shared_pool_size=163577856
+ASM1.asm_diskgroups=’DATA’,’FAST’#Manual
Mount
+ASM2.asm_diskgroups=’DATA’,’FAST’#Manual
Mount
*.asm_diskstring=’/dev/emcpower*’
*.asm_power_limit=1
*.diagnostic_dest=’/u01/app/oracle’
*.instance_type=’asm’
*.large_pool_size=12M
*.remote_login_passwordfile=’EXCLUSIVE’
Taking the contents of this file (apart from the underscore-underscore parameter), we can create a new ASM parameter file (/home/oracle/pfile.ora) on our server (dbserver01a).
Now let’s startup the +ASM1 instance.
SQL> shutdown abort
SQL> startup pfile=’/home/oracle/pfile.ora’
ASM instance started
As soon as the OCR and Voting disks become accessible, CRS starts automatically. With the Clusterware and ASM instance running on node 1, we can now configure +ASM1 through SQL*Plus. We must convert ASM to use a shared Server Parameter File (SPFILE). This can be done as follows from the SQL prompt:
SQL> create spfile=’+MGMT’
from pfile=’/home/oracle/pfile.ora’;
File created.
SQL> exit
The ASM instance must be restarted to pick up the configuration. With the dependency between ASM and CRS, it is easier to restart the Clusterware. On each node execute the following commands as root user:
$ /u01/app/product/11.2.0/grid/bin/crsctl stop crs
$ /u01/app/product/11.2.0/grid/bin/crsctl start crs
After a short period of time, all CRS resources should appear ONLINE.
$ /u01/app/product/11.2.0/grid/bin/crsctl stat res -t
--------------------------------------------------------------------------------
NAME
TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE dbserver01a
ONLINE ONLINE dbserver01b
ora.FAST.dg
ONLINE ONLINE dbserver01a
ONLINE ONLINE dbserver01b
ora.LISTENER.lsnr
ONLINE ONLINE dbserver01a
ONLINE ONLINE dbserver01b
ora.MGMT.dg
ONLINE ONLINE dbserver01a
ONLINE ONLINE dbserver01b
ora.asm
ONLINE ONLINE dbserver01a Started
ONLINE ONLINE dbserver01b
ora.gsd
ONLINE ONLINE dbserver01a
ONLINE ONLINE dbserver01b
ora.net1.network
ONLINE ONLINE dbserver01a
ONLINE ONLINE dbserver01b
ora.ons
ONLINE ONLINE dbserver01a
ONLINE ONLINE dbserver01b
ora.registry.acfs
ONLINE ONLINE dbserver01a
ONLINE ONLINE dbserver01b
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE dbserver01a
ora.cvu
1 ONLINE ONLINE dbserver01a
ora.dbserver01a.vip
1 ONLINE ONLINE dbserver01a
ora.dbserver01b.vip
1 ONLINE ONLINE dbserver01b
ora.orcl.db
1 ONLINE ONLINE dbserver01a Open
2 ONLINE ONLINE dbserver01b Open
There may be a legitimate reason for increasing the processes parameter on your ASM instances, typically if your nodes are hosting multiple database instances. Should this be the case, remember to increase all of the following parameters, stop all CRS resources and restart your cluster:
·
processes (default 100)
·
memory_target (default 272M)
·
large_pool (default 12M)
For example
SQL> alter system set processes=200 scope=spfile sid=’*’;
System altered.
SQL> alter system set memory_target=512M
scope=spfile sid=’*’;
System altered.
SQL> alter system set large_pool_size=24M
scope=spfile sid=’*’;
System altered.
SQL> exit
$ su
–
$ /u01/app/product/11.2.0/grid/bin/crs_stop –all
and on each node:
$ /u01/app/product/11.2.0/grid/bin/crsctl stop crs
$ /u01/app/product/11.2.0/grid/bin/crsctl start crs
_______________________________________________________________________________
Did you find the article useful?
Please provide your feedback by voting now.
If you have a comment or question, please complete and submit the form below.
_______________________________________________________________________________