Cannot Start CRS 11gR2 Due to ASM Startup Failure?

Oracle 11gR2 was first released on Linux in September 2009, a year later Oracle 11.2.0.2 became available. As you are probably aware by now, Oracle Clusterware (CRS) 11gR2 has a very different architecture to the 11gR1 version, the main difference being the tight integration with Oracle Automatic Storage Management (ASM). Here the OCR and Voting disks are located in a dedicated ASM management (+MGMT) diskgroup, as shown in the example below:

Voting Disk

[oracle@dbserver01a ~]$ crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group

--  -----    -----------------                --------- ---------

 1. ONLINE   04b2a8c3c11b4f46bf1d37a65940b47e (/dev/emcpowerf) [MGMT]

Located 1 voting disk(s).

OCR Disk

[oracle@dbserver01 ~]$ cat /etc/oracle/ocr.loc

ocrconfig_loc=+MGMT

local_only=FALSE

When I first learnt of this approach, I did think this to be a “chicken and egg” situation, where CRS cannot start if there was a problem with ASM and vice-versa. So how do you fix this if you cannot access your ASM instance?.

This short article reveals just how you can “fix” ASM should it not start and get your Oracle cluster up and running.

Case Study

Firstly, for this case study we need to break ASM.

To stop the ASM instance from starting up when CRS starts, I introduced a memory problem by increasing the maximum number of processes to some large value. ASM by default operates in a relatively small SGA, in fact the default memory_target is only 272MB. By setting the processes parameter to 1000 should exhaust the allocated memory.

[oracle@dbserver1a ~]$ . oraenv

ORACLE_SID = [ORCL1] ? +ASM1

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@dbserver1a ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.2.0 Production on Mon Apr 2 13:47:44 2012

SQL> show parameter memory_target

 

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

memory_target                        big integer 272M

 

SQL> show parameter spfile

 

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

spfile                               string      +MGMT/dbserver-orcl/asmparameterfi

                                                 le/registry.253.756994263

SQL> alter system set processes=1000 scope=spfile sid=’*’;

System altered.

SQL> exit

Stopping and then attempting to restart CRS resulted in the following errors seen in their respective log files:

CRS Alert Log

[/u01/app/product/11.2.0/grid/bin/oraagent.bin(29649)]CRS-5019:All OCR locations are on ASM disk groups [MGMT], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/product/11.2.0/grid/log/dbserver01a/agent/ohasd/oraagent_oracle/oraagent_oracle.log".

Oracle Agent Log

2012-03-30 14:58:43.924: [    AGFW][1119709504] {0:0:2} sending status msg [CRS-5017: The resource action "ora.asm start" encountered the following error:

ORA-04031: unable to allocate 32 bytes of shared memory ("shared pool","ALTER DISKGROUP ALL MOUNT /*...","SQLA","tmp")] for start for resource: ora.asm 1 1

Obviously this error will affect all RAC instances as the cluster is down. The next step is therefore to get at least 1 ASM instance running so we can start CRS on that node. In order to start the failed ASM instance we must first obtain the initialization parameters from a good instance. This can be done by executing the following command on a working environment (11.2.0.2 ASM instance).

SQL> create pfile=’/home/oracle/pfile.ora’ from spfile;

File created.

SQL> exit

$ cat /home/oracle/pfile.ora

+ASM1.__shared_pool_size=163577856

+ASM1.asm_diskgroups=’DATA’,’FAST’#Manual Mount

+ASM2.asm_diskgroups=’DATA’,’FAST’#Manual Mount

*.asm_diskstring=’/dev/emcpower*’

*.asm_power_limit=1

*.diagnostic_dest=’/u01/app/oracle’

*.instance_type=’asm

*.large_pool_size=12M

*.remote_login_passwordfile=’EXCLUSIVE’

Taking the contents of this file (apart from the underscore-underscore parameter), we can create a new ASM parameter file (/home/oracle/pfile.ora) on our server (dbserver01a).

Now let’s startup the +ASM1 instance.

SQL> shutdown abort

SQL> startup pfile=’/home/oracle/pfile.ora’

ASM instance started

As soon as the OCR and Voting disks become accessible, CRS starts automatically. With the Clusterware and ASM instance running on node 1, we can now configure +ASM1 through SQL*Plus. We must convert ASM to use a shared Server Parameter File (SPFILE). This can be done as follows from the SQL prompt:

SQL> create spfile=’+MGMT’ from pfile=’/home/oracle/pfile.ora’;

File created.

SQL> exit

The ASM instance must be restarted to pick up the configuration. With the dependency between ASM and CRS, it is easier to restart the Clusterware. On each node execute the following commands as root user:

$ /u01/app/product/11.2.0/grid/bin/crsctl stop crs

$ /u01/app/product/11.2.0/grid/bin/crsctl start crs

After a short period of time, all CRS resources should appear ONLINE.

$ /u01/app/product/11.2.0/grid/bin/crsctl stat res -t

--------------------------------------------------------------------------------

NAME           TARGET  STATE        SERVER                   STATE_DETAILS

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.DATA.dg

               ONLINE  ONLINE       dbserver01a

               ONLINE  ONLINE       dbserver01b

ora.FAST.dg

               ONLINE  ONLINE       dbserver01a

               ONLINE  ONLINE       dbserver01b

ora.LISTENER.lsnr

               ONLINE  ONLINE       dbserver01a

               ONLINE  ONLINE       dbserver01b

ora.MGMT.dg

               ONLINE  ONLINE       dbserver01a

               ONLINE  ONLINE       dbserver01b

ora.asm

               ONLINE  ONLINE       dbserver01a          Started

               ONLINE  ONLINE       dbserver01b

ora.gsd

               ONLINE  ONLINE       dbserver01a

               ONLINE  ONLINE       dbserver01b

ora.net1.network

               ONLINE  ONLINE       dbserver01a

               ONLINE  ONLINE       dbserver01b

ora.ons

               ONLINE  ONLINE       dbserver01a

               ONLINE  ONLINE       dbserver01b

ora.registry.acfs

               ONLINE  ONLINE       dbserver01a

               ONLINE  ONLINE       dbserver01b

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

      1        ONLINE  ONLINE       dbserver01a

ora.cvu

      1        ONLINE  ONLINE       dbserver01a

ora.dbserver01a.vip

      1        ONLINE  ONLINE       dbserver01a

ora.dbserver01b.vip

      1        ONLINE  ONLINE       dbserver01b

ora.orcl.db

      1        ONLINE  ONLINE       dbserver01a          Open

      2        ONLINE  ONLINE       dbserver01b          Open

Setting ASM Instance Parameters

There may be a legitimate reason for increasing the processes parameter on your ASM instances, typically if your nodes are hosting multiple database instances. Should this be the case, remember to increase all of the following parameters, stop all CRS resources and restart your cluster:

·          processes (default 100)

·          memory_target (default 272M)

·          large_pool (default 12M)

For example

SQL> alter system set processes=200 scope=spfile sid=’*’;

System altered.

 

SQL> alter system set memory_target=512M scope=spfile sid=’*’;

System altered.

 

SQL> alter system set large_pool_size=24M scope=spfile sid=’*’;

System altered.

SQL> exit

$ su

$ /u01/app/product/11.2.0/grid/bin/crs_stop –all

and on each node:

$ /u01/app/product/11.2.0/grid/bin/crsctl stop crs

$ /u01/app/product/11.2.0/grid/bin/crsctl start crs

 

_______________________________________________________________________________

Did you find the article useful?

Please provide your feedback by voting now.

If you have a comment or question, please complete and submit the form below.

_______________________________________________________________________________