It is every
DBA’s nightmare to be given the task of patching an Oracle environment during a
tight scheduled downtime window, to be rewarded with problems, resulting in failure.
Maybe even having to resort to an Oracle Home backup with no new patches
applied. The following paragraphs aim to explain and provide solutions to some
of the “gotchas” I have experienced when patching Oracle 11.1.0.7.
Patch Set Updates
(PSUs) are proactive cumulative patches containing recommended bug fixes that
are released on a regular and predictable schedule. PSUs include recommended
Patch Bundles and Critical Patch Updates (CPUs). The 5th digit in
the Oracle version is incremented for each PSU released against the patch set
version.
When installing
interim patches on top of 11.1.0.7.1 PSU, it is possible to encounter
an Opatch error code 73 as seen in the example below.
[oracle@g###01a patches]$ unzip p7006588_111071_Linux-x86-64.zip
Archive: p7006588_111071_Linux-x86-64.zip
creating: 7006588/
creating: 7006588/files/
creating: 7006588/files/lib/
creating: 7006588/files/lib/libserver11.a/
inflating: 7006588/files/lib/libserver11.a/kfn.o
inflating: 7006588/files/lib/libserver11.a/kfnc.o
inflating: 7006588/files/lib/libserver11.a/ksmp.o
inflating: 7006588/files/lib/libserver11.a/kspt.o
creating: 7006588/etc/
creating: 7006588/etc/config/
inflating: 7006588/etc/config/inventory.xml
inflating: 7006588/etc/config/actions.xml
creating: 7006588/etc/xml/
inflating: 7006588/etc/xml/GenericActions.xml
inflating: 7006588/etc/xml/ShiphomeDirectoryStructure.xml
inflating: 7006588/README.txt
[oracle@g###01a patches]$ cd 7006588/
[oracle@g###01a 7006588]$ opatch apply
Invoking OPatch 11.1.0.6.2
Oracle Interim Patch Installer version 11.1.0.6.2
Copyright 2007, Oracle Corporation. All rights reserved.
Central Inventory : /u01/app/oraInventory
from : /etc/oraInst.loc
OPatch version : 11.1.0.6.2
OUI version : 11.1.0.7.0
OUI location : /u01/app/oracle/product/11.1.0/asm/oui
Log file location : /u01/app/oracle/product/11.1.0/asm/cfgtoollogs/opatch/opatch2009-12-08_15-07-33PM.log
ApplySession failed: Patch ID is null.
System intact, OPatch will not attempt to restore the system
OPatch failed with error code 73
Upgrade OPatch to
11.1.0.6.8 or higher will resolve the issue.
Oracle recommends
that all customers be on the latest version of OPatch.
Oracle Doc ID: 224346.1 provides the instructions to update OPatch to the
latest version.
The CRS Bundle
Patch has been renamed as CRS PSU. CRS PSU and RDBMS PSU are two separate
patches.
For 11.1.0.7, the
CRS PSU is patch 8287931 and the RDBMS PSU is patch 8833297.
Despite what the
README.txt file says in patch 8287931, using opatch napply to install and
invoking [pre/post]rootpatch.sh may result in numerous errors, even causing
corruption to your CRS Oracle Home.
As stated in
Oracle Documentation (Oracle®
Universal Installer and OPatch User's Guide), “The
opatch auto command automates all of these tasks for patching the CRS home and
all other applicable RDBMS homes”. The auto option has been available since
Oracle 10gR2. Although a great utility, the auto option is not without
problems.
<CRSHome>/Opatch/opatch auto <patch_directory>
Opatch is
bundled with Oracle Configuration Manager. The OCM path must include an actual
ocm.rsp response file and not just path.
E.g.
<CRSHome>/OPatch/ocm/bin/ocm.rsp
instead
of:
<CRSHome>/OPatch/ocm/bin
To generate
a OCM response file, execute the following commands as root user
cd <CRSHome>/OPatch/ocm/bin
<CRSHome>/OPatch/ocm/bin/emocmrsp
[root@d###01a
PSUpatches]$ <CRSHome>/OPatch/opatch auto /u02/PSUpatches/8287931
auto_patch.pl:
log file is /u02/PSUpatches/opatchauto_2010-05-27_15:12:45.log
Discovering
environment to patch
Checking
if Clusterware is up
Looking
for configured cluster nodes
. .
Oracle
version for Oracle Home /u01/app/oracle/product/11.1.0/asm/ is 11.1.0.7.0
The
patch is applicable for this Oracle Home /u01/app/oracle/product/11.1.0/asm/
verify
failure: exit code 32512 for cat
Action failed, do you want to retry it? (y/n/abort/N/N1-N2/help):
n
ignored at user request, continuing
This error
can be ignored as confirmed in Doc ID: 844983.1
You cannot
continue from this error, Opatch will exit. To discover why I had hit this
particular error I ran the following command manually from the opatch auto
screen output:
Executing
/u01/app/oracle/product/11.1.0/crs/OPatch/opatch napply -local -silent -ocmrf
/u01/app/oracle/product/11.1.0/crs/OPatch/ocm/bin/ocm.rsp -oh
/u01/app/oracle/product/11.1.0/crs -id 8287931 as oracle on d****01a
verify
failure: exit code 18688 for patch_crs
Action
failed, do you want to retry it? (y/n/abort/N/N1-N2/help):
Re-executing
command manually as oracle user:
[oracle@D****01A
111070]$ /u01/app/oracle/product/11.1.0/crs/OPatch/opatch napply -local -silent
-ocmrf /u01/app/oracle/product/11.1.0/crs/OPatch/ocm/bin/ocm.rsp -oh
/u01/app/oracle/product/11.1.0/crs -id 8287931
Invoking
OPatch 11.1.0.6.9
Oracle
Interim Patch Installer version 11.1.0.6.9
Copyright
(c) 2009, Oracle Corporation. All rights
reserved.
UTIL
session
Oracle
Home :
/u01/app/oracle/product/11.1.0/crs
Central
Inventory : /u01/app/oraInventory
from : /etc/oraInst.loc
OPatch
version : 11.1.0.6.9
OUI
version : 11.1.0.7.0
OUI
location :
/u01/app/oracle/product/11.1.0/crs/oui
Log
file location :
/u01/app/oracle/product/11.1.0/crs/cfgtoollogs/opatch/opatch2010-05-27_15-16-23PM.log
Patch
history file:
/u01/app/oracle/product/11.1.0/crs/cfgtoollogs/opatch/opatch_history.txt
Invoking
utility "napply"
Checking
conflict among patches...
Checking
if Oracle Home has components required by patches...
Checking
conflicts against Oracle Home...
OPatch
continues with these patches: 8287931
Do
you want to proceed? [y|n]
Y
(auto-answered by -silent)
User
Responded with: Y
Running
prerequisite checks...
Prerequisite
check "CheckApplicable" failed.
The
details are:
Patch
8287931:
Copy
Action: Destination File
"/u01/app/oracle/product/11.1.0/crs/install/onsconfig" is not
writeable.
'oracle.crs,
11.1.0.7.0': Cannot copy file from 'onsconfig' to
'/u01/app/oracle/product/11.1.0/crs/install/onsconfig'
UtilSession
failed: Prerequisite check "CheckApplicable" failed.
OPatch
failed with error code 73
From the
above output we see that file: /u01/app/oracle/product/11.1.0/crs/install/onsconfig is not writeable.
To fix the
permission problem, execute the following command as root user:
chmod
754 /u01/app/oracle/product/11.1.0/crs/install/onsconfig
Then startup
CRS stack and re-execute opatch auto.
You cannot
continue from this error, Opatch will fail and exit. This error relates to a
problem with ssh between the nodes.
The Opatch
log file shows the error which occurs after the perl script auto_patch.pl invokes the “cat” command:
Wed
Jun 9 08:50:17 2010: Oracle version for Oracle Home /u01/app/oracle/product/11.1.0/asm
is 11.1.0.7.0
Wed
Jun 9 08:53:39 2010: remote: invoked with -v cat oracle d****01a n
/bin/cat /u01/app/oracle/product/11.1.0/crs/a.txt
Wed
Jun 9 08:53:39 2010: run: invoked with -v cat oracle n
/usr/bin/ssh d****01a /u01/app/oracle/product/11.1.0/crs/OPatch/crs/auto_patch.pl
-x -v 0 -oh /u01/app/oracle/product/11.1.0/crs -och
/u01/app/oracle/product/11.1.0/crs -patchn 8287931 -patchdir /patchdir /bin/cat
/u01/app/oracle/product/11.1.0/crs/a.txt
Wed
Jun 9 08:53:39 2010: run: exit
code 65280 after /usr/bin/ssh d****01a
/u01/app/oracle/product/11.1.0/crs/OPatch/crs/auto_patch.pl -x -v 0 -oh
/u01/app/oracle/product/11.1.0/crs -och /u01/app/oracle/product/11.1.0/crs
-patchn 8287931 -patchdir /patchdir /bin/cat /u01/app/oracle/product/11.1.0/crs/a.txt
Wed
Jun 9 08:53:39 2010: verify: invoke with cat /patchdir/opatchauto_2010-06-09_08:39:42.2.tmp
65280
Wed
Jun 9 08:53:39 2010: osimulate is
Wed Jun 9 08:53:39 2010: verify failure: exit code 65280 for cat
Wed
Jun 9 08:53:39 2010: Action failed, do you want to retry it?
(y/n/abort/N/N1-N2/help):
Interestingly
the tmp file provides the real error:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: POSSIBLE DNS SPOOFING
DETECTED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The
RSA host key for d****01a has changed,
and
the key for the according IP address 196.6.204.10
is
unknown. This could either mean that
DNS
SPOOFING is happening or the IP address for the host
and
its host key have changed at the same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS
CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT
IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone
could be eavesdropping on you right now (man-in-the-middle attack)!
It
is also possible that the RSA host key has just been changed.
The
fingerprint for the RSA key sent by the remote host is
50:94:28:e6:b1:18:7b:ac:d2:ee:d0:33:4c:b8:6c:1f.
Please
contact your system administrator.
Add
correct host key in /home/oracle/.ssh/known_hosts to get rid of this message.
Offending
key in /home/oracle/.ssh/known_hosts:4
RSA
host key for ****01a has changed and you have requested strict checking.
Host
key verification failed.
From the
above output we see there is an issue with RSA key fingerprint #4 causing problems
with ssh to remote node. The solution is:
_______________________________________________________________________________
Did you find the article useful?
Please provide your feedback by voting now.
If you have a comment or question, please complete and submit the form below.
_______________________________________________________________________________