Grid and Oracle Home Housekeeping, in 11.2 with cron jobs and opatch util cleanup

Grid and oracle home filesystem space exhaust problems, is a real headache which needs to be handled before it happens.

It is not a "nice to have" job; because, you will find yourself in following situations ultimately:

- getting up at midnight because of a call coming from 7*24 personnel that your db archive process can not write redo

- you can find yourself in a situation that trying to find out who ate my disk space in the middle of patch apply/PSU upgrade scenario.

If you are not motivated enough, so here comes the results:

Below there is the filesystem size before the operation:
grid@node1:/u01/app/11.2.0:>df -g /u01
/dev/fslv00 99.00 28.50 72% 120470 2% /u01

After the cleanup:
grid@node1:/u01/app/11.2.0:>df -g /u01
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/fslv01 96.88 49.21 50% 89422 1% /u01



So what I did?

First of all I had set the critical alarm level of filesystem usage to 70 pecent, which was 95 before. And I got the alarms...

In order for cleaning up of u01 filesystem, I had used "opatch util cleanup" for unnecessary opatch stored files and cron jobs to age out or remove old trace or log files.

1. CRONTAB SCRIPTS:

oracle@node1:/u01/app/11.2.0/grid>crontab –l

## trace cleanup
0 15 * * * /usr/bin/find /u01/app/oracle/diag/rdbms/mydb/MYDB1/trace -type f -name "MYDB1*tr*" -mtime +10 -exec rm {} \; > /dev/null 2>&1

## audit cleanup
0 15 * * * /usr/bin/find /u01/app/oracle/admin/MYDB/adump -type f -name "*aud" -mtime +10 -exec rm {} \; > /dev/null 2>&1

## agent emd cleanup
0 15 * * * /usr/bin/find /u01/app/agent/agent11g/node1/sysman/emd -type f -name "*dmp" -mtime +3 -exec rm {} \; > /dev/null 2>&1
0 15 * * * /usr/bin/find /u01/app/agent/agent11g/node1/sysman/emd -type f -name "*trc" -mtime +3 -exec rm {} \; > /dev/null 2>&1
0 15 * * * /usr/bin/find /u01/app/agent/agent11g/node1/sysman/emd -type f -name "*core*" -mtime +3 -exec rm {} \; > /dev/null 2>&1

grid@node1:/home/grid:>crontab -l

## asm trace files cleanup
14 0 * * * /usr/bin/find /u01/app/grid/diag/asm/+asm/+ASM1/trace -type f -name "*ASM*.tr*" -mtime +20 -exec rm {} \; > /dev/null 2>&1

## listener log cleanup
15 0 * * * /usr/bin/find /u01/app/grid/diag/tnslsnr/node1/listener*/alert -type f -name "log*.xml" -mtime +3 -exec rm {} \; > /dev/null 2>&1

## audit cleanup
16 0 * * * /usr/bin/find /u01/app/11.2.0/grid/rdbms/audit -type f -name "*aud" -mtime +1 -exec rm {} \; > /dev/null 2>&1

## listener log aging
17 0 * * 0 /usr/bin/compress -f /u01/app/grid/diag/tnslsnr/node1/listener/trace/listener.log
17 0 * * 0 /usr/bin/compress -f /u01/app/11.2.0/grid/log/diag/tnslsnr/node1/listener_scan1/trace/*log
17 0 * * 0 /usr/bin/compress -f /u01/app/11.2.0/grid/log/diag/tnslsnr/node1/listener_scan2/trace/*log
17 0 * * 0 /usr/bin/compress -f /u01/app/11.2.0/grid/log/diag/tnslsnr/node1/listener_scan3/trace/*log

# rdbms log cleanup
0 15 * * * /usr/bin/find /u01/app/11.2.0/grid/rdbms/log -type f -name "*.tr*" -mtime +3 -exec rm {} \; > /dev/null 2>&1



2. OPATCH CLEANUP

DB HOME
oracle@node1:/u01/app/11.2.0/grid>opatch util cleanup
Invoking OPatch 11.2.0.1.5

Oracle Interim Patch Installer version 11.2.0.1.5
Copyright (c) 2010, Oracle Corporation. All rights reserved.

UTIL session

Oracle Home : /u01/app/oracle/product/11.2.0/dbhome_1
Central Inventory : /u01/app/oraInventory
from : /etc/oraInst.loc
OPatch version : 11.2.0.1.5
OUI version : 11.2.0.2.0
OUI location : /u01/app/oracle/product/11.2.0/dbhome_1/oui
Log file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2012-01-25_11-41-54AM.log

Patch history file: /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch_history.txt

Invoking utility "cleanup"
OPatch will clean up 'restore.sh,make.txt' files and 'rac,scratch,backup' directories.
You will be still able to rollback patches after this cleanup.
Do you want to proceed? [y|n]
y
User Responded with: Y
Size of directory "/u01/app/oracle/product/11.2.0/dbhome_1/.patch_storage" before cleanup is 1159144429 bytes.
Size of directory "/u01/app/oracle/product/11.2.0/dbhome_1/.patch_storage" after cleanup is 526164263 bytes.

UtilSession: Backup area for restore has been cleaned up. For a complete list of files/directories
deleted, Please refer log file.

OPatch succeeded.

GRID HOME
grid@node1:/u01/app/11.2.0:>opatch util cleanup
Invoking OPatch 11.2.0.1.5

Oracle Interim Patch Installer version 11.2.0.1.5
Copyright (c) 2010, Oracle Corporation. All rights reserved.

UTIL session

Oracle Home : /u01/app/11.2.0/grid
Central Inventory : /u01/app/oraInventory
from : /etc/oraInst.loc
OPatch version : 11.2.0.1.5
OUI version : 11.2.0.2.0
OUI location : /u01/app/11.2.0/grid/oui
Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2012-01-25_11-38-10AM.log

Patch history file: /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch_history.txt

Invoking utility "cleanup"
OPatch will clean up 'restore.sh,make.txt' files and 'rac,scratch,backup' directories.
You will be still able to rollback patches after this cleanup.
Do you want to proceed? [y|n]
y
User Responded with: Y
Size of directory "/u01/app/11.2.0/grid/.patch_storage" before cleanup is 40399439700 bytes.
Size of directory "/u01/app/11.2.0/grid/.patch_storage" after cleanup is 23080457091 bytes.

UtilSession: Backup area for restore has been cleaned up. For a complete list of files/directories
deleted, Please refer log file.

OPatch succeeded.

Opatch cleanup MOS Note:
How To Avoid Disk Full Issues Because OPatch Backups Take Big Amount Of Disk Space. [ID 550522.1]

4. ) In preparation for patching systems, is there any way to predict the amount of space that will be required in $ORACLE_HOME/.patch_storage?