Tuesday, May 22, 2012

E-Business Suite RMAN based Backup Implementation Strategy




Implemented Backup Strategy
Production Machine
10.10.0.89 – erp.mydomain.com is production server on which Sun Solaris Release 10 Operating System is being used. EBS R12 has been installed in multi user environment. APPDEV is the owner of EBS Application Tier and ORADEV is the owner of Database Tier.

Standby Machine

10.40.0.89 – erp2.mydomain.com is standby server on which Sun Solaris Release 10 Operating System is being used. EBS R12 has been cloned in multi user environment. APPDEV is the owner of EBS Application Tier and ORADEV is the owner of Database Tier. Database is synchronized through Oracle Standby Database Technology and EBS Application is synchronized through Operating System utility named “rsync”. Standby Database is operating in Maximum Availability protection mode and DataGurad Broker is enabled too.
In our EBS R12 environment three types of backups are being performed,
1.       Daily and Hourly RMAN Based Full and Incremental Backups of DEV database
2.       Full Physical Offline Backup of Database and Application tier performed from Monday to Friday
3.       EBS Application Tier Synchronization at Operating System Level through Crontab


RMAN Based Backups

Full online RMAN based backup is performed on daily basis. RMAN is configured in catalog mode, the backup of RMAN schema is also performed using export utility. After execution of backup procedure created backup files are transferred to tape media.
rman target sys/sysdev@devprmy rcvcat rman/rman@devprmy
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION ON;
CONFIGURE DEFAULT DEVICE TYPE TO 'SBT_TAPE';
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE 'SBT_TAPE' TO '%F';
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE SBT_TAPE PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE 'SBT_TAPE' TO 1;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE 'SBT_TAPE' TO 1;
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT   'ORA_DF%t_s%s_s%p';
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' FORMAT   'ORA_DF%t_s%s_s%p' PARMS  'ENV=(TDPO_OPTFILE=/opt/tivoli/tsm/client/oracle/bin64/tdpo.opt)';
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BZIP2'; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/development/oradev/snapcf_DEV.f';



Crontab Settings

root@erp.mydomain.com # crontab -l oradev
0 6 * * * /export/home/oradev/devbkp.sh
0 8-6 * * * /export/home/oradev/devarchbkp.sh

Full Backup

RMAN based Full Online Backup is performed at 06:00 a.m. on daily basis.

Incremental Backup

RMAN based Incremental Backup of Archive Log Files is performed on hourly basis.

Backup Script

/export/home/oradev/devbkp.sh
LOG_FILE="DEVBKP_`hostname`_`date '+%d%m%y_%H%M'`.txt"
echo "**Start Date: `date '+%d%m%y_%H%M'`*********************">>$LOG_FILE
cd /export/home/oradev>>$LOG_FILE
. .profile>>$LOG_FILE
rman target sys/sysdev@devprmy rcvcat rman/rman@devprmy cmdfile devbkpscr.scr msglog devbkplog.log>>$LOG_FILE
echo "**End Date: `date '+%d%m%y_%H%M'`*********************">>$LOG_FILE
FILE_CT="DEVBKP_`hostname`_DATA"
echo "Catalog Start Date:`date '+%d%m%y_%H%M'`******************">>$FILE_CT.txt
expdp rman/rman@devprmy DIRECTORY=expdp DUMPFILE=$FILE_CT.dmp LOGFILE=$FILE_CT.log>>$FILE_CT.txt
dsmc selective "/development/oradev/DEV/db/apps_st/data/$FILE_CT.dmp">>$FILE_CT.txt
rm -f /development/oradev/DEV/db/apps_st/data/$FILE_CT.dmp>>$FILE_CT.txt
echo "Catalog End Date:`date '+%d%m%y_%H%M'`******************"
/export/home/oradev/devbkpscr.scr
run
{
allocate channel t1 type 'sbt_tape';
backup database;
backup archivelog all format 'archivelogs_%d_%u_%s_%T' not backed up 2 times;
delete noprompt archivelog all backed up 2 times to device type 'sbt_tape' completed before 'sysdate-3';
release channel t1;
}


/export/home/oradev/devarchbkp.sh
echo "**Start Date: `date '+%d%m%y_%H%M'`*********************">>$LOG_FILE
cd /export/home/oradev>>$LOG_FILE
. .profile>>$LOG_FILE
rman target sys/sysdev@devprmy rcvcat rman/rman@devprmy cmdfile devarchbkpscr.scr msglog devarchbkplog.log>>$LOG_FILE
echo "**End Date: `date '+%d%m%y_%H%M'`*********************">>$LOG_FILE
FILE_CT="DEVBKP_`hostname`_ARC"
echo "Catalog Start Date:`date '+%d%m%y_%H%M'`******************">>$FILE_CT.txt
expdp rman/rman@devprmy DIRECTORY=expdp DUMPFILE=$FILE_CT.dmp LOGFILE=$FILE_CT.log>>$FILE_CT.txt
dsmc selective "/development/oradev/DEV/db/apps_st/data/$FILE_CT.dmp">>$FILE_CT.txt
rm -f /development/oradev/DEV/db/apps_st/data/$FILE_CT.dmp>>$FILE_CT.txt
echo "Catalog End Date:`date '+%d%m%y_%H%M'`******************">>$FILE_CT.txt
/export/home/oradev/devarchbkpscr.scr
run
{
allocate channel t2 type 'sbt_tape';
backup incremental level 1 database plus archivelog all format 'archivelogs_%d_%u_%s_%T' not backed up 2 times;
release channel t2;
}

Log Files Location

/export/home/oradev/
Backup Locations
Backup sets are directly generated on tape media


Full Physical Offline backup of EBS R12 Application and Database instance on 10.10.0.89 named DEV is performed five days a week from Monday to Friday. This activity script is scheduled to be executed by OS crontab utility. This script shutdown all Application and Database Services, Create TAR file of DEV instance, starts all Application and Database Services, copy generated TAR file to USB, TAPE and keeps two days Backup Files on local machine as well.
Backup times
This activity is starts at 23:00 and completes at 07:15 AM next day.
Crontab Entry
0 23 * * 1,2,3,4,5 /devbkp.sh
Backup scripts
/devbkp.sh
LOG_FILE="DEVBK_`date '+%d%m%y_%H%M'`.txt"
BKORANAME="DEV_BK_ORADEV_`date '+%d%m%y_%H_%M'`.tar"
BKAPPNAME="DEV_BK_APPDEV_`date '+%d%m%y_%H_%M'`.tar"
cd /development/
echo "Apps Services Down: Start*****`date '+%d%m%y_%H%M'`*****">>$LOG_FILE
su appdev -c "/development/appdev/DEV/inst/apps/DEV_erp/admin/scripts/adstpall.sh apps/appsdevadmin098" >>$LOG_FILE
sleep 45
echo "adstpall executed successfully">>$LOG_FILE
echo "******************************">>$LOG_FILE
echo "DB Service Down: Start****">>$LOG_FILE
su oradev -c "/development/oradev/DEV/db/tech_st/11.1.0/appsutil/scripts/DEV_erp/addbctl.sh stop immediate" >>$LOG_FILE
echo "addbctl stop executed successfully">>$LOG_FILE
sleep 60
su oradev -c "/development/oradev/DEV/db/tech_st/11.1.0/appsutil/scripts/DEV_erp/addlnctl.sh stop DEV" >>$LOG_FILE
echo "addlnctl stop executed successfully">>$LOG_FILE
echo "tar creation start `date '+%d%m%y_%H%M'`">>$LOG_FILE
rm /test/devbackup/oldcopy/*
mv /test/devbackup/localcopy/* /test/devbackup/oldcopy/
cd /development/
tar -cvf /test/devbackup/externalcopy/$BKORANAME oradev/
tar -cvf /test/devbackup/externalcopy/$BKAPPNAME appdev/
echo "tar created end `date '+%d%m%y_%H%M'`">>$LOG_FILE
echo "***********************************">>$LOG_FILE
echo "DB Services Startup: Start****">>$LOG_FILE
echo "*********************************">>$LOG_FILE
su oradev -c "/development/oradev/DEV/db/tech_st/11.1.0/appsutil/scripts/DEV_erp/addbctl.sh start" >>$LOG_FILE
echo "addbctl started successfully">>$LOG_FILE
su oradev -c "/development/oradev/DEV/db/tech_st/11.1.0/appsutil/scripts/DEV_erp/addlnctl.sh start DEV" >>$LOG_FILE
sleep 60
echo "addlnctl started successfully">>$LOG_FILE
echo "*****************************">>$LOG_FILE
echo "Apps Services Startup: Start**">>$LOG_FILE
echo "*****************************">>$LOG_FILE
su appdev -c "/development/appdev/DEV/inst/apps/DEV_erp/admin/scripts/adstrtal.sh apps/appsdevadmin098" >>$LOG_FILE
echo "adstrtal started successfully">>$LOG_FILE
mount -F ufs /dev/dsk/c6t0d0s0 /USB-Storage
cp /test/devbackup/externalcopy/* /USB-Storage/devbackup
umount /USB-Storage
echo "tar file transferred to USB `date '+%d%m%y_%H%M'`">>$LOG_FILE
cd /test/devbackup/externalcopy/
dsmc selective * -subdir=yes
cd /development/
echo "tar file transferred to tape `date '+%d%m%y_%H%M'`">>$LOG_FILE
mv /test/devbackup/externalcopy/* /test/devbackup/localcopy/
echo "tar file moved ">>$LOG_FILE
echo "End*****`date '+%d%m%y_%H%M'`*****">>$LOG_FILE

Backup locations

/test/devbackup/externalcopy/
/test/devbackup/localcopy/
/test/devbackup/oldcopy/
/USB-Storage/devbackup/



Operating System Utility named “rsync” is used to synchronize EBS Application Tier between the Primary and Standby Site. Crontab utility is used to execute “rsync” script twice a day, first at 13:00 and then at 22:00.

Script To Be executed

The script executed by “crontab” is,
/export/home/appdev/sncdev.sh
LOG_FILE="DEVAPPS_SYNC_`hostname`_`date '+%d%m%y_%H%M'`.txt"
cd /export/home/appdev
. .profile
echo "**CONC LOG SYNC- Start Date: `date '+%d%m%y_%H%M'`*********************">>$LOG_FILE
rsync -av -e ssh /development/appdev/DEV/inst/apps/DEV_erp/logs/appl/conc/ erp2:/development/appdev/DEV/inst/apps/DEV_erp/logs/appl/conc>>$LOG_FILE
echo " ">>$LOG_FILE
echo "**CONC LOG SYNC-End Date: `date '+%d%m%y_%H%M'`*********************">>$LOG_FILE
echo " ">>$LOG_FILE
echo " ">>$LOG_FILE
echo "**APPL SYNC- Start Date: `date '+%d%m%y_%H%M'`*********************">>$LOG_FILE
rsync -av -e ssh /development/appdev/DEV/apps/apps_st/appl/ erp2:/development/appdev/DEV/apps/apps_st/appl>>$LOG_FILE
echo " ">>$LOG_FILE
echo "**APPL SYNC-End Date: `date '+%d%m%y_%H%M'`*********************">>$LOG_FILE
echo " ">>$LOG_FILE
echo " ">>$LOG_FILE
echo "**COMMNTOP SYNC- Start Date: `date '+%d%m%y_%H%M'`*********************">>$LOG_FILE
rsync -av -e ssh /development/appdev/DEV/apps/apps_st/comn/ erp2:/development/appdev/DEV/apps/apps_st/comn>>$LOG_FILE
echo " ">>$LOG_FILE
echo "**COMMNTOP SYNC-End Date: `date '+%d%m%y_%H%M'`*********************">>$LOG_FILE
echo " ">>$LOG_FILE
echo " ">>$LOG_FILE

Crontab Entry for appdev

The execution scheduling details is,
0 13 * * * /export/home/appdev/sncdev.sh                       (to be executed at 13:00 everyday)
0 22 * * * /export/home/appdev/sncdev.sh                       (to be executed at 22:00 everyday)

Log File Location

                /export/home/appdev/

1 comment: