HA概述
HA(High Available)即由两台计算机组成并对外提供一组相同的服务，也叫做一主一备模式。正常情况下由主服务器提供服务，备服务器处于待机备用，备机可以随时接替主服务器的工作。也就是当主服务器宕机或所提供的服务不可用时，备用服务器会主动激活并且替换主服务器继续提供服务，这时主服务器上的服务资源包括网络(IP)、存储、服务(Web/数据库)就会转移到备机接管，从而提供不间断的服务。基于此，便可以将服务器的宕机时间减少到最低，对业务实现不中断或短暂中断。
由一组计算机(多台)组成一个整体并向用户提供相同的网络资源或服务，这种模式叫做HA集群(High Available Cluster)。
Oracle HA
Oracle HA高可用，又叫做双机热备，一般用于关键性业务。
Oracle双机热备模式
常用的有Data Guard、RAC(Real Application Clusters)、基于HA软件实现的双机热备。
作用和区别
l Data Guard采用重做日志复制技术，对主业务数据进行实时的异步同步备份，有一主一备、一主多备模式，一般常用于容灾(异地备份、灾难恢复)；存储独立、数据完整备份；备机可以做数据分析、报表统计等；主备自由切换，能实现自动快速故障转移；备用角色强制转换为主用角色会破环Data Guard架构；设计复杂、维护难度大
l RAC多个实例同时运行，无主备概念，有集群负载功能，其中某一台down机，不影响整体服务，不存在故障切换时间，可以提供高性能服务；存储共享，由ASM管理存储；硬件成本低；设计简单、维护难度大
l HA双机热备可以提供高可用性，保证业务的持续稳定运行，可以实现自动快速故障转移，存在短暂的切换时间(10-30s)。一般用于关键性业务；存储共享；有商业的HA方案，也可以使用开源的高可用软件keepalved、heartbeat搭建实现HA方案；设计复杂、维护难度小
自由组合
l HA双机和Data Guard
l Data Guard和RAC
Oracle HA+Keepalived架构
l Oracle HA双机热备+Keepalived
l Oracle HA双机+DataGuard+Keepalived集群
搭建Oracle HA主备服务器
软件版本：
Orace 11g R2
Keepalived 1.3.2
主服务器: DB1
备服务器: DB2
环境介绍
|
Hostname
IP
OS
Role
hmdg-db1
172.16.10.25
CentOS6.9
MASTER
hmdg-db2
172.16.10.26
CentOS6.9
BACKUP
VIP
172.16.10.130、172.16.10.131
Share Disk
/dev/sdb1 mount on /oradata
ORACLE_BASE
/u01/app/oracle
ORACLE_HOME
/u01/app/oracle/product/11.2.0/db_1
ORACLE_SID
HMODB
Datadir
/oradata/HMODB
Controlfiles
/oradata/HMODB/control01.ctl,/u01/app/oracle/flash_recovery_area/HMODB/control02.ctl,
/home/oracle/rman/HMODB/control03.ctl
Oracle软件安装步骤
1、 同时在主备上安装Oracle数据库软件，数据库的安装目录和环境变量保持一致
2、 仅在主服务器上建立数据库实例(主服务器上挂载共享存储并使用DBCA新建库)
3、 主服务创建数据库实例后，将主库上的控制文件、参数文件(SPFILE)、以及密码文件传输到备用服务器上相应的目录
l 备用服务器DB2
##创建对应的数据库文件目录
[root@hmdg-db2~]#su-oracle
[oracle@hmdg-db2~]$mkdir/u01/app/oracle/admin/HMODB
[oracle@hmdg-db2~]$mkdir/u01/app/oracle/admin/HMODB/{adump,dpdump,pfile}
[oracle@hmdg-db2~]$mkdir/u01/app/oracle/flash_recovery_area/HMODB
##密码文件
[oracle@hmdg-db2~]$scporacle@hmdg-db1:/u01/app/oracle/product/11.2.0/db_1/dbs/orapwHMODB/u01/app/oracle/product/11.2.0/db_1/dbs/orapwHMODB
##SPFILE参数文件
[oracle@hmdg-db2~]$scporacle@hmdg-db1:/u01/app/oracle/product/11.2.0/db_1/dbs/spfileHMODB.ora/u01/app/oracle/product/11.2.0/db_1/dbs/spfileHMODB.ora
##控制文件，仅需要将位于非共享存储的数据目录下的控制文件传输到其他磁盘或目录位置下
[oracle@hmdg-db2~]$scporacle@hmdg-db1:/u01/app/oracle/flash_recovery_area/HMODB/control02.ctl/u01/app/oracle/flash_recovery_area/HMODB/control02.ctl
[oracle@hmdg-db2~]$cp/u01/app/oracle/flash_recovery_area/HMODB/control02.ctl/home/oracle/rman/HMODB/control03.ctl
在备用服务器上启动数据库
1、 主服务器上关闭数据库实例
2、 主服务器上卸载共享存储
3、 备服务器上挂载共享存储
4、 备服务器上启动监听
5、 备服务器上启动数据库实例
注意：不要同时挂载共享存储启动数据库实例，否则会导致数据不一致
l 主服务器DB1
##关闭数据库实例
[oracle@hmdg-db1~]$sqlplus/assysdba
SQL>shutdownimmediate
##关闭数据库监听
[oracle@hmdg-db1~]$lsnrctlstop
##卸载共享存储
[oracle@hmdg-db1~]$umount/oradata
l 备服务器DB2
##挂载/dev/sdb1共享存储到/oradta下
[root@hmdg-db2~]#mount/dev/sdb1/oradata/
##启动数据库监听
[oracle@hmdg-db2~]$lsnrctlstart
[oracle@hmdg-db2~]$exportORACLE_SID=HMODB
##启动数据库实例
[oracle@hmdg-db2~]$sqlplus/assysdba
SQL>startup
ORACLEinstancestarted.
TotalSystemGlobalArea1603411968bytes
FixedSize2213776bytes
VariableSize1056966768bytes
DatabaseBuffers536870912bytes
RedoBuffers7360512bytes
Databasemounted.
DatabaSEOpened.
SQL>
##检验数据库
SQL>SHOWPARAMETERDB_NAME
NAMETYPEVALUE
-----------------------------------------------------------------------------
db_namestringHMODB
SQL>SELECTOPEN_MODE,NAME,DATABASE_ROLEFROMV$DATABASE;
OPEN_MODENAMEDATABASE_ROLE
---------------------------------------------
READWRITEHMODBPRIMARY
再次手动切换步骤
由于安装数据库实例时，创建了多个控制文件，并且这些控制文件都保存在不同的磁盘上，在主备进行切换的时，位于非共享存储的控制文件需要手动进行同步。这是由于在关闭数据库实例时Oracle同时将当前数据库状态信息记录到控制文件中，在没有启动数据库实例的备机就无法同步自身的其他的控制文件。这时就需要使用共享存储里的控制文件进行恢复。
1、 在DB2关闭正在运行数据库实例
##DB2:
[oracle@hmdg-db2~]$sqlplus/assysdba
SQL>shutdownimmediate
[root@hmdg-db2~]#umount/oradata/
2、在DB1上进行恢复启动数据库实例
##DB1：
[root@hmdg-db1~]#mount/dev/sdb1/oradata/
[oracle@hmdg-db1~]$exportORACLE_SID=HMODB
[oracle@hmdg-db1~]$sqlpus/assysdba
SQL>startupnomount
SQL>alterdatabasemount;
SQL>alterdatabaSEOpen;
如果控制文件在不同的磁盘目录，那么就需要执行这一步恢复控制文件
[oracle@hmdg-db1~]$rmantarget/
RMAN>restorecontrolfilefrom'/oradata/HMODB/control01.ctl';
启动数据库监听
[oracle@hmdg-db1~]$lsnrctlstart
或者直接只有操作系统的cp、scp等命令进行拷贝恢复
[oracle@hmdg-db1~]$cp/oradata/HMODB/control01.ctl/u01/app/oracle/flash_recovery_area/HMODB/control02.ctl
SQL>SELECTNAME,OPEN_MODE,DATABASE_ROLEFROMV$DATABASE;
NAMEOPEN_MODENAMEDATABASE_ROLE
------------------------------------------------------
HMODBREADWRITEHMODBPRIMARY
主备服务SSH密钥配置
配置oracle用户的ssh密钥，用于主备通过oracle身份自动检测对方的数据共享存储状态。在keepalived脚本管理中自动检查对方的挂载状态，并在主备切换的过程中，待接管的服务器等待对方umount共享存储之后，正常启动数据库实例。（如果主服务异常关机或网络中断，不需要等待对方umount存储，主服务器异常关机时keepalived会即时检测到并自动切换到备机上）
[oracle@hmdg-db1~]$ssh-keygen-trsa-b2048
[oracle@hmdg-db1~]$ssh-copy-id-i~/.ssh/id_rsa.puboracle@hmdg-db2
[oracle@hmdg-db2~]$ssh-keygen-trsa-b2048
[oracle@hmdg-db2~]$ssh-copy-id-i~/.ssh/id_rsa.puboracle@hmdg-db1
添加主备防火墙互通规则
#iptables-IINPUT-ptcp-s172.16.10.0/24-jACCEPT
#iptables-IINPUT-pudp-s172.16.10.0/24-jACCEPT
Keepalived配置和管理脚本
keepalived配置
主服务器的keepalived配置，备用服务器需要做相应的修改
!ConfigurationFileforkeepalived
global_defs{
notification_email{
mail@huangming.org
}
notification_email_fromAlexandre.Cassen@firewall.loc
smtp_server127.0.0.1
smtp_connect_timeout30
router_idORACLE_HA_MASTER#backup为ORACLE_HA_BACKUP
!vrrp_skip_check_adv_addr
}
vrrp_sync_groupORACLE_HA_GROUP{
group{
Oracle_HA_1#定义一个vrrp实例组
}
}
vrrp_scriptmonitor_oracle_status{#oracle数据库服务器状态检查脚本
script"/usr/local/keepalived/scripts/monitor.sh"
interval10#脚本执行时间间隔
fall3#脚本执行的最大失败次数
rise1#脚本执行成功一次则返回正常状态
weight20#脚本执行成功时的附加权值
}
vrrp_scriptchange_monitor_status{#服务器状态监控并修改主备检查脚本
script"/usr/local/keepalived/scripts/change_monitor_status.sh"
interval2
}
vrrp_instanceOracle_HA_1{
stateBACKUP#主备都设置为BACKUP
interfaceeth1#绑定vip的网络接口
virtual_router_id200#主备保存一致
priority100#备为90
advert_int2#主备状态同步时间间隔
nopreempt#主设置为不抢占模式
authentication{
auth_typePASS
auth_pass11112222
}
virtual_ipaddress{
172.16.10.130/24deveth1#需要绑定的VIP
172.16.10.131/24deveth1
}
track_script{
monitor_oracle_status
change_monitor_status
}
track_interface{
eth0#同时监控检测其他的网接口
}
notify_master"/usr/local/keepalived/scripts/keepalived_notify.shmaster"
notify_backup"/usr/local/keepalived/scripts/keepalived_notify.shbackup"
notify_fault"/usr/local/keepalived/scripts/keepalived_notify.shfault"
notify_stop"/usr/local/keepalived/scripts/keepalived_notify.shstop"
!notify管理脚本的定义
!分别对应keepalived转换为master、backup、fault、stop状态时执行的脚本
!在oracle的双机管理中主要是依赖这些脚本实现资源的接管
}
keepalived监控脚本
[root@hmdg-db1scripts]#ls-l/usr/local/keepalived/scripts/
-rwxr-xr-x1rootroot818Apr1420:13change_monitor_status.sh
-rw-r--r--1rootroot57Apr1420:13Controlfile.sql
-rw-r--r--1rootroot1308Apr1420:13keepalived
-rwxr-xr-x1rootroot10549Apr1423:18keepalived_notify.sh
-rwxr-xr-x1rootroot926Apr1423:12monitor_primary_oracle.sh
-rwxr-xr-x1rootroot895Apr1516:46monitor.sh
-rwxr-xr-x1rootroot895Apr1420:13monitor_standby_oracle.sh
-rw-r--r--1rootroot1657Apr1515:13oracle_init.sh
-rwxr-xr-x1rootroot8441Apr1420:13notify.sh_20180413
关于这些keepalived脚本，由于篇幅有限我没有全部贴到本文中，大家可以访问我的Github仓库找到：https://github.com/hmlinux/oracle-keepalived
l keepalived_notify.sh #keepalived的notify脚本，用于实现主备的资源自动切换
l monitor.sh #keepalved的监控脚本，监控oracle数据库运行状态、共享存储状态
l change_monitor_status.sh #异常或正常检测脚本，当备服务器处于正常运行状态，并且处于keepalived的BACKUP状态时，将自身的监控脚本修改为在备机就绪时执行都返回正常结果，主要作用时让正常的备机可以随时接管主服务器的资源
l monitor_primary_oracle.sh #处于MASTER状态时的服务器监控脚本
l monitor_standby_oracle.sh #处于BACKUP状态时的服务器监控脚本
l oracle_init.sh #notify脚本调用oracle启动、停止、状态检测命令
#!/bin/bash
#author:hm@huangming.org
#keepalivednotifyscript
#bindvipinterface
interface="eth1"
#keepalivedvirtual_ipaddress
virtual_ipaddress="172.16.10.130","172.16.10.131"
MASTER_HOSTNAME="hmdg-db1"#DB1
BACKUP_HOSTNAME="hmdg-db2"#DB2
LOGDIR="/usr/local/keepalived/log"
LOGFILE="$LOGDIR/keepalived_haswitch.log"
TMPLOG=/tmp/notify_.log
sharedisk="/dev/sdb1"
sharedisk_mount_point="/oradata"
#oracle_datadir="/oradata/path/HMODB"
oracle_datadir="/oradata"
#Sourceoracledatabaseinstancestartup/shutdownscript
./usr/local/keepalived/scripts/oracle_init.sh
control_files="/$oracle_datadir/$ORACLE_SID/control01.ctl","/u01/app/oracle/flash_recovery_area/$ORACLE_SID/control02.ctl","/home/oracle/rman/$ORACLE_SID/control03.ctl"
RETVAL=0
OLD_IFS="$IFS"
IFS=","
[-d$LOGDIR]||mkdir$LOGDIR
controlfile_backpath="/backup/oracle/control"
controlfile_back=$controlfile_backpath/control_$(date'+%Y%d%m%H%M%S')
[-d$controlfile_backpath]||mkdir-p$controlfile_backpath&&chown-Roracle:oinstall$controlfile_backpath
info_log(){
printf"$(date'+%b%d%T%a')$HOSTNAME[keepalived_notify]:$1"
}
control01_ctl=`printf${control_files[0]}`
backup_controlfile(){
su-oracle<<EOF
exportORACLE_SID=$ORACLE_SID
$ORACLE_HOME/bin/sqlplus-S"/assysdba"
alterdatabasebackupcontrolfileto'$controlfile_back';
exit
EOF
}
#runuser-loracle-c"exportORACLE_SID=$ORACLE_SID;rmantarget/cmdfile=/usr/scripts/ControlfileRestore.sql"
restore_controlfile(){
su-oracle<<EOF
exportORACLE_SID=$ORACLE_SID
$ORACLE_HOME/bin/rmantarget/nocatalog
RESTORECONTROLFILEFROM'$control01_ctl';
exit
EOF
}
ssh_p=$(netstat-ntp|awk'/sshd/{print$4}'|awk-F':''{print$2}'|head-1)
chk_remote_node_sharedisk(){
if[$(hostname)==$MASTER_HOSTNAME];then
ifping-c1-w2$BACKUP_HOSTNAME&>/dev/null;then
runuser-loracle-c"ssh-p$ssh_p$BACKUP_HOSTNAME2>/dev/nulldf|grep$sharedisk|wc-l"
fi
fi
if[$(hostname)==$BACKUP_HOSTNAME];then
ifping-c1-w2$MASTER_HOSTNAME&>/dev/null;then
runuser-loracle-c"ssh-p$ssh_p$MASTER_HOSTNAME2>/dev/nulldf|grep$sharedisk|wc-l"
fi
fi
}
master(){
info_log"DatabaseSwitchoverToMASTERn"
info_log"Checkremotenodesharediskmounted.n"
i=1
while(($i<=30))
do
chk_status=$(chk_remote_node_sharedisk)
if[$chk_status-ge1];then
info_log"$sharediskisalreadymountedonremotenodeorbusy.checking[$i]...n"
else
info_log"$sharediskcheckpassed.n"
break
fi
if[$i-eq20];then
info_log"Diskstatusabnormal.n"
exit1
fi
sleep1
i=$(($i+1))
done
ismount=$(df-h|grep$sharedisk|grep$sharedisk_mount_point|wc-l)
if[$ismount-eq0];then
info_log"mount$sharediskon$sharedisk_mount_pointn"
mount$sharedisk$sharedisk_mount_point
RETVAL=$?
if[$RETVAL-eq0];then
#shutdown_instance
info_log"restorecontrolfile1n"
startup_nomount
restore_controlfile
else
info_log"Error:$sharediskcannotmountor$sharedisk_mount_pointbusyn"
exit$RETVAL
fi
else
disk=$(df-h|grep$sharedisk_mount_point|awk'{print$1}')
if[$disk==$sharedisk];then
info_log"mount:$sharediskisalreadymountedon$sharedisk_mount_pointn"
else
info_log"Warning:$sharediskalreadymountedon$diskn"
fi
fi
status=$(check_instance_status|grep-Eio-e"bOPENb"-e"bMOUNTEDb"-e"bSTARTEDb")
if["$status"=="OPEN"];then
info_log"adatabasealreadyopenbytheinstance.n"
elif["$status"=="MOUNTED"];then
info_log"re-opendatabaseinstancen"
open_instance|tee$TMPLOG
opened=$(cat$TMPLOG|grep-Eio"bDatabasealteredb")
if["$opened"!="Databasealtered"];then
info_log"Error:databaseinstanceopenfail!n"
exit2
fi
elif["$status"=="STARTED"];then
info_log"alterdatabasetomountn"
mount_instance|tee$TMPLOG
mounted=$(cat$TMPLOG|grep-Eio"bDatabasealteredb")
if["$mounted"=="Databasealtered"];then
info_log"alterdatabasetoopenn"
open_instance|tee$TMPLOG
opened=$(cat$TMPLOG|grep-Eio"bDatabasealteredb")
if["$opened"=="Databasealtered"];then
info_log"DatabaSEOpened.n"
else
info_log"DatabaSEOpenfailedn"
exit4
fi
else
info_log"Databasemountfailedn"
exit3
fi
else
info_log"Startupdatabaseandopeninstancen"
shutdown_instance&>/dev/null
startup_instance|tee$TMPLOG
started=$(cat$TMPLOG|grep-Eio"bDatabaSEOpenedb")
if["$started"!="DatabaSEOpened"];then
info_log"Databaseinstanceopenfail.n"
info_log"restorecontrolfile2n"
shutdown_instance|tee$TMPLOG
startup_nomount|tee$TMPLOG
restore_controlfile
shutdown_instance|tee$TMPLOG
startup_instance|tee$TMPLOG
started=$(cat$TMPLOG|grep-Eio"bDatabaSEOpenedb")
if["$started"!="DatabaSEOpened"];then
info_log"Databaserestorefail!n"
exit5
else
info_log"DatabaSEOpened.n"
fi
else
info_log"DatabaSEOpened.n"
fi
fi
info_log"Startuplistenern"
runuser-loracle-c"lsnrctlstatus&>/dev/null"
if[$?-eq0];then
info_log"listeneralreadystarted.n"
else
info_log"startinglistener...n"
runuser-loracle-c"lsnrctlstart&>/dev/null"
if[$?-eq0];then
info_log"Thelistenerstartupsuccessfullyn"
else
info_log"Listenerstartfailure!n"
fi
fi
echo
}
backup(){
info_log"DatabaseSwitchoverToBACKUPn"
ismount=$(df-h|grep$sharedisk|grep$sharedisk_mount_point|wc-l)
if[$ismount-ge1];then
disk=$(df-h|grep$sharedisk_mount_point|awk'{print$1}')
if[$disk==$sharedisk];then
status=$(check_instance_status|grep-Eio-e"bOPENb"-e"bMOUNTEDb"-e"bSTARTEDb")
if["$status"=="OPEN"-o"$status"=="MOUNTED"];then
info_log"Databaseinstancestateismountedn"
info_log"Backupcurrentcontrolfile.n"
echo-e"nSQL>alterdatabasebackupcontrolfileto'$controlfile_back';n"
backup_controlfile
info_log"Shutdowndatabaseinstance,pleasewait...n"
shutdown_instance|tee$TMPLOG
shuted=$(cat$TMPLOG|grep-Eio"binstanceshutdownb")
if["$shuted"=="instanceshutdown"];then
info_log"Databaseinstanceshutdownsuccessfully.n"
else
info_log"Databaseinstanceshutdownfailed.n"
info_log"shutdownabort.n"
shutdown_abort
fi
elif["$status"=="STARTED"];then
info_log"DatabaseinstancestateisSTARTEDn"
info_log"Shutdowndatabaseinstance,pleasewait...n"
shutdown_instance|tee$TMPLOG
shuted=$(cat$TMPLOG|grep-Eio"binstanceshutdownb")
if["$shuted"=="instanceshutdown"];then
info_log"Databaseinstanceshutdownsuccessfully.n"
else
info_log"Databaseinstanceshutdownfailed.n"
info_log"shutdownabort.n"
shutdown_abort
fi
else
shutdown_instance|tee$TMPLOG
info_log"Databaseinstancenotavailable.n"
fi
echo
info_log"umountsharediskn"
echo
umount$sharedisk_mount_point&&RETVAL=$?
if[$RETVAL-eq0];then
info_log"umount$sharedisk_mount_pointsuccess.n"
else
info_log"umount$sharedisk_mount_pointfail!n"
fi
else
info_log"$sharediskisnotmounton$sharedisk_mount_pointorbusy.n"
fi
else
info_log"$sharedisk_mount_pointisnomountn"
fi
info_log"stoppinglistener...n"
runuser-loracle-c"lsnrctlstatus"&>/dev/null
RETVAL=$?
if[$RETVAL-eq0];then
runuser-loracle-c"lsnrctlstop"&>/dev/null
RETVAL=$?
if[$RETVAL-eq0];then
info_log"Thelistenerstopsuccessfullyn"
else
info_log"Listenerstopfailure!n"
fi
else
info_log"listenerisnotstarted.n"
fi
echo
}
notify_master(){
echo-e"n-------------------------------------------------------------------------------"
echo"`date'+%b%d%T%a'`$(hostname)[keepalived_notify]:Transitionto$1STATE";
echo"`date'+%b%d%T%a'`$(hostname)[keepalived_notify]:SetuptheVIPon$interface$virtual_ipaddress";
}
notify_backup(){
echo-e"n-------------------------------------------------------------------------------"
echo"`date'+%b%d%T%a'`$HOSTNAME[keepalived_notify]:Transitionto$1STATE";
echo"`date'+%b%d%T%a'`$HOSTNAME[keepalived_notify]:removingtheVIPon$interfacefor$virtual_ipaddress";
}
case$1in
master)
notify_masterMASTER|tee-a$LOGFILE
master|tee-a$LOGFILE
;;
backup)
notify_backupBACKUP|tee-a$LOGFILE
backup|tee-a$LOGFILE
;;
fault)
notify_backupFAULT|tee-a$LOGFILE
backup|tee-a$LOGFILE
;;
stop)
notify_backupSTOP|tee-a$LOGFILE
/etc/init.d/keepalivedstart
#sleep6&&backup|tee-a$LOGFILE
;;
*)
echo"Usage:`basename$0`{master|backup|fault|stop}"
RETVAL=1
;;
esac
exit$RETVAL
手动执行notify切换脚本
l 将主服务器切换为BACKUP状态
执行./keepalived_notify.sh backup
[root@hmdg-db1scripts]#./keepalived_notify.shbackup
-------------------------------------------------------------------------------
Apr1517:23:05Sunhmdg-db1[keepalived_notify]:TransitiontoBACKUPSTATE
Apr1517:23:05Sunhmdg-db1[keepalived_notify]:removingtheVIPoneth1for172.16.10.130,172.16.10.131
Apr1517:23:05Sunhmdg-db1[keepalived_notify]:DatabaseSwitchoverToBACKUP
Apr1517:23:05Sunhmdg-db1[keepalived_notify]:/oradataisnomount
Apr1517:23:05Sunhmdg-db1[keepalived_notify]:stoppinglistener...
Apr1517:23:05Sunhmdg-db1[keepalived_notify]:listenerisnotstarted.
观察执行脚本的输出结果，这是由于在本机上并没有启动oracle实例，所以脚本最终提示本机并没有启动oracle数据库实例
l 将主服务器切换为MASTER状态
执行./keepalived_notify.sh master
[root@hmdg-db1scripts]#./keepalived_notify.shmaster
-------------------------------------------------------------------------------
Apr1517:23:43Sunhmdg-db1[keepalived_notify]:TransitiontoMASTERSTATE
Apr1517:23:43Sunhmdg-db1[keepalived_notify]:SetuptheVIPoneth1172.16.10.130,172.16.10.131
Apr1517:23:43Sunhmdg-db1[keepalived_notify]:DatabaseSwitchoverToMASTER
Apr1517:23:43Sunhmdg-db1[keepalived_notify]:Checkremotenodesharediskmounted.
Apr1517:23:43Sunhmdg-db1[keepalived_notify]:/dev/sdb1checkpassed.
Apr1517:23:43Sunhmdg-db1[keepalived_notify]:mount/dev/sdb1on/oradata
Apr1517:23:43Sunhmdg-db1[keepalived_notify]:restorecontrolfile1
ORACLEinstancestarted.
TotalSystemGlobalArea1603411968bytes
FixedSize2213776bytes
VariableSize1056966768bytes
DatabaseBuffers536870912bytes
RedoBuffers7360512bytes
RecoveryManager:Release11.2.0.1.0-ProductiononSunApr1517:23:442018
Copyright(c)1982,2009,Oracleand/oritsaffiliates.Allrightsreserved.
connectedtotargetdatabase:HMODB(notmounted)
usingtargetdatabasecontrolfileinsteadofrecoverycatalog
RMAN>
Startingrestoreat15-APR-18
allocatedchannel:ORA_DISK_1
channelORA_DISK_1:SID=572devicetype=DISK
channelORA_DISK_1:copiedcontrolfilecopy
outputfilename=/oradata/HMODB/control01.ctl
outputfilename=/u01/app/oracle/flash_recovery_area/HMODB/control02.ctl
outputfilename=/home/oracle/rman/HMODB/control03.ctl
Finishedrestoreat15-APR-18
RMAN>
RecoveryManagercomplete.
Apr1517:23:46Sunhmdg-db1[keepalived_notify]:alterdatabasetomount
Databasealtered.
Apr1517:23:50Sunhmdg-db1[keepalived_notify]:alterdatabasetoopen
Databasealtered.
Apr1517:23:57Sunhmdg-db1[keepalived_notify]:DatabaSEOpened.
Apr1517:23:57Sunhmdg-db1[keepalived_notify]:Startuplistener
Apr1517:23:57Sunhmdg-db1[keepalived_notify]:startinglistener...
Apr1517:23:57Sunhmdg-db1[keepalived_notify]:Thelistenerstartupsuccessfully
观察执行脚本的输出结果
1、脚本接收转换到MASTER信息
2、首先提示绑定的VIP
3、然后检测远程备用节点的磁盘是否挂载，或本机是否挂载了磁盘，检测成功则把磁盘挂载
4、启动数据库到nomount状态，并恢复控制文件
5、启动数据库实例到open状态
6、启动数据库监听，切换完成
注意：如果使用系统命令手动切换，应该要保证正在运行oracle服务的主节点先正常关闭，并且卸载共享存储之后，再启动备用节点的oracle服务(即不能同时挂载共享存储)
l 再次将主服务器切换到BACKUP状态
执行./keepalived_notify.sh backup
[root@hmdg-db1scripts]#./keepalived_notify.shbackup
-------------------------------------------------------------------------------
Apr1517:34:54Sunhmdg-db1[keepalived_notify]:TransitiontoBACKUPSTATE
Apr1517:34:54Sunhmdg-db1[keepalived_notify]:removingtheVIPoneth1for172.16.10.130,172.16.10.131
Apr1517:34:54Sunhmdg-db1[keepalived_notify]:DatabaseSwitchoverToBACKUP
Apr1517:34:55Sunhmdg-db1[keepalived_notify]:Databaseinstancestateismounted
Apr1517:34:55Sunhmdg-db1[keepalived_notify]:Backupcurrentcontrolfile.
SQL>alterdatabasebackupcontrolfileto'/backup/oracle/control/control_20181504173454';
Databasealtered.
Apr1517:34:55Sunhmdg-db1[keepalived_notify]:Shutdowndatabaseinstance,pleasewait...
Databaseclosed.
Databasedismounted.
ORACLEinstanceshutdown.
Apr1517:35:04Sunhmdg-db1[keepalived_notify]:Databaseinstanceshutdownsuccessfully.
Apr1517:35:04Sunhmdg-db1[keepalived_notify]:umountsharedisk
Apr1517:35:05Sunhmdg-db1[keepalived_notify]:umount/oradatasuccess.
Apr1517:35:05Sunhmdg-db1[keepalived_notify]:stoppinglistener...
Apr1517:35:08Sunhmdg-db1[keepalived_notify]:Thelistenerstopsuccessfully
观察执行脚本的输出结果
1、 脚本接收转换到BACKUP信息
2、 首先移除VIP(由keepalived绑定到主机上浮动IP地址，同时也是对外提供服务的IP地址)，实际上是由keepalived检测到异常或本身的服务器不可用时自动剔除VIP
3、 备份控制文件到/backup/oracle/control/目录下
4、 正常关闭数据库实例
5、 卸载共享存储
6、 关闭数据库监听程序
l notify脚本切换演示
如果备机主动切换到MASTER状态时，脚本首先或检测对方磁盘是否umount，如果没有则最多等待20秒，提示异常并正常推出切换到MASTER的请求
[root@hmdg-db1scripts]#./keepalived_notify.shmaster
-------------------------------------------------------------------------------
Apr1517:40:08Sunhmdg-db1[keepalived_notify]:TransitiontoMASTERSTATE
Apr1517:40:08Sunhmdg-db1[keepalived_notify]:SetuptheVIPoneth1172.16.10.130,172.16.10.131
Apr1517:40:08Sunhmdg-db1[keepalived_notify]:DatabaseSwitchoverToMASTER
Apr1517:40:08Sunhmdg-db1[keepalived_notify]:Checkremotenodesharediskmounted.
Apr1517:40:08Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[1]...
Apr1517:40:09Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[2]...
Apr1517:40:10Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[3]...
Apr1517:40:12Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[4]...
Apr1517:40:13Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[5]...
Apr1517:40:14Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[6]...
Apr1517:40:15Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[7]...
Apr1517:40:16Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[8]...
Apr1517:40:17Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[9]...
Apr1517:40:18Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[10]...
Apr1517:40:20Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[11]...
Apr1517:40:21Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[12]...
Apr1517:40:22Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[13]...
Apr1517:40:23Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[14]...
Apr1517:40:24Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[15]...
Apr1517:40:25Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[16]...
Apr1517:40:26Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[17]...
Apr1517:40:27Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[18]...
Apr1517:40:29Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[19]...
Apr1517:40:30Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[20]...
Apr1517:40:30Sunhmdg-db1[keepalived_notify]:Diskstatusabnormal.
这时如果需要手动切换，则需要先将远程备用节点的服务关闭并umount存储之后，才能正常切换
[root@hmdg-db1scripts]#./keepalived_notify.shmaster
-------------------------------------------------------------------------------
Apr1517:45:29Sunhmdg-db1[keepalived_notify]:TransitiontoMASTERSTATE
Apr1517:45:29Sunhmdg-db1[keepalived_notify]:SetuptheVIPoneth1172.16.10.130,172.16.10.131
Apr1517:45:29Sunhmdg-db1[keepalived_notify]:DatabaseSwitchoverToMASTER
Apr1517:45:29Sunhmdg-db1[keepalived_notify]:Checkremotenodesharediskmounted.
Apr1517:45:29Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[1]...
Apr1517:45:30Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[2]...
Apr1517:45:31Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[3]...
Apr1517:45:32Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[4]...
Apr1517:45:33Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[5]...
Apr1517:45:35Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[6]...
Apr1517:45:36Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[7]...
Apr1517:45:37Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[8]...
Apr1517:45:38Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[9]...
Apr1517:45:39Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[10]...
Apr1517:45:40Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[11]...
Apr1517:45:41Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[12]...
Apr1517:45:42Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[13]...
Apr1517:45:44Sunhmdg-db1[keepalived_notify]:/dev/sdb1checkpassed.
Apr1517:45:44Sunhmdg-db1[keepalived_notify]:mount/dev/sdb1on/oradata
Apr1517:45:44Sunhmdg-db1[keepalived_notify]:restorecontrolfile1
ORACLEinstancestarted.
TotalSystemGlobalArea1603411968bytes
FixedSize2213776bytes
VariableSize1056966768bytes
DatabaseBuffers536870912bytes
RedoBuffers7360512bytes
RecoveryManager:Release11.2.0.1.0-ProductiononSunApr1517:45:452018
Copyright(c)1982,Oracleand/oritsaffiliates.Allrightsreserved.
connectedtotargetdatabase:HMODB(notmounted)
usingtargetdatabasecontrolfileinsteadofrecoverycatalog
RMAN>
Startingrestoreat15-APR-18
allocatedchannel:ORA_DISK_1
channelORA_DISK_1:SID=572devicetype=DISK
channelORA_DISK_1:copiedcontrolfilecopy
outputfilename=/oradata/HMODB/control01.ctl
outputfilename=/u01/app/oracle/flash_recovery_area/HMODB/control02.ctl
outputfilename=/home/oracle/rman/HMODB/control03.ctl
Finishedrestoreat15-APR-18
RMAN>
RecoveryManagercomplete.
Apr1517:45:46Sunhmdg-db1[keepalived_notify]:alterdatabasetomount
Databasealtered.
Apr1517:45:51Sunhmdg-db1[keepalived_notify]:alterdatabasetoopen
Databasealtered.
Apr1517:45:58Sunhmdg-db1[keepalived_notify]:DatabaSEOpened.
Apr1517:45:58Sunhmdg-db1[keepalived_notify]:Startuplistener
Apr1517:45:58Sunhmdg-db1[keepalived_notify]:startinglistener...
Apr1517:45:58Sunhmdg-db1[keepalived_notify]:Thelistenerstartupsuccessfully
启动keepalived服务接管Oracle服务
启动keepalived服务时，keepalived首先进行资源检测，并根据权值设定MASTER角色
1、启动主服务器DB1的keepalived
##启动keepalived
[root@hmdg-db1scripts]#/etc/init.d/keepalivedstart
##设置开机启动
[root@hmdg-db1scripts]#chkconfigkeepalivedon
2、查看主服务器的keepalived系统日志
[root@hmdg-db1scripts]#tail-f/var/log/messages
Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:RegisteringKernelnetlinkreflector
Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:RegisteringKernelnetlinkcommandchannel
Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:RegisteringgratuitousARPsharedchannel
Apr1517:51:25hmdg-db1Keepalived_healthcheckers[82071]:UsingLinkWatchkernelnetlinkreflector...
Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:Openingfile'/usr/local/keepalived/etc/keepalived/keepalived.conf'.
Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:VRRP_Instance(Oracle_HA_1)removingprotocolVIPs.
Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:SECURITYVIOLATION-scriptsarebeingexecutedbutscript_securitynotenabled.
Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:SyncgroupORACLE_HA_GROUPhasonly1virtualrouter(s)-removing
Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:UsingLinkWatchkernelnetlinkreflector...
Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:VRRP_Instance(Oracle_HA_1)EnteringBACKUPSTATE
Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:VRRPsockpool:[ifindex(3),proto(112),unicast(0),fd(10,11)]
Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:VRRP_Script(change_monitor_status)succeeded
Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:VRRP_Script(monitor_oracle_status)succeeded
Apr1517:51:27hmdg-db1Keepalived_vrrp[82072]:VRRP_Instance(Oracle_HA_1)Changingeffectivepriorityfrom100to120
Apr1517:51:31hmdg-db1Keepalived_vrrp[82072]:VRRP_Instance(Oracle_HA_1)TransitiontoMASTERSTATE
Apr1517:51:33hmdg-db1Keepalived_vrrp[82072]:VRRP_Instance(Oracle_HA_1)EnteringMASTERSTATE
Apr1517:51:33hmdg-db1Keepalived_vrrp[82072]:VRRP_Instance(Oracle_HA_1)settingprotocolVIPs.
Apr1517:51:33hmdg-db1Keepalived_vrrp[82072]:SendinggratuitousARPoneth1for172.16.10.130
Apr1517:51:33hmdg-db1Keepalived_healthcheckers[82071]:NetlinkreflectorreportsIP172.16.10.130added
Apr1517:51:33hmdg-db1Keepalived_healthcheckers[82071]:NetlinkreflectorreportsIP172.16.10.131added
Apr1517:51:33hmdg-db1Keepalived_vrrp[82072]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.130
Apr1517:51:33hmdg-db1Keepalived_vrrp[82072]:SendinggratuitousARPoneth1for172.16.10.131
Apr1517:51:33hmdg-db1Keepalived_vrrp[82072]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.131
Apr1517:51:33hmdg-db1Keepalived_vrrp[82072]:SendinggratuitousARPoneth1for172.16.10.130
Apr1517:51:33hmdg-db1Keepalived_vrrp[82072]:SendinggratuitousARPoneth1for172.16.10.131
Apr1517:51:33hmdg-db1Keepalived_vrrp[82072]:SendinggratuitousARPoneth1for172.16.10.130
Apr1517:51:33hmdg-db1Keepalived_vrrp[82072]:SendinggratuitousARPoneth1for172.16.10.131
3、启动备服务器DB2的keepalived
##启动keepalived
[root@hmdg-db2scripts]#/etc/init.d/keepalivedstart
##设置开机启动
[root@hmdg-db2scripts]#chkconfigkeepalivedon
4、查看备服务器的keepalived系统日志
Apr1517:51:41hmdg-db2Keepalived[71709]:StartingKeepalivedv1.3.2(04/14,2018)
Apr1517:51:41hmdg-db2Keepalived[71709]:WARNING-defaultuser'keepalived_script'forscriptexecutiondoesnotexist-pleasecreate.
Apr1517:51:41hmdg-db2Keepalived[71709]:Openingfile'/usr/local/keepalived/etc/keepalived/keepalived.conf'.
Apr1517:51:41hmdg-db2Keepalived[71710]:StartingHealthcheckchildprocess,pid=71711
Apr1517:51:41hmdg-db2Keepalived[71710]:StartingVRRPchildprocess,pid=71712
Apr1517:51:41hmdg-db2Keepalived_healthcheckers[71711]:NetlinkreflectorreportsIP192.168.6.26added
Apr1517:51:41hmdg-db2Keepalived_healthcheckers[71711]:NetlinkreflectorreportsIP172.16.10.26added
Apr1517:51:41hmdg-db2Keepalived_healthcheckers[71711]:NetlinkreflectorreportsIPfe80::20c:29ff:fedd:d175added
Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:NetlinkreflectorreportsIP192.168.6.26added
Apr1517:51:41hmdg-db2Keepalived_healthcheckers[71711]:NetlinkreflectorreportsIPfe80::20c:29ff:fedd:d17fadded
Apr1517:51:41hmdg-db2Keepalived_healthcheckers[71711]:NetlinkreflectorreportsIPfe80::20c:29ff:fedd:d189added
Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:NetlinkreflectorreportsIP172.16.10.26added
Apr1517:51:41hmdg-db2Keepalived_healthcheckers[71711]:NetlinkreflectorreportsIPfe80::20c:29ff:fedd:d193added
Apr1517:51:41hmdg-db2Keepalived_healthcheckers[71711]:RegisteringKernelnetlinkreflector
Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:NetlinkreflectorreportsIPfe80::20c:29ff:fedd:d175added
Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:NetlinkreflectorreportsIPfe80::20c:29ff:fedd:d17fadded
Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:NetlinkreflectorreportsIPfe80::20c:29ff:fedd:d189added
Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:NetlinkreflectorreportsIPfe80::20c:29ff:fedd:d193added
Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:RegisteringKernelnetlinkreflector
Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:RegisteringKernelnetlinkcommandchannel
Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:RegisteringgratuitousARPsharedchannel
Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:Openingfile'/usr/local/keepalived/etc/keepalived/keepalived.conf'.
Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:VRRP_Instance(Oracle_HA_1)removingprotocolVIPs.
Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:SECURITYVIOLATION-scriptsarebeingexecutedbutscript_securitynotenabled.
Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:SyncgroupORACLE_HA_GROUPhasonly1virtualrouter(s)-removing
Apr1517:51:41hmdg-db2Keepalived_healthcheckers[71711]:RegisteringKernelnetlinkcommandchannel
Apr1517:51:41hmdg-db2Keepalived_healthcheckers[71711]:Openingfile'/usr/local/keepalived/etc/keepalived/keepalived.conf'.
Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:UsingLinkWatchkernelnetlinkreflector...
Apr1517:51:41hmdg-db2Keepalived_healthcheckers[71711]:UsingLinkWatchkernelnetlinkreflector...
Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:VRRP_Instance(Oracle_HA_1)EnteringBACKUPSTATE
Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:VRRPsockpool:[ifindex(3),11)]
Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:pid71721exitedwithstatus1
Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:VRRP_Script(change_monitor_status)succeeded
Apr1517:52:01hmdg-db2Keepalived_vrrp[71712]:VRRP_Script(monitor_oracle_status)succeeded
Apr1517:52:01hmdg-db2Keepalived_vrrp[71712]:VRRP_Instance(Oracle_HA_1)Changingeffectivepriorityfrom90to110
5、主备Oracle、VIP、磁盘服务状态检查
##DB1
[root@hmdg-db1scripts]#ps-ef|grepora_
oracle824391017:51?00:00:00ora_pmon_HMODB
oracle824411017:51?00:00:00ora_vktm_HMODB
oracle824451017:51?00:00:00ora_gen0_HMODB
oracle824471017:51?00:00:00ora_diag_HMODB
oracle824491017:51?00:00:00ora_dbrm_HMODB
oracle824511017:51?00:00:00ora_psp0_HMODB
oracle824531017:51?00:00:00ora_dia0_HMODB
oracle824551017:51?00:00:00ora_mman_HMODB
oracle824571017:51?00:00:00ora_dbw0_HMODB
oracle824591017:51?00:00:00ora_lgwr_HMODB
oracle824611017:51?00:00:00ora_ckpt_HMODB
oracle824631017:51?00:00:00ora_smon_HMODB
oracle824651017:51?00:00:00ora_reco_HMODB
oracle824671017:51?00:00:00ora_mmon_HMODB
oracle824691017:51?00:00:00ora_mmnl_HMODB
oracle824711017:51?00:00:00ora_d000_HMODB
oracle824731017:51?00:00:00ora_s000_HMODB
oracle826381017:51?00:00:00ora_arc0_HMODB
oracle826531017:51?00:00:00ora_arc1_HMODB
oracle826551017:51?00:00:00ora_arc2_HMODB
oracle826571017:51?00:00:00ora_arc3_HMODB
oracle827551017:51?00:00:00ora_qmnc_HMODB
oracle828071017:51?00:00:00ora_cjq0_HMODB
oracle828351017:51?00:00:00ora_vkrm_HMODB
oracle828631017:51?00:00:00ora_q000_HMODB
oracle828651017:51?00:00:00ora_q001_HMODB
oracle840991117:54?00:00:00ora_j000_HMODB
oracle841011017:54?00:00:00ora_j001_HMODB
root842683823017:54pts/200:00:00grepora_
[root@hmdg-db1scripts]#df-Th
FilesystemTypeSizeUsedAvailUse%Mountedon
/dev/sda3ext450G19G29G40%/
tmpfstmpfs3.9G909M3.0G24%/dev/shm
/dev/sda1ext4190M40M141M22%/boot
/dev/sdb1ext430G6.0G22G22%/oradata
[root@hmdg-db1scripts]#ipaddrshowdeveth1
3:eth1:<BROADCAST,MULTICAST,UP,LOWER_UP>mtu1500qdiscpfifo_faststateUPqlen1000
link/ether00:0c:29:59:d5:91brdff:ff:ff:ff:ff:ff
inet172.16.10.25/24brd172.16.10.255scopeglobaleth1
inet172.16.10.130/24scopeglobalsecondaryeth1
inet172.16.10.131/24scopeglobalsecondaryeth1
inet6fe80::20c:29ff:fe59:d591/64scopelink
valid_lftforeverpreferred_lftforever
##DB2
[root@hmdg-db2scripts]#ps-ef|grepora_
root7325131390017:55pts/200:00:00grepora_
[root@hmdg-db2scripts]#df-Th
FilesystemTypeSizeUsedAvailUse%Mountedon
/dev/sda3ext450G18G30G37%/
tmpfstmpfs3.9G03.9G0%/dev/shm
/dev/sda1ext4190M40M141M22%/boot
[root@hmdg-db2scripts]#ipaddrshowdeveth1
3:eth1:<BROADCAST,LOWER_UP>mtu1500qdiscpfifo_faststateUPqlen1000
link/ether00:0c:29:dd:d1:7fbrdff:ff:ff:ff:ff:ff
inet172.16.10.26/24brd172.16.10.255scopeglobaleth1
inet6fe80::20c:29ff:fedd:d17f/64scopelink
valid_lftforeverpreferred_lftforever
测试Keepalived故障切换
停止或重启主服务器的keepalibed服务
##系统日志
Apr1518:08:02hmdg-db1Keepalived[89311]:Stopping
Apr1518:08:02hmdg-db1Keepalived_healthcheckers[89313]:Stopped
Apr1518:08:02hmdg-db1Keepalived_vrrp[89314]:VRRP_Instance(Oracle_HA_1)sent0priority
Apr1518:08:02hmdg-db1Keepalived_vrrp[89314]:VRRP_Instance(Oracle_HA_1)removingprotocolVIPs.
Apr1518:08:03hmdg-db1Keepalived_vrrp[89314]:Stopped
Apr1518:08:03hmdg-db1Keepalived[89311]:StoppedKeepalivedv1.3.2(04/14,2018)
##nitify脚本的输出日志
-------------------------------------------------------------------------------
Apr1518:08:03Sunhmdg-db1[keepalived_notify]:TransitiontoSTOPSTATE
Apr1518:08:03Sunhmdg-db1[keepalived_notify]:removingtheVIPoneth1for172.16.10.130,172.16.10.131
Apr1518:08:09Sunhmdg-db1[keepalived_notify]:DatabaseSwitchoverToBACKUP
Apr1518:08:09Sunhmdg-db1[keepalived_notify]:Databaseinstancestateismounted
Apr1518:08:09Sunhmdg-db1[keepalived_notify]:Backupcurrentcontrolfile.
SQL>alterdatabasebackupcontrolfileto'/backup/oracle/control/control_20181504180803';
Databasealtered.
Apr1518:08:09Sunhmdg-db1[keepalived_notify]:Shutdowndatabaseinstance,pleasewait...
Databaseclosed.
Databasedismounted.
ORACLEinstanceshutdown.
Apr1518:08:18Sunhmdg-db1[keepalived_notify]:Databaseinstanceshutdownsuccessfully.
Apr1518:08:18Sunhmdg-db1[keepalived_notify]:umountsharedisk
Apr1518:08:18Sunhmdg-db1[keepalived_notify]:umount/oradatasuccess.
Apr1518:08:18Sunhmdg-db1[keepalived_notify]:stoppinglistener...
Apr1518:08:19Sunhmdg-db1[keepalived_notify]:Thelistenerstopsuccessfully
##服务资源检查
[root@hmdg-db1scripts]#ps-ef|grepora_
root907582451018:09pts/000:00:00grepora_
[root@hmdg-db1scripts]#df-Th
FilesystemTypeSizeUsedAvailUse%Mountedon
/dev/sda3ext450G19G29G40%/
tmpfstmpfs3.9G03.9G0%/dev/shm
/dev/sda1ext4190M40M141M22%/boot
[root@hmdg-db1scripts]#ipaddrshowdeveth1
3:eth1:<BROADCAST,LOWER_UP>mtu1500qdiscpfifo_faststateUPqlen1000
link/ether00:0c:29:59:d5:91brdff:ff:ff:ff:ff:ff
inet172.16.10.25/24brd172.16.10.255scopeglobaleth1
inet6fe80::20c:29ff:fe59:d591/64scopelink
valid_lftforeverpreferred_lftforever
由于主服务的keepalived异常，keepalived实例组将会重新分配MASTER角色，并将资源转移到BACKUP上(即重新选举后新的MASTER)
##系统日志
Apr1518:08:03hmdg-db2Keepalived_vrrp[77811]:VRRP_Instance(Oracle_HA_1)TransitiontoMASTERSTATE
Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:VRRP_Instance(Oracle_HA_1)EnteringMASTERSTATE
Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:VRRP_Instance(Oracle_HA_1)settingprotocolVIPs.
Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.130
Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.131
Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:08:05hmdg-db2Keepalived_healthcheckers[77810]:NetlinkreflectorreportsIP172.16.10.130added
Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:08:05hmdg-db2Keepalived_healthcheckers[77810]:NetlinkreflectorreportsIP172.16.10.131added
Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.130
Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.131
Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.131
##notify脚本日志
-------------------------------------------------------------------------------
Apr1518:08:05Sunhmdg-db2[keepalived_notify]:TransitiontoMASTERSTATE
Apr1518:08:05Sunhmdg-db2[keepalived_notify]:SetuptheVIPoneth1172.16.10.130,172.16.10.131
Apr1518:08:05Sunhmdg-db2[keepalived_notify]:DatabaseSwitchoverToMASTER
Apr1518:08:05Sunhmdg-db2[keepalived_notify]:Checkremotenodesharediskmounted.
Apr1518:08:05Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[1]...
Apr1518:08:06Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[2]...
Apr1518:08:08Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[3]...
Apr1518:08:09Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[4]...
Apr1518:08:10Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[5]...
Apr1518:08:11Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[6]...
Apr1518:08:12Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[7]...
Apr1518:08:13Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[8]...
Apr1518:08:14Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[9]...
Apr1518:08:15Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[10]...
Apr1518:08:17Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[11]...
Apr1518:08:18Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[12]...
Apr1518:08:19Sunhmdg-db2[keepalived_notify]:/dev/sdb1checkpassed.
Apr1518:08:19Sunhmdg-db2[keepalived_notify]:mount/dev/sdb1on/oradata
Apr1518:08:19Sunhmdg-db2[keepalived_notify]:restorecontrolfile1
ORACLEinstancestarted.
TotalSystemGlobalArea1603411968bytes
FixedSize2213776bytes
VariableSize1056966768bytes
DatabaseBuffers536870912bytes
RedoBuffers7360512bytes
RecoveryManager:Release11.2.0.1.0-ProductiononSunApr1518:08:202018
Copyright(c)1982,Oracleand/oritsaffiliates.Allrightsreserved.
connectedtotargetdatabase:HMODB(notmounted)
usingtargetdatabasecontrolfileinsteadofrecoverycatalog
RMAN>
Startingrestoreat15-APR-18
allocatedchannel:ORA_DISK_1
channelORA_DISK_1:SID=572devicetype=DISK
channelORA_DISK_1:copiedcontrolfilecopy
outputfilename=/oradata/HMODB/control01.ctl
outputfilename=/u01/app/oracle/flash_recovery_area/HMODB/control02.ctl
outputfilename=/home/oracle/rman/HMODB/control03.ctl
Finishedrestoreat15-APR-18
RMAN>
RecoveryManagercomplete.
Apr1518:08:22Sunhmdg-db2[keepalived_notify]:alterdatabasetomount
Databasealtered.
Apr1518:08:26Sunhmdg-db2[keepalived_notify]:alterdatabasetoopen
Databasealtered.
Apr1518:08:33Sunhmdg-db2[keepalived_notify]:DatabaSEOpened.
Apr1518:08:33Sunhmdg-db2[keepalived_notify]:Startuplistener
Apr1518:08:33Sunhmdg-db2[keepalived_notify]:startinglistener...
Apr1518:08:33Sunhmdg-db2[keepalived_notify]:Thelistenerstartupsuccessfully
##服务检查
[root@hmdg-db2scripts]#ps-ef|grepora_
oracle786321018:08?00:00:00ora_pmon_HMODB
oracle786341018:08?00:00:00ora_vktm_HMODB
oracle786381018:08?00:00:00ora_gen0_HMODB
oracle786401018:08?00:00:00ora_diag_HMODB
oracle786421018:08?00:00:00ora_dbrm_HMODB
oracle786441018:08?00:00:00ora_psp0_HMODB
oracle786461018:08?00:00:00ora_dia0_HMODB
oracle786481018:08?00:00:00ora_mman_HMODB
oracle786501018:08?00:00:00ora_dbw0_HMODB
oracle786521018:08?00:00:00ora_lgwr_HMODB
oracle786541018:08?00:00:00ora_ckpt_HMODB
oracle786561018:08?00:00:00ora_smon_HMODB
oracle786581018:08?00:00:00ora_reco_HMODB
oracle786601018:08?00:00:00ora_mmon_HMODB
oracle786621018:08?00:00:00ora_mmnl_HMODB
oracle786641018:08?00:00:00ora_d000_HMODB
oracle786661018:08?00:00:00ora_s000_HMODB
oracle788441018:08?00:00:00ora_arc0_HMODB
oracle788471018:08?00:00:00ora_arc1_HMODB
oracle788491018:08?00:00:00ora_arc2_HMODB
oracle788511018:08?00:00:00ora_arc3_HMODB
oracle788791018:08?00:00:00ora_qmnc_HMODB
oracle789681018:08?00:00:00ora_cjq0_HMODB
oracle790311018:08?00:00:00ora_vkrm_HMODB
oracle790591018:08?00:00:00ora_q000_HMODB
oracle790611018:08?00:00:00ora_q001_HMODB
oracle816521018:13?00:00:00ora_smco_HMODB
oracle816671018:13?00:00:00ora_w000_HMODB
root8180831390018:13pts/200:00:00grepora_
[root@hmdg-db2scripts]#df-Th
FilesystemTypeSizeUsedAvailUse%Mountedon
/dev/sda3ext450G18G30G37%/
tmpfstmpfs3.9G909M3.0G24%/dev/shm
/dev/sda1ext4190M40M141M22%/boot
/dev/sdb1ext430G6.0G22G22%/oradata
[root@hmdg-db2scripts]#ipaddrshowdeveth1
3:eth1:<BROADCAST,LOWER_UP>mtu1500qdiscpfifo_faststateUPqlen1000
link/ether00:0c:29:dd:d1:7fbrdff:ff:ff:ff:ff:ff
inet172.16.10.26/24brd172.16.10.255scopeglobaleth1
inet172.16.10.130/24scopeglobalsecondaryeth1
inet172.16.10.131/24scopeglobalsecondaryeth1
inet6fe80::20c:29ff:fedd:d17f/64scopelink
valid_lftforeverpreferred_lftforever
Apr1518:15:14hmdg-db1Keepalived_vrrp[90792]:Openingfile'/usr/local/keepalived/etc/keepalived/keepalived.conf'.
Apr1518:15:14hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)removingprotocolVIPs.
Apr1518:15:14hmdg-db1Keepalived_vrrp[90792]:SECURITYVIOLATION-scriptsarebeingexecutedbutscript_securitynotenabled.
Apr1518:15:14hmdg-db1Keepalived_vrrp[90792]:SyncgroupORACLE_HA_GROUPhasonly1virtualrouter(s)-removing
Apr1518:15:14hmdg-db1Keepalived_vrrp[90792]:UsingLinkWatchkernelnetlinkreflector...
Apr1518:15:14hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)EnteringBACKUPSTATE
Apr1518:15:14hmdg-db1Keepalived_vrrp[90792]:VRRPsockpool:[ifindex(3),11)]
Apr1518:15:14hmdg-db1Keepalived_vrrp[90792]:pid90794exitedwithstatus1
Apr1518:15:14hmdg-db1Keepalived_vrrp[90792]:VRRP_Script(change_monitor_status)succeeded
Apr1518:15:34hmdg-db1Keepalived_vrrp[90792]:VRRP_Script(monitor_oracle_status)succeeded
Apr1518:15:34hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)Changingeffectivepriorityfrom100to120
##系统日志
Apr1518:16:24hmdg-db2Keepalived_vrrp[83304]:VRRP_Instance(Oracle_HA_1)EnteringBACKUPSTATE
Apr1518:16:24hmdg-db2Keepalived_vrrp[83304]:VRRPsockpool:[ifindex(3),11)]
Apr1518:16:24hmdg-db2Keepalived_vrrp[83304]:VRRP_Script(change_monitor_status)succeeded
Apr1518:16:25hmdg-db2Keepalived_vrrp[83304]:VRRP_Script(monitor_oracle_status)succeeded
Apr1518:16:26hmdg-db2Keepalived_vrrp[83304]:VRRP_Instance(Oracle_HA_1)Changingeffectivepriorityfrom90to110
##notify日志
-------------------------------------------------------------------------------
Apr1518:16:24Sunhmdg-db2[keepalived_notify]:TransitiontoSTOPSTATE
Apr1518:16:24Sunhmdg-db2[keepalived_notify]:removingtheVIPoneth1for172.16.10.130,172.16.10.131
-------------------------------------------------------------------------------
Apr1518:16:24Sunhmdg-db2[keepalived_notify]:TransitiontoBACKUPSTATE
Apr1518:16:24Sunhmdg-db2[keepalived_notify]:removingtheVIPoneth1for172.16.10.130,172.16.10.131
Apr1518:16:24Sunhmdg-db2[keepalived_notify]:DatabaseSwitchoverToBACKUP
Apr1518:16:24Sunhmdg-db2[keepalived_notify]:Databaseinstancestateismounted
Apr1518:16:25Sunhmdg-db2[keepalived_notify]:Backupcurrentcontrolfile.
SQL>alterdatabasebackupcontrolfileto'/backup/oracle/control/control_20181504181624';
Databasealtered.
Apr1518:16:25Sunhmdg-db2[keepalived_notify]:Shutdowndatabaseinstance,pleasewait...
Databaseclosed.
Databasedismounted.
ORACLEinstanceshutdown.
Apr1518:16:34Sunhmdg-db2[keepalived_notify]:Databaseinstanceshutdownsuccessfully.
Apr1518:16:34Sunhmdg-db2[keepalived_notify]:umountsharedisk
Apr1518:16:34Sunhmdg-db2[keepalived_notify]:umount/oradatasuccess.
Apr1518:16:34Sunhmdg-db2[keepalived_notify]:stoppinglistener...
Apr1518:16:38Sunhmdg-db2[keepalived_notify]:Thelistenerstopsuccessfully
##服务状态
[root@hmdg-db2scripts]#ps-ef|grepora_
root8461931390018:18pts/200:00:00grepora_
[root@hmdg-db2scripts]#df-Th
FilesystemTypeSizeUsedAvailUse%Mountedon
/dev/sda3ext450G18G30G37%/
tmpfstmpfs3.9G03.9G0%/dev/shm
/dev/sda1ext4190M40M141M22%/boot
[root@hmdg-db2scripts]#ipaddrshowdeveth1
3:eth1:<BROADCAST,LOWER_UP>mtu1500qdiscpfifo_faststateUPqlen1000
link/ether00:0c:29:dd:d1:7fbrdff:ff:ff:ff:ff:ff
inet172.16.10.26/24brd172.16.10.255scopeglobaleth1
inet6fe80::20c:29ff:fedd:d17f/64scopelink
valid_lftforeverpreferred_lftforever
##DB1的系统日志
Apr1518:15:14hmdg-db1Keepalived_vrrp[90792]:VRRP_Script(change_monitor_status)succeeded
Apr1518:15:34hmdg-db1Keepalived_vrrp[90792]:VRRP_Script(monitor_oracle_status)succeeded
Apr1518:15:34hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)Changingeffectivepriorityfrom100to120
Apr1518:16:23hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)TransitiontoMASTERSTATE
Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)EnteringMASTERSTATE
Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)settingprotocolVIPs.
Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.130
Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:16:25hmdg-db1Keepalived_healthcheckers[90790]:NetlinkreflectorreportsIP172.16.10.130added
Apr1518:16:25hmdg-db1Keepalived_healthcheckers[90790]:NetlinkreflectorreportsIP172.16.10.131added
Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.131
Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:16:30hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:16:30hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.130
Apr1518:16:30hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:16:30hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.131
Apr1518:16:30hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:16:30hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.131
##DB1的notify脚本日志
-------------------------------------------------------------------------------
Apr1518:16:25Sunhmdg-db1[keepalived_notify]:TransitiontoMASTERSTATE
Apr1518:16:25Sunhmdg-db1[keepalived_notify]:SetuptheVIPoneth1172.16.10.130,172.16.10.131
Apr1518:16:25Sunhmdg-db1[keepalived_notify]:DatabaseSwitchoverToMASTER
Apr1518:16:25Sunhmdg-db1[keepalived_notify]:Checkremotenodesharediskmounted.
Apr1518:16:26Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[1]...
Apr1518:16:27Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[2]...
Apr1518:16:28Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[3]...
Apr1518:16:29Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[4]...
Apr1518:16:30Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[5]...
Apr1518:16:31Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[6]...
Apr1518:16:32Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[7]...
Apr1518:16:34Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[8]...
Apr1518:16:35Sunhmdg-db1[keepalived_notify]:/dev/sdb1checkpassed.
Apr1518:16:35Sunhmdg-db1[keepalived_notify]:mount/dev/sdb1on/oradata
Apr1518:16:35Sunhmdg-db1[keepalived_notify]:restorecontrolfile1
ORACLEinstancestarted.
TotalSystemGlobalArea1603411968bytes
FixedSize2213776bytes
VariableSize1056966768bytes
DatabaseBuffers536870912bytes
RedoBuffers7360512bytes
RecoveryManager:Release11.2.0.1.0-ProductiononSunApr1518:16:382018
Copyright(c)1982,Oracleand/oritsaffiliates.Allrightsreserved.
connectedtotargetdatabase:HMODB(notmounted)
usingtargetdatabasecontrolfileinsteadofrecoverycatalog
RMAN>
Startingrestoreat15-APR-18
allocatedchannel:ORA_DISK_1
channelORA_DISK_1:SID=572devicetype=DISK
channelORA_DISK_1:copiedcontrolfilecopy
outputfilename=/oradata/HMODB/control01.ctl
outputfilename=/u01/app/oracle/flash_recovery_area/HMODB/control02.ctl
outputfilename=/home/oracle/rman/HMODB/control03.ctl
Finishedrestoreat15-APR-18
RMAN>
RecoveryManagercomplete.
Apr1518:16:40Sunhmdg-db1[keepalived_notify]:alterdatabasetomount
Databasealtered.
Apr1518:16:44Sunhmdg-db1[keepalived_notify]:alterdatabasetoopen
Databasealtered.
Apr1518:16:50Sunhmdg-db1[keepalived_notify]:DatabaSEOpened.
Apr1518:16:50Sunhmdg-db1[keepalived_notify]:Startuplistener
Apr1518:16:50Sunhmdg-db1[keepalived_notify]:startinglistener...
Apr1518:16:50Sunhmdg-db1[keepalived_notify]:Thelistenerstartupsuccessfully
##服务状态
[root@hmdg-db1scripts]#ps-ef|grepora_
oracle917651018:16?00:00:00ora_pmon_HMODB
oracle917671018:16?00:00:00ora_vktm_HMODB
oracle917711018:16?00:00:00ora_gen0_HMODB
oracle917731018:16?00:00:00ora_diag_HMODB
oracle917751018:16?00:00:00ora_dbrm_HMODB
oracle917771018:16?00:00:00ora_psp0_HMODB
oracle917791018:16?00:00:00ora_dia0_HMODB
oracle917811018:16?00:00:00ora_mman_HMODB
oracle917831018:16?00:00:00ora_dbw0_HMODB
oracle917851018:16?00:00:00ora_lgwr_HMODB
oracle917871018:16?00:00:00ora_ckpt_HMODB
oracle917891018:16?00:00:00ora_smon_HMODB
oracle917911018:16?00:00:00ora_reco_HMODB
oracle917931018:16?00:00:00ora_mmon_HMODB
oracle917951018:16?00:00:00ora_mmnl_HMODB
oracle917971018:16?00:00:00ora_d000_HMODB
oracle917991018:16?00:00:00ora_s000_HMODB
oracle919651018:16?00:00:00ora_arc0_HMODB
oracle919671018:16?00:00:00ora_arc1_HMODB
oracle919691018:16?00:00:00ora_arc2_HMODB
oracle919711018:16?00:00:00ora_arc3_HMODB
oracle919991018:16?00:00:00ora_qmnc_HMODB
oracle920901018:16?00:00:00ora_cjq0_HMODB
oracle921051018:16?00:00:00ora_vkrm_HMODB
oracle921811018:16?00:00:00ora_q000_HMODB
oracle921831018:16?00:00:00ora_q001_HMODB
oracle938171018:20?00:00:00ora_j000_HMODB
oracle938191018:20?00:00:00ora_j001_HMODB
root940302451018:20pts/000:00:00grepora_
[root@hmdg-db1scripts]#df-Th
FilesystemTypeSizeUsedAvailUse%Mountedon
/dev/sda3ext450G19G29G40%/
tmpfstmpfs3.9G909M3.0G24%/dev/shm
/dev/sda1ext4190M40M141M22%/boot
/dev/sdb1ext430G6.0G22G22%/oradata
[root@hmdg-db1scripts]#ipaddrshowdeveth1
3:eth1:<BROADCAST,LOWER_UP>mtu1500qdiscpfifo_faststateUPqlen1000
link/ether00:0c:29:59:d5:91brdff:ff:ff:ff:ff:ff
inet172.16.10.25/24brd172.16.10.255scopeglobaleth1
inet172.16.10.130/24scopeglobalsecondaryeth1
inet172.16.10.131/24scopeglobalsecondaryeth1
inet6fe80::20c:29ff:fe59:d591/64scopelink
valid_lftforeverpreferred_lftforever
测试当主服务器Oracle服务关闭时
1、手动关闭DB1的数据库实例
[oracle@hmdg-db1~]$exportORACLE_SID=HMODB
[oracle@hmdg-db1~]$sqlplus/assysdba
SQL*Plus:Release11.2.0.1.0ProductiononSunApr1518:21:522018
Copyright(c)1982,Oracle.Allrightsreserved.
Connectedto:
OracleDatabase11gEnterpriseEditionRelease11.2.0.1.0-64bitProduction
WiththePartitioning,OLAP,DataMiningandRealApplicationTestingoptions
SQL>SELECTOPEN_MODE,DATABASE_ROLEFROMV$DATABASE;
OPEN_MODENAMEDATABASE_ROLE
---------------------------------------------
READWRITEHMODBPRIMARY
SQL>shutdownimmediate
Databaseclosed.
Databasedismounted.
ORACLEinstanceshutdown.
2、查看keepalived日志
##
Apr1518:23:34hmdg-db1Keepalived_vrrp[90792]:pid96098exitedwithstatus2
Apr1518:23:48hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)Receivedadvertwithhigherpriority110,ours100
Apr1518:23:48hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)EnteringBACKUPSTATE
Apr1518:23:48hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)removingprotocolVIPs.
Apr1518:23:48hmdg-db1Keepalived_healthcheckers[90790]:NetlinkreflectorreportsIP172.16.10.130removed
Apr1518:23:48hmdg-db1Keepalived_healthcheckers[90790]:NetlinkreflectorreportsIP172.16.10.131removed
Apr1518:23:54hmdg-db1Keepalived_vrrp[90792]:pid96486exitedwithstatus2
Apr1518:24:14hmdg-db1Keepalived_vrrp[90792]:VRRP_Script(monitor_oracle_status)succeeded
Apr1518:24:15hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)Changingeffectivepriorityfrom100to120
-------------------------------------------------------------------------------
Apr1518:23:48Sunhmdg-db1[keepalived_notify]:TransitiontoBACKUPSTATE
Apr1518:23:48Sunhmdg-db1[keepalived_notify]:removingtheVIPoneth1for172.16.10.130,172.16.10.131
Apr1518:23:48Sunhmdg-db1[keepalived_notify]:DatabaseSwitchoverToBACKUP
Apr1518:23:49Sunhmdg-db1[keepalived_notify]:Databaseinstancestateismounted
Apr1518:23:49Sunhmdg-db1[keepalived_notify]:Backupcurrentcontrolfile.
SQL>alterdatabasebackupcontrolfileto'/backup/oracle/control/control_20181504182348';
Databasealtered.
Apr1518:23:49Sunhmdg-db1[keepalived_notify]:Shutdowndatabaseinstance,pleasewait...
Databaseclosed.
Databasedismounted.
ORACLEinstanceshutdown.
Apr1518:23:55Sunhmdg-db1[keepalived_notify]:Databaseinstanceshutdownsuccessfully.
Apr1518:23:55Sunhmdg-db1[keepalived_notify]:umountsharedisk
Apr1518:23:55Sunhmdg-db1[keepalived_notify]:umount/oradatasuccess.
Apr1518:23:55Sunhmdg-db1[keepalived_notify]:stoppinglistener...
Apr1518:23:56Sunhmdg-db1[keepalived_notify]:Thelistenerstopsuccessfully
[root@hmdg-db1scripts]#ps-ef|grepora_
root975202451018:26pts/000:00:00grepora_
[root@hmdg-db1scripts]#df-Th
FilesystemTypeSizeUsedAvailUse%Mountedon
/dev/sda3ext450G19G29G40%/
tmpfstmpfs3.9G03.9G0%/dev/shm
/dev/sda1ext4190M40M141M22%/boot
[root@hmdg-db1scripts]#ipaddrshowdeveth1
3:eth1:<BROADCAST,LOWER_UP>mtu1500qdiscpfifo_faststateUPqlen1000
link/ether00:0c:29:59:d5:91brdff:ff:ff:ff:ff:ff
inet172.16.10.25/24brd172.16.10.255scopeglobaleth1
inet6fe80::20c:29ff:fe59:d591/64scopelink
valid_lftforeverpreferred_lftforever
3、查看DB2是否转换为MASTER并接管服务
##系统日志
Apr1518:23:49hmdg-db2Keepalived_vrrp[88134]:VRRP_Instance(Oracle_HA_1)forcinganewMASTERelection
Apr1518:23:51hmdg-db2Keepalived_vrrp[88134]:VRRP_Instance(Oracle_HA_1)TransitiontoMASTERSTATE
Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:VRRP_Instance(Oracle_HA_1)EnteringMASTERSTATE
Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:VRRP_Instance(Oracle_HA_1)settingprotocolVIPs.
Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.130
Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:23:53hmdg-db2Keepalived_healthcheckers[88133]:NetlinkreflectorreportsIP172.16.10.130added
Apr1518:23:53hmdg-db2Keepalived_healthcheckers[88133]:NetlinkreflectorreportsIP172.16.10.131added
Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.131
Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:23:57hmdg-db2kernel:EXT4-fs(sdb1):warning:maximalmountcountreached,runninge2fsckisrecommended
Apr1518:23:57hmdg-db2kernel:EXT4-fs(sdb1):mountedfilesystemwithordereddatamode.Opts:
Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.130
Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.131
Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.131
Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.130
Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.131
##notify脚本日志
-------------------------------------------------------------------------------
Apr1518:23:53Sunhmdg-db2[keepalived_notify]:TransitiontoMASTERSTATE
Apr1518:23:53Sunhmdg-db2[keepalived_notify]:SetuptheVIPoneth1172.16.10.130,172.16.10.131
Apr1518:23:53Sunhmdg-db2[keepalived_notify]:DatabaseSwitchoverToMASTER
Apr1518:23:53Sunhmdg-db2[keepalived_notify]:Checkremotenodesharediskmounted.
Apr1518:23:53Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[1]...
Apr1518:23:54Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[2]...
Apr1518:23:56Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[3]...
Apr1518:23:57Sunhmdg-db2[keepalived_notify]:/dev/sdb1checkpassed.
Apr1518:23:57Sunhmdg-db2[keepalived_notify]:mount/dev/sdb1on/oradata
Apr1518:23:57Sunhmdg-db2[keepalived_notify]:restorecontrolfile1
ORACLEinstancestarted.
TotalSystemGlobalArea1603411968bytes
FixedSize2213776bytes
VariableSize1056966768bytes
DatabaseBuffers536870912bytes
RedoBuffers7360512bytes
RecoveryManager:Release11.2.0.1.0-ProductiononSunApr1518:23:582018
Copyright(c)1982,Oracleand/oritsaffiliates.Allrightsreserved.
connectedtotargetdatabase:HMODB(notmounted)
usingtargetdatabasecontrolfileinsteadofrecoverycatalog
RMAN>
Startingrestoreat15-APR-18
allocatedchannel:ORA_DISK_1
channelORA_DISK_1:SID=572devicetype=DISK
channelORA_DISK_1:copiedcontrolfilecopy
outputfilename=/oradata/HMODB/control01.ctl
outputfilename=/u01/app/oracle/flash_recovery_area/HMODB/control02.ctl
outputfilename=/home/oracle/rman/HMODB/control03.ctl
Finishedrestoreat15-APR-18
RMAN>
RecoveryManagercomplete.
Apr1518:23:59Sunhmdg-db2[keepalived_notify]:alterdatabasetomount
Databasealtered.
Apr1518:24:04Sunhmdg-db2[keepalived_notify]:alterdatabasetoopen
Databasealtered.
Apr1518:24:11Sunhmdg-db2[keepalived_notify]:DatabaSEOpened.
Apr1518:24:11Sunhmdg-db2[keepalived_notify]:Startuplistener
Apr1518:24:11Sunhmdg-db2[keepalived_notify]:listeneralreadystarted.
##服务状态检查(Oracle进程、磁盘、VIP状态)
[root@hmdg-db2scripts]#ps-ef|grepora_
oracle886291018:23?00:00:00ora_pmon_HMODB
oracle886311018:23?00:00:00ora_vktm_HMODB
oracle886351018:23?00:00:00ora_gen0_HMODB
oracle886371018:23?00:00:00ora_diag_HMODB
oracle886391018:23?00:00:00ora_dbrm_HMODB
oracle886411018:23?00:00:00ora_psp0_HMODB
oracle886431018:23?00:00:00ora_dia0_HMODB
oracle886451018:23?00:00:00ora_mman_HMODB
oracle886471018:23?00:00:00ora_dbw0_HMODB
oracle886491018:23?00:00:00ora_lgwr_HMODB
oracle886511018:23?00:00:00ora_ckpt_HMODB
oracle886531018:23?00:00:00ora_smon_HMODB
oracle886551018:23?00:00:00ora_reco_HMODB
oracle886571018:23?00:00:00ora_mmon_HMODB
oracle886591018:23?00:00:00ora_mmnl_HMODB
oracle886611018:23?00:00:00ora_d000_HMODB
oracle886631018:23?00:00:00ora_s000_HMODB
oracle888281018:24?00:00:00ora_arc0_HMODB
oracle888431018:24?00:00:00ora_arc1_HMODB
oracle888451018:24?00:00:00ora_arc2_HMODB
oracle888471018:24?00:00:00ora_arc3_HMODB
oracle889441018:24?00:00:00ora_qmnc_HMODB
oracle890111018:24?00:00:00ora_cjq0_HMODB
oracle890261018:24?00:00:00ora_vkrm_HMODB
oracle890541018:24?00:00:00ora_q000_HMODB
oracle890561018:24?00:00:00ora_q001_HMODB
root9079031390018:27pts/200:00:00grepora_
[root@hmdg-db2scripts]#df-Th
FilesystemTypeSizeUsedAvailUse%Mountedon
/dev/sda3ext450G18G30G37%/
tmpfstmpfs3.9G909M3.0G24%/dev/shm
/dev/sda1ext4190M40M141M22%/boot
/dev/sdb1ext430G6.0G22G22%/oradata
[root@hmdg-db2scripts]#ipaddrshowdeveth1
3:eth1:<BROADCAST,LOWER_UP>mtu1500qdiscpfifo_faststateUPqlen1000
link/ether00:0c:29:dd:d1:7fbrdff:ff:ff:ff:ff:ff
inet172.16.10.26/24brd172.16.10.255scopeglobaleth1
inet172.16.10.130/24scopeglobalsecondaryeth1
inet172.16.10.131/24scopeglobalsecondaryeth1
inet6fe80::20c:29ff:fedd:d17f/64scopelink
valid_lftforeverpreferred_lftforever
结尾
本篇中主要的重点再在于测试部分。关于如何设计数据库高可用集群方案可作为参考
（编辑：李大同）
【声明】本站内容均来自网络，其相关言论仅代表作者个人观点，不代表本站立场。若无意侵犯到您的权利，请及时与联系站长删除相关内容!
