1. Check backup of OCR
ocrconfig -showbackup
crsctl query css votedisk
Location to check - /cdata/cluster_name
2.Take a backup of the gpnp profile on all nodes before starting.
cd /gpnp/dbnode1/profiles/peer
cp -p profile.xml profile.xml.bk
3.Determine if the CLUSTER_INTERCONNECT parameter is used in the
Oracle Database and Oracle ASM instances using the following command
set linesize 300
col NAME form a50
col VALUE form a50
SELECT inst_id, name,value FROM
gv$parameter WHERE name ='cluster_interconnects';
INST_ID
NAME
VALUE
----------
--------------------------------------------------
--------------------------------------------------
1
cluster_interconnects
192.168.10.31
2
cluster_interconnects
192.168.10.32
ASM
ALTER SYSTEM SET
CLUSTER_INTERCONNECTS='192.168.20.31' SCOPE=SPFILE SID='+ASM1';
ALTER SYSTEM SET
CLUSTER_INTERCONNECTS='192.168.20.32' SCOPE=SPFILE SID='+ASM2';
DATABASE
ALTER SYSTEM SET
CLUSTER_INTERCONNECTS='192.168.20.31' SCOPE=SPFILE SID='DATABASE1';
ALTER SYSTEM SET
CLUSTER_INTERCONNECTS='192.168.20.32' SCOPE=SPFILE SID='DATABASE2';
4.Shut down all cluster-managed services on each database server
as the oracle user using the following command:
srvctl stop home -o -s /tmp/dbnode1_dbhome -n
dbnode1
srvctl stop home -o -s /tmp/dbnode2_dbhome -n
dbnode2
5. Modify the cluster interconnect interface to use the BONDIB0
interface on the first database server as follows - NEEDS TO BE DONE ON FIRST NODE
ONLY
a. Log in as the oracle user.
b. Set ORACLE_HOME to the Grid Infrastructure home.
c. Set the base for the ORACLE_SID environment variable using
the following command. The
ORACLE_HOME environment variable must be set to the Grid
Infrastructure home.
5.1 oifcfg iflist
eth0 10.240.64.0
bondeth0 10.240.48.0
bondib0 192.168.8.0
bondib0 169.254.0.0
5.2 oifcfg getif
bondeth0 10.240.48.0 global public
bondib0 192.168.8.0 global
cluster_interconnect
Assign BONDIB0 as the global cluster interconnect interface
using the following command:
5.3 oifcfg setif
-global bond0/192.168.20.0:cluster_interconnect
5.4 Check Ips again
oifcfg getif
Output
bondeth0 10.240.48.0 global public
Bondib0 192.168.8.0 global
cluster_interconnect
Bondib0 192.168.20.0 global
cluster_interconnect
6.Shut down Oracle Clusterware and
Oracle Clusterware CRS on each database server as follows
a./bin/crsctl stop crs -f
b./bin/crsctl disable crs
7.Change the InfiniBand
IP addresses on each Exadata Storage Server as follows:
7.1 login as root
7.2 service celld
stop
7.3 service ocrvottargetd stop
7.4 ipconf
[root@cellnode1 ~]# service celld stop
Stopping the RS, CELLSRV, and MS
services...
The SHUTDOWN of services was successful.
[root@cellnode1 ~]# service
ocrvottargetd stop
[root@cellnode1 ~]# service
ocrvottargetd status
[root@cellnode1 ~]# ipconf
Logging started to
/var/log/cellos/ipconf.log
Interface ib0 is Linked. hca:
mlx4_0
Interface ib1 is Linked. hca:
mlx4_0
Interface eth0 is Linked.
driver/mac: igb/00:21:28:d7:cf:82
Interface eth1 is ... Unlinked.
driver/mac: igb/00:21:28:d7:cf:83
Interface eth2 is ... Unlinked.
driver/mac: igb/00:21:28:d7:cf:84
Interface eth3 is ... Unlinked.
driver/mac: igb/00:21:28:d7:cf:85
Network interfaces
Name State
IP address Netmask
Gateway Net type Hostname
ib0 Linked
ib1 Linked
eth0 Linked
eth1 Unlinked
eth2 Unlinked
eth3 Unlinked
Warning. Some network interface(s) are
disconnected. Check cables and swicthes and retry
Do you want to retry (y/n) [y]: n
The current nameserver(s): 10.239.32.5
10.38.9.31 10.239.96.36 10.8.130.191
Do you want to change it (y/n) [n]:
The current timezone: Asia/Kolkata
Do you want to change it (y/n) [n]:
The current NTP server(s): 10.46.192.98
Do you want to change it (y/n) [n]:
Network interfaces
Name State
IP address Netmask
Gateway Net type Hostname
eth0 Linked
10.240.64.237 255.255.255.0 10.240.64.1 Management
cellnode1.enterprisenet.org
eth1 Unlinked
eth2 Unlinked
eth3 Unlinked
bondib0 ib0,ib1
192.168.10.33 255.255.252.0
Private
cellnode1-priv.enterprisenet.org
Select interface name to configure or
press Enter to continue: bondib0
Selected interface. bondib0
IP address or none [192.168.10.33]:
192.168.20.33
Netmask [255.255.252.0]:
Fully qualified hostname or none
[cellnode1-priv.enterprisenet.org]:
Continue configuring or re-configuring
interfaces? (y/n) [y]: n
Select canonical hostname from the list
below
1: cellnode1.enterprisenet.org
2: cellnode1-priv.enterprisenet.org
Canonical fully qualified domain name
[1]:
Select default gateway interface from
the list below
1: eth0
Default gateway interface [1]:
Canonical hostname:
cellnode1.enterprisenet.org
Nameservers: 10.239.32.5 10.38.9.31
10.239.96.36 10.8.130.191
Timezone: Asia/Kolkata
NTP servers: 10.46.192.98
Default gateway device: eth0
Network interfaces
Name State
IP address Netmask
Gateway Net type Hostname
eth0 Linked
10.240.64.237 255.255.255.0 10.240.64.1 Management
cellnode1.enterprisenet.org
eth1 Unlinked
eth2 Unlinked
eth3 Unlinked
bondib0 ib0,ib1
192.168.20.33 255.255.252.0
Private
cellnode1-priv.enterprisenet.org
Is this correct (y/n) [y]:
Do you want to configure basic ILOM
settings (y/n) [y]: n
Info. IP address of bondib0 changed to
"192.168.20.33"
Info. Run
/opt/oracle.cellos/validations/init.d/saveconfig
Info. Custom changes have been detected
in /etc/sysconfig/network-scripts/ifcfg-bondib0
Warning. Config
/etc/sysconfig/network-scripts/ifcfg-bondib0. Existing line
"IPADDR=192.168.10.33" will be overwritten with
"IPADDR=192.168.20.33"
Info. Config
/etc/sysconfig/network-scripts/ifcfg-bondib0. Original line
"IPADDR=192.168.10.33" will be commented
Warning. Config /etc/sysconfig/network-scripts/ifcfg-bondib0.
Existing line "NETWORK=192.168.8.0" will be overwritten with
"NETWORK=192.168.20.0"
Info. Config
/etc/sysconfig/network-scripts/ifcfg-bondib0. Original line
"NETWORK=192.168.8.0" will be commented
Warning. Config /etc/sysconfig/network-scripts/ifcfg-bondib0.
Existing line "BROADCAST=192.168.11.255" will be overwritten with
"BROADCAST=192.168.23.255"
Info. Config
/etc/sysconfig/network-scripts/ifcfg-bondib0. Original line
"BROADCAST=192.168.11.255" will be commented
Info. Original file will be saved in
/opt/oracle.cellos/conf/network-scripts/backup_by_Exadata_ipconf
Info. Original file will be saved in
/etc/ssh/sshd_config.backupbyExadata
Info. Generate /etc/ssh/sshd_config with
ListenAddress(es) 10.240.64.237, 192.168.20.33
Stopping sshd:
[ OK ]
Starting sshd:
[ OK ]
bash
/opt/oracle.cellos/ib_set_node_desc.sh
Re-login using new IP address
10.240.64.237 if you were disconnected after following commands
echo -n
ifdown bondib0
sleep 1
/opt/oracle.cellos/reset_rule_table.sh
200
sleep 1
ifup bondib0
sleep 3
cellcli -e "alter cell startup
services rs"
cellcli -e "alter cell startup
services ms"
cellcli -e "alter cell
interconnect1=bondib0"
cellcli -e "alter cell shutdown
services all force"
service ocrvottargetd restart
sleep 1
tgtadm: can't find the target
Starting the RS services...
Getting the state of RS services...
running
Starting MS services...
The STARTUP of MS services was
successful.
A restart of all services is required to
put new network configuration into effect. MS-CELLSRV communication may be
hampered until restart.
Cell cellnode1 successfully altered
Stopping the RS, CELLSRV, and MS
services...
The SHUTDOWN of services was successful.
ipaddress1=192.168.20.33/22
[root@cellnode1 ~]#
7.5 as root user - reboot
7.6 Verify
the newly-assigned InfiniBand address on Exadata Storage Server using the
following command
cellcli -e list cell detail | grep
ipaddress1
Current output :
[root@cellnode1 ~]# cellcli -e list cell detail | grep
ipaddress1
ipaddress1:
192.168.10.33/22
Output after change
[root@cellnode1 ~]# cellcli -e list cell detail | grep ipaddress1
ipaddress1:
192.168.20.33/22
7.7. If there is an error indicating that one or more cell
services are not running, then restart the cell services using the
following command:
cellcli -e alter cell restart services
all
8.Change the InfiniBand IP addresses on each
database server as follows - NEEDS
TO BE PERFORMED ON ALL NODES ONE BY ONE
8.1 login as root
8.2 cd /etc/sysconfig/network-scripts
8.3 cp
ifcfg-bondib0 orig_ifcfg-bondib0
8.4 Edit the ifcfg-bond0 file to update the IPADDR, NETMASK, NETWORK and BROADCAST
fields.
[root@dbnode1 network-scripts]# cat
ifcfg-bondib0
#### DO NOT REMOVE THESE LINES ####
#### %GENERATED BY CELL% ####
DEVICE=bondib0
USERCTL=no
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.10.31
NETMASK=255.255.252.0
NETWORK=192.168.8.0
BROADCAST=192.168.11.255
BONDING_OPTS="mode=active-backup
miimon=100 downdelay=5000 updelay=5000 num_grat_arp=100"
IPV6INIT=no
MTU=65520
[root@dbnode2 network-scripts]# cat
ifcfg-bondib0
#### DO NOT REMOVE THESE LINES ####
#### %GENERATED BY CELL% ####
DEVICE=bondib0
USERCTL=no
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.10.32
NETMASK=255.255.252.0
NETWORK=192.168.8.0
BROADCAST=192.168.11.255
BONDING_OPTS="mode=active-backup
miimon=100 downdelay=5000 updelay=5000 num_grat_arp=100"
IPV6INIT=no
MTU=65520
UPDATED VALUE for DBNODE1 and
DBNODE2
FOR DBNODE1?
IPADDR=192.168.20.31
NETMASK=255.255.252.0
NETWORK=192.168.20.0
BROADCAST=192.168.23.255
FOR DBNODE2?
IPADDR=192.168.20.32
NETMASK=255.255.252.0
NETWORK=192.168.20.0
BROADCAST=192.168.23.255
8.5 reboot
8.6 Verify the InfiniBand IP address
information using the following command
Current value - dbnode1
bondib0 Link
encap:InfiniBand HWaddr
80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
inet addr:192.168.10.32 Bcast:192.168.11.255 Mask:255.255.252.0
inet6 addr: fe80::221:2800:1cf:2b97/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:65520 Metric:1
RX packets:62232895 errors:0 dropped:0 overruns:0 frame:0
TX packets:67053494 errors:3 dropped:14 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:41159776048 (38.3 GiB) TX bytes:34502152558 (32.1 GiB)
Current value - dbnode2
bondib0 Link
encap:InfiniBand HWaddr
80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
inet addr:192.168.10.31 Bcast:192.168.11.255 Mask:255.255.252.0
inet6 addr: fe80::221:2800:1cf:1e67/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:65520 Metric:1
RX packets:67191396 errors:0 dropped:0 overruns:0 frame:0
TX packets:63617557 errors:0 dropped:11 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:34394734207 (32.0 GiB) TX bytes:41626158309 (38.7 GiB)
ifconfig -a
bondib0 Link
encap:InfiniBand HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
inet addr:192.168.20.32
Bcast:192.168.23.255 Mask:255.255.252.0
inet6 addr: fe80::221:2800:1cf:2b97/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:65520 Metric:1
RX packets:62232895 errors:0 dropped:0 overruns:0 frame:0
TX packets:67053494 errors:3 dropped:14 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:41159776048 (38.3 GiB) TX bytes:34502152558 (32.1 GiB)
Current value - dbnode2
bondib0 Link
encap:InfiniBand HWaddr
80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
inet addr:192.168.20.31
Bcast:192.168.23.255 Mask:255.255.252.0
inet6 addr: fe80::221:2800:1cf:1e67/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:65520 Metric:1
RX packets:67191396 errors:0 dropped:0 overruns:0 frame:0
TX packets:63617557 errors:0 dropped:11 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:34394734207 (32.0 GiB) TX bytes:41626158309 (38.7 GiB)
9.Modify Cellinit.ora (Do not edit the cellinit.ora or cellip.ora files when the database or Oracle ASM
instance are running)
9.1 cd /etc/oracle/cell/network-config/
Current value on dbnode1
[root@dbnode1 network-config]# cat
cellinit.ora
ipaddress1=192.168.10.31/22
[root@dbnode2 network-config]# cat
cellinit.ora
ipaddress1=192.168.10.32/22
9.1 cp
cellinit.ora cellinit.new
9.2 Modify cellinit.new to
ipaddress1=192.168.20.31/22
ipaddress1=192.168.20.32/22
9.3 rename the file back
mv cellinit.new cellinit.ora
10.Modify cellip.ora
10.1 cd /etc/oracle/cell/network-config/
Current Values :
[root@dbnode1 network-config]# cat
cellip.ora
cell="192.168.10.33"
cell="192.168.10.34"
cell="192.168.10.35"
[root@dbnode2 network-config]# cat
cellip.ora
cell="192.168.10.33"
cell="192.168.10.34"
cell="192.168.10.35"
10.2 cp cellip.ora orig_cellip.ora
10.3 modify cellip.ora to following
on both nodes
cell="192.168.20.33"
cell="192.168.20.34"
cell="192.168.20.35"
11.Update the /etc/hosts file on each database server and Exadata
Storage Servers
11.1 login as root
11.2 Current
value on both dbnode1 and dbnode2
### Compute Node Private Interface details
#####192.168.10.1
dbnode1-priv.enterprisenet.org dbnode1-priv
#####192.168.10.2
dbnode2-priv.enterprisenet.org dbnode2-priv
192.168.10.31 dbnode1-priv.enterprisenet.org
dbnode1-priv
192.168.10.32 dbnode2-priv.enterprisenet.org
dbnode2-priv
### CELL Node Private Interface details
#####192.168.10.3
cellnode1-priv.enterprisenet.org cellnode1-priv
#####192.168.10.4
cellnode2-priv.enterprisenet.org cellnode2-priv
#####192.168.10.5
cellnode3-priv.enterprisenet.org cellnode3-priv
192.168.10.33 cellnode1-priv.enterprisenet.org
cellnode1-priv
192.168.10.34 cellnode2-priv.enterprisenet.org
cellnode2-priv
192.168.10.35 cellnode3-priv.enterprisenet.org
cellnode3-priv
On CELLNODE1
#### END Generated by Exadata ####
### Compute Node Private Interface details
192.168.10.1 dbnode1-priv.enterprisenet.org
dbnode1-priv
192.168.10.2 dbnode2-priv.enterprisenet.org
dbnode2-priv
### CELL Node Private Interface details
# 192.168.10.3 cellnode1-priv.enterprisenet.org
cellnode1-priv
192.168.10.4 cellnode2-priv.enterprisenet.org
cellnode2-priv
192.168.10.5 cellnode3-priv.enterprisenet.org
cellnode3-priv
On CELLNODE2
#### END Generated by Exadata ####
### Compute Node Private Interface details
192.168.10.1 dbnode1-priv.enterprisenet.org
dbnode1-priv
192.168.10.2 dbnode2-priv.enterprisenet.org
dbnode2-priv
### CELL Node Private Interface details
192.168.10.3 cellnode1-priv.enterprisenet.org
cellnode1-priv
# 192.168.10.4 cellnode2-priv.enterprisenet.org
cellnode2-priv
192.168.10.5 cellnode3-priv.enterprisenet.org
cellnode3-priv
Updated value
On CELLNODE3
#### END Generated by Exadata ####
### Compute Node Private Interface
details
192.168.10.1
dbnode1-priv.enterprisenet.org dbnode1-priv
192.168.10.2
dbnode2-priv.enterprisenet.org dbnode2-priv
### CELL Node Private Interface details
192.168.10.3
cellnode1-priv.enterprisenet.org cellnode1-priv
192.168.10.4
cellnode2-priv.enterprisenet.org cellnode2-priv
# 192.168.10.5 cellnode3-priv.enterprisenet.org
cellnode3-priv
11.3 cp /etc/hosts /etc/orig_hosts
11.4 update to
On both dbnode1 and dbnode2
#####192.168.10.1
dbnode1-priv.enterprisenet.org dbnode1-priv
#####192.168.10.2
dbnode2-priv.enterprisenet.org dbnode2-priv
#####192.168.10.31 dbnode1-priv.enterprisenet.org
dbnode1-priv
#####192.168.10.32 dbnode2-priv.enterprisenet.org
dbnode2-priv
192.168.20.31 dbnode1-priv.enterprisenet.org
dbnode1-priv
192.168.20.32 dbnode2-priv.enterprisenet.org
dbnode2-priv
### CELL Node Private Interface details
#####192.168.10.3
cellnode1-priv.enterprisenet.org cellnode1-priv
#####192.168.10.4
cellnode2-priv.enterprisenet.org cellnode2-priv
#####192.168.10.5
cellnode3-priv.enterprisenet.org cellnode3-priv
#####192.168.10.33 cellnode1-priv.enterprisenet.org
cellnode1-priv
#####192.168.10.34 cellnode2-priv.enterprisenet.org
cellnode2-priv
#####192.168.10.35 cellnode3-priv.enterprisenet.org
cellnode3-priv
192.168.20.33 cellnode1-priv.enterprisenet.org cellnode1-priv
192.168.20.34 cellnode2-priv.enterprisenet.org
cellnode2-priv
192.168.20.35 cellnode3-priv.enterprisenet.org
cellnode3-priv
On CELLNODE1
#### END Generated by Exadata ####
### Compute Node Private Interface details
###192.168.10.1 dbnode1-priv.enterprisenet.org
dbnode1-priv
###192.168.10.2 dbnode2-priv.enterprisenet.org
dbnode2-priv
192.168.20.31 dbnode1-priv.enterprisenet.org
dbnode1-priv
192.168.20.32 dbnode2-priv.enterprisenet.org
dbnode2-priv
### CELL Node Private Interface details
### 192.168.10.3 cellnode1-priv.enterprisenet.org
cellnode1-priv
##192.168.10.4 cellnode2-priv.enterprisenet.org
cellnode2-priv
##192.168.10.5 cellnode3-priv.enterprisenet.org
cellnode3-priv
# 192.168.20.33 cellnode1-priv.enterprisenet.org
cellnode1-priv
192.168.20.34 cellnode2-priv.enterprisenet.org
cellnode2-priv
192.168.20.35 cellnode3-priv.enterprisenet.org
cellnode3-priv
On CELLNODE2
#### END Generated by Exadata ####
### Compute Node Private Interface details
###192.168.10.1 dbnode1-priv.enterprisenet.org
dbnode1-priv
###192.168.10.2 dbnode2-priv.enterprisenet.org
dbnode2-priv
192.168.20.31 dbnode1-priv.enterprisenet.org
dbnode1-priv
192.168.20.32 dbnode2-priv.enterprisenet.org
dbnode2-priv
### CELL Node Private Interface details
##192.168.10.3 cellnode1-priv.enterprisenet.org
cellnode1-priv
### 192.168.10.4 cellnode2-priv.enterprisenet.org
cellnode2-priv
##192.168.10.5 cellnode3-priv.enterprisenet.org
cellnode3-priv
192.168.20.33 cellnode1-priv.enterprisenet.org
cellnode1-priv
# 192.168.20.34 cellnode2-priv.enterprisenet.org
cellnode2-priv
192.168.20.35 cellnode3-priv.enterprisenet.org
cellnode3-priv
On CELLNODE3
#### END Generated by Exadata ####
### Compute Node Private Interface details
###192.168.10.1 dbnode1-priv.enterprisenet.org
dbnode1-priv
###192.168.10.2 dbnode2-priv.enterprisenet.org
dbnode2-priv
192.168.20.31 dbnode1-priv.enterprisenet.org
dbnode1-priv
192.168.20.32 dbnode2-priv.enterprisenet.org dbnode2-priv
### CELL Node Private Interface details
##192.168.10.3 cellnode1-priv.enterprisenet.org
cellnode1-priv
## 192.168.10.4 cellnode2-priv.enterprisenet.org
cellnode2-priv
###192.168.10.5 cellnode3-priv.enterprisenet.org
cellnode3-priv
192.168.20.33
cellnode1-priv.enterprisenet.org cellnode1-priv
192.168.20.34
cellnode2-priv.enterprisenet.org cellnode2-priv
#192.168.20.35 cellnode3-priv.enterprisenet.org
cellnode3-priv
12. Start Oracle Clusterware on each server using the following
command
/bin/crsctl start crs
13. Verify the cluster interconnect is using
the RDS protocol on each database server by examining the Oracle ASM
alert.log
cd
/u01/app/oracle/diag/asm/+asm/+ASM1/trace for dbnode2
cd /u01/app/oracle/diag/asm/+asm/+ASM2/trace
for dbnode2
view alert_+ASM1.log
view alert_+ASM2.log
Search for "CELL
interconnect IPC version: Oracle RDS/IP (generic)"
13.1 Check using skgxpinfo
Current ouput
oracle >skgxpinfo
rds
13.2 If the instance is not using the
RDS protocol over InfiniBand, then relink the Oracle binary as follows(if it is
using RDS already, it does not change while chaging IP):
13.2.1 :As the oracle user, shut down any processes using the Oracle binary
crscrtl stop crs -f
13.2.2 As the root user
/crs/install/rootcrs.pl -unlock
13.2.3 as oracle user
cd
/u01/app/oracle/product/11.2.0/dbhome_1/rdbms/lib
make -f ins_rdbms.mk ipc_rds ioracle
13.2.4 run the following command if relinking the grid infrastructure
home. Do not perform this step if you are not relinking the grid infrastructure
home.
GRID_HOME/crs/install/rootcrs.pl
-patch
14.Start
all cluster-managed services using the SRVCTL utility as follows
14.1 Login as oracle
14.2 srvctl start
home -o /u01/app/oracle/product/11.2.0/dbhome_1 -s /tmp/dbnode1_dbhome -n
dbnode1
14.3 srvctl status database -d DATABASE
15. Verifiy using on both ASM and
DB instance
set linesize 300
col NAME form a50
col VALUE form a50
SELECT inst_id, name,value FROM
gv$parameter WHERE name ='cluster_interconnects';
16. Delete the Old
Private Network
oifcfg delif -global bondib0/192.168.8.0
16.1 Verify Old Interface
is no longer present
oifcfg getif
==> oifcfg getif
bondeth0 10.240.48.0 global public
bondib0 192.168.30.0 global cluster_interconnect
17.1 Enable Oracle Clusterware CRS automatic
restart on each database server as follows
u01/app/11.2.0/grid/bin/crsctl enable
crs on both nodes
18. Restart the cluster one by one
and verify
