I'm installing RAC (12.1.0.2) on RHEL 7.2 and running the cluster verify utility to check the setup before installation of grid.
I'm having difficulty getting past the various network connectivity checks unless I essentially turn off the firewall on both nodes. Almost all of the google hits I've seen on this issues simply say "turn off your firewall" but that seems much too "unsecure". In the Oracle docs and notes, I see that it is recommended to turn off the firewall on the private network interface and that makes sense. I've done that but I still get errors from cluster verify during the testing of the public interfaces.
I've opened the following ports (I didn't start with all of these but I've been trying to add ones that looked like they were being blocked):
firewall-cmd --list-ports
1521/tcp 443/tcp 5500/tcp 42424/udp 1630/tcp 5353/tcp 42414/udp 33887/udp 3872/tcp 22/tcp 40338/udp 137/udp 5353/udp 138/udp 53/udp 6100/tcp 6200/tcp
Here is the output of cluster verify from the connectivity check
Checking node reachability...
Check: Node reachability from node "ora-ch589"
Destination Node Reachable?
------------------------------------ ------------------------
ora-ch589 yes
ora-ch588 yes
Result: Node reachability check passed from node "ora-ch589"
Checking user equivalence...
Check: User equivalence for user "grid"
Node Name Status
------------------------------------ ------------------------
ora-ch589 passed
ora-ch588 passed
Result: User equivalence check passed for user "grid"
Checking node connectivity...
Checking hosts config file...
Node Name Status
------------------------------------ ------------------------
ora-ch589 passed
ora-ch588 passed
Verification of the hosts config file successful
Interface information for node "ora-ch589"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eno16780032 10.109.2.151 10.109.2.0 0.0.0.0 10.109.2.1 00:50:56:85:02:41 1500
eno33559296 192.168.1.11 192.168.1.0 0.0.0.0 10.109.2.1 00:50:56:85:10:BE 1500
Interface information for node "ora-ch588"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eno16780032 10.109.2.148 10.109.2.0 0.0.0.0 10.109.2.1 00:50:56:85:5D:0F 1500
eno33559296 192.168.1.10 192.168.1.0 0.0.0.0 10.109.2.1 00:50:56:85:3C:67 1500
Check: Node connectivity of subnet "10.109.2.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
ora-ch589[10.109.2.151] ora-ch588[10.109.2.148] yes
Result: Node connectivity passed for subnet "10.109.2.0" with node(s) ora-ch589,ora-ch588
Check: TCP connectivity of subnet "10.109.2.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
ora-ch589 : 10.109.2.151 ora-ch589 : 10.109.2.151 passed
ora-ch588 : 10.109.2.148 ora-ch589 : 10.109.2.151 failed
ERROR:
PRVG-11850 : The system call "connect" failed with error "113" while executing exectask on node "ora-ch588"
No route to host
ora-ch589 : 10.109.2.151 ora-ch588 : 10.109.2.148 failed
ERROR:
PRVG-11850 : The system call "connect" failed with error "113" while executing exectask on node "ora-ch589"
No route to host
ora-ch588 : 10.109.2.148 ora-ch588 : 10.109.2.148 passed
Result: TCP connectivity check failed for subnet "10.109.2.0"
Check: Node connectivity of subnet "192.168.1.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
ora-ch589[192.168.1.11] ora-ch588[192.168.1.10] yes
Result: Node connectivity passed for subnet "192.168.1.0" with node(s) ora-ch589,ora-ch588
Check: TCP connectivity of subnet "192.168.1.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
ora-ch589 : 192.168.1.11 ora-ch589 : 192.168.1.11 passed
ora-ch588 : 192.168.1.10 ora-ch589 : 192.168.1.11 passed
ora-ch589 : 192.168.1.11 ora-ch588 : 192.168.1.10 passed
ora-ch588 : 192.168.1.10 ora-ch588 : 192.168.1.10 passed
Result: TCP connectivity check passed for subnet "192.168.1.0"
Interfaces found on subnet "10.109.2.0" that are likely candidates for VIP are:
ora-ch589 eno16780032:10.109.2.151
ora-ch588 eno16780032:10.109.2.148
Interfaces found on subnet "192.168.1.0" that are likely candidates for a private interconnect are:
ora-ch589 eno33559296:192.168.1.11
ora-ch588 eno33559296:192.168.1.10
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "10.109.2.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.
Result: Node connectivity check failed
Checking multicast communication...
Checking subnet "10.109.2.0" for multicast communication with multicast group "224.0.0.251"...
PRVG-11138 : Interface "10.109.2.151" on node "ora-ch589" is not able to communicate with interface "10.109.2.148" on node "ora-ch588" over multicast group "224.0.0.251"
PRVG-11138 : Interface "10.109.2.148" on node "ora-ch588" is not able to communicate with interface "10.109.2.151" on node "ora-ch589" over multicast group "224.0.0.251"
Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251" passed.
Check of multicast communication passed.
As noted, I've gone so far as to log the ports that the firewall decides to drop and try opening them up. It seems that some ports used are dynamic so it's a never-ending chase that leads me nowhere.
What *does* work is to disable the firewall for communication between the two nodes by IP address. So I create a firewall rule such as this
firewall-cmd --zone=public --add-rich-rule='rule family="ipv4" source address="10.109.2.148" accept'
firewall-cmd --zone=public --add-rich-rule='rule family="ipv4" source address="10.109.2.151" accept'
which allows anything between the nodes and the connectivity check passes.
Checking node reachability...
Check: Node reachability from node "ora-ch589"
Destination Node Reachable?
------------------------------------ ------------------------
ora-ch589 yes
ora-ch588 yes
Result: Node reachability check passed from node "ora-ch589"
Checking user equivalence...
Check: User equivalence for user "grid"
Node Name Status
------------------------------------ ------------------------
ora-ch589 passed
ora-ch588 passed
Result: User equivalence check passed for user "grid"
Checking node connectivity...
Checking hosts config file...
Node Name Status
------------------------------------ ------------------------
ora-ch589 passed
ora-ch588 passed
Verification of the hosts config file successful
Interface information for node "ora-ch589"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eno16780032 10.109.2.151 10.109.2.0 0.0.0.0 10.109.2.1 00:50:56:85:02:41 1500
eno33559296 192.168.1.11 192.168.1.0 0.0.0.0 10.109.2.1 00:50:56:85:10:BE 1500
Interface information for node "ora-ch588"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eno16780032 10.109.2.148 10.109.2.0 0.0.0.0 10.109.2.1 00:50:56:85:5D:0F 1500
eno33559296 192.168.1.10 192.168.1.0 0.0.0.0 10.109.2.1 00:50:56:85:3C:67 1500
Check: Node connectivity of subnet "10.109.2.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
ora-ch589[10.109.2.151] ora-ch588[10.109.2.148] yes
Result: Node connectivity passed for subnet "10.109.2.0" with node(s) ora-ch589,ora-ch588
Check: TCP connectivity of subnet "10.109.2.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
ora-ch589 : 10.109.2.151 ora-ch589 : 10.109.2.151 passed
ora-ch588 : 10.109.2.148 ora-ch589 : 10.109.2.151 passed
ora-ch589 : 10.109.2.151 ora-ch588 : 10.109.2.148 passed
ora-ch588 : 10.109.2.148 ora-ch588 : 10.109.2.148 passed
Result: TCP connectivity check passed for subnet "10.109.2.0"
Check: Node connectivity of subnet "192.168.1.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
ora-ch589[192.168.1.11] ora-ch588[192.168.1.10] yes
Result: Node connectivity passed for subnet "192.168.1.0" with node(s) ora-ch589,ora-ch588
Check: TCP connectivity of subnet "192.168.1.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
ora-ch589 : 192.168.1.11 ora-ch589 : 192.168.1.11 passed
ora-ch588 : 192.168.1.10 ora-ch589 : 192.168.1.11 passed
ora-ch589 : 192.168.1.11 ora-ch588 : 192.168.1.10 passed
ora-ch588 : 192.168.1.10 ora-ch588 : 192.168.1.10 passed
Result: TCP connectivity check passed for subnet "192.168.1.0"
Interfaces found on subnet "10.109.2.0" that are likely candidates for VIP are:
ora-ch589 eno16780032:10.109.2.151
ora-ch588 eno16780032:10.109.2.148
Interfaces found on subnet "192.168.1.0" that are likely candidates for a private interconnect are:
ora-ch589 eno33559296:192.168.1.11
ora-ch588 eno33559296:192.168.1.10
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "10.109.2.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.
Result: Node connectivity check passed
Checking multicast communication...
Checking subnet "10.109.2.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "10.109.2.0" for multicast communication with multicast group "224.0.0.251" passed.
Check of multicast communication passed.
So the question is: Is this the recommended way to fix this connectivity test? It seems odd that Oracle docs don't really address this issue (as far as I can tell) and that almost everyone just disables the entire firewall to get around it.
Thanks for any insight.