Issue: node2 was not reachable to the cluster , hence node2 db instance, rdbms, grid software is deleted as per the below links
https://docs.oracle.com/database/121/CWADD/adddelclusterware.htm#CWADD90997
https://docs.oracle.com/database/121/CWADD/adddelclusterware.htm#CWADD1167
node2 was successfully deleted..
We started re-adding the node2 back to the cluster where node1 is perfectly working..
Extend Clusterware as gid owner is done - from node1
./addnode.sh -silent "CLUSTER_NEW_NODES={node2 }" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={node2 -vip}"
cluster on node2 has come up.. then started extending database software
cd $ORACLE_HOME/addnode ---$RDBMS_ORACLE_HOME/addnode
./addnode.sh "CLUSTER_NEW_NODES={node2}
It hangs forever.. install log not showing any errors.
olsnodes command output showsn both nodes on node2
olsnodes command execution hangs forever on node1 and also even after terminating above addnode.sh execution.
on node1 where everything working fine as of now, one process get started automatically with name /bin/sh $grid_home/bin/olsnodes, we killed this process with root user but it comes again after couple of minutes and one more process with same command .. we ended up having >10. killed all these.. /bin/sh $grid_home/bin/olsnodes getting started automatically and appears to be running endlessly.
how can I make olsnodes works and it lists both nodes on node1..
if i restart cluster on node1 , will everything comes okay? .. how come olsnodes , ./addnode.sh hangs forever ( we did let it run > 6 hours).
i need to make sure at least node1 is healthy in all aspect where olsnodes command hangs now and new /bin/sh gridhome/../olsnodes process get started automatically often..
can someone help me plz..