Database Software

OS: OEL 5 U4 x86_64
DB: Oracle 11.2.0.1 EE
Grid Infrastructure: Oracle 11.2.0.1
CRS and Voting disk Storage: ASM
Datafile and FRA storage: ASM

I'm not sure exactly what caused this, but anyways, I changed MTU from 1500 to 900 online. After some time, 3 out of 4 nodes in the cluster went down and CRS refuses to start on these nodes after trying the switch back from MTU 9000 to 1500, reboots, and making sure disk permissions and ownership are correct. The logs are not too helpful (and cryptic) so I'm at a loss and appreciate any ideas or help.

The installation was successful, the RAC was up for a few days while running some tests (including restart of a node). Currently only a single node has everything up and functional, the others are not working. Below are some output that might help:

[root@ucstst11 bin]# ./crsctl check cluster -n ucstst11
**************************************************************
ucstst11:
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4533: Event Manager is online
**************************************************************
[root@ucstst11 bin]# ./crsctl start cluster -n ucstst11
CRS-2672: Attempting to start 'ora.cssd' on 'ucstst11'
CRS-2672: Attempting to start 'ora.diskmon' on 'ucstst11'
CRS-2676: Start of 'ora.diskmon' on 'ucstst11' succeeded
CRS-4404: The following nodes did not reply within the allotted time:
ucstst11

[root@ucstst11 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4533: Event Manager is online

[root@ucstst11 bin]# ./crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.

[root@ucstst11 bin]# oracleasm querydisk -p CRSVOL01
Disk "CRSVOL01" is a valid ASM disk
/dev/sdz1: LABEL="CRSVOL01" TYPE="oracleasm"
/dev/sdcj1: LABEL="CRSVOL01" TYPE="oracleasm"

[root@ucstst11 bin]# ll /dev/sdz1 /dev/sdcj1
brw-rw---- 1 oracle dba 69, 113 Mar 27 19:00 /dev/sdcj1
brw-rw---- 1 oracle dba 65, 145 Mar 27 19:00 /dev/sdz1

[root@ucstst11 bin]# oracleasm querydisk -d CRSVOL01
Disk "CRSVOL01" is a valid ASM disk on device [65, 145]

From the functional node:
[root@ucstst12 bin]# ./crsctl check cluster -all
**************************************************************
ucstst12:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

Cluster verification now hangs when it tries to contact the other nodes.

Please help!