Single infiniband dual port card configuration in RAC
I am dealing with a site that is using a single dual port Infiniband card per node. They want to run the storage array conneciton through one port and the interconnect using RDS through the other on each node. They are using a single infiniband switch (Mullinox). This is 10.2.0.4 on RH4 ES usinf 1.5.2 OFED.
We started with just the storage configured through a multi-port connection to the switch. It seemed to be working ok (other than one of the three nodes kept rebooting, whcih seems to be a hardware issue) when we then configured to remove the multiport storage configuration and use the second port for the interconnect (not relinking for RDS yet, just using normal UDP protocol) after a couple of hours we got reboots on all servers due to ocfs2 timeouts. I suggested changing the o2cb O@CB_HEARTBEAT_THRESHOLD to 60000 from 30000.
A SUN document seemed to indicate that with multiple ports on a single card they should not be attached to the same switch but should use different switches, however, the logic for this suggestion was not given.
Has anyone used a similar configuration? Any tips?
We are looking at returning to just having the storage on Infiniband and going back to 10gb Ethernet for the interconnect.
Mike