Database Software

Hello,
Whilst testing Dataguard with FSFO, I seem to have managed to achieve a split brain situation where 2 databases in the Data Guard configuration were considered as primary databases and both were available to receive client connections. I dont understand why this happened and I'm looking for some assurance that this is not a bug in Oracle.

My Data Guard Configuration is as follows:-

1 primary database (DBa)
3 physical standby databases (DBb, DBc, DBd)

All databases are single instance, i.e no RAC, and are running Oracle 11g (11.1.0.7) on RHEL 5.3

DBb is the Fast Start Failover Target for DBa. The FSFO observer process is running on a stand-alone server called OBS1,

To simulate a 'data centre disaster' i did the following :-

1) Kill the SMON processes on the servers running DBa and DBb (Note I did not kill the Observer process)
2) From DGMGRL on the server running DBc issue the following commands :-

DGMGRL> disable fast_start failover force (Without doing this I could not issue the subsequent failover command)
DGMGRL> failover to DBc ;

This worked as expected and DBc was established as the new primary database in the configuration. DBd continued to function correctly as a stand db. Subsequent client connections were routed to DBc as expected.

3) I then attempted to simulate the two failed databases DBa and DBb rejoining the configuration. Firstly I put DBa into MOUNT status using the STARTUP MOUNT command from the SQLPLUS command line.

4) Before I did anything with DBb, the Observer process that was still running on OBS1, detected that DBa was 'active' again and OPENed the database. In doing this it took no notice of the fact that DBc was already open and acting as the primary database in the configuration. The result of this was that two databases - DBa and DBc in the configuration were in an an OPEN state and acting as a primary database i.e Split Brain. The TNSNAMES.ora configuration on the Oracle machines meant that it was now perfectly possible for client connection to be spread over both these machines.

I am very concerned as to why Oracle allowed the above situation to happen. Was my test unreasonable or should Oracle have detected that DBc was the new primary database after I attmepted to restart DBa in MOUNT state?? I now understand that if I had also issued the STOP OBSERVER command in DGMGRL, after issuing the FAILOVER ro DBc command, then the FSFO Observer could not have OPENed DBa once DBc had become the primary database, so is this the only mechanism that must be used in the above scenario to prevent a Split brain ?

Any advice would be greatly appreciated.

Thanks,
Shaun