Hi pros,
I have a problem while replacing a new controller. My storage 6180 have 2 controller A ( OK ) - B ( failed ). I power off storage than replacing controller B by a new controller.
Now, i cannot access to controller A through serial console. F/W controller A: 7.60.x.x
I can access to controller B through serial console and CAM which show all drive is incompatible and unassigned, no volumes, no virtual disk. F/W controller B: 7.80.x.x
Controller B message shows Failed to open Inter-Controller Communication Channels!
I searched google and MOS but still have not find any solution for this problem. I will appreciate for any helps.
This is log file from serial controller B
09/22/02-12:03:37 (tRAID): WARN: Failed to open Inter-Controller Communication Channels!
09/22/02-12:03:37 (tRAID): NOTE: LockMgr Role is Master
09/22/02-12:03:38 (utlTimer): NOTE: fcnChannelReport ==> 0 1
09/22/02-12:03:40 (tHckReset): NOTE: HealthCheck: Alt Ctl: 1 Reset_Failure, state: 0 Start
0x8f9084 (tNetCfgInit): miiPhyInit check cable connection
09/22/02-12:03:49 (tNetCfgInit): NOTE: eth1: LinkDown event
09/22/02-12:03:49 (tNetCfgInit): NOTE: Network Ready
09/22/02-12:04:07 (tRAID): NOTE: WWN baseName 00040080-e5185e0e (valid==>SigMatch)
09/22/02-12:04:07 (tRAID): NOTE: spmEarlyData: No data available
09/22/02-12:04:08 (tRAID): SOD: Pre-Initialization Phase Complete
09/22/02-12:04:09 (utlTimer): NOTE: fcnChannelReport ==> 0 1 ~2 -3
09/22/02-12:04:14 (utlTimer): NOTE: fcnChannelReport ==> 0 1 ~2 =3
09/22/02-12:04:39 (utlTimer): NOTE: fcnChannelReport ==> 0 1 2 =3
09/22/02-12:04:40 (tHckReset): NOTE: HealthCheck: Alt Ctl: 4 Norun_Failure, state: 0 Start
09/22/02-12:04:40 (tHckReset): NOTE: HealthCheckManager: Notify Event 6 Ctl_Not_Running
09/22/02-12:04:40 (cmgrEvent): WARN: Alt Ctl Reboot:
Reboot CompID: 0x407
Reboot reason: 0x6
Reboot reason extra: 0x0
09/22/02-12:04:40 (cmgrEvent): NOTE: holding alt ctl in reset
09/22/02-12:04:40 (cmgrEvent): NOTE: HealthCheck: Alt Ctl: 1 Reset_Failure, state: 0 Start
09/22/02-12:04:42 (utlTimer): NOTE: fcnChannelReport ==> +0 1 2 =3
09/22/02-12:04:43 (utlTimer): NOTE: fcnChannelReport ==> 0 1 2 =3
09/22/02-12:04:46 (tRAID): WARN: pcm::CapabilityManager::validateAdoption exception (DbmWriteException: recType: 85, status: -12)
09/22/02-12:04:46 (tRAID): WARN: dbm::RWFileSystem::initialize: Exception caught, ConstructorIOException: -16
09/22/02-12:04:46 (tRAID): ERROR: In PersistenceManager::initialize: catch DbmNoFileSystemException: recType: 84
09/22/02-12:04:46 (tRAID): ERROR: ADM Load Reservations failed with error (5) Exception
09/22/02-12:04:46 (tRAID): NOTE: ACS: Icon ping to alternate failed: -2, resp: 0
09/22/02-12:04:46 (tRAID): NOTE: ACS: autoCodeSync(): Process start. Comm Mode: 0, Status: 0
09/22/02-12:04:46 (tRAID): WARN: ACS: autoCodeSync(): Skipped since alt not communicating.
09/22/02-12:04:46 (tRAID): WARN: DbmNoFileSystemException: recType: 59 Line 1819 File cmgrControllerMgr.cc
09/22/02-12:04:46 (tRAID): WARN: DbmNoFileSystemException: recType: 59 Line 1819 File cmgrControllerMgr.cc
09/22/02-12:04:46 (tRAID): SOD: Code Synchronization Initialization Phase Complete
09/22/02-12:04:46 (sntpEvent): NOTE: sntpEventHandler: VNI_GET_SYNC_TIME failed
09/22/02-12:04:47 (tRAID): WARN: USM Exception caught in processUsmHeader - DbmNoFileSystemException: recType: 30
09/22/02-12:04:47 (tRAID): WARN: USM Error allocating UsmStableStorageHeader in processUsmHeader() - DbmNoFileSystemException: recType: 30
09/22/02-12:04:47 (tRAID): NOTE: SOD failure in evf::VolumeCfgManager::initialize
09/22/02-12:04:47 (tRAID): NOTE: DbmNoFileSystemException in evf::VolExtentMgr::initialize
09/22/02-12:04:47 (tRAID): NOTE: DbmNoFileSystemException in safe::initialize
09/22/02-12:04:47 (tRAID): ERROR: Caught DbmNoFileSystemException: recType: 95 in initialize
09/22/02-12:04:47 (tRAID): WARN: edrSOD: No Config File System
09/22/02-12:04:48 (tRAID): WARN: snrProcessDatabase: No File System Found
09/22/02-12:04:48 (tRAID): WARN: spm::SPMManager::initialize NoFileSystem
09/22/02-12:04:48 (tRAID): NOTE: fcn: Peering Disabled (Alt Unavailable)
09/22/02-12:04:48 (tRAID): NOTE: ion: Peering Disabled (Alt Unavailable)
09/22/02-12:04:48 (tRAID): WARN: Error in updating in memory dq cfg!
09/22/02-12:04:48 (tRAID): WARN: Caught DbmNoFileSystemException: recType: 65 in fbm::initialize
09/22/02-12:04:48 (tRAID): WARN: MelMgr::initialize FAILED: Exception Line 2197 File mlmMelMgr.cc
09/22/02-12:04:48 (tRAID): WARN: readDatabaseSyslogConfig: Caught Exception DbmNoFileSystemException: recType: 74 Line 420 File mlmSyslogMgr.cc
09/22/02-12:04:48 (tRAID): WARN: Syslog database access failed: Exception at Line 324 File mlmSyslogMgr.cc
09/22/02-12:04:49 (tRAID): WARN: Unable to intialize mirror device
09/22/02-12:04:49 (tRAID): NOTE: CacheMgr::cacheOpenMirrorDevice:: mirror device 0xfffffff
09/22/02-12:04:49 (tRAID): WARN: CacheStore::read(,): Exception DbmNoFileSystemException: recType: 24
09/22/02-12:04:49 (tRAID): WARN: CCM: readAndValidateCacheStore() caught CacheStoreDataException
09/22/02-12:04:49 (tRAID): NOTE: CCM: readAndValidateCacheStore() recovering
09/22/02-12:04:49 (tRAID): NOTE: CCM: readAndValidateCacheStore() partitioning for no mirroring
09/22/02-12:04:49 (tRAID): WARN: CacheStoreExt::read(,): Exception DbmNoFileSystemException: recType: 82
09/22/02-12:04:49 (tRAID): WARN: CCM: readAndValidateCacheStore() cacheStoreExt read exception
09/22/02-12:04:49 (tRAID): NOTE: CCM: readAndValidateCacheStore() initializes with default values
09/22/02-12:04:49 (tRAID): NOTE: CCM: Changing default demand cache flush values
09/22/02-12:04:49 (tRAID): NOTE: CCM: validateCacheMem() cache memory is invalid
09/22/02-12:04:49 (tRAID): NOTE: CCM: validateCacheMem() Initializing my partition
09/22/02-12:04:50 (tRAID): NOTE: CacheStore::write(): Exception DbmNoFileSystemException: recType: 24
09/22/02-12:04:50 (tRAID): WARN: CCM: initialize() caught exception(2) CacheStoreDataException
09/22/02-12:04:50 (tRAID): NOTE: CCM: sodClearMOSIntentsAlt(), failure clearing MOS intents on alt
09/22/02-12:04:52 (tRAID): WARN: Exception caught in presMgrInit - DbmNoFileSystemException: recType: 12
09/22/02-12:04:52 (tRAID): NOTE: DbmNoFileSystemException in vdm::CrushDriveManager::initialize
09/22/02-12:04:52 (tRAID): NOTE: DbmNoFileSystemException in vdm::DriveManager::initialize
09/22/02-12:04:52 (tRAID): NOTE: DbmNoFileSystemException in vdm::VolumeGroupManager::initialize
09/22/02-12:04:52 (tRAID): NOTE: DbmNoFileSystemException in vdm::DacstoreVolManager::initialize
09/22/02-12:04:52 (tRAID): NOTE: DbmNoFileSystemException in vdm::PieceManager::initialize
09/22/02-12:04:52 (tRAID): NOTE: DbmNoFileSystemException in vdm::CrushStripeManager::initialize
09/22/02-12:04:52 (tRAID): NOTE: DbmNoFileSystemException in vdm::exop::ExclOpManager::initialize
09/22/02-12:04:52 (tRAID): WARN: CCM: initComplete() - isRestoreInProgressAlt caught IconSendInfeasibleException Error
09/22/02-12:04:52 (tRAID): NOTE: CacheStore::write(): Exception DbmNoFileSystemException: recType: 24
09/22/02-12:04:52 (tRAID): WARN: CCM: initComplete() caught(2) CacheStoreDataException
09/22/02-12:04:53 (tRAID): NOTE: DiagVolManager::initialize: Exception - Alt controller not ready
09/22/02-12:04:53 (tRAID): ERROR: dbm: SubRecInterface::save caught DbmNoFileSystemException: recType: 58 - recType: 58, txn: 0x03b0de7c
09/22/02-12:04:53 (tRAID): ERROR: dbm: SubRecInterface::save caught DbmNoFileSystemException: recType: 58 - recType: 58, txn: 0x03b0de7c
09/22/02-12:04:53 (tRAID): WARN: Caught DbmNoFileSystemException: recType: 57 in sam::StorageArrayManager::initialize
09/22/02-12:04:55 (tRAID): SOD: Initialization Phase Complete
09/22/02-12:04:53 (ProcessHandlers): WARN: Exception caught in WWNStorage iosReleased: DbmNoFileSystemException: recType: 9
09/22/02-12:04:53 (ProcessHandlers): WARN: dbm::WWNStorage::iosReleased() FAILED.
==============================================
Title: Disk Array Controller
Copyright 2008-2011 LSI Logic Corporation, All Rights Reserved.
Name: RC
Version: 07.80.51.10
Date: 10/03/2011
Time: 15:49:14 CDT
Models: 4980 4981 4985 4988
Manager: devmgr.v1080api06.Manager
==============================================
09/22/02-12:04:56 (tRAID): sodMain Normal sequence finished, elapsed time = 112 seconds
09/22/02-12:04:56 (tRAID): sodMain complete
09/22/02-12:05:06 (ProcessHandlers): WARN: Drive 0x10000 will be marked incompatible as DSM detected error: 12
09/22/02-12:05:06 (ProcessHandlers): ERROR: dbm: SubRecInterface::read caught DbmNoFileSystemException: recType: 84 - recType: 84, txn: 0x03b0df9c
09/22/02-12:05:08 (utlTimer): WARN: Extended Link Down Timeout on channel 3
09/22/02-12:05:10 (ProcessHandlers): WARN: DbmNoFileSystemException: recType: 35 Line 6404 File cmgrControllerMgr.cc
09/22/02-12:05:10 (ProcessHandlers): WARN: DbmNoFileSystemException: recType: 35 Line 5444 File cmgrControllerMgr.cc
09/22/02-12:05:10 (ProcessHandlers): WARN: CCM: backupStorageAvailableAlt() caught IconSendInfeasibleException Error
09/22/02-12:05:10 (ProcessHandlers): NOTE: SYMbol available
09/22/02-12:05:12 (ProcessHandlers): SOD: sodComplete Notification Complete
09/22/02-12:05:10 (PersistentRestore): NOTE: PSTOR: PstorRecordManager::readRecord data block not found
09/22/02-12:05:10 (PersistentRestore): WARN: IOManager::readBackupStatus - Pstor record does not exsit
09/22/02-12:05:10 (PersistentRestore): NOTE: IOManager::getBackupDataSize - read to pstor failed
09/22/02-12:05:10 (PersistentRestore): WARN: ddcDq & ddcTrace restore abandoned: nothing to recover
09/22/02-12:05:10 (PersistentRestore): WARN: PSTOR: PstorRecordMgr: removeRecord failed
09/22/02-12:05:10 (PersistentRestore): WARN: IOManager::readBackupStatus - Pstor record does not exist
09/22/02-12:05:10 (PersistentRestore): NOTE: DDC Restore Failed
09/22/02-12:05:10 (PersistentRestore): NOTE: IOManager::restoreData - m_DataSize:0x500000, m_StartAddress:0x3b0e320
09/22/02-12:05:10 (PersistentRestore): NOTE: IOManager::restoreData - Successful
09/22/02-12:05:10 (PersistentRestore): NOTE: ncb::IOManager::restoreData - Successful
09/22/02-12:05:10 (PersistentRestore): NOTE: DQ Restore Completed
09/22/02-12:06:10 (ProcessEvents): WARN: RAIDVolumeManager::updateAltMountStates, caught IconSendInfeasibleException Error
09/22/02-12:18:25 (utlTimer): NOTE: fcnChannelReport ==> 0 +1 2 =3
09/22/02-12:18:32 (utlTimer): NOTE: fcnChannelReport ==> 0 -1 2 =3
09/22/02-12:18:35 (IOSched): NOTE: Extended Link Down ==> Chan 1
09/22/02-12:18:42 (utlTimer): NOTE: fcnChannelReport ==> 0 +1 2 =3
09/22/02-12:21:23 (utlTimer): NOTE: fcnChannelReport ==> +0 1 2 =3
09/22/02-12:21:38 (utlTimer): NOTE: fcnChannelReport ==> +0 1 2 =3
09/22/02-12:22:37 (utlTimer): NOTE: fcnChannelReport ==> +0 1 2 =3
09/22/02-12:22:55 (tSubSys): NOTE: HealthCheck: Alternate controller removal
09/22/02-12:22:55 (tSubSys): NOTE: HealthCheckManager: Notify Event 5 Ctl_Removed
09/22/02-12:22:56 (utlTimer): NOTE: fcnChannelReport ==> -0 1 2 =3
09/22/02-12:23:00 (IOSched): NOTE: Extended Link Down ==> Chan 0
09/22/02-12:23:01 (utlTimer): NOTE: fcnChannelReport ==> =0 1 2 =3
09/22/02-12:23:24 (IOSched): WARN: Extended Link Down is over on channel 0 - lasted 29 seconds
09/22/02-12:23:25 (utlTimer): NOTE: fcnChannelReport ==> +0 1 2 =3