Database Software

Hi,
I installed 11gr2 with ASM on one node (solaris SPARC). Then I added another node to this cluster (via addNode.sh script).
Than got strange error: If my first node is up, second node is started fine and run well. If I shutdown first node - crsd on second node dump to core and fails to restart. I get the same error if I try to start second node when the first one is down.
In the crsd.log I see the following:

---
[ clsdmt][2]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=mskbkp2DBG_CRSD))
2010-03-03 17:31:35.330: [ clsdmt][2]PID for the Process [18669], connkey 1
2010-03-03 17:31:35.331: [ clsdmt][2]Creating PID [18669] file for home /u01/grid/11.2.0 host mskbkp2 bin crs to /u01/grid/11
.2.0/crs/init/
2010-03-03 17:31:35.331: [ clsdmt][2]Writing PID [18669] to the file [u01/grid/11.2.0/crs/init/mskbkp2.pid]
2010-03-03 17:31:35.925: [ default][1] CRS Daemon Starting
2010-03-03 17:31:35.933: [ default][1] ENV Logging level for Module: AGENT 1
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: AGFW 0
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: CLSFRAME 0
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: CLSVER 0
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: CLUCLS 0
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: COMMCRS 0
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: COMMNS 0
2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSAPP 0
2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSCCL 0
2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSCEVT 0
2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSCOMM 1
2010-03-03 17:31:35.936: [ CRSD][1] ENV Debug Level(CRSD): 50
2010-03-03 17:31:35.936: [ CRSD][1] ENV Logging level for Module: CRSD 50
2010-03-03 17:31:35.937: [ CRSD][1] ENV Debug Level(CRSEVT): 0
2010-03-03 17:31:35.937: [ CRSD][1] ENV Logging level for Module: CRSEVT 0
2010-03-03 17:31:35.937: [ CRSD][1] ENV Debug Level(CRSMAIN): 1
2010-03-03 17:31:35.937: [ CRSD][1] ENV Logging level for Module: CRSMAIN 1
2010-03-03 17:31:35.937: [ CRSD][1] ENV Debug Level(CRSOCR): 0
2010-03-03 17:31:35.937: [ CRSD][1] ENV Logging level for Module: CRSOCR 0
2010-03-03 17:31:35.939: [ CRSD][1] ENV Debug Level(CRSPE): 0
2010-03-03 17:31:35.939: [ CRSD][1] ENV Logging level for Module: CRSPE 0
2010-03-03 17:31:35.939: [ CRSD][1] ENV Debug Level(CRSPLACE): 0
2010-03-03 17:31:35.939: [ CRSD][1] ENV Logging level for Module: CRSPLACE 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Debug Level(CRSRES): 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Logging level for Module: CRSRES 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Debug Level(CRSRPT): 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Logging level for Module: CRSRPT 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Debug Level(CRSRTI): 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Logging level for Module: CRSRTI 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Debug Level(CRSSE): 0
2010-03-03 17:31:35.941: [ CRSD][1] ENV Logging level for Module: CRSSE 0
2010-03-03 17:31:35.941: [ CRSD][1] ENV Debug Level(CRSSEC): 0
2010-03-03 17:31:35.941: [ CRSD][1] ENV Logging level for Module: CRSSEC 0
2010-03-03 17:31:35.941: [ CRSD][1] ENV Debug Level(CRSSHARED): 0
2010-03-03 17:31:35.941: [ CRSD][1] ENV Logging level for Module: CRSSHARED 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Debug Level(CRSTIMER): 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Logging level for Module: CRSTIMER 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Debug Level(CRSUI): 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Logging level for Module: CRSUI 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Debug Level(CSSCLNT): 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Logging level for Module: CSSCLNT 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Debug Level(OCRAPI): 1
2010-03-03 17:31:35.942: [ CRSD][1] ENV Logging level for Module: OCRAPI 1
2010-03-03 17:31:35.942: [ CRSD][1] ENV Debug Level(OCRASM): 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Logging level for Module: OCRASM 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Debug Level(OCRCAC): 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Logging level for Module: OCRCAC 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Debug Level(OCRCLI): 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Logging level for Module: OCRCLI 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Debug Level(OCRMAS): 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Logging level for Module: OCRMAS 1
2010-03-03 17:31:35.944: [ CRSD][1] ENV Debug Level(OCRMSG): 1
2010-03-03 17:31:35.944: [ CRSD][1] ENV Logging level for Module: OCRMSG 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Debug Level(OCROSD): 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Logging level for Module: OCROSD 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Debug Level(OCRRAW): 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Logging level for Module: OCRRAW 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Debug Level(OCRSRV): 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Logging level for Module: OCRSRV 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Debug Level(OCRUTL): 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Logging level for Module: OCRUTL 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Debug Level(SuiteTes): 1
2010-03-03 17:31:35.946: [ CRSD][1] ENV Logging level for Module: SuiteTes 1
2010-03-03 17:31:35.946: [ CRSD][1] ENV Debug Level(UiServer): 0
2010-03-03 17:31:35.946: [ CRSD][1] ENV Logging level for Module: UiServer 0
2010-03-03 17:31:35.946: [ CRSMAIN][1] Checking the OCR device
2010-03-03 17:31:35.948: [ CRSMAIN][1] Connecting to the CSS Daemon
2010-03-03 17:31:35.976: [ CRSMAIN][1] Initializing OCR
2010-03-03 17:31:35.981: [ OCRAPI][1]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
2010-03-03 17:31:35.981: [ OCRAPI][1]Check namebufs
2010-03-03 17:31:35.981: [ OCRAPI][1]Finished checking namebufs
2010-03-03 17:31:35.982: [ GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsinet.c : 3232
], original from [clsss.c : 5026]
2010-03-03 17:31:36.036: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:405] gpnp tracelevel 3, component tracelevel 0
2010-03-03 17:31:36.037: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:535] '/u01/grid/11.2.0' in effect as GPnP home base.
2010-03-03 17:31:36.059: [ GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsgpnp0.c : 680
], original from [clsss.c : 5026]
2010-03-03 17:31:36.067: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3867] Init gpnp local security key providers (2)
fatal if both fail
2010-03-03 17:31:36.068: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3870] Init gpnp local security key proveders 1 o
f 2: file wallet (LSKP-FSW)
2010-03-03 17:31:36.068: [ GPnP][1]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/grid/11.2.0/
gpnp/mskbkp2/wallets/peer/

2010-03-03 17:31:36.069: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3892] Init gpnp local security key provider 1 of
2: file wallet (LSKP-FSW) OK
2010-03-03 17:31:36.069: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3898] Init gpnp local security key proveders 2 o
f 2: OLR wallet (LSKP-CLSW-OLR)
[ CLWAL][1]clsw_Initialize: OLR initlevel [30000]
2010-03-03 17:31:36.080: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3921] Init gpnp local security key provider 2 of
2: OLR wallet (LSKP-CLSW-OLR) OK
2010-03-03 17:31:36.081: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;7. (2
providers - fatal if all fail)
2010-03-03 17:31:36.081: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:36.152: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:36.152: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:36.152: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:36.172: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;4. (2
providers - fatal if all fail)
2010-03-03 17:31:36.172: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:36.238: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:36.239: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:36.239: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:36.239: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:840] GPnP client pid=18669, tl=3, f=0
2010-03-03 17:31:36.541: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1735], ret gipcretSuccess
(0)
2010-03-03 17:31:36.552: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSucces
s (0)
2010-03-03 17:31:36.771: [ OCRRAW][1]proprioo: for disk 0 (+DR2_BIN), id match (1), total id sets, (1) need recover (0), my v
otes (0), total votes (0), commit_lsn (9), lsn (9)
2010-03-03 17:31:36.771: [ OCRRAW][1]proprioo: my id set: (833490748, 1028247821, 0, 0, 0)
2010-03-03 17:31:36.772: [ OCRRAW][1]proprioo: 1st set: (833490748, 1028247821, 0, 0, 0)
2010-03-03 17:31:36.772: [ OCRRAW][1]proprioo: 2nd set: (0, 0, 0, 0, 0)
2010-03-03 17:31:36.830: [ OCRSRV][1]th_init: Successfully retrieved CSS misscount [31].
2010-03-03 17:31:36.830: [ OCRSRV][1]th_init: Successfully query CLSS mode [3].
[ OCRMAS][20]th_calc_av:5': Rturn persisted AV [186646784] [11.2.0.1.0]
2010-03-03 17:31:36.920: [ OCRSRV][20]th_not_master_change: Master change callback not registered
2010-03-03 17:31:36.920: [ OCRMAS][20]th_master:12: I AM THE NEW OCR MASTER at incar 1. Node Number 2
2010-03-03 17:31:37.134: [ OCRASM][20]proprasmo: ASM cache size is [5MB]
2010-03-03 17:31:37.142: [ OCRASM][20]proprasmo: ASM cache [5MB] enabled for disk group [DR2_BIN].
2010-03-03 17:31:37.155: [ OCRRAW][20]proprioo: for disk 0 (+DR2_BIN), id match (1), total id sets, (1) need recover (0), my
votes (0), total votes (0), commit_lsn (9), lsn (9)
2010-03-03 17:31:37.155: [ OCRRAW][20]proprioo: my id set: (833490748, 1028247821, 0, 0, 0)
2010-03-03 17:31:37.155: [ OCRRAW][20]proprioo: 1st set: (833490748, 1028247821, 0, 0, 0)
2010-03-03 17:31:37.155: [ OCRRAW][20]proprioo: 2nd set: (0, 0, 0, 0, 0)
2010-03-03 17:31:37.214: [ OCRMAS][20]proath_master:18: Spawned connection mgr thread
2010-03-03 17:31:37.214: [ OCRMAS][20]proath_master:20: Spawned upgrade thread
2010-03-03 17:31:37.214: [ OCRMAS][20]th_master:19.1: Wake up upgrade thread
2010-03-03 17:31:37.216: [ OCRSRV][1]th_snap_local_spawn: Inside snap local spawn. host is [mskbkp2]
2010-03-03 17:31:37.219: [ CRSMAIN][1] Running as user: root
2010-03-03 17:31:37.219: [ CRSMAIN][1] CRSD running as the Privileged user

2010-03-03 17:31:37.219: [ CLSVER][1] Static Version 11.2.0.1.0
2010-03-03 17:31:37.226: [ OCRMAS][20]th_master:1': Recvd pubdata event from node [2]
2010-03-03 17:31:37.227: [ OCRMAS][20]th_master:2': Recvd pubdata event for self. Do nothing.
2010-03-03 17:31:37.227: [ CLSVER][1] Daemon version: 11.2.0.1.0 Software version: 11.2.0.1.0
2010-03-03 17:31:37.231: [ CLSVER][1] Active Version from OCR:11.2.0.1.0
2010-03-03 17:31:37.232: [ CLSVER][1] Active Version and Software Version are same
2010-03-03 17:31:37.232: [ CLSVER][1] Active Version changed to 11.2.0.1.0
2010-03-03 17:31:37.232: [ OCRSRV][1]th_reg_master_change: Master change callback registered
2010-03-03 17:31:37.232: [ OCRAPI][1]a_reg_master_change: Registered master change callback
2010-03-03 17:31:37.232: [ OCRSRV][1]th_not_master_change: Invoking master change callback. Master [2] Inc [1]
2010-03-03 17:31:37.232: [ OCRAPI][1]a_reg_master_change: Notified master change
2010-03-03 17:31:37.232: [ CRSMAIN][1] CAA Node Group Pri Data size: 128
2010-03-03 17:31:37.233: [ CRSMAIN][1] CAA Node Group Pub Data size: 128
2010-03-03 17:31:37.247: [ CRSMAIN][1] Getting private data of booted nodes
2010-03-03 17:31:37.247: [ CRSMAIN][1] Checking for booted param on nodenum: 2
2010-03-03 17:31:37.306: [ CLSE][1]clse_get_auth_loc: Returning default authloc: /u01/grid/11.2.0/auth/crs/mskbkp2
2010-03-03 17:31:37.306: [ CRSMAIN][1] Using Authorizer location: /u01/grid/11.2.0/auth/crs/mskbkp2
2010-03-03 17:31:37.314: [ OCRSRV][23]th_upgrade: Starting upgrade calculation
2010-03-03 17:31:37.364: [ CLSCLU][1]clsclu_init: rc 0
2010-03-03 17:31:37.381: [ OCRSRV][23]th_upgrade:10.1 AV [186646784]. State [11]. Already upgraded.Updated global data to the
crs version group. Return [0]
2010-03-03 17:31:37.385: [ CRSMAIN][1] Initializing RTI
2010-03-03 17:31:37.433: [ CRSMAIN][1] Initializing ResouceStateListener
2010-03-03 17:31:37.433: [CRSTIMER][37] Timer Thread Starting.
2010-03-03 17:31:37.433: [ CRSMAIN][1] Initializing EVMMgr
2010-03-03 17:31:37.446: [ CRSMAIN][1] Initializing ResourceMap Map
2010-03-03 17:31:37.461: [ CRSMAIN][1] Subscribing to EVM events for apps
2010-03-03 17:31:37.504: [ CRSMAIN][1] CRSD locked during state recovery, please wait.
2010-03-03 17:31:37.516: [ CRSMAIN][1] CRSD recovered, unlocked.
2010-03-03 17:31:37.525: [ default][1]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
2010-03-03 17:31:37.525: [ default][1]Check namebufs
2010-03-03 17:31:37.525: [ default][1]Finished checking namebufs
2010-03-03 17:31:37.526: [ GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsinet.c : 3232
], original from [clsss.c : 5026]
2010-03-03 17:31:37.569: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:405] gpnp tracelevel 3, component tracelevel 0
2010-03-03 17:31:37.569: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:535] '/u01/grid/11.2.0' in effect as GPnP home base.
2010-03-03 17:31:37.587: [ GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsgpnp0.c : 680
], original from [clsss.c : 5026]
2010-03-03 17:31:37.595: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3867] Init gpnp local security key providers (2)
fatal if both fail
2010-03-03 17:31:37.595: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3870] Init gpnp local security key proveders 1 o
f 2: file wallet (LSKP-FSW)
2010-03-03 17:31:37.596: [ GPnP][1]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/grid/11.2.0/
gpnp/mskbkp2/wallets/peer/

2010-03-03 17:31:37.596: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3892] Init gpnp local security key provider 1 of
2: file wallet (LSKP-FSW) OK
2010-03-03 17:31:37.596: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3898] Init gpnp local security key proveders 2 o
f 2: OLR wallet (LSKP-CLSW-OLR)
[ CLWAL][1]clsw_Initialize: OLR initlevel [30000]
2010-03-03 17:31:37.607: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3921] Init gpnp local security key provider 2 of
2: OLR wallet (LSKP-CLSW-OLR) OK
2010-03-03 17:31:37.607: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;7. (2
providers - fatal if all fail)
2010-03-03 17:31:37.607: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:37.673: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:37.673: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:37.673: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:37.690: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;4. (2
providers - fatal if all fail)
2010-03-03 17:31:37.690: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:37.754: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:37.754: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:37.754: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:37.755: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:840] GPnP client pid=18669, tl=3, f=0
2010-03-03 17:31:37.806: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1735], ret gipcretSuccess
(0)
2010-03-03 17:31:37.817: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSucces
s (0)
2010-03-03 17:31:37.822: [ CRSMAIN][1] CRSD listening on 10 style E2E port (ADDRESS=(PROTOCOL=tcp)(HOST=172.31.25.112)(PORT=38
983))
2010-03-03 17:31:37.835: [ CRSMAIN][1] Starting Threads
2010-03-03 17:31:37.858: [ CLSE][1]clse_get_auth_loc: Returning default authloc: /u01/grid/11.2.0/auth/crs/mskbkp2
2010-03-03 17:31:37.858: [ CRSD][1] AuthLoc /u01/grid/11.2.0/auth/crs/mskbkp2
2010-03-03 17:31:37.859: [ CRSD][1] PE active version: 11.2.0.1.0
2010-03-03 17:31:37.859: [ CRSD][1] PE Engine: NEW
2010-03-03 17:31:37.859: [ CRSD][1] Using OCR batch ops : ENABLED
2010-03-03 17:31:37.860: [ CRSMAIN][1] Initializing Node Down Monitor
2010-03-03 17:31:37.860: [ CRSMAIN][1] CRS Daemon Started.
2010-03-03 17:31:37.860: [ CRSD][1] Connecting to the CSS Daemon
2010-03-03 17:31:37.861: [ CRSD][1] Local CSS Node Number is: 2
2010-03-03 17:31:37.863: [ CRSD][1] Local Css Node Name is: mskbkp2
2010-03-03 17:31:37.863: [ CRSD][1] CRSDPersonality initialized
2010-03-03 17:31:37.864: [ CRSMAIN][1] Process member data: CRSD:mskbkp2
2010-03-03 17:31:37.864: [ CRSD][1][F-ALGO] getIpcPath returning (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_IPC_SOCKET_11))
2010-03-03 17:31:37.865: [CLSFRAME][1] Inited lsf context 102b3f670
2010-03-03 17:31:37.865: [CLSFRAME][1] Initing CLS Framework messaging
2010-03-03 17:31:37.869: [ CRSD][1][F-ALGO] getIpcPath returning (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_IPC_SOCKET_11))
2010-03-03 17:31:37.873: [UiServer][1] UI Comms initalize() 1
2010-03-03 17:31:37.873: [CLSFRAME][1] New Framework state: 2
2010-03-03 17:31:37.873: [CLSFRAME][1] M2M is starting...
2010-03-03 17:31:37.873: [ CRSCCL][1]clsCclInit called by process: 18669

2010-03-03 17:31:37.885: [ CRSCCL][1]USING CLSC ============
2010-03-03 17:31:37.895: [ default][1]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
2010-03-03 17:31:37.895: [ default][1]Check namebufs
2010-03-03 17:31:37.895: [ default][1]Finished checking namebufs
2010-03-03 17:31:37.950: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:405] gpnp tracelevel 3, component tracelevel 0
2010-03-03 17:31:37.951: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:535] '/u01/grid/11.2.0' in effect as GPnP home base.
2010-03-03 17:31:37.970: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3867] Init gpnp local security key providers (2)
fatal if both fail
2010-03-03 17:31:37.970: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3870] Init gpnp local security key proveders 1 o
f 2: file wallet (LSKP-FSW)
2010-03-03 17:31:37.970: [ GPnP][1]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/grid/11.2.0/
gpnp/mskbkp2/wallets/peer/

2010-03-03 17:31:37.970: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3892] Init gpnp local security key provider 1 of
2: file wallet (LSKP-FSW) OK
2010-03-03 17:31:37.970: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3898] Init gpnp local security key proveders 2 o
f 2: OLR wallet (LSKP-CLSW-OLR)
[ CLWAL][1]clsw_Initialize: OLR initlevel [70000]
2010-03-03 17:31:37.980: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3921] Init gpnp local security key provider 2 of
2: OLR wallet (LSKP-CLSW-OLR) OK
2010-03-03 17:31:37.980: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;7. (2
providers - fatal if all fail)
2010-03-03 17:31:37.980: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:38.049: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:38.049: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:38.049: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:38.068: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;4. (2
providers - fatal if all fail)
2010-03-03 17:31:38.068: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:38.134: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:38.135: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:38.135: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:38.135: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:840] GPnP client pid=18669, tl=3, f=3
2010-03-03 17:31:38.184: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1735], ret gipcretSuccess
(0)
2010-03-03 17:31:38.194: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSucces
s (0)
2010-03-03 17:31:38.200: [ CRSCCL][1]Listening endpoint created sucessfully @ (ADDRESS=(PROTOCOL=tcp)(DEV=54)(HOST=172.31.25.
112)(PORT=38984)).con = 10359a0d0
2010-03-03 17:31:38.209: [ CRSCCL][48]CSS Group Registration complete.

2010-03-03 17:31:38.213: [ CRSCCL][48]cclGetMemberData called
2010-03-03 17:31:38.215: [ CRSCCL][48]Obtained first membership map.

2010-03-03 17:31:38.215: [ CRSCCL][48]Dumping member data ------------------
2010-03-03 17:31:38.215: [ CRSCCL][48]Member (2, 603412550) on node port=.
2010-03-03 17:31:38.216: [ CRSCCL][48]Done ------------------
2010-03-03 17:31:38.216: [ CRSCCL][48]Waiting for reconfigs
2010-03-03 17:31:38.216: [ CRSCCL][49]cclCommunicationHandler started.
2010-03-03 17:31:38.220: [ CRSCOMM][1] Ipc: m_pClscCtx=1020c4850m_pUgblm=1035b2a50
2010-03-03 17:31:38.220: [ CRSCOMM][1] Ipc: Starting send thread
2010-03-03 17:31:38.220: [ CRSCOMM][1] IpcL: Listener instantiated for: (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_IPC_SOCKET_11))
2010-03-03 17:31:38.221: [ CRSCOMM][52] Ipc: sendWork thread started.
2010-03-03 17:31:38.222: [ CRSCOMM][1] IpcL: Listener started listening.
2010-03-03 17:31:38.223: [ CRSCOMM][53] IpcL: thread started listening
2010-03-03 17:31:38.223: [CLSFRAME][1] Starting thread model named: AgfwProxySrvTM
2010-03-03 17:31:38.224: [CLSFRAME][1] Starting thread model named: OcrModuleTM
2010-03-03 17:31:38.225: [CLSFRAME][1] Starting thread model named: PolicyEngineTM
2010-03-03 17:31:38.225: [CLSFRAME][1] Starting thread model named: SharedThreadTM
2010-03-03 17:31:38.225: [CLSFRAME][1] Starting thread model named: UiServerTM
2010-03-03 17:31:38.225: [CLSFRAME][1] New Framework state: 3
2010-03-03 17:31:38.227: [ CRSRPT][62] Enabled
2010-03-03 17:31:38.228: [ CRSPE][61] PE Role|State Update: old role [INVALID] new [INVALID]; old state [Not yet initialized
] new [Enabling: waiting for role]
2010-03-03 17:31:38.229: [ CRSSE][62] Master Change Event; New Master Node ID:2 This Node's ID:2
2010-03-03 17:31:38.230: [ CRSPE][61] PE Role|State Update: old role [INVALID] new [MASTER]; old state [Enabling: waiting fo
r role] new [Configuring]
2010-03-03 17:31:38.230: [ CRSPE][61] PE MASTER NAME: mskbkp2
2010-03-03 17:31:38.230: [ CRSPE][61] Starting to read configuration
2010-03-03 17:31:38.260: [ CRSPE][61] Reading (2) servers
2010-03-03 17:31:38.459: [ CRSPE][61] DM: set global config version to: 150
2010-03-03 17:31:38.459: [ CRSPE][61] DM: set pool freeze timeout to: 60000
2010-03-03 17:31:38.459: [ CRSPE][61] DM: Set event seq number to: 13900000
2010-03-03 17:31:38.459: [ CRSPE][61] DM: Set threshold event seq number to: 13980000
2010-03-03 17:31:38.460: [ CRSPE][61] Sent request to write event sequence number 14000000 to repository
2010-03-03 17:31:38.483: [ CRSPE][61] Wrote new event sequence to repository
2010-03-03 17:31:38.568: [ CRSPE][61] Reading (15) types
2010-03-03 17:31:38.593: [ CRSPE][61] Reading (3) server pools
2010-03-03 17:31:38.624: [ CRSPE][61] Reading (21) resources
2010-03-03 17:31:39.987: [ CRSPE][61] Finished reading configuration. Parsing...
2010-03-03 17:31:39.988: [ CRSPE][61] Parsing resource types...
2010-03-03 17:31:40.030: [ CRSD][61] Initializing the config version for type ora.asm.type to: 1
2010-03-03 17:31:40.035: [ CRSD][61] Initializing the config version for type ora.cluster_resource.type to: 1
2010-03-03 17:31:40.040: [ CRSD][61] Initializing the config version for type ora.cluster_vip.type to: 1
2010-03-03 17:31:40.044: [ CRSD][61] Initializing the config version for type ora.cluster_vip_net1.type to: 1
2010-03-03 17:31:40.048: [ CRSD][61] Dump State Starting ...
2010-03-03 17:31:40.048: [ CRSD][61] State Dump for RTILock
2010-03-03 17:31:40.048: [ CRSD][61] Lock State List is busy, skipping ..
2010-03-03 17:31:40.048: [ CRSD][61] State Dump for Timer
2010-03-03 17:31:40.049: [ CRSD][61] Timer map size=0
2010-03-03 17:31:40.049: [ CRSPE][61] Dumping PE Data Model...:DM has [0 resources][0 types][0 servers][0 spools]
------------- RESOURCES:

------------- TYPES:

------------- SERVERS:

------------- SERVER POOLS:

2010-03-03 17:31:40.049: [ CRSPE][61] Dumping ICE contents...:ICE operation count: 0
2010-03-03 17:31:40.049: [ CRSD][61] Dump State Done.
---

I guess that there is some thing wrong in configuration, but cannot find out what.

Any help would be appreciated.

Thanks