Database Software

hi,
timesten 7.05, host1:134.133.1.71 tt1,host2:134.133.1.70 tt2;
oracle db:RAC
Host2(tt2) drop in the cache group after group create cache after a while, timesten offline the;
------------------------------------------------
error_log：
23:12:14.40 Err : REP: 10422: HB_OCS:transmitter.c(3048): TT16121: Failed to flush transaction queue. Restarting log read loo
p
23:12:14.40 Err : REP: 10422: HB_OCS:transmitter.c(3048): TT16121: Failed to flush transaction queue. Restarting log read loo
p
23:12:14.40 Warn: REP: 10422: HB_OCS:receiver.c(1931): TT16060: Failed to read data from the network. TimesTen replication ag
ent is stopping
23:12:14.51 Warn: REP: 10422: HB_OCS:receiver.c(1931): TT16060: Failed to read data from the network. TimesTen replication ag
ent is stopping
23:13:27.09 Err : ORA: 9592: ora-9592-0009-raStuff09836: Unexpected row count. Expecting 1. Got 0.
23:49:25.50 Warn: REP: 9651: HB_OCS:receiver.c(1931): TT16060: Failed to read data from the network. select() timed out
23:49:30.01 Warn: REP: 9651: HB_OCS:transmitter.c(5028): TT16060: Failed to read data from the network. select() timed out
23:49:30.01 Err : REP: 9651: HB_OCS:transmitter.c(3048): TT16121: Failed to flush transaction queue. Restarting log read loo
p
23:49:53.54 Warn: REP: 9651: HB_OCS:receiver.c(1931): TT16060: Failed to read data from the network. TimesTen replication ag
ent is stopping
23:54:40.14 Warn: : 6151: 7759 exited while connected to data store '/tt_data/hb_ocs/hb_ocs' shm 757334180 count=10
23:54:40.27 Warn: : 6151: 7546 exited while connected to data store '/tt_data/hb_ocs/hb_ocs' shm 757334180 count=7
23:54:40.38 Warn: : 6151: 6137 exited while connected to data store '/tt_data/hb_ocs/hb_ocs' shm 757334180 count=7
23:54:40.56 Warn: : 6151: 5029 exited while connected to data store '/tt_data/hb_ocs/hb_ocs' shm 757334180 count=10
23:54:40.67 Warn: : 6151: 6367 exited while connected to data store '/tt_data/hb_ocs/hb_ocs' shm 757334180 count=10
23:54:40.70 Warn: : 6151: 6249 exited while connected to data store '/tt_data/hb_ocs/hb_ocs' shm 757334180 count=7
23:54:40.81 Warn: : 6151: 5150 exited while connected to data store '/tt_data/hb_ocs/hb_ocs' shm 757334180 count=7
23:54:40.92 Warn: : 6151: 6016 exited while connected to data store '/tt_data/hb_ocs/hb_ocs' shm 757334180 count=10
23:54:44.70 Warn: : 6151: 8168 exited while connected to data store '/tt_data/hb_ocs/hb_ocs' shm 757334180 count=7
23:54:44.80 Warn: : 6151: 8053 exited while connected to data store '/tt_data/hb_ocs/hb_ocs' shm 757334180 count=4
23:54:44.91 Warn: : 6151: 4869 exited while connected to data store '/tt_data/hb_ocs/hb_ocs' shm 757334180 count=10
00:00:54.02 Err : : 6151: 6165/60000000000ae160: Validity Error (pgDir.c: 2992) sbDirNodeIsValid: Page 13340614272 is ful
l but found in non-full list
00:00:54.02 Err : : 6151: 6165/60000000000ae160: Validity Error (pgDir.c: 2726) sbPgDirIsValid: Node (id 2556776512, ix 9
) is not valid
00:00:54.02 Err : : 6151: 6165/60000000000ae160: Validity Error (table.c: 29707) sbTblIsValid: Table has invalid physical
page dir (id 4611686017073389424)
00:00:54.02 Err : : 6151: 6165/60000000000ae160: Validity Error (db.c: 29151) sbDbIsValid(): sbTblIsValid() failed for ta
ble 735848
00:00:54.04 Err : : 6151: 6165/60000000000ae160: Data store marked invalid [db.c:4.1598:sbDbCkpt:19087] PID 6165 (timeste
nsubd) CONN 2042 (Worker) Context 0x60000000000ae160
00:00:54.20 Err : : 6151: 6165/60000000000ae160: Checkpoint failure (db.c, line 19089). 1 errors/warnings follow.
00:00:54.20 Err : : 6151: 6165/60000000000ae160: TT0994: Data store connection terminated. Please reconnect. -- file "db.
c", lineno 19089, procedure "sbDbCkpt"
00:00:54.20 Err : : 6151: TT14006: TimesTen daemon disconnect failed: 6165: Disconnect failure reported by client
00:00:54.21 Err : : 6165: subd: Error identified in [sub.c: line 4301]
00:00:54.21 Err : : 6165: subd: (Warning 994): TT0994: Data store connection terminated. Please reconnect. -- file "db.c"
, lineno 19089, procedure "sbDbCkpt"
00:00:54.21 Err : : 6165: file "db.c", lineno 19089, procedure "sbDbCkpt"
00:00:54.21 Err : : 6165: subd: (Warning 601): TT0601: Checkpoint failure -- file "db.c", lineno 15587, procedure "sbDbDi
sconnect"
00:00:54.21 Err : : 6165: file "db.c", lineno 15587, procedure "sbDbDisconnect"
00:00:54.21 Err : : 6165: subd: (Error 994): TT0994: Data store connection terminated. Please reconnect. -- file "dbAPI.c
", lineno 3178, procedure "sb_dbDisconnect()"
00:00:54.21 Err : : 6165: file "dbAPI.c", lineno 3178, procedure "sb_dbDisconnect()"
00:00:54.21 Err : : 6165: subd: disconnect failed rc 994
00:00:54.22 Err : : 6151: TT14000: TimesTen daemon internal error: 6165: Error processing 'unmanage' in subdaemon rc 400
00:00:54.22 Err : : 6151: TT14000: TimesTen daemon internal error: Error in subdaemon sb_err_t 994
00:00:54.22 Err : : 6151: TT14006: TimesTen daemon disconnect failed: error -1 in stopManaging
00:00:54.29 Warn: : 6151: 6165 ------------------: subdaemon process exited
00:03:55.54 Err : : 6151: 16745/60000000000ae160: Validity Error (pgDir.c: 2992) sbDirNodeIsValid: Page 13340614272 is fu
ll but found in non-full list
00:03:55.54 Err : : 6151: 16745/60000000000ae160: Validity Error (pgDir.c: 2726) sbPgDirIsValid: Node (id 2556776512, ix
9) is not valid
00:03:55.54 Err : : 6151: 16745/60000000000ae160: Validity Error (table.c: 29707) sbTblIsValid: Table has invalid physica
l page dir (id 4611686017073389424)
00:03:55.54 Err : : 6151: 16745/60000000000ae160: Validity Error (db.c: 29151) sbDbIsValid(): sbTblIsValid() failed for t
able 735848
00:03:55.55 Err : : 6151: 16745/60000000000ae160: Data store marked invalid [db.c:4.1598:sbDbCkpt:19087] PID 16745 (times
tensubd) CONN 2042 (Worker) Context 0x60000000000ae160
00:03:55.60 Err : : 6151: 16745/60000000000ae160: Checkpoint failure (db.c, line 19089). 1 errors/warnings follow.
00:03:55.72 Err : : 6151: 16745/60000000000ae160: TT0994: Data store connection terminated. Please reconnect. -- file "db
.c", lineno 19089, procedure "sbDbCkpt"
00:03:55.72 Err : : 6151: TT14006: TimesTen daemon disconnect failed: 16745: Disconnect failure reported by client
00:03:55.72 Err : : 16745: subd: Error identified in [sub.c: line 4301]
00:03:55.72 Err : : 16745: subd: (Warning 994): TT0994: Data store connection terminated. Please reconnect. -- file "db.c"
, lineno 19089, procedure "sbDbCkpt"
00:03:55.72 Err : : 16745: file "db.c", lineno 19089, procedure "sbDbCkpt"
00:03:55.72 Err : : 16745: subd: (Warning 601): TT0601: Checkpoint failure -- file "db.c", lineno 15587, procedure "sbDbDi
sconnect"
00:03:55.72 Err : : 16745: file "db.c", lineno 15587, procedure "sbDbDisconnect"
00:03:55.72 Err : : 16745: subd: (Error 994): TT0994: Data store connection terminated. Please reconnect. -- file "dbAPI.c
", lineno 3178, procedure "sb_dbDisconnect()"
00:03:55.72 Err : : 16745: file "dbAPI.c", lineno 3178, procedure "sb_dbDisconnect()"
00:03:55.72 Err : : 16745: subd: disconnect failed rc 994
00:03:55.72 Err : : 6151: TT14000: TimesTen daemon internal error: 16745: Error processing 'unmanage' in subdaemon rc 400
00:03:55.72 Err : : 6151: TT14000: TimesTen daemon internal error: Error in subdaemon sb_err_t 994
00:03:55.72 Err : : 6151: TT14006: TimesTen daemon disconnect failed: error -1 in stopManaging
00:03:56.01 Warn: : 6151: 16745 ------------------: subdaemon process exited
01:17:18.05 Err : ORA: 27108: ora-27108-0009-raStuff09836: Unexpected row count. Expecting 1. Got 0.
03:12:21.36 Warn: ORA: 27108: ora-27108-0013-refresh10886: Autorefresh was not able to acquire lock on one of the cache group
s, may be because a DDL transaction is open on the cache group. Autorefresh will be retried again
03:27:40.23 Err : ORA: 27108: ora-27108-0013-raStuff09836: Unexpected row count. Expecting 1. Got 0.
03:41:47.35 Err : ORA: 27108: ora-27108-0013-Refresh03385: The sequence number for table HB_ODS.TB_BIL_PRESENT_RESOURCE had c
hanged but the table was not refreshed. The cache group in datastore /tt_data/hb_ocs/hb_ocs is out of sync.