Hi Gurus,
Oracle Version:10.2.0.3
Operating system:Linux
Yesterday night there was a call from my onsite member but I was unable to lift it and morning i recognize that there was a problem and i saw following in my alert.log file
Thu Jan 27 13:31:33 2011
Thread 1 cannot allocate new log, sequence 58704
Private strand flush not complete
Current log# 1 seq# 58703 mem# 0: /u02/oradata/acaprod/redo01a.log
Current log# 1 seq# 58703 mem# 1: /u03/oradata/acaprod/redo01b.log
Thread 1 advanced to log sequence 58704
Current log# 3 seq# 58704 mem# 0: /u02/oradata/acaprod/redo03a.log
Current log# 3 seq# 58704 mem# 1: /u03/oradata/acaprod/redo03b.log
Thu Jan 27 13:56:33 2011
Thread 1 cannot allocate new log, sequence 58705
Private strand flush not complete
Current log# 3 seq# 58704 mem# 0: /u02/oradata/acaprod/redo03a.log
Current log# 3 seq# 58704 mem# 1: /u03/oradata/acaprod/redo03b.log
Thread 1 advanced to log sequence 58705
Current log# 2 seq# 58705 mem# 0: /u02/oradata/acaprod/redo02a.log
Current log# 2 seq# 58705 mem# 1: /u03/oradata/acaprod/redo02b.log
Thu Jan 27 14:13:59 2011
Starting background process EMN0
EMN0 started with pid=65, OS id=27464
Thu Jan 27 14:13:59 2011
Shutting down instance: further logons disabled
Thu Jan 27 14:14:04 2011
Stopping background process QMNC
Thu Jan 27 14:14:04 2011
Stopping background process CJQ0
Thu Jan 27 14:14:06 2011
Stopping background process MMNL
Thu Jan 27 14:14:07 2011
Stopping background process MMON
Thu Jan 27 14:14:09 2011
Errors in file /home/oracle/oracle/admin/acaprod/udump/acaprod_ora_27083.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-12663: Services required by client not available on the server
ORA-36961: Oracle OLAP is not available.
ORA-06512: at "SYS.OLAPIHISTORYRETENTION", line 1
ORA-06512: at line 15
Thu Jan 27 14:14:09 2011
Shutting down instance (normal)
License high water mark = 93
Thu Jan 27 14:14:09 2011
Stopping Job queue slave processes
Thu Jan 27 14:14:09 2011
Job queue slave processes stopped
Thu Jan 27 14:19:08 2011
Active process 16124 user 'oracle' program 'oracleacaprod@virqts3ora001.localdomain'
Active process 21258 user 'oracle' program 'oracleacaprod@virqts3ora001.localdomain'
Active process 14522 user 'oracle' program 'oracleacaprod@virqts3ora001.localdomain'
Active process 14557 user 'oracle' program 'oracleacaprod@virqts3ora001.localdomain'
Active process 32428 user 'oracle' program 'oracleacaprod@virqts3ora001.localdomain'
Active process 15986 user 'oracle' program 'oracle@virqts3ora001.localdomain (TNS V1-V3)'
Active process 32430 user 'oracle' program 'oracleacaprod@virqts3ora001.localdomain'
Active process 32424 user 'oracle' program 'oracleacaprod@virqts3ora001.localdomain'
Active process 21891 user 'oracle' program 'oracleacaprod@virqts3ora001.localdomain'
Active process 18495 user 'oracle' program 'oracleacaprod@virqts3ora001.localdomain'
Active process 32426 user 'oracle' program 'oracleacaprod@virqts3ora001.localdomain'
Active process 18467 user 'oracle' program 'oracleacaprod@virqts3ora001.localdomain'
Active process 18480 user 'oracle' program 'oracleacaprod@virqts3ora001.localdomain'
Active process 18463 user 'oracle' program 'oracleacaprod@virqts3ora001.localdomain'
Active process 18457 user 'oracle' program 'oracleacaprod@virqts3ora001.localdomain'
Active process 18459 user 'oracle' program 'oracleacaprod@virqts3ora001.localdomain'
Active process 32432 user 'oracle' program 'oracleacaprod@virqts3ora001.localdomain'
SHUTDOWN: waiting for logins to complete.
Thu Jan 27 14:20:10 2011
Shutting down instance: further logons disabled
Thu Jan 27 14:20:10 2011
Errors in file /home/oracle/oracle/admin/acaprod/udump/acaprod_ora_27083.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-12663: Services required by client not available on the server
ORA-36961: Oracle OLAP is not available.
ORA-06512: at "SYS.OLAPIHISTORYRETENTION", line 1
ORA-06512: at line 15
Thu Jan 27 14:20:10 2011
Shutting down instance (immediate)
License high water mark = 93
Thu Jan 27 14:20:27 2011
All dispatchers and shared servers shutdown
Thu Jan 27 14:20:28 2011
Process OS id : 18457 alive after kill
Errors in file /home/oracle/oracle/admin/acaprod/udump/acaprod_ora_27083.trc
Thu Jan 27 14:20:28 2011
Process OS id : 18463 alive after kill
Errors in file /home/oracle/oracle/admin/acaprod/udump/acaprod_ora_27083.trc
Thu Jan 27 14:20:28 2011
Process OS id : 18467 alive after kill
Errors in file /home/oracle/oracle/admin/acaprod/udump/acaprod_ora_27083.trc
Thu Jan 27 14:20:28 2011
Process OS id : 18480 alive after kill
Errors in file /home/oracle/oracle/admin/acaprod/udump/acaprod_ora_27083.trc
Thu Jan 27 14:20:28 2011
Process OS id : 32424 alive after kill
Errors in file /home/oracle/oracle/admin/acaprod/udump/acaprod_ora_27083.trc
Thu Jan 27 14:20:30 2011
ALTER DATABASE CLOSE NORMAL
Thu Jan 27 14:20:30 2011
SMON: disabling tx recovery
SMON: disabling cache recovery
Thu Jan 27 14:20:30 2011
Shutting down archive processes
Archiving is disabled
Thu Jan 27 14:20:40 2011
ARCH shutting down
Thu Jan 27 14:20:40 2011
ARC1: Becoming the heartbeat ARCH
ARC0: Archival stopped
Thu Jan 27 14:20:40 2011
ARC1: Archiving disabled
ARCH shutting down
ARC1: Archival stopped
Thu Jan 27 14:20:41 2011
Thread 1 closed at log sequence 58705
Successful close of redo thread 1
Thu Jan 27 14:20:41 2011
Completed: ALTER DATABASE CLOSE NORMAL
Thu Jan 27 14:20:41 2011
ALTER DATABASE DISMOUNT
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Thu Jan 27 14:21:23 2011
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 2
Autotune of undo retention is turned on.
IMODE=BR
ILAT =18
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.3.0.
System parameters with non-default values:
processes = 150
__shared_pool_size = 301989888
__large_pool_size = 16777216
__java_pool_size = 16777216
__streams_pool_size = 33554432
sga_target = 1610612736
control_files = /u02/oradata/acaprod/control01.ctl, /u02/oradata/acaprod/control02.ctl, /u03/oradata/acaprod/control03.ctl
db_block_size = 8192
__db_cache_size = 1224736768
compatible = 10.2.0.3.0
log_archive_dest_1 = LOCATION=/u02/oradata/arch/acaprod
log_archive_format = %t_%s_%r.dbf
archive_lag_target = 2700
db_file_multiblock_read_count= 16
db_recovery_file_dest = /u02/oradata/flash_recovery_area
db_recovery_file_dest_size= 2147483648
standby_file_management = AUTO
undo_management = AUTO
undo_tablespace = UNDOTBS1
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=acaprodXDB)
utl_file_dir = /u02/utl_file_dir
job_queue_processes = 12
background_dump_dest = /home/oracle/oracle/admin/acaprod/bdump
user_dump_dest = /home/oracle/oracle/admin/acaprod/udump
core_dump_dest = /home/oracle/oracle/admin/acaprod/cdump
audit_file_dest = /home/oracle/oracle/admin/acaprod/adump
db_name = acaprod
open_cursors = 500
pga_aggregate_target = 839909376
PMON started with pid=2, OS id=29045
PSP0 started with pid=3, OS id=29047
MMAN started with pid=4, OS id=29050
DBW0 started with pid=5, OS id=29052
LGWR started with pid=6, OS id=29054
CKPT started with pid=7, OS id=29056
SMON started with pid=8, OS id=29058
RECO started with pid=9, OS id=29060
CJQ0 started with pid=10, OS id=29062
MMON started with pid=11, OS id=29064
Thu Jan 27 14:21:24 2011
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=12, OS id=29066
Thu Jan 27 14:21:24 2011
starting up 1 shared server(s) ...
Oracle Data Guard is not available in this edition of Oracle.
Thu Jan 27 14:21:24 2011
ALTER DATABASE MOUNT
Thu Jan 27 14:21:28 2011
Setting recovery target incarnation to 2
Thu Jan 27 14:21:28 2011
Successful mount of redo thread 1, with mount id 1921081444
Thu Jan 27 14:21:28 2011
Database mounted in Exclusive Mode
Completed: ALTER DATABASE MOUNT
Thu Jan 27 14:21:28 2011
ALTER DATABASE OPEN
Thu Jan 27 14:21:28 2011
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=16, OS id=29090
Thu Jan 27 14:21:28 2011
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=17, OS id=29092
Thu Jan 27 14:21:28 2011
Thread 1 opened at log sequence 58705
Current log# 2 seq# 58705 mem# 0: /u02/oradata/acaprod/redo02a.log
Current log# 2 seq# 58705 mem# 1: /u03/oradata/acaprod/redo02b.log
Successful open of redo thread 1
Thu Jan 27 14:21:28 2011
ARC0: Becoming the 'no FAL' ARCH
ARC0: Becoming the 'no SRL' ARCH
Thu Jan 27 14:21:28 2011
ARC1: Becoming the heartbeat ARCH
Thu Jan 27 14:21:28 2011
SMON: enabling cache recovery
Thu Jan 27 14:21:29 2011
Successfully onlined Undo Tablespace 1.
Thu Jan 27 14:21:29 2011
SMON: enabling tx recovery
Thu Jan 27 14:21:29 2011
Database Characterset is WE8ISO8859P1
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=18, OS id=29094
Thu Jan 27 14:21:30 2011
Errors in file /home/oracle/oracle/admin/acaprod/udump/acaprod_ora_29088.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-12663: Services required by client not available on the server
ORA-36961: Oracle OLAP is not available.
ORA-06512: at "SYS.OLAPIHISTORYRETENTION", line 1
ORA-06512: at line 15
Thu Jan 27 14:21:30 2011
db_recovery_file_dest_size of 2048 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Thu Jan 27 14:21:30 2011
Completed: ALTER DATABASE OPEN
Thu Jan 27 14:21:30 2011
Thread 1 advanced to log sequence 58706
Current log# 1 seq# 58706 mem# 0: /u02/oradata/acaprod/redo01a.log
Current log# 1 seq# 58706 mem# 1: /u03/oradata/acaprod/redo01b.log
Thu Jan 27 14:41:31 2011
Thread 1 cannot allocate new log, sequence 58707
Private strand flush not complete
Current log# 1 seq# 58706 mem# 0: /u02/oradata/acaprod/redo01a.log
Current log# 1 seq# 58706 mem# 1: /u03/oradata/acaprod/redo01b.log
Thread 1 advanced to log sequence 58707
Current log# 3 seq# 58707 mem# 0: /u02/oradata/acaprod/redo03a.log
Current log# 3 seq# 58707 mem# 1: /u03/oradata/acaprod/redo03b.log
Thu Jan 27 15:06:31 2011
Thread 1 cannot allocate new log, sequence 58708
Private strand flush not complete
Current log# 3 seq# 58707 mem# 0: /u02/oradata/acaprod/redo03a.log
Current log# 3 seq# 58707 mem# 1: /u03/oradata/acaprod/redo03b.log
Thread 1 advanced to log sequence 58708
Current log# 2 seq# 58708 mem# 0: /u02/oradata/acaprod/redo02a.log
Current log# 2 seq# 58708 mem# 1: /u03/oradata/acaprod/redo02b.log
Thu Jan 27 15:31:32 2011
Thread 1 cannot allocate new log, sequence 58709
Private strand flush not complete
Current log# 2 seq# 58708 mem# 0: /u02/oradata/acaprod/redo02a.log
Current log# 2 seq# 58708 mem# 1: /u03/oradata/acaprod/redo02b.log
Thread 1 advanced to log sequence 58709
Current log# 1 seq# 58709 mem# 0: /u02/oradata/acaprod/redo01a.log
Current log# 1 seq# 58709 mem# 1: /u03/oradata/acaprod/redo01b.log
Thu Jan 27 15:47:57 2011
ALTER DATABASE DATAFILE '/u02/oradata/acaprod/ACAUAT_FEB18_2.dbf' AUTOEXTEND ON NEXT 128M
Thu Jan 27 15:47:57 2011
Completed: ALTER DATABASE DATAFILE '/u02/oradata/acaprod/ACAUAT_FEB18_2.dbf' AUTOEXTEND ON NEXT 128M
Thu Jan 27 15:48:11 2011
ALTER DATABASE DATAFILE '/u03/oradata/acaprod/ACAUAT_FEB18.dbf' AUTOEXTEND ON NEXT 16M
Thu Jan 27 15:48:11 2011
Completed: ALTER DATABASE DATAFILE '/u03/oradata/acaprod/ACAUAT_FEB18.dbf' AUTOEXTEND ON NEXT 16M
Thu Jan 27 15:48:32 2011
ALTER DATABASE DATAFILE '/u02/oradata/acaprod/ACA_CSO_02.dbf' AUTOEXTEND ON NEXT 16M
Thu Jan 27 15:48:32 2011
Completed: ALTER DATABASE DATAFILE '/u02/oradata/acaprod/ACA_CSO_02.dbf' AUTOEXTEND ON NEXT 16M
Thu Jan 27 15:48:42 2011
ALTER DATABASE DATAFILE '/u02/oradata/acaprod/ACA_CSO_01.dbf' AUTOEXTEND ON NEXT 16M
Completed: ALTER DATABASE DATAFILE '/u02/oradata/acaprod/ACA_CSO_01.dbf' AUTOEXTEND ON NEXT 16M
Thu Jan 27 15:56:33 2011
Thread 1 cannot allocate new log, sequence 58710
Private strand flush not complete
Current log# 1 seq# 58709 mem# 0: /u02/oradata/acaprod/redo01a.log
Current log# 1 seq# 58709 mem# 1: /u03/oradata/acaprod/redo01b.log
Thread 1 advanced to log sequence 58710
Current log# 3 seq# 58710 mem# 0: /u02/oradata/acaprod/redo03a.log
Current log# 3 seq# 58710 mem# 1: /u03/oradata/acaprod/redo03b.log
Thu Jan 27 16:06:28 2011
Immediate Kill Session#: 71, Serial#: 8
Immediate Kill Session: sess: 0x7f341858 OS pid: 24483
Thu Jan 27 16:07:07 2011
Immediate Kill Session#: 117, Serial#: 227
Immediate Kill Session: sess: 0x7f3779b8 OS pid: 24485
Thu Jan 27 16:21:33 2011
Thread 1 cannot allocate new log, sequence 58711
Private strand flush not complete
Current log# 3 seq# 58710 mem# 0: /u02/oradata/acaprod/redo03a.log
Current log# 3 seq# 58710 mem# 1: /u03/oradata/acaprod/redo03b.log
Thread 1 advanced to log sequence 58711
Current log# 2 seq# 58711 mem# 0: /u02/oradata/acaprod/redo02a.log
Current log# 2 seq# 58711 mem# 1: /u03/oradata/acaprod/redo02b.log
Thu Jan 27 16:29:30 2011
Immediate Kill Session#: 62, Serial#: 697
Immediate Kill Session: sess: 0x7f336f08 OS pid: 23609
Immediate Kill Session#: 63, Serial#: 448
Immediate Kill Session: sess: 0x7f3381d8 OS pid: 30029
Thu Jan 27 16:29:30 2011
Process OS id : 30029 alive after kill
Errors in file /home/oracle/oracle/admin/acaprod/udump/acaprod_ora_29115.trc
Immediate Kill Session#: 64, Serial#: 334
Immediate Kill Session: sess: 0x7f3394a8 OS pid: 32597
Immediate Kill Session#: 65, Serial#: 503
Immediate Kill Session: sess: 0x7f33a778 OS pid: 31408
Immediate Kill Session#: 68, Serial#: 685
Immediate Kill Session: sess: 0x7f33dfe8 OS pid: 32591
Thu Jan 27 16:29:31 2011
Process OS id : 32591 alive after kill
Errors in file /home/oracle/oracle/admin/acaprod/udump/acaprod_ora_29115.trc
Immediate Kill Session#: 70, Serial#: 1457
Immediate Kill Session: sess: 0x7f340588 OS pid: 31043
Immediate Kill Session#: 71, Serial#: 482
Immediate Kill Session: sess: 0x7f341858 OS pid: 31485
Immediate Kill Session#: 72, Serial#: 225
Immediate Kill Session: sess: 0x7f342b28 OS pid: 30043
Immediate Kill Session#: 74, Serial#: 524
Immediate Kill Session: sess: 0x7f3450c8 OS pid: 30031
Immediate Kill Session#: 77, Serial#: 435
Immediate Kill Session: sess: 0x7f348938 OS pid: 1537
Immediate Kill Session#: 78, Serial#: 1785
can any one please help me out what might be the problem.
Thanks & Regards,
Poorna Prasad.S