Hello All,
We are having a weird problem and we are having difficult time to find the cause so we cannot even fix it.
Environment is, Oracle 11g on RHEL6
> env | grep -E 'ORACLE|LD'
LD_LIBRARY_PATH=/usr/sap/SEQ/SYS/exe/run:/usr/sap/SEQ/SYS/exe/uc/linuxx86_64:/oracle/SEQ/112_64/lib
ORACLE_SID=SEQ
ORACLE_BASE=/oracle
ORACLE_HOME=/oracle/SEQ/112_64
We cannot start listener;
> lsnrctl start
LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 15-OCT-2015 12:23:29
Copyright (c) 1991, 2011, Oracle. All rights reserved.
Message 1070 not found; No message file for product=network, facility=TNSTNS-12545: Message 12545 not found; No message file for product=network, facility=TNS
TNS-12560: Message 12560 not found; No message file for product=network, facility=TNS
TNS-00515: Message 515 not found; No message file for product=network, facility=TNS
Linux Error: 2: No such file or directory
and we cannot start database;
> sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Thu Oct 15 12:28:50 2015
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.
Total System Global Area 2305060864 bytes
Fixed Size 2230392 bytes
Variable Size 1157629832 bytes
Database Buffers 1124073472 bytes
Redo Buffers 21127168 bytes
Database mounted.
ORA-00600: internal error code, arguments: [kcrf_resilver_log_1],
[0x0E9E8DC78], [2], [], [], [], [], [], [], [], [], []
sqlnet.log file is empty, listener.log has nothing useful, only thing at last is this;
Thu Oct 15 08:47:27 2015
No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=SEQ.WORLD)))
No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=SEQ)))
No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=erpqa.do.main)(PORT=1521)))
Listener completed notification to CRS on stop
only alert.log file has some information to re-create the controlfile and open the database;
Thu Oct 15 03:18:44 2015
********************* ATTENTION: ********************
The controlfile header block returned by the OS
has a sequence number that is too old.
The controlfile might be corrupted.
PLEASE DO NOT ATTEMPT TO START UP THE INSTANCE
without following the steps below.
RE-STARTING THE INSTANCE CAN CAUSE SERIOUS DAMAGE
TO THE DATABASE, if the controlfile is truly corrupted.
In order to re-start the instance safely,
please do the following:
(1) Save all copies of the controlfile for later
analysis and contact your OS vendor and Oracle support.
(2) Mount the instance and issue:
ALTER DATABASE BACKUP CONTROLFILE TO TRACE;
(3) Unmount the instance.
(4) Use the script in the trace file to
RE-CREATE THE CONTROLFILE and open the database.
*****************************************************
Thu Oct 15 03:18:45 2015
System state dump requested by (instance=1, osid=20813 (CKPT)), summary=[abnormal instance termination].
System State dumped to trace file /oracle/SEQ/saptrace/diag/rdbms/seq/SEQ/trace/SEQ_diag_20801.trc
CKPT (ospid: 20813): terminating the instance
Non critical error ORA-48913 caught while writing to trace file "/oracle/SEQ/saptrace/diag/rdbms/seq/SEQ/trace/SEQ_diag_20801.trc"
Error message: ORA-48913: Writing into trace file failed, file size limit [10240000] reached
Writing to the above trace file is disabled for now on...
Instance terminated by CKPT, pid = 20813
After using the commands in these steps, we can open database but listener is still the same. What can we do to fix the listener ?