Oracle Database Discussions

Hi,
I've an Oracle Database 11g Release 11.1.0.6.0 - 64bit Production With the Real Application Clusters option.

Yesterday instance on node1 crashed.
This is the extract of alert log:

Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc (incident=195198):
ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195198/EVODB1_pmon_9297_i195198.trc
Sat Jan 22 16:24:47 2011
Trace dumping is performing id=[cdmp_20110122162447]
Sat Jan 22 16:24:50 2011
Sweep Incident[195198]: completed
Sat Jan 22 16:25:00 2011
Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc (incident=195199):
ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195199/EVODB1_pmon_9297_i195199.trc
Sat Jan 22 16:25:10 2011
Trace dumping is performing id=[cdmp_20110122162510]
Sat Jan 22 16:25:12 2011
Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc (incident=195200):
ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195200/EVODB1_pmon_9297_i195200.trc
Sat Jan 22 16:25:23 2011
DBRM (ospid: 9310): terminating the instance due to error 472
Sat Jan 22 16:25:23 2011
System state dump is made for local instance
System State dumped to trace file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_diag_16320.trc
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3724_47445977853072)
Sat Jan 22 16:25:23 2011
SUCCESS: diskgroup ARCHIVELOG was dismounted
Sat Jan 22 16:14:22 2011
DDE: Problem Key 'ORA 600 [17183]' was completely flood controlled (0x6)
Further messages for this problem key will be suppressed for up to 10 minutes
Sat Jan 22 16:15:56 2011
Thread 1 advanced to log sequence 171322
Current log# 18 seq# 171322 mem# 0: +ONLINELOG/evodb/onlinelog/group_18.264.729348055
Sat Jan 22 16:15:56 2011
SUCCESS: diskgroup ARCHIVELOG was mounted
Sat Jan 22 16:16:15 2011
SUCCESS: diskgroup ARCHIVELOG was dismounted
Sat Jan 22 16:20:52 2011
Thread 1 advanced to log sequence 171323
Current log# 12 seq# 171323 mem# 0: +ONLINELOG/evodb/onlinelog/group_12.267.729333861
Sat Jan 22 16:20:53 2011
SUCCESS: diskgroup ARCHIVELOG was mounted
Sat Jan 22 16:21:14 2011
SUCCESS: diskgroup ARCHIVELOG was dismounted
Sat Jan 22 16:24:23 2011
DDE: Problem Key 'ORA 600 [17183]' was completely flood controlled (0x2)
Further messages for this problem key will be suppressed for up to 10 minutes
Sat Jan 22 16:24:38 2011
Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc (incident=195198):
ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195198/EVODB1_pmon_9297_i195198.trc
Sat Jan 22 16:24:47 2011
Trace dumping is performing id=[cdmp_20110122162447]
Sat Jan 22 16:24:50 2011
Sweep Incident[195198]: completed
Sat Jan 22 16:25:00 2011
Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc (incident=195199):
ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195199/EVODB1_pmon_9297_i195199.trc
Sat Jan 22 16:25:10 2011
Trace dumping is performing id=[cdmp_20110122162510]
Sat Jan 22 16:25:12 2011
Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc (incident=195200):
ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195200/EVODB1_pmon_9297_i195200.trc
Sat Jan 22 16:25:23 2011
DBRM (ospid: 9310): terminating the instance due to error 472
Sat Jan 22 16:25:23 2011
System state dump is made for local instance
System State dumped to trace file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_diag_16320.trc
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3724_47445977853072)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3650_47673206152336)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (19709_47962498952336)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3715_47239602621584)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3718_47120460606608)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3646_47899813694608)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3671_47700480982160)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3637_47223057443984)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3695_47109627223184)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3679_47262512999568)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3733_46976527121552)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3667_47474016554128)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3669_47585103858832)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3658_47992462539920)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3654_47736794324112)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3686_47295241862288)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3706_47055349939344)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3630_47331091332240)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3713_47194150415504)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3726_47289949199504)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3677_47842172337296)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (19131_47557370236048)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (27091_47096969530512)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3688_47422639070352)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3652_47944422886544)
Sat Jan 22 16:25:23 2011
ORA-1092 : opiodr aborting process unknown ospid (3648_47426149817488)
Sat Jan 22 16:25:24 2011
Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j005_9360.trc:
ORA-00472: PMON process terminated with error
ORA-1092 : opidrv aborting process J005 ospid (9360_47420959304848)
Sat Jan 22 16:25:24 2011
ORA-1092 : opiodr aborting process unknown ospid (18393_47566684101776)
Sat Jan 22 16:25:24 2011
Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j003_7054.trc:
ORA-00472: PMON process terminated with error
ORA-1092 : opidrv aborting process J003 ospid (7054_47244451724432)
Sat Jan 22 16:25:24 2011
ORA-1092 : opiodr aborting process unknown ospid (19719_47840732109968)
Sat Jan 22 16:25:24 2011
ORA-1092 : opiodr aborting process unknown ospid (22184_47575322619024)
Sat Jan 22 16:25:24 2011
ORA-1092 : opiodr aborting process unknown ospid (19721_47679094451344)
Sat Jan 22 16:25:24 2011
ORA-1092 : opiodr aborting process unknown ospid (30170_47106572815504)
Sat Jan 22 16:25:24 2011
ORA-472 : opidrv aborting process S000 ospid (9367_46990649617552)
Sat Jan 22 16:25:24 2011
Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j002_7052.trc:
ORA-00472: PMON process terminated with error
ORA-1092 : opidrv aborting process J002 ospid (7052_47411695402128)
Sat Jan 22 16:25:24 2011
ORA-1092 : opiodr aborting process unknown ospid (18493_47469336738960)
Sat Jan 22 16:25:24 2011
Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j006_10011.trc:
ORA-00472: PMON process terminated with error
ORA-1092 : opidrv aborting process J006 ospid (10011_47836408102032)
Sat Jan 22 16:25:24 2011
ORA-1092 : opiodr aborting process unknown ospid (19714_47958179912848)
Sat Jan 22 16:25:24 2011
Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j007_10289.trc:
ORA-00472: PMON process terminated with error
ORA-1092 : opidrv aborting process J007 ospid (10289_47356480347280)
Sat Jan 22 16:25:24 2011
ORA-1092 : opiodr aborting process unknown ospid (9076_47765392362640)
Sat Jan 22 16:25:24 2011
ORA-1092 : opitsk aborting process
Sat Jan 22 16:25:25 2011
ORA-1092 : opiodr aborting process unknown ospid (22167_47949919177872)
Sat Jan 22 16:25:25 2011
ORA-1092 : opiodr aborting process unknown ospid (24320_47408077532304)
Sat Jan 22 16:25:25 2011
ORA-1092 : opiodr aborting process unknown ospid (22130_47998542101648)
Sat Jan 22 16:25:25 2011
ORA-472 : opidrv aborting process N000 ospid (14774_47466200525968)
Sat Jan 22 16:25:25 2011
Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_q000_14701.trc:
ORA-00472: PMON process terminated with error
ORA-1092 : opidrv aborting process Q000 ospid (14701_47693688122512)
Sat Jan 22 16:25:25 2011
ORA-1092 : opiodr aborting process unknown ospid (18386_47900707175568)
Sat Jan 22 16:25:25 2011
ORA-1092 : opiodr aborting process unknown ospid (27553_46921398619280)
Sat Jan 22 16:25:26 2011
Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j001_7050.trc:
ORA-00472: PMON process terminated with error
ORA-1092 : opidrv aborting process J001 ospid (7050_47290016251024)
Sat Jan 22 16:25:26 2011
Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_o001_5297.trc:
ORA-00472: PMON process terminated with error
ORA-1092 : opidrv aborting process O001 ospid (5297_47702958673040)
Sat Jan 22 16:25:26 2011
Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_o000_5277.trc:
ORA-00472: PMON process terminated with error
ORA-1092 : opidrv aborting process O000 ospid (5277_47599668255888)
Sat Jan 22 16:25:26 2011
ORA-1092 : opiodr aborting process unknown ospid (19745_47560156560528)
Sat Jan 22 16:25:26 2011
ORA-1092 : opiodr aborting process unknown ospid (27044_47388543339664)
Sat Jan 22 16:25:26 2011
Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j004_7061.trc:
ORA-00472: PMON process terminated with error
ORA-1092 : opidrv aborting process J004 ospid (7061_47341656689808)
Sat Jan 22 16:25:27 2011
ORA-1092 : opiodr aborting process unknown ospid (14431_47662297960112)
Sat Jan 22 16:25:27 2011
ORA-1092 : opiodr aborting process unknown ospid (14433_47826608635568)
Sat Jan 22 16:25:31 2011
ORA-1092 : opitsk aborting process
Sat Jan 22 16:25:32 2011
License high water mark = 269
Instance terminated by DBRM, pid = 9310
USER (ospid: 10539): terminating the instance
Instance terminated by USER, pid = 10539
Sat Jan 22 16:25:49 2011
Starting ORACLE instance (normal)

This is the trace /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195200/EVODB1_pmon_9297_i195200.trc
(the one regarding the error DBRM (ospid: 9310): terminating the instance due to error 472)

Oracle Database 11g Release 11.1.0.6.0 - 64bit Production
With the Real Application Clusters option
ORACLE_HOME = /u01/app/oracle/product/11.1.0/db1
System name: Linux
Node name: dcsrv-evodb01
Release: 2.6.18-164.10.1.el5
Version: #1 SMP Wed Dec 30 18:35:28 EST 2009
Machine: x86_64
Instance name: EVODB1
Redo thread mounted by this instance: 1
Oracle process number: 2
Unix process pid: 9297, image: oracle@dcsrv-evodb01 (PMON)


*** 2011-01-22 16:25:12.984
*** SESSION ID:(884.1) 2011-01-22 16:25:12.984
*** CLIENT ID:() 2011-01-22 16:25:12.984
*** SERVICE NAME:(SYS$BACKGROUND) 2011-01-22 16:25:12.984
*** MODULE NAME:() 2011-01-22 16:25:12.984
*** ACTION NAME:() 2011-01-22 16:25:12.984

Dump continued from file: /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc
ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []

+========= Dump for incident 195200 (ORA 600 [17183]) ========+
----- Beginning of Customized Incident Dump(s) -----

And this is the trace /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j005_9360.trc
(the one regarding the error ORA-00472: PMON process terminated with error)

Trace file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j005_9360.trc
Oracle Database 11g Release 11.1.0.6.0 - 64bit Production
With the Real Application Clusters option
ORACLE_HOME = /u01/app/oracle/product/11.1.0/db1
System name: Linux
Node name: dcsrv-evodb01
Release: 2.6.18-164.10.1.el5
Version: #1 SMP Wed Dec 30 18:35:28 EST 2009
Machine: x86_64
Instance name: EVODB1
Redo thread mounted by this instance: 1
Oracle process number: 95
Unix process pid: 9360, image: oracle@dcsrv-evodb01 (J005)


*** 2011-01-22 16:25:24.057
*** SESSION ID:(649.45863) 2011-01-22 16:25:24.057
*** CLIENT ID:() 2011-01-22 16:25:24.057
*** SERVICE NAME:(SYS$USERS) 2011-01-22 16:25:24.057
*** MODULE NAME:() 2011-01-22 16:25:24.057
*** ACTION NAME:() 2011-01-22 16:25:24.057

error 472 in job queue process
ORA-00472: PMON process terminated with error

My questions are:
- how can I discover the nature of the ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []?
- what does error 472 in job queue proces mean?
- is it normal the the error 472 in job queue proces crash the instance?

Thanks in advanca for any kind of help!

Samuel