Slow Database response--getiing ora-00604,ora-04021....
AchyotMar 6 2009 — edited Mar 6 2009Hello experts,
I am currently working as a junior DBA in a 24x7 project having 10g Database RAC having two nodes on RHEL 4 platform.
Due to slow database response time yesterday I checked alert log and found the below errors:
Thu Mar 5 08:33:52 2009
Errors in file /u02/app/oracle/product/database/admin/tims/bdump/tims1_smon_10657.trc:
ORA-00604: error occurred at recursive SQL level 2
ORA-04021: timeout occurred while waiting to lock object
.....................................
....................................
GES: Potential blocker (pid=10659) on resource DR-0x0-0x0;
enqueue info in file /u02/app/oracle/product/database/admin/tims/bdump/tims1_lmd0_10639.trc and DIAG trace file
.......................................
.....................................
Thu Mar 5 11:09:51 2009
Errors in file /u02/app/oracle/product/database/admin/tims/bdump/tims1_j001_29104.trc:
ORA-12012: error on auto execute of job 42568
ORA-22303: type "ORA-22303: type "EXFSYS"."NAMELIST" not found
ORA-00604: error occurred at recursive SQL level 3
ORA-04021: timeout occurred while waiting to lock object
ORA-06508: PL/SQL: could not find program unit being called: "EXFSYS.DBMS_RLMGR_DR"
ORA-06512: at line 1
"."" not found
.....................................
Due to 24x7 site I have to take quick decision and I did the below steps:
1. checked the cluster status and every thing found ok.
2. checked the DR database archive log status and found ok.
3. at node 1 I try to shutdown instance one and got below error
sql>shutdown immediate;
ORA-00604: error occurred at recursive SQL level 2
ORA-04021: timeout occurred while waiting to lock object
sql>
4. So unable to shutdown instance 1 I tried to shutdown instance 2
after 15 minute I checked
#crs_stat -t and found instance 2 is offline
but got the below error at sql prompt
sql>shutdown immediate;
ORA-03113 end-of-file on communication channel
sql>
5. after this I tried to shutdow instance 1 again and this time I am successfull to shut down it
6. after step 5 I was able to successfully restart both the instance
7. immediately took online backup
Now both the application as well as database is running ok. But my question is why we are getting these errors?? I am suspecting due to improperlly written sql quries by the application develoment team we are having these errors in the alert log. So being a junior DBA what should I do so that in future we don't get these type of errors.
Thanx
Achyot