CPU load goes very high(any node/any time) hanging all the apps (any node)!
jeev007Jul 9 2010 — edited Jul 18 2010Hi,
We have production servers running on
RHEL 5.3(Red Hat Enterprise Linux Server release 5.3 (Tikanga)) on
Oracle 11g Standard edition Release2 (11.2.0.1.0) RAC with ASM.
It is a 2 node RAC setup which is connected by Java & Ruby on Rails applications through oracle client(TNSNAMES).
In normal time the load is below 1 on both nodes. But in between some spikes happening in DB server (any node) and the load shoots upto 8-10 and hangs the entire applications. Then we are forced to restart the application servers and then restart the DB server and things get back to normal. This happens randomly on any one of the nodes. Same time the load does not shoot up on other node, i.e if the load on node 1 is 9-10, the load on other node is 1.2/1.4. The spikes happen daily 3-4 times minimum. .ie It happens either on Node 1 or Node 2 and it happens adhoc. We have enough memory on our servers (32GB RAM on each server).
We have noticed that,
From a session perspective sessions are distributed (say 87 sessions on node1, 85 on node2).
In normal time, even though the SGA max size is 10G, oracle takes almost 32G memory and free memory is only in MBs say 112MB free.
We have a scheduled rman full backup runs on everyday 3AM in cron, and noticed that once th RMAN is started, the memory is not getting released.
We have collected the statspack,sessions details but could not help..
We dont get any error as such in alert logs
There is no background/scheduled jobs running on server.
Has anyone faced this issue with 11g RAC? If so how it has been solved. It is very critical as it directly impacts our business. Any help or hints would be appreciated.
Thanks
Jaise.