ONS process and high paging on AIX
ebh-OCMar 16 2013 — edited Mar 21 2013Dear all,
I would like to share with you a problem that we've been facing in the past few days, and hope to get some suggestions or solutions from your side:
First of all we are running an Oracle RAC 10g (10.2.0.3) with two database nodes on two dedicated IBM Power servers with AIX 5.3 as OS.
We have executed a memory upgrade during last week by Replacing on both servers our old 8GB RAM with new 16GB RAM, and after that we increased SGA_target to 8GB and SGA_max_size to 9GB on both instances. Unfortunately the next day one of the instances crashed, after investigation we have noted that there was very high paging activity on the server, so we immediately increased the swap space from 16GB to 48GB and restarted the server.
Also the next day the swap space was full and the instance crashed again, so we decided to decrease the sga on that instance to 5 GB, but once again the paging space was filled so we flushed the swapping space into another target to avoid the crash.
We noted that a processes on the server was consuming most of the memory (and paging): and that is (ONS) /oracle/opmn/bin, and we found in ons.log a repeating message:
(Passive connection: 0,<IP of localhost>,6200 invalid connect server IP fromat)
and below that we have: (hostaname:<name of the second server>)
Noting that we have changed nothing in our cluster configuration.
I would appreciate any suggestions here, I need to know if is it normal that the ons process consumes high memory (The highest consuming process on the server), If not what could be the problem. Or could it be that we are facing defected hardware with the new installed RAMs noting that using topas on AIX, the memory capacity is exact and during the installation all went smoothly and the server started normally.
Thank you for your help