Integration

Our client's application server has been live for 6 months now and currently there is an issue.

This server is a Dell 6850 running Red Hat Linux ES 3.4 (Taroon Update 5) and kernel 2.4.21-32 with 2 CPUs and 8Gb of ram. Our J2EE web application runs in Oracle Application Server and currently 5 OC4Js are up and running.

The issue we are facing is the server has frequently gone to a hang state. When this happened, we won't be able to telnet into the server but there are ping responses. So the only thing to do then would be to reboot the server.

Now I have setup nmon for linux to run 24 hours in the server taking snap shots of the server's performance every 10 minutes. From the nmon output, I could see free memory is always around 20Mb - 50 Mb and sometimes would shoot up to 1Gb and then in 3 hours back down to 20Mb - 50Mb. Is this a normal pattern for a linux server?

From the Oracle console under memory usage, I saw that around 1.5Gb is being used by the Application Server, 36Mb free and 6.5Gb used by other processes. Now, I am curious as what this other processes are and why are they using 6.5Gb.

I am not sure if this is caused by a faulty hardware, OS/kernel bug, or our web application.

What I have done:
i) monitor /var/log/messages but no suspicious error being logged not even before or during the server hang.
ii) monitor Oracle Application Sever log files but no suspicious errors logged.
iii) monitor nmon output daily
iv) searched for similar cases using Google and Oracle's Metalink but found nothing.

Thanks.