'ps -ef' command hangs - server needs hard boot
I have a strange issue that has now occurred twice on two different physical servers running OEL 5.6. One server runs Oracle databases and one runs Netbackup.
Server one ran to 100% on the oracle binary directory. The DBA did an 'rm' to remove files and create space. At this point the server seemed to become unstable and when issueing a 'ps -ef' or an 'ls' I got partial output then the server would hang. crtl-c would not even break it. A hard reboot cleared the issue.
Server two had an issue where the Netbackup binary library went above 91% and the admin used 'rm' to clean up some large log files. There was no issue immediately, but sometime druing the night the server stopped running some processes. We could not connect through the Netbackup Admin Console. I issued the 'ps -ef' and got the same unstable response. A hard reboot was needed.
In both cases the server was not sowing any issue with cpu, memory, or io. the load was minimal. also, a 'ps -e' would work fine. In both cases the 'reboot' command never worked. I would get 'shutting down for system reboot' but it would not proceed, requiring a hard reboot.
There were messages about 'clearing orphaned inodes' during the post of each reboot.
I would like to understand what may have been occuring, if it was related to the binary filesystems becoming full and then having files removed, and what would be a method to troubleshoot as the issue is occuring.
Thank you.