Infrastructure Software

Hello All ,

We are frequently getting "cannot fork: Resource temporarily unavailable" in one of our oracle database server.


Messages from /var/adm/messages
===============================

Oct 26 14:48:09 node01 sendmail[2660]: [ID 801593 mail.crit] NOQUEUE: SYSERR(root): daemon: cannot fork: Resource temporarily unavailable
Oct 26 14:48:09 node01 sendmail[2660]: [ID 801593 mail.crit] NOQUEUE: SYSERR(root): daemon: cannot fork: Resource temporarily unavailable
Oct 26 15:01:09 node01 sendmail[2660]: [ID 801593 mail.crit] NOQUEUE: SYSERR(root): daemon: cannot fork: Resource temporarily unavailable
Oct 26 15:01:09 node01 sendmail[2660]: [ID 801593 mail.crit] NOQUEUE: SYSERR(root): daemon: cannot fork: Resource temporarily unavailable
Oct 26 15:57:06 node01 svc.startd[7]: [ID 462725 daemon.warning] svc:/system/name-service-cache:default: Couldn't fork to execute method /lib/svc/method/svc-nscd: Resource temporarily unavailable
Oct 26 15:57:06 node01 svc.startd[7]: [ID 462725 daemon.warning] svc:/system/name-service-cache:default: Couldn't fork to execute method /lib/svc/method/svc-nscd: Resource temporarily unavailable
Oct 26 16:50:13 node01 sshd[7134]: [ID 800047 auth.error] error: fork: Resource temporarily unavailable
Oct 27 07:30:10 node01 sendmail[7451]: [ID 801593 mail.crit] o9R6UAar007451: SYSERR(root): deliver: fork 1: Resource temporarily unavailable
Oct 27 07:30:10 node01 sendmail[7451]: [ID 801593 mail.crit] o9R6UAar007451: SYSERR(root): deliver: fork 1: Resource temporarily unavailable
Oct 27 08:05:00 node01 sendmail[18044]: [ID 801593 mail.crit] o9R750nJ018044: SYSERR(root): doworklist: cannot fork: Resource temporarily unavailable
Oct 27 08:05:00 node01 sendmail[18044]: [ID 801593 mail.crit] o9R750nJ018044: SYSERR(root): doworklist: cannot fork: Resource temporarily unavailable
Oct 27 08:25:09 node01 sendmail[2660]: [ID 801593 mail.crit] NOQUEUE: SYSERR(root): daemon: cannot fork: Resource temporarily unavailable
Oct 27 08:25:09 node01 sendmail[2660]: [ID 801593 mail.crit] NOQUEUE: SYSERR(root): daemon: cannot fork: Resource temporarily unavailable
Oct 27 10:40:48 node01 inetd[2523]: [ID 702911 daemon.error] Unable to fork inetd_start method of instance svc:/network/vnetd/tcp:default: Resource temporarily unavailable
Oct 27 10:40:48 node01 inetd[2523]: [ID 702911 daemon.error] Unable to fork inetd_start method of instance svc:/network/vnetd/tcp:default: Resource temporarily unavailable



Few months back, we have applied latest patch cluster and increased swap space. But still we getting "cannot fork: Resource temporarily unavailable".


We confused about this messages, when we login to server and checked the swap and memory usages are normal.


So, we planned to monitor the process usage. We can monitor the swap and physical memory using sar, vmstat


Please let me know how to monitor the below


1) Total no. of process can run in the server
2) no. of process running in the server at a given point of time
3) no. of process currently running for each and every users.
4) no. of file descriptor opened at a given time
5) no. of files opened by a process
6) file descriptor limit, process limit


Please help me to identify the Root cause for this incident.


Regards,
Rajesh.