The application is Oracle process, and it took 7 hours long,versus about 3 hours normally. When the slowness is happening, the load is very high. There are a lot of CPU's and memory, so they are not the issue. Also the database is located on a NFS file system, but no latency on the storage based on the storage tool. Oracle OEM looks fine as well on Oracle performance.
As you can see the following, a lot of Oracle processes are hanging on I/O, "D" on the state, but are they the cause and the result of the slowness?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
40415 root 20 0 0 0 0 S 18.5 0.0 0:03.72 flush-0:37
2449 root 20 0 0 0 0 S 11.6 0.0 63:23.41 rpciod/0
40225 cuorsa 20 0 24.5g 55m 30m D 5.6 0.1 0:30.20 oracle_40225_fd
2464 root 20 0 0 0 0 S 4.3 0.0 34:43.50 nfsiod
189 root 20 0 0 0 0 S 3.3 0.0 13:46.78 kswapd0
9454 cuorsa 20 0 24.4g 1.2g 1.2g D 2.0 2.6 0:44.37 ora_p00u_appprd
9462 cuorsa 20 0 24.4g 1.5g 1.4g D 2.0 3.1 1:25.53 ora_p00y_appprd
9476 cuorsa 20 0 24.4g 1.1g 1.0g D 2.0 2.2 0:16.94 ora_p015_appprd
2458 root 20 0 0 0 0 S 1.7 0.0 1:03.90 rpciod/9
9440 cuorsa 20 0 24.4g 1.7g 1.6g D 1.7 3.5 1:36.53 ora_p00o_appprd
9442 cuorsa 20 0 24.5g 1.7g 1.7g D 1.7 3.7 1:33.58 ora_p00p_appprd
9446 cuorsa 20 0 24.4g 1.6g 1.6g D 1.7 3.3 1:32.11 ora_p00q_appprd
9448 cuorsa 20 0 24.5g 1.7g 1.6g D 1.7 3.5 1:31.14 ora_p00r_appprd
9458 cuorsa 20 0 24.5g 1.5g 1.5g D 1.7 3.1 1:24.59 ora_p00w_appprd
9466 cuorsa 20 0 24.4g 1.2g 1.2g D 1.7 2.5 0:29.43 ora_p010_appprd
9472 cuorsa 20 0 24.4g 1.1g 1.1g D 1.7 2.4 0:29.69 ora_p013_appprd
9474 cuorsa 20 0 24.4g 1.1g 1.1g D 1.7 2.3 0:17.59 ora_p014_appprd
9482 cuorsa 20 0 24.4g 994m 984m D 1.7 2.1 0:17.98 ora_p018_appprd
9488 cuorsa 20 0 24.4g 896m 885m D 1.7 1.9 0:18.51 ora_p01b_appprd
9450 cuorsa 20 0 24.4g 1.4g 1.4g D 1.3 3.0 0:44.31 ora_p00s_appprd
9452 cuorsa 20 0 24.4g 1.4g 1.4g D 1.3 3.0 0:45.82 ora_p00t_appprd
9456 cuorsa 20 0 24.5g 1.4g 1.4g D 1.3 2.9 0:43.30 ora_p00v_appprd
9460 cuorsa 20 0 24.4g 1.5g 1.5g D 1.3 3.2 1:25.59 ora_p00x_appprd
9464 cuorsa 20 0 24.4g 1.3g 1.3g D 1.3 2.7 1:21.41 ora_p00z_appprd
9468 cuorsa 20 0 24.4g 1.2g 1.2g D 1.3 2.5 0:29.69 ora_p011_appprd
We simulated a new test environment, and the application is running fine there. So, my question, what can I do to identify the root cause of the slowness?
Thank you!