I am a newbie DBA and faced with a task for performance improvement of the database as users are complaining of response time of the report
Our database is running on Release 11.2.0.3.0 running on AIX based systems ( 64 bits) .
I tried running the below query dealing with performance metrics on the database and below is the result.
select METRIC_NAME,
VALUE
from SYS.V_$SYSMETRIC
where METRIC_NAME IN ('Database CPU Time Ratio',
'Database Wait Time Ratio') AND
INTSIZE_CSEC =
(select max(INTSIZE_CSEC) from SYS.V_$SYSMETRIC);
| Date | Database Wait Time Ratio | Database CPU Time Ratio |
|---|
| 25th March 19:30 | 84.3903814040024 | 15.6096185959976 |
| 26th March 11:00 | 90.0173756663731 | 9.98262433362687 |
| 26th March 14:30 | | 5.46906001616506 |
| 26th March 18:00 | 95.8742566434613 | |
| 27th March 15:00 | | 3.15104090052205 |
I read in one Oracle blog that in a healthy database the Database CPU Time Ratio should be around 90-95% but in my case it is between 3 to 15%.So it seems that the database us busy waiting than processing.
When I tried to drill down I found the I/O class to the "biggest" wait event, always at the top. I used the following query
select WAIT_CLASS,
TOTAL_WAITS,
round(100 * (TOTAL_WAITS / SUM_WAITS),2) PCT_WAITS,
ROUND((TIME_WAITED / 100),2) TIME_WAITED_SECS,
round(100 * (TIME_WAITED / SUM_TIME),2) PCT_TIME
from
(select WAIT_CLASS,
TOTAL_WAITS,
TIME_WAITED
from V$SYSTEM_WAIT_CLASS
where WAIT_CLASS != 'Idle'),
(select sum(TOTAL_WAITS) SUM_WAITS,
sum(TIME_WAITED) SUM_TIME
from V$SYSTEM_WAIT_CLASS
where WAIT_CLASS != 'Idle')
order by 5 desc;
| Total No of Wait | % of Wait Time | Waited in Secs | % of Wait Time |
| 1089724089 | 27.38 | 3641194.36 | 84.46 |
| 1133729747 | 27.42 | 3832058.76 | 84.79 |
| 1172166725 | 27.73 | 3990623.83 | 84.86 |
| 1216231331 | 28.18 | 4106292.07 | 84.97 |
| 1338971693 | 28.85 | 4564967.07 | 85.74 |
After more or less identifying that I/O is the main bottel neck, I executed the worst performing ( heavy reports) at the same time and took the AWR report.Below is the snapshot of the report. I could see from the Load Profile that "Physical Reads" is around 5000/sec and Hard parsing in 1.2 per secs. Top 5 Waiting Events are all of class "I/O" with direct path read having highest wait time ( 56 ms). Is there anything else I can infer from the "Top 5 Waiting Events". I have also attached the Instance Stats related to physical read I/O request , bytes etc as well as break up of IO stats. Is my understanding correct that I/O is the main constraint ?....
2nd Question is how to improve/eliminate the bottle neck ? Is it hardware or SQL query ? If it is the SQL query, which section of SQL statistics should U be looking ? SQL ordered by "elapsed time" or "User I/O wait time" or "Read" or "Physical read"
Looking forward for co operation from all.Thanks in advance
Load Profile
| Per Second | Per Transaction | Per Exec | Per Call |
|---|
| DB Time(s): | 28.0 | 37.9 | 0.09 | 0.44 |
| DB CPU(s): | 2.1 | 2.8 | 0.01 | 0.03 |
| Redo size: | 6,183,381.2 | 8,366,555.1 | | |
| Logical reads: | 115,584.9 | 156,394.5 | | |
| Block changes: | 16,055.8 | 21,724.6 | | |
| Physical reads: | 5,877.4 | 7,952.6 | | |
| Physical writes: | 1,287.0 | 1,741.4 | | |
| User calls: | 63.6 | 86.0 | | |
| Parses: | 9.5 | 12.8 | | |
| Hard parses: | 1.2 | 1.7 | | |
| W/A MB processed: | 18.3 | 24.7 | | |
| Logons: | 0.7 | 1.0 | | |
| Executes: | 324.6 | 439.3 | | |
| Rollbacks: | 0.0 | 0.0 | | |
| Transactions: | 0.7 | | | |
Instance Efficiency Percentages (Target 100%)
| Buffer Nowait %: | 99.27 | Redo NoWait %: | 100.00 |
| Buffer Hit %: | 98.85 | In-memory Sort %: | 99.56 |
| Library Hit %: | 99.06 | Soft Parse %: | 86.84 |
| Execute to Parse %: | 97.09 | Latch Hit %: | 98.11 |
| Parse CPU to Parse Elapsd %: | 24.57 | % Non-Parse CPU: | 79.16 |
Shared Pool Statistics
| Begin | End |
|---|
| Memory Usage %: | 73.84 | 69.52 |
| % SQL with executions>1: | 99.68 | 99.17 |
| % Memory for SQL w/exec>1: | 96.86 | 96.23 |
Top 5 Timed Foreground Events
| Event | Waits | Time(s) | Avg wait (ms) | % DB time | Wait Class |
|---|
| read by other session | 2,690,761 | 23,477 | 9 | 23.35 | User I/O |
| db file sequential read | 2,946,396 | 13,238 | 4 | 13.16 | User I/O |
| DB CPU | | 7,503 | | 7.46 | |
| direct path read | 117,938 | 6,617 | 56 | 6.58 | User I/O |
| direct path read temp | 308,572 | 5,340 | 17 | 5.31 | User I/O |
The foreground Wait calss shows that User I/O accounting to 53% of DB time
Foreground Wait Class
- s - second, ms - millisecond - 1000th of a second
- ordered by wait time desc, waits desc
- %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
- Captured Time accounts for 64.6% of Total DB time 100,564.61 (s)
- Total FG Wait Time: 57,498.11 (s) DB CPU time: 7,502.80 (s)
| Wait Class | Waits | %Time -outs | Total Wait Time (s) | Avg wait (ms) | %DB time |
|---|
| User I/O | 6,322,726 | 0 | 54,060 | 9 | 53.76 |
| DB CPU | | | 7,503 | | 7.46 |
| Concurrency | 603,649 | 0 | 2,208 | 4 | 2.20 |
| Other | 43,872,316 | 100 | 681 | 0 | 0.68 |
| Commit | 36,222 | 0 | 424 | 12 | 0.42 |
| Network | 323,394 | 0 | 68 | 0 | 0.07 |
| Configuration | 190 | 17 | 30 | 156 | 0.03 |
| Application | 97 | 0 | 26 | 266 | 0.03 |
| System I/O | 531 | 0 | 2 | 4 | 0.00 |
Instance Activity Stats
| physical read IO requests | 5,329,617 | 1,484.14 | 2,008.15 |
| physical read bytes | 691,604,488,192 | 192,591,107.23 | 260,589,483.12 |
| physical read total IO requests | 5,357,500 | 1,491.90 | 2,018.65 |
| physical read total bytes | 714,057,507,328 | 198,843,599.64 | 269,049,550.61 |
| physical read total multi block requests | 610,350 | 169.96 | 229.97 |
| physical reads | 21,106,094 | 5,877.41 | 7,952.56 |
| physical reads cache | 4,599,106 | 1,280.71 | 1,732.90 |
| physical reads cache prefetch | 1,577,898 | 439.40 | 594.54 |
| physical reads direct | 16,506,988 | 4,596.70 | 6,219.66 |
| physical reads direct (lob) | 35 | 0.01 | 0.01 |
| physical reads direct temporary tablespace | 2,702,275 | 752.50 | 1,018.19 |
| physical reads prefetch warmup | 0 | 0.00 | 0.00 |
IOStat by Function summary
- 'Data' columns suffixed with M,G,T,P are in multiples of 1024 other columns suffixed with K,M,G,T,P are in multiples of 1000
- ordered by (Data Read + Write) desc
| Function Name | Reads: Data | Reqs per sec | Data per sec | Writes: Data | Reqs per sec | Data per sec | Waits: Count | Avg Tm(ms) |
|---|
| Direct Reads | 498.2G | 527.18 | 142.073 | 10G | 23.47 | 2.86016 | 0 | |
| Buffer Cache Reads | 140.4G | 938.72 | 40.0334 | 0M | 0.00 | 0M | 3020.7K | 4.54 |
| Direct Writes | 2G | 1.20 | .577268 | 95.3G | 388.36 | 27.1608 | 0 | |
| Others | 24.5G | 24.91 | 6.99600 | 27.3G | 14.78 | 7.78908 | 68.4K | 7.10 |
| DBWR | 1M | 0.01 | .000278 | 29.7G | 193.02 | 8.46855 | 40 | 0.73 |
| LGWR | 4M | 0.08 | .001113 | 21.4G | 26.55 | 6.10712 | 57.3K | 14.11 |
| Streams AQ | 1M | 0.00 | .000278 | 0M | 0.00 | 0M | 8 | 13.13 |
| TOTAL: | 665.2G | 1492.11 | 189.682 | 183.7G | 646.18 | 52.3857 | 3146.4K | 4.77 |
IOStat by Filetype summary
- 'Data' columns suffixed with M,G,T,P are in multiples of 1024 other columns suffixed with K,M,G,T,P are in multiples of 1000
- Small Read and Large Read are average service times, in milliseconds
- Ordered by (Data Read + Write) desc
| Filetype Name | Reads: Data | Reqs per sec | Data per sec | Writes: Data | Reqs per sec | Data per sec | Small Read | Large Read |
|---|
| Data File | 561.6G | 1079.24 | 160.135 | 39.8G | 204.89 | 11.3615 | 3.92 | 89.82 |
| Temp File | 82.6G | 405.31 | 23.5529 | 101.2G | 407.13 | 28.8606 | 61.39 | 50.23 |
| Log File | 21G | 6.01 | 5.98988 | 21.4G | 26.48 | 6.10545 | 3.05 | 110.75 |
| Archive Log | 0M | 0.00 | 0M | 21G | 5.99 | 5.98710 | | |
| Control File | 110M | 1.71 | .030631 | 67M | 1.18 | .018657 | 3.34 | 21.88 |
| Other | 1M | 0.07 | .000278 | 50M | 0.48 | .013923 | 157.08 | |
| TOTAL: | 665.3G | 1492.35 | 189.709 | 183.6G | 646.17 | 52.3473 | 20.39 | 84.38 |
IOStat by Function/Filetype summary
- 'Data' columns suffixed with M,G,T,P are in multiples of 1024 other columns suffixed with K,M,G,T,P are in multiples of 1000
- Ordered by (Data Read + Write) desc for each function
| Function/File Name | Reads: Data | Reqs per sec | Data per sec | Writes: Data | Reqs per sec | Data per sec | Waits: Count | Avg Tm(ms) |
|---|
| Direct Reads | 498.2G | 527.23 | 142.074 | 10G | 23.47 | 2.86016 | 0 | |
| Direct Reads (Data File) | 498.2G | 527.23 | 142.074 | 10G | 23.47 | 2.86016 | 0 | |
| Buffer Cache Reads | 140.4G | 938.75 | 40.0361 | 0M | 0.00 | 0M | 2974.3K | 4.28 |
| Buffer Cache Reads (Data File) | 140.4G | 938.75 | 40.0361 | 0M | 0.00 | 0M | 2974.3K | 4.28 |
| Direct Writes | 2G | 1.20 | .577268 | 95.3G | 388.36 | 27.1616 | 0 | |
| Direct Writes (Data File) | 2G | 1.20 | .577268 | 95.3G | 388.36 | 27.1616 | 0 | |
| Others | 24.6G | 24.85 | 7.00574 | 27.3G | 14.31 | 7.78295 | 66K | 6.47 |