Skip to Main Content

Oracle Database Discussions

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

Help required with AWR report, I/O bound database.Need recommendation

User_3ZQRHApr 8 2015 — edited Apr 15 2015

I am a newbie DBA and faced with a task for performance improvement of the database as users are complaining of response time of the report

Our database is running on Release 11.2.0.3.0 running on AIX based systems ( 64 bits) .

I tried running the below query dealing with performance metrics on the database and below is the result.

select METRIC_NAME,

  VALUE

from SYS.V_$SYSMETRIC

where METRIC_NAME IN ('Database CPU Time Ratio',

  'Database Wait Time Ratio') AND

  INTSIZE_CSEC =

  (select max(INTSIZE_CSEC) from SYS.V_$SYSMETRIC);

DateDatabase Wait Time RatioDatabase CPU Time Ratio
25th March 19:3084.390381404002415.6096185959976
26th March 11:0090.01737566637319.98262433362687
26th March 14:30
94.53093998
5.46906001616506
26th March 18:0095.8742566434613
4.125743357
27th March 15:00
96.8489591
3.15104090052205

I read in one Oracle blog that in a healthy database the Database CPU Time Ratio should be around 90-95% but in my case it is between 3 to 15%.So it seems that the database us busy waiting than processing.

When I tried to drill down I found the I/O class to the "biggest" wait event, always at the top. I used the following query

select WAIT_CLASS,

  TOTAL_WAITS,

  round(100 * (TOTAL_WAITS / SUM_WAITS),2) PCT_WAITS,

  ROUND((TIME_WAITED / 100),2) TIME_WAITED_SECS,

  round(100 * (TIME_WAITED / SUM_TIME),2) PCT_TIME

from

(select WAIT_CLASS,

  TOTAL_WAITS,

  TIME_WAITED

from V$SYSTEM_WAIT_CLASS

where WAIT_CLASS != 'Idle'),

(select sum(TOTAL_WAITS) SUM_WAITS,

  sum(TIME_WAITED) SUM_TIME

from V$SYSTEM_WAIT_CLASS

where WAIT_CLASS != 'Idle')

order by 5 desc;

   

Total No of  Wait% of Wait TimeWaited in Secs% of Wait Time
108972408927.383641194.3684.46
113372974727.423832058.7684.79
117216672527.733990623.8384.86
121623133128.184106292.0784.97
133897169328.854564967.0785.74

After more or less identifying that I/O is the main bottel neck, I executed the worst performing ( heavy reports) at the same time and took the AWR report.Below is the snapshot of the report. I could see from the Load Profile that "Physical Reads" is around 5000/sec and Hard parsing in 1.2 per secs. Top 5 Waiting Events are all of class "I/O" with direct path read having highest wait time ( 56 ms). Is there anything else I can infer from the "Top 5 Waiting Events". I have also attached the Instance Stats related to physical read I/O request , bytes etc as well as break up of IO stats. Is my understanding correct that I/O is the main constraint ?....

2nd Question is how to improve/eliminate the bottle neck ? Is it hardware or SQL query ? If it is the SQL query, which section of SQL statistics should U be looking ? SQL ordered by "elapsed time" or "User I/O wait time" or "Read" or "Physical read"

Looking forward for co operation from all.Thanks in advance

Load Profile

Per SecondPer TransactionPer ExecPer Call
DB Time(s):28.037.90.090.44
DB CPU(s):2.12.80.010.03
Redo size:6,183,381.28,366,555.1
Logical reads:115,584.9156,394.5
Block changes:16,055.821,724.6
Physical reads:5,877.47,952.6
Physical writes:1,287.01,741.4
User calls:63.686.0
Parses:9.512.8
Hard parses:1.21.7
W/A MB processed:18.324.7
Logons:0.71.0
Executes:324.6439.3
Rollbacks:0.00.0
Transactions:0.7

Instance Efficiency Percentages (Target 100%)

Buffer Nowait %:99.27Redo NoWait %:100.00
Buffer Hit %:98.85In-memory Sort %:99.56
Library Hit %:99.06Soft Parse %:86.84
Execute to Parse %:97.09Latch Hit %:98.11
Parse CPU to Parse Elapsd %:24.57% Non-Parse CPU:79.16

Shared Pool Statistics

BeginEnd
Memory Usage %:73.8469.52
% SQL with executions>1:99.6899.17
% Memory for SQL w/exec>1:96.8696.23

Top 5 Timed Foreground Events

EventWaitsTime(s)Avg wait (ms)% DB timeWait Class
read by other session2,690,76123,477923.35User I/O
db file sequential read2,946,39613,238413.16User I/O
DB CPU 7,503 7.46
direct path read117,9386,617566.58User I/O
direct path read temp308,5725,340175.31User I/O

The foreground Wait calss shows that User I/O accounting to 53% of DB time

Foreground Wait Class

  • s - second, ms - millisecond - 1000th of a second
  • ordered by wait time desc, waits desc
  • %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
  • Captured Time accounts for 64.6% of Total DB time 100,564.61 (s)
  • Total FG Wait Time: 57,498.11 (s) DB CPU time: 7,502.80 (s)
Wait ClassWaits%Time -outsTotal Wait Time (s)Avg wait (ms)%DB time
User I/O6,322,726054,060953.76
DB CPU 7,503 7.46
Concurrency603,64902,20842.20
Other43,872,31610068100.68
Commit36,2220424120.42
Network323,39406800.07
Configuration19017301560.03
Application970262660.03
System I/O5310240.00

Instance Activity Stats

physical read IO requests5,329,6171,484.142,008.15
physical read bytes691,604,488,192192,591,107.23260,589,483.12
physical read total IO requests5,357,5001,491.902,018.65
physical read total bytes714,057,507,328198,843,599.64269,049,550.61
physical read total multi block requests610,350169.96229.97
physical reads21,106,0945,877.417,952.56
physical reads cache4,599,1061,280.711,732.90
physical reads cache prefetch1,577,898439.40594.54
physical reads direct16,506,9884,596.706,219.66
physical reads direct (lob)350.010.01
physical reads direct temporary tablespace2,702,275752.501,018.19
physical reads prefetch warmup00.000.00

IOStat by Function summary

  • 'Data' columns suffixed with M,G,T,P are in multiples of 1024 other columns suffixed with K,M,G,T,P are in multiples of 1000
  • ordered by (Data Read + Write) desc
Function NameReads: DataReqs per secData per secWrites: DataReqs per secData per secWaits: CountAvg Tm(ms)
Direct Reads498.2G527.18142.07310G23.472.860160
Buffer Cache Reads140.4G938.7240.03340M0.000M3020.7K4.54
Direct Writes2G1.20.57726895.3G388.3627.16080
Others24.5G24.916.9960027.3G14.787.7890868.4K7.10
DBWR1M0.01.00027829.7G193.028.46855400.73
LGWR4M0.08.00111321.4G26.556.1071257.3K14.11
Streams AQ1M0.00.0002780M0.000M813.13
TOTAL:665.2G1492.11189.682183.7G646.1852.38573146.4K4.77

IOStat by Filetype summary

  • 'Data' columns suffixed with M,G,T,P are in multiples of 1024 other columns suffixed with K,M,G,T,P are in multiples of 1000
  • Small Read and Large Read are average service times, in milliseconds
  • Ordered by (Data Read + Write) desc
Filetype NameReads: DataReqs per secData per secWrites: DataReqs per secData per secSmall ReadLarge Read
Data File561.6G1079.24160.13539.8G204.8911.36153.9289.82
Temp File82.6G405.3123.5529101.2G407.1328.860661.3950.23
Log File21G6.015.9898821.4G26.486.105453.05110.75
Archive Log0M0.000M21G5.995.98710
Control File110M1.71.03063167M1.18.0186573.3421.88
Other1M0.07.00027850M0.48.013923157.08
TOTAL:665.3G1492.35189.709183.6G646.1752.347320.3984.38

IOStat by Function/Filetype summary

  • 'Data' columns suffixed with M,G,T,P are in multiples of 1024 other columns suffixed with K,M,G,T,P are in multiples of 1000
  • Ordered by (Data Read + Write) desc for each function
Function/File NameReads: DataReqs per secData per secWrites: DataReqs per secData per secWaits: CountAvg Tm(ms)
Direct Reads 498.2G527.23142.07410G23.472.860160
Direct Reads (Data File) 498.2G527.23142.07410G23.472.860160
Buffer Cache Reads 140.4G938.7540.03610M0.000M2974.3K4.28
Buffer Cache Reads (Data File) 140.4G938.7540.03610M0.000M2974.3K4.28
Direct Writes 2G1.20.57726895.3G388.3627.16160
Direct Writes (Data File) 2G1.20.57726895.3G388.3627.16160
Others 24.6G24.857.0057427.3G14.317.7829566K6.47
Comments
Locked Post
New comments cannot be posted to this locked post.
Post Details
Locked on May 13 2015
Added on Apr 8 2015
40 comments
14,407 views