Hi we are using version 12.1.0.2.0 of oracle exadata. In this database we used to see delayed block cleanout effect during monthend processing and to avoid that we manually scan the table/index blocks before hand. Also index contention is a known issue in thid DB on which we are working on design changes. But suddenly this monthend(1st nov) we experience the "delayed block cleanout effect" pretty extensively making the read queries to fail multiple times throughout the day which was not the case in previous monthends. I am trying to understand what has caused this drastic degradation in performance this time. My initial thought was it must be because of increase in number of transactions, but I did compare the "user commits"/rollbacks and not seeing significant difference as compared to last month.
1)When i compare the AWR for the last monthend to current monthend. I found the two odd top wait events were related to "system i/o(DB file parallel write)" and "configuration(flashback buf free by RVWR)" wait classes. We do have flashback enabled in this database as per business agreement, but i am trying to understand what has triggered this wait to be so high this time as compared to earlier?
2)Not much idea about the backup strategies but i see we have UNDO retention is 3600 and db_flashback_retention_target is 4320. We do have RMAN backup(L0 and L1) scheduled as per standard. So my question was as we can recover individual objects from rman backup itself so is it okay to switch off the flashback to get rid of these additional bottlenecks during write/dmls mainly?
3)I see in the "top timed event" section the number of "flashback buf free by RVWR" has been increased from ~140k to ~350k. ANd "DB file parallel write" in creased from ~22million to ~46million. So Is it correct to say that starting point is "db file parallel write" whose response time degraded and due to that the "flashback buf free by RVWR" has been coming on top and others(delayed block cleanout etc) are result of this?
Below is the comparison of AWR between two monthends, i have only posted below few key sections of the AWR comparison and not all.
| Snapshot Set | Begin Snap Id | Begin Snap Time | End Snap Id | End Snap Time | Avg Active Users | Elapsed Time (min) | DB time (min) |
| 1st | 272055 | 01-Oct-19 05:00:10 (Tue) | 272127 | 01-Oct-19 23:00:30 (Tue) | 21.3 | 1,080.30 | 23,022.60 |
| 2nd | 275023 | 01-Nov-19 05:00:02 (Fri) | 275095 | 01-Nov-19 23:00:04 (Fri) | 28.8 | 1,080.00 | 31,122.50 |
| %Diff | | | | | 35.2 | 0 | 35.2 |
| Host Configuration Comparison | | |
| 1st | 2nd | Diff | %Diff |
| Number of CPUs: | 32 | 32 | 0 | 0 |
| Number of CPU Cores: | 16 | 16 | 0 | 0 |
| Number of CPU Sockets: | 2 | 2 | 0 | 0 |
| Physical Memory: | 257973.7M | 257973.7M | 0M | 0 |
| Load at Start Snapshot: | 3.81 | 3.19 | -0.62 | -16.3 |
| Load at End Snapshot: | 3.27 | 2.38 | -0.89 | -27.2 |
| %User Time: | 26.42 | 27.35 | 0.93 | 3.5 |
| %System Time: | 4.97 | 5.21 | 0.24 | 4.8 |
| %Idle Time: | 67.52 | 66.02 | -1.49 | -2.2 |
| %IO Wait Time: | 0.01 | 0.01 | 0 | 0 |
| Cache Sizes | | | |
| | | |
| 1st (M) | 2nd (M) | Diff (M) | %Diff |
| Memory Target | | | | |
| SGA Target | | | |
| 45,056.00 | 45,056.00 | 0 | 0 |
| Buffer Cache | | | |
| 35,200.00 | 35,200.00 | 0 | 0 |
| Shared Pool | | | |
| 8,192.00 | 8,192.00 | 0 | 0 |
| Large Pool | | | |
| 1,024.00 | 1,024.00 | 0 | 0 |
| Java Pool | | | |
| 128 | 128 | 0 | 0 |
| Streams Pool | | | |
| 256 | 256 | 0 | 0 |
| PGA Target | | | |
| 46,080.00 | 46,080.00 | 0 | 0 |
| Log Buffer | 128 | 128 | 0 | 0 |
| In-Memory Area | | | | |
| Load Profile | | | | | |
| 1st per sec | 2nd per sec | %Diff | 1st per txn | 2nd per txn | %Diff |
| DB time: | 21.3 | 28.8 | 35.2 | 0 | 0.1 | 20 |
| CPU time: | 8.2 | 8.4 | 3.3 | 0 | 0 | 0 |
| Background CPU time: | 1.2 | 1.5 | 28.6 | 0 | 0 | 0 |
| Redo size (bytes): | 18,337,036.50 | 17,500,030.90 | -4.6 | 39,452.70 | 37,503.50 | -4.9 |
| Logical read (blocks): | 573,369.80 | 552,181.80 | -3.7 | 1,233.60 | 1,183.40 | -4.1 |
| Block changes: | 91,955.60 | 84,981.60 | -7.6 | 197.8 | 182.1 | -7.9 |
| Physical read (blocks): | 97,379.80 | 103,043.60 | 5.8 | 209.5 | 220.8 | 5.4 |
| Physical write (blocks): | 4,750.90 | 6,774.00 | 42.6 | 10.2 | 14.5 | 42.1 |
| Read IO requests: | 7,553.30 | 10,041.80 | 32.9 | 16.3 | 21.5 | 32.4 |
| Write IO requests: | 2,100.60 | 4,042.10 | 92.4 | 4.5 | 8.7 | 91.6 |
| Read IO (MB): | 760.8 | 805 | 5.8 | 1.6 | 1.7 | 5.5 |
| Write IO (MB): | 37.1 | 52.9 | 42.6 | 0.1 | 0.1 | 37.5 |
| IM scan rows: | 0 | 0 | 0 | 0 | 0 | 0 |
| Session Logical Read IM: | | | | | | |
| Global Cache blocks received: | 771.1 | 694.5 | -9.9 | 1.7 | 1.5 | -10.2 |
| Global Cache blocks served: | 219.7 | 186.2 | -15.2 | 0.5 | 0.4 | -14.9 |
| User calls: | 20,240.10 | 20,888.90 | 3.2 | 43.5 | 44.8 | 2.8 |
| Parses (SQL): | 2,341.60 | 2,334.50 | -0.3 | 5 | 5 | -0.8 |
| Hard parses (SQL): | 0.3 | 0.2 | -35.7 | 0 | 0 | 0 |
| SQL Work Area (MB): | 1,329.30 | 1,341.70 | 0.9 | 2.9 | 2.9 | 0.9 |
| Logons: | 1.2 | 1.5 | 18.9 | 0 | 0 | 0 |
| Executes (SQL): | 6,188.80 | 6,093.80 | -1.5 | 13.3 | 13.1 | -2 |
| Transactions: | 464.8 | 466.6 | 0.4 | | | |
| 1st | 2nd | Diff | | |
| % Blocks changed per Read: | 16 | 15.4 | -0.6 | | |
| Recursive Call %: | 7.9 | 7 | -0.9 | | |
| Rollback per transaction %: | 0 | 0 | 0 | | |
| Rows per Sort: | 2,164.90 | 1,637.50 | -527.4 | | |
| Avg DB time per Call (sec): | 0 | 0 | 0 |
| Top Timed Events | | | | | | | | | | |
| Events with a "-" did not make the Top list in this set of snapshots, but are displayed for comparison purposes | | | | |
| | | | | | | | | | |
| 1st | 2nd | | | | | | | | | |
| Event | Wait Class | Waits | Time(s) | Avg Time(ms) | %DB time | Event | Wait Class | Waits | Time(s) | Avg Time(ms) | %DB time |
| CPU time | | | 528,463.26 | | 38.26 | cell single block physical read | User I/O | 552,992,032 | 575,983.20 | 1.04 | 30.84 |
| cell single block physical read | User I/O | 399,804,976 | 463,012.67 | 1.16 | 33.52 | CPU time | | | 545,675.33 | | 29.22 |
| db file parallel write | System I/O | 22,301,041 | 222,674.79 | 9.98 | 16.12 | db file parallel write | System I/O | 46,409,738 | 522,686.29 | 11.26 | 27.99 |
| buffer busy waits | Concurrency | 30,172,532 | 117,027.57 | 3.88 | 8.47 | flashback buf free by RVWR | Configuration | 349,855 | 220,141.23 | 629.24 | 11.79 |
| flashback buf free by RVWR | Configuration | 139,899 | 82,831.01 | 592.08 | 6 | enq: TX - index contention | Concurrency | 9,578,098 | 219,165.55 | 22.88 | 11.74 |
| enq: TX - index contention | Concurrency | 6,869,458 | 65,227.55 | 9.5 | 4.72 | buffer busy waits | Concurrency | 40,070,745 | 177,680.91 | 4.43 | 9.52 |
| flashback log file write | System I/O | 3,726,607 | 39,787.97 | 10.68 | 2.88 | flashback log file write | System I/O | 3,259,125 | 54,879.57 | 16.84 | 2.94 |
| log file sequential read | System I/O | 1,354,233 | 32,101.16 | 23.7 | 2.32 | log file sequential read | System I/O | 1,350,478 | 44,940.87 | 33.28 | 2.41 |
| log file sync | Commit | 30,502,742 | 32,088.99 | 1.05 | 2.32 | Log archive I/O | System I/O | 313,570 | 42,509.85 | 135.57 | 2.28 |
| Log archive I/O | System I/O | 293,760 | 27,461.50 | 93.48 | 1.99 | log file sync | Commit | 30,215,374 | 34,641.36 | 1.15 | 1.86 |
| Time Model Statistics | | | | | | | | | | | |
| Ordered by absolute value of 'Diff' column of '% of DB time', descending (DB time statistic first, background statistics last) | | | | |
| | | | | | | | | | | |
| % of DB time | % of Total CPU time | Time (seconds) | Time per Trans (seconds) | | | | | |
| Statistic Name | 1st | 2nd | Diff | 1st | 2nd | Diff | 1st | 2nd | %Diff | 1st | 2nd | %Diff |
| DB CPU | 38.26 | 29.22 | -9.03 | 87.22 | 84.65 | 2.57 | 528,463.26 | 545,675.33 | 3.26 | 0.02 | 0.02 | 0 |
| sql execute elapsed time | 93.95 | 93.09 | -0.86 | | | | 1,297,821.48 | 1,738,320.73 | 33.94 | 0.04 | 0.06 | 50 |
| PL/SQL execution elapsed time | 0.05 | 0.29 | 0.24 | | | | 742.08 | 5,507.91 | 642.23 | 0 | 0 | 0 |
| parse time elapsed | 0.37 | 0.34 | -0.03 | | | | 5,089.26 | 6,349.44 | 24.76 | 0 | 0 | 0 |
| hard parse elapsed time | 0.08 | 0.06 | -0.02 | | | | 1,093.06 | 1,071.31 | -1.99 | 0 | 0 | 0 |
| sequence load elapsed time | 0.05 | 0.03 | -0.02 | | | | 711.38 | 591.2 | -16.89 | 0 | 0 | 0 |
| hard parse (sharing criteria) elapsed time | 0.02 | 0.04 | 0.01 | | | | 287.64 | 661.2 | 129.87 | 0 | 0 | 0 |
| connection management call elapsed time | 0.02 | 0.02 | 0 | | | | 299.81 | 315.93 | 5.38 | 0 | 0 | 0 |
| hard parse (bind mismatch) elapsed time | 0.01 | 0.01 | 0 | | | | 125.51 | 121.33 | -3.33 | 0 | 0 | 0 |
| repeated bind elapsed time | 0 | 0 | 0 | | | | 8.46 | 6.29 | -25.65 | 0 | 0 | 0 |
| failed parse elapsed time | 0 | 0 | 0 | | | | 1.66 | 4.29 | 158.43 | 0 | 0 | 0 |
| PL/SQL compilation elapsed time | 0 | 0 | 0 | | | | 5.45 | 5.54 | 1.65 | 0 | 0 | 0 |
| background elapsed time | | | | | | | 464,512.95 | 847,358.50 | 82.42 | 0.02 | 0.03 | 50 |
| background cpu time | | | | 12.78 | 15.35 | -2.57 | 77,431.40 | 98,972.32 | 27.82 | 0 | 0 | 0 |
| DB time | | | | | | | 1,381,356.23 | 1,867,348.56 | 35.18 | 0.05 | 0.06 | 20 |
| total CPU time | 43.86 | 34.52 | -9.34 | | | | 605,894.67 | 644,647.65 | 6.4 | 0.02 | 0.02 | 6.01 |
| Wait Classes | | | | | | | | | | | |
| Ordered by absolute value of 'Diff' column of '% of DB time' descending | | | | | | | | | |
| | | | | | | | | | | |
| % of DB time | # Waits/sec (Elapsed Time) | Total Wait Time (sec) | Avg Wait Time (ms) | | | | | | |
| Wait Class | 1st | 2nd | Diff | 1st | 2nd | %Diff | 1st | 2nd | %Diff | 1st | 2nd | %Diff |
| System I/O | 24.87 | 37.19 | 12.32 | 1,091.04 | 1,863.14 | 70.77 | 343,525.77 | 694,524.19 | 102.18 | 4.86 | 5.75 | 18.31 |
| Concurrency | 13.51 | 21.57 | 8.06 | 689.19 | 825.23 | 19.74 | 186,591.93 | 402,793.72 | 115.87 | 4.18 | 7.53 | 80.14 |
| User I/O | 40.24 | 34.32 | -5.92 | 6,606.87 | 8,929.27 | 35.15 | 555,877.70 | 640,826.28 | 15.28 | 1.3 | 1.11 | -14.62 |
| Configuration | 6.48 | 12.25 | 5.76 | 5.23 | 8.6 | 64.44 | 89,576.64 | 228,685.11 | 155.3 | 264.48 | 410.37 | 55.16 |
| Other | 4.1 | 5.95 | 1.85 | 1,888.72 | 4,147.47 | 119.59 | 56,629.39 | 111,176.86 | 96.32 | 0.46 | 0.41 | -10.87 |
| Cluster | 1.85 | 1.34 | -0.5 | 819.6 | 1,045.45 | 27.56 | 25,489.94 | 25,075.63 | -1.63 | 0.48 | 0.37 | -22.92 |
| Commit | 2.32 | 1.86 | -0.47 | 470.58 | 466.27 | -0.92 | 32,088.99 | 34,641.36 | 7.95 | 1.05 | 1.15 | 9.52 |
| Application | 0.13 | 0.25 | 0.12 | 0.97 | 1.5 | 54.64 | 1,850.05 | 4,656.25 | 151.68 | 29.44 | 48.01 | 63.08 |
| Network | 0.07 | 0.04 | -0.03 | 18,096.89 | 18,724.98 | 3.47 | 940.24 | 776.87 | -17.38 | 0 | 0 | 0 |
| Wait Events | | | | | | | | | | | | |
| Ordered by absolute value of 'Diff' column of '% of DB time' descending (idle events last) | | | | | | | | |
| | | | | | | | | | | | |
| | % of DB time | # Waits/sec (Elapsed Time) | Total Wait Time (sec) | Avg Wait Time (ms) | | | | | | |
| Event | Wait Class | 1st | 2nd | Diff | 1st | 2nd | %Diff | 1st | 2nd | %Diff | 1st | 2nd | %Diff |
| db file parallel write | System I/O | 16.12 | 27.99 | 11.87 | 344.05 | 716.18 | 108.16 | 222,674.79 | 522,686.29 | 134.73 | 9.98 | 11.26 | 12.83 |
| enq: TX - index contention | Concurrency | 4.72 | 11.74 | 7.01 | 105.98 | 147.81 | 39.47 | 65,227.55 | 219,165.55 | 236 | 9.5 | 22.88 | 140.84 |
| flashback buf free by RVWR | Configuration | 6 | 11.79 | 5.79 | 2.16 | 5.4 | 150 | 82,831.01 | 220,141.23 | 165.77 | 592.08 | 629.24 | 6.28 |
| cell single block physical read | User I/O | 33.52 | 30.84 | -2.67 | 6,167.96 | 8,533.58 | 38.35 | 463,012.67 | 575,983.20 | 24.4 | 1.16 | 1.04 | -10.34 |
| direct path read | User I/O | 1.97 | 0 | -1.96 | 80.93 | 0.75 | -99.07 | 27,180.64 | 87.36 | -99.68 | 5.18 | 1.79 | -65.44 |
| wait for a undo record | Other | 0 | 1.5 | 1.5 | 0 | 6.58 | 100 | 0 | 27,931.83 | 100 | 0 | 65.48 | 100 |
| flashback log switch | Other | 0.36 | 1.45 | 1.09 | 0.13 | 0.48 | 269.23 | 5,016.02 | 27,102.46 | 440.32 | 583.05 | 867.83 | 48.84 |
| buffer busy waits | Concurrency | 8.47 | 9.52 | 1.04 | 465.48 | 618.36 | 32.84 | 117,027.57 | 177,680.91 | 51.83 | 3.88 | 4.43 | 14.18 |
| reliable message | Other | 1.1 | 0.23 | -0.86 | 15.42 | 4.11 | -73.35 | 15,143.18 | 4,371.77 | -71.13 | 15.15 | 16.4 | 8.25 |
| direct path read temp | User I/O | 1.67 | 0.9 | -0.76 | 18.78 | 13.89 | -26.04 | 23,061.74 | 16,894.29 | -26.74 | 18.95 | 18.76 | -1 |
| cell multiblock physical read | User I/O | 1.65 | 0.99 | -0.67 | 163.17 | 150.24 | -7.92 | 22,816.59 | 18,403.96 | -19.34 | 2.16 | 1.89 | -12.5 |
| log file sync | Commit | 2.32 | 1.86 | -0.47 | 470.58 | 466.27 | -0.92 | 32,088.99 | 34,641.36 | 7.95 | 1.05 | 1.15 | 9.52 |
| PX Deq: Slave Session Stats | Other | 0.47 | 0.02 | -0.45 | 1.76 | 2.67 | 51.7 | 6,463.75 | 368.36 | -94.3 | 56.56 | 2.13 | -96.23 |
| log file parallel write | System I/O | 1.32 | 0.97 | -0.35 | 427.42 | 393.68 | -7.89 | 18,240.47 | 18,039.00 | -1.1 | 0.66 | 0.71 | 7.58 |
| read by other session | User I/O | 0.26 | 0.55 | 0.29 | 38.92 | 40.16 | 3.19 | 3,572.25 | 10,279.20 | 187.75 | 1.42 | 3.95 | 178.17 |
| Log archive I/O | System I/O | 1.99 | 2.28 | 0.29 | 4.53 | 4.84 | 6.84 | 27,461.50 | 42,509.85 | 54.8 | 93.48 | 135.57 | 45.03 |
| ASM file metadata operation | Other | 0.73 | 0.99 | 0.26 | 9.74 | 9.7 | -0.41 | 10,106.34 | 18,480.02 | 82.86 | 16.01 | 29.41 | 83.7 |
| Key Instance Activity Stats | | | | | | | | | | | |
| Ordered by statistic name | | | | | | | | | | | |
| | | | | | | | | | | |
| Value | per Second (DB time) | per Second (Elapsed Time) | per Trans | | | | | | | |
| Statistic | 1st | 2nd | %Diff | 1st | 2nd | %Diff | 1st | 2nd | %Diff | 1st | 2nd | %Diff |
| db block changes | 5,960,523,024 | 5,506,966,155 | -7.61 | 4,314.98 | 2,949.08 | -31.65 | 91,955.55 | 84,981.58 | -7.58 | 197.84 | 182.12 | -7.95 |
| execute count | 401,158,419 | 394,888,894 | -1.56 | 290.41 | 211.47 | -27.18 | 6,188.84 | 6,093.79 | -1.54 | 13.32 | 13.06 | -1.95 |
| gc cr block receive time | 1,669,707 | 1,766,843 | 5.82 | 1.21 | 0.95 | -21.49 | 25.76 | 27.27 | 5.86 | 0.06 | 0.06 | 0 |
| gc cr blocks received | 23,064,693 | 23,417,669 | 1.53 | 16.7 | 12.54 | -24.91 | 355.83 | 361.37 | 1.56 | 0.77 | 0.77 | 0 |
| gc current block receive time | 2,755,570 | 1,104,899 | -59.9 | 1.99 | 0.59 | -70.35 | 42.51 | 17.05 | -59.89 | 0.09 | 0.04 | -55.56 |
| gc current blocks received | 26,915,903 | 21,589,244 | -19.79 | 19.49 | 11.56 | -40.69 | 415.24 | 333.16 | -19.77 | 0.89 | 0.71 | -20.22 |
| logons cumulative | 79,146 | 94,178 | 18.99 | 0.06 | 0.05 | -16.67 | 1.22 | 1.45 | 18.85 | 0 | 0 | 0 |
| opened cursors cumulative | 163,696,327 | 159,404,749 | -2.62 | 118.5 | 85.36 | -27.97 | 2,525.41 | 2,459.88 | -2.59 | 5.43 | 5.27 | -2.95 |
| parse count (total) | 151,781,535 | 151,278,949 | -0.33 | 109.88 | 81.01 | -26.27 | 2,341.60 | 2,334.48 | -0.3 | 5.04 | 5 | -0.79 |
| parse time elapsed | 420,941 | 508,405 | 20.78 | 0.3 | 0.27 | -10 | 6.49 | 7.85 | 20.96 | 0.01 | 0.02 | 100 |
| physical reads | 6,312,117,715 | 6,677,416,939 | 5.79 | 4,569.51 | 3,575.88 | -21.74 | 97,379.75 | 103,043.56 | 5.82 | 209.52 | 220.83 | 5.4 |
| physical writes | 307,952,792 | 438,965,566 | 42.54 | 222.94 | 235.07 | 5.44 | 4,750.92 | 6,773.96 | 42.58 | 10.22 | 14.52 | 42.07 |
| redo size | 1,188,599,573,792 | 1,134,034,937,892 | -4.59 | 860,458.40 | 607,296.87 | -29.42 | 18,337,036.51 | 17,500,030.91 | -4.56 | 39,452.66 | 37,503.47 | -4.94 |
| session cursor cache hits | 16,018,389 | 15,796,254 | -1.39 | 11.6 | 8.46 | -27.07 | 247.12 | 243.76 | -1.36 | 0.53 | 0.52 | -1.89 |
| session logical reads | 37,165,610,353 | 35,782,418,780 | -3.72 | 26,905.16 | 19,162.15 | -28.78 | 573,369.85 | 552,181.78 | -3.7 | 1,233.62 | 1,183.35 | -4.07 |
| user calls | 1,311,953,295 | 1,353,637,036 | 3.18 | 949.76 | 724.9 | -23.68 | 20,240.07 | 20,888.85 | 3.21 | 43.55 | 44.77 | 2.8 |
| user commits | 30,127,137 | 30,237,993 | 0.37 | 21.81 | 16.19 | -25.77 | 464.78 | 466.62 | 0.4 | 1 | 1 | 0 |
| user rollbacks | 100 | 141 | 41 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| workarea executions - onepass | 185 | 61 | -67.03 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| workarea executions - optimal | 32,184,548 | 32,572,258 | 1.2 | 23.3 | 17.44 | -25.15 | 496.52 | 502.64 | 1.23 | 1.07 | 1.08 | 0.93 |
| Other Instance Activity Stats | | | | | | | | | | | |
| Ordered by statistic name | | | | | | | | | | | |
| | | | | | | | | | | |
| Value | per Second (DB time) | per Second (Elapsed Time) | per Trans | | | | | | | |
| Statistic | 1st | 2nd | %Diff | 1st | 2nd | %Diff | 1st | 2nd | %Diff | 1st | 2nd | %Diff |
| transaction rollbacks | 1,148,774 | 1,144,077 | -0.41 | 0.83 | 0.61 | -26.51 | 17.72 | 17.65 | -0.4 | 0.04 | 0.04 | 0 |
| transaction tables consistent read rollbacks | 382,182 | 383,790 | 0.42 | 0.28 | 0.21 | -25 | 5.9 | 5.92 | 0.34 | 0.01 | 0.01 | 0 |
| transaction tables consistent reads - undo records applied | 94,081,724 | 174,332,736 | 85.3 | 68.11 | 93.36 | 37.07 | 1,451.44 | 2,690.24 | 85.35 | 3.12 | 5.77 | 84.94 |
| rollback changes - undo records applied | 31,833,455 | 63,051,939 | 98.07 | 23.05 | 33.77 | 46.51 | 491.11 | 973 | 98.12 | 1.06 | 2.09 | 97.17 |
| rollbacks only - consistent read gets | 2,463,723 | 3,084,067 | 25.18 | 1.78 | 1.65 | -7.3 | 38.01 | 47.59 | 25.2 | 0.08 | 0.1 | 25 |
| cleanout - number of ktugct calls | 468,851,008 | 470,343,109 | 0.32 | 339.41 | 251.88 | -25.79 | 7,233.17 | 7,258.17 | 0.35 | 15.56 | 15.55 | -0.06 |
| cleanouts and rollbacks - consistent read gets | 49,846,256 | 17,909,477 | -64.07 | 36.09 | 9.59 | -73.43 | 769 | 276.37 | -64.06 | 1.65 | 0.59 | -64.24 |
| cleanouts only - consistent read gets | 320,381,864 | 286,709,747 | -10.51 | 231.93 | 153.54 | -33.8 | 4,942.67 | 4,424.40 | -10.49 | 10.63 | 9.48 | -10.82 |
| | | | | | | | | | | | |
| Operating System Statistics | | | | | | | | | | |
| Ordered by 'Statistic Name' | | | | | | | | | | |
| | | | | | | | | | |
| Value (centi-seconds) | per Second (DB time) | per Second (Elapsed Time) | per Trans | | | | | | |
| Statistic Name | 1st | 2nd | 1st | 2nd | %Diff | 1st | 2nd | %Diff | 1st | 2nd | %Diff |
| BUSY_TIME | 66,313,641 | 69,228,136 | 48.01 | 37.07 | -22.79 | 1,023.05 | 1,068.30 | 4.42 | 2.2 | 2.29 | 4.09 |
| FREE_MEMORY_BYTES | 1,997,733,888 | -646,643,712 | 1,446.21 | -346.29 | -123.94 | 30,819.90 | -9,978.78 | -132.38 | 66.31 | -21.39 | -132.26 |
| GLOBAL_RECEIVE_SIZE_MAX | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| GLOBAL_SEND_SIZE_MAX | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| IDLE_TIME | 137,841,014 | 134,531,079 | 99.79 | 72.04 | -27.81 | 2,126.53 | 2,076.04 | -2.37 | 4.58 | 4.45 | -2.84 |
| INACTIVE_MEMORY_BYTES | -120,803,328 | -137,617,408 | -87.45 | -73.7 | -15.72 | -1,863.68 | -2,123.66 | 13.95 | -4.01 | -4.55 | 13.47 |
| IOWAIT_TIME | 25,364 | 26,263 | 0.02 | 0.01 | -50 | 0.39 | 0.41 | 5.13 | 0 | 0 | 0 |
| NICE_TIME | 242,236 | 240,236 | 0.18 | 0.13 | -27.78 | 3.74 | 3.71 | -0.8 | 0.01 | 0.01 | 0 |
| RSRC_MGR_CPU_WAIT_TIME | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| SWAP_FREE_BYTES | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| SYS_TIME | 10,147,376 | 10,610,918 | 7.35 | 5.68 | -22.72 | 156.55 | 163.74 | 4.59 | 0.34 | 0.35 | 2.94 |
| TCP_RECEIVE_SIZE_DEFAULT | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| TCP_RECEIVE_SIZE_MAX | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| TCP_RECEIVE_SIZE_MIN | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| TCP_SEND_SIZE_DEFAULT | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| TCP_SEND_SIZE_MAX | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| TCP_SEND_SIZE_MIN | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| USER_TIME | 53,940,227 | 55,728,285 | 39.05 | 29.84 | -23.59 | 832.16 | 859.98 | 3.34 | 1.79 | 1.84 | 2.79 |
| VM_IN_BYTES | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| VM_OUT_BYTES | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |