Oracle Process is reading control01, control02 and control03 endlessly
PatJun 22 2011 — edited Jun 22 2011My Oracle database is version 11.1.0.6, this is a simple test installation, only 1 user connected and doing a INSERT /*+ APPEND */ INTO A SELECT * FROM B;
Table B contains only 1.8 million rows and about 20 rather simple fields (numeric, string, etc), no blobs or clobs.
For testing purposes Table A had 1 primary key constraint and 4 indexes (on different fields).
As this sometimes takes a long time to complete, I wanted to know what was going on,
It seems oracle.exe is only reading the same data over and over again from its control files.
Checking with Process Monitor (SysInternals Suite) we see
It is doing the following for about 15 minutes (but sometimes for 30 minutes) and no other activity (e.g. reading/writing from other files).
9:57:36.7382739 ORACLE.EXE 4252 ReadFile C:\Oracle\oradata\orcl\CONTROL01.CTL SUCCESS Offset: 262,144, Length: 16,384, I/O Flags: Non-cached, Priority: Normal
9:57:36.7384804 ORACLE.EXE 4252 ReadFile C:\Oracle\oradata\orcl\CONTROL01.CTL SUCCESS Offset: 294,912, Length: 16,384, I/O Flags: Non-cached, Priority: Normal
9:57:36.7386945 ORACLE.EXE 4252 ReadFile C:\Oracle\oradata\orcl\CONTROL01.CTL SUCCESS Offset: 327,680, Length: 16,384, I/O Flags: Non-cached, Priority: Normal
9:57:36.7389160 ORACLE.EXE 4252 ReadFile C:\Oracle\oradata\orcl\CONTROL01.CTL SUCCESS Offset: 344,064, Length: 16,384, I/O Flags: Non-cached, Priority: Normal
9:57:36.7391664 ORACLE.EXE 4252 ReadFile C:\Oracle\oradata\orcl\CONTROL01.CTL SUCCESS Offset: 16,384, Length: 16,384, I/O Flags: Non-cached, Priority: Normal
9:57:36.7393785 ORACLE.EXE 4252 ReadFile C:\Oracle\oradata\orcl\CONTROL02.CTL SUCCESS Offset: 16,384, Length: 16,384, I/O Flags: Non-cached, Priority: Normal
9:57:36.7395935 ORACLE.EXE 4252 ReadFile C:\Oracle\oradata\orcl\CONTROL03.CTL SUCCESS Offset: 16,384, Length: 16,384, I/O Flags: Non-cached, Priority: Normal
9:57:36.7398094 ORACLE.EXE 4252 ReadFile C:\Oracle\oradata\orcl\CONTROL01.CTL SUCCESS Offset: 262,144, Length: 16,384, I/O Flags: Non-cached, Priority: Normal
9:57:36.7400154 ORACLE.EXE 4252 ReadFile C:\Oracle\oradata\orcl\CONTROL01.CTL SUCCESS Offset: 294,912, Length: 16,384, I/O Flags: Non-cached, Priority: Normal
9:57:36.7402253 ORACLE.EXE 4252 ReadFile C:\Oracle\oradata\orcl\CONTROL01.CTL SUCCESS Offset: 327,680, Length: 16,384, I/O Flags: Non-cached, Priority: Normal
9:57:36.7404429 ORACLE.EXE 4252 ReadFile C:\Oracle\oradata\orcl\CONTROL01.CTL SUCCESS Offset: 344,064, Length: 16,384, I/O Flags: Non-cached, Priority: Normal
9:57:36.7406946 ORACLE.EXE 4252 ReadFile C:\Oracle\oradata\orcl\CONTROL01.CTL SUCCESS Offset: 16,384, Length: 16,384, I/O Flags: Non-cached, Priority: Normal
9:57:36.7409890 ORACLE.EXE 4252 ReadFile C:\Oracle\oradata\orcl\CONTROL02.CTL SUCCESS Offset: 16,384, Length: 16,384, I/O Flags: Non-cached, Priority: Normal
9:57:36.7412177 ORACLE.EXE 4252 ReadFile C:\Oracle\oradata\orcl\CONTROL03.CTL SUCCESS Offset: 16,384, Length: 16,384, I/O Flags: Non-cached, Priority: Normal
I've performed a hang analysis as explained in: http://www.dba-oracle.com/t_fix_hanging.htm
The trace log file contains the following.
*** 2011-06-22 10:30:25.764
===============================================================================
HANG ANALYSIS:
instances (db_name.oracle_sid): orcl.orcl
no oradebug node dumps
===============================================================================
Chains most likely to have caused the hang:
[a] Chain 1 Signature: 'log file switch (checkpoint incomplete)'<='buffer busy waits'
Chain 1 Signature Hash: 0x25872b6
Chain 2 Signature: 'log file switch (checkpoint incomplete)'<='buffer busy waits'
Chain 2 Signature Hash: 0x25872b6
[c] Chain 3 Signature: 'log file switch (checkpoint incomplete)'<='buffer busy waits'
Chain 3 Signature Hash: 0x25872b6
I've also browsed the result (which is quite large) of the 'dump systemstate 10' command and came across.
Line 1247: 0 commit cleanout failures: write disabled
Line 1248: 42 commit cleanout failures: block lost
Line 1249: 28 commit cleanout failures: cannot pin
Line 1250: 0 commit cleanout failures: hot backup in progress
Line 1251: 1 commit cleanout failures: buffer being written
Line 1252: 16 commit cleanout failures: callback failure
Line 1252: 16 commit cleanout failures: callback failure
Line 1344: 0 shared io pool buffer get failure
Line 1412: 1699 IMU- failed to get a private strand
Line 1450: 11551 failed probes on index block reclamation
Line 1453: 0 native hash arithmetic fail
Line 1576: 345 parse count (failures)
Line 1594: 0 OTC commit optimization failure - setup
In the end it (oracle.exe) has read > 200 Gbyte, while the system was just rebooted and this was about the only thing that was performed.
This is running on Windows 7 64 bit with 64 bit Oracle.
Do we need to give oracle more memory?
Any help appreciated.
Patrick