Oracle RAC 11.1.0.7 RDBMS using ASM
Oracle Enterprise Linux 5.6
I have multiple databases on each of our RAC clusters, with an ASM Disk Group for data and another for FRA.
Over the weekend a node was evicted. The CPU had little activity but I/O was high. Trace files show:
WARNING:io_submit failed due to kernel limitations MAXAIO for process=257 pending aio=197
WARNING:asynch I/O kernel limits is set at AIO-MAX-NR=3145728 AIO-NR=136486
WARNING:Oracle process running out of OS kernel I/O resources
Now that this eviction has happened, I dug into the trace directory for all of the instances, and the occurrence is a few times a day, usually without an eviction.
I also see:
WARNING:Could not set the asynch I/O limit to 8193 for SQL direct I/O. It is set to 4096
Our system administrator did some AIO_MAX-NR research and found recommendations that 64K is enough for most applications. I'm not sure how that relates to Oracle, information has been hard to find.
For most asm init and instance init parameters I take the default that Oracle defines based on the os.
disk_asynch_io TRUE
Any insights would be appreciated.