2 node Oracle 11.2.0.3 RAC on OL 5.9.
Have something strange happening to a job process, and happening more than once.
Process shows SQL "insert /*+APPEND*/ into some_table select * from some_view". Around 1700 rows in the view at the time.
Process events shows a "db file scattered read" (or sometimes a sequential read or a gc current request), and spins on this single event - until process is killed. After which the job runs again - successfully.
Previous times I ran a strace, I saw no kernel calls being made at all. This morning though I saw the process setting a timer signal, and when it fires, making a poll() kernel call on a file descriptor, that is a UDP socket on localhost. Confirmed that is waiting on a UDP read by using the nc command to send a UDP test packet on that localhost port - and seeing via strace the UDP data being read (and discarded as incorrect, obviously). With the process going back to poll() on timer signal that localhost UDP port.
Do not see any other process on that server attempting to write to that localhost UDP port - thus it seems that the job process spinning on the I/O event, lost comms with a fellow server process (ASM?) via UDP. Until it is killed. After which the 2nd time around the processing works fine.
Any suggestions on what this could be? I find it puzzling that Oracle reports db read I/O when spinning on a UDP read on a localhost port. There are ASM and RAC and Interconnect events that seem to be a better fit for describing what the process is waiting for, on a UDP read, to be completed.