Oracle core dumps from /LIB/LIBC.SO.6 after upgrade to 10.2.0.3
Database 10.2.0.3 on Linux x86
(from SR 6366561.993) - also created RFA:771217
- Oracle occassionally core dumps from /LIB/LIBC.SO.6
- This behaviour was noticed after upgrading from 10.1.0.5.0 to 10.2.0.3.0.
- On 26th june 2007 at 6.08 PM and 20.02PM, a database server produced many core dumps in only 2 minutes until the hardisk was full and database stops working because of this.
- The coredump files are 256 MB each.
- The alert.log does not show any errors.
- The database was upgraded from 10.1 to 10.2 a few weeks ago.
- There is no workarounds, but deleted some core dumps in cdump directory and restarted Oracle database ("startup").
- customer said that Setting the parameter : optimizercost_based_transformation=off
did not stop the core dumps from generating.
- The error: ORA-00600: [qctcte1] was reported in the alert.log, but this is not related to this issue, and this is no longer a problem since customer placed a workaround for this bug/error.
- For the uploaded core file (core_32478), customer manually created a stack trace from the core file which resulted in the following:
Function List
__kernel_vsyscall kill slcra
ssexhd ksdfmw ksdwrf slcra ssexhd ksdfmw
ksdwrf ssexhd ksdfmw ksdwrf ssexhd ksdfmw
ksdwrf ssexhd ksdfmw ksdwrf ssexhd ksdfmw
ksdwrf ksdout ksddst ksdout ksddst ksdfmw
ksdwrf ssexhd opitsk opiino opiodr opidrv
sou2o opimai_real main
- I asked for an ARE review who said:
This is an OS related problem.
Regarding to the failed functions in the stack I assume that the root cause is within
the OS code, not on Oracle side.
We are making a system call through the system call interface (__kernel_vsyscall)
when Oracle writes the core dump.
Normally the reason is that Oracle does not get the expected return from OS. Here we get signal 6 (abort) from OS.
Probably the library from where we call the OS is
'/lib/libc.so.6'.
I think that the disability of the linux system to write core dumps in such situations
cannot be taken as argument that this is not an OS problem !
The following bugs seem to show slightly related problems (more than the former
noted Bug 5901669 :
Bug.5904793 /5984581 (36) 10.2.0.2 32-BIT AGENT DIES MONITORING
Bug.6057813 (90) ORA-07445 EXCEPTION ENCOUNTERED CORE DUMP [__KERNEL_VSYSCALL()+16] [SIGIOT]
That does not mean that they match our problem. But all of them seem to be OS
related (port specific) to LINUX, which would confirm our suggestion made above.
- Customer confirmed: The operating system is certified (SUSE Linux Enterprise Server 10 (i586)),
the Kernel is the original distribution kernel: kernel-bigsmp-2.6.16.21-0.8
All required libs are installed.
The install log on this machine was not saved, so they couldn't be checked for errors.
This is an official kernel as per Note 395353.1 .
Error(s)
--------------------
core dump
Similar Bugs:
---------------
Bug 5901669 : occurs in the same environment and the failed OS functions seem to be similar.
Bug 5904793 /5984581 (36) 10.2.0.2 32-BIT AGENT DIES MONITORING
Bug 6057813 (90) ORA-07445 EXCEPTION ENCOUNTERED CORE DUMP [__KERNEL_VSYSCALL()+16] [SIGIOT]
Current Requirement :
----------------------------------
Need to clarify the below points/actions:
- Please review the following trace uploaded to the SR which is : mimastack.trc - this is manually created stack trace from the core file: core_32478.
- What's the reason for producing the core dumps?
- How to avoid this?
- is this really an OS problem?