Hi guys,
i currently try to deploy a complete fresh Oracle RAC 12.1.0.2 (GI stack) installation on on OEL 7 (3.8.13-35.3.4.el7uek) with 2 nodes for validation purpose. However the installation itself went fine, but the step "Creating Container Database for Oracle Grid Infrastructure Management Repository" always fails with "ORA-01034: ORACLE not available / ORA-27102: out of memory / Linux-x86_64 Error: 12: Cannot allocate memory". The validation steps by runInstaller finished successfully, but i never get over this configuration step without skipping.
Here is my setting which usually should avoid such ORA errors.
Host
-bash-4.2$ uname -a
Linux OELRAC1 3.8.13-35.3.4.el7uek.x86_64 #2 SMP Tue Jul 29 23:24:14 PDT 2014 x86_64 x86_64 x86_64 GNU/Linux
"Creating Container Database for Oracle Grid Infrastructure Management Repository" error in /oracle/base/cfgtoollogs/dbca/_mgmtdb/trace.log
Starting restore at 09-AUG-14
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=12 device type=DISK
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00003 to +GRID
channel ORA_DISK_1: reading from backup piece /oracle/grid/12102/assistants/dbca/templates/MGMTSeed_Database.dfb
channel ORA_DISK_1: ORA-19870: error while restoring backup piece /oracle/grid/12102/assistants/dbca/templates/MGMTSeed_Database.dfb
ORA-19504: failed to create file "+GRID"
ORA-17502: ksfdcre:4 Failed to create file +GRID
ORA-15001: diskgroup "GRID" does not exist or is not mounted
ORA-01034: ORACLE not available
ORA-27102: out of memory
Linux-x86_64 Error: 12: Cannot allocate memory
Additional information: 2663
Additional information: 1565392897
Additional information: 161480704
failover to previous backup
creating datafile file number=3 name=+GRID
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 08/09/2014 19:41:47
ORA-01119: error in creating database file '+GRID'
ORA-17502: ksfdcre:4 Failed to create file +GRID
ORA-15001: diskgroup "GRID" does not exist or is not mounted
ORA-01034: ORACLE not available
ORA-27102: out of memory
Linux-x86_64 Error: 12: Cannot allocate memory
Additional information: 2663
Additional information: 1565392897
Additional information: 1614807040
RMAN-06956: create datafile failed; retry after removing +GRID from OS
SHM / IPC (check the shm id with the previous additional information)
-bash-4.2$ ipcs -a
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 1565360128 grid 640 4096 0
0x00000000 1565392897 grid 640 4096 0
0xfba47600 1565425666 grid 640 24576 29
Memory setting of ASM instance
SQL> show parameter memory
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
memory_max_target big integer 1076M
memory_target big integer 1076M
ASM disk groups
SQL> select NAME,STATE, TOTAL_MB, USABLE_FILE_MB from v$asm_diskgroup;
NAME STATE TOTAL_MB USABLE_FILE_MB
------------------------------ ----------- ---------- --------------
GRID MOUNTED 6144 4868
/dev/shm for memory_target (more than enough free space)
-bash-4.2$ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 3.0G 630M 2.4G 21% /dev/shm
Kernel parameter for SHM limits set to unlimited
-bash-4.2$ sysctl -a | grep shm
kernel.shmall = 1152921504606846720
kernel.shmmax = 922337203685477580
User limits set to unlimited for memory
-bash-4.2$ cat /etc/security/limits.conf
# Oracle settings
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
* hard memlock unlimited
* soft memlock unlimited
-bash-4.2$ su - grid
-bash-4.2$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 23953
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 2047
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
So what the heck is wrong in here? Why does the ASM (+ASM1) instance return the error below, even if there is no memory limit / issue. The error with the disk group is also absolutely non sense.
------------------------------------------------------------------------------------------------------------------------
ORA-01119: error in creating database file '+GRID'
ORA-17502: ksfdcre:4 Failed to create file +GRID
ORA-15001: diskgroup "GRID" does not exist or is not mounted
ORA-01034: ORACLE not available
ORA-27102: out of memory
Linux-x86_64 Error: 12: Cannot allocate memory
Additional information: 2663
Additional information: 1565392897
Additional information: 1614807040
------------------------------------------------------------------------------------------------------------------------
Has anybody experience the same issue with 12.1.0.2 by deploying the CDB for Grid Infrastructure Management Repository? Is there something special with UEK3 (3.8.13-35.3.4.el7uek) on OEL 7? Any special kernel parameter needed (even if the runInstaller checks do not mention it)? I am totally bugged by 12.1.0.2 GI.
Thank you.
Best Regards
Stefan