Hello Everybody!
I am struggling with a RAC setup for quite sometime now. Only this time I am facing issues that I never faced before.
OS OL 6.8
Grid/DB Version 11.2.0.4
ASM
Grid Infrastructure installed and working fine. All services are running. Checked after reboots and services come up with no issues.
Database Software installed without any problems.
Attempt to create a database using dbca failed with below errors. These are just the error relevant portion of the trace but if you want, I will paste the entire trace file here for your review. Appreciate your help. If it is not the right forum then please move it to the correct forum.
[InstanceStepOPS.performACFSCheck:1796] +DATA, Not an ACFS
[StreamReader.run:65] ERROR>acfsutil info fs: ACFS-03037: not an ACFS file system
PRCT-1114 : Execution of acfsutil command failed on node localnode for location /
PRCT-1011 : Failed to run "acfsutil". Detailed error: acfsutil info fs: ACFS-03037: not an ACFS file system
at oracle.cluster.deployment.ClusterwareInfo.do1_isOFSPartition(ClusterwareInfo.java:1774)
Caused by: PRCT-1114 : Execution of acfsutil command failed on node localnode for location /
PRCT-1011 : Failed to run "acfsutil". Detailed error: acfsutil info fs: ACFS-03037: not an ACFS file system
Caused by: PRCT-1011 : Failed to run "acfsutil". Detailed error: acfsutil info fs: ACFS-03037: not an ACFS file system
[Thread-270] [ 2016-11-12 20:24:53.703 GST ] [Verifier.isACFSRegistered:2158] /u02/app/oracle is ACFS :false
[Thread-270] [ 2016-11-12 20:24:53.703 GST ] [InstanceStepOPS.performACFSCheck:1806] /u02/app/oracle, Not an ACFS.
[Thread-270] [ 2016-11-12 20:24:53.703 GST ] [InitParamAttributes.removeDuplicateOPSParams:938] InitParamAttributes->removeDuplicateOPSParams: Start of function, number of parameters=328
[Thread-270] [ 2016-11-12 20:24:53.711 GST ] [StepContext.getSelectedDiskGroups:3453] Data Diskgroups OMF+DATA
[Thread-270] [ 2016-11-12 20:24:53.711 GST ] [StepContext.getSelectedDiskGroups:3470] Recovery Diskgroups +DATA
[Thread-270] [ 2016-11-12 20:24:53.712 GST ] [DiskGroupImpl.<init>:108] name = ora.DATA.dg, display name = DATA
[Thread-270] [ 2016-11-12 20:24:53.728 GST ] [DiskGroupImpl.<init>:138] name = ora.DATA.dg, display name = DATA
[Thread-270] [ 2016-11-12 20:24:54.160 GST ] [DatabaseImpl.create:465] Checking server group configuration
[Thread-270] [ 2016-11-12 20:24:54.202 GST ] [DatabaseFactoryImpl.getDatabases:1145] No database was found hosted by server pool =ora.TEST
[Thread-270] [ 2016-11-12 20:24:54.224 GST ] [DatabaseFactoryImpl.getServices:2261] No service was found hosted by server pool =ora.TEST
[Thread-270] [ 2016-11-12 20:24:54.224 GST ] [DatabaseImpl.create:523] no existing db or service hosted by server group, TEST
[Thread-270] [ 2016-11-12 20:24:54.224 GST ] [DatabaseImpl.create:528] Done with server group check: AdminManaged=true
[Thread-270] [ 2016-11-12 20:24:54.224 GST ] [Version.isPre:583] version to be checked 11.2.0.4.0 major version to check against10
[
[Thread-278] [ 2016-11-12 20:24:54.246 GST ] [StreamReader.run:65] ERROR>acfsutil info fs: ACFS-03037: not an ACFS file system
[Thread-270] [ 2016-11-12 20:24:54.246 GST ] [UnixSystem.dorunRemoteExecCmd:3232] retval = 1
[Thread-270] [ 2016-11-12 20:24:54.247 GST ] [CmdToolUtil.doexecute:474] nativeSystem.runRemoteExecCmd failed. Command = /sbin//acfsutil arguments = [info, fs, -o, mountpoints, /database] env = null error = acfsutil info fs: ACFS-03037: not an ACFS file system
[Thread-270] [ 2016-11-12 20:24:54.247 GST ] [DatabaseImpl.getACFS:3898] non fatal, exception encountered when getting mountpoint for oracle home
[Thread-270] [ 2016-11-12 20:24:54.248 GST ] [DatabaseImpl.create:613] instance Name = TEST1
[Thread-270] [ 2016-11-12 20:24:54.248 GST ] [DatabaseImpl.create:613] instance Name = TEST2
[Thread-292] [ 2016-11-12 20:24:55.645 GST ] [StreamReader.run:65] ERROR>/bin/cat: /database/dbs/CFSFileName1478967895469326579103688289580.tmp: No such file or directory
[Thread-270] [ 2016-11-12 20:24:55.814 GST ] [DBEntryStep.executeImpl:254] undoMgmt=null
[Thread-270] [ 2016-11-12 20:24:55.814 GST ] [DBEntryStep.executeImpl:255] undoMgmt=true
[Thread-270] [ 2016-11-12 20:24:55.815 GST ] [DBEntryStep.executeImpl:274] undoTBS=UNDOTBS1
[Thread-270] [ 2016-11-12 20:25:03.961 GST ] [DBEntryStep.executeImpl:390] undoTBS create SQL=CREATE SMALLFILE UNDO TABLESPACE "UNDOTBS1" DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
[Thread-270] [ 2016-11-12 20:25:03.961 GST ] [DBEntryStep.executeImpl:439] 0th redoLogText = GROUP 1 SIZE 51200K
[Thread-270] [ 2016-11-12 20:25:03.962 GST ] [DBEntryStep.executeImpl:439] 1th redoLogText = GROUP 2 SIZE 51200K
[Thread-270] [ 2016-11-12 20:25:03.962 GST ] [DBEntryStep.executeImpl:456] CREATEDB without passwords = CREATE DATABASE "TEST"
MAXINSTANCES 32
MAXLOGHISTORY 1
MAXLOGFILES 192
MAXLOGMEMBERS 3
MAXDATAFILES 1024
DATAFILE SIZE 700M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
EXTENT MANAGEMENT LOCAL
SYSAUX DATAFILE SIZE 600M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
SMALLFILE DEFAULT TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
SMALLFILE UNDO TABLESPACE "UNDOTBS1" DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
CHARACTER SET AL32UTF8
NATIONAL CHARACTER SET AL16UTF16
LOGFILE GROUP 1 SIZE 51200K,
GROUP 2 SIZE 51200K
[Finalizer] [ 2016-11-12 20:25:18.832 GST ] [Util.finalize:126] Util: finalized called for oracle.ops.mgmt.has.Util@2e18ff27
[Finalizer] [ 2016-11-12 20:25:18.832 GST ] [Util.finalize:126] Util: finalized called for oracle.ops.mgmt.has.Util@6b177115
oracle.sysman.assistants.util.step.StepExecutionException: ORA-03113: end-of-file on communication channel
at oracle.sysman.assistants.dbca.backend.DBEntryStep.executeImpl(DBEntryStep.java:536)
at oracle.sysman.assistants.util.step.BasicStep.execute(BasicStep.java:210)
at oracle.sysman.assistants.util.step.Step.execute(Step.java:140)
at oracle.sysman.assistants.util.step.StepContext$ModeRunner.run(StepContext.java:2711)
at java.lang.Thread.run(Thread.java:637)
[Thread-270] [ 2016-11-12 20:30:11.862 GST ] [OsUtilsBase.deleteFile:1863] OsUtilsBase.deleteFile: +DATA/TEST/OMF_1_REDOLOG_MEMBER_0
[Thread-270] [ 2016-11-12 20:30:11.862 GST ] [CommonAttributes.replaceVariables:1329] var2 TEST
[Thread-270] [ 2016-11-12 20:30:11.862 GST ] [OsUtilsBase.deleteFile:1863] OsUtilsBase.deleteFile: +DATA/TEST/OMF_2_REDOLOG_MEMBER_0
[Thread-270] [ 2016-11-12 20:30:11.862 GST ] [CommonAttributes.replaceVariables:1329] var2 TEST
[Thread-270] [ 2016-11-12 20:30:11.862 GST ] [OsUtilsBase.deleteFile:1863] OsUtilsBase.deleteFile: +DATA/TEST/OMF_3_REDOLOG_MEMBER_0
[Thread-270] [ 2016-11-12 20:30:11.862 GST ] [CommonAttributes.replaceVariables:1329] var2 TEST
[Thread-270] [ 2016-11-12 20:30:11.863 GST ] [OsUtilsBase.deleteFile:1863] OsUtilsBase.deleteFile: +DATA/TEST/OMF_4_REDOLOG_MEMBER_0
[Thread-270] [ 2016-11-12 20:30:11.863 GST ] [CommonAttributes.replaceVariables:1329] var2 TEST
[Thread-270] [ 2016-11-12 20:30:11.863 GST ] [OsUtilsBase.deleteFile:1863] OsUtilsBase.deleteFile: +DATA/TEST/<OMF_CONTROL_0>
[Thread-270] [ 2016-11-12 20:30:11.863 GST ] [CommonAttributes.replaceVariables:1329] var2 TEST
[Thread-270] [ 2016-11-12 20:30:11.863 GST ] [OsUtilsBase.deleteFile:1863] OsUtilsBase.deleteFile: +DATA/TEST/<OMF_CONTROL_1>
[Thread-270] [ 2016-11-12 20:30:11.863 GST ] [OsUtilsBase.deleteFile:1863] OsUtilsBase.deleteFile: <OMF_SYSTEM_DATAFILE_0>
[Thread-270] [ 2016-11-12 20:30:11.863 GST ] [DBCreateFilesStep.cancel:447] DBCreateFilesStep.cancel(): useRawDevices =false
[Thread-270] [ 2016-11-12 20:30:11.866 GST ] [OsUtilsBase.deleteFile:1863] OsUtilsBase.deleteFile: <OMF_SYSAUX_DATAFILE_0>
[Thread-270] [ 2016-11-12 20:30:11.866 GST ] [OsUtilsBase.deleteFile:1863] OsUtilsBase.deleteFile: <OMF_TEMP_DATAFILE_0>
[Thread-270] [ 2016-11-12 20:30:11.866 GST ] [OsUtilsBase.deleteFile:1863] OsUtilsBase.deleteFile: <OMF_UNDOTBS1_DATAFILE_0>
[Thread-270] [ 2016-11-12 20:30:11.867 GST ] [OsUtilsBase.deleteFile:1863] OsUtilsBase.deleteFile: <OMF_UNDOTBS2_DATAFILE_0>
[Thread-270] [ 2016-11-12 20:30:11.867 GST ] [OsUtilsBase.deleteFile:1863] OsUtilsBase.deleteFile: <OMF_USERS_DATAFILE_0>
[Thread-270] [ 2016-11-12 20:30:11.867 GST ] [StepErrorHandler.setFatalErrors:322] setting Fatal Error: ORA-00942
[Thread-270] [ 2016-11-12 20:30:11.867 GST ] [StepErrorHandler.setFatalErrors:322] setting Fatal Error: ORA-00955
[Thread-270] [ 2016-11-12 20:30:11.867 GST ] [StepErrorHandler.setFatalErrors:322] setting Fatal Error: ORA-01432
[Thread-270] [ 2016-11-12 20:30:11.867 GST ] [StepErrorHandler.setFatalErrors:322] setting Fatal Error: ORA-01434
[Thread-270] [ 2016-11-12 20:30:11.867 GST ] [StepErrorHandler.setFatalErrors:322] setting Fatal Error: ORA-01918
[Thread-270] [ 2016-11-12 20:30:11.867 GST ] [StepErrorHandler.setFatalErrors:322] setting Fatal Error: ORA-01919
[Thread-270] [ 2016-11-12 20:30:11.867 GST ] [StepErrorHandler.setFatalErrors:322] setting Fatal Error: ORA-01921
[Thread-270] [ 2016-11-12 20:30:11.867 GST ] [StepErrorHandler.setFatalErrors:322] setting Fatal Error: ORA-02289
[Thread-270] [ 2016-11-12 20:30:11.868 GST ] [StepErrorHandler.setFatalErrors:322] setting Fatal Error: ORA-04043
[Thread-270] [ 2016-11-12 20:30:11.868 GST ] [StepErrorHandler.setFatalErrors:322] setting Fatal Error: ORA-29807
[Thread-270] [ 2016-11-12 20:30:11.868 GST ] [StepErrorHandler.setFatalErrors:322] setting Fatal Error: ORA-29833
[Thread-270] [ 2016-11-12 20:30:11.868 GST ] [StepErrorHandler.setIgnorableErrors:250] setting Ignorable Error: ORA-00942
[Thread-270] [ 2016-11-12 20:30:11.868 GST ] [StepErrorHandler.setIgnorableErrors:250] setting Ignorable Error: ORA-00955
[Thread-270] [ 2016-11-12 20:30:11.868 GST ] [StepErrorHandler.setIgnorableErrors:250] setting Ignorable Error: ORA-01432
[Thread-270] [ 2016-11-12 20:30:11.868 GST ] [StepErrorHandler.setIgnorableErrors:250] setting Ignorable Error: ORA-01434
[Thread-270] [ 2016-11-12 20:30:11.868 GST ] [StepErrorHandler.setIgnorableErrors:250] setting Ignorable Error: ORA-01918
[Thread-270] [ 2016-11-12 20:30:11.868 GST ] [StepErrorHandler.setIgnorableErrors:250] setting Ignorable Error: ORA-01919
[Thread-270] [ 2016-11-12 20:30:11.868 GST ] [StepErrorHandler.setIgnorableErrors:250] setting Ignorable Error: ORA-01921
[Thread-270] [ 2016-11-12 20:30:11.868 GST ] [StepErrorHandler.setIgnorableErrors:250] setting Ignorable Error: ORA-02289
[Thread-270] [ 2016-11-12 20:30:11.868 GST ] [StepErrorHandler.setIgnorableErrors:250] setting Ignorable Error: ORA-04043
[Thread-270] [ 2016-11-12 20:30:11.869 GST ] [StepErrorHandler.setIgnorableErrors:250] setting Ignorable Error: ORA-29807
[Thread-270] [ 2016-11-12 20:30:11.869 GST ] [StepErrorHandler.setIgnorableErrors:250] setting Ignorable Error: ORA-29833
[Thread-270] [ 2016-11-12 20:30:11.869 GST ] [BasicStep.configureSettings:304] messageHandler being set=oracle.sysman.assistants.util.UIMessageHandler@57f55cae
[Thread-270] [ 2016-11-12 20:30:11.869 GST ] [StepErrorHandler.setFatalErrors:322] setting Fatal Error: ORA-03113