I just ran into this issue while trying to run apxremov_con.sql which is why it is referenced below. I'm not totally sure if everything below is correct or not, but I think it is...
Hypothesis: catcon.pl needs a bequeath connection to run code in multitenant RAC databases
When using catcon.pl and/or SQL scripts that call catcon.pl with a RAC database (including RAC One Node), make sure that you have the ORACLE_SID set to the instance name of the instance running on the machine where you are trying to run the code and make sure that you use a bequeath connection and not an Oracle Net connection when running catcon.pl or a script that calls catcon.pl. If you don't do this then you will see the following cryptic error:
catconInit: database is not open on the default instance
Unexpected error encountered in catconInit; exiting
That's the only error you will see. There won't be anything in the database alert logs.
[Aside: I wonder if you could use an Oracle Net connection to run apxremov_con.sql if the ORACLE_SID was set to orcl_1 before you called it... I doubt it, but I'm not totally sure and I don't think I'll have time to test it.]
Details:
Scripts like apxremov_con.sql (used in APEX 4.x to remove APEX from the root container and any pluggable databases in a multitenant database) are wrappers for catcon.pl, a pearl script that runs the passed script in each container database. For example here's the specific lines in apxremov_con.sql script:
host &OH_HOME/perl/bin/perl -I &OH_HOME/rdbms/admin &OH_HOME/rdbms/admin/catcon.pl -b apxremov1_con apxremov1.sql
host &OH_HOME/perl/bin/perl -I &OH_HOME/rdbms/admin &OH_HOME/rdbms/admin/catcon.pl -b apxremov2_con apxremov2.sql
catcon.pl can also be called directly and is referenced here in the Oracle Documentation:
http://docs.oracle.com/database/121/ADMIN/cdb_admin.htm#ADMIN14074 <-- This needs also needs a doc bug if this is true.
When using catcon.pl with a RAC multitenant database then you'll need to set the ORACLE_SID environment variable to the instance name running on the machine where you are running the script. Let's assume you are on RAC Node 1 and there is an orcl database that has two instances accessing it: orcl_1 and orcl_2 (which is running on Node 2 and really isn't important in this case).
From Node 1 the following will NOT work:
. oraenv
orcl
sqlplus sys@orcl as sysdba
@apxremov_con.sql
But the following WILL work:
. oraenv
orcl_1
/u01/app/oracle/product/12.1.2/dbhome_1
sqlplus / as sysdba
@apxremov_con.sql
End of hypothesis.
So we also need a doc bug for the APEX 4.X documentation (which is where apxremov_con.sql was introduced).
We also need someone to test the new apxremov.sql script that ships with APEX 5.X which is supposed to dynamically detect if it is being run in a PDB or a CDB with a RAC Multitenant Database to see if a bequeath connection is necessary or if it will work with an Oracle Net connection.