I teach Oracle Education courses at Austin Community College through something called the Workforce Development Program (WDP). It's the official OU training material, but I have to build the machines for my students. We host them on AWS and only bring them up two days a week for three hours and the rest of the time they are down. With the Database 11g training, Oracle used the Grid Infrastructure and Oracle Restart to manage Oracle database startup and shutdown and also used EM Database Control instead of EM Grid Control, so everything was 'fine' out of the box (once I built the old init.d startup scripts). With the 12c content they made two changes: They removed the Grid Infrastructure and they moved largely to EM Cloud Control for management. I'm using the latest Oracle Linux for the class (7.2) and Oracle Linux 7 is now 'systemd based'. I do understand that the old init.d scripts will work with systemd, but I really would like to build my systemd skills and systemd (which I'm not sure really needed to happen... I always thought that init.d stuff worked fine...) is the 'new thing'.
In an effort to 'modernize' my Linux knowledge, I've been attempting to use systemd to manage Database and Middleware startup and shutdown.
I've got a 'server' that is going to be bounced quite often and it has 4 databases running on it, one of which is EMREP, the repository database for Oracle Enterprise Manager Cloud Control. We all know that OEM CC doesn't want to start up if the repository database isn't already up and running, so systemd actually seems to be 'ideal' for managing this dependency. Note that I'm aware that Oracle doesn't officially support systemd for starting and stopping Oracle Middleware, but I'd like to get this to work.
All is well with the Oracle Database service (and I did notice that someone recently asked about this for Oracle Database here: Systemd and Oracle start/stop ):
# /etc/systemd/system/oracle-database.service
# Startup and shutdown database instances listed in /etc/oratab
# Also starts and stops the database listener
[Unit]
Description=Oracle Database Listener and Instances
Requires=network.target
[Service]
User=oracle
Type=forking
Restart=no
ExecStart=/u01/app/oracle/product/12.1.0/dbhome_1/bin/dbstart /u01/app/oracle/product/12.1.0/dbhome_1
ExecStop=/u01/app/oracle/product/12.1.0/dbhome_1/bin/dbshut /u01/app/oracle/product/12.1.0/dbhome_1
[Install]
WantedBy=multi-user.target
Next I'd like to have an oracle-emcc service that starts up after the database. Again, systemd (Requires=oracle-database, After=oracle-database) seems to be ideal for this, but unfortunately it doesn't work. I've been testing out various combinations and it always seems to die after starting everything up or just refusing to start (Type=forking). If I 'hurry' I can log into EM CC for a minute or two, but then systemd tells me that something 'timed out' and it shuts down everything. Since I have a TimeoutSec=infinity I'm really surprised to be getting this.
# /etc/systemd/system/oracle-em.service
# Start and stop Oracle Enterprise Manager
[Unit]
Description=Oracle Enterprise Manager Cloud Control
Requires=oracle-database.service
After=oracle-database.service
[Service]
User=oracle
#Type=forking <--- Generates an instant 'timeout' error, so commented out.
Restart=no
ExecStart=/u01/app/oracle/product/middleware/bin/emctl start oms
ExecStop=/u01/app/oracle/product/middleware/bin/emctl stop oms -all
TimeoutSec=infinity
[Install]
WantedBy=multi-user.target
At just around 4.5 minutes we go from this:
[root@multitenant ~]# systemctl status oracle-emcc
[0m oracle-emcc.service - Oracle Enterprise Manager Cloud Control
Loaded: loaded (/etc/systemd/system/oracle-emcc.service; enabled; vendor preset: disabled)
Active: active (running)[0m since Wed 2016-08-17 12:00:33 CDT; 4min 12s ago
Main PID: 20358 (emctl)
CGroup: /system.slice/oracle-emcc.service
├─20358 /bin/sh -f /u01/app/oracle/product/middleware/bin/emctl start oms
├─20421 /u01/app/oracle/product/middleware/perl/bin/perl /u01/app/oracle/prod...
├─20559 /u01/app/oracle/product/middleware/oracle_common/jdk/bin/java -cp /u0...
├─20663 /bin/bash /u01/app/oracle/product/gc_inst/user_projects/domains/GCDom...
├─20665 /bin/sh /u01/app/oracle/product/middleware/wlserver/server/bin/startN...
├─20708 /u01/app/oracle/product/middleware/oracle_common/jdk/bin/java -Djava....
├─20727 /bin/sh /u01/app/oracle/product/gc_inst/user_projects/domains/GCDomai...
├─20730 /bin/sh /u01/app/oracle/product/gc_inst/user_projects/domains/GCDomai...
├─20786 /u01/app/oracle/product/middleware/oracle_common/jdk/bin/java -server...
├─21163 /bin/sh /u01/app/oracle/product/gc_inst/user_projects/domains/GCDomai...
├─21166 /bin/sh /u01/app/oracle/product/gc_inst/user_projects/domains/GCDomai...
├─21222 /u01/app/oracle/product/middleware/oracle_common/jdk/bin/java -server...
├─21896 /u01/app/oracle/product/middleware/ohs/bin/httpd.worker -k start -f /...
├─21898 /u01/app/oracle/product/middleware/ohs/bin/odl_rotatelogs -l /u01/app...
├─21899 /u01/app/oracle/product/middleware/ohs/bin/odl_rotatelogs /u01/app/or...
├─21900 /u01/app/oracle/product/middleware/ohs/bin/odl_rotatelogs /u01/app/or...
├─21901 /u01/app/oracle/product/middleware/ohs/bin/odl_rotatelogs /u01/app/or...
├─21902 /u01/app/oracle/product/middleware/ohs/bin/odl_rotatelogs /u01/app/or...
├─21903 /u01/app/oracle/product/middleware/ohs/bin/odl_rotatelogs /u01/app/or...
├─21904 /u01/app/oracle/product/middleware/ohs/bin/httpd.worker -k start -f /...
├─21924 /u01/app/oracle/product/middleware/ohs/bin/odl_rotatelogs -l -h:/u01/...
├─21925 /u01/app/oracle/product/middleware/ohs/bin/httpd.worker -k start -f /...
├─21926 /u01/app/oracle/product/middleware/ohs/bin/httpd.worker -k start -f /...
├─22118 /bin/sh /u01/app/oracle/product/middleware/oracle_common/common/bin/w...
├─22165 /u01/app/oracle/product/middleware/oracle_common/jdk/bin/java -Dprod....
├─22229 /bin/sh /u01/app/oracle/product/gc_inst/user_projects/domains/GCDomai...
├─22232 /bin/sh /u01/app/oracle/product/gc_inst/user_projects/domains/GCDomai...
└─22289 /u01/app/oracle/product/middleware/oracle_common/jdk/bin/java -server...
Aug 17 12:00:33 multitenant.localdomain systemd[1]: Started Oracle Enterprise Manager C....
Aug 17 12:00:33 multitenant.localdomain systemd[1]: Starting Oracle Enterprise Manager ....
Aug 17 12:00:33 multitenant.localdomain emctl[20358]: Oracle Enterprise Manager Cloud C...1
Aug 17 12:00:33 multitenant.localdomain emctl[20358]: Copyright (c) 1996, 2015 Oracle C....
Aug 17 12:00:34 multitenant.localdomain emctl[20358]: Starting Oracle Management Server...
Aug 17 12:03:45 multitenant.localdomain emctl[20358]: WebTier Successfully Started
Aug 17 12:04:00 multitenant.localdomain emctl[20358]: Oracle Management Server Successf...d
Aug 17 12:04:00 multitenant.localdomain emctl[20358]: Oracle Management Server is Up
Aug 17 12:04:00 multitenant.localdomain emctl[20358]: JVMD Engine is Up
Aug 17 12:04:00 multitenant.localdomain emctl[20358]: Starting BI Publisher Server ...
Hint: Some lines were ellipsized, use -l to show in full.
To this:
[root@multitenant ~]# systemctl status oracle-emcc
[0m oracle-emcc.service - Oracle Enterprise Manager Cloud Control
Loaded: loaded (/etc/systemd/system/oracle-emcc.service; enabled; vendor preset: disabled)
Active: failed[0m (Result: timeout) since Wed 2016-08-17 12:05:00 CDT; 470ms ago
Process: 22693 ExecStop=/u01/app/oracle/product/middleware/bin/emctl stop oms -all (code=killed, signal=TERM)
Process: 20358 ExecStart=/u01/app/oracle/product/middleware/bin/emctl start oms (code=exited, status=0/SUCCESS)
Main PID: 20358 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/oracle-emcc.service
Aug 17 12:04:00 multitenant.localdomain emctl[20358]: Oracle Management Server is Up
Aug 17 12:04:00 multitenant.localdomain emctl[20358]: JVMD Engine is Up
Aug 17 12:04:00 multitenant.localdomain emctl[20358]: Starting BI Publisher Server ...
Aug 17 12:05:00 multitenant.localdomain emctl[20358]: BI Publisher Server Successfully ...d
Aug 17 12:05:00 multitenant.localdomain emctl[20358]: BI Publisher Server is Up
Aug 17 12:05:00 multitenant.localdomain systemd[1]: oracle-emcc.service stopping timed ....[0m
Aug 17 12:05:00 multitenant.localdomain systemd[1]: oracle-emcc.service stop-final-sigt....[0m
Aug 17 12:05:00 multitenant.localdomain systemd[1]: oracle-emcc.service still around af....[0m
Aug 17 12:05:00 multitenant.localdomain systemd[1]: Unit oracle-emcc.service entered fa....0m
Aug 17 12:05:00 multitenant.localdomain systemd[1]: moracle-emcc.service failed.[0m
Hint: Some lines were ellipsized, use -l to show in full.
I'm flummoxed at this point...
Does anyone have any thoughts on why this might not be working?
(I also asked over in the MOS community systemd for Oracle DB and Middleware (especially EMCC 13c) , but didn't get any responses there.)