Thread: Answers to "Why are my jobs not running?"


Permlink Replies: 7 - Pages: 1 - Last Post: Jan 14, 2009 1:55 PM Last Post By: Ronald Rood
RnR

Posts: 988
Registered: 12/11/01
Answers to "Why are my jobs not running?"
Posted: Apr 22, 2008 5:52 PM
Click to report abuse...   Click to reply to this thread Reply
ANSWERS TO "WHY ARE MY JOBS NOT RUNNING ?"

This is one of the most common Scheduler questions asked.
Here we list some of the common problems and their solutions.

1) job_queue_processes may be too low (this is the most common problem)
The value of job_queue_processes limits the total number of dbms_scheduler
and dbms_job jobs that can be running at a given time.
To check whether this is the case check the current value of
job_queue_processes with
SQL> select value from v$parameter where name='job_queue_processes';
Then check the number of running jobs
SQL> select count(*) from dba_scheduler_running_jobs;
SQL> select count(*) from dba_jobs_running;

If this is the problem you can increase the parameter using
SQL> alter system set job_queue_processes=1000;

2) max_job_slave_processes may be too low
If this parameter is not NULL then it limits how many dbms_scheduler jobs can
be running at a time. To check whether this is the problem, check the current
value using
SQL> select value from dba_scheduler_global_attribute
where attribute_name='MAX_JOB_SLAVE_PROCESSES';
Then check the number of running jobs
SQL> select count(*) from dba_scheduler_running_jobs;

If this is the problem you can increase the number or just NULL it out using
SQL> exec dbms_scheduler.set_scheduler_attribute('max_job_slave_processes',null)

3) sessions may be too low
This parameter limits the number of sessions at any time. Every Scheduler job
requires 2 sessions. To check whether this is the problem, check the current
valule using
SQL> select value from v$parameter where name='sessions';
Then check the current number of sessions using
SQL> select count(*) from v$session ;

If the numbers are too close you can increase the maximum using
SQL> alter system set job_queue_processes=200;

4) Have you recently applied a timezone update patch or upgraded the database
to a version with newer timezone information ? If you skipped any steps when
updating the timezone information, jobs may not run. To check whether this
is the case try doing
SQL> select * from sys.scheduler$_job;
and
SQL> select * from sys.scheduler$_window;
and make sure they finish without errors.

If it throws a timezone warning, reapply the upgrade or
timezone patch making sure to follow all the steps.

5) Is the database running in restricted mode ?
If the database is running in restricted mode then no jobs will run (unless
you are using 11g and use the ALLOW_RUNS_IN_RESTRICTED_MODE attribute).
To check this use
SQL> select logins from v$instance ;

If logins is restricted you can disable the restricted mode using
SQL> ALTER SYSTEM DISABLE RESTRICTED SESSION;

6) Has the Scheduler been disabled ? This is not a supported action
but it is possible that someone has done it anyway. To check this do
SQL> select value from dba_scheduler_global_attribute where attribute_name='SCHEDULER_DISABLED'

If this query returns TRUE then you can fix this using
SQL> exec dbms_scheduler.set_scheduler_attribute('scheduler_disabled','false');

Reasons why jobs may run late

1) The first thing to check is the timezone that the job is scheduled with
SQL> select owner, job_name, next_run_date from dba_scheduler_jobs ;

If the jobs are in the wrong timezone they may not run at the expected
time. If the next_run_date is using an absolute timezone offset (like
+08:00) instead of a named timezone (like US/PACIFIC) then the jobs may not
run as expected if daylight savings is in effect - they may run an hour
early or late.

2) It may be that at the time the job was scheduled to run, one of the several
limits above may have been temporarily reached causing the job to be delayed.
Check if the limits above are high enough and if possible check them during
the time that the job is being delayed.

3) One possible reason that one of the above limits may be hit is that a
maintenance window may have come into effect. Maintenance windows are Oracle
Scheduler windows that belong to the window group named
MAINTENANCE_WINDOW_GROUP. During a scheduled maintenance window, several
maintenance tasks are run using jobs. This may cause one of the limits listed
above to be hit and user jobs to be delayed. See the admin guide for more info
about this (chapter 24).

To get a list of maintenance windows use
SQL> select * from dba_scheduler_wingroup_members;

To see when the windows run use
SQL> select * from dba_scheduler_windows;

To fix this you can either increase the limits or reschedule the maintenance
windows to run at more convenient times.

Diagnosing other Problems

If none of this works, here are some further steps you can take to try to
figure out what is going on.

1) Check whether there are any errors in the alert log. If the database is
having trouble allocating memory or has run out of disk space or any other
catastrophic errors have occurred, you should resolve those first. You can
find the location of the alert log by using
SQL> select value from v$parameter where name = 'background_dump_dest';
The alert log will be in this directory with a name starting with "alert".

2) Check whether if a job coordinator trace file and if it does, check if it
contains any errors. If this exists, it will be located in the
'background_dump_dest' directory which you can find as above and will look
something like SID-cjq0_nnnn.trc . If there are any errors here they may
hint at why jobs are not running.

3) If either of the above indicates that the SYSAUX tablespace (where the scheduler stores its logging tables) is full, you can use the dbms_scheduler.purge_log procedure to clear out old log entries.

4) See if there is a window currently open. If there is, you can try closing it to see if that helps .
SQL> select * from DBA_SCHEDULER_GLOBAL_ATTRIBUTE where
attribute_name='CURRENT_OPEN_WINDOW';
SQL> exec DBMS_SCHEDULER.close_window ('WEEKNIGHT_WINDOW');

5)try running a simple run-once job and see if it runs
SQL>begin
dbms_scheduler.create_job (
job_name => 'test_job',
job_type => 'plsql_block',
job_action => 'null;',
enabled => true);
end;
/
SQL> -- wait a while
SQL> select * from user_scheduler_job_run_details where job_name='TEST_JOB';

6) If a simple run-once job doesn't run, you can try restarting the scheduler as follows.

SQL> exec dbms_scheduler.set_scheduler_attribute('SCHEDULER_DISABLED', 'TRUE');
SQL> alter system set job_queue_processes=0;
SQL> exec dbms_ijob.set_enabled(FALSE);
SQL>
SQL> alter system flush shared_pool;
SQL> alter system flush shared_pool;
SQL>
SQL> exec dbms_ijob.set_enabled(TRUE);
SQL> alter system set job_queue_processes=99;
SQL> exec dbms_scheduler.set_scheduler_attribute('SCHEDULER_DISABLED', 'FALSE');
fdelacuadra

Posts: 2
Registered: 01/23/08
Re: Answers to "Why are my jobs not running?"
Posted: Jun 26, 2008 9:50 AM   in response to: RnR in response to: RnR
Click to report abuse...   Click to reply to this thread Reply
Very good info. Thanks. One more thing, though. How about jobs that hang in the "running" state? We have a job that does an export of a user's partitioned tables by invoking a shell script on the local file system. When we run the script from the OS command line, it completes fine. When we run the exact same script from the scheduler, it hangs. I'd love to see any troubleshooting pointers for jobs that don't complete you may have put together. It's almost like it times out...

Thanks

F
RnR

Posts: 988
Registered: 12/11/01
Re: Answers to "Why are my jobs not running?"
Posted: Jun 26, 2008 1:09 PM   in response to: fdelacuadra in response to: fdelacuadra
Click to report abuse...   Click to reply to this thread Reply
Hi,

Hanging PL/SQL jobs and hanging external jobs need to be treated differently.

- hanging pl/sql jobs are typically associated with a real background session which is hanging and issues can be tracked down using normal tools/methods to track down oracle hangs (see what sql statement is being run, see what locks are being held/waited on etc).

- hanging external jobs may be the result of a hanging script or may be hanging because the external job never returned results. Both of these issues are covered in the post on running external jobs here
http://forums.oracle.com/forums/thread.jspa?threadID=555102

First check whether the script is running (look through the process list for the name of the script or the commands that the script runs).

If it is, then you need to debug the script itself or you may be running into the bug where the scheduler hangs on large amounts of stderr (fixed in version 10.2.0.2) in which case you should redirect stderr to a file.

If nothing is running then you should check your permissions, in particular, the user that the job runs as must have execute access on $ORACLE_HOME/bin and all parent directories. So for e.g. if your $ORACLE_HOME is /opt/oracle/db then you would have to make sure that

chmod a+rx /opt
chmod a+rx /opt/oracle
chmod a+rx /opt/oracle/db
chmod a+rx /opt/oracle/db/bin

Hope this helps,
Ravi.
fdelacuadra

Posts: 2
Registered: 01/23/08
Re: Answers to "Why are my jobs not running?"
Posted: Jun 26, 2008 4:04 PM   in response to: RnR in response to: RnR
Click to report abuse...   Click to reply to this thread Reply
You nailed it, Ravi. The script was calling the exp command. The command had "> /dev/null" appended to it. Once I changed it to ">& /dev/null", it all worked fine.

Thanks a lot

F
Ronald Rood

Posts: 529
Registered: 01/21/99
Re: Answers to "Why are my jobs not running?"
Posted: Nov 23, 2008 6:48 AM   in response to: RnR in response to: RnR
Click to report abuse...   Click to reply to this thread Reply
A job that has auto_drop = false and end_time < sysdate also won't run. From what I see, resetting the end_date to null makes the job look like able to run but the job log keeps showing 'end time reached'.

Ronald.
http://ronr.nl/unix-dba
RnR

Posts: 988
Registered: 12/11/01
Re: Answers to "Why are my jobs not running?"
Posted: Nov 24, 2008 5:02 PM   in response to: Ronald Rood in response to: Ronald Rood
Click to report abuse...   Click to reply to this thread Reply
Hi Ronald,

As you probably know. Jobs should not be running if end_date < sysdate since this is the purpose of the end_date attribute as documented.

However, it looks like you have run into a bug in that resetting the end_date and then reenabling the job still doesn't force the job to start running again. This is the first time that this bug has been reported.

I will need some info from you to be able to better track this as a bug and get it fixed in the next release (and possibly backported). I will try to contact you directly.

Thanks,
Ravi.
RnR

Posts: 988
Registered: 12/11/01
Re: Answers to "Why are my jobs not running?"
Posted: Dec 2, 2008 4:51 PM   in response to: Ronald Rood in response to: Ronald Rood
Click to report abuse...   Click to reply to this thread Reply
Hi,

The bug Ronald found (resetting end_date for a completed job does not work) has been filed internally as bug #7607696 and is being investigated.

-Ravi
Ronald Rood

Posts: 529
Registered: 01/21/99
Re: Answers to "Why are my jobs not running?"
Posted: Dec 11, 2008 8:32 AM   in response to: RnR in response to: RnR
Click to report abuse...   Click to reply to this thread Reply
Hi Ravi,

an other reason for a job not to run is that the database is 10.2.0.4, has plenty resources available and is using resource manager. I see this happen on systems that have lots of cpu idle, a resource manager plan that has no limits on cpu. What I see is that the job just keeps status scheduled. In the resource manager monitors there is nothing to see what is waiting for resources. It all starts running as soon as we switch to the INTERNAL_PLAN, effectively switching off resource manager.
Strange is there are no waiters for resources ...

regards,
Ronald.
Legend
Guru Guru : 2500 - 1000000 pts
Expert Expert : 1000 - 2499 pts
Pro Pro : 500 - 999 pts
Journeyman Journeyman : 200 - 499 pts
Newbie Newbie : 0 - 199 pts
Oracle ACE Director
Oracle ACE Member
Oracle Employee ACE
Helpful Answer (5 pts)
Correct Answer (10 pts)

Point your RSS reader here for a feed of the latest messages in all forums