ScheduledThreadPoolExecutor Deadlock
I've been trying to use ScheduledThreadPoolExecutor instead of the Timer class to handle a large amount of tasks with a thread pool. However, I'm running into a deadlock situation.
I basically schedule tasks at a fixed rate. And in the run() method of these ScheduledFuture tasks, I decide whether a task is finished and do so by doing the following:
scheduler.remove(this);
task.cancel(false);
Previously I used to just do task.cancel() without any issues. But I'm forced to the scheduler.remove() because the unlike the Timer task, the scheduler does not remove cancelled tasks when it sees them which makes the queue grow as a memory leak.
However, during heavy load, all my new schedule()'s get stuck at offer() on the queue and all the worker threads are in take() waiting for tasks in the queue. It seems like somehow a lock was not unlocked if that makes sense because I've done a dump on the threads and I see no thread in a runnable state holding any locks in the ScheduledThreadPoolExecutor or its DelayQueue.
Has anyone seen anything like this before?