best way to schedule lots of small threads
807606May 17 2007 — edited May 17 2007Hello,
I'm working on a large set of data and trying to split up the job between multiple threads, to be more exact about 1000 threads.
The maximum number of threads I can run simultaneously is defined by MAX_THREADS.
To be most efficient I would need to know when a thread finishes to start the next one.
I've tried a solution where I start MAX_THREADS, then sleep for a certain time, then check if all threads are still running and start new threads in place of the ones finished. I found it really hard to figure out a good timing for polling the threads.
My next solution would be to create MAX_THREADS number of helper threads that kick off a thread from a pool, then join them and start new ones as those finish.
Another idea is to create a listener that would be activated when a certain thread finishes so it can kick off a new thread.
Could anyone suggest a good solution for this problem? Or best practice?
Thanks,
Lajos