Limiting the number of threads that are created
795672Jul 20 2010 — edited Jul 20 2010I am trying to write a web crawler as a way of learning about multi-threading and I have hit a stumbling block (probably my design). The way the application I have started works is that I provide a single URL, which is passed to a thread and that thread then parses the web page building a list of links (If the links are not already held in a 'visited' list then they are saved in a list in the Runnable object that is parsing the html page).
When the page has been parsed the list of links are then passed to a method in a Utilities class that creates a new group of threads, 1 for each link and each of these threads then does the same thing. The problem I have is that as more and more links are captured more and more threads are created eventually I either get a out of memory exception or I get an operating system 'cannot create native thread' message. I am not sure if the threads are staying in memory after they have done their tasks, I am not sure why I am running out of memory so quickly.
What I would like to try and do is to set a limit for the maximum number of threads created and then only create new threads if the limit is not breached. If someone could point me in the right direction that would be good, I have googled around but cant find an example of what I want to do.
Thanks