Java Programming

Hi, my application reads from a repository of 1 million documents and then do some processing (very simple and requires little memory or computation) then outputs to another repository, where output files are placed in a flat structure, i.e., 1 million files in a single folder, not organised into any subfolders. I know this is a bad design and Im about to change it, but not sure if this is the cause of my problem (see following).

My program haulted and threw exception *"insufficent existing resources to complete requested service"* every time it reached about 800k documents. And after I got this exception, my WinXP OS starts to misbehave, like sometimes I cannot open applications properly, and sometimes memory reference address issues, and once I tried to run JAVA IDE (intellij 7.0) again and I got this message:

"ERROR: transport error 202: recv failed during handshake: No buffer space available (maximum connections reached?)"

Also I tried to list all open files in a console, but the command hangs and without responding, looks like there are too many things to list or too little resources available to complete the operation.

All these problems go away if I restart system.


I have confirmed the following possible causes and they do not apply:

- unclosed IO's , I checked the code and all IO operations properly instantiated and closed;
- memory leak, java profiler shows memory consumption well below allocated max, and very stable (test on 100k documents, not whole of 1million though)


I do not have any clue now as how this problem happend and how I can solve it. I wonder if placing all 800k documetns in a single folder causes problem? I know that if I try to call File.listFiles() on a folder with too many files that will cause memory problem. But I didnt have this kind of method call at all, but simply writing a particular file to that folder. Does this cause problem too? I am about to change this and to organise the output folder to subfolders, but is this the right dirrection to take?

I am sorry that my description may be insufficient to truely debug the problem, but that is all I can get and it does look strange to myself too. Please could you throw in your ideas - any thoughts about the exception message, even any guess on what would be causes to this etc...

Any thoughts are very much appreciated!

Thanks so much inadvance!