Skip to Main Content

How to avoid OutOfMemory exception in a memory intensive application in Java?

1065078Jan 16 2014 — edited Jan 23 2014

down vote  favorite  

We have developed a java application, whose primary objective is to read a file(input file), process it and convert it into set of output files.

(I have given a generic description of our solution, to avoid irrelevant details).

This program works perfectly fine when the input file is of 4 GB, with memory settings of -Xms4096m -Xmx16384m in a 32 GB RAM

Now we need to run our application with the input file of size 130 GB.

We used a linux box with 250GB RAM and with memory setting of -Xms40g -Xmx200g (also tried couple of other variations) to run the application and hit OutOfMemory Exception.

At this stage of our project it's very hard to consider redesigning the code to accommodate hadoop ( or someother large scale data processing framework), also the current hardware configuration which we can afford is 250GB of RAM.

Can you please suggest us ways to avoid OutOfMemory Exceptions, what is the general practice when developing these kind of applications.?

Thanks in advance

Comments
Post Details
Added on Jan 16 2014
8 comments
3,827 views