Java SE (Java Platform, Standard Edition)

This problem is encountered on a Windows XP box. Simple source code that encounters the problem is included below.

I am creating a very large 2D image of floating point numbers. Often too large to support from the heap. It is useful with only a few adjacent image columns in memory. So, as I create the image from left to right by columns, I write them (float [] array) to disk using an ObjectOutputStream. After closing that stream, I re-open as an ObjectInputStream to read the arrays back in, a few columns at a time.

This scheme works on small images. On large, heap space threatening images I note that the ObjectOutputStream appears to accumulate written objects in the heap, so that the program eventually runs out of heap space. Using "flush()" periodically does not help. Inviting garbage collection by setting the array variable = "null" does not help. Watching the Task Manager memory usage supports this interpretation.

What does "help" is to periodically flush() and close() the ObjectOutputStream. Then reopen it, each time in "append" mode. This clears out the heap, heap memory use stabilizes, and running out of heap space is no longer a problem. Unfortunately, after using this "cure" on output, I can not read the float [] arrays back into memory. Darn. What am I doing wrong?

The attached skeleton of the full Java program shows the problem. The class name is "virtualImageTest". The method "virtualMemory(int,int)" registers the size of the desired image and creates the ObjectOutputStream. The method "fill(float [], int)" writes individual columns to the ObjectOutputStream. Finally, the method "readyRead()" flushes and closes the output stream, and reopens it as an ObjectInputStream. The critical two lines of code, surrounded by much white space, are in "fill". Here "wos" is the ObjectOutputStream stream and "weights" is the File output stream:

wos.close();
wos = new ObjectOutputStream(new FileOutputStream(weights,true));

Comment out these two lines for small images and the program works. But, it runs out of heap memory for large images. Uncomment these lines of code and the program stores very large images on disk without running out of heap space. But, large or small, I can not then read columns back from the file.

Suggestions?

Steven Zoraster

=================== CODE ==================

/*
* virtualImageTest.java
*
*
*/


import java.io.ObjectOutputStream;
import java.io.ObjectInputStream;

import java.io.FileOutputStream;
import java.io.FileInputStream;

import java.io.IOException;
import java.io.File;

/**
*
* @author Steven
*/
public class virtualImageTest {

static int mprint = 25;

int jlast = -1;
int nrows,ncols,nrows2,ncols2;

float [] g0,g1,g2,g3;

File weights=null;

ObjectOutputStream wos;
ObjectInputStream wis;

/* ===================================== */
/* Creates a new instance of virtualImage */
/* ===================================== */
public virtualImageTest() {
}


public static void main(String[] args) {

/* ====================================================== */
/* Set these numbers to 10000 each to see heap space used */
/* ====================================================== */
int nr = 100,nc=100;

/* ============================ */
/* Create a virtual grid object */
/* ============================ */

virtualImageTest vg = new virtualImageTest();
vg.virtualMemory(nr,nc);

/* ========================================= */
/* Now add columns to the virtual image file */
/* ========================================= */

float [] temp1;
for(int j=0;j<=nc+1;j++) {

temp1 = new float[nr+2];
for(int i=0;i<=nr+1;i++) temp1=(float)j;

vg.fill(temp1,j);
if(j/mprint*mprint==j) System.out.println(" Creating data with j = " + j );

}

/* ==================================== */
/* Okay, we wrote the data, now read it */
/* ==================================== */

if(!vg.readyRead()) System.exit(0);

}

public boolean virtualMemory(int nr,int nc)
{
nrows = nr; nrows2 = nrows+2;
ncols = nc; ncols2 = ncols+2;

/* ======================================= */
/* We are going to need temp file */
/* As shown in Java Cookbook, page 295-296 */
/* ======================================= */

try {

weights = File.createTempFile("weights","tmp");

} catch(IOException e) {System.out.println("Temp file problems " + e);System.exit(0);}

/* ======================================================== */
/* Open an output file and create a Buffered writer handles */
/* ======================================================== */

try {

wos = new ObjectOutputStream(new FileOutputStream(weights));

} catch (IOException e) {
System.out.println("Failed to open output streams " + e); return false;}
return true;
}

public boolean fill(float [] w,int j)
{
/* ==================================================== */
/* This method accepts columns of the grid in order */
/* and writes them out to disk files for use later. */
/* ==================================================== */
jlast = j;

try {

wos.writeObject(w);
w=null;

} catch(IOException e)
{ System.out.println("Failure to write array in virtualImage.fill " + e);
return false;}

/* ====================================================== */
/* Okay, don't completely trust this as a buffered stream */
/* ====================================================== */

if(j/20*20==j&&j!=100)
try {
wos.flush();

/* ============================================ */
/* Good grief! On output a file stream seems */
/* to accumulate heap memory. Not believeable */
/* but true. Closing and re-opening for */
/* append seems to handle this. */
/* ============================================ */


long wl = weights.length();
System.out.println("\n At periodic close and open, length of weights = " + wl);
System.out.flush();




wos.close();
wos = new ObjectOutputStream(new FileOutputStream(weights,true));





} catch(IOException e)
{ System.out.println("Failure to flush output file buffers in virtual.Grid.fill: " + e);
return false; }

return true;
}


public boolean readyRead() {
/* ========================================================= */
/* We will now flush and close all three original disk files */
/* Then, we read the first four columns into memory. */
/* ========================================================= */

try {

wos.flush(); wos.close();

} catch(IOException e)
{ System.out.println("Failure to close array file in virtualImage.readyRead: " + e);
return false;}

/* ================================================= */
/* Now we will read the first four columns from file */
/* THIS IS WHERE IT WILL BLOW UP IF STREAM HAS BEEN */
/* CLOSED AND OPENED During output */
/* ================================================= */

try {

wis = new ObjectInputStream(new FileInputStream(weights));

} catch (IOException e) {
System.out.println("Failed to open input grid file in virtualImage.readyRead: " + e);
return false;}

try {
g0 = (float[])wis.readObject();
g1 = (float[])wis.readObject();
g2 = (float[])wis.readObject();
g3 = (float[])wis.readObject();

} catch(IOException e) {
System.out.println("Failed to read input grid file in virtualImage.readyRead: " + e);
return false;}
catch(ClassNotFoundException e) { }

jlast = 1;
return true;
}

}