Java APIs

Hello,

I've recently been playing with a program whose intent is to grab files from URL calls, and then send the files via a servlet to another web page. For simplicity's sake, I've been using "MultiPartFormOutputStream" to handle the post requests to my servlet (thanks to bsampieri, I think his code can be found here).

To give a quick summary, MultiPartFormOutputStream will essentially let me take a URL and some field/value pairs, and make POST data prepared to be sent to a web servlet. Additionally, it also allows for file upload requests via POST data, through several implementations of a writeFile method. The method I'm current using is the one that takes an InputStream as the location of the file to be uploaded. For simplicity's sake, here's the code for writing the file to the POST request ('out' is a DataOutputStream):

/**
	 * Writes a input stream's contents.  If the input stream is null, a 
	 * <code>java.lang.IllegalArgumentException</code> will be thrown.  
	 * 
	 * @param  name      the field name
	 * @param  mimeType  the file content type (optional, recommended)
	 * @param  fileName  the file name (required)
	 * @param  is        the input stream
	 * @throws  java.io.IOException  on input/output errors
	 */
	public void writeFile(String name, String mimeType, 
			String fileName, InputStream is) 
			throws java.io.IOException {
		if(is == null) {
			throw new IllegalArgumentException("Input stream cannot be null.");
		}
		if(fileName == null || fileName.length() == 0) {
			throw new IllegalArgumentException("File name cannot be null or empty.");
		}
		/*
		--boundary\r\n
		Content-Disposition: form-data; name="<fieldName>"; filename="<filename>"\r\n
		Content-Type: <mime-type>\r\n
		\r\n
		<file-data>\r\n
		*/
		// write boundary
		out.writeBytes(PREFIX);
		out.writeBytes(boundary);
		out.writeBytes(NEWLINE);
		// write content header
		out.writeBytes("Content-Disposition: form-data; name=\"" + name + 
			"\"; filename=\"" + fileName + "\"");
		out.writeBytes(NEWLINE);
		if(mimeType != null) {
			out.writeBytes("Content-Type: " + mimeType);
			out.writeBytes(NEWLINE);
		}
		out.writeBytes(NEWLINE);
		// write content
		byte[] data = new byte[1024];
		int r = 0;
		while((r = is.read(data, 0, data.length)) != -1) {
			out.write(data, 0, r);
		}
		// close input stream, but ignore any possible exception for it
		try {
			is.close();
		} catch(Exception e) {}
		out.writeBytes(NEWLINE);
		out.flush();
	}

To create the InputStream which contains the file, I'm making a web call which would retrieve a file as the returned InputStream from URLConnection.getInputStream(). The code I have written to do this works perfectly, except that the inputStream I get seems to be loaded to the heap all at once. If you don't have a very large heap for this (I'm using -Xms512M and -Xmx768M to set the min and max heap sizes) to perform the file download/uploads, you're almost guaranteed to throw outOfMemoryExceptions on larger files.

This all being said, I would like to know if there's a way to read only a small amount of my InputStream (from URLConnection.getInputStream() ) at a time, and give said data chunk to the DataOutputStream, without overloading my memory heap. My guess is that this involves finding a way to buffer my InputStream, but I am unsure as to how I would get this to work properly without getting the whole thing loaded to the heap at once.