FileInputStream and what sizes of files it's capable of handling
807606Apr 3 2007 — edited Apr 3 2007Hey all -
I've got a few enormous data files (7gigs, 9 gigs, etc...) that I'm working with. I've got a program that will just blindly copy the first N bytes of an input file into a separate file. For my purposes, it's easier for me to work with the first ~3 gigabytes of one of my big data files than working with the whole 7 or 9 gigs.
I'm using a FileInputStream and all appears to go well using the 7 gig file. I can copy out the first 2 gigs into a separate file and everything works great.
However, when I try to do the same on the 9 gigabyte file, the copying stops after just shy of 1 gigabyte (~930mb or so). My loop is still going from 0 bytes to 2 gigabytes, no actual bytes are read from the input file after ~930 megabytes.
I threw in some print statements and it appears that at the beginning of program execution, the FileInputStream.available() call returns only 993718656 (~900 megs) when I'm running on the 9 gigabyte file.
On the 7 gigabyte file, FileInputStream.available() first returns 2147483647 - considerably larger than what .avaialble() starts at on the 9 gigabyte file.
I've read in a few posts that the .available() call tends to be unreliable. I'm not using it by choice here, but more as a method of exploring why my program only copies the first ~900megs instead of the full amount I'd like copied.
Additionally, if theres a better Java.io type I should be using for working with monstrous files, please let me know!
Thanks in advance,
Brad