Parse lines out of a StringBuffer and/or InputStream
807588Feb 24 2009 — edited Feb 25 2009I am loading and parsing a ~40MB text file. I am retrieving this file using the JCraft JSch SFTP library. This has given me no problems so far.
Since I need to process this file line by line, my first approach was to read a character at a time off the InputStream provided by the SFTP library, collect characters until a new line was reached, then parse the line. This proved to be extremely slow. Now I'm pulling 1024 characters at a time into a byte[] from the stream and pushing that array into a StringBuffer until the end of the stream has been reached. It has sped up considerably, but now I don't know what the best way would be to process the buffer.
What would be the best way to split off and process every line of this file? It seems redundant to pull the whole thing into a string buffer then parse it again by using charAt() and substring() to find and chop off lines. Reading character by character from the stream works in theory but is much too slow. Trying to parse the byte[] with fragments of lines is impractical and error prone.
Memory is not an issue since it is running on a dedicated batch server, but speed is important.
Thank you.