Skip to Main Content

New to Java

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

Why is STDOUT to STDIN so dang slow in Windows?

802457Jan 26 2013 — edited Jan 30 2013
I have a JAR file that reads a specific file format and writes the data to STDOUT.

I have another JAR file that reads this data from STDIN and processes it.

This data has hundreds of millions of records, each of which are ~2K bytes long.

When i read this data from a text file i can read it at a blazing speed, like multiple million records per minute.

However, when i run:

java -jar DataReader.jar | java -jar DataProcessor.jar (this is an example, not the EXACT name of the jars...)

the processing time is MUCH MUCH MUCH less. We're talking MAYBE 20-30K per minute.

I use BufferedOutputStream in the class that writes to STDOUT AND for the class that reads from STDIN.

I have tried increasing the buffersize for the above classes in multiple increments, but it doesn't seem to change anything by much.

Is there some sort of an Achilles Heel that i'm missing that can cripple the speed of this operation if i'm not aware of it, or is this just a naturally doomed process because of something in Windows?

What kind of throughput should i expect in this situation? What is fair/expected?

Any info is greatly appreciated!
Comments
Locked Post
New comments cannot be posted to this locked post.
Post Details
Locked on Feb 27 2013
Added on Jan 26 2013
6 comments
1,096 views