Java double floating point precision vs MS Excel
807588May 8 2009 — edited May 10 2009Hello,
I know that in floating point computations there are precission issues related to decimal-to-binary conversions and also related to IEEE 754 specification restrictions (basically the number of bits used for fp numbers) shared with a lot of languages and computer systems and that Java language makes simply more evident because it shows doubles with full precision by default when you print them to screen.
A good sample is this calculation: 100.39 - 100.35 which doesn't gives the exact result that a human (thinking in base 10 arithmetic) could expect.
This substraction is a problem that hapens also in MS Excel ... but you can see it only if you shows the result with Scientific full precision or if you do a conditional formula that compares the cell with the substaction result with 0.04
But now I've at hand another calculation that shows a little (decimal) inaccuracy in Java and shows an EXACT result in Excel, and I dont understand why. This is the sample:
double a = 0.21;
double b = 28.5;
double c = a * b;
The result, converted to decimal, shows in screen like 5.984999999999999 but in MS Excel 2000 this time I get the exact result even showing it at full scientific precision and even if I do a conditional formula that compares the result with 5.985.
Both Java and Excel use the IEEE 754 floating point format with double precission, so I could expect the same roundoff problems in both.
Some hint?
Thanks.