Consider the following code:
Dim doubleResult = (21 / 88) * 11
Dim decimalResult = Decimal.Divide(21, 88) * 11
The doubleResult
is 2.625, as it should.
The decimalResult
is 2.6249999999999996, so when rounding this to two decimal places will give an incorrect result.
When I change the assignment to:
Dim decimalResult = Decimal.Divide(21 * 11, 88)
The result is 2.625!
We adopted the decimal type in our application hoping that it would give us increased accuracy. However, it seems that the decimal type just gives slightly incorrect results on other calculations than the double type does, due to the the fact that it is ten based, rather than two based.
So, how do we have to deal with this idiosyncrasies to avoid rounding errors as above?
Aucun commentaire:
Enregistrer un commentaire