dimanche 28 juin 2015

Accuracy of the decimal number type versus the double type in .Net

Consider the following code:

    Dim doubleResult = (21 / 88) * 11
    Dim decimalResult = Decimal.Divide(21, 88) * 11

The doubleResult is 2.625, as it should.

The decimalResult is 2.6249999999999996, so when rounding this to two decimal places will give an incorrect result.

When I change the assignment to:

    Dim decimalResult = Decimal.Divide(21 * 11, 88) 

The result is 2.625!

We adopted the decimal type in our application hoping that it would give us increased accuracy. However, it seems that the decimal type just gives slightly incorrect results on other calculations than the double type does, due to the the fact that it is ten based, rather than two based.

So, how do we have to deal with this idiosyncrasies to avoid rounding errors as above?

Aucun commentaire:

Enregistrer un commentaire