Why is this number rounded down instead of up?

[alessia.lamanna]

In my TestStand 2019 application the number 0.18325, with 4 decimal digits (format string %.4f), is rounded to 0.1832. I would expect 0.1833,

as indeed happens correctly for 0.18326, ... , 0.18329, since the fifth decimal digit is greater than or equal to 5. Am I missing something? These are my custom number format settings.

Could anyone help me, please?

Thanks in advance, and best regards!

Alessia

[santo_13 · Accepted solution]

It would be because it would use Bankers rounding where a number in the middle of two numbers is rounded to the nearest even number.

So, anything at x.5 will use this particular case.

[crossrulz · Accepted solution]

Also note that Banker's Rounding is actually the IEEE recommended rounding due to statistical analysis.

[altenbach]

Bankers rounding is only part of the story here and is probably not the correct explanation in this edge case.

A format of %.4f is a purely cosmetic property. Assuming the underlying numeric is in DBL representation, the closest number to 0.18325 that can be represented correctly in binary is 0.183249999999999996 and thus will be rounded to 0.1832 in any typical rounding scheme.

Where does the value come from?

[alessia.lamanna]

@altenbach:

This is a limit value for a Numeric Limit Test step, and has been calculated as follows:

0.175+0.003+0.175*0.03