How do i avoid rounding

[blessedk]

Hello,

I am trying to subtract an array of very small numbers from 1. The resulting output ends up rounding up as an array of 1s. I am expecting to get numbers very close to 1 (say 0.9999999997 for exampe) but not exactly 1.  I have tried to ensure that both inputs to the numeric operator have the same type DBL/SGL etc. I have also tried scaling both inputs up and dividing after.

I know this must be easy but I dont have a ready solution  🙂  VI attached. I need both probes 64 and 65 to maintain a good number of precision digits. Thanks.

[Norbert_B]

As you didn't store your "very small numbers" as default values, the attached VI doesn't show much except the code itself.

However, your numeric control/indicator is still in default configuration for showing 6 digits of precision. You might want to increase that in the controls/indicators property dialog to show e.g. 15 digits of precision.

Norbert

[Ken_Naylor]

Increase the digits of precision

to get

Ken

[altenbach]

• When attaching an example VI, make sure the controls contain typical default data so we can see what you are talking about.
• Not the cause of your problem, but mind your representations. that "1" diagram constant should be DBL, not SGL. Notice the coercion dot?
• DBL values have a precision of about 15 decimal digits. You seem to be worried about the displayed precision of controls and indicators.
• The displayed precision is only cosmetic and you can change the display format to anything you want (right-click...display format"). The display format does not affect the underlying data, so if you would square the result and subtract  from 1 again, you would get the original data back (within the limits of DBL computations). Try it.
• Your VI does not contain any "probes", but if you want a probe that shows values with higher precision, you can create custom probes.
• You cannot change the format of execution highlighting.