11-29-2017 11:28 PM
Can anyone please explain what does 'Approximate number of Decimal digits' indicated in 4th column of the image.??
11-29-2017 11:48 PM
How many digits you can see if you wrote that number out in decimal, i.e. base 10.
Think significant digits.
11-29-2017 11:59 PM
Thanks for the reply.
just for clarification, taking an example 'I8' numeric data type, we have '2' as approximate number of decimal digit written in 4th column of the image. The range is -128 to 127 , which involves three significant number..!! Ex: 123.
Am i right ?
11-30-2017 12:11 AM - edited 11-30-2017 12:18 AM
I think it has to do with the fact that all floating point numbers cannot perfectly represent decimal digits or fractions of a whole number. The chart is a guide to where you will start to see error in the number in terms of number of digits to the right of the decimal.
Example: Divide 1 by 10 and show the result in a double indicator, the chart says you get about 15 digits of precision, in this case we got 17 digits before error.
11-30-2017 12:22 AM
@vishal777 wrote:
Thanks for the reply.
just for clarification, taking an example 'I8' numeric data type, we have '2' as approximate number of decimal digit written in 4th column of the image. The range is -128 to 127 , which involves three significant number..!! Ex: 123.
Am i right ?
Yes. But it also says "approximate". That third digit isn't really a full digit. It is only 0, 1, or 2 for an unsigned U8, and -1, 0, or 1 for a signed 8-bit. So it is really 2+ significant digits.