Tricky Rounding Issuie

[Steve_G]

Hi,

I have a file of numbers that I read as a spreadsheet.  The numbers are all three decimal places long, such as 0.466, however when the file is read some of the numbers get extraneous extra values, like 0.4660000000000000250.  I've tried all sorts of things to round the values to 3 decimals (ie, back to 0.466) but nothing works.  See the attached vi in Labview 8.6 to see what I'm talking about.  Does anyone have any suggestions?  I feel like there's something really obvious that I'm missing.

Steve 

[smercurio_fc · Accepted solution]

What you are seeing is something that has been brought up in this forum more times than I can count, and I learned to count really high in school. Computers cannot store floating points numbers exactly. See http://en.wikipedia.org/wiki/Floating_point_number#Accuracy_problems.

[jyang72211]

Attached is what I did.  I just change it into a string and back to a number.  Does it work for you?  The problem is indeed strange.  Maybe someone else can comment on this too?

Yik

[jyang72211]

Cool.  I didn't know that.  It kind of make sense though.

Yik

[Matt_W1]

The relative precision of doubles is 16 decimal digits, asking for more than that (your indicators are trying to display 20) will likely cause problems. Going over that limit will affect the displays of the few numbers that don't suffer from any accuracy problems in floating point. Lower the precision to 16 and hide the trailing zeros and its display will be what you're looking for.

another thread about the "issue"

Message Edited by Matt W on 03-12-2010 08:21 PM

[jyang72211]

Hi,

I found the solution.  You can use "In Range and Coerce" function to fix the problem.

Yik

http://forums.ni.com/ni/board/message?board.id=170&thread.id=435330

[dan_u]

---

jyang72211 wrote:

Hi,

I found the solution.  You can use "In Range and Coerce" function to fix the problem.

Yik

http://forums.ni.com/ni/board/message?board.id=170&thread.id=435330

---

Solution to what? This is not a solution to the display problem. The solution to the display "problem" is just to limit the number of significant digits of the numerical controls/indicators.

It makes sense to use "In Range and Coerce" to compare floating point numbers (instead of "Equal"), but this was not the problem here.

Message Edited by dan_u on 03-13-2010 11:14 AM

[nitad54448]

hi

for this kind of cosmetic problem I am using multiplication by 10 to the power of number of digits, conversion to integer then division by 10 to the power of number of digits

 say, for your case 

0.46500234, if you want to remove the 000234 do 0.46500234*1000=465.00234, then convert to integer 465 then divide by 1000... 

is this what you want ?

N

[James@Home]

Steve_G

I had a need several years ago to modify the precision of numbers.

My need was based on limiting precision prior to specification limits.

My solution is what jyang72211 suggested, but thought I'd share my VI.

This won't help identify the cause of you problem, but may be a useful until corrected.

 

Hope this helps.

James

[smercurio_fc]

---

jrjones wrote:

Steve_G

 

This won't help identify the cause of you problem, but may be a useful until corrected.

---

The "cause of the problem" is a lack of understanding. There is no problem. Did you read the Wikipedia page? There is nothing to correct. You must understand the way numbers are stored in computers and simply deal with it.