LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Not showing precision in extended numbers

I am working on a program that records all the input information into the header of a data file, then begins recording the input information from my measuring devices. The data in the header format will not read past the decimal, it will round the number to its closest whole number - I need to be able to record up to the hundreths place (.01). Ive been trying a number of different things and an truly stumped.

Download All
0 Kudos
Message 1 of 6
(2,658 Views)

That's because you use the Number to Decimal String.  That can only output integer numbers in a string format.  You need to use Number to Fractional String like you did for the actual data.


GCentral
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines
"Not that we are sufficient in ourselves to claim anything as coming from us, but our sufficiency is from God" - 2 Corinthians 3:5
Message 2 of 6
(2,654 Views)

That did it! Thank you

0 Kudos
Message 3 of 6
(2,642 Views)

Ahhh, my favorite pet peeve.  Of course, LabVIEW is exactly correct in the naming of the function - it follows exactly the convention used to name the others - e.g., "Number to Hexadecimal String," "Numnber to Octal String," etc... but it is SO CONFUSING because we tend to think of decimal meaning "x.y" when in reality it means "base10."  And even more confusing is, we tend to think of fractions as numerator/denominator, so "Fractional String" would intuitively mean a string written in either proper or improper fractional notation.

 

There is no real fix, because it's not really broken.  It's just something that probably trips up just about every LabVIEW programmer once.

Bill
CLD
(Mid-Level minion.)
My support system ensures that I don't look totally incompetent.
Proud to say that I've progressed beyond knowing just enough to be dangerous. I now know enough to know that I have no clue about anything at all.
Humble author of the CLAD Nugget.
Message 4 of 6
(2,616 Views)

or dozens of times.......

Doug

"My only wish is that I am capable of learning each and every day until my last breath."
0 Kudos
Message 5 of 6
(2,597 Views)

Think of it as the radix point and not the decimal point.  There is a binary point, and a hexadecimal point, but we do not really think naturally in terms of 1/2 or 1/16.  Perhaps the word integer should have been chosen in place of number in those functions...

0 Kudos
Message 6 of 6
(2,591 Views)