efficient way to read a hex file to an array of U16 data?

[mlloyd]

I have a large file (~4.2 Megs) that I want to read in and store as an array of U16 data. Right now, I'm reading the whole file to a string. The string is inputted to a for loop that picks out every four characters (string subset.vi) and stores them in an array. I then convert the entire array to U16 using the hex string to number VI.

The main problem is that the for loop takes about 2 minutes to iterate... are there any faster was to convert a very long string into a hex array?

[JeanPierre]

Taking a whole file in a temporary copy is memory-wise inefficient.
Taking this copy to create another temporary array or 4 byte-strings is also inefficient.


Maybe the most direct way to accomplish the task without allocating temporary strings/arrays is reading the file 4 byte at a time and making the conversion using "Scan from File". See attached picture.

[rpursley8]

You shouldn't have to use a loop. Take a look at the attached. I haven't tried it with a really large file, but it should work fine.

[mlloyd]

Thanks for the suggestions. I'm still running into a few snags...

The Scan from File doesn't provide a way to input an offset if it's going to be used in a loop, so it has to be used in conjunction with Read Characters from File. I tried this and found it to also be slow.

The non-loop version gives me numeric ASCII code for my characters instead of the characters themselves. That is, if my file in hex is "10FFFFFF..." I get in my array "3031 4646 4646 4646...". What I really want in the array is "10FF FFFF..."

Let me know if you have any more suggestions...

[rpursley8]

OK try this one. Since you really don't have a hex file, the hard part is converting to hex numbers from ASCII characters without a for loop. I think this will work.

[rpursley8]

If you don't have LabVIEW 7.1 here is an image of the vi.

[JeanPierre]

---

mlloyd a écrit:
The Scan from File doesn't provide a way to input an offset if it's going to be used in a loop, so it has to be used in conjunction with Read Characters from File.

---

ARGhhhh! That stooopid Scan from File function doesn't update the file index! How useless it is!!! That means one reads a complex sting with variable length strings items and numbers but can't know how much characters you have used!

Sorry I thought that Scan From File worked like Scan From String which returns the offset after scan. I've modified the example using the file function Seek to position the file index before each Scan.

I agree it is slow but loading the whole file and make temporary copies is also slow. Maybe you could try to read the file few kilobytes at a time and avoid splitting the string into 4 bytes strings: that makes a lot of memory allocations.

Message Edité par Jean-Pierre Drolet le 03-10-2005 03:42 PM

[mlloyd]

No problem about the Scan From File... I was surprised to find that's how it worked as well! I've tried the second solution Randall offered with stunning success: my runtime is down from two minutes to about 6 seconds! I think this will work well. 🙂

Thanks a ton...
Michelle

[altenbach]

Michelle,

I had the gut feeling that Randalls version could still be improved, too much pink!!! 😮

Please try the attached modification. On my system it is about 14x faster. Basically, it converts the string to U8, then uses a lookup table on the ASCII code to get the real number. Adjacent nibbles are then merged creating a U8 Array of half the size which is cast as U16 array, yielding the final result. WHewwww!

I think it gives the same result, but please test with your own data. The slightly more complex code is probably worth the speed gain. Please try it. (If you want to use it as subVI, change to subroutine priority for another speedup). 🙂

I only spend a few minutes on this so I'm sure it could be further optimized. This might be a topic of one of the next LabVIEW Challenges 😉

Have you explored Jean-Pierre's suggestions?

[altenbach]

Sorry, the above is still way too complex. Here's a streamlined version that is another 30% faster. 🙂

(Let me know if you want it in LabVIEW 7.0).