Append data to binary file as concatenated array

[pilina]

Hello,

I have a problem that may has been discussed few times, but I do not have clear answer.

Normally I have camera producing 2D arrays as images. I send them to gathering loop with a queue and then index them as a 3D array and at the end save to binary file.

It is working fine. But I am starting to struggle with some memory issues, when number of those images exceeds more than ca. 2000.

So I am trying to take advantage of fast SSD drive and save images by bulks (eg. by 300) into binary file.

In the attached diagram where I am simulating camera with reading some file taken before. The program works well, but when I am trying to open the new file in secondary diagram, I see only first 300 (in this case) images.

I've read on the forum, that I need to set Count as -1 in Read Binary File and then I can read data as cluster of arrays. It isn't very good for me, because I need to work with the data with Matlab and I would like to have the same format as before (e.g. 3D array - 320 x 240 x 4000). Is there any way to append 3D array to existing file as concatenated?

I hope it makes sense 🙂

Thanks,

Honza

[Bob_Schor]

Honza,

     Since this is your first Post on the Forum, I won't yell at you.  Suppose I asked you for help with a big Matlab program, and I sent you a picture of 200 lines of code.  Would you be happy about that?  Probably not, since what you'd really like to do would be to open it in Matlab, look at it, maybe even execute it ...

     In the LabVIEW Forum, we deal with LabVIEW Code, expressed as VIs.  If you show us a picture of your code, we cannot examine it closely, we cannot tell what Version of LabVIEW you are using, and we cannot test or modify your code.  Some of us (not including me) are patient enough to look at pictures, but having actual code, as attached VIs, is much, much better.

     As you get more experienced with LabVIEW, you will start using LabVIEW Project and multiple VIs in your work.  You can attach your entire Project by compressing the file that contains it, and attaching the resulting .ZIP file.

     I look forward to seeing your code.  It will help me understand your question better, and I can test any suggestion that I make to be sure that it works for you.

Bob Schor

[pilina]

Hello.

First of all, thanks for quick response.

I saw bunch of less difficult things solved by the pictures, so I though it could be enough. My bad.

I've changed camera simulation (in LT_save.vi) with generation of 2D array of random numbers, so I do not need to send a lot of data.

The second code I use is the one I am normally using for reading those binary file without any analysis.

Thanks again.

[Bob_Schor · Accepted solution]

• Good for You to simulate Image Creation using a 2D array of random numbers!  Always good to model the actual problem (file I/O) without "complicating details" (Camera handling).
• Good use of Producer/Consumer in LT_Save.  Do you know about Sentinels?  You only need a single Queue, the Data Queue, sending an array of data to the Consumer.  When the Producer exits (because the Stop button is pushed), it enqueues an empty Array (you can just right-click on the Element input and choose "Create Constant").  In the Consumer, when you dequeue, test to see if you got an Empty Array.  If you do, stop the Consumer Loop and Release the Queue (since you know the Producer has already stopped, and you've stopped, too).
• I'm not sure what you are trying to do in the File_Read_3D routine, but I'm going to say "it's wrong".  So let's analyze the situation.  In some ways, your two routines form a Producer/Consumer "pair" -- LT_Save "produces" a file of 3D arrays (mostly 300 pages, unless it is the final bunch of data) and file_read_3D "consumes" them and "does something", as yet somewhat poorly defined.  So you could (and maybe should?) merge these two routines into a single "Simulator".  Here's what I mean:

This is taken directly from your code.  I replaced the "Stop Button" Queue with Sentinel code (which I'll go over), and have added a "Producer Queue", Sim File, to simulate writing these data to a file (it also uses a Sentinel).

Your existing Producer code puts single 2D arrays into the Data Queue.  This routine takes them out and "accumulates" up to 300 of them at a time before "doing something with them", in your code, writing to a file, here, simulating this by writing to a 3D Sim File Queue.  Let's consider the "easy" case first, where we get all 300 elements.  The For loop exits, returning a 3-D array consisting of 300 2D arrays, which we simply enqueue into our Sim File, our simulated file.  You might notice that there is an Empty Array? function (which, in this instance, is never True, always False) whose value is inverted (to be always True) and wired to the Conditional Terminal of the Indexing Tunnel.  The reason for this strange logic will become clear in the next paragraph.

Now, consider what happens when you push the Stop Button and your Producer (not shown) exits.  Since we're using Sentinels, it enqueues an empty 2D array.  Well, we dequeue it and detect it with the "Empty Array?" function, which we use to do three things:  stop the For Loop early, prevent adding the Empty Array to the output Indexing Tunnel by using the Conditional terminal (Empty Array = True, Negate changes to False, so empty array is not appended to the growing array), and it also cause the While loop to exit.  What happens when we exit the While loop?  Well, we're done with Data Queue, so we Release it.  We also know that we've enqueued the last "good" data in the Sim File Queue, so we create a Sentinel (an empty 3D array) and enqueue it for the to-be-developed Sim File Consumer Loop.

Now, here's where you come it.  Write this final Consumer loop.  Should be pretty straight forward -- you Dequeue, and if you don't have a 3D empty array, you proceed as follows:

• Your array consists of N (up to 300) 2D Images.  In a single For Loop, you extract each image and do whatever you want to do with it (display it, save to file, etc.).  Note that if you write a sub-VI called "Process One Image" that takes a 2D array and does something with it, you will "declutter" your code by "hiding the details".
• If you did have an empty Array, you simply exit the While loop and release the Sim File Queue.

OK, now translate this to Files.  You're off to a good start by writing your file with Array Size headers, which implies that if you read a file into a 3D array, you will have a 3D array (just as you did in the Sim File Consumer) and can carry out the same processing as above.  All you need to worry about is the Sentinel -- how do you know when you've reached the end of the file?  I'm sure you can figure this out, if you don't already know ...

Bob Schor

P.S. -- you should know that the Snippet I posted was not created "correctly" all at once.  I actually pasted about 6 versions here, as I kept finding errors as I was writing the description to you (like forgetting the "Not" function in the Conditional terminal).  This illustrates the virtue of Writing Documentation -- it "slows you down", makes you review your code, and you say "Oops, I forgot to ..."

[pilina]

Thank you very much for your advice.

Manually writing down Array Size Header did the job.

Also thanks for knowledge of queues handling, I did my code acc. to LabVIEW Queue tutorial (stopping conditions and stuff).

Honza

edit: The working LT_save attached.

note - LT save is a part of larger code which also handles the camera and stuff, but it contains quite a few 3rd party commercial VIs, so I thounght I was not able to produce them. Producing part of that code (loop reading CCD data) still generates only 2D array of U16.