append .mat array

[cmdettmar]

I am using the Save .mat lev. 5 application to convert 3D arrays to .mat file and it works fine. However, I would like to collect very large arrays, (160x128x250000) which is way over may RAM capabilities, is there a way to write every 50 or so layers to the hard drive and append the file in a loop?

Thank you.

[RavensFan]

Where does LabVIEW fit into this question?

Perhaps you should be asking this question of whichever company makes "Save .mat lev. 5 application".

[cmdettmar]

I misspoke, save to matlab isnt an application, its a labview conversion tool for converting arrays into matlab format. I was refered to it on this forum.

[RavensFan]

Ahhh, okay.  Matlab.  That tidbit of information was left out of the original message.  Perhaps you can post a link where you found the conversion tool on the forums.

[stefan!]

Ok, there is still no answer. I tried two tools to write continuous to a mat-file. I tried: 

- CONV2MAT.VI   by Albert K. Wijnja (http://digital.ni.com/public.nsf/websearch/2f8ed0f588e06be1862565a90066e9ba)

and I tried

matio v0.1-8 by Author: James R Cook (http://matio-labview.sourceforge.net/)

But I failed to write continuous to a mat-file. Maybe it is the logic of the mat data container (I haven't checked that)

that makes it difficult to write line by line of a 2D-Array.

Therefore, I guess I will write to a standard text file, read the data into octave via text file and save it as *.mat.

Would be nice, if someone knows a more sophisticated way to do the job.

Stefan

[mcduff]

Appending to an existing .mat file can be difficult, especially depending on how it has been previously saved and how you want to append to the data.

Added to this post is the pdf for Matlab file format, if you do not have it already.

1.) Matlab can be compressing using the zlib library. Obviously, if you want to append to this file you may need to uncompress first. if you are writing the VIs yourself, I suggest not to compress your data, as such a large data compression would take a lot of memory and CPU usuage.

2.) It depends on how you are appending. If you are appending to an existing array, then besides appending the data, you would also have to modify the header to the array in order to tell Matlab how big the array is. If you are just adding an additional array to Matlab, ie one with a new variable name, then you can just append that structure (Header and data) to the already open Matlab file. (This is the simple way to do this.)

Cheers,

mcduff

[stefan!]

Thanks for the document. You are right, each new element (e.g. a 2D-Array) is described also by its size in bytes and if you want to store an array line by line, you do not know which size you will have at the end. Is there another file format that works line by line and has advantages if you work with octave/matlab?

Stefan

[mcduff]

I am not sure about other file formats. Personally, I really do not work in Matlab, some colleagues of mine wanted the data in mat fromat though.

If Matlab/Octave can import a binary file whose format you specifiy, then you can just add data to the file.

mcduff