LabVIEW and large data sets are incompatible. T/F? I say T.

[*3d0g]

Oops! 100M times in a row.

[altenbach]

Sorry, posting by phone, so I may have missed some subtleties.

First of all, there is no oscilloscope in the world that deserves DBL data. What is the data type of the driver output ( you said something about "waveform object" or similar. Also, if you want the difference of tho channels, can't you just wire it up differently and get that directly?).

It is important to know that arrays are contiguous in memory, and you can run out of contiguous memory way before you run out of free memory. Bad coding habits such as constantly growing or resizing arrays, changing representations and similar can be very detrimental. In-placeness is the key here.

Once you can show us how you are doing things we might be able to give specific advice.

What kind of indicators do you use and how much data they contain.

[*3d0g]

Thank you for entering the discussion, altenbach.

I can try to answer your question(s).

These are dirt simple arrays I'm working with, here.  (I can provide a picture or a vi if you must have such.)  I just

1. declare/initialize an array of 32,000,000 elements of type double and create an indicator to display it on the FP.

2. I run the vi and LabVIEW has no issue.

3. I ctrl-copy the block, creating a duplicate.  I save this as a new vi.

I repeat the above four times, each time adding another copy.  I found I was good until that fourth vi ran.  At that point I have a vi that won't finish because LabVIEW has no more memory.

Three is fine. Four is not.  I interpret this as I need to use the heap to be able to work with the oscilloscope, for my vi really does nothing and LabVIEW can't work with it. 

Understand this is research for future need.  I'm trying to prepare.  I have looked at the OEM's driver, their sample vi.  I looked at a simple capture vi provided and note that the response is given to me via a waveform indicator.  The assumption is I would/could do the same four times, somehow.  Another assumption is that the indicator itself can actually contain 32,000,000 samples as claimed. 

Ok, fine, I have 32,000,000 samples in an indicator/object.  Now what?  This is the question I am trying to answer with my little experiment.  So far I'm seeing LabVIEW not being able to handle two differential measurements of 32MSa.  I am going to have to figure out another way to do it.  My present thought process towards a suitable algorithm is to use the heap and do all/most work outside of LabVIEW, for if LabVIEW can't even build the simple arrays, I can't see it being used to work with the data, after.

I know there are professionals, here.  I'd like to hear what they have to say.  Have I missed something?  Is there some tool(s) in LabVIEW that can handle this situation?

[*3d0g]

The indicator I'm using for each array is simple: right-click array output terminal and create indicator.  It's dirt simple.

I need to work with four 32MSa arrays.  Granted, it's possible I could use the math of the scope and send out two 32MSa waveforms instead of four.  But there's a bigger question here, I think.

Has anyone here received a 32MSa waveform from a scope into LabVIEW?  How did that work out for you?  How about multiple such waveforms?  Once into an array it'd seem reasonable to be able to break each data set into smaller arrays and work with them as needed.  Waveform display creation could be handled by (I think the term is) decimation, display of less samples by grabbing only a single sample over a defined interval multiple times.  But such comes after the waveforms can actually get into LabVIEW in the first place.  There is only so much memory LabVIEW can work with, so it seems.

[GerdW · Accepted solution]

Hi 3d0g,

---

@*3d0g wrote:

The indicator I'm using for each array is simple: right-click array output terminal and create indicator.  It's dirt simple.

I need to work with four 32MSa arrays.  Granted, it's possible I could use the math of the scope and send out two 32MSa waveforms instead of four.  But there's a bigger question here, I think.

Has anyone here received a 32MSa waveform from a scope into LabVIEW?  How did that work out for you?  How about multiple such waveforms?

---

An indicator uses it's own buffer so it will double the memory requirement for your 32MS*8bytes large arrays…

Keep in mind: in LabVIEW the variable is the wire!

You can stream all data into (TDMS) files: no need to hold ALL data in memory.

You can analyze your data in smaller chunks: no need to hold ALL data in memory.

---

@*3d0g wrote:

There is only so much memory LabVIEW can work with, so it seems.

---

This is valid for all 32bit executables: they all can handle at max 2GB of memory!

In your special problem you also have to keep in mind: arrays need contiguous memory blocks, so each 256MB large arrays needs a memory block of atleast this size. Each operation, that requires a new memory buffer will need an additional memory chunk of that size! And that 2GB limit only provides <=8 blocks of 256MB available…

So the simple solution is to switch to LabVIEW-64bit as it is recommended for operations with large memory requirements.

The second simple option is to modify your program to use less memory…

[rolfk]

There are two possible limits you can run into in LabVIEW. The first is the total amount of memory that a process can allocate. For a 32-bit process that is at most 3GB of memory. This must hold all the data and code for you program. That means not just your large arrays but also the LabVIEW VIs, its data structures, LabVIEW support libraries, drivers and DLLs.

Typically when your array data hits a few 100MB, Windows runs out of continuous memory in a 3GB process heap. That can be a little higher just after starting your program but as longer as you run your program, the more memory will tend to get fragmented and the less there will be available as a continuous chunk of memory. But a LabVIEW array must fit in a continuous memory area. There are no sparse LabVIEW arrays.

If you want to have a bigger chance for larger memory blocks to be available, you absolutely should go with the 64-bit version of LabVIEW! Trying to create several 100MB arrays and complaining that you run in out of memory situation but insisting to use the 32-bit version of LabVIEW, is like complaining that you can't drive faster than 50 m/h but insisting on using a Ford Model T car because of its antique feeling.

The second limit is independent of the bit-size of LabVIEW. A LabVIEW array can not have more than 2^31 elements! But if you run into that limit you are using 64-bit LabVIEW, and run on a machine with probably more than 32 or 64 GB of physical memory and most likely use a data model that is highly inefficient in more than just one way.

[altenbach]

---

@*3d0g wrote:

I can try to answer your question(s).

---

So why the need for DBL? If that "Driver" outputs DBL, these guys are unclear of the concept. There is no A/D converter that has 50+bits resolution! Most likely I16 would be plenty, i.e. 4x less!

There is also absolutely no need for an indicator (which will ~triple the memory foot print adding copies for the display and transfer buffer). You can subtract two arrays without having to display the two inputs first. The wire is the variable!

No human can look at an indicator with 32M samples and efficiently understand it. This is much more than the number of pixels in your display!

You still have not said if your LabVIEW is 32 or 64bit. (I assume the OS is 64bit, right?)

[cstorey]

>> An oscilloscope is capable of capturing 50MSa/channel.  You want to work with the captures, differentially; therefore, you need two such captures for a single signal capture.

Perhaps the question should be do you need to work with extremely large datasets?   When working with scopes and LabVIEW the biggest issue I always face is the time it takes to transfer data to the computer.   Often times people want to transfer 4 traces to the computer and then process them, and then repeat that as fast as possible.  Because of the data transfer bottleneck, I always advice a few things when designing the data capture LabVIEW code..

- Make full use of the instrument processing!  Do as much data processing on the scope as possible since "modern" scopes have dedicated hardware for fast processing. Your program design shouldn't try to duplicate your scope features. Its usually orders of magnitude faster (even on 20 year old instruments) to have the scope do the math before transferring as much data s you really need.  

- Make use of cursors and peak finding features on the scope.  If you only need to get a single feature out of the program you can write a very different program; do a hi-res scan for very accurate peak stats then a second lower-resolution scan for transfer to PC.  Or use the peak find to zoom in on region of interest and throw away unneeded samples that way.

- Only transfer as much data at the required resolution to the computer.  Sure it sounds nice to have 300M samples, but no matter how fast the computer you will run into issues processing and storing that much data.   If the feature you are truly interested in can be characterised in 4M/1M/512 or 256 samples use the minimum until there's a good reason to increase.  While scopes advertise high sample/high speed capabilities its rare that you need both at the same time to get good data.

So, of course here are limitation to working with huge datasets, but there are good alternatives to doing so as many have pointed out in this discussion already.  

[*3d0g]

The LabVIEW is 32-bit and the system is 64-bit.  Changing this is not an option.

That looks like a waveform holding doubles, to me.  This was snagged from the multiple capture sample vi.

The good news is I might be seeing a way to pull this off, without leaving LabVIEW.  The "in place element" concept may help.  It dawned on me that if the samples are indeed in the waveforms, then they're already in LabVIEW.  Now I just need a pointer (the pointer concept, as a LabVIEW tool.) ...that made sense to me, at least -- it's another path to explore.

[*3d0g]

BTW, this driver I'm looking at is in the form of a LabVIEW project -- the folder is just dropped into instr.lib and the tools show in instrument I/O drivers.  The project has an Examples folder, and that's where I'm getting the vis and learning about what I'd be working with.  Of course I started with the OEM; I asked them what to do.  However, they couldn't help, saying they had no LabVIEW license, which I find odd being is they're saying their equipment works with LabVIEW.  It seems very reasonable to me to question how would they know it it works as claimed if they can't use LabVIEW?  I can only figure that years and years and years ago someone wrote something that worked enough for them to tag on product after product, after, claiming they'll all work with LabVIEW but without actually testing any of it.  Is there really 32MSa in those waveforms?  I'm not so sure about that.