Functional global with too many terminals

[Bublina]

Hello,

I am working on a project where I have to manage large sets of data. I used to keep these datasets in FGs with simple R/W functionality, but soon figured out, that passing in and out big data in/out of FGs consumes a lot of CPU and takes some time.

Therefor, I implemented all functionality on top of these data into the FGs. This solved the performance problem, but as more needed functionality occured, I started to run out of terminals and had to merge input parameters, that had nothing to do with each other into clusters. This resulted in a bit less readable code + I cannot help myself, this is the wrong way to do it.

Please help me answer my following question:

What is the correct way to manage large datasets to properly encapsulate them and get the best performance when working with them.

p.s. I was thinking about implementing it using LVOOP, but was afraid of some performance penalties from unwanted datacopies. (I am still learning the LVOOP, so I rather decided to stick to sth. I knew)

Thank you.

[crossrulz]

What kind of data do you have?  What operations are you trying to do with it?

This is kind of sounding like a good place to use a Data Value Reference.  Then you make a VI for each of your operations.  You just need to pass the DVR around.

[JÞB]

Thats a tough one.  The best thing I can imagine is to wrap that FGV inside a set of subvis for each "Method" and only expose the terminals required for that method.

[Bublina]

It is an array.

If I use the data reference, wont LabVIEW make some unwanted copies, if I access the reference in different VIs? (The app has at least two main windows - two monitor app, that make heavy use of the global simultaneously)

[Bublina]

---

@JÞB wrote:

Thats a tough one.  The best thing I can imagine is to wrap that FGV inside a set of subvis for each "Method" and only expose the terminals required for that method.

---

Thanks, that is a good idea. I will probably stick to it. Is it possible to create this way (from those wrapper VIs) a polymorphic VI ? Would this preserve the data encapsulation (I see no problem, just asking; never used those polymorphic VIs before)?

This would be perfect as I could have same VI icon with different terminals and still keep data on single heap.

Edit : ++ I could read the function called in the selector, it is sweet. I already started conversion 🙂

[Intaris]

No, the DVR does NOT make copies.

You need the Inplace Element (IPE) structure to access the data within the DVR at which point you are working "in place" but any other code must wait before it can attain a lock on the resource (at the start if the IPE)

[Bublina]

---

@Intaris wrote:

No, the DVR does NOT make copies.

You need the Inplace Element (IPE) structure to access the data within the DVR at which point you are working "in place" but any other code must wait before it can attain a lock on the resource (at the start if the IPE)

---

Thanks for answer. I tried to find out how does the structure that allows reading (and forces writing) work, but found nothing.

When trying a simple stupid experiment, LabVIEW shows it allocates new memory for every access to the reference.

Is this OK? (I had problems with passing those large data through terminals from function to function or VI to VI, where these dots also appeared)

[Intaris]

I'm no Expert ont he DVR.

I am led to believe that IF a copy is made, it's only the DVR itself (a pointer) which is either 32-bit or 64-bit depending on your LV bitness.

This is essentially negligible, especially when working with large datasets, but I will make way for someone who knows what they are talking about to answer.

I do know however that the visual feedback regarding buffer allocations, while usually reliable, is not 100% correct all the time.

[DFGray]

I second using the DVR.  It is far more scaleable, and performance will be much better than an FGV for large data sets.

The DVR does not make a full copy of the data at each access.  The buffer dot is a pointer to default data in case the actual data fails to materialize, so that no error will occur.  Technically, it is correct.  Practically, it is negligible.  You can verify this yourself by creating an array of a million data points, loading into the DVR, then using the In Place Element to unload and reload the data.  Now step through the VI while watching your operating system memory monitor for LabVIEW.  You will get a memory increase when you allocate the data, but you should not get a spike when you go through the In Place Element structure (TIP: put a single frame sequence in the In Place Element structure and run your data wire through it so you can stop your single step inside the structure).

Good luck.  Let us know if you run into issues.

P.S.  I consider the fact that you have to pass the DVR reference around a plus.  It increases data encapsulation.  You can only use it where you explicitly allow it to be used.

[crossrulz]

I'm in Intaris.  You pass a reference to a place in memory.  The manipulation is done in place, so there should be no copying, other than possibly the DVR itself, which is just a small pointer.

I too have noticed some freaky things with the buffer allocation.  I don't trust it with many of the situations like this.