Create and commit large number of shared variables - dsc

[Tyri]

Dear all,

I have a small problem with the DSC engine and the creation and commitment of many shared variables.

I guess it is well known that these two processes can take a lot of time depending on the number of variables. I have no problems with that. As long as one can abort this process and / or show the user that the system has not crashed.

For this I wrote a small VI (in LV 2014 SP1) to check the behavior of the Create Shared Variable.vi and the Commit Shared Variable.vi.

First the variables are created in buffer mode (the commit mode takes far to long for a larger number of variables). Then I collect all and try to commit them.

First, I will say a few words about what we expected: Before testing these VIs we actually thought that the Commit Shared Variable.vi is somehow not delaying further execution of the VI. So we thought Commit Shared Variables.vi only triggers the actual commit procedure in the background and then the code in this VI continues. This could lead to the problem that at a later point other parts of the source code try to connect to a variable which is not there. To overcome this we thought we can just use the Get Shared Variable List.vi in a loop and check wether all variables are there and delay further execution. This could also include a button for the user to cancel the operation and show him for example how many variables are already there and how long the whole commitment process will approcimately take.

Nice idea, but obviously it is not working like that. You can use the VI in the attachments and set probes at the marked positions to obtain time information. I tried it with 1000 to 30000 variables. Unlike our assumption the Commit Shared Variables.vi seems to stop the VI for several seconds. On my PC and with 30k variables it needs 9 s, which is much faster then expected by us. Actually we expected several minutes. For us this is a good sign.

But: The Get Shared Variable List.vi which is executed in the next step needs increadible long (over 3 minutes in the case of 30k variables). And once it is finished, it shows all variables. The conclusion would be that the variables are in fact NOT commited when the Commit Shared Variable.vi finished its execution. Something is still happening in the background preventing the Get Shared Variable List.vi from working.

As a next step I tried to write a number to one of the variables once the Commit Shared Variable.vi is finished (this is the other case in the conditional disable structure in the loop). Try to use this with 10 variables and it works (at least on my PC). With 1000 it is not working. He cannot find the variable.

So, my question at the end of this long text would be: Is there any way to find out when all variables are commited and ready?

Thx for reading and for your help

Greetings Dennis

[DesasterMaster]

Hi Dennis,

Are you getting any error message while running the VI with so much variables?

In your VI you wire the bufered Variables to the get List VI.

So you will not get the output of the Commit VI of course.

When the Variables are commited they are already listed.

The more variables you write in the buffer the earlier you get a buffer overrun and the Commit Variables VI will not be able to find the variables that are not in the buffer because of that.

But to get more insight in your project, why do you want to create 30k shared Variables?

Maybe there is a better approach to your root idea to solve something?

Best regards

Tenshinhan

[Tyri]

Dear Tenshinhan,

first to your question about our project: In fact, we are right now developing a framework based on LabVIEW and the Actor framework for small and medium size experiments in physics. We have already several experiments using an older version of our framework (this one had no Actor framework included) so we know that there can be cases with some 10k of process variables. The point is: I just don´t know what exact requirements future experiments will have, so I would like to have our framework as robust as possible and I would like to know its limits. For example: we have a laser facility using our old framework. They have roughly 13k process variables but distributed over something like 20 nodes. Then the performance of the shared variable engine may not be an issue anymore. I also found out yesterday that it is even possible to create more than 700k variables (doubles in this case) on the same PC if they are distributed in processes of 20k variables. It just takes ages to create so many variables (but ok no one will want to do that...:)).

Because I am writing to much and I am afraid that my point is not clear enough: My aim is to check when all shared variables a created, commited and ready.

In the attachment you can find a second version of my test VI (there are only small changes). The "GetList&write" was my last test. The idea was once all variables are created and commited I use the GetList vi to delay further execution of the VI until the shared variable engine is ready and then I open a connection to one of the variables, write a value and close it again. I should not get an error here.

Unfortunatly sometimes I get error messages. Even with only 40 NofServices I sometimes (not always, just continously run the VI) get:

Error -1950678964 at the OpenConnection.vi (Failed to load NI Scan Engine software required for I/O Variable access. )

With 50k variables in one process (NofCommits=1) I get error:

Error -2147220719 at GetVarList.vi (The client process has been disconnected from the configuration server.)

So this way is also not working reliable. Is there any other way to delay the execution until the engine is ready? I could include a 1 second waiting time. This would maybe work for 1000 variables, but nor for 10k. For this I could inluce a 1 min waiting time which would be probably not enough for 20k and so on...

Thx &

Greetings Dennis

[DesasterMaster]

Hi Dennis,

thanks a lot for your explanation. The information about why you need this much shared variables in your framework is still open, but anyways.

You could use the last created variable and triy to write something in it. You should get an error if the variable is not ready yet and process this error to wait a few seconds and try again.

Maybe this approach helps.

[mikeporter]

Nothing will help this train wreck. This application is serious off the rails and things will not end well. Someday you will have to:

A. Rethink your data structures/framework
B. Rethink your choice of using shared variables
C. Find a new job and let someone else cleanup your mess

One thing I learned many, many years ago is that LabVIEW will tell you when you are doing things incorrectly. It will start making things hard to do.

Mike...