multiple producer loops with one consumer loop

[emi1010]

Hello, 

I have a problem with my VI. I'm using LabView 21.0.1f2 (32-bit) with the NI cDaq-9174 with the PT100 Temperature Input Module NI-9216. With the NI-9216 I have connected PT100 temperature sensors.

For my VI I want two or more Producer loops and one Consumer loop. The Producer loops should acquire the data and store it in the queue. The Consumer loop should take the data out of the loop and write it into a file.

All of the above is working in my VI, the problem occurs if the two Producer loops are timed differently. In the final VI there should be different sensors (not only temperature), that’s the reason why I need different timed loops. The VI only writes the data with the timing of the Producer loop with the higher wait time, which is 2000ms in this example.

It would also be nice if the data from different sensors are sorted (see picture below).

I think I know why this problem occurs I just can’t find a way to solve it.

Thank you in advance!

[GerdW]

Hi emi,

---

@emi1010 wrote:

The VI only writes the data with the timing of the Producer loop with the higher wait time, which is 2000ms in this example.

---

Without being able to look at your VI right now I guess you use queues and more than one dequeue function inside your consumer loop!?

When you don't provide a timeout value for Dequeue then it will wait for new elements in the queue - and so the slower producer will dictate when your consumer loop is able to run the next iteration!

I guess all producers should write to the same queue, so the consumer only has to read one queue. The queue element should contain the information which producer sent the message, like cluster of [sender, timestamp, value, additional data]…

[Bob_Schor · Accepted solution]

OK, I'm not completely sure why your code behaves as it does, but it is oddly backwards.  Here are my suggestions:

• The Producer loops are examples of "timed" actions.  You've chosen to do this as a standard While Loop with a "Wait (ms)" function to do the timing.  A not-unreasonable alternate configuration (particularly when running in a cDAQ, with which I'm not very familiar, so this might be "wrong") is to use a Timed Loop (assuming the cDAQ has a reasonable timer).
• Conversely, the Consumer loop should not be timed, as it is "gated" (or "timed") by the receipt of data from its Producers -- if no data arrives, the Consumer simply waits (as determined by its TimeOut, often set to -1 by default).
• In a Producer/Consumer configuration, the Producer should never destroy the transmission mechanism (I use Stream Channels, you use Queues, different mechanisms, same rule).  The Producer(s) "know" when they are done, and stop themselves.  When they exit, they need to send a signal to the Consumer saying "We've exited, so you should, too, and then we'll all be done and you can clean up, if needed, by Releasing the Queue".  Note that in your two-Producer model, you'll need to merge the Error Outs (which does the "And" for you) and, as a "signal" (which I've seen called a "Sentinel"), you could enqueue an Empty Array to the Consumer.  Now, you need to modify the Consumer to recognize this "special signal" and get it to do two actions:  if set (empty Array), do not save the data, but do exit the Consumer, and on doing so, Release the Queue.
• I see you use a Stream Channel to connect the two Producers.  You can (and probably should) use Stream Channels to connect your Producers to separate Consumers, otherwise you will have data from two different temperature sensors "intermixed" at different rates, which complicates the data file and makes parsing the results (it would seem to me) more difficult.  Note, however, that the Stream Channel Wire has "built-in" the "Sentinel" behavior that you might want to safely bring your two Producer/Consumer designs to a halt.
• I would put the Stop button in an Event Structure, designed so that the Stop "Value Change" Event stops the Event Loop and generate a Tag Channel Wire that branches (Tags can Branch) and goes to both Producer Loops.  The output of the Tag reader goes to the two Stop Controls, stopping the two Producers.  If you follow my suggestion of having two Consumers, you would use the two Producer-to-Consumer Stream Channels to stop the two Consumers, so everything stops "naturally".
• Note I'm just writing a "How to" piece, without actually trying to generate the LabVIEW code I'm describing.  I leave that to you.  There may be a "bug" in what I've described, but you should be able to follow the basic ideas and "find the bug in the code" if it is present. 
• If if you really do want to merge the data streams from the Producers to a single Consumer (in which case  a Stream Channel wouldn't work for you), you can use an alternative Channel Wire, the Messenger Channel.  A Messenger Channel doesn't have the "Sentinel" inputs that the Stream has, but you can merge Messenger Channels (as you have merged the Queue references).  But how do you handle having two Producers telling the single Consumer to exit?  You use another Tag Channel that sends "Exit" after both Producers have exited.  You need to make a change to the Consumer and put a TimeOut (maybe 1 second) in the Messenger Reader, which you wire to a Case Statement which does the Write to File only if there was no TimeOut.  This allows the Tag Channel Read (wired to the Stop Control) to run and stop the Consumer (but only if both Producers have exited and have set a "True" into the Tag).

Bob (Can you tell I'm a Nut about Channel Wires?) Schor

[Kevin_Price]

I'm at LV 2020 and can't open your code.  Here are some general thoughts about having multiple data producers delivering to a single consumer at different rates.

1. First make a decision about your file format.  A format like TDMS is very good at handling different "channels" at different rates.  But you can't open such files in simple apps like Notepad.  There *is* however a free plugin for Excel that will painlessly import a tdms file to Excel, with separate worksheets as needed for the different-rate channels.

2. If you're sure you want to stick with simple ASCII text files (such as CSV format), there's another approach I've been known to use.  It has generally been built on top of a QMH-style framework where the queue datatype is a cluster of string message and variant data.

    Each of the producers is contained in its own parallel loop, and the producers get into a free running state where they repeatedly "push" their data and timing info into a single shared queue.  The string message identifies what kind of data it is, the variant holds the data and timing info.  Only the parallel consumer loop ever performs dequeues.

    Meanwhile the consumer maintains a (typedef'ed) cluster of state variables, including fields for the data from each of the producers.  Each time the consumer dequeues, the string message identifies who the data is from (and by implication, what to convert the variant data into), and that data goes into the correct field of the big state variable cluster.  (Sometimes I may accumulate data into an array, sometimes I simply replace prior stale values -- it depends on the needs of the app.)

    I'll have decided ahead of time what's going to trigger me to write 1 CSV line to a file.  It's usually every time I get an update from one of the producers.   Which one?  Well again, that's very app dependent.   When I write based on a faster producer, I'll end up with many lines in the file where the slower producers' data are repeated because I'm always simply writing the most recent known value.  When I write based on a slower producer, I'll typically accumulate data from the faster producers in an array so when it comes time to write I'll have options.  I can do averaging, filtering, most recent value, etc.  In some cases I might do 2 or more of those things in separate "columns" of the CSV.

There are pros and cons and a lot depends on the kind of workflow you need to support when analyzing the data and creating reports.  I think TDMS is a better inherent fit for multi-rate data, but I've more often written to CSV simply to support internal customers' preferred workflow.

-Kevin P

[Yamaeda]

I guess you don't want the empty rows on the slower signal?

It makes perfect sense currently, as the consumer only have 1 sample every second time. 

The easy solution is to write to a TDMS file instead, then you can extract and defragment and stuff afterwards.

[t.n14]

Hi,

I'm not sure about that, i never used tunnels, but it seems your tunnel blocks the execution of the second (faster) producer loop. Use a different way to stop your loops.

If you want your data sorted like in the picture, you will have 40 'sensor 2' values per 'sensor 1' value, just fyi.

You need to your consumer which 'sensor' the data is coming from, i.e. add a constant to the build array with '1' for the first sensor (edit: I'd make this a Cluster), '2' for the secound and add 2 columns to the data of the second sensor before writing to the file.

good luck

Timo

[emi1010]

Thanks to you all for your suggestions. 

I will try some of these solutions next week and will update you!

[emi1010]

Hello everyone, 

after some trys I managed to fix the error thanks to your help. I used an tag instead of the stream channel wire and it worked. 

But I still have some problems with sorting the data as shown in the picture I attached above. I tried some things, but in both solutions the data of the different sensors are not aligned through the time. I want the data which is aquired within the same timestamp to be written in the same row of the array. 

Both solutions i have come up with are attached below. 

At this point I'm also good with using TDMS-file instead of csv, but I couldn't figure that out as well. 

Thank you!

Emi

[t.n14]

That's a hard one, as you generate the timestamp with fraction of seconds (like '14:04:41.947') they are most likely not the same.

What comes into my mind, is to save the first result in a shift register, with the next result, compare if the time matches (without the fractions, like in your picture), if not, write the first one to the file and put the latest result in the shift register. If they match, concatenate the strings and write them into the file.

With that you have a delay of writing the file, but same time will be in the same row. As you write the time of every sensor in it's own column, you don't lose data with that comparison.

Timo

[Bob_Schor]

Sorry, I haven't been following this discussion so closely, but I wonder if it wouldn't be easier to save the "raw" data files that come from different sources at different timings in different files (particularly if at least some of the data are "regular in time" so that they can be saved in a compact form such as a Waveform).  Once the data files are all written, they can be examined and "merged" if this is necessary.

There could well be interaction between the "regular" and the "irregular" data channel, but that might have nothing to do with the format of the data files, rather with how the two channels interact.  It might be that you can describe a condition where "when Channel 2 shows this, then we need to be on the lookout for Channel 1 to do that".  This almost sounds like a parallel task handling these data, maybe via a separate Producer/Consumer design.  You, of course, are in the best position to determine "What" you want to do, or "What" you have to do.  Try to get that clearly delineated before getting lost in the Weeds of "how" you do that -- therein lies Spaghetti Code

(and with its color-coded Wires, LabVIEW can make awesome Spaghetti).

Bob Schor