Waiting on any of multiple queues to have data

[InfiniteNothing]

Is there a slick way to have a loop that waits on any of multiple queues to have data before executing.

I can only think of convoluted methods like firing a notifier when I enqueue to any of the queues and have the loop dequeue the right one or having one loop per dequeue and using a number of local variables to share data between them

[dan07]

Hello

If I understood your question, you can have two while loops (producer and consumer), and have a Event Structure inside producer loop. When an event fires, one event case will enqueue something  to queue and the consumer loop will run just after that.

Find attached a picture from a code that I wrote in the past using this kind of structure. Hope it helps.

Dan07

[Mark_Yedinak]

Do you want your code to execute when a single queue receives data or after both have received data? If both than what you wrote will work just fine. If you want to execute whenever either queue receives data the only way you can really accomplish that would be to use timeouts. You would have to allow the queues to timeout. You would have to check if both queues timed out and not execute any code. I'm not quite sure why you would want to create this situation though. I see that you have different data types however you can accomplish that using a cluster as your queue data. The cluster would contain an ID to identify the type of data and the other element would be a variant. That way you could pass any type of data and the ID would be used to determine the type of data.

[D Robertson]

Try using a Rendezvous

[InfiniteNothing]

I'm looking to execute when either gets data. The rendezvous is the opposite of what I want I think.

The problem with timeouts is the queues are fed at different rates. While one queue is waiting, the other will stack up messages and I'd have to write a "catch up" mechanism for it. It seems inelegant. Using variants seems inefficient (because of packing and unpacking) and a little inelegant. Maybe just cluster all the types together (in my example the array and the integer) along with a flag of which part of the cluster has valid data. It still seems inelegant. There's also a bit of a problem if the producers are enqueuing lossily. Then the faster data source gets priority over which producer gets more samples in the pipe.

[D Robertson]

I understand your problem better now, and I agree the rendezvous is the wrong mechanism.

I do like the idea with a notifier, that could be really slick.  I would recommend using a queue as that would allow multiple queues to add elements simultaneously and still get handled (see code idea below).

Dave

[johnsold]

InfiniteNothing,

I think your login ID may have the clue you need: Zero times Infinity may be undefined, depending on how you generate your Zero and Infinity.

Perhaps you are asking the wrong question.  Can you tell us more about what you are trying to do?  What are the sources of the data being enqueued?  What are the rates? What are you doing with the data after it is dequeued? What is the consequence of missing a data point or getting data in both queues simultaneously?

I suspect that a change in your basic program architecture may be better that trying to solve the exact question you asked.

Lynn

[Ben]

---

@InfiniteNothing wrote:

I'm looking to execute when either gets data. The rendezvous is the opposite of what I want I think.

The problem with timeouts is the queues are fed at different rates. While one queue is waiting, the other will stack up messages and I'd have to write a "catch up" mechanism for it. It seems inelegant. Using variants seems inefficient (because of packing and unpacking) and a little inelegant. Maybe just cluster all the types together (in my example the array and the integer) along with a flag of which part of the cluster has valid data. It still seems inelegant. There's also a bit of a problem if the producers are enqueuing lossily. Then the faster data source gets priority over which producer gets more samples in the pipe.

---

I do that all of the time to handle data stream at different rates. I read the number of elements in the queue and use that to drive a For loop to process the backlog.

Ben

[InfiniteNothing]

---

@D Robertson wrote:

I understand your problem better now, and I agree the rendezvous is the wrong mechanism.

I do like the idea with a notifier, that could be really slick.  I would recommend using a queue as that would allow multiple queues to add elements simultaneously and still get handled (see code idea below).

Dave

---

 I like it. I'll take a look at what the enqueuing would look like. Maybe I need a queue of queues, and then after the first dequeue, I can just dequeue from what ever pops out. 

---

@johnsold wrote:

InfiniteNothing,

I think your login ID may have the clue you need: Zero times Infinity may be undefined, depending on how you generate your Zero and Infinity.

Perhaps you are asking the wrong question.  Can you tell us more about what you are trying to do?  What are the sources of the data being enqueued?  What are the rates? What are you doing with the data after it is dequeued? What is the consequence of missing a data point or getting data in both queues simultaneously?

I suspect that a change in your basic program architecture may be better that trying to solve the exact question you asked.

Lynn

---

It's just an architecture puzzle that I run into often. I'd rather not specify a data rate because I want the architecture to be flexible if the customer changes their data rates. This sort of problem is never impossible to solve but there's many solutions and I was trying to gauge what people did and what the best practice is. In one of the problems I'm thinking of, after the data is dequeued, it is processed, thresholded, and firing an alarm if the data is out of range. Missing a data point is OK as long as we know we missed it. Getting data in both queues at the same time is to be expected. I hope to just process both new data points right after one another.

---

@Ben wrote:

I do that all of the time to handle data stream at different rates. I read the number of elements in the queue and use that to drive a For loop to process the backlog.

Ben

---

Would you dequeue, then check the status of the queue? Otherwise you're back to a polling method which I'd prefer to avoid.

[Ben]

---

@InfiniteNothing wrote:

...

@Ben wrote:

I do that all of the time to handle data stream at different rates. I read the number of elements in the queue and use that to drive a For loop to process the backlog.

Ben

---

Would you dequeue, then check the status of the queue? Otherwise you're back to a polling method which I'd prefer to avoid.

---

I'll jump the gun and ask about your application.

An engineer I worked with once stated tht "a queue is always almost empty or almost full." It has been years since I had to deal with the "almost full" sie of the game and the reason is PC are fast enough to keep up when the code is implemented properly. So my queues get emptied almost as fast as they get filled.

A glaring exceptin is "command queues" that only get processed one at a time to force sequential operation. These are usually "polled" when the rest of the control loop is done with what it has to do and need something to entertain itself.

So if you have a process that occationally gets bogged down with numbr crunching that creates a back log in the form of a growing queue, I first want to know about the code that is creating the back log.

Ben