How to write a very large waveform in the "Finite Samples" mode

[D_mitriy]

We have a process where we need to write a multiple repeated multichannel waveforms* to the digital and analog outputs (synchronized).

*I.e. the data looks like:

• Waveform pair #1 (analog and digital, both N channels, M1 samples, where M1 is small, <20k), repeated K1 times,
• Waveform pair #2 (analog and digital, both N channels, M2 samples, where M2 is small, <20k), repeated K2 times,
• ...
• Waveform pair #X (analog and digital, both N channels, Mx samples, where Mx is small, <20k), repeated Kx times.

All these smaller waveforms with all the repetitions were then combined into a pair of the large waveforms and fed into a DAQmx Write (the Digital Output and Analog Output tasks were synchronised using the Digital task's OnboardClock and the clocks were set to Finite Samples mode).

The advantage of this approach is that it is simple, we get a guaranteed number of samples at the output and there are no "hiccups" when switching between the different waveforms as they are all combined before writing. And all of this was worked fine, when the number of samples was relatively small (under ~50M per channel).

But now we need to increase the number of samples to at least 1B (one billion per channel) or even more, while maintaining a nice "hiccup-free" behaviour and a guaranteed number of output samples. Of course, the old approach does not work: the CompactDAQ require too much time to write the data before the task can be started, and the CompactRIO is simply run out of memory.

Question: what would be the best/most efficient approach in this case? How do I tell DAQmx "Please repeat waveform pair #1 K1 times, then proceed to the next without interruption, and so on and so forth until the last waveform pair #X"? Thanks!

[Kevin_Price]

Well, it probably won't be *easy*...

I don't have hardware (or time) for experimenting, so take these thoughts as educated guesses.

Here's my working understanding of typical DAQmx output tasks (using fairly "standard" MIO devices as opposed to very special-purpose ones)

A. The buffer size is fixed and equal to the # samples written to the task prior to starting the task.

B. I'm not 100% sure whether you can write twice (or more times) prior to start and let DAQmx deal with growing the task buffer and appending the new data.  And maybe this will depend on whether you're in Finite or Continuous sampling mode?

C. By default, output will regenerate, repeating the contents of the task buffer.  You can run a Finite task for a million samples even if you only write a thousand to the buffer.  The buffer will just be repeated a thousand times.

D. Once you start a Finite task, I *don't think* you're allowed to write to the task again to try to start replacing buffer contents before the generation task wraps back around to repeat.

So I agree that there's a dilemma when you want to generate a very specific uninterrupted pattern that's very long but not repetitive as a whole.  This is where we get sneaky by bringing in a counter task for the rescue!

We use a Finite counter task to generate the clock that both DO and AO will use as their sample clock.  Set it up for a billion samples or whatever you need for your sequence of repeated patterns.  (Note: the typical 32-bit counters will max out at ~2^32 finite samples, which is something like 4.3 billion.)

Now we're gonna kinda lie and tell DAQmx that the DO and AO tasks are Continuous.   This is the key trick that will let us make a reasonable-sized task buffer and then replace its contents on-the-fly a little at a time over the course of the entire generation.

There's still gonna be work to do to manage this process, but here's a rough outline of the first approach I'd take:

1. before starting the task, compute and write several seconds of data to the task to set the task buffer size

2. after the task starts, keep computing and writing smaller chunks of data to the task.  I'd recommend chunks that are no bigger than about 1/3 of the buffer size.

3. DAQmx should help manage this process.  It knows which data has been transferred down to the DAQ device, so it will wait until your smaller chunk of data will only be overwriting data that has already been transferred.  By having a pretty decent sized buffer that's at least several seconds long and replacing less than 1/2 of it at a time, you'll be helping DAQmx to have an easier time of this.

I'm out of time right now, hopefully that gets you started on the key ideas.

-Kevin P

[D_mitriy]

Hi Kevin,

Thank you for your reply and for the ideas. Approach with the counter is definitely worth trying.

One way is as you have suggested - with the online data feeding into a buffer.

Another one is a variation of your idea - to use the counter as a time source, but in the "Finite Implicit Buffered Pulse Train Generation" mode (i.e. unevenly spaced). A single-channel counter takes up much less memory, and even 2 billion samples should fit on a CompactRIO with a good margin. And since the data we write is (fortunately!) quite sparse, the writes could be squeezed into just a few kilobytes if the counter could be used as a timing source. This (if it works) allows the option of very fast writing of the new data into a buffer. The nature of the processes allows for standardized short intervals between the subsequent steps of the process (e.g., less than 5, or better, 1 ms; I should mention this option in the post...).

So thanks again, now I have a lot of things to test and try out 🙂

[Kevin_Price]

Much depends on the nature of your M1, M2, ..., Mx "waveforms" for AO and DO.  Because you used the term "waveforms", I pictured that each of those would be time-varying outputs whose particular pattern would repeat K1, K2, ..., Kx times.

Your idea of using a sparsely populated buffered CO output seems like it'd only work if your so-called "waveforms" are actually just constant setpoint values for each of the corresponding AO, DO channels in the task.  Such that each output signal remains constant throughout K1 repetitions, then new constant values throughout K2 repetitions, etc.

What are some typical waveforms, sample rates, # repetitions you need to support?

-Kevin P

[D_mitriy]

Thank you, I understand that the "waveform" is a bit confusing and it is worth explaining in more detail.

Here is an simulated example of a pair of waveforms for a single channel. All channels (usually in the range of 4 to 8 ) could be treated as independent.

The time scale is in the milliseconds and the sampling rate is 10 kHz. In most scenarios we can say that the Mx has less than 20k samples per channel (i.e. less than 2 seconds) and the Kx is around 25k, i.e. the number of DO+AO samples generated per channel is 500M or less. The number of "blocks" X is usually less than 5, the total number of samples per channel will never exceed 2B.

As you can see, the number of transitions is small (only 8 for the DO and 3 for the AO in the example, and it rarely goes far beyond that). For the 8 channels combined, it is safe to say that the total number of transitions is always less than 500 at all times.

[Kevin_Price]

I'm back to my first remark that it won't be *easy*, but now I can see that it should be *feasible*.

I'm not so sure the "sparse array" is gonna be the way to go.  That would involve a buffered counter output which only produces active edges for instants when an AO or DO channel had a transition.  Since each Mx pattern will have a potentially *different* sequence of time intervals between transitions, you can't just define a counter pulse timing pattern that will regenerate.  You would need to define the *ENTIRE* time sequence of transitions ahead of time for the counter task.   (Remember, the counter task needs to be Finite so that you can define an EXACT total # samples to generate while the AO, DO tasks must be Continuous so you can redefine their outputs on the fly.)   So 500 transitions * 25k repetitions * 5 waveform pairs * 8 bytes per pulse interval gets you to 500 MB worth of counter buffer.  That's getting to be a pretty big ask.

The alternative would probably be a non-buffered constant 10kHz finite pulse train running for up to 2 billion cycles, but then you'll also need to (eventually) feed 2 billion sample values to your AO, DO tasks.

Bear in mind that the actual physical buffer size for continuous AO, DO will be set by the # samples you write before starting up the M1 * K1 portion.  If you choose a # that's an integer multiple of M1, that'll seem convenient BUT you'll still need to be tracking progress carefully to be sure to start the transition to M2 after *exactly* K1 repetitions.  Then you'll likely find that the physical buffer size is *not* an integer multiple of M2, so you'll need to not only track progress but also keep feeding "next chunks" of M2 to the task on the fly.  And so on through Mx.

Not a trivial task.  I would very highly advise making a whole dedicated sub-project out of exploring this task buffer management operation, apart from the much bigger overall app.  There are a lot of little pitfalls and edge conditions to get worked out and you need to do that with as few variables as possible.

-Kevin P

[D_mitriy]

Thanks for sharing your thoughts! 500 MB is indeed quite a big chunk of a memory, but at least it might work (with some delay at the beginning).

I agree that this problem deserves its own small sub-project. We will see how it goes.