High-speed data acquisition - Low frame rate

[abhi_vt]

Hello,

I have a question regarding acquiring and recording data at very high speeds in LabVIEW 2013 environment. We are trying to build an imaging system that acquires data from two modalities. We have an Alazar digitizer ATS9350 operating at its full capacity (500MS/sec) - digitizing an analog signal coming from a photodetector. Additionally, we also have a  cystoscope camera (ethernet connected), capable of acquiring at 60 fps. Both of these systems get triggered by a software generated external trigger at 15 Hz. We are trying to drive these two to acquire and record data simultaneously. 

We developed a VI that allows us to generate this trigger signal, and acquire data at each trigger. Within this VI, three main blocks:

1. Trigger generator 

2. block driving ATS9350 for data acquisition

3. Block operating the cystoscope camera. 

We have two separate queues (for 2nd and 3rd block) for writing (saving) acquired data to an SSD. Since we want the digitizer and the camera to operate and acquire in synchrony with each other, we added a Rendezvous block so that both modalities will collect the same number of frames. 

Based on such design, this system should be able to acquire two data sets (one from the cystoscope camera and the other one from photodetector) at 15 frames each per second. However, we have not been able to achieve the frame rate of 15 frames per second. Following table describes the frame rate under different conditions:

	Saving Disabled	Saving Enabled
Only photodetector	13	11
Only cystoscope	15.12	13
Both Simultaneously	8	6

Based on these numbers, it appears that saving the data (writing to a hard drive) may be an issue, which could be slowing the imaging down.  With the digitizer operating at its full capacity, we acquire 1.2Gbits/sec. Therefore, we recently replaced the traditional HDD with an SSD to achieve higher write-speed. However, 

Following are the current system specifications:

Intel Core 2 Duo, E8400 3GHz , Windows 7 (x64), 4GB RAM, Samsung SSD (SATA III).

I was wondering if there are any VI design modifications we can do to address this low frame rate issue, or is the issue causing due to hardware limitations such as low RAM, processor etc. 

Please let me know if you need any additional information.

Thanks,

Abhi

[billko]

How can we hope to suggest modifications to your VI when we haven't even seen it yet?

[abhi_vt]

I am sorry for not posting the code earlier. However, I am not sure if this single file will help. 

[billko]

Since I asked for the code, I feel it is my responibility to do the commenting on it.  I don't know how to put this gently, but I reviewed the code and the best advice that I can give is that you start over with someone with more LabVIEW experience there to guide you.  Before we can discuss your original problem, we need to resolve some basic issues.

[johnsold]

Abhi,

As Bill said, your VI cannot be fixed.  There are too many fundamental problems to make it worth trying to modify.

Sequence structures defeat LabVIEW's inherent parallelism. Stacked sequence structures obscure code and force right to left wiring for sequence locals. They are almost never required in good LV code, and when they are necessary it is almost always a single frame to enforce dataflow.

Local and global variables should almost never be used to pass changing data among parts of the program. They also often make extra copies of the data. Use wires, queues, or user events.

Value property nodes are the slowest way to move data, run in the UI thread (= slows things even more), and are rarely the best way to get access to the data.

Nested loops have their places but in your code are likely complicating things more than they simplify them.

Do not dequeue from a queue in more than one place.

Solutions:

Look at the Producer/Consumer Design Patterns. They show the best practices methods for operating loops in parallel.

When optimizing for speed and memory usage consider these factors:

Do nothing else in the acquisition loops except the actual acquisition and enqueuing the data. All analysis and saving should be done in parallel loops with sufficient buffering (usually the queues) that the speed of those loops does not limit overall performance.

Do not do any formatting or calculations on the data in the acquisition loop.

Do not display images on the front panel. This likely makes an extra copy and is certainly slow.

Do not configure or change parameters on the image devices while in the acquisition state.

Read the White Paper on the NI web site on Managing Large Data Sets. It was written several years ago and some parts are now out of date but the concepts are still valid.

Consider a binary file format such as TDMS. The files will be smaller and you can write much faster than with text files. (On looking again it appears that you may be writing binary files.)

Lynn

[abhi_vt]

Hi Bill and Lynn,

Thank you very much for the response. However, I am a bit confused about some of the concerns you raised. I admit that my code is not memory- and execution-efficient - possibly because I learned LabVIEW coding merely by looking at example codes. Following is my response to your comments.  I would appreciate it if you could help me one more time. 

"Sequence structures defeat LabVIEW's inherent parallelism. Stacked sequence structures obscure code and force right to left wiring for sequence locals. They are almost never required in good LV code, and when they are necessary it is almost always a single frame to enforce dataflow."

I needed sequence structures for initializing the digitizer. Also, some of the sequence structures were provided in the SDK of the digitizer card. What alternative do I use to Sequence Structures?

"Local and global variables should almost never be used to pass changing data among parts of the program. They also often make extra copies of the data. Use wires, queues, or user events."

 I thought local variables make the code tidy. However, it makes sense not to use them if the program makes extra copies of data. 

"Value property nodes are the slowest way to move data, run in the UI thread (= slows things even more), and are rarely the best way to get access to the data."

I remember reading somewhere on this forum, that Value Property node is more efficient way than Local variables. 

"Nested loops have their places but in your code are likely complicating things more than they simplify them."

What alternatives should I follow?

"Do not dequeue from a queue in more than one place."

I have two queues in the code, which get dequeued at two locations. The third dequeue segment is disabled. I meant to delete it earlier but never got around to doing that. Do structures under "Diagram Disable" get compiled when executing the code?

Solutions:

Look at the Producer/Consumer Design Patterns. They show the best practices methods for operating loops in parallel.

I tried to follow the P/C design. The second block in the design (which contains a lot of nested loops) is the producer where the data is being acquired and the third block all the way at the bottom is where the data is being dequeued and written to the disk. 

When optimizing for speed and memory usage consider these factors:

Do nothing else in the acquisition loops except the actual acquisition and enqueuing the data. All analysis and saving should be done in parallel loops with sufficient buffering (usually the queues) that the speed of those loops does not limit overall performance.

Do not do any formatting or calculations on the data in the acquisition loop.

I can revise the code to implement these suggestions.

Do not display images on the front panel. This likely makes an extra copy and is certainly slow.

Although there are multiple display panels, there is only one (or two) that work. All others are disabled.

Do not configure or change parameters on the image devices while in the acquisition state.

We never actively make any changes to the configurations of the imaging devices. However I realized that the code would actually permit that activity, which implies that some CPU resources may be polling for these parameters. I will change the code to avoid this.

I look forward to your response. Thank you very much for your help, in advance.

[billko]

---

@abhi_vt wrote:

Hi Bill and Lynn,

Thank you very much for the response. However, I am a bit confused about some of the concerns you raised. I admit that my code is not memory- and execution-efficient - possibly because I learned LabVIEW coding merely by looking at example codes. Following is my response to your comments.  I would appreciate it if you could help me one more time. 

"Sequence structures defeat LabVIEW's inherent parallelism. Stacked sequence structures obscure code and force right to left wiring for sequence locals. They are almost never required in good LV code, and when they are necessary it is almost always a single frame to enforce dataflow."

I needed sequence structures for initializing the digitizer. Also, some of the sequence structures were provided in the SDK of the digitizer card. What alternative do I use to Sequence Structures?

Sequence structures have limited use.  They should only be used when no other method of enforcing dataflow are available.  Stacked sequence structures have NO use, IMHO.  They have all the disadvantages of sequence structures in general, plus they just obfuscate code because you have to figure out where things enter and exit and what frames they belong to.  A state machine design pattern works better because, even though the individual frames in the case structure are hidden - as in the stacked sequence structure - wires are entering and exiting in an easily understood way.

"Local and global variables should almost never be used to pass changing data among parts of the program. They also often make extra copies of the data. Use wires, queues, or user events."

 I thought local variables make the code tidy. However, it makes sense not to use them if the program makes extra copies of data. 

 Even though LabVIEW is a graphically-oriented programming language, never sacrifice dataflow for "neatness."  Besides memory issues, global and local variable do NOT follow dataflow principles and are therefore notoriosly difficult to harness correctly.

"Value property nodes are the slowest way to move data, run in the UI thread (= slows things even more), and are rarely the best way to get access to the data."

I remember reading somewhere on this forum, that Value Property node is more efficient way than Local variables. 

I read that somewhere, also.  It's wrong.  I think if you read further down in that topic, you would have seen that the OP had that mistaken belief based on the fact that they misread the post he was quoting.

"Nested loops have their places but in your code are likely complicating things more than they simplify them."

What alternatives should I follow?

This is a tough one.  All I can say is that the more experience I gained from using LabVIEW, the less I used nested loops.  I can't exacty say why, though.  It just seemed that I used them less often.

"Do not dequeue from a queue in more than one place."

I have two queues in the code, which get dequeued at two locations. The third dequeue segment is disabled. I meant to delete it earlier but never got around to doing that. Do structures under "Diagram Disable" get compiled when executing the code?

Once you dequeue something, it is gone from the queue forever.  It's on a first-come first-serve basis, so multiple dequeues will always be racing each other to see who gets the data and who is left out - i.e., race condition.  It's likely you are missing pieces of data and you don't even know it!

[edit] I think I msunderstood this part.  If you meant "two different queues with each being dequeued only once" then my advice - while still true - doesn't apply.  [/edit]

Solutions:

Look at the Producer/Consumer Design Patterns. They show the best practices methods for operating loops in parallel.

I tried to follow the P/C design. The second block in the design (which contains a lot of nested loops) is the producer where the data is being acquired and the third block all the way at the bottom is where the data is being dequeued and written to the disk. 

When optimizing for speed and memory usage consider these factors:

Do nothing else in the acquisition loops except the actual acquisition and enqueuing the data. All analysis and saving should be done in parallel loops with sufficient buffering (usually the queues) that the speed of those loops does not limit overall performance.

Do not do any formatting or calculations on the data in the acquisition loop.

I can revise the code to implement these suggestions.

This is perhaps the BIGGEST thig you can do to optimize your code.  However, this is an optimization.  This may or may not be useful in your present code and may even be detrimental or damaging with the code in its current state.

Do not display images on the front panel. This likely makes an extra copy and is certainly slow.

Although there are multiple display panels, there is only one (or two) that work. All others are disabled.

Do not configure or change parameters on the image devices while in the acquisition state.

We never actively make any changes to the configurations of the imaging devices. However I realized that the code would actually permit that activity, which implies that some CPU resources may be polling for these parameters. I will change the code to avoid this.

I look forward to your response. Thank you very much for your help, in advance.

---

I have comments in the sections I felt qualified to answer.  🙂

[johnsold]

Abhi,

BIll made some excellent comments.

Sequence structures: Just because the manufacturer of the digitizer card provided an SDK with sequence structures it does not mean that it is good code. As you spend more time on the Forums you will find that many drivers provided by device manufacturers are poorly written - often by someone who knows very little about LV. They are often written just so the vendor can claim that his device supports LV.

Sequence structures are often a clue that the programmer has a background in text-based languages and has not understood the LV dataflow paradigm. In a text-based language the control of the program goes from the current line to the next line unless a branching instruction is executed. In a dataflow language any node which has current data availalble at its inputs may execute at any time and no node produces data at its output teminals until all code within the node has completed execution. This inherent parallelism is one of the great advantages of LV. The sequence structure defeats that advantage.

Moving data: Wires are always the most efficient way to move data. Local variables are really not variables in the sense that they are used in text-based programming languages. It may be better to think of them as remote access and duplicate terminals to a control or indicator. The only time that they should be used is for setting controls to an initial state as read from a configuration file when starting the program or very similar situations. Value property nodes are slower as has been mentioned before and should only be used in similar situations to locals but in a subVI where a local cannot be used. If wires cannot be used (parallel loops), queues or User Events are the appropriate methods to move data.

Nested loops - alternatives: In programs like yours a state machine is usually a good alternative. In place of the inner loop go back to the same state repeatedly in the outer loop. One execution of the state case for each frame read from your camera is appropriate. This allows the state machine to respond to user commands, conditions in the data, or errors promptly without requiring a complicated process within an inner loop. Why? Because these are handled in other states.

Dequeue: I think your code is OK. The disabled part does not affect performance. I did not attempt to determine whether the disabled code was an alternative to the enabled part or had been disabled because of problems with two dequeues.

Producer/Consumer: Yes, you have something like a producer/consumer pattern. Because of your requirement for increased speed you must streamline the Producer. Either the Producer or the Consumer or both may contain state machines or other architectures. While in acquistion mode the Producer should be in a condition (state!) which does absolutely nothing except acquire from the camera or sensor and enqueue the data. If no errors and no Stop command, then repeat the acquisition state.  Because you have two data sources which generate large amounts of data, they should probably be in separate Producers. You may also need an Event based Producer to handle User Inputs. Make sure that any image displays which you decide to use have the indicator terminals in the Consumer, not in the Producer. That way if there are any delays it only affects the display and not the acquisition.

Lynn

[abhi_vt]

Bill and Lynn,

Thank you very much for taking time to respond in detail. Your comments and pointers will certainly help me in revising and optimizing the code.

I truly appreciate your efforts.

Best regards,

Abhi