3D Vision
LabVIEW User Events for Acquisition of Clouds of Points from the Kinect
Overview
One of the feature of the Haro3D library, available through the LabVIEW Tool Network, is the ability to acquire colored clouds of points from the Microsoft Kinect V2. Those 3D clouds of points represent a large quantity of data and hence require significantly more processing than regular images. For that reason, one request has been to improve the acquisition rates of those clouds of points.
This document presents how improvements in acquisition rates of cloud of points from the Kinect when using the Haro3D library can be achieved by using parallel processing and LabVIEW User Events.
Video
A video associated with this document can be seen below.
Parallel Processing and threads
To achieve an improvement in acquisition rates, processing the data in parallel with the data acquisition appeared as the best approach.
When attempting to work in parallel, the acquisition rate barely improved, generating the following Microsoft errors:
0x8000000A (-2147483638) : E_Pending, The data necessary to complete this operation is not yet available.
or
0x80004005 (-2147467259) : E_FAIL, Unspecified error
(see Microsoft MSD reference).
I rapidly realized that the cloud data processing and the data acquisition from the Kinect never executed in parallel, even when those tasks executed in two separate loops.
This problem was solved by configuring the acquisition DLL (through a Call Library Function Node) to “Run in any thread” instead of the usual “Run in UI thread” (see figure below). My understanding is that when a common memory space needs to be accessed by several calls to the same DLL, “Run in UI thread” should be used, while “Run in any thread” could be used for independent DLL calls (there would be no warranty that two calls would be in the same thread). However, using the "Run in any thread" option does not appear to be a problem, so far.
Figure 1 Configuration dialog of a Call Library Function Node. The thread options are highlighted.
LabVIEW User Events
The parallel approach proposed here uses LabVIEW user events and two loops, each one containing an event structure (see figure below).
Figure 2 Block diagram of the Cloud User Event Example VI.
A first user event is created using a Haro3D cloud data cluster. That event is registered with the event structure of the data processing loop (top loop). The reference to that event is passed to the Haro3D Cloud API that uses a new command, “Acq User Event”. The event reference is passed to the DLL that uses the reference to send the data directly to the data processing loop. This process could be implemented using the Cloud API with the old command “Acquire” and the “Generate User Event” function with no significant loss of performance but the current approach with the new command is simpler to implement.
A second user event is created using a Boolean named “Acq”. That event is registered with the event structure of the data acquisition loop (bottom loop). Its reference is passed to the data processing loop.
The cloud data is acquired in one event case configured for both “Timeout” and “<Acq> User Event” events . The acquisition starts when after am initial timeout value of 500 ms. The data is acquired by the DLL that sends it to the processing loop in the “<Cloud> User Event” case. The data processing loop receives the data and start processing the data for display. In parallel in the same case, a <Acq> event is generated to initiate a new data acquisition while data processing is still going on. In this manner, a second <Acq> event cannot be generated until the data processing has completed. This approach prevents the data acquisition to generate data faster than the data processing can handle them. If for some reason (acquisition error for example), the <Acq> User Event is not fired, the acquisition will restart after the timeout time.
The feedback mechanism to prevent data overflow could be implemented with a queue instead of a user event. The benefit of using a user event is that the event structure can be configure to stop the data acquisition loop by creating a case for the stop button. A queue would require to implement a mechanism to stop the data acquisition loop.
Requirements
Hardware
Microsoft Kinect V2.
Computer meeting requirements for the Microsoft Kinect V2 (http://msdn.microsoft.com/en-us/library/dn782036.aspx).
Software
Software requirements for Microsoft Kinect V2 (http://msdn.microsoft.com/en-us/library/dn782036.aspx).
Haro3D library (downloaded and installed using VIPM).
Installation
The new cloud examples for the Haro3D library version 1.3 are installed in the example folder of the LabVIEW folder of your computer. Examples can be accessed using the
Example Finder, as shown in the video.
Results
Using a parallel approach and “Run in any thread” for the DLL calls bring the cloud acquisition rates to approximately 30 Hz for the 480x270 resolution and between 12 and 15 Hz for the 960x540 resolution, concurrently with data processing and display. Those figures compare to 23 to 26 Hz and 6 to 8 Hz for the 480x270 and 960x540 resolutions, respectively, for the Cloud Example VI acquisition of Haro3D version 1.3, when running on a I7 2.7 GHz laptop.
Summary
Parallel processing, as shown in an example VI, can improve acquisition rates of cloud data from the Kinect when using the Haro3D library. Such improvements required to configure the DLL calls for the cloud acquisition to to run in any thread instead of running in the user interface thread.
The cloud acquisition using LabVIEW user events will be included as an additional example of the Haro3D library version 1.3.
The use of “run in any thread” for the DLL might lead to unexpected problems. If you encounter some issues, please let me know in the comment section or directly by email.
Marc Dubois
