Example of Robot Application with the Hololens

[MarcDub]

Overview

This document presents an example of application using the Haro3D library version 2 that includes an API for the Microsoft Hololens.

In a previous document, it was shown how to use the OBJ file format and the Haro3D library to build and operate a virtual 3D robot from CAD files. The present document, and its associated code, is using the same approach to load the virtual robots in both LabVIEW and the Microsoft Hololens space. The virtual robot is then operated from the Hololens (or augmented reality) space.

The goal of the current document is twofold. First, the code is provided for anyone interested to use it to upload and control any type of robot, as long as the CAD files are converted and organized as explained in the document referenced above. Second, the provided code illustrates the general concepts of programming behind the Haro3D library that can be used for any other type of application.

VIDEO

A video showing the use of the VI described by the current document along with the Hololens operations is shown below.

Class structure

The Haro3D library uses classes to create the 3D objects that will be uploaded either in a LabVIEW 3D picture control or in a Microsoft Hololens (or in any other future 3D display interface). The class structure of the Haro3D library along with the classes created for the current project are shown in the UML diagram below. The UML diagram was created from within LabVIEW using the NI GOOP Development Suite (the Haro3D library box and the legend were subsequently added).

 

UML diagram of the classes of the Haro3D library and of classes extended from it for the current project.

In the Haro3D library, all objects to be uploaded in a 3D space are derived from the abstract class 3D_Object. Two classes inherit from the 3D_Object class, the abstract class 3D_Shape and the OBJ_Mesh class. Generic 3D objects inherit from the abstract 3D_Shape class: Cylinder, Panel, Box, Sphere, and Mesh. The OBJ_Mesh class contains the attributes and methods to handle meshes described in a file of the Wavefront .obj format (Wikipedia link).

While inheriting from the 3D_Object class, the OBJ_Mesh class is actually an aggregation of objects of the Mesh class. The 3D virtual robots in the current project are described by .obj files. It was therefore decided that the robots would need to be described by OBJ_Mesh objects, or objects that inherited from that class. An actual robot can be defined as an assembly of axes, each of which is an assembly of different objects (robot parts, cabling, base, etc.). Each of those objects can be defined in the virtual world as one or more meshes, defined within an .obj file. A virtual robot for the current project is therefore an object of the class Holo_Robot that is a composite aggregation (or containment) of axis objects of the Holo_Robot_Axis class, that is itself a composite aggregation of objects of the Holo_Object class.

Within the VI, each Holo_Robot_Axis object is a collection of Holo_Object objects that can be handled in the 3D picture control and Hololens spaces using the methods defined for the OBJ_Mesh and the 3D_Object classes methods. The spatial relationships between the different Holo_Robot_Axis defining the robot axis movements, positions, and orientations, must be defined within the Holo_Robot and Holo_Robot_Axis classes.

VI Description

The front panel and block diagram of the Hololens Robot Demo VI are shown below.

Front Panel of the Hololens Robot Demo VI. The 3D picture control, indicators, and controls are not essential but are provided for convenience. Robot in 3D picture control, position, angles, and axis values are updated as the axes are moved from the Hololens.

Block diagram of the Hololens Robot Demo VI. The architecture is a single-loop event-based Queued Message Handler.

Requirements

1. The attached VI requires Haro3D library version 2. The Haro3D library can be downloaded from VIPM (vipm://harotek_lib_haro3d).
2. The Hololens Robot Demo VI and supporting files (attached to the current document).
3. LabVIEW 2015 or later.
4. Microsoft Hololens (or emulator).
5. The Haro3D app version 2.0.3 or later must be installed on the Hololens and configured with the correct LabVIEW computer IP address.
6. The Hololens and the computer running the LabVIEW VI must be on the same network and the firewalls and other security settings must be configured for mutual network access. The Microsoft Hololens requires a 5-GHz Wifi connection.

Installation

Just unzip the attached file in a given directory.

Operation of the VI

1. Start the VI and start the Haro3D Hololens app. The Haro3D app on the Hololens must be configured with the IP Address corresponding to the computer running the LabVIEW VI. 
2. Once the VI is started, the application is controlled by the Hololens user.
3. A dialog appears when the LabVIEW application is connected, which is typically fractions of seconds after the splash screen.
4. Make the Hold gesture, this brings a multi-choice dialog. Select the desired robot.
5. Once the robot is loaded, you can select an axis, which becomes highlighted in green. Move the axis by using Manipulation gestures.
6. After robot has been loaded, you can make the Hold gesture, which brings a multi-choice dialog to select Reset axes, Display axes values, Display Position Info, Scale, Reset scale.
7. Quit at any time by using the Bloom gesture. This gesture also terminates the LabVIEW application.

Summary

This document presents an example of code uploading different robots into a Microsoft Hololens. The robot can be moved, scaled, and its axes independently moved by the Hololens user.

In a first time, the code presented here can be used as-is to upload any other type of robot as long as the CAD files are being converted and configured as described in a previous document.

In a second time, the code presented illustrates how to use the Haro3D library and how to extend the classes for any other type of application.

Please post question, feedback, and other comments by replying to the current post.