3D Vision
Sharing Holograms between Multiple Hololens using LabVIEW
Overview
The Microsoft Hololens (www.hololens.com) is an augmented reality headset that offers to its users to see 3D objects, or holograms, in real environments. The users can walk around the holograms and interact with the holograms.
It is therefore advantageous for multiple Hololens users to be able to see the same holograms simultaneously and at the same physical location in the real environments. The present document shows an example of a new feature available with version 2.1 of the Haro3D library: sharing between multiple hololens users of holograms that are located at the same physical location.
This new feature is demonstrated using in a VI that is a modification of a robot demo that has been shown in a previous document.
VIDEO
A video illustrating the sharing of holograms can be seen using the link below.
Sharing of holograms: Spaces
Each Hololens has its own coordinate system. By default, the position and orientation of the headset at the start of the Hololens app defines the origin and orientation of that coordinate system. However, it is possible to create an anchor. An anchor is a point in space not defined by the coordinate system but by 3D features of the environment as mapped by the Hololens. This anchor can then be used to define a new coordinate system for each Hololens.
The approach consists in defining an anchor in one Hololens and to send this anchor to another Hololens. If the anchor can be found by this other Hololens from the features of the mapped surfaces, the anchor becomes a common point between the two Hololens. Defining that point as the origin of the coordinate system makes those Hololens share a common coordinate system.
One issue is that an anchor shared by one Hololens must be found by the other Hololens. The anchor is defined using the 3D data mapped by the operating system of the Hololens, called spaces by Microsoft. To successfully share an anchor, both Hololens must currently be in the same room, and the mapped spaces of that room must be similar.
In general, simply sharing an anchor between Hololens that are in the same room does not work. Typically, Hololens that are in the same room map slightly different areas of that room. Relatively small differences in the mapped spaces will result in failure of sharing an anchor. Fortunately, Microsoft provides a tool to see a space as currently mapped by a Hololens. This tool can be accessed using the device web portal of a Hololens (https://developer.microsoft.com/en-us/windows/mixed-reality/using_the_windows_device_portal). The currently mapped space can be seen in the 3D View page by clicking the button Update under the Surface Mapping heading.
The figure below shows the mapped surfaces of the same room mapped by two different Hololens. The small differences shown between the mapped surfaces shown in that figure prevent the successful sharing of an anchor.
A successful approach consists in systematically mapping a room by walking in the room along the same path with each Hololens. First, it is preferable to erase the current space to make sure that no residual mapped surfaces would be associated with the current room. The spaces can be erased in the Settings -> System -> Spaces page of each Hololens. Notice that each space is associated with a wi-fi connection.
The next figure shows two mapped surfaces of the same room by two different Hololens. In that case, the mapped surfaces of the room appear to cover the same zones and the sharing of an anchor is successful.
For large rooms, it might be easier to map only the section of the room where the Hololens will be confined, as long as the mapped surfaces appear similar in the 3D View page of the device web portal of each Hololens. Notice that the spaces are continuously updated while the Hololens is in the operating system, sometimes resulting in a new area of the room being mapped by one Hololens but not the other.
Connection mechanism
After the LabVIEW application is started on a separate computer a first Hololens connects to the LabVIEW application. The first Hololens to connect is the primary Hololens.
The next Hololens to connect is a secondary Hololens. It is important to make sure the Hololens that is intended to be used as the primary is fully connected before attempting to connect secondary Hololens. All subsequent Hololens that connect to the LabVIEW application are also secondary Hololens. Any secondary Hololens can disconnect without impacting the other Hololens. If the primary Hololens terminates, connections to all secondary Hololens also terminate.
Notice that all holograms uploads and hologram modifications can be observed by the current secondary Hololens. However, any actions carried out by the primary Hololens prior to the connection of a secondary Hololens will not be observed by that secondary Hololens.
VI Description
The VI presented here is based on one VI that was successfully modified in two previous documents https://forums.ni.com/t5/3D-Vision/Example-of-Robot-Application-with-the-Hololens/gpm-p/3638192 and https://forums.ni.com/t5/3D-Vision/OBJ-File-Format-for-3D-Robot-Simulations/gpm-p/3535101. The architecture of the VI is a single-loop Queued Message Handler.
The front panel of the new VI is shown below.
The main difference in the front panel is the presence of two indicators: Primary and Secondaries. The Primary indicator is a string giving the IP address or network name of the primary Hololens connected to the current VI. The Secondaries indicator is an array of string giving the IP addresses or network names of the secondary Hololens connected to the current VI.
A new kind of event was added to the different events coming from the Hololens: Sharing. The Sharing events are a family of events generated by secondary Hololens. Each Sharing event is accompanied by data identifying which secondary Hololens generated the event, the kind of event, and data associated with that kind of event. The kinds of events possible for a Sharing event are the same than those available from the primary Hololens, except for one exception: Secondary connect. This kind of event occurs when a secondary Hololens connect to the LabVIEW application. This event is the equivalent of a Connection event from a Primary Hololens.
The figure below shows the code of the current VI for the Secondary connect event from the Secondary Hololens.
Once a secondary Hololens is connected, most actions sent to the primary Hololens will also transparently be sent to all secondary Hololens connected at the time of the action. A few VIs were added that apply specifically to secondary Hololens only: Move Secondary Origin.vi, Update Secondary Origins.vi, Display Info All Secondaries.vi, and Voice Ouput All Secondaries.vi. The figure below shows the code associated with the Receive Anchor event that uses Update Secondary Origins.vi to send the anchor generated by the primary Hololens to be used as the origin point of the coordinate system of all secondary Hololens.
Notice that in the case of the failure of a secondary Hololens to apply the anchor, no feedback is provided to the LabVIEW application.
Requirements
The attached VI requires Haro3D library version 2.1 The Haro3D library can be downloaded using VIPM (vipm://harotek_lib_haro3d).
The Holo_Demo_Robot_Sharing_LV2015.vi and supporting files (attached to the current document).
LabVIEW 2015 or later.
The Haro3D app version 2.1 must be installed on the Hololens (or Hololens emulator) and configured with the correct LabVIEW computer IP address.
The Hololens and the computer running the LabVIEW VI must be on the same network and the firewalls and other security settings must be configured for mutual network access.
Installation
Just unzip the attached file in a given directory.
Operation of the VI
If a common coordinate system between all Hololens is desired, make sure that the current Spaces as observed from the 3D View of the web portal of the Hololens, look similar on all Hololens.
Start the VI and start the Haro3D Hololens app on the primary Hololens. The Haro3D app on the primary Hololens must be configured with the IP Address corresponding to the computer running the LabVIEW VI. The Hololens and the computer must be on the same network, with the firewalls and other network securities appropriately configured for mutual access.
Once the VI is started, the application is controlled by the primary Hololens user.
A dialog appears when the LabVIEW application is connected.
Once the primary Hololens is connected, secondary Hololens users can connect as well. The secondary Hololens must be configured like the primary Hololens. Make sure that all secondary Hololens are connected before further proceeding.
On the primary Hololens, make the Hold gesture. This gesture brings a multi-choice dialog. Select the desired robot.
Once the robot is loaded, still on the primary Hololens, map at least one surface of the current room by using the voice command “Start Mapping”. Once the desired surface is mapped, stop the mapping by the voice command “Stop Mapping”.
Make the Hold gesture and select “Move Origin”. Move the cursor on the desired mapped surface (cursor should turn orange) and air tap. The robot should instantaneously move to the location of the cursor. After a few seconds, the robot should move to the same location in the room for the secondary Hololens that successfully found the anchor within their own mapped spaces.
At any time after loading the robot, the Hold gesture on the primary Hololens brings a multi-choice dialog to select Reset axes, Display axes values, Display Position Info, Scale, Reset scale, 3D Markers, Move Origin.
Notice that if the move of the origin is not successful (or if the origin had not been moved), the secondary Hololens users can still observe the operations applied to the robot by the primary Hololens. In that case, the robot is located at the position of the headset where the Haro3D app was launched for that particular secondary Hololens.
Any secondary Hololens can quit at any time by using the Bloom gesture without affecting the other Hololens. The Bloom gesture by the primary Hololens closes the connections for all secondary Hololens and terminates the LabVIEW application.
Summary
This document presents an example of code using the Haro3D library version 2.1 where a robot hologram uploaded into a Microsoft Hololens is shared among several Hololens. The robot can be moved, scaled, and its axes independently moved by one Hololens user while observed at the same physical location by several other Hololens user.
