Main menu


New and Improved - Microsoft's Augmented Reality Headset

Earlier in May last year, Microsoft announced new features for its HoloLens 2 augmented reality headset. Prior to the upgrade, the headset had issues with the headset’s sensors getting confused in mobile environments.

The problem at hand was that augmented reality headset sensors were competing with each other due to camera tracking behavior and sensor tracking behavior. When used in a car, the headset sensed motion, but saw the static environment around it and caused “confusion”.

The new extension was developed in collaboration with Volkswagen and is designed to not only present the driver with virtual objects inside and outside the car, but also display a virtual map on the car’s dashboard. HoloLens 2 can also display navigation arrows at major intersections to warn drivers of pedestrians crossing in front of their vehicle.

We asked Marc Pollefeys, Director of the Microsoft Mixed Reality and AI Lab Zurich, to learn more about this new technology.

Just Auto (EN): How did the partnership with Volkswagen come about?

Mark Polfieth (MP): In 2015, Volkswagen launched a research project using self-driving cars and augmented reality to teach driving on a racing circuit.

Volkswagen wanted to use HoloLens for this project, but found that when the device was mounted in a moving vehicle, the sensors lost track and the holograms that would normally appear disappeared. .

The Volkswagen researchers turned to Microsoft for help and connected me to Microsoft’s Director of Science and an expert in 3D computer vision and machine learning.

The Volkswagen team visited Microsoft’s headquarters in Redmond, Washington to see the team’s work with HoloLens and their vision for augmented reality technology. The two teams began working together around 2018 to develop HoloLens 2’s movable platform capabilities.

How does augmented reality work when used in vehicles?

To address the problem of sensors losing track, my team developed algorithms that model discrepancies between sensors and allow HoloLens to continue tracking.

HoloLens 2’s new “moving platform” mode overcomes a major limitation of mixed reality headsets and opens up possibilities for using this technology in new ways. That is, training drivers to handle difficult road conditions.

Can you describe the virtual map technology you see on your car’s dashboard and what it does?

This is a demo application that explores what future navigation systems that leverage mobile platform capabilities might look like. Note that it does not necessarily represent plans for future technology, nor is it currently in widespread use.

What are the main benefits of this technology for car occupants?

In the future, augmented reality tools such as moving platform modes will enable seamless and connected mobility experiences, such as receiving information, navigation aids, and entertainment as people travel on a daily basis.

Do you have any plans for future technology development or partnerships with other companies?

The technology is already attracting interest from shipping companies and organizations that are using HoloLens to remotely connect maritime workers with mechanical experts via Microsoft Dynamics 365 Remote Assist.

Microsoft plans to further refine the moving platform mode for use in elevators, trains, cars, and other moving environments. This technology can be used to help truck and bus drivers navigate narrow streets, identify points of interest along driving routes, and provide entertainment to passengers. There are multiple ways to connect to the outside world.