To understand the answer you must first understand the question… Really we should first take a look at what might be considered the base technologies in the “realities” world. Before the the term, Mixed Reality, came up in what some assume might have been concocted in a marketing meeting we primarily talked about two other technologies.

First we had Virtual Reality (VR). This is a method of taking a user and completely immersing them in a virtual world. This often includes a headset to block out the real world. The blocking out of the real world has this effect of convincing the user’s brain that the world they’re seeing is real. Although convincing the brain like this can have side effects. An example would be simulator sickness that many people experience.

Another technology that has been commonly talked about is Augmented Reality (AR). This is where things get a little muddled unfortunately. For the purpose of this article I will define it as simply the visible world you see with a layer of digital information. This digital information can be anything as simple as some numbers or as complicated as a 3D dancing character.

I can’t believe how much fun I’m having!!! – Chris Grelik

Mixed Reality is the two technologies combined. With mixed reality you have a virtual world and the real world which is weaved together using an array of sensors. This way if you walk around in the real world your movements could be 1 to 1 in the virtual world. Virtual objects then will seem to be pinned to physical locations. This differs from Augmented Reality where the system has no mapping to the physical world; objects in that case would just seem to be placed in relation to the screen space.

Mixed Reality Sensors

Inertial Sensors

These sensors are more commonly described as gyroscopic sensors and accelerometers. The basic premise behind this is that the accelerometer allows the system to track physical linear motion. This way when you walk past a virtual object, the system can then calculate out how far you have traveled in the real world and apply that same translation to the physical world. Gyroscopic sensors are used to track the rotation of the device as the user moves around. These two sensors have to be calibrated well and limit error at a large range of temperatures; this is one of the reasons that Google has limited the devices to Galaxy S8 and Pixel devices for its ARCore framework.

Optics Sensors

Optical sensors are basically cameras. But not just any cameras; these must be very high-fidelity cameras. They should be able to capture fine details in the world and evaluate their features. For example ARKit (Apple) and ARCore use fine details in flat surfaces to better hone surface movement. An optical sensor can also be used to correct for ‘drift’ in the inertial sensors; meaning as you move around with the device the inertial sensors may tend to lose exact track of placement the optical sensor can then be used to reevaluate the surface details and correct the motion tracking.

A subset of optical sensors are the ‘Time of Flight’ sensors. These sensors combine a infrared projector and a camera. A short burst of infrared light is emitted and the sensor then calculates the amount of time it takes for the light to bounce back to the device. The result is a point cloud which can be used to create a 3D map of the world.

Getting Started Today

Currently I’m only familiar with the ARCore/ARKit development. For these two SDKs the companies have made it very simple to create an MR application by working with game engine developers to integrate the SDKs into game engine systems.

There are two major game engines that work with both of these frameworks: Unreal Engine and Unity. Currently Unreal 4.18 Preview allows for developers to develop for ARCore and ARKit without having to rebuild the game engine. With Unity, on the other hand, you can use a plugin to develop for both platforms.

Mixed Reality in the Future

Over the years many different technologies have come and gone. Google Glass promised to be an immersive technology for the general consumer. Oculus announced the Rift and it seemed like that was the future and the world would never go back. Over time these technologies have faded from public light or found different niches than what was originally intended. Time will tell what the fate of Mixed Reality will be; currently it seems that Mixed Reality has a promising future with consumer facing products appearing more frequently.