I was looking at a window filled with puffy clouds on a clear blue sky when I noticed a little boy walking up to it and started drawing. With his tiny fingers, he diligently covered the windows with his art and I asked him what he was drawing. “A train.” Why was he drawing a steam train? “Imagine it traveling across and above those clouds!” I stared at him with disbelief, realizing that ever since I met another child in the subway station trying to control the digital signage, I hadn’t had that sense of genuine amazement: These kids were already experiencing the world with extended reality (XR).

What is XR? XR is an abbreviation for extended reality, which covers any immersive technologies that merge the physical and virtual worlds, such as virtual reality (VR), mixed reality (MR), and augmented reality (AR).

The term ‘XR’ was first used by the Khronos Group consortium in March 2017 with the development of OpenXR - an open, royalty-free standard that provides high-performance access to virtual reality (VR) and augmented reality (AR) platforms and devices. Industry terminologies are generally categorized based on technology implementation and academic purposes, but XR is the overarching concept. There is comfort knowing that we could simply label everything as “XR” without having to name/rename every rapid technological change involved. 

Difference between VR, AR and MR

But there are probably some people wondering “Do Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) really fall under XR?” Long story short, the answer is “Yes”. All three technologies depend on computer graphics and display technology. The key difference is how they operate. How?

If you feel like you stepped into another reality, similar to getting into the game from reality, that’s VR: Virtual Reality exists to present users with simulated environments that are different from the real world.

 On the other hand, Augmented Reality adds a layer on top of the reality we exist in. Think of the Pokemon Go craze a few years ago or photo editing apps that lets you filter your skin or add bunny ears on your selfies. Unfortunately, AR currently is unable to ‘interact’ with reality real-time so it has a limited use to identifying objects and adding graphics on top. Should you cover your face, the virtual bunny ears would disappear; instead of petting your favorite Pokemon, your hands end up in the back hovering.

Last but not least, Mixed Reality is a hybrid of reality and virtual reality. In March 2021,  Niantic (the creators of Pokemon Go) released a demo using HoloLens 2 where Pikachu was able to hide behind trees or physically follow the app user realtime. It feels like an uncanny replication of magical creatures are in real life that only used to exist behind game consoles/computer screens. Imagine you pointing at a desk and shouting “Fire” - with Mixed Reality, we could make it happen virtually. Minecraft Earth, which shut down on June 30, 2021, was an augmented reality sandbox game that used geographic information to let users build Minecraft-style architecture in real life. 

Living in the Era of XR

Foundations of XR technology depend on computer graphics (CG) and display technology, so technological growth in all three areas is encouraged. From the computer game we obsess over to indoor car/golf simulators, our lives are already filled with virtual reality technology. Welcome to the era of Extended Reality.

Remember Second Life, the first Metaverse service back in 2003, allowing people to create an avatar for themselves and have a second life in a virtual world? Looking back, it had a lot of similarities to The Sims. Another example to consider is the world’s first virtual reality mapping system: The Aspen Movie Map was created in 1978, giving users an interactive virtual tour of the real-world city of Aspen, Colorado. It used the first person perspective commonly used in games, which can be comparable to Google Street View.

One of the earliest virtual reality simulators, the Sensorama Simulator (1956) has a clear resemblance to the modern-day 4DX* movie theaters. Using technology such as three-dimensional imaging, stereophonic sound, and aromas, the Sensorama Simulator provided immersive, multi-sensory experiences.  

*4D movie technology that allows films to be augmented with various practical effects (ex: motion seats, simulated wind/snow, scents)

The AR Translator allows travellers to easily take snapshots of restaurant menus and have them translated within seconds. Fortnite, Roblox, ZEPETO are all famous examples of Metaverse that rely on Extended Reality technology on flat-displays. 

Aside from gaming, Extended Reality is also utilized in various industries. There are case studies of educational content using AR/VR simulations and tools integrating AI, location, and translation functions utilizing improved hardware performance. For instance, the US Army has used a virtual reality dome for military training for many years. The army recently ordered 120,000 HoloLens augmented reality headsets for training and for combat. A helmet-mounted display (HMD) training has been superseding simulators for pilot training due to its affordability.

AGCO Corporation, an American agricultural machinery manufacturer, implemented the use of AR glasses and saw massive outcomes in efficiency such as reduction of work hours by 30% and training hours by 50%. In the medical field, XR technologies are extensively used to train and assist with surgical procedures. Building automobiles, virtual travels, virtual orthodontics consultations, virtual house tours and virtual fitting rooms are just some of the many active applications. Filming at digital studios, based on VR technology with projection mapping, has become more common. Think of Disney+ Original ‘The Mandalorian’.

XR, Matrix and Metaverse

What can we expect from XR technology moving forward? Despite the number of commercialized XR technologies, it’s true that many people hold on to the ideal of immersive Virtual Reality. In fact, the ideal virtual reality that Jaron Lanier, founding father of virtual reality who coined the term, introduced us was a future based around immersive experiences. Lanier actually released the first models of helmet-mounted displays (HMD). Contributing editor at WIRED, Peter Rubin noted that Jaron Lanier’s works had a tremendous impact on cultural content since earlier versions of VR headsets were not easily accessible.  

This explains the advent of Snow Crash (1992), a science fiction novel that introduced the word ‘Metaverse’ for the first time, V.R. Troopers (American TV Series 1994–1996), and The Lawnmower Man (Movie 1992). Arcade video games swept away people with immersive VR experience and even Nintendo released the Virtual Boy, a portable video game console, in 1995. Iconic films such as Mamoru Oshii’s ‘Ghost in the Shell (1995)’ and ‘The Matrix (1999)’ became massive hits. Tim Sweeney, founder and CEO of Epic Games, said “Be patient. The Metaverse will come” because in his opinion, Metaverse is defined by immersive VR capabilities. He believes that a VR headset with 8K resolution will open doors to the ‘real’ Metaverse one day. 

Despite Sweeney’s wishes, development of XR technology has been rather different. The next generation of VR headset developers envision a future that seamlessly blends reality and virtual worlds. They hope to replace PC monitors with XR glasses/headsets and keep cell phones in back pockets. They dream of a device that uses voice recognition and gesture recognition technology - easing users into the contactless communication and information lifestyle without PCs.   

If Metaverse represents a set of spaces online in a social/cultural setting, Extend Reality is the underpinning of that virtual reality. One thing is clear: Our lives will never be the same once Extended Reality replaces existing PCs and transforms the way we work, play, and interact on a daily basis.