CO-MR MAGIC
When done right, community mixed reality (Co-MR) is magic. When done wrong it is literally nauseating. Those are the stakes for Co-MR and why we put such a premium on designing and creating the highest quality experiences.
Our perception of physical reality is constructed from our sensory inputs. MR’s core proposition is that if you can digitally override those sensory inputs, (primarily visual and auditory using a HMD) then you can trick the user’s lizard brain into believing that they have been transported to a virtual reality. Some vocabulary is helpful for understanding how this magic happens, and for how it can fail when poorly executed.
The Realities
Physical Reality (PR): We’ll hand this one off to the physicists out there.
Virtual Reality (VR): Technology that creates a computer-generated stereoscopic 3D world with spatial audio. Using devices such as a six degrees of freedom (6DoF) head mounted display (HMD) and motion captured (mocap) controllers or hand tracking, users can look around, move, and interact with objects in the virtual environment.
Social Virtual Reality (SVR): People embodied with avatars interacting inside VR.
Augmented Reality (AR): Created with simi-transparent glasses that present digital elements such as 3D objects and animations, virtual screens, images, and text. They allow users to see and interact with digital content while still viewing their PR surroundings.
Mixed Reality (MR): The idealized form of MR is VR that is indistinguishable from PR. The reality is we get as close to that ideal as technology and costs will allow. Examples of how that is achieved include using distributed AI supercomputer node powered community hyper mixed reality (Node Co-HMR) for photorealistic graphics, HiFi digital audio, and full body mocap. It can also include VR HMDs with color passthrough video that can create an enhanced form of AR.
Community Mixed Reality (Co-MR): A MR platform where people come together for events that center on shared passions. It is best experienced with a distributed AI supercomputer node powered hyper mixed reality system, but it is fully compatible with personal computer powered mixed reality (PC MR), console powered MR (CMR), mobile powered VR (MVR), AR, personal computers, mobile phones/tablets, and smart glasses.
Community Hyper Mixed Reality (Co-HMR): A blending of mixed reality with matching physical reality props like stairs or walls, and multi-sensory feedback like fans or heaters. Users wear MR gear with optional body haptic feedback, and move through the space where what they feel in PR matches what they see in MR.
Extended Reality (XR): Also known as spatial computing, is an industry description that can include any or all of the above.
The Illusions
Place Illusion (PI): The subjective experience of perceiving a MR environment, and the objects it contains, as real. Place illusion arises when the MR system accurately updates the scene in response to the user's head and body movements (sensorimotor contingencies), causing their lizard brain to accept a MR world as the place they actually are at. PI is strongly influenced by visual and audio fidelity.
Embodiment Illusion (EI): The feeling that a MR 3D mocaped avatar, is one's own body. The EI mocap can be just the head (from the HMD) and hands (from controllers), or detailed full body tracking. EI occurs when an avatar moves in synchrony with the user's real movements, producing a sense of body ownership and self-location. Mocap not matching up with an avatar’s movement is a common reason for EI to break.
Co-Presence Illusion (CI): A sense of sharing the MR space with other people. Co-presence occurs when avatars are perceived as genuinely occupying the same environment and responding naturally through body language and spatial audio.
Plausibility Illusion (PiI): The union of the first three illusions being achieved and working in harmony. Other secondary factors like the quality of interactivity and in-world physics also influence it. A failure of one or more of the foundation illusions will break plausibility. Another way PiI can break is from VR simulation sickness. That occurs with 3DoF HMDs if the user's head moves in x, y, or z, or worse if HMD visual movement is forced on the user and it doesn't match their proprioception. An example of that kind of simulation sickness happens with VR rollercoaster experiences at arcades.
The Tech
There are three foundational elements to any mixed reality system. The computer that runs it, a head mounted display, and a way to interact. Other technologies that can enhance MR are full body motion capture systems and/or hyper reality elements.
MR Computer: The brains of the system. It processes graphics, sound, movement, and user input in real time, then sends this information to the MR headset. Generally speaking the more powerful the computer the higher the plausibility of the experience.
Head Mounted Display (HMD): A xDoF headset that has stereoscopic visuals, and spatial audio speakers. It is used to display 3D environments and avatars, while also tracking the user’s head movements. That data both updates what the user sees and hears, as well as expressing their avatar’s head movement. HMD’s can be tethered where a data/video/audio cable is connected to a computer, or wireless (some of which are tethered to an external battery).
Interaction options include: A pair of 6DoF handheld controllers that track their position and rotation, allowing users to navigate their environment, express body language, interact with menus, and grab MR objects; Hand tracking that uses sensors to mocap the hands and fingers for body language and interactions; A single 3DoF controller that tracks rotation and pointing direction but not x,y,z movement; The Google Cardboard button was an input control used to interact with VR content while a phone was inside the headset. Users would use the 3DoF HMD to aim, and then pressing the button allowed them to select objects they were looking at; Traditional game controllers use buttons and joysticks for movement and actions.
DEFINITIONS
Community Mixed Reality (Co-MR): A MR platform where people come together for events that center on shared passions. It is best experienced with a distributed AI supercomputer node powered hyper mixed reality system, but it is fully compatible with personal computer powered MR, console powered MR, mobile powered VR, AR, personal computers, mobile phones/tablets, and smart glasses.
Head Mounted Display (HMD): A xDoF headset that has stereoscopic visuals, and spatial audio speakers. It is used to display 3D environments and avatars, while also tracking the user’s head movements. That data both updates what the user sees and hears, as well as expressing their avatar’s head movement. HMD’s can be tethered where a data/video/audio cable is connected to a computer, or wireless (some of which are tethered to an external battery).
Lizard brain: The popular shorthand for the most primitive, instinctive parts of the human brain. In the context of Mixed Reality, it means the automatic, subconscious systems that process basic sensory input and generate a felt sense of reality, presence, and embodiment, without needing higher-level rational analysis. The idea is that if visual and auditory sensory signals are overridden convincingly enough, these primal circuits can be fooled into treating the virtual environment as physically real, producing the subjective experience of having been transported there.
Stereoscopic 3D: The technique that creates the illusion of depth and three dimensional space by presenting two slightly different images, one to each eye, that mimic the way human binocular vision works.
Spatial audio: A technology that reproduces sound so listeners can perceive its direction, distance, and movement in three-dimensional space, mimicking how we naturally hear the real world.
Motion capture (mocap): The process of recording a person’s real world movements, typically of the body, face, and/or hands, and converting that data into digital animation that can drive a 3D character or avatar.
Avatar: A computer generated 3D representation of a person (or character) that appears and moves within a XR environment.
Distributed AI Supercomputer Node Powered Community Hyper Mixed Reality (Node Co-HMR): The highest plausibility form of community mixed reality, rendered and processed by a high performance compute node. Each one is equipped with enterprise-grade, liquid-cooled NVIDIA GPUs and can deliver photorealistic graphics, high-fidelity spatial audio, and full-body motion capture at the highest levels of speed and visual fidelity. Node Co-HMR aims to approach the ideal of mixed reality that feels nearly indistinguishable from physical reality. It supports shared experiences (vercasts and social gatherings) with strong place illusion, embodiment, co-presence, and overall plausibility.
Personal Computer powered Mixed Reality (PC MR): High plausibility mixed reality experiences that are rendered by a personal computer. In this setup, a PC handles the intensive computing tasks such as generating high quality 3D graphics, spatial audio, and real time interactions, while sending the output to a mixed reality headset (via cable or wireless connection).
Console powered Mixed Reality (CMR): Medium plausibility mixed reality experiences that are rendered by a video game console like the PlayStation 5 or Valve's Steam Machine. In this setup, the console handles the computing tasks including generating 3D graphics, spatial audio, and real-time interactions, while outputting to a mixed reality HMD (via cable or wireless connection).
Mobile-powered VR (MVR): Lower plausibility virtual reality rendered by built-in mobile hardware.
Sensorimotor contingencies: The expected changes in what you see and experience in response to your movements and actions.
VR Simulation Sickness: A form of motion sickness triggered by a sensory conflict in virtual reality. The user's eyes perceive motion while their body’s proprioception detect a mismatched lack of movement.
Proprioception: The body’s ability to sense the position, movement, and orientation of its limbs and other body parts in space. Specialized sensory receptors (proprioceptors) located in muscles, tendons, joints, and the inner ear continuously send information to the brain about: joint angles, muscle length and tension, force being applied, and overall body posture and movement.
Degrees of Freedom (xDoF): This describes how many different ways a HMD or controller can move and be tracked in XR. 0DoF (Zero Degrees of Freedom): No movement or rotation is tracked. The view remains fixed. 3DoF (Three Degrees of Freedom): Tracks head/hand rotation for looking/pointing left/right, up/down, and tilting side to side. 6DoF (Six Degrees of Freedom): Tracks both rotation and position, allowing you to look/point around as well as move forward/backward, left/right, and up/down. Therefore, 0DoF doesn't track anything; 3DoF tracks your head or controller’s pitch, yaw, and roll (PYR) rotation; and 6DoF tracks your PYR rotation, and x,y,z movement.