How Mixed Reality Will Change the Way we See the World

Autor:
Sebastian Demmerle, CEO at NMY

Mixed Reality

Put on futuristic (albeit still relatively bulky) glasses and suddenly you are transported to a stadium, you can steer a spaceship, or even dive into the sea. You can use your smartphone to go on the hunt for Pokémons literally everywhere. Snapchat filters allow you to wear funny goggles or cat ears. You can also check out how well the new sofa fits the rest of your interior — all thanks to mixed reality technologies.

Mixed reality will make a lot of things easier. For patients as well as physicians

Not for Entertainment (Only)

The fields of application for mixed reality technologies, such as augmented and virtual reality, are endless. They can be used for entertainment purposes, but also for more serious, business, and even scientific use. More and more, we find out about how augmented and virtual reality apps help change the way we navigate our lives, and how businesses optimize workflows and processes by employing new technologies.

Mixed Reality for Visually Impaired People

Mixed reality technologies have an important feature in common: they attract the visual system. But what about the people, who no longer have a fully functioning visual system? Can visually impaired people participate in mixed reality technologies? Or, can they even benefit from them? Let’s focus on a target audience, that does not spring to mind as soon as state-of-the-art technologies are implied: The over-60s. Even though 60 is the new 40, older generations are not at the forefront of VR and AR users. This could potentially change very soon.

Age-Related Macular Degeneration (AMD)

AMD can lead to partial or even complete loss of vision in one eye or both. The causes are destructive metabolic products that accumulate in the sensory cells of the macula lutea as is common due to the aging process of humans. As the macula is responsible for color, central, and high-resolution vision, any kind of damage is obviously not good news. Early stages of AMD occur in every fifth 70-year-old and in 30–40% of 85-year-old people. To support therapy and relieve symptoms of AMD, a conglomerate of scientific and technological partners collaborates on the so-called IDeA project: Integrated Diagnosis- and E-assistance System for Patients with Age-related Macular Degeneration.

Science to the Rescue. Mixed Reality for AMD Patients

Since December 2018, NMY has been involved in the IDeA project that features partners such as the universities of Munich and Tübingen, as well as optics company ZEISS. The main objective of the government-funded research project is to develop solutions that improve quality of life for people suffering from AMD. By developing a system that combines several crucial functions specifically tailored to AMD patients, the IDeA partners have come together for a duration of 3 years. IDeA revolves around three main strategies for diagnosis, assistance, and training.

What remote diagnostics could look like in the (near) future

Diagnose. Assist. Train.

Clinical diagnoses can be supported by inexpensive and portable tests in virtual reality or the analysis of eye movements in everyday life. Furthermore, our system enables learning and improvement of visual coping strategies for patients. Last but not least, those affected in difficult situations will be offered an easily recognizable visual extension of reality to make information more easily accessible.

To summarize: The IDeA-project aims to…

improve the quality of life of AMD patients
develop an assistance system for AMD patients
facilitate diagnostic processes
facilitate training processes

Diagnosis with Mixed Reality: Early Detection Is Key

Early diagnosis is crucial. The development of AMD takes years and since it often affects one eye, the healthy eye compensates for the loss of vision for a long time, before the patient becomes aware of it. In the meantime, the disease has progressed. Regular check-ups at the ophthalmologist are essential for early diagnosis and therapy because they prevent the disease from developing further and can even lead alleviate and overall improve symptoms.

How Can We Improve Diagnostics for Patients?

In the context of IDeA, we have identified two approaches. First, we will take advantage of the increasing popularity and availability of mixed reality technologies, to integrate diagnostic tests into everyday life. Plus, we want to create slimmed-down versions of classic clinical tests to be used at home. This helps to prevent unnecessary, time-consuming (or, due to AMD symptoms, often difficult) trips to the doctor’s office. We will investigate how eye movements in real and virtual spaces can be used to draw conclusions about the existence, size, and location of visual field defects. For this purpose, the user wears a mixed reality headset with integrated eye tracker and environment camera that allow continuous reading and evaluation of eye movements. The aim is to detect pathological deviations from normal gaze behavior associated with AMD in “normal” activities like reading, cooking, or shopping as early and automatically as possible, in order to detect the extend of defects. This comes with a degree of accuracy that allows the adaptation of AR-based assistance functions.

Support and Substitute Clinical Diagnosis

Standardized procedures for the examination of visual function in AMD will be made available for telemedical and everyday use. For this purpose, computer-aided, AR/VR-based variants of the procedures will be developed and validated. The focus will be on procedures that are both technically feasible and promise the greatest knowledge gain during a telemedical examination. Furthermore, it will be examined to what extent the duration of individual test sessions can be shortened by integrating test items into everyday activities without reducing the validity of the testing procedure.

To summarize: Diagnosis with Mixed Reality…

enables earlier diagnosis which can lead to a milder course in AMD patients
renders frequent visits at the doctor’s unnecessary
will use technology that is easily available
increases the frequency of measurements taken of the eyes
improves the database doctors use to make decisions

An Assistance System that Actually Helps

For the development of user-centered assistance functions, we have identified augmented reality (AR) as eligible technology to support AMD patients with everyday activities. The assistance system that will be implemented in AR glasses, will compensate declining vision as well as additionally projecting information into the intact visual field. At later stages, the assistance software will also enable context awareness, i.e. the system will selectively and automatically apply functions at the exact moment they are needed by the patient.

The headset recognizes the clock and projects the time in digital numbers unto the view field

Moreover, the system will respond to individual visual needs of users. To achieve this goal, functional AR demonstrators for AR glasses are needed that use object recognition. There are endless applications for AR-based assistance in daily situations: Purchasing tickets from a machine, recognizing familiar faces at events, reading clocks, etc.

First prototypes have already been presented at conferences

At the end of the day, let’s not forget we are talking future developments here. AR hardware is not up to speed as of today. We are developing the strategies and software matching hardware that does, in fact, not exist yet. However, we are confident that it is not a matter of “how” but “when”. AR hardware will be a common device for people of all age groups.

To summarize: AR Assistance…

helps patients in their everyday life
depends on the development of more practical hardware
needs to adapt to individual patient requirements

XR Training — An Eye Trainer of/for the Future

A process we are hoping to support with our training strategy is fixation behavior, because AMD patients tend to alter it, leading to further deterioration of vision. In order to create a valid training program, we conduct experiments to evaluate “free” eye movement behavior with respect to its “fixation quality”. The next step will be to test various training methods for efficiency and level of invasiveness.

Sneak peek into VR prototype for training simulation (complete with virtual assistant)

At a later stage, we will develop a training concept for everyday use. Initially, visual field defects will be simulated in VR. Training methods include the additional manipulation of visual content which optimizes the development of an eccentric fixation site (e.g. augmentation of the defect by binocular presentation, presentation of defect edge areas, or systematic masking of image content). The training program will later be applied and validated on AMD patients in real environments.

To summarize: VR training…

can be a real asset in the therapy of AMD patients
will be based on real needs and real situations
has to be validated based on virtual data and real people
can be done remotely (e.g. at home)

Developing Software for the Future of Mixed Reality Hardware

As mentioned above, the IDeA project is focused on developing concepts and strategies in anticipation of hardware that is currently not available on the market. However, since mixed reality hardware development has been taken on by all big software and hardware players, it is good to know that the software will be ready to go the moment the right MR hardware is launched. AMD patients will benefit from diagnostics, training, and assistance systems in a way that is unheard of today — but will be standard in the (near) future.