Bringing Health Care’s Vision of Tomorrow Into Focus

Ultrasound data displayed directly on a patient via augmented reality headsets. Immersive “grand rounds” for medical students and faculty even when they’re in different locations. Virtual reality landscapes matched with classical opera to transport people with painful injuries outside of themselves, reducing the need for potentially addictive opioids.

These medical examples of extended reality (XR)—the umbrella term used for technology based in virtual and augmented reality or other immersive media—are already being prototyped or tested in clinical trials. But its widespread use in hospitals and other health care settings is currently hampered by technical challenges and sparse regulatory guidelines.

Now, with $5 million from the National Science Foundation (NSF) and technology titans including Google, Microsoft and Meta (formerly known as Facebook), the University of Maryland is leading a multi-institutional effort that will focus on developing, testing and certifying XR technologies in medicine and health care.

The new Center for Medical Innovations in Extended Reality, known as MIXR, joins UMD computer scientists and engineers with physicians and clinicians at the University of Maryland School of Medicine in Baltimore and the University of Michigan to improve medical training, patient management and health care outcomes across all areas of clinical practice.

Behrooz Shirazi, acting deputy division director of the NSF’s Division of Computer and Network Systems, called MIXR one of the first national centers at the intersection of medical and computing sciences, and said, “We expect this vibrant collaboration to produce significant societal and health care impacts.”

In addition to Google, Microsoft and Meta, other technology companies involved in MIXR are Sony, Magic Leap, Health2047, GigXR, Brainlab and apoQlar.

Another key partner in the MIXR initiative will be regulatory experts at the U.S. Food and Drug Administration, ensuring that safe, effective and innovative clinical solutions make it to patients as soon as possible.

“We’ll work closely with our industry and government partners to answer any scientific questions regarding regulatory evaluations and decisions needed for the wide-scale clinical use of these devices,” said Amitabh Varshney, professor and dean of the College of Computer, Mathematical, and Natural Sciences at the University of Maryland.

Varshney is the lead-site principal investigator on the project, and is joined by partner-site PIs Sarah Murthi, M.D., an associate professor of surgery at the University of Maryland School of Medicine, and Mark Cohen, M.D., a professor and vice chair of surgery at the University of Michigan Medical School. All three have extensive experience developing and using immersive technologies in a medical setting.

Varshney and Murthi co-direct the Maryland Blended Reality Center (MBRC), launched in 2017 as part of MPowering the State, the strategic partnership between the University of Maryland, College Park and the University of Maryland, Baltimore.

Early projects out of MBRC focused on prototyping new diagnostic tools to assist physicians at the renowned R Adams Cowley Shock Trauma Center in Baltimore, where Murthi is a critical care doctor and director of the critical care ultrasound program. This includes innovative AR medical displays that could improve how bedside procedures are done.

MBRC clinicians also teamed up with the Maryland Institute College of Art to test a virtual reality platform that can help patients deal with physical and emotional trauma through immersion in another world, with a focus on quadriplegic patients who are hospitalized with acute spinal cord injury.

Cohen, who leads the Center for Surgical Innovation and trains new physicians at Michigan, said that using XR in medical rounds—adding in virtual reality-based illustrations or augmented reality data overlaid on a patient—makes for a richer experience for both teacher and student, both in-person or virtually observing from anywhere in the world.

“We realized that having this ability to interact virtually with both the patients and residents—pulling up holographic windows and showing diagnostic imaging and labs—would greatly enhance the educational experience in these traditional grand rounds,” he said.

The MIXR initiative is heavily dependent on powerful computing resources. At Maryland, those resources will be handled by the University of Maryland Institute for Advanced Computer Studies. This includes building and maintaining a soon-to-be-unveiled “HoloCamera” studio, where more than 300 immersive cameras are fused together to bring unique visualization technology to bear on immersive medical environment captures.

The new camera system can be used to record cinematic-quality 3D demos of surgeons teaching intricate procedures like a lower extremity fasciotomy, a limb-saving technique of cutting the sheath of tissue encasing a muscle to treat for loss of circulation.

Other high-end technology will come from the MIXR industry partners, who are keen to see their latest hardware and software tools used to save lives and improve medical training.

“The synergy in MIXR will be contagious,” Varshney said. “Our industry partners will push forward new ideas and novel technologies. The scientists and physicians will help refine and test those ideas. And we both will work with the FDA to bring these technologies from the lab to the proper health care setting where they can have an exponential impact.”