Virtual reality (VR) and augmented reality (AR) have seen an increased utilisation in healthcare education in the past two decades. There has been much published in literature about the effectiveness of VR and AR in higher education, especially in areas of anatomy and simulation of hospital environments. Research in this area has been shown to reduce learning time, minimise errors, and lower mental workload of the participants. However, there are significant drawbacks associated with VR and AR. Users are unable to perceive the real-world while using VR devices, and this causes many issues including motion sickness, visual fatigue, lack of realism in the virtual scenarios, and lack of inter‑user engagement.

Though a step up from VR devices, AR devices experience some of the same issues faced by VR, including limited user engagement, restricted controls, and the inability to render fully functional scenarios.

Mixed Reality (MR) technology incorporates the best of both VR and AR into a single device. It uses transparent visors to project holographic renderings into the user’s eyes, and as the images are projected to both eyes (as opposed to just one eye in most AR devices), it allows for the user to have a full stereoscopic perception of the image, including distance, depth, and spatial orientation. The key advantages of MR are:

MR is also able to achieve a greater level of interaction compared to AR as it recognises free hand gesture control and is not limited to a fixed set of control gestures.

Since April 2022, medical and nursing students at NUS Yong Loo Lin School of Medicine started using MR technology to help enhance their learning experience and education journey.

The collaboration, which spans NUS Medicine, the National University Health System and Microsoft adds MR to the learning experience, named Project Polaris. Through holographic technology, medical and nursing undergraduates would better hone their skills through training enabled by the Microsoft HoloLens 2.

From left: Year 2 Nursing student Kerwin Chia, Phase V Medicine student Nicholas Sim, Dr Gao Yujia, Co-lead of Project Horizon and Deputy Director of Undergraduate Medical Education, Year 2 Nursing student Lim Hwee Xin and Phase IV Medicine student Sarvananthan Rajragavan are part of the pilot batch of nursing and medical students who have been using the Microsoft HoloLens 2 as part of the nursing and medical curriculum.

A name derived from the North Star, or Pole Star, Project Polaris focuses on the development of modules aimed at enhancing how procedural skills are taught to students. In Project Polaris, the HoloLens 2 will be used to project three-dimensional holograms to give medical and nursing students a visual appreciation of actual clinical scenarios in practice. Using the already established procedural skills manual, POLARIS will seek to transform the current text-based manual into a fully interactive MR experience with video, pictorial, and audio guidance to enhance the student’s learning experience, with the eventual aim of converting most procedural skills, typically done manually, onto the MR platform.

The suite of instructional software developed by the team from NUS Medicine and Microsoft Industry Solutions is broken down into four components, namely learning tasks, supportive information, procedural information and part-task practice. It provides 3D instructional images, which will be used to help students practice clinical procedural skills such as inserting a cannula, as well as inserting catheters in male and female urinary tracts. Separate windows are visible to show recorded videos, as well as step-by-step interactive guides on the procedure at hand. This guide will include a list of all equipment required for the task, equipment familiarisation, indications, and contraindications for the procedures, as well as common pitfalls and tips/tricks on how to avoid or troubleshoot them.

Assoc Prof Alfred Kow, Assistant Dean of Education, is the co-lead of Project Horizon that introduces holographic mixed reality to create a more holistic and interactive learning experience for aspiring medical professionals.

Project Polaris comes with three levels of difficulty, with a goal to train and provide sufficient direction to allow students at varying levels of competence to achieve the highest standards of clinical practice in a safe space, said the School’s Assistant Dean for Education and Project Co-lead, Associate Professor Alfred Kow. “We are truly excited to see how far this collaboration with Microsoft can go in terms of offering our students innovative and effective visual aids and teaching mechanisms. With the ongoing pandemic, VR and MR have been identified as must-have tools for teaching and learning in onsite and remote environments.”

At the beginning, students will be allowed access to the complete set of user-assistance guides. As they progress through their learning journey and gain more experience with these procedures, the guides can either be hidden automatically, or the tutor may choose to disable selected guides to provide the student with a more challenging task. Project Polaris aims to integrate their non-recurrent and recurrent skills by providing an authentic, whole-task experience-based learning module based on real-life tasks.

The COVID-19 pandemic also forcefully brought home the importance of building in versatility in the medical undergraduate curriculum and teaching pedagogy. Social distancing and home-based learning negated much of the traditional teaching modalities, and new methods had to be adopted to adapt to the situation. Even so, much of the adaptation involves only the use of online conference tools as a replacement for face-to-face interactions. This still lacks the interaction that is so desired and crucial when building interpersonal skills. Project Polaris provides an avenue for us to create content and teaching methodologies that can transcend distance and space, where students and educators are still able to ‘come together’ in an artificially rendered world regardless of where they are, yet still feel like they are right next to each other. Negating physical distance by ‘teleporting’ participants to the same shared environment provides a much more interactive experience for everyone.

Project Polaris is part of a larger programme called Project Horizon, which consists of more initiatives such as Project Delphinus and Project Mira. These aim to train students in clinical soft skills and clinical anatomy respectively, positioning NUS Medicine as the first institute in Southeast Asia to introduce holographic MR as a teaching tool to train medical and nursing students. The medical and technical expertise of NUS Medicine and Microsoft will pave the way for the development of a niche technological competency, in which clinical training tools can be developed to introduce realistic clinical scenarios for use in medical education.

“We are continually pursuing new and innovative teaching methods to help medical and nursing students better understand the medical curriculum and gain a new appreciation for healthcare and health, while striving to maintain a balance with time-tested traditional approaches,” said Associate Professor Lau Tang Ching, the School’s Vice-Dean for Education. “This incorporation of holographic mixed-reality learning fits in well with our teaching initiatives and we hope to see this collaboration with Microsoft flourish in the coming years.”

“From delivering better healthcare experiences at the frontlines to helping neurosurgeons keep patients better informed of what could happen during their surgeries, technology has been an empowering tool for healthcare workers as they protect and save the lives of patients. As one of the few hospitals in Southeast Asia that has a tertiary education arm which collaborates with a training hospital, NUS Medicine is in a unique position to use MR solutions and the Microsoft HoloLens 2 to aid in the transformation of healthcare education, for years to come,” said Mr Richard Koh, National Technology Officer, Microsoft Singapore.

Professor Chong Yap Seng, Dean of NUS Medicine, said he looked forward to the impact that technology would bring to medical education, “I’m glad to see that our educators have found like-minded industry partners to work with in educating the next generation of medical professionals. With such interdisciplinary partnerships, we are even more confident that our graduates will be future-ready clinicians.”