ISSN: 2161-0940
+44 1300 500008
Review Article - (2019)Volume 9, Issue 1
Virtual Reality (VR), Augmented Reality (AR) and Head Mounted Displays (HMDs) are revolutionizing the way we view and interact with the world, affecting nearly every industry. These technologies are allowing 3D immersive display and understanding of anatomy never before possible. Medical applications are wide-reaching and affect every facet of medical care from learning gross anatomy and surgical technique to patient-specific pre-procedural planning and intra-operative guidance, as well as diagnostic and therapeutic approaches in rehabilitation, pain management, and psychology. The FDA is beginning to approve these approaches for clinical use. In this review article, we summarize the application of VR and AR in clinical medicine. The history, current utility and future applications of these technologies are described.
Virtual reality; Augmented reality; Surgery; Medicine; Anatomy; Therapy
Recalling the 1966 science fiction film “Fantastic Voyage”, watching in amazement as the doctors traveled inside the patient’s body to repair a brain injury. VR and AR, particularly with HMDs, are now allowing a similar, albeit safer, immersive interactive display of the patient’s anatomy, never before possible. AR and VR are now allowing the operator to virtually “travel” in the body while fusing the real world with a virtual depiction of anatomy and disease processes. The role of these paradigm-shifting technologies to affect every aspect of patient care is emerging as the FDA is beginning to approve related technologies with applications in all disciplines of surgery, as well as medicine and psychology. A large body of literature over at least the past 20 years substantiates the use of these technologies in medical training and practice. VR and AR HMD applications in medical practice are considered disruptive technologies poised to bring about a paradigm shift in how medical care is provided in nearly every medical specialty. These technologies have an increasingly meaningful role in training, procedural planning, inter-procedural guidance and even therapy such as peri-surgical pain management and rehabilitation [1].
The human visual system
Our visual system allows us to understand the world with incredible accuracy and efficiency. At a high level, it has two functional stages: sensing and perceiving. Your eyes act as sensors that capture light similar to how a camera captures a picture. VR and AR HMDs trick the human visual system at this stage by sending light from displays in a way that forces your brain to think the light is coming from real objects. At the second stage, our brain processes what is seen.
Two categories of visual processing are applied: monocular processing and binocular processing. Monocular processing requires a single image, such as the image from your computer screen, while binocular processing requires two images, such as the images seen by each eye [2]. While both categories of processing are important, binocular processing is the most effective way for us to understand depth and structure and is the fundamental reason why VR and AR HMDs, which create stereopsis through dualimage displays, are more effective at demonstrating anatomy than traditional flat screens that display a single image. Aspects of image display that constitute the 3D immersive sense in AR and VR are listed in Table 1.
Perspective Projection | Objects seem smaller as they get farther away |
Occlusion | An object in front of another object blocks, or occludes |
Familiar size | Understand the relative size of objects based on other objects we know |
Shading | Using light and shadows, this cue helps us understand shapes |
Structure from motion | Helps us understand an object’s shape as the object rotates |
Motion parallax | Describes shape and depth as we, the viewer, moves |
Table 1: Cues related to visual depth.
Virtual reality (VR)
The following represents a review of the literature and related websites, meant to outline the history of the development of VR. In the mid-1960’s, Sutherland first described VR as “a window through which a user perceives the virtual world that looked, felt, sounded real and in which the user could act realistically” [3]. VR provides a viewer-centered experience which is immersive and interactive, involving multiple sensory experiences [4]. The observer experiences a “stereoscopic” three dimensional (3D) image, coupled with head positioning which gives a sense of interactive motion [5]. Morton Heilig has been said to be the “Father of Virtual Reality”. In 1960 Morton received his first related patent for the “Telesphere Mask”. This was the first ever HMD. It enabled stereoscopic (3D) display and “surround” sound. In 1962, he invented the Sensorama, the first commercially available 3D immersive simulator in which the observer experienced a “motorcycle ride”. In the 70s, the GROPE system represented the use of “force-feedback” incorporating haptics, the sensation of feeling in VR. “VIDEOPLACE” was said to be an “Artificial Reality” in which an image of the operator, captured by cameras, was projected onto a screen and incorporated into a video of a scene [6]. In 1982, the US’ Air Force created the first flight simulator for pilot training, “The Visually Coupled Airborne System Simulator (VCASS)” with HMD. Commercial devices began to emerge in the ‘80s. The DataGlove added to VR incorporating hand gestures and position for interacting with the VR environment. At the end of the 80’s, Fake Space Labs created a Binocular-Omni-Orientational Monitor (BOOM), incorporating stereoscopic-display and a mechanical arm which is tracked and represented in the virtual images. BOOM offered reduced latency resulting in quicker response to movements then displayed in the HMD display [7]. The first clinical systems for surgery, laparoscopic procedures, telemedicine, and therapy began to appear in the 90’s. Medical VR applications research has emphasized surgical training/ planning and even therapeutic applications such as rehabilitation. Numerous recent clinical trials are demonstrating the utility of VR simulation and pre-operative planning in nearly every surgical discipline.
Augmented reality (AR)
By definition, AR with HMDs adds digital information to the real world while VR completely replaces the real world with a digital reality, implying an immersive experience that completely shuts out the physical world. When using AR artificial information is incorporated into the actual world as perceived by one or more senses. In this manner, video or computer generated images are superimposed onto the real world. According to Azuma, et al. [8] an AR system should:
1) Combine real and virtual objects in the actual environment
2) Run interactively and in real-time
3) Register real and virtual objects with each other
An AR system typically includes a camera to track the observer’s movement which is merged with the virtual objects. A visual display allows the observer to see virtual objects overlaid on top of the actual physical world. To date, two types of AR HMDs exist Video See-Through (VST) and Optical See-Through (OST) [9]. VST AR composites, or overlays, virtual objects on top of a video feed of the real world, while OST AR shines light into the retina by bouncing the light off of see-through a glass, such as the lenses of a pair of glasses.
The following represents a review of the literature and related websites, meant to outline the history of the development of AR. The first prototype of an AR system was introduced by the Boeing Corporation in the early 90’s. The system was created to aid employees in setting up a wiring tool [10]. Another AR system was introduced for maintenance assistance with resultant improved task performance [11]. The first medical application for AR was introduced by Loomis in 1993 comprising an AR GPS-based system to help the blind get around by adding audio information to assist sensing of spatial information [12]. Fuchs, et al. [13] demonstrated the clinical benefits of AR with their system that overlaid medical images on top of a patient receiving a biopsy. In the last 10-years tools for software developers, such as the HoloLens SDK and ARToolkit, have accelerated the development of AR applications. D’Fusion’s Total Immersion enabled operators to design AR system applications making AR more accessible to developers [14]. Google developed Google Glass and in 2016 Microsoft developed the HoloLens, both of which have been used in clinical research and FDA approved commercial applications. Their usability is currently being tested in multiple fields of application.
The expanding scope of AR and VR
The global AR and VR healthcare industry was estimated to be worth $641 million by 2018 and recently Medtech Boston estimated that by 2020 the global market for VR would reach $3.8 billion [15]. During the last 20 years, 1000s of scientific papers illustrated the processes, effects, and applications of AR and VR in medicine. Cipresso, et al. [7] searched VR in the “Web of Science Core Collection scientific database” and found 21,667 records for VR and 9,944 for AR.
There has been an incremental and steady progression towards immersive AR/VR platforms and mass-market appeal. However, computer-mediated systems are still relatively immature, with related techniques and applications waiting to be implemented and explored. This year the FDA approved the first AR and virtual systems for medical/surgical use. Recently Medtech Boston reported the top 7 companies developing VR and AR systems for medical applications demonstrating the diversity of medical applications beginning to be used in patient care [16].
The value of 3D depiction of medically relevant anatomy is obvious and can be seen in the now routine inclusion of multiplanar and 3D segmented images for axial cross-sectional imaging Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) that was traditionally displayed only in 2D cross-sectional planes [17]. Furthermore, cone beam rotational angiography with 3D segmentation is now common place during complex angiographic procedures [18]. Immersive holographic VR and AR display of patient anatomy can be thought of like a more effective way to display data from these technologies. Today, AR and VR have made significant inroads in the medical field: training, pre-procedural planning, and therapeutic interventions.
With respect to training, immersive holographic stereoscopic 3D/4D display:
1) Aids the detection of diagnostically relevant detail, orientation, and position of anatomic structures and related pathology
2) Helps novice surgeons orient themselves in the surgical landscape and perform complicated tasks
3) Improve the three-dimensional anatomical understanding, particularly in less experienced operators who may have low visualspatial skills [19]
3D computer modeling and interactive VR simulation has been validated as effective teaching adjuncts and is now being used throughout most medical disciplines. Surgeons need to have a detailed understanding of the surgical anatomy and pathology as possible. Much of this information is obtained from a review of recent CT or MRI images. 3D segmented reconstructed images of these scans provide a more realistic depiction of the relevant anatomy. VR with HMDs also has the capacity to add new methods for collecting information relevant to the response to training such as gaze tracking, body movement, a physiological response such as heart rate.
When training medical students, AR and VR, particularly when interactive, is suggested to be an effective adjunct for teaching gross anatomy [20,21]. Usual surgical training has been for the trainee to observe a senior surgeon and then perform the surgery under senior supervision. This approach of observation and training during the performance of the surgery is an age-old practice which may be improved upon, particularly with advanced techniques such as laparoscopic and robotic-assisted surgery. An ever-increasing number of increasingly complex VR based simulators have been used for surgical/procedural training for more than a decade. VR based training is now becoming commonplace. The trainee can interact with the surgical anatomy and experience changes that occur during surgical procedures in a lifelike but simulated/ immersive environment without associated risk of actual surgical complications. Thus the learning curve is significantly reduced. Furthermore, VR display of a patient’s actual anatomy, obtained from a prior CT or MRI can be used for patient-specific preoperative planning and training, performing the actual surgery in a simulation of the actual anatomy the surgeon/proceduralist will encounter. This pre-surgical experience would make the actual surgery safer and more efficient [22,23]. Meta-analyses have shown improved operative time, error rate and accuracy when new trainees, with little or no prior experience, use VR training based laparoscopic training [22,24].
VR based simulators have been used for training and surgical planning for nearly all surgical discipline, some of which are listed by Li, et al. including: “orthopedic surgery (Procedicus virtual arthroscopy simulator (Mentice AB, Sweden), Insight Arthro VR Shoulder Simulator (3D Systems, USA), and Trauma Vision (Swemac Orthopaedics, Sweden) [25], neurosurgery [24] Laparoscopic surgery (LapSim (Surgical Science, Sweden), Lap Mentor (3D System, USA), MIST-VR (minimally invasive surgical trainer-VR, Mentice AB, Sweden), Simendo (Simendo, Holland) VR and laparoscopic surgery [22], and plastic surgery [26-28]. Most recently Urologists demonstrated the use of pre-operative planning with VR to better understand the patient-specific renal anatomy and location of renal stones prior to performing percutaneous Nephrolithotomy. This use of VR altered the operative approach in 40% of cases, decreased fluoroscopy time and blood loss along with a trend toward fewer access tracts and higher stone-free rates [29]. Using VR based training, the trainee’s actions could be recorded and then evaluated and reviewed with trainees. This objective assessment of the operators’ psychomotor skills is valuable.
An ever-increasing number of VR simulators have been introduced into medical/surgical training. Multiple studies have concluded that both doctors and patients benefit from VR based medical simulators [7].
On the forefront of AR/VR technology is intra-operative guidance. The virtue of AR is that the virtual object, such as the relevant 3D anatomy, is superimposed on the view of the actual world, such as the patient. In this manner, AR may be better suited than VR for use as intra-operative guidance. AR has been used in neurosurgery for years. In this setting stereotactic surgery was used in combination with previously obtained radiographic images superimposed on patient surgical sites to allow accurate and safer “neuronavigation” which had a significant impact on operating time [29,30]. These AR-based navigation systems used in plastic surgery and neurosurgery provide a 3D depiction of the patient’s specific anatomy and pathology in a submersive interactive display. Intra-procedural guidance or initial planning is improved as images of patient-specific anatomy or virtual surgical plans can be seen “fused” with the operative site. Unlike VR systems, AR systems allow one to see the 3D patient images (digital) and the patient (reality) at the same time. AR also lets the operator see complete real-world surroundings, avoiding the disorientation of VR. AR has even been utilized by pathologists during the gross and histopathological assessment. This AR based examination could even be done remotely [31].
VR is also being used as part of therapy in psychology and rehabilitation medicine. VR is being used to diminish acute and chronic pain by affecting pain perception. The immersive virtual world takes the patient’s attention from the painful stimulus [32]. These “distraction interventions” result in measurable increased pain tolerance and pain threshold. Moreover, VR related distraction reduced pain intensity, time spent thinking about pain and related anxiety [33].
Finally, VR has been used for the actual treatment of psychological disorders including phobias and anxiety disorders. In this therapy, patients confront the situations they fear in a realistic and immersive/interactive environment. This “exposure therapy”, helps patients to curtail related anxiety and recognize that the feared the catastrophic result would likely not occur, thus sensitizing the patient to their phobia. VR approaches have also been successfully utilized to help patients with autism and attention deficit [34].
An important application of AR, and VR is to facilitate telemedicine, whereby procedures or other medical care, provided in a distant or remote location, can be observed in an immersive holographic display. In some cases, a distant operator may even interact with remote actual devices such as ultrasound or surgical robotic systems [35]. The principle of telemedicine is to transmit medical information and provide patient care from a distance. AR and VR can obviously enhance how we view this medical information, providing and immersive communication interface enhancing the feeling of being physically present at a distant site during patient care provided through telemedicine [36].
Though tremendous strides have been made over the past 10 years in the usability of AR and VR in medical applications, there is still room for improvement. A major limitation of AR is miss-registration between the real world being viewed and the superimposed virtual object [37]. This can occur as a result of “system latency, tracker error, calibration error, and optical distortion” [38]. AR Latency currently approximates 1 millimeter of misalignment for every 1 millisecond of latency. More powerful microcomputers are likely required to solve this. An AR HMD needs to be a natural extension of the surgeon’s senses. For this to be accomplished it has to be light, mobile, comfortable and functional for potentially long periods of time [5]. Currently, HMDs are somewhat large with cumbersome cables. In addition, AR HMDs are unable to fully immerse users in a digital world which limits what the user can see and how they interact with digital information. This may be ideal for certain scenarios but less ideal for others. Conversely, VR HMDs and their requirement to fully immerse users in a digital world can be limiting for certain tasks while ideal for others. Current AR and VR HMDs may be uncomfortable. Operators have reported being dizzy, disoriented, or nausea similar to motion sickness. For genuine adoption to occur, operators must be willing to embrace the challenges of these new technologies and guide their future development to meet the constraints of medical application [8].
In October 2018, The FDA granted 510(k) approval to the OpenSight AR System. This represents the first FDA approved AR medical application for Microsoft HoloLens, now cleared for use in pre-operative surgical planning [39]. This system interactively overlays 2D, 3D and 4D images of patient’s relevant anatomy, obtained from pre-operative MRI, CT or PET imaging onto the patient's body. “Persistent Registration™ technology” continuously aligns images with the actual patient. It is suggested that medical/ surgical procedures could be performed with decreased operative times, with improved precision, and safety. A multi-user experience is possible if multiple operators wear the HMDs. The system also allows AR-based intra-procedural guidance, which has been shown when applies to percutaneous biopsies using 3D freehand biopsy guidance utilizing needle sensors combined with scan-head localization sensors.
In September 2018 Leica Microsystems received FDA 510(k) gave clearance to market its AR based “GLOW800 surgical fluorescence for vascular neurosurgery” [40]. In combination with ICG (Indocyanine Green) and GLOW800, surgeons can view cerebral anatomy in natural color, augmented with superimposed coregistered real-time vascular flow allowing full depth perception. This AR application provides the surgeon with a never before possible immersive view of anatomy and physiology aiding in additional understanding during vascular neurosurgery.
In 2017, EchoPixel gained 510(k) FDA clearance for the True3D Viewer, the first platform to convert anatomical data from patients into fully interactive, 3-D VR images, allowing one to interact with patient-specific anatomy, from prior MRI or CT scans, in an open 3D space emanating from a desktop display [41]. The system has been used by interventional Radiologist pre-procedural planning of Trans-Hepatic Portal Systemic Shunt placement for portal hypertension-related variceal bleeding/ascites. Radiologists have applied it to CT Colonography to better understand colon lumen morphology and increase polyp detection sensitivity, speed, and accuracy. It was shown that Cardiologists could more accurately and quickly diagnose complex congenital cardiac pathology using EchoPixel [42]. The “body VR” is also one of the first medical commercially available VR visualization tools enabling an immersive holographic view of previously obtained MRI, CT, or PET images. The VR display of patient-specific anatomy can be used for preprocedural planning to affect how procedures are performed.
Recently MedTech Boston listed the top 7 VR digital health companies, which included the following companies. Osso VR which uses VR and HMD to track movements in a simulator environment to train orthopedic surgeons. SentiAR has developed an AR system for the visualization during interventional procedures. An interactive 3D hologram of the patient’s anatomy and catheter location is displayed. “Through the Microsoft HoloLens, physicians have full control of their view using real-time navigation data, rather than MRI/CT, for procedures like catheter-based electrophysiological ablations for the treatment of arrhythmias.” The AR platform enables real-time mapping/catheter location outputs into a real-time hologram in the clinical field of view in an immersive patient-specific cardiac model resulting in significantly faster and more accurate procedures. FDA approval is currently pending. MedTech Boston stated that ImmersiveTouch has been a premier provider of Medical VR and AR products for over 13 years. Their proprietary haptic technology, incorporating tactile feedback into VR based simulation is FDA approved for pre-procedural planning.
The technology offered by Augmented Intelligence Inc. creates a “patients’ ‘digital twins’ based on their CT/MRI” this can then be used for educational and clinical applications as described in this review.
In conclusion, the virtues of VR and AR in medicine and surgery were well summarized by Kim, et al. [25] including:
1) VR/AR allows the virtual presentation of clinically relevant anatomy and physiology to all human senses in a way identical to their natural counterparts
2) As a diagnostic tool, the simulated 3D reconstruction of organs and related pathology, based on radiological data, can provide a more naturalistic view of a patient's anatomy and clinical presentation
3) Preoperative surgical planning can provide a more realistic prediction of the outcome
4) Computerized 3D atlases of human anatomy, physiology, and pathology can provide better learning and training systems
5) Intraoperative navigation reduces the possibility of major complications and increases the possibility of the best surgical results
6) VR/AR technology can play an important role in telemedicine, from remote diagnosis to complex tele-interventions
The FDA is beginning to approve the use of these technologies continuing to herald the fact that a paradigm shift is underway with the potential for these technologies to affect every aspect of medical care. Though there is still room for improvement, it is clear that there is potential for these technologies to affect every aspect of medical care.
Citation: Monsky WL, James R, Seslar SS (2019) Virtual and Augmented Reality Applications in Medicine and Surgery-The Fantastic Voyage is here. Anat Physiol 9:1.
Received: 09-Jan-2019 Accepted: 08-Feb-2019 Published: 15-Feb-2019
Copyright: © 2019 Monsky WL, et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.