X-Ray Vision: Enhancing Liver Surgery with Augmented Reality

Liver cancer has the fastest growth of incidence and the second highest mortality of all cancers in the United States. Worldwide, it is estimated that over one million people will die from liver cancer in 2030. Liver resection (hepatectomy) is the paradigm for treating liver cancer. A crucial part of a partial hepatectomy is understanding where the tumors, vessels, and other important landmarks are located. To aid in this, the patient typically undergoes preoperative cross-sectional imaging (e.g., CT/MR scans). Surgeons use these images to determine resectability based upon the location of important structures (e.g., veins), analyze tumor margins, accurately compute future liver remnant volumes, and generally aid in surgical planning and navigation.

An augmented reality image guidance system for enhancing liver surgery.

However, it is challenging for the surgeon to mentally register preoperative cross-sectional images to the surface of the liver at the time of operation since surgical actions cause significant and sometimes permanent liver deformations that lead to mismatches with cross-sectional images. Mentally integrating preoperative data into the operative field is time consuming and error prone. This can make it difficult to accurately localize smaller tumors intra-operatively, which can affect surgical decision making and adequate resection of primary and metastatic liver tumors.

Dr. Michael Barrow‘s PhD thesis developed augmented Reality (AR) image guidance techniques that merge preoperative data directly into the surgeons view during surgery. The goal is to provide surgeons with what Michael describes as “X-ray vision” — allowing them to see through tissues and better understand where blood vessels, tumors, and other important surgical landmarks lie.

Current scenario: The surgeon has to estimate internal vessel positions
X-Ray vision: the surgeon is presented an AR overlay of internal landmarks.

The research brings together many state-of-the-art technologies. It requires computer vision approaches to track the surgical scene, real-time mechanical modeling of the organ to accurately place the important unseen surgical landmarks, augmented reality to visualize the landmarks, and hardware accelerated compute systems to process the high throughput sensor data. He showed that patient specific biomechanical modeling results in clinically significant increases in accuracy. Specifically, he built a system that uses magnetic resonance elastography to create a patient-specific mechanical model. The system works in real-time to provide accurate positions of unknown landmarks. He physically validated the techniques by creating a phantom mechanical platform to demonstrate it is possible to track landmarks internal to the phantom liver.

Left: Overview of the complete AR surgical system. Right: Experimental platform used to validate the AR accuracy

Michael took an unconventional path to his PhD. Unlike most PhDs, he laid out his research topic almost solely on his own. He spent a lot of time shadowing medical doctors to understand their problems. He deftly maneuvered through many different fields, seeking out and finding key collaborators. The result is an amazing example of an interdisciplinary thesis that has tremendous potential value in a clinical setting.

Michael developed a number of other technologies that are not reflected in his thesis. Most recently he is focusing on developing technologies to help into COVID-19 crisis which was awarded an UCSD Institute of Engineering in Medicine Galvanizing Engineering in Medicine award. He lead a team of undergraduates to build systems to better scale the care of COVID-19 patients (for more information see CSE Research Highlight).

Michael was a real tour de force in pushing collaborations between the School of Engineering and the School of Medicine. In addition to his Phd thesis project, he developed a close collaboration with Dr. Shanglei Liu and made many other connections between our research group and the medical school that will certainly create more future fruitful collaborations.

After graduation, Michael started a post-doctoral position at Lawrence Livermore National Labs.