La Trobe University in Australia has used CT scans of a coronavirus-infected patient’s lungs to build a virtual 3D model of the infected organs, providing unique insight of the disease viewed through Microsoft resources such as a HoloLens 2 mixed reality (MR) headset.
Believed to be a world first, the visualisations enable enhanced exploration inside the patient’s lungs, allowing the user to have a closer interaction with particular areas of interest.
The system has been developed over eight weeks by a team led by Henry Duh, La Trobe’s head of Computer Science and Information Technology.
Duh said 3D modelling can show the transformation from healthy lungs to those infected by coronavirus – a sequence which is hard to pick up with 2D biomedical images – and provide a better understanding of how the virus invades the body.
“Medical imaging is indispensable to the delivery of healthcare and an essential component of clinical research,” said Duh. “However, there is currently a lack of appropriate software to assimilate and derive maximum benefit from multimodal interactions with medical images in mixed reality environments.”
La Trobe University seeks to overcome that by building immersive medical tools using AR – the ability to superimpose 3D graphics over real-world views seen through a camera lens.
Leveraging Microsoft resources including Windows10 and HoloLens2, the University hopes the technology will one day be adopted in clinical diagnosis.
The team is now working toward using AI to automatically segment the lungs – or indeed any organs of interest – and render 3D scans from 2D medical images, to provide a deeper analysis of what it reveals.
“AI and the HoloLens2 are combined here to provide insights that would be hard to grasp in any other way – and the speed at which La Trobe was able to spin up this application is extremely impressive,” said Tiffany Wright, education director for Microsoft Australia.
Wright siad that starting in 2021, La Trobe is going to offer a new course – a Masters in Digital Media which will explore issues such as user experience, interaction design and virtual and mixed reality environments.
“The ultimate goal of our project is to visualise the infected lungs and at the same time link to the outcomes of AI analysis,” said Duh. “Clinicians will be able to see the analytics and interact with the model to gain insights from the system.”
