Combining vision and touch in augmented reality to support surgeons in reconstructing complex fractures. This is the focus of a team of researchers from the Department of Information Engineering at the University of Pisa who have created a prototype and tested it on a displaced knee fracture. The prototype comprises a simple, low-cost tactile device created in the university’s laboratories that complements the augmented reality headset by integrating visual and tactile stimuli. Consisting of two rings applied to the fingers, it generates vibrotactile stimuli. These stimuli provide valuable information, enabling operators to manipulate virtual bone fragments correctly through natural gestures such as pushing, grabbing and aligning.
“For surgeons, it is not always easy to understand how to arrange the fragments of a fracture,” explains Vincenzo Ferrari, Professor of Bioengineering at the University of Pisa. “Our system recreates the fracture in augmented reality and uses vision and tactile stimuli to guide the doctor in planning the operation.”
“The device provides tactile feedback through vibration,” says Matteo Bianchi, Professor of Robotics at the University of Pisa, “improving spatial awareness and immersion during surgical training or preoperative planning. The absence of tactile feedback is in fact a significant limitation of virtual and extended reality training systems for surgery because, in common clinical practice, medical staff are largely guided by what they feel with their hands as well as by what they see. This aspect is particularly important when dealing with delicate and hidden anatomical structures. Our results show that tactile feedback significantly improves accuracy and efficiency in fracture fragment reduction tasks in virtual and extended reality contexts. Therefore, the integration of vision and touch will be fundamental in future augmented reality-based surgical training and assistance systems.”
Research into the medicine of the future is being conducted at the Centre 5.0 of the Department of Information Engineering. This centre is dedicated to developing technologies for society, medicine and next-generation industry. It is working in collaboration with the EndoCAS Centre for computer-assisted surgery, the “E. Piaggio” Research Centre, and the School of Specialisation in Orthopaedics and Traumatology at the University of Pisa.
