Synergies and Conflicts in Multi-Task Surgical Vision: Stereo Matching, Segmentation, and Classification of Stereoendoscopic Video Frames in Mitral Valve Surgery
Inminimally invasive and robot-assisted mitral valve surgery, direct vision is replaced by endoscopic imaging, which limits depth per ception and increases the cognitive demands on surgeons. AI-based as sistance systems could address this issue by allowing a holistic surgical understanding. We introduce MiTriSurv, a multi-task framework that si multaneously performs the classification and segmentation of surgical in struments, the segmentation of the mitral valve leaflets, and stereo-based depth estimation meeting near real time intraoperative requirements. We present an annotated stereo-endoscopic dataset from 20 patients. We per form a systematic analysis of the interactions between tasks, which shows that tissue segmentation benefits from geometric depth cues and that in strument classification benefits from joint training with instrument seg mentation. However, combining mitral valve leaflet segmentation with instrument classification is counterproductive, demonstrating that task compatibility is a critical factor in multitask surgical vision.
