Abstract: PD46-04
Sources of Funding: Blue Cross Blue Shield of Michigan; Intuitive Surgical

Introduction

A surgeon's technical skill may be a major determinant of patient outcomes. Because robotic surgery can be recorded, computer-vision video analysis of skill and technique (VAST) methods may have advantages for assessment that is objective and scalable. To test the hypothesis that specific features in a video can categorize skill, we studied crowdsourced annotated videos of surgeons performing robotic prostatectomy and applied machine learning to determine skill.

Methods

Videos of the anastomosis from 12 surgeons in the Michigan Urological Surgery Improvement Collaborative underwent blinded review by 25 peer surgeons using the Global Evaluative Assessment of Robotic Skills (GEARS) tool (max score 25). Surgeons were categorized into low and high skill based on &[prime]bimanual dexterity&[prime] and &[prime]efficiency&[prime]. Robotic instruments were annotated by crowdworkers via a custom-designed Mechanical Turk platform. Using the videos we trained a linear support vector machine (SVM), sampling consecutive frames to study VAST metrics for instruments including velocity, trajectory, smoothness of movement, and relationship to contralateral instrument. We applied the SVM to learn and classify videos into high/low skill. To evaluate performance we used 11 videos as training, and tested on the remaining 1 video, repeating it 12 times and averaged the accuracy.

Results

GEARS scores ranged from 15.75 to 23.11, with 9 and 3 surgeons categorized into high and low skill, respectively. In total, 146,309 video frames were annotated by 925 crowdworkers. Instrument annotation included individual points as well as wristed joint movement (Figure). SVM accuracy in skill categorization using individual points on an instrument was 83.3%. Accuracy improved to 91.7% when we assessed joint movement. When we combined assessment with the contralateral instrument, accuracy was 100% in categorizing binary skill level. Instrument metrics most closely related to skill prediction were relationship between needle driver forceps and joint, acceleration, and velocity.

Conclusions

Computer video analysis can be used to predict skill in practicing robotic surgeons. In the future, methods utilizing deep learning to track instruments and calculate skill, may have significant implications for credentialing and quality improvement.

Funding

Blue Cross Blue Shield of Michigan; Intuitive Surgical