Introduction

Pilot work from the Michigan Urological Surgery Improvement Collaborative (MUSIC) showed that crowdsourced (i.e. lay-person) video ratings of skill for surgeons performing robot-assisted radical prostatectomy (RARP) demonstrated strong correlation with peer surgeon ratings. In this project, we assessed the association between crowd review of surgeon skill with patient outcomes following RARP.

Methods

Surgeons in MUSIC were invited to submit a representative video of a nerve-sparing RARP. Edited video clips of the urethrovesical anastomosis from surgeons underwent evaluation of global skill by crowdworkers via C-SATS Inc, (Seattle, WA) using the Global Evaluative Assessment of Robotic Skills (GEARS) tool (maximum score 25). A mean GEARS score for each video clip was derived using a linear mixed effects model. Surgeons were ranked on their skill scores and sorted into quartiles of skill. Using data from a prospective registry involving 2,256 patients, we compared risk-adjusted peri-operative complication rates at the patient level between surgeons (n=7) in the lowest (Q1) skill quartile to surgeons (n=8) in the highest (Q4) quartile (n=8). Odds ratios (OR) were calculated using logistic regression models. Likelihood ratio tests were used to statistically assess whether outcomes differed between ratings quartiles.

Results

Crowdworkers (n=285) provided 867 video ratings on anastomosis procedures from 29 MUSIC surgeons within 3 hours and 48 minutes. Crowd ratings for global robotic skill of the anastomosis ranged from 16.5 to 18.0 in the lowest quartile (Q1), and from 20.2 to 21.9 in the highest quartile (Q4). Compared to surgeons in Q1 for anastomosis skill, surgeons in Q4 for skill had significantly lower rates of urethral catheter replacement (OR=0.45, p<0.001) and readmission rates (OR=0.54, p=0.002) after RARP, but not lower rates of excess (>400 cc) blood loss (OR=1.03, P=0.901) (Table).

Conclusions

Crowdsourced assessments of technical skill for practicing robotic surgeons performing the anastomosis varied widely, and correlated with clinically relevant post-surgical outcomes for RARP. In the future, the technical ability of a surgeon may become a significant performance measurement directly linked to patient outcomes.

Funding

Blue Cross Blue Shield of Michigan