Login to Access Video or Poster Abstract: MP92-12
Sources of Funding: Blue Cross Blue Shield of Michigan

Introduction

Systematic, automated methods for monitoring physician performance are necessary if changes in behavior are to be detected promptly and acted on. In the Michigan Urological Surgery Improvement Collaborative (MUSIC), we evaluated several statistical process control (SPC) methods to determine the sensitivity and ease of interpretation for assessing adherence to imaging guidelines for patients with newly diagnosed prostate cancer.

Methods

MUSIC is a quality improvement consortium consisting of 43 urology practices in Michigan. Following dissemination of imaging guidelines, MUSIC set a target rate of <10% for non-indicated bone and computed tomography scans. We compared four SPC methods: p chart, Bernoulli cumulative sum (CUSUM), weighted binomial CUSUM, and exponentially weighted moving average (EWMA). We studied non-indicated bone scan rates ranging from 11% (out of control rate) to 6% (in control rate) for the median MUSIC practice using each method. SPC method sensitivity was determined using the average run length (ARL): time taken to signal a change, per quarter. Using the Monte Carlo method (n = 10,000), we determined the ARL at the 11% out of control rate. Statistical analysis was performed using R (v 3.3.1). Non-indicated bone scan rates for a single MUSIC practice were plotted using each SPC method to qualitatively assess interpretation.

Results

When bone scan rates were in an out of control phase, EWMA and Bernoulli CUSUM methods were each found to have lower out of control ARL values (4.5 and 4.4 respectively) than weighted binomial CUSUM (5.0) or p chart (7.4; p<0.001); thus able to detect significant changes in imaging rates earlier. EWMA and p charts were easier to interpret graphically than CUSUM methods due to ability to display prior imaging rates. EWMA was the most suitable method due to its fast response time and ease of interpretation (Figure).

Conclusions

For the purposes of assessing adherence to guidelines in a statewide collaborative, we found the EWMA method most suited for detecting changes in imaging rates. This technique may have important implications for prospective automated monitoring of patient safety, such as tracking complications following prostate biopsy.

Funding

Blue Cross Blue Shield of Michigan