Objective: To estimate the likelihood of poor surgical results being explained by chance rather than under-performance.
Methods: The 30-day mortality rates after radical cystectomy for bladder cancer were analysed theoretically. Surgical competence was defined as a mortality rate of 4%, excellence as 2% and under-performance as 8%, 12%, 20% or 40%. Four scenarios were explored for surgeons of different competence: first, the sample size required to show that a given level of under-performance is very unlikely to be due to chance; second, the likelihood of two or more consecutive deaths in a series of cases; third, the likelihood of clustering of deaths, defined as two deaths in five or in 10 cases; and last, the likelihood of outstanding surgical results (i.e. no deaths) being achieved in small cohorts by surgeons of differing competence.
Results: For surgeons with a mortality rate of 8%, 12%, 20% or 40%, the sample sizes needed to prove under-performance are 211, 65, 21 and seven, respectively. For consecutive deaths, 0.4% of excellent, 1.4% of competent and 21% of surgeons with a mortality rate of 12% will experience two or more consecutive deaths in the next 10 cases. For clustered deaths, 1% of excellent, 5% of competent and 23% of seriously under-performing surgeons (mortality rates > or = 12%) will experience two deaths in their next 10 cases. Lastly, for the likelihood of outstanding results, only 3.6% of surgeons with an 8% mortality rate and < 1% of surgeons with a mortality rate > or = 12% will experience no deaths over 40 consecutive cases.
Conclusions: Very large cohorts are needed to confirm even significant under-performance. Consecutive deaths are very unlikely events for competent surgeons. Clustered deaths (two deaths in 10 cases) are very unlikely events for excellent surgeons but plausible for competent ones. Analysis of consecutive/clustered deaths is limited by low statistical sensitivity; only up to a quarter of seriously under-performing surgeons are identified. No deaths in 40 consecutive cases implies competence.