The value (and limitations) of surgical rate variation analysis
===============================================================

* Véd Tandan
* Bernard Langer

In 1973, Wennberg and Gittelsohn published an article entitled “Small area variations in health care delivery.”1 This work spawned a large volume of research into rate variations in health care. Surgical procedures lend themselves well to this type of study because they are more discrete and quantifiable than medical practices. As a result, variations in surgical procedure rates have become a focus of much research, with the objective being to identify the reasons for these rate variations.

In this issue (pages 361 to 367) Gentleman and associates present a study of variations in the rates of surgical procedures in census divisions across Canada. They have selected 39 procedures for study and classified them into “primarily discretionary” and “primarily non-discretionary” to test the hypothesis that there is greater variation in the rates of discretionary than non-discretionary procedures. They found that the rate variations between regions tended to be higher for operations they defined as discretionary. They suggest that the results of this study be used by the provinces and medical regulatory bodies to “focus on certain primarily discretionary operations with unusually high rates” and that a reduction in those rates will improve health status and reduce costs.

This paper contains much interesting information. However, it is more useful as background material for developing a hypothesis than for drawing conclusions or making recommendations. It is not the rate of performance of a surgical procedure but the outcome on the health of the population that is important. Although we are moving steadily toward more evidence-based medical practice, many common problems have not been subjected to the studies necessary to determine which treatment is best in terms of important outcomes such as survival, quality of life and cost. It is not surprising that in problems for which strong evidence of the efficacy of one treatment over another is lacking, there is more variation in the choices of which treatment to offer patients. For such problems, clinician and patient preferences often play a larger role than they would if better efficacy data were available.

If the approach of Gentleman and associates is to be used, the methodology for selecting procedures for study and classifying them as primarily discretionary or non-discretionary must be sound. According to Gentleman and associates, “We selected operations that are common or relatively common and are fully covered by health insurance . . . and we included examples from general surgery and the subspecialties.” These authors examined variation in surgical procedure rates in a previous publication2 and are thus biased in both the selection and classification of procedures. The classification was performed by only one of these authors, a non-surgeon. Although the classification was done independently of and prior to the data analysis, the author could not be blinded because of his knowledge of the data from the previous study. Despite the attempts to develop criteria for classification of procedures, the classification is still arbitrary and would be better performed by a panel of physicians and surgeons than a single unblinded (and possibly biased) person.

The very use of the term discretionary implies that the primary factor in decision making regarding the choice of operation is the surgeon’s opinion. This totally ignores other factors such as the availability of resources (donor kidneys in relation to transplantation, cardiac surgical operating time in relation to heart operations, hospital budget allocations in relation to knee replacement surgery and surgical subspecialist availability in relation to orthopedic, urologic and ophthalmologic surgery). To conclude, as the authors have, that where there are wide variations, health will be improved by bringing the high end rates down is unsupported by their data. It may be that factors such as those listed above are actually responsible for rates being too *low* in some areas. The authors’ focus on the discretionary issue also ignores the roles of indications for operation. Hall and Cohen3 showed that when variations in hysterectomy rates were stratified by indications, there was significant variation for indications such as menstrual hemorrhage, but the rate of variation of hysterectomy for cancer was not significant.

Another major problem in the data is the selection of procedures that may be performed on either an inpatient or outpatient basis. Since data on outpatient procedures are not included in the data file used for this study, and since hospitals may have widely varying rates of inpatient versus outpatient performance of individual operations, reporting only inpatient procedures may give totally erroneous data. Eight of the 16 discretionary procedures listed in the top half of the authors’ Table I are frequently performed in an outpatient setting. Inclusion of the procedures appears to strengthen the authors’ case, but at the expense of using seriously flawed data.

It is often assumed in small-area analysis studies that when high rates of variation are found, the higher surgical rates in some geographic areas are too high rather than the converse. The authors reveal their own bias when they state “. . . we suspect, the high surgical procedures rates are too high . . . .” The question of “which rate is right” cannot be answered by this type of study. This is the point at which small-area analysis ends and outcome research begins.

Gentleman and associates suggest that the results of their study be used by the provinces and medical regulatory bodies to “focus on certain primarily discretionary operations with unusually high rates.” Although these data may be useful to identify procedures with high rates of variation, the classification of procedures as primarily discretionary or non-discretionary independent of indication is unimportant in terms of where to focus future studies. All procedures with unusually high rates of variation should be examined further, whether they are classified as discretionary or not. The variations in the rates of these procedures must be examined by indication. Those indications that exhibit high rates of variation, independent of the factors listed above, represent the areas of uncertainty in the medical community. These are the areas where future outcome research is needed.

Only with studies using important outcomes as end points can this issue be resolved and appropriate practice guidelines developed. Although government has an interest in seeing that health care delivery is appropriate, effective and cost-effective, it is the health care providers who need to become involved in developing the evidence-based standards of practice that will achieve these ends. Experiences show that this approach can have an impact on practice.4

The study of Gentleman and associates provides data on national rates of variation of 39 surgical procedures using an index (I95) that eliminates outliers. They have identified several procedures for which the indications need to be examined and alternative treatments need to be compared in future clinical studies. These data could be greatly improved by including outpatient surgical procedures to complete the picture and by stratifying procedures according to indications. The use of the arbitrary classification of procedures as discretionary or non-discretionary is not supported by the data in the paper and distracts attention from other more important issues that contribute to when an operation is (or should be) carried out.

## Footnotes

*   [A response from Drs. Gentleman and Vayda and Mr. Parsons can be found in the correspondence section on [pages 425 and 426](http://canjsurg.ca/lookup/volpage/39/429).]

## References

1.  Wennberg JE, Gittelsohn AM. Small area variations in health care delivery. Science 1973;183:1102–8.
    
    

2.  Gentleman JF, Parsons GF, Walsh MN, Vayda E. High and low surgical procedure rates in census divisions across Canada. Health Rep 1994;6: 403–40.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=7795179&link_type=MED&atom=%2Fcjs%2F39%2F5%2F351.atom) 

3.  Hall RE, Cohen MM. Variations in hysterectomy rates in Ontario: Does the indication matter? Can Med Assoc J 1994;151:1713–9.
    
    [Abstract](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMToiMTUxLzEyLzE3MTMiO3M6NDoiYXRvbSI7czoxODoiL2Nqcy8zOS81LzM1MS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

4.  Soliman SR, Burrows RF. Cesarean section: analysis of the experience before and after the National Consensus Conference on Aspects of Cesarean Birth. Can Med Assoc J 1993;148:1315–20.
    
    [Abstract](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMDoiMTQ4LzgvMTMxNSI7czo0OiJhdG9tIjtzOjE4OiIvY2pzLzM5LzUvMzUxLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==)