Intended for healthcare professionals

Education And Debate

Need for expertise based randomised controlled trials

BMJ 2005; 330 doi: https://doi.org/10.1136/bmj.330.7482.88 (Published 06 January 2005) Cite this as: BMJ 2005;330:88
  1. P J Devereaux, assistant professor (philipj{at}mcmaster.ca)1,
  2. Mohit Bhandari, orthopaedic surgeon2,
  3. Mike Clarke, director3,
  4. Victor M Montori, assistant professor4,
  5. Deborah J Cook, professor1,
  6. Salim Yusuf, professor5,
  7. David L Sackett, director6,
  8. Claudio S Cinà, vascular surgeon2,
  9. S D Walter, professor1,
  10. Brian Haynes, professor1,
  11. Holger J Schünemann, associate professor7,
  12. Geoffrey R Norman, professor1,
  13. Gordon H Guyatt, professor1
  1. 1 Department of Clinical Epidemiology and Biostatistics, McMaster University, 1200 Main Street, West Hamilton ON, Canada L8N 3Z5,
  2. 2 Department of Surgery, McMaster University,
  3. 3 UK Cochrane Centre, Oxford,
  4. 4 Department of Medicine, Mayo Clinic College of Medicine, Rochester, Minnesota, United States,
  5. 5 Population Health Research Institute, McMaster University,
  6. 6 Trout Research and Education Centre of Irish Lake, Markdale, Ontario, Canada,
  7. 7 Departments of Medicine and Social and Preventive Medicine, University at Buffalo, Buffalo, New York, United States
  1. Correspondence to: P J Devereaux
  • Accepted 24 October 2004

Surgical procedures are less likely to be rigorously evidence based than drug treatments because of difficulties with randomisation. Expertise based trials could be the way forward

Introduction

Although conventional randomised controlled trials are widely recognised as the most reliable method to evaluate pharmacological interventions,1 2 scepticism about their role in nonpharmacological interventions (such as surgery) remains.36 Conventional randomised controlled trials typically randomise participants to one of two intervenions (A or B) and individual clinicians give intervention A to some participants and B to others. An alternative trial design, the expertise based randomised controlled trial, randomises participants to clinicians with expertise in intervention A or clinicians with expertise in intervention B, and the clinicians perform only the procedure they are expert in. We present evidence to support our argument that increased use of the expertise based design will enhance the validity, applicability, feasibility, and ethical integrity of randomised controlled trials in surgery, as well as in other areas. We focus on established surgical interventions rather than new surgical procedures in which clinicians have not established expertise.

Use of expertise based trials

Investigators have used the expertise based design when conventional randomised controlled trials were impossible because different specialty groups provided the interventions under evaluation—for example, percutaneous transluminal coronary angioplasty versus coronary artery bypass graft surgery.79 In 1980, Van der Linden suggested randomising participants to clinicians committed to performing different interventions in an area in which a conventional randomised controlled trial was possible.10 Since that time, however, the expertise based design has been little used, even in areas where it has high potential (such as, surgery, physiotherapy, and chiropractic).

Problems with validity of conventional randomised controlled trials

Differential expertise between procedures

Because it takes training and experience to develop expertise in surgical interventions, individual surgeons tend to solely or primarily use a single surgical approach to treat a specific problem.10 11 The restricted expertise that results can compromise the validity of conventional randomised controlled trials. For example, in a conventional randomised controlled trial, if surgeons with expertise in intervention A treat 70% of the patients in both groups A and B, and surgeons with expertise in intervention B treat 30% of those in both groups A and B, the trial results will be biased towards intervention A (fig 1). We will refer to this type of bias as differential expertise bias. The more disproportionate the number of cases being performed by surgeons with expertise in procedure A compared with surgeons with expertise in procedure B, the greater the impact of differential expertise bias on the trial results.


Embedded Image

Differential expertise bias in a conventional randomised controlled trial in which 70% of surgeons are expert in procedure A and 30% in procedure B

We estimated the potential for differential expertise bias through a survey of 139 surgeons in a large (> 1000 patients) conventional randomised controlled trial comparing two surgical procedures for treating a tibial shaft fracture (reaming versus no reaming before insertion of an intramedullary nail).12 Seventy four surgeons completed the survey. Significantly more surgeons had no or limited experience with the non-reamed procedure (which is more technically challenging) than the reamed procedure in the year before they joined the randomised controlled trial (table 1). The median number of cases surgeons performed in the year before randomised controlled trial participation was 12 reamed procedures and 2 non-reamed procedures (median difference 7 procedures, 95% confidence interval 5 to 11).

Table 1

Experience of 74 surgeons with reamed and non-reamed procedure in the year before participating in the randomised controlled trial

View this table:

This example shows the potential for differential expertise bias. Three key considerations suggest that this problem is likely to be common in surgical trials. Firstly, trialists rarely, if ever, institute measures to ensure that the number of participating surgeons with expertise in each procedure is equal. Secondly, although some conventional randomised controlled trials try to reduce bias by requiring participating surgeons to perform a minimum number of both the experimental and control procedures before participating in the trial, this measure is unlikely to eliminate bias because outcomes often improve with extensive experience with a procedure. Thirdly, even if these two problems are overcome, one of the procedures (let us say procedure A) may be more technically challenging. If this is the case, after doing the required numbers of unfamiliar procedures, surgeons who have to acquire expertise in procedure A will remain more technically challenged than those who have to acquire skills in procedure B. In this situation, the trial will be biased towards B, the less technically challenging procedure.

The second of these three issues deserves further discussion. All surgical interventions have a learning curve. Depending on how technically challenging the procedure is the number of cases required to achieve basic and advanced competence will vary. For many procedures, surgeons are unlikely to reach the plateau of the learning curve until they undertake many more procedures than are required to participate in a conventional surgical randomised controlled trial. For example, an observational study of 15 427 segmental colon resections showed that surgeons with five or more years' experience after surgical residency had lower risk adjusted death rates than surgeons during their first four years after residency training.13 Similarly, an observational study of 5678 isolated coronary artery surgeries showed a progressive decrease in observed and risk adjusted mortality during the first four years of independent practice.14 Therefore, differential expertise bias probably persists even when surgeons perform a minimum number of cases with the procedure in which they lack expertise before participating in a conventional randomised controlled trial. Exceptions may exist for procedures so technically simple that experienced surgeons acquire expertise after a small number of cases.

Potential problems related to unblinded surgeons

Surgeons participating in conventional surgical randomised controlled trials usually have opinions about the relative effectiveness of the procedures under investigation. Surgeons solely or primarily using procedure A probably do so because they believe it gives better outcomes. As a result, they probably expect and hope that the randomised controlled trial testing the outcomes of procedure A versus procedure B will affirm their belief.

Thus, surgeons, who are necessarily unblinded to the procedure they perform, may subconsciously systematically bias trial findings in a conventional randomised controlled trial. This bias may manifest itself through several mechanisms, including surgeons being more meticulous when performing one procedure than the other or differentially prescribing effective cointerventions.2 Although it is preferable for independent blinded individuals to collect data and assess outcomes, in some trials it is done by the surgeons. When outcome evaluation is open to judgment and surgeons are involved in the process, they may differentially record data, repeat measurements, or interpret outcomes depending on whether a patient received procedure A or procedure B.15

We asked surgeons participating in the randomised controlled trial of different strategies for nailing tibial fractures whether they thought a reamed procedure or non-reamed procedure was superior before participating in the randomised controlled trial and at the time of the survey (that is, when about 900 patients had been randomised).12 Surgeons rated their confidence about the superiority of the procedure they selected on a seven point scale, with 1 representing no confidence, 4 representing moderate confidence, and 7 representing extreme confidence. Before participating in the randomised controlled trial, 87% (95% confidence interval 77% to 94%) of respondents believed that a reamed procedure was superior and 86% of respondents indicated their confidence about the superiority of a reamed procedure was in the moderate to extreme range. After 900 patients were randomised, responses remained similar.


Embedded Image

The results of this survey reflect the possible magnitude of treatment preference among surgeons participating in a randomised controlled trial comparing surgical procedures. This may lead to bias for reasons outlined above. As is the case with balancing expertise, trialists are unlikely to be able to ensure the absence of a dominant treatment preference among participating surgeons.

Procedural crossovers

Our ability to determine if patients have a better outcome when they receive one of two procedures will be enhanced if patients actually receive the procedures to which they were randomised. If this is not the case because of procedural crossovers, the trial's ability to determine the true effect will be compromised.

We evaluated the number of crossovers in the reamed and non-reamed groups in the trial we surveyed. Of the 510 patients allocated to a reamed intervention, five received a non-reamed procedure, whereas of the 498 patients allocated to a non-reamed intervention, 40 received a reamed procedure (P < 0.0001). These findings show the large potential for differential crossovers in a conventional randomised controlled trial. Procedural crossovers initiated by surgeons are more common when surgeons have limited experience with a procedure than when they have more extensive experience.1618 Except for the unlikely event that exactly the same number of participating surgeons have expertise in the experimental and control procedures (and both groups are allocated to perform an equal number of procedures A and B), there is a potential for differential crossover in the two arms.

Validity of surgical expertise based randomised controlled trials

In the surgical expertise based randomised controlled trial, patients are randomised to different surgeons with expertise in the relevant intervention. The first advantage of the expertise based randomised controlled trial is that surgeons will perform only the procedure in which they have expertise, avoiding the problem of differential expertise.

As in the conventional randomised controlled trial, surgeons in the expertise based randomised controlled trial will be unblinded. However, in the expertise based randomised controlled trial surgeons are likely to be subconsciously biased toward the procedure in which they have expertise. Consequently, the likelihood of differential procedural performance, cointerventions, data collection, and outcome assessment decreases. A third advantage of the expertise based randomised controlled trial is that procedural crossovers are less likely to occur because surgeons are doing the procedures with which they are most comfortable.1618

Applicability of expertise based randomised controlled trials

Enrolling a large number of centres and surgeons will enhance the applicability of both expertise based and conventional randomised controlled trials. Surgical randomised controlled trials, which often include only a few surgeons at a single centre, frequently neglect this issue.19 20 It is, however, encouraging that many of the expertise based randomised controlled trials performed when this was the only viable design have recruited many centres and included large numbers of surgeons.79 If this capacity for multicentre trials with large surgeon samples can be extended to expertise based randomised controlled trials when both designs are possible, this will enhance their applicability.

If an expertise based randomised controlled trial shows that one procedure is superior to another, it does not follow that all surgeons with expertise in the less effective procedure and little or no experience in the more effective procedure can expect their patients to have better outcomes if they immediately start performing the superior procedure. Rather, if these surgeons acquire the same skill set and expertise as the surgeons who participated in the randomised controlled trial, they can expect their patients to have improved outcomes when they switch procedures.

The applicability of the results of a surgical randomised controlled trial further relates to whether a trial is an explanatory trial that uses only surgeons with advanced expertise in ideal clinical settings or if it is a pragmatic trial that uses surgeons with at least basic competence in routine clinical practice settings. Both conventional and expertise based randomised controlled trials can be explanatory or pragmatic trials (table 2).

Table 2

Conditions for pragmatic and explanatory randomised controlled trials using conventional and expertise based methods

View this table:

Feasibility

Randomised controlled trials done in the past few decades show that conventional surgical randomised controlled trials are feasible. However, surgical research is still dominated by observational studies,19 21 and general surgery interventions are half as likely to be based on the results of randomised controlled trials as medical interventions.22 23 To investigate the feasibility of the expertise based randomised controlled trial in areas in which both designs are an option we searched three electronic bibliographic databases: MEDLINE (1966 to September Week 2 2003), EBM Reviews—Cochrane Central Register of Controlled Trials (issue 2, 2003), and EMBASE (1980 to 2003 Week 38). Complete listings of search strategies are available from the authors.

This search identified 162 citations, and two researchers independently reviewed each citation to determine if the report was a surgical expertise based randomised controlled trial. We contacted one of the trial authors in cases of disagreements. We identified five surgical expertise based randomised controlled trials that were conducted in an area in which both designs were a potential option.2427 Table 3 summaries the characteristics of these expertise based randomised controlled trials, showing that such trials are feasible in both emergency and elective surgery.

Table 3

Surgical expertise based randomised controlled trials

View this table:

Surgical expertise based randomised controlled trials may be more feasible than conventional randomised controlled trials. Surgeons may be more willing to participate in an expertise based randomised controlled trial because they have to perform only the procedure for which they have developed expertise. Furthermore, surgeons do not have to do a minimum number of operations with the unfamiliar intervention before participating in the trial. This is likely to appeal to both surgeons and investigators and could prevent delays in starting trials.

The feasibility of expertise based randomised controlled trials will be enhanced when an intermediary physician randomises patients before they are seen by the participating surgeon. For example, in trials of patients admitted to accident and emergency, a surgical resident or an emergency physician could randomise the patient. Expertise based randomised controlled trials may also work well in group practices. Some cardiac surgeons, for instance, have group practices and accept patients for surgery without assignment to a specific surgeon. Patients at such practices could be randomised into an expertise based randomised controlled trial when they are accepted for surgery. Greater practical experience with the expertise based design is needed to evaluate the effectiveness of these approaches to recruitment.

Enrolling patients into an expertise based randomised controlled trial may be more challenging after a patient has seen a specific surgeon because randomising such a patient to another surgeon may be awkward. Even under these circumstances, an expertise based randomised controlled trial may prove feasible. For instance, in the largest randomised trial of treatment for subarachnoid haemorrhage, in which 2143 patients were randomised to neurosurgical clipping versus endovascular coiling,29 patients were seen first by neurosurgeons, who then randomised them to neurosurgery or endovascular coiling by an interventional radiologist.

A potential challenge to undertaking an expertise based randomised controlled trial is that it is highly desirable to have surgeons from both treatment groups at all participating hospitals. This avoids the possible negative impact on recruitment resulting from patients having to travel to other hospitals and the potential confounding related to variations in non-surgical care in the different hospitals. If, in an expertise based randomised controlled trial, it is not possible to have surgeons from both treatment groups at all participating hospitals, the effect of centre is likely to vary depending on the nature of the procedure. The centre may not have an important effect for outpatient procedures, but the effect may be large for major inpatient procedures. For trials of major procedures, trialists will have to weigh the potential influence of differential expertise bias, surgeon unblinding, and differential crossover versus centre effect when deciding on the optimal trial design. The need to have two groups of surgeons on call presents a further challenge to undertaking expertise based randomised controlled trials of 24 hour acute surgical care.

A surgical expertise based randomised controlled trial must ensure satisfactory competence among the surgeons doing each procedure. Strategies to achieve this goal will include selecting qualified surgeons who have attained a specified level of post training experience, who fulfil requirements established by professional guidelines, or who have documented their expertise is at the plateau of the learning curve.

Ethics

Although the medical community accepts conventional surgical randomised controlled trials as ethical, some surgeons may have ethical problems with enrolling patients in a trial when they know they may have to do a procedure with which they feel inexperienced.10 11 This problem does not arise in expertise based randomised controlled trials because surgeons perform only the procedures in which they have established expertise.

The consent process for expertise based randomised controlled trials can inform patients that, regardless of the procedure to which they are allocated, a surgeon with specific expertise will do the assigned intervention. Although rarely acknowledged, this is not the case for most conventional surgical randomised controlled trials. Obtaining consent for the reamed versus non-reamed trial in which we conducted our survey might have been problematic had patients been informed that they might be randomised to a procedure in which their surgeon was both inexperienced and sceptical of its effectiveness.

Applicability to non-surgical areas

The issues of validity, applicability, feasibility, and the ethics of expertise based randomised controlled trials relative to conventional randomised controlled trials are also relevant to the evaluation of interventions in many other fields including rehabilitation, behaviour modification, physiotherapy, chiropractic, radiation oncology, occupational therapy, and education. The issues are relevant to any area in which the skill set that a clinician requires to perform the experimental and control interventions varies importantly. The issues become increasingly germane when clinicians administering the interventions cannot be blinded.

Summary points

Questions remain about the use of randomised controlled trials to evaluate non-pharmacological interventions such as surgery

An alternative is to use expertise based randomised controlled trials, in which participants are randomised to clinicians with expertise in intervention A or intervention B

Interventions are performed only by clinicians with expertise in the procedure, which reduces both bias and ethical concerns

Expertise based randomised controlled trials may have greater applicability and feasibility than conventional trials

Acknowledgments

We thank the study to prospectively evaluate reamed intramedullary nails in tibial shaft fractures (SPRINT) investigators for completing our survey and providing data related to their trial. We also thank D Altman, I Boutron, M Gent, C Meinert, K Schulz, and W Van der Linden for helpful comments and Neera Bhatnagar for the electronic database search.

Footnotes

  • Contributors and sources PJD originated the idea for this paper and brought together all the authors to formulate and debate the points included in the text. PJD, MB, SDW, and GHG conducted the survey presented in this paper, PJD undertook the data analysis, and all authors contributed to the data interpretation. PJD wrote the first draft of the paper and all authors provided critical revisions to the manuscript. All authors have expertise in randomised controlled trial methodology and most haveled large international randomised controlled trials, including surgical randomised controlled trials. PJD will act as the guarantor

  • Funding PJD is supported by a Canadian Institutes of Health Research senior research fellowship award, MB holds a Canada research chair in surgical outcomes, DJC holds a Canada research chair, SY holds an endowed chair of the Heart and Stroke Foundation of Ontario and is a senior scientist of the Canadian Institues of Health Research, GRN holds a Canadian research chair from the Social Sciences and Humanities Research Council. VMM is a Mayo Foundation scholar

  • Competing interests None declared

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.
  24. 24.
  25. 25.
  26. 26.
  27. 27.
  28. 28.
  29. 29.