Clinical scenario
After a long night on call, a general surgery resident on a vascular surgery rotation meets a radiology resident at the hospital coffee shop, and they begin to discuss interesting consults from the previous night. She mentions to her radiology colleague that she had been called to the emergency department to see an elderly gentleman with peripheral vascular disease, and she had found an asymptomatic carotid bruit during her clinical examination. After hearing the case, the keen radiology resident suggests ordering an angiogram of the carotid and vertebral arteries, stating that the clinical exam likely indicates the presence of carotid stenosis, heralding the need for surgical intervention. The surgical resident feels that the presence of a carotid bruit is a poor predictor of a surgically important carotid lesion and is unsure whether any investigation or surgical intervention would provide clinical benefit in an asymptomatic patient. Both residents admit that they needed more information, so they set out to find the nearest computer to do a literature search.
Literature search
Before beginning the literature search, the 2 residents refined their search criteria: they were investigating the appropriateness of screening patients who have no focal neurologic findings but are at increased risk for carotid disease. On the basis of an article they had reviewed in medical school from the Rational Clinical Exam series in the Journal of the American Medical Association,1 they agreed that, in the general population, the presence of an asymptomatic carotid bruit has little bearing on the risk of perioperative stroke and is not predictive of the degree of stenosis. Therefore, the clinical question they outlined was, “Which asymptomatic patients would most benefit from carotid artery imaging?”
Once online, the 2 residents searched the MEDLINE database, using the methodology described in the User’s Guide to the Surgical Literature series.2 The search terms “asymptomatic. tw” and “carotid stenosis,” combined with the term “carotid endarterectomy” were used. Their search resulted in 1078 references, in which they identified 6 published randomized controlled trials looking specifically at surgical intervention in patients with asymptomatic carotid arterial disease.3–8 The conclusions from these trials provided evidence regarding the results of surgery versus medical management of asymptomatic carotid stenosis, but they were unclear regarding the specific question of what is the best diagnostic and therapeutic strategy to be used in these patients. The attending surgeon suggested that this question may be addressed with a decision analysis approach. A search of the literature identified a decision analysis that the residents reviewed.9
Background
Clinical decision-making is a process that relies on personal experience, formal training and a clinician’s ability to critically review the relevant literature and determine its applicability to patients. Decision-making becomes more complex when multiple variables influence outcome, when the heterogeneity of populations in the literature makes the interpretation of the outcomes difficult or when the available evidence is discrepant. In these circumstances, formal decision analyses are useful tools that may help clinical decision-making.
A decision analysis is a quantitative method used to assess the relative value of different options to deal with a clinical choice under conditions of uncertainty. This uncertainty may be a clinical scenario that differs from the specific conditions of clinical trials or conflicting results from multiple trials. It has been defined as,
a method of describing complex clinical problems in an explicit fashion, identifying the available courses of action, assessing the probability and value of all possible outcomes, and then making a simple calculation to select the optimal course of action.10
The goal of a decision analysis is to identify the most effective therapeutic or diagnostic approach. It is most valuable in areas where randomized controlled trials are lacking, or where consensus for definitive management has not been reached.
Components of decision analysis
Decision analysis begins with a problem defined as a specific clinical question. The question must have a focused population, intervention and outcome.
The literature review
A decision analysis requires multiple reviews of the literature, each one addressing a specific question in the decision tree. These reviews are used to determine probabilities associated with outcomes and the uncertainties, or confidence intervals, surrounding point estimates of outcomes. The term base case is used in the field of decision analysis to define the best estimate of the probabilities for each variable. It is the value closest to the “truth” in the authors’ estimation. This value can be determined by combining all the known estimates of outcome, therefore conducting a meta-analysis. The term sensitivity analysis may also appear in decision analyses; this simply means that the above analyses are repeated, exchanging base case variables with reasonable alternative values for each variable. These alternative values would be derived from the confidence intervals around point estimates or from the probability distributions of the outcomes. An example illustrating sensitivity analysis is provided in this article.
The decision tree
For a decision to exist, there must be at least 2 options in response to a problem. There are the elements of the problem and all of the options available; these are used to create a decision tree. The decision tree illustrates the complexities of the clinical scenario (i.e., all the possible clinical outcomes or options) and becomes more complex if multiple decisions are required to reach the final outcome. The detailed theory behind the construction of a decision tree is beyond the scope of this article and can be found elsewhere.10 In brief, a decision tree is a standardized method for reporting options: each node or fork in the tree represents a choice made either purposefully or by random chance. Probabilities are then assigned to the branches that grow from each node, along with the utilities associated with each option (Fig. 1). In this context, a utility is the quantitative expression of the value or importance of a given variable. They are typically expressed as a single number between 0 (representing death or worst outcome) and 1 (representing perfect health or best outcome). These values can be derived by polling patients in the similar clinical scenarios to determine their satisfaction with the outcome. When multiplied by the probability of a particular event, the expected utility of the decision node can be determined. In more complex decision analyses, it is necessary to place weights or relative values on different health states to determine the utilities. 10 By following this decision tree, a clinical decision can be subjected to the rigors of the accumulated data on outcomes to determine the appropriate diagnostic or therapeutic path in different circumstances.
The outcomes
All clinically relevant outcomes for the available management options must be defined and analyzed. Uncertainties (or confidence intervals) in the probabilities are identified and substituted for base case probabilities to evaluate the impact that changing these values will have on the final analysis. This is known as a sensitivity analysis because it uses variation in the variables that contribute to the outcomes to determine the robustness of the conclusions derived from the decision analysis. Conclusions are not as strong if they change when subjected to the extreme values.
Clinical application
When applying a decision analysis, a clinician must carefully consider the characteristics of the patient that will affect outcome, based on tacit knowledge of the field, and the individual values that the patient may have with respect to different outcomes; the clinician then uses this framework to apply the decision tree.
Users’ guides: a framework
Using the principles of decision analysis theory stated above and a previously devised methodological framework for use in interpreting the literature,10–13 we can now critically appraise the article chosen by the 2 residents (Box 1).
Framework for interpreting the literature
Are the results valid? |
|
|
|
|
What are the results? |
|
|
|
Will the results help me in caring for my patients? |
|
|
Are the results valid?
Before applying the decision analysis to the patient in question, it is important to determine whether the results of the article are applicable in this clinical scenario. This means examining the strategies and outcomes used by the authors, the methods used to combine the evidence, the source and credibility of utilities chosen and the potential impact of the uncertainties surrounding point estimates.
Were all important strategies and outcomes included?
The first step in appraising a decision analysis is determining whether the authors have studied the field broadly and included all clinically relevant treatment strategies and outcomes. The authors should consider both quantitative gains and qualitative changes that will have an effect on the patient.
The decision analysis by Cinà and Clase9 reviews the literature around diagnosis and therapy for asymptomatic carotid stenosis. The clinical question they address compares medical management to an “investigate and operate” strategy for patients at increased risk of having carotid disease and examines the outcomes of stroke or death. In this article, a detailed literature review is reflected in a clear tabulation of the evidence available for each variable used in the analysis and the range of plausible values around the point estimates to be used in the sensitivity analysis. However, the authors did not include their specific search strategy, which would have made this literature search reproducible. On the basis of strategies used in randomized controlled trials and suggested in guidelines, the authors constructed several models that fit the clinical scenario usually encountered in clinical practice. They compared 5 different investigate-and-operate strategies with the strategy of medical management alone.
The choice of outcomes aids our clinical decision-making process. In this review, main outcomes were the number of patients who would need to undergo duplex ultrasound (DUS) to prevent a stroke and the number of patients who would need to undergo carotid endarterectomy (CE) to prevent a stroke. For the clinical outcomes, they included stroke or death after angiography, stroke or death in the perioperative period and all strokes occurring during medical management or after CE. The authors, however, did not explore other clinical outcomes (e.g., nerve injuries during surgery, hematomas, etc.) and did not use different utilities dependent on the severity of stroke outcome.
Was an explicit and sensible process used?
The methods used to combine the evidence in a decision analysis are predictive of the clinical value of the final result. Providing details of the proper methodological steps is necessary to prevent the introduction of bias. The authors were guided by results from clinical trials and guidelines. 4,14–23 The 2 coauthors undertook a joint review, and through a consensus process that they do not describe, they assigned values for variables used for the base case. They determined values for sensitivity analysis using the confidence intervals of the point estimate of the variables. A table in which they summarized the input variables assists the reader in understanding how the input variables were summarized and calculated (Table 14–6,24). The authors do not provide details of their literature search, which means that potential bias cannot be excluded. A decision analysis is most useful when the review is done systematically and is reproducible by interested readers. The authors then used Data 3.5 software (Tree Age Software, Williamstown, Mass.) to construct the decision trees (Fig. 2).
Were the utilities obtained from credible sources?
The authors used a simple type of decision analysis with 2 possible outcomes (presence or absence of stroke or death). In this model, by convention, the value 1 is assigned to the best outcome and 0 to the worst outcome: the outcome of stroke is less preferable than the alternate of “no stroke,” and the final analysis is represented in an absolute fashion. As a variation on the current reporting technique, the value of the outcome could have been expressed as adjusted quality of life, since a major stroke is different than a perfect health state but also differs from a minor stroke. Therefore, for an individual living 5 years in normal health state or with full return to function after suffering a minor stroke, the number of quality adjusted life years (QALYs) is 5 (utility = 1 multiplied by the number of years). For somebody with a major disabling stroke, the utility value is 0.5 and the QALYs decrease to 2.5 years. QALYs provide a summary that aggregates the outcome with length of life.
Was the potential impact of any uncertainty in the evidence determined?
Uncertainty in the evidence can contribute to miscalculation of both the utilities assigned to outcomes and probabilities at the decision nodes. As in any critical appraisal of clinical literature, the highest quality evidence, by way of randomized controlled trials or meta-analyses of RCTs, should be included preferentially over lesser studies. As well, the authors should review the data in such a way that they acknowledge deficiencies in the evidence and state them explicitly.
In this article, the authors identified various assumptions that were required to complete the analysis. Specifically, an assumption was made that the perioperative risk of stroke was uniform regardless of the burden of carotid disease found. As well, there was no consideration given to the risk of morbidity or mortality from other diseases in this population of vascular patients. Their analysis included data from a thorough review of the literature, and the references list evidence from clinical trials and meta-analyses.4,14–23 Using the confidence intervals of the input variables, the authors assigned intervals that were used to conduct sensitivity analyses. This is an accepted way to test the decision trees that are created, by noting whether outcomes change appreciably within a clinically relevant interval of values.
To explore the robustness of the results of the base case, the authors used a 1-way sensitivity analyses for several clinically important variables. This technique tests the results by using the uncertainties in the base case by modifying 1 variable in the decision tree at a time, until all the variables have been tested. Both the probabilities assigned and the utilities should be subjected to the sensitivity analysis. In this case, the authors used sensitivity and specificity of DUS, number of years of follow-up, annual stroke rate after successful CE, probability of perioperative stroke or death, probability of angiographic stroke or death and baseline probability of carotid stenosis.
The authors have used a decision-analytic approach where the probability of an outcome was fixed, rather than one in which the probability changes with time. The latter is called a Markov model. This would have permitted incorporating stroke rates that changed over time, the competing risk of death from other conditions or the possibility of becoming symptomatic with transient ischemic attack or minor stroke and being offered carotid endarterectomy for symptomatic disease.
Now that the article has been shown to be usable, it is time to interpret the results.
In the baseline analysis, does one strategy result in a clinically important gain for patients? If not, is the result a toss-up?
Ultimately, the goal of a decision analysis is to guide care by determining the harmful and beneficial treatment strategies for a given patient. Authors of decision analyses will often summarize the best strategy for management based on utilities and preferred outcomes. This may be expressed as a quantitative measure, such as the number needed to treat; this is the number of patients that need to be treated with the technique being investigated to prevent an unwanted outcome. Normally, the probability from one branch of the tree is multiplied by the utility to obtain an expected utility. The expected utilities are then summed in each of the competing interventions, and a decision is made based on which intervention provides the highest utility value. Because dichotomous outcomes of stroke and death were used in this decision analysis, the expected utilities are simply 1 or 0.
In this article, Figure 2 has been constructed to depict the number needed to screen and the number needed to treat for the various strategies used for different levels of stenosis in the base population. Cinà and Clase determine that the strategy that manages stenosis of over 70% with angiography, and then carotid endarterectomy after confirmation, is the best. This also allows the reader to assess strategies that may cause harm. Specifically, the harmful strategies were identified as those with an angiographic stroke rate of 1.8% or higher, or a perioperative stroke or death rate of 2.5% or greater. The strategy that called for angiography for lesions above 50% and carotid endarterectomy for lesion with a proven stenosis of over 70% swings the outcomes to the unfavourable side and, therefore, is not recommended. Additionally, the strategy that employed angiography for lesions over 50% and CE for lesions over 50% have a number needed to treat that is much higher than other strategies and is therefore not ideal.
How strong is the evidence in the analysis?
The strength of evidence in a decision analysis determines the accuracy of the probabilities used and can be gauged by the methodological rigour used in the studies from which the probabilities are derived. The strength of evidence cited by Cinà and Clase9 is good: it reflected a systematic review of the numerous large multicentre trials and meta-analyses. It should be remembered that this study was a deterministic analysis based on secondary data collected from the literature rather than a randomized controlled trial that directly compared the 2 approaches under consideration. Deterministic analysis uses indirect information about outcome event rates to aid in decision-making. Randomized controlled trials, or meta-analyses of randomized controlled trials provide the strongest evidence on which clinical decisions can be based.
Could the uncertainty in the evidence change the result?
After interpreting the data in a decision analysis, it is important to test whether the results are vulnerable to variation in the prevalence of disease or probability of key outcomes. Sackett and colleagues10 suggest using clinically relevant changes to test the decision that has lead to the most desirable outcome: a sensitivity analysis must be undertaken.10 If minor changes in a given probability will affect the outcome in a way that is likely important to the patient, then the decision analysis should be reconsidered.
In our article, Cinà and Clase explore this potential vulnerability in the discussion by providing the interpretation of results through a sensitivity analyses. After presenting the analysis for all 5 strategies, the authors concluded that, for the strategy of angiography for lesions above 50% on DUS with CE reserved for lesions above 50% on angiography, the base case predicted the prevention of 19 strokes for the cohort as a whole for patients managed by investigate-and-operate rather than medically. Using sensitivity analyses, they found that the mild fluctuations in sensitivity and specificity of DUS did not have large bearings on outcomes but that the angiographic or perioperative stroke or death rates were “critical to the outcomes with any strategy.”9 In day-to-day clinical practice with the potential for poor patient compliance, screening of patients who are not operative candidates and longer perioperative follow-up, the decision to aggressively screen patients with asymptomatic carotid artery stenosis may not be as clearly beneficial when these algorithms are applied in routine practice.
Will the results help me in caring for my patients?
This last section alludes to an important concept in decision analysis: can the results of the decision analysis, with all of the intrinsic assumptions and evidence-based probabilities, be applied to real clinical scenarios and be predictive and helpful. This is similar to the debate of efficacy versus effectiveness in controlled trials of therapeutic interventions. The artificial environment in which the decision algorithm is constructed must still be focused in reality if it is to be useful. For example, the utilities and probabilities assigned must reflect the values and risks present in the decision makers’ environment. Also, the assumptions made when the decision analysis is constructed should not exclude the target patient population. The generalizability of decision analysis to daily clinical activity also requires the clinician to extract the pertinent characteristics of their patient (or population) to know whether that particular patient falls within the group described in the article.
Do the probability estimates fit my patients’ clinical features?
Every decision analysis should contain a description of the patient population on which it is based, including pertinent positive and negative characteristics. This allows the clinician to match their patient with the one described in the article. This is best done by reviewing the sensitivity analysis in the article and using it as a gauge to determine similarities with the patient in question. The differences will inform the clinician as to how much weight can be placed on the outcomes of the decision analysis.
In this article, the authors clearly describe their target population as asymptomatic individuals with risk factors for carotid disease, such as peripheral vascular disease or carotid bruits. Therefore, the generalizability of this decision analysis is limited to this group of patients.
Do the utilities reflect how my patients will value the decision outcomes?
Although quantitative values are often universal, the effect on quality of life will vary depending on individual values and social context. This paper assigns dichotomous utilities to stroke and death and does not consider the effect that a major or minor stroke would have on quality of life. As stated previously, a more complex decision analysis would further subdivide the outcome “stroke” by acknowledging the different impact that a minor or major stroke would have on daily functioning.
Resolution of the scenario
After working through the above steps, the 2 residents had a much clearer understanding of the approach to asymptomatic carotid stenosis. Together they ordered a Doppler ultrasound of their patients’ carotid arteries, and when the results showed a 50% stenosis, they were confident in their decision to do no further testing. They parted ways, both pleased that their time had been wisely invested. Over the course of the next few years, the 2 residents kept tabs on the patient that spurned their initial interest in decision analyses: he remained asymptomatic and thankful for their sage advice.
Acknowledgements
We acknowledge the authors of the original papers12,13 addressing how to read a decision analysis paper published in the Journal of the American Medical Association.
Footnotes
The Evidence-based Surgery Working Group members include Stuart Archibald, MD;*†‡ Frank Baillie, MD;†‡¶ Mohit Bhandari, MD;†‡¶ Margherita Cadeddu, MD;*†‡ Claudio Cinà, MD;†‡§¶ Sylvie Cornacchi, MSc;†‡ Forough Farrokhyar, PhD;†‡§ Anna Garnett, BA;†‡ Charles H. Goldsmith, PhD;†‡§ Ted Haines, MD;‡§ Robert Hansebout, MD;*†‡ Carolyn Levis, MD;*†‡ Marko Simunovic, MD, MPH;†‡§¶ Ved Tandan, MD;*†‡§ Achilleas Thoma, MD*†‡§
* Department of Surgery, St. Joseph’s Healthcare, the †Department of Surgery, ‡Surgical Outcomes Research Centre and §Department of Clinical Epidemiology and Biostatistics, McMaster University, and ¶Hamilton Health Sciences, Hamilton, Ont.
Competing interests: None declared.
- Accepted November 7, 2006.