Prevention of postoperative pancreatic fistula after pancreatectomy: results of a Canadian RAND/UCLA appropriateness expert panel
=================================================================================================================================

* Julie Hallet
* Evangelia Theodosopoulos
* Jad Abou-Khalil
* Kimberley Bertens
* Jean-Sébastien Pelletier
* Maja Segedi
* Jean-François Ouellet
* Jeffrey Barkun
* Natalie Coburn

## Abstract

**Background:** We aimed to define the appropriateness of interventions for the prevention of postoperative pancreatic fistulas (POPF) after pancreatectomy, given the lack of consistent data on this topic.

**Methods:** Using the RAND/UCLA appropriateness method, we assembled an expert panel to rate clinical scenarios for interventions to prevent POPF after pancreaticoduodenectomy (PD) and distal pancreatectomy (DP).

**Results:** The following interventions were rated appropriate: individualized risk prediction for all patients; perioperative pasireotide administration for patients undergoing PD who have a soft pancreatic gland and a pancreatic duct size of less 3 mm and for patients undergoing DP; pancreaticogastrostomy for patients undergoing PD who have a soft pancreatic gland and pancreaticojejunostomy for PD for patients with a pancreatic duct size of 6 mm or greater regardless of pancreatic gland texture; duct-to-mucosa anastomosis for all patients undergoing PD and dunking anastomosis for patients undergoing PD who have a pancreatic duct size of less than 3 mm with a firm pancreatic gland; simple stapled and reinforced stapled transection for all DP; surgical drains for PD and DP in patients with a soft pancreatic gland; and open and minimally invasive surgery for DP and open surgery for PD. The following were rated inappropriate: gastrointestinal anastomosis for stump closure in all DP and omission of surgical drain in PD for patients with a pancreatic duct diameter less than 3 mm and a soft pancreatic gland.

**Conclusion:** The expert panel identified appropriate and inappropriate scenarios for POPF prevention following pancreatectomy, to provide guidance to clinicians. However, the appropriateness of the interventions in the majority of the clinical scenarios was rated as uncertain, demonstrating equipoise.

Pancreatectomy remains a substantial undertaking for patients, health care providers and health systems because of its complexity and the associated morbidity. Despite improvements in perioperative care and reductions in mortality, it remains associated with considerable postoperative symptoms that can persist and affect quality of life.1–3 This is mostly related to high morbidity rates following pancreatectomy occurring in 40%–60% of patients.2–5 Postoperative pancreatic fistulas (POPF), which affect 27% of patients after pancreaticoduodenectomy (PD) and 34% after distal pancreatectomy, are particularly challenging to prevent and manage. They are responsible for most of the postoperative complication burden.6,7

While a large body of literature has addressed techniques and approaches to prevent POPF, the available studies have yielded conflicting results, even when randomized controlled trials were conducted.8–11 The quality of the data and the conclusions are inconsistent, and evidence-informed clinical guidelines on how to prevent POPF are lacking. In the absence of guidelines to convey the importance of surgical technique and perioperative interventions to prevent POPF, there are clinically important variations in the care provided by surgeons who perform pancreatectomy, particularly across centres with different clinical volumes.12 Given that POPF continue to occur frequently and have short- and long-term repercussions for patients, it is important to identify optimal processes of care and the clinical usefulness of interventions to prevent POPF after pancreatectomy to facilitate decision-making, optimize health care resources utilization and improve patient outcomes.

The RAND/UCLA appropriateness method (RAM) is a well-described methodology developed to determine appropriate care in situations where evidence-based guidelines are not feasible.13–19 It has been proven to generate treatment criteria that have face and construct validity as well as excellent predictive ability.20–23 Indeed, RAM results have shown excellent correlation with the results of subsequent randomized controlled trials.24 In the present study, an expert panel was organized to clarify the appropriate use of medical and surgical therapies to prevent clinically relevant POPF following pancreatectomy and to create an expert statement, using the RAM.

## Methods

The RAM involves a comprehensive literature review to determine the evidence regarding interventions, followed by the creation of clinical scenarios to use the identified interventions and then rating of the appropriateness of the interventions for each clinical scenario by a panel of experts.16,19 The structure of RAM is presented in Figure 1. This study was approved by the Research Ethics Board at Sunnybrook Health Sciences Centre.

![Fig. 1](http://canjsurg.ca/https://www.canjsurg.ca/content/cjs/65/2/E135/F1.medium.gif)

[Fig. 1](http://canjsurg.ca/content/65/2/E135/F1)

Fig. 1 
RAND-UCLA methodology process. POPF = postoperative pancreatic fistulas.

The scope of the RAM process and the interventions to be examined were defined by the steering committee using an iterative process. Important domains and specific interventions within those domains were identified. Items were first generated without restrictions and subsequently reduced to retain only the most relevant ones, using Likert scales for priority ranking.25 The final list of interventions included risk assessment for POPF, perioperative somatostatin analogs, anastomosis techniques for PD, stenting for PD, parenchymal transection techniques for distal pancreatectomy, topical adjuncts for PD and DP (fibrin sealants, polyglycolic acid [PGA] mesh), use of surgical drains, and surgical approach of minimally invasive surgery (MIS; robotic or laparoscopic) and open surgery. This process was established to focus the scope of the statement and to limit the number of scenarios so that the scoring process would be reasonable for the panellists while covering the most relevant aspects of POPF prevention.

### Literature review

A literature review was first performed on the following topics identified by the steering committee. Where systematic reviews and meta-analyses existed, they were retained and the literature search was updated. If systematic reviews had not previously been performed, they were conducted (for items 1 and 7).

### Selection of panel of experts

A panel of 13 pancreatic surgeons from 12 hospitals in Canada and the United States was assembled, who represented the pancreatic surgery landscape. The size of the panel was chosen following RAM guidance that there be an adequate number of panellists representing a range of opinions and expertise while keeping the panel manageable and allowing participation of each member in the discussion.19,26 These experts were all fellowship-trained hepato pancreatico-biliary surgeons practising in dedicated hepato-pancreatico-biliary surgery centres in academic and community settings. In Canada, hepato-pancreaticobiliary surgery is mostly regionalized to high-volume centres of expertise, so all of the Canadian panellists were high-volume pancreatic surgeons. They were selected to ensure diversity and inclusion so that the statement they created would be representative of all voices and opinions. Information on the panellists is presented in Table 1.

View this table:
[Table 1](http://canjsurg.ca/content/65/2/E135/T1)

Table 1 
Members of the expert hepato-pancreaticobiliary surgeons panel

### Creation of clinical scenarios

Clinical scenarios were created to evaluate each intervention to prevent POPF from the list generated by the steering committee, through an iterative process. The results of the literature review informed this process. The clinical scenarios were based on 2 main clinical encounters, PD and DP, within which important clinical modifiers were used to create a matrix of clinical scenarios. The modifiers were known risk factors for POPF, including pancreatic duct size, pancreatic gland texture and estimated blood loss.27 The modifiers were selected by the steering committee via an iterative process similar to the one they had previously used to identify the interventions to examine. The matrices were piloted with 3 pancreatic surgeons and revised to a final number of 161 unique scenarios. Definition of key terms were provided to the panellists to ensure uniform understanding and interpretation of the scenarios. The initial scenarios for ratings are presented in Appendix 1 (available at [www.canjsurg.ca/lookup/doi/10.1503/cjs.001520/tab-related-content](http://www.canjsurg.ca/lookup/doi/10.1503/cjs.001520/tab-related-content)).

### Rating of clinical scenarios

The 161 scenarios were circulated to the panellists along with the literature review, instructions and definitions. The panellists were asked to rate the appropriateness of each scenario independently on the basis of their clinical expertise and the available evidence. Appropriateness was defined as “the expected health benefit of an intervention exceeding the expected negative consequences by a wide enough margin that the procedure is worth doing, regardless of cost.”28 It was rated using a 9-point Likert-scale from 1 (highly inappropriate) to 9 (highly appropriate). Ratings were completed using a self-administered electronic survey (SurveyMonkey Inc.) in May 2019 over a 4-week period to allow sufficient time for the panellists to go over the literature review and fill out the survey. Median scores and level of agreement were computed for each scenario as recommended by RAM.19

A second and final round of ratings happened during a moderated in-person meeting in Calgary, Alberta, Canada, in June 2019. The moderator (N.C.) was chosen on the basis of her experience with RAM and her expertise in pancreatic surgery.16,19–31 For the meeting, the panellists were provided with the initial clinical scenarios, their own ratings and the group median scores, as well as the literature review. They discussed the clinical scenarios, the evidence and the rationale for their ratings. If necessary, the scenarios were modified on the basis of the discussions. The panellists then confidentially rated the scenarios a second time for appropriateness. They were not required to reach consensus. All panellists attended the meeting in person.

### Data analysis

Median scores were computed for each scenario, using the panellists’ scores. In the final analysis, procedures were classified as appropriate if the median score was between 7 and 9 with agreement, inappropriate if it was between 1 and 3 with agreement and uncertain if it was between 4 and 6 with agreement. Agreement was defined as 4 or fewer panellists rating outside the 3-point region containing the median (1–3, 4–6, 7–9), and disagreement if 4 or more panellists rated in each extreme 3-point region (1–3 or 7–9). The level of agreement was considered indeterminate if it did not satisfy either of these criteria.19

## Results

### Interrater agreement

Agreement was obtained for 147 of the 161 scenarios. There was no disagreement, but 14 scenarios had an indeterminate level of agreement: using perioperative pasireotide for DP in a patient with a firm pancreatic gland (1) or with blood loss of 400 mL or more (2), using pancreaticojejunostomy for PD in a patient with a pancreatic duct of less than 3 mm and a firm gland (3), dunking pancreatic anastomosis for PD in the presence of a soft gland with a duct of less than 3 mm (4) or a duct of 6 mm or greater (5) or in the presence of a firm gland with a duct of 6 mm or greater (6) or with blood loss of 400 mL or greater (7), using hand-sewn stump closure for DP in patients with a firm gland (8) or blood loss of 400 mL or greater (9), using reinforced stapling transection for DP in patients with a firm gland (10), not using surgical drains for PD in the presence of a soft gland with a duct of 6 mm or greater (11) or a firm gland with a duct less of than 3 mm (12), and placing surgical drains using gravity drainage drains for PD in the presence of a soft gland with a duct of less than 3 mm (13) or a 6 mm or greater (14).

### Appropriateness ratings

Overall, the panel rated 26 scenarios as appropriate and 5 as inappropriate. The appropriateness of the remaining 130 scenarios was rated as uncertain.

Three clinical scenarios addressed the use of predictive tools to estimate the risk of POPF. The panel of experts defined the assessment of individualized risk of POPF as appropriate (median score 8). However, the specific use of the only prospectively externally validated risk score, the Fistula Risk Score, and utilization of its results to direct perioperative management were rated as uncertain (both median scores 6), because of lack of evidence regarding its clinical usefulness and impact on postoperative outcomes.

The surgical approach for pancreatectomy was evaluated in 4 clinical scenarios. Open surgery was determined to be appropriate for both PD and DP (both median scores 8). Minimally invasive surgery was defined as appropriate for DP (median score 7) but as uncertain for PD (median score 5).

The final appropriateness category rating and median scores for perioperative care, surgical technique for PD and surgical technique for DP are presented in Table 2, Table 3 and Table 4, respectively. For perioperative care, the prophylactic use of pasireotide was determined appropriate for patients at higher risk of POPF (those undergoing PD who had both a small duct and soft gland and those undergoing DP who had a soft gland), but uncertain in other circumstances. Prophylactic perioperative octreotide was deemed to be of uncertain appropriateness in all PD and DP scenarios.

View this table:
[Table 2](http://canjsurg.ca/content/65/2/E135/T2)

Table 2 
Median appropriateness scores for perioperative care

View this table:
[Table 3](http://canjsurg.ca/content/65/2/E135/T3)

Table 3 
Median appropriateness scores for surgical techniques for pancreaticoduodenectomy

View this table:
[Table 4](http://canjsurg.ca/content/65/2/E135/T4)

Table 4 
Median appropriateness scores for surgical techniques for distal pancreatectomy

Anastomotic techniques for pancreatic reconstruction in PD that were determined to be appropriate were pancreaticogastrostomy for patients with a soft gland, regardless of the size of the pancreatic duct, and pancreaticojejunostomy for patients with a larger pancreatic duct with either a soft or firm gland. Duct-to-mucosa anastomosis was rated appropriate in all scenarios, while the dunking anastomotic technique was rated as appropriate in the setting of a small pancreatic duct with a firm gland. Pancreatic duct stenting was considered to be of uncertain appropriateness in all PD scenarios. With regard to surgical drains, the omission of drains was considered inappropriate only in patients with the highest risk of POPF (small pancreatic duct with soft gland); conversely, the use of drains was rated as appropriate in those patients. The panellists did not favour either the gravity or suction drainage mechanism, with use rated as uncertain in both of those scenarios.

For DP, both the simple stapling and reinforced stapling techniques were defined as appropriate in all scenarios. Performing a gastrointestinal anastomosis to manage the pancreatic stump was determined to be inappropriate in all DP scenarios. The use of surgical drains was considered appropriate in patients at highest risk of POPF, with a soft gland.

Finally, the use of topical adjuncts, including autologous patches, such as serosal or round ligament patches, fibrin glue and PGA mesh, was considered of uncertain appropriateness in all PD and DP scenarios.

## Discussion

As part of a RAM study, panellists combined available evidence with expert opinion to create guidance in perioperative pancreatectomy care by defining appropriate and inappropriate interventions to prevent POPF after pancreatectomy. Twenty-six scenarios were rated as appropriate, including individualized risk prediction for all patients undergoing pancreatectomy, perioperative pasireotide for high-risk patients, reconstruction with pancreaticogastrostomy for high-risk PD and with pancreaticojejunostomy for lower-risk PD, duct-to-mucosa pancreatic anastomosis for PD, simple or reinforced stapled transection for DP, use of surgical drains for high-risk PD and DP, open surgery for both PD and DP and minimally invasive surgery for DP. Five scenarios were defined as inappropriate: these involved use of gastrointestinal anastomosis to manage the pancreatic stump in DP and omission of surgical drains for high-risk PD.

Lack of consensus and variation in processes of care have been shown to lead to suboptimal care delivery, decision-making and outcomes.32–34 Individual physicians’ preferences are the main driver of unwarranted variation affecting outcomes in medical care.35,36 It is thus critical to identify and support optimal processes of care to guide individual practices, which is associated with better outcomes.37–39

In this study, the appropriateness of processes of care, referring to activities performed by physicians or patients, to prevent POPF following pancreatectomy was assessed. Management and decision-making in perioperative pancreatectomy care can be challenging for surgeons at all career stages. Considerable variation in practice has been demonstrated for pancreatic surgery along with variation in POPF rates at the surgeon and institution levels.3,12 In recent years, surgical quality improvement efforts have advanced greatly, particularly with the implementation of large-scale initiatives such as the American College of Surgeons National Surgical Quality Improvement Program, which provides monitoring of surgical performance. However, such programs cannot improve outcomes without guidance regarding optimal processes of care.40

Despite a large body of literature on the topic, the evidence regarding interventions to reduce POPF for pancreatectomy is conflicting; the existing randomized controlled trials are limited by small sample sizes and single-centre designs and have yielded conflicting results.8–11 Thus, probably owing to the paucity of high-quality evidence and the lack of practice guidelines on the topic, practices in pancreatic surgery have relied mostly on individuals’ or practice groups’ experiences with specific techniques; indeed, preferences for anastomotic techniques are related to years in practice rather than other patient and provider factors.41 However, individual or group experience is not sufficient to standardize practice and does not guarantee better outcomes, as demonstrated by the conflicting literature regarding the experience–outcome relationship in surgery.42–46 Indeed, for most surgical procedures, the surgeon’s experience is not associated with improved outcomes.44

The only practice guideline addressing processes of care for pancreatectomy comes from the International Study Group of Pancreatic Surgery (ISGPS) in 2017.47 Although production of the guideline involved international experts in pancreatic surgery and an extensive literature review, this statement did not use formal consensus methodology, nor did it define how levels of agreement were determined, and as a result its generalizability and validity cannot be assessed. Therefore, there are currently no standardized recommendations to guide perioperative care and surgical technique for pancreatectomy with the goal of preventing POPF. The current RAM exercise fills this important gap in pancreatic surgery.

The RAM is a well-described, validated and standardized methodology that has been proven to provide valid and reliable results regarding optimal care where evidence is lacking or conflicting.13–15,19,21–23 It does not include a rating of the level of evidence supporting each appropriateness rating. It assesses experts’ rating of appropriateness integrating their interpretation of the literature and clinical experience, supported by in-person panel-wide discussions. Furthermore, it provides an anonymous scoring mechanism that favours the expression of every panellist’s opinion and avoids dominance by a few. In contrast to other consensus methods, RAM also allows for disagreements among the panellists, who are not required to reach consensus; rather, the level of agreement for each scenario is highlighted.19 Finally, this process allows for the evaluation of multiple scenarios considering all relevant clinical factors when determining the appropriateness of care. RAM has been used to study various other interventions in health care, including processes of care for gastric cancer, colorectal liver metastases, endarterectomy and hepatectomy.14,16,18,21,22,24,29,30 Thus, RAM was an ideal method to use to define optimal processes of care to prevent POPF following pancreatectomy.

There was remarkable homogeneity in the panellists’ scores. They agreed on the appropriateness or inappropriateness of processes of care for the vast majority of the scenarios. The RAM process permitted the panel’s ratings to reflect grey areas in the evidence as well as the heterogeneity of patients undergoing pancreatectomy and the need for risk assessment and individualization of care taking patient factors into account.4,48 Appropriate scenarios indicate the need for selective use of some interventions for higher-risk patients (those with a small duct with a soft gland for PD, and those with a soft gland for DP): perioperative pasireotide, pancreaticogastrostomy reconstruction and surgical drains. Interventions deemed inappropriate are also of special relevance in light of the agreement among the expert panellists and should be avoided. Finally, interventions of interest were rated to be of uncertain appropriateness in the majority of scenarios. Interestingly, these included a number of scenarios assessing interventions for pancreatectomy with higher blood loss, a variable included in the original Fistula Risk Score. This highlights the limited usefulness of estimated blood loss as a prognostic factor in clinical practice, probably owing to the difficulty of accurately measuring it; an alternative Fistula Risk Score excluding estimated blood loss has been published since we completed this RAM exercise.49 Overall, most of the uncertain scenarios highlight the equipoise in the field of pancreatic surgery and the need to continue investigating perioperative care and surgical techniques for pancreatectomy with higher-quality randomized controlled trials. Such work will require the pancreatic surgery community to collaborate in designing and conducting large multi-institutional trials that focus on patient-centred end points and address the methodology and generalizability limitations of prior studies.

### Limitations

There are limitations to this study. Although an extensive literature review was performed, the appraisal of clinical scenarios relied mostly on retrospective studies or small randomized controlled trials of lower quality. It is in areas where the evidence is limited that the opinion of experts may be most influential, and the RAM provided a structured, validated and systematic way to elicit those opinions. Even though we sought to include experts from various practice settings and with diverse training and knowledge on the panel, which provides generalizability, only experts practising in North America were included. However, variation in care for pancreatectomy and rates of POPF have been reported worldwide and similar views have been documented among international experts.12,47 Finally, the RAM appropriateness ratings are meant to represent care for an average patient in an average health care system. They may not be applicable to all patients and systems in the same way, such that available resources and patient preferences must be taken into consideration in their interpretation and implementation.

## Conclusion

Using RAM, a panel of experts has determined that interventions to prevent POPF following pancreatectomy are appropriate in 26 clinical scenarios and inappropriate in 5 clinical scenarios. These suggestions should contribute to improving the care of patients undergoing pancreatectomy by providing guidance for optimal care, with a focus on higher risk patients with a small pancreatic duct and soft gland. The panellists also found that the majority of the scenarios were of uncertain appropriateness, outlining areas of uncertainty on which to focus future efforts. Ongoing improvements in pancreatectomy outcomes require an emphasis on better level 1 evidence. This expert statement provides valuable information to inform challenging decision-making in pancreatectomy care.

## Acknowledgements

The authors would like to thank Dr. Calvin Law for organizational support for this work.

## Footnotes

*   Presented in part at the 2020 Annual Meeting of the Americas Hepato-Pancreato-Biliary Association, Miami Beach, Fla., Mar. 5–8, 2020.

*   **Competing interests:** None declared.

*   **Contributors:** J. Hallet, J. Abou-Khalil, J. Barkun and N. Coburn designed the study. J. Hallet, E. Theodosopoulos, J. Abou-Khalil, K. Bertens, M. Segedi and J.-F. Ouellet acquired the data, which J. Hallet, K. Bertens, J.-S. Pelletier, M. Segedi, J. Barkun and N. Coburn analyzed. J. Hallet wrote the manuscript, which E. Theodosopoulos, J. Abou-Khalil, K. Bertens, J.-S. Pelletier, M. Segedi, J.-F. Ouellet, J. Barkun and N. Coburn critically revised. All authors gave final approval of the version to be published.

*   **Members of the Canadian Hepato-Pancreato-Biliary Association working group:** Sean Cleary, MD MSc MPH; Kenneth Leslie, MD; John Shaw, MD; Edward Solano, MD; Melanie Tsang, MD MSc; Alice Wei, MD MSc; George Zogopoulos, MD PhD.

*   Accepted August 27, 2020.

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY-NC-ND 4.0) licence, which permits use, distribution and reproduction in any medium, provided that the original publication is properly cited, the use is noncommercial (i.e., research or educational use), and no modifications or adaptations are made. See: [https://creativecommons.org/licenses/by-nc-nd/4.0/](https://creativecommons.org/licenses/by-nc-nd/4.0/)

## References

1.  Tung S, Davis LE, Hallet J, et al. Population-level symptom assessment following pancreaticoduodenectomy for adenocarcinoma. JAMA Surg 2019;154:e193348.
    
    

2.  Gouma DJ, van Geenen RC, van Gulik TM, et al. Rates of complications and death after pancreaticoduodenectomy: risk factors and the impact of hospital volume. Ann Surg 2000;232:786–95.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1097/00000658-200012000-00007&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=11088073&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000165599100012&link_type=ISI) 

3.  McMillan MT, Soi S, Asbun HJ, et al. Risk-adjusted outcomes of clinically relevant pancreatic fistula following pancreatoduodenectomy: a model for performance evaluation. Ann Surg 2016;264: 344–52.
    
    

4.  Ecker BL, McMillan MT, Asbun HJ, et al. Characterization and optimal management of high-risk pancreatic anastomoses during pancreatoduodenectomy. Ann Surg 2018;267:608–16.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 

5.  Ecker BL, McMillan MT, Allegrini V, et al. Risk factors and mitigation strategies for pancreatic fistula after distal pancreatectomy: analysis of 2026 resections from the International, Multi-institutional Distal Pancreatectomy Study Group. Ann Surg 2019;269:143–9.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1097/sla.0000000000002491&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=28857813&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 

6.  McMillan MT, Christein JD, Callery MP, et al. Comparing the burden of pancreatic fistulas after pancreatoduodenectomy and distal pancreatectomy. Surgery 2016;159:1013–22.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/j.surg.2015.10.028&link_type=DOI) 

7.  Miller BC, Christein JD, Behrman SW, et al. Assessing the impact of a fistula after a pancreaticoduodenectomy using the Post-operative Morbidity Index. HPB (Oxford) 2013;15:781–8.
    
    

8.  Gong J, He S, Cheng Y, et al. Fibrin sealants for the prevention of postoperative pancreatic fistula following pancreatic surgery. Cochrane Database Syst Rev 2018;6:CD009621.
    
    

9.  Kilambi R, Singh AN. Duct-to-mucosa versus dunking techniques of pancreaticojejunostomy after pancreaticoduodenectomy: Do we need more trials? A systematic review and meta-analysis with trial sequential analysis. J Surg Oncol 2018;117:928–39.
    
    

10. Hallet J, Zih FSW, Deobald RG, et al. The impact of pancreaticojejunostomy versus pancreaticogastrostomy reconstruction on pancreatic fistula after pancreaticoduodenectomy: meta-analysis of randomized controlled trials. HPB (Oxford) 2015;17:113–22.
    
    

11. Dong Z, Xu J, Wang Z, et al. Stents for the prevention of pancreatic fistula following pancreaticoduodenectomy. Cochrane Database Syst Rev 2016;5: CD008914.
    
    

12. McMillan MT, Malleo G, Bassi C, et al. Pancreatic fistula risk for pancreatoduodenectomy: an international survey of surgeon perception. HPB (Oxford) 2017;19:515–24.
    
    

13. Bilimoria KY, Raval MV, Bentrem DJ, et al. National assessment of melanoma care using formally developed quality indicators. J Clin Oncol 2009;27:5445–51.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiamNvIjtzOjU6InJlc2lkIjtzOjEwOiIyNy8zMi81NDQ1IjtzOjQ6ImF0b20iO3M6MTk6Ii9janMvNjUvMi9FMTM1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

14. Poston GJ, Adam R, Alberts S, et al. OncoSurge: a strategy for improving resectability with curative intent in metastatic colorectal cancer. J Clin Oncol 2005;23:7125–34.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiamNvIjtzOjU6InJlc2lkIjtzOjEwOiIyMy8yOC83MTI1IjtzOjQ6ImF0b20iO3M6MTk6Ii9janMvNjUvMi9FMTM1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

15. Spencer BA, Miller DC, Litwin MS, et al. Variations in quality of care for men with early-stage prostate cancer. J Clin Oncol 2008;26: 3735–42.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiamNvIjtzOjU6InJlc2lkIjtzOjEwOiIyNi8yMi8zNzM1IjtzOjQ6ImF0b20iO3M6MTk6Ii9janMvNjUvMi9FMTM1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

16. Coburn N, Seevaratnam R, Paszat L, et al. Optimal management of gastric cancer: results from an international RAND/UCLA expert panel. Ann Surg 2014;259:102–8.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1097/SLA.0b013e318288dd2b&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=23478525&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 

17. Brar SS, Mahar AL, Helyer LK, et al. Processes of care in the multidisciplinary treatment of gastric cancer. JAMA Surg 2014;149:18–25.
    
    

18. Bennett S, Tinmouth A, McIsaac DI, et al. Ottawa criteria for appropriate transfusions in hepatectomy: using the RAND/UCLA Appropriateness Method. Ann Surg 2018;267:766–74.
    
    

19. Fitch K, Bernstein SJ, Aguilar MD. The RAND/UCLA Appropriateness Method user’s manual. Santa Barbara (CA): RAND Corporation; 2001.
    
    

20. Shekelle PG, Park RE, Kahan JP, et al. Sensitivity and specificity of the RAND/UCLA Appropriateness Method to identify the overuse and underuse of coronary revascularization and hysterectomy. J Clin Epidemiol 2001;54:1004–10.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/S0895-4356(01)00365-1&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=11576811&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000171047800006&link_type=ISI) 

21. Hemingway H, Crook AM, Banerjee S, et al. Hypothetical ratings of coronary angiography appropriateness: Are they associated with actual angiographic findings, mortality, and revascularisation rate? The ACRE study. Heart 2001;85:672–9.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiaGVhcnRqbmwiO3M6NToicmVzaWQiO3M6ODoiODUvNi82NzIiO3M6NDoiYXRvbSI7czoxOToiL2Nqcy82NS8yL0UxMzUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

22. Kravitz RL, Laouri M, Kahan JP, et al. Validity of criteria used for detecting underuse of coronary revascularization. JAMA 1995;274:632–8.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1001/jama.1995.03530080048040&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=7637144&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=A1995RP70200031&link_type=ISI) 

23. Merrick NJ, Fink A, Park RE, et al. Derivation of clinical indications for carotid endarterectomy by an expert panel. Am J Public Health 1987;77:187–90.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.2105/AJPH.77.2.187&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=3799858&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=A1987F631000012&link_type=ISI) 

24. Shekelle PG, Chassin MR, Park RE. Assessing the predictive validity of the RAND/UCLA appropriateness method criteria for performing carotid endarterectomy. Int J Technol Assess Health Care 1998;14:707–27.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1017/S0266462300012022&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=9885461&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000077841800013&link_type=ISI) 

25. American Association for Public Opinion Research. American Association for Public Opinion Research standard definitions: final dispositions of case codes and outcome rates for surveys. Deerfield (IL): American Association for Public Opinion Research; 2011.
    
    

26. Ayanian JZ, Landrum MB, Normand SL, et al. Rating the appropriateness of coronary angiography – Do practicing physicians agree with an expert panel and with each other? N Engl J Med 1998;338: 1896–1904.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1056/NEJM199806253382608&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=9637811&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000074352800008&link_type=ISI) 

27. Callery M, Pratt W, Ken TT, et al. A prospectively validated clinical risk score accurately predicts pancreatic fistula after pancreatoduodenectomy. J Am Coll Surg 2013;216:1–14.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/j.jamcollsurg.2012.09.002&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=23122535&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 

28. Brook RH, Chassin MR, Fink A, et al. A method for the detailed assessment of the appropriateness of medical technologies. Int J Technol Assess Health Care 1986;2:53–63.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1017/S0266462300002774&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=10300718&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 

29. Dixon M, Seevaratnam R, Wirtzfeld D, et al. A RAND/UCLA appropriateness study of the management of familial gastric cancer. Ann Surg Oncol 2013;20:533–41.
    
    

30. Brar SS, Mahar AL, Helyer LK, et al. Processes of care in the multi-disciplinary treatment of gastric cancer: results of a RAND/UCLA expert panel. JAMA Surg. 2014;149:18–25.
    
    

31. Brar S, Law C, McLeod R, et al. Defining surgical quality in gastric cancer: a RAND/UCLA appropriateness study. J Am Coll Surg 2013;217:347–57.e1.
    
    

32. Chassin MR. Explaining geographic variations. The enthusiasm hypothesis. Med Care 1993;31:YS37–44.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=8492584&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=A1993LC93600006&link_type=ISI) 

33. Wennberg J, Gittelsohn A. Variations in medical care among small areas. Sci Am 1982;246:120–34.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1038/scientificamerican0482-120&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=7079718&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=A1982NG07000008&link_type=ISI) 

34. Wennberg JE. Unwarranted variations in healthcare delivery: implications for academic medical centres. BMJ 2002;325:961–4.
    
    [FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMjUvNzM3MC85NjEiO3M6NDoiYXRvbSI7czoxOToiL2Nqcy82NS8yL0UxMzUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

35. Goodman DC. Unwarranted variation in pediatric medical care. Pediatr Clin North Am 2009;56:745–55.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/j.pcl.2009.05.007&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=19660625&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 

36. Mercuri M, Gafni A. Medical practice variations: what the literature tells us (or does not) about what are warranted and unwarranted variations. J Eval Clin Pract 2011;17:671–7.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1111/j.1365-2753.2011.01689.x&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=21501341&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 

37. Bravata DM, Wells CK, Lo AC, et al. Processes of care associated with acute stroke outcomes. Arch Intern Med 2010;170: 804–10.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1001/archinternmed.2010.92&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=20458088&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000277357800010&link_type=ISI) 

38. Guadagnoli E, Shapiro CL, Weeks JC, et al. The quality of care for treatment of early stage breast carcinoma: Is it consistent with national guidelines? Cancer 1998;83:302–9.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1002/(SICI)1097-0142(19980715)83:2<302::AID-CNCR14>3.0.CO;2-X&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=9669813&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000074651200014&link_type=ISI) 

39. Browman GP, Levine MN, Mohide EA, et al. The practice guidelines development cycle: a conceptual tool for practice guidelines development and implementation. J Clin Oncol 1995;13:502–12.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiamNvIjtzOjU6InJlc2lkIjtzOjg6IjEzLzIvNTAyIjtzOjQ6ImF0b20iO3M6MTk6Ii9janMvNjUvMi9FMTM1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

40. Osborne NH, Nicholas LH, Ryan AM, et al. Association of hospital participation in a quality reporting program with surgical outcomes and expenditures for Medicare beneficiaries. JAMA 2015;313:496–504.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1001/jama.2015.25&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=25647205&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 

41. Kennedy GT, McMillan MT, Maggino L, et al. Surgical experience and the practice of pancreatoduodenectomy. Surgery 2017;162:812–22.
    
    

42. Prystowsky JB, Bordage G, Feinglass JM. Patient outcomes for segmental colon resection according to surgeon’s training, certification, and experience. Surgery 2002; 132:663–70 discussion 670–2.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1067/msy.2002.127550&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=12407351&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000179023200030&link_type=ISI) 

43. Sosa JA, Bowman HM, Tielsch JM, et al. The importance of surgeon experience for clinical and economic outcomes from thyroidectomy. Ann Surg 1998;228:320–30.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1097/00000658-199809000-00005&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=9742915&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000075918600008&link_type=ISI) 

44. Waljee JF, Greenfield LJ, Dimick JB, et al. Surgeon age and operative mortality in the United States. Ann Surg 2006;244:353–62.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1097/01.sla.0000234803.11991.6d&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=16926561&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 

45. Karakiewicz PI, Bazinet M, Aprikian AG, et al. Thirty-day mortality rates and cumulative survival after radical retropubic prostatectomy. Urology 1998;52:1041–6.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/S0090-4295(98)00350-1&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=9836552&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000077130400020&link_type=ISI) 

46. O’Neill L, Lanska DJ, Hartz A. Surgeon characteristics associated with mortality and morbidity following carotid endarterectomy. Neurology 2000;55:773–81.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1212/WNL.55.6.773&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=10993995&link_type=MED&atom=%2Fcjs%2F65%2F2%2FE135.atom) 

47. Shrikhande SV, Sivasanker M, Vollmer CM, et al. Pancreatic anastomosis after pancreatoduodenectomy: a position statement by the International Study Group of Pancreatic Surgery (ISGPS). Surgery 2017;161:1221–34.
    
    

48. Ecker BL, McMillan MT, Maggino L, et al. Taking Theory to practice: quality improvement for pancreaticoduodenectomy and development and integration of the Fistula Risk Score. J Am Coll Surg 2018;227:430–438.e431.
    
    

49. Mungroop TH, van Rijssen LB, van Klaveren D, et al. Alternative Fistula Risk Score for Pancreatoduodenectomy (a-FRS): design and international external validation. Ann Surg 2019;269:937–43.