Predictors of mortality and cost among surgical patients requiring rapid response team activation
=================================================================================================

* Alexandre Tran
* Shannon M. Fernando
* Daniel I. McIsaac
* Bram Rochwerg
* Garrick Mok
* Andrew J.E. Seely
* Dalibor Kubelik
* Kenji Inaba
* Dennis Y. Kim
* Peter M. Reardon
* Jennifer Shen
* Peter Tanuseputro
* Kednapa Thavorn
* Kwadwo Kyeremanteng

## Abstract

**Background:** Prior studies of rapid response team (RRT) implementation for surgical patients have demonstrated mixed results with respect to reductions in poor outcomes. The aim of this study was to identify predictors of in-hospital mortality and hospital costs among surgical inpatients requiring RRT activation.

**Methods:** We analyzed data prospectively collected from May 2012 to May 2016 at The Ottawa Hospital. We included patients who were at least 18 years of age, who were admitted to hospital, who received either preoperative or postoperative care, and and who required RRT activation. We created a multivariable logistic regression model to describe mortality predictors and a multivariable generalized linear model to describe cost predictors.

**Results:** We included 1507 patients. The in-hospital mortality rate was 15.9%. The patient-related factors most strongly associated with mortality included an Elixhauser Comorbidity Index score of 20 or higher (odds ratio [OR] 3.60, 95% confidence interval [CI] 1.96–6.60) and care designations excluding admission to the intensive care unit and cardiopulmonary resuscitation (OR 3.52, 95% CI 2.25–5.52). The strongest surgical predictors included neurosurgical admission (OR 2.09, 95% CI 1.17–3.75), emergent surgery (OR 2.04, 95% CI 1.37–3.03) and occurrence of 2 or more operations (OR 1.73, 95% CI 1.21–2.46). Among RRT factors, occurrence of 2 or more RRT assessments (OR 2.01, 95% CI 1.44–2.80) conferred the highest mortality. Increased cost was strongly associated with admitting service, multiple surgeries, multiple RRT assessments and medical comorbidity.

**Conclusion:** RRT activation among surgical inpatients identifies a population at high risk of death. We identified several predictors of mortality and cost, which represent opportunities for future quality improvement and patient safety initiatives.

Many hospitals have adopted rapid response teams (RRTs), often comprising multidisciplinary critical care providers including nurses, respiratory therapists and physicians, to provide urgent critical care expertise when activated.1 The aim of these teams is to intervene when patients admitted to hospital have abnormal vital signs and biochemical abnormalities in the hope of preventing further deterioration.2 Reasons for RRT activations typically include hypotension, respiratory distress, arrhythmias, altered level of consciousness, or concern of the clinical ward team of impending deterioration.

A recent systematic review evaluating RRTs concluded that implementation of RRTs was associated with a reduction in cardiac arrests among medical ward patients.3 However, prior studies of RRT implementation for surgical patients have demonstrated mixed results with regard to reduction of critical events such as cardiac arrests, intensive care unit (ICU) admissions and in-hospital mortality.4–7 Potential reasons for this variability between medical and surgical patients include attitude differences among physicians, attitude differences among nurses and fundamental differences in disease pathophysiology.5–10

Despite these critical differences, studies specifically examining the surgical population are limited. When institutional protocols for activation are inefficient or not implemented with input from front-line staff, studies have demonstrated noncompliance rates surpassing 75% even when activation criteria are met, and median times to activation as high as 16 hours.11 Interventions tailored to the individual needs of each institution have been shown to be more effective in terms of promoting safety culture and improving patient outcomes.12,13 To identify the most vulnerable patients in this surgical subpopulation who may potentially benefit from tailored RRT care and follow-up, we sought to identifiy specific predictors of poor outcome. The objective of this study was to identify the predictors of both in-hospital mortality and hospital costs among surgical inpatients requiring RRT activation.

## Methods

### Cohort

This is a retrospective analysis of a prospectively collected patient cohort from 2 hospitals in The Ottawa Hospital network, a tertiary care hospital network in Ottawa, Canada. We included all patients who were admitted to the hospitals between May 2012 and May 2016 and met the following inclusion criteria: aged 18 years or older, admitted to a surgical service at either of the study sites, required RRT activation at any time during the index hospital admission, and received either preoperative or postoperative care. Patients were included regardless of their level of care designation. The 3 levels are as follows: active medical treatment including ICU admission and cardiopulmonary resuscitation (CPR), active medical treatment including ICU admission without CPR, and active medical treatment excluding ICU admission or CPR. Patients with cardiac arrests were excluded as the institutional “code blue” response team responds to such patients. In the event of an RRT activation that escalates to a code blue for cardiac arrest, the patient is tended to by both the RRT and the code blue response team; such patients were included in this cohort. Ethics approval was obtained from the Ottawa Health Science Network Research Ethics Board.

### Setting and data collection

The Ottawa Hospital is a 1163-bed health care network that handles 160 000 emergency visits, 50 000 hospital admissions, 2500 ICU admissions and 35 000 surgical cases per year. The hospital network serves as the regional referral centre for oncology, vascular surgery, neurosurgery and thoracic surgery. Each campus has a tertiary-level, mixed medical–surgical ICU with a 28-bed capacity. The campuses use the same RRT activation criteria and institutional database although each caters to a different patient population mix.

The RRT at the Ottawa Hospital comprises a critical care physician, a critical care registered nurse with at least 5 years of critical care experience, and a respiratory therapist; the team can be activated to provide urgent medical care at the bedside. Nighttime RRT activations are managed by a resident physician on the critical care service with support from the registered nurse and respiratory therapist. Institutional criteria for RRT activation, presented in Appendix 1 (available at [canjsurg.ca/017319-a1](http://canjsurg.ca/017319-a1)), include airway concerns, decreased level of consciousness, tachypnea, hypoxia, hypotension, arrhythmias and general worry of impending deterioration.12 Health care providers are also encouraged to activate the RRT if they have concerns that are not outlined in the institutional criteria. Data for the RRT database are prospectively gathered by the responding registered nurse using a standardized data collection form at the time of patient assessment.

We linked prospectively collected data sets from the ICU RRT database with the institutional database, termed the Ottawa Hospital Data Warehouse (OHDW). The OHDW is a central repository of clinical and health administrative data that includes the patient registration system, clinical data repository, case-costing system and patient discharge abstracts. Linkage is performed using unique anonymized patient-level identifiers. The Ottawa Hospital’s health administrative database has been widely used in previous health services research.1,13–15 Data quality assessments were performed during development and are executed routinely as new data are added to ensure completeness and accuracy.

### Predictors and outcome variables

Predictors of interest were selected on the basis of their clinical importance as guided by clinical expert opinion and the published literature. These included patient demographic characteristics, such as age and sex; medical comorbidity, defined using the Elixhauser Comorbidity Index;16 medical fitness at the time of surgery, defined using the American Society of Anesthesiologists (ASA) classification system;17 surgical characteristics, such as admitting service, occurrence of elective or emergent surgery and occurrence of multiple operations; and RRT activation characteristics, such as reason for activation (grouped by organ system) and occurrence of multiple activations. The Elixhauser Comorbidity Index score was extracted from the institutional database; it was calculated on the basis of the *International Statistical Classification of Diseases and Related Health Problems, 10th revision* diagnosis codes in administrative data and has been previously validated for the prediction of hospital costs and in-hospital mortality.16 Previous work has demonstrated its superior discriminative ability over the Charlson Comorbidity Index score or modified Frailty Index score for the prediction of adverse events after orthopedic surgery.18 The ASA class and surgical procedure data were extracted directly from the operative record of the first operation.

Data points for the RRT database are collected at the time of RRT assessment by the responding critical care nurse, while data points for the health administrative database are entered at the time of admission and discharge by trained clerical staff. For continuous variables, we investigated their functional forms and assessed for potential transformation with restricted cubic splines, fit with 3 knots. If the transformation improved the model fit, we categorized the continuous variable at the spline infection points. This minimizes inappropriate assumptions of linearity and arbitrary categorization thresholds that result in data waste and statistical inefficiency.19 The remaining predictors were treated as categorical variables.

The dependent variables of interest were in-hospital mortality and total hospital admission costs. Total costs were determined from indirect costs (overhead operational fees) and direct costs (salaries and materials not including physician remuneration). The Ottawa Hospital uses standardized case-costing methodology developed by the Ontario Case Costing Initiative20 and based on the Canadian Institute for Health Information Management Information System guidelines.14 This case-costing algorithm, used in previous health services literature,14,15,21 calculates total hospital costs on a hospital-specific basis using standardized methods and includes both direct and indirect costs standardized to 2018 Canadian dollars.

### Statistical analysis

For descriptive statistics, categorical variables are presented as counts and percentages, and continuous variables are presented as medians with interquartile ranges (IQRs) to account for nonnormal distributions. Variables with normal distributions are presented as means with standard deviations (SDs). To identify predictors of in-hospital mortality, a multivariable logistic regression model was created on the basis of a priori selection of clinically important variables, including patient demographics, medical comorbidity, surgical characteristics and RRT activation characteristics, with associations presented as unadjusted and adjusted odds ratios (ORs) with 95% confidence intervals (CIs). We adhered to an event per variable ratio greater than 10,22 and we intentionally avoided selection by univariate testing of association to minimize potential overfitting.19 We used the C-statistic to characterize goodness of fit and corrected for optimism using 1000 bootstrap resamples for internal validation. Unadjusted associations are presented separately in Appendix 1. We used a multivariable generalized linear model (GLM) with a γ distribution to describe predictors of hospital costs, as this method accounts for highly skewed data without the need for transformation.23–25 We similarly used a priori selection of clinically important variables and present associations as adjusted cost ratios (CRs). All statistical analyses were performed using SAS version 9.4 (SAS Institute) and SPSS version 24.0 (IBM) software. We used *p* < 0.05 as the threshold for statistical significance for all analyses.

## Results

Between May 2012 and May 2016, 1507 adult surgical inpatients met our eligibility criteria. The characteristics of the cohort are presented in Table 1. The median age was 71 (IQR 60–81) years, and 50.9% of patients were male. The median Elixhauser Comorbidity Index score was 4 (IQR 0–10), and the median ASA score was 3 (IQR 3–4). The most common admitting services were general surgery (23.2%) and orthopedics (29.0%). There were 898 (59.6%) surgeries classified as emergent, and 300 (19.9%) patients underwent 2 or more operations. The most common reasons for RRT activation were arrhythmias (23.0%), respiratory distress (18.0%) and decreased level of consciousness (17.3%).

View this table:
[Table 1](http://canjsurg.ca/content/63/6/E598/T1)

Table 1 
Patient characteristics

Patient outcomes are presented in Table 2. There were 318 (21.1%) patients requiring ICU admission, and the overall in-hospital mortality rate was 15.9%. Among patients requiring RRT activation and ICU admission, the median hospital length of stay was 22.5 (IQR 11–41) days.

View this table:
[Table 2](http://canjsurg.ca/content/63/6/E598/T2)

Table 2 
Clinical outcomes

The results of the multivariable regression for predictors of in-hospital mortality among surgical patients requiring RRT activation are presented in Table 3. The patient factors most strongly associated with increased in-hospital mortality included an Elixhauser Comorbidity Index score of 1–19 (OR 1.89, 95% CI 1.28–2.77), an Elixhauser Comorbidity Index score of 20 or higher (OR 3.60, 95% CI 1.96–6.60), increasing ASA class (OR 1.39, 95% CI 1.07–1.80) and designation of care excluding ICU admission and CPR (OR 3.52, 95% CI 2.25–5.52). The strongest surgical predictors included admission to a neurosurgical service (OR 2.09, 95% CI 1.17–3.75), occurrence of emergent surgery (OR 2.04, 95% CI 1.37–3.03) and occurrence of 2 or more operations (OR 1.73, 95% CI 1.21–2.46). Among RRT factors, occurrence of 2 or more RRT assessments (OR 2.01, 95% CI 1.44–2.80) or activation for respiratory distress (OR 1.95, 95% CI 1.24–3.07) conferred the highest associated odds of mortality. The naïve C-statistic was 0.78 and remained as such when corrected for optimism with bootstrap validation over 1000 repeated samples.

View this table:
[Table 3](http://canjsurg.ca/content/63/6/E598/T3)

Table 3 
Predictors of mortality

The total hospital costs and cost allocation breakdown are presented in Table 4, indexed to 2018 Canadian dollars. The median total cost was $30 051 (IQR $17 496–$56 274), primarily driven by $22 440 (IQR $13 171–$41 890) in direct costs. The top 10% of high resource users had a mean cost of $199 567 (SD $106 380). The total cost for the cohort was $76 484 573 and the cost per survivor was $60 367 for 1267 survivors. Nursing costs and operating room costs accounted for most of the total hospital costs.

View this table:
[Table 4](http://canjsurg.ca/content/63/6/E598/T4)

Table 4 
Resource utilization and cost allocation

The results of GLM modelling of costs are presented in Table 5. Patient factors most strongly associated with increasing cost included male sex (CR 1.10, 95% CI 1.02–1.18), an Elixhauser Comorbidity Index score of 20 or higher (CR 1.39, 95% CI 1.19–1.62) and increasing ASA class. In particular, an ASA class 5 level of preoperative medical fitness conveyed a 3.55-fold increase (95% CI 2.26–5.60) in expected hospital admission costs. Patients with a level of care designation excluding ICU admission and CPR had an associated reduction in total costs (CR 0.84, 95% CI 0.75–0.95). Among admitting services, trauma team care was associated with higher cost (CR 2.19, 95% CI 1.57–3.05). The occurrence of multiple operations was associated with a 2.44-fold increase (95% CI 2.23–2.67) in costs. Among RRT factors, occurrence of 2 or more RRT assessments (CR 1.54, 95% CI 1.42–1.68) and activation for respiratory distress (CR 1.29, 95% CI 1.16–1.44) conferred the highest associated increase in costs.

View this table:
[Table 5](http://canjsurg.ca/content/63/6/E598/T5)

Table 5 
Predictors of hospital admission cost

## Discussion

In this observational study of surgical inpatients requiring RRT assistance, we examined predictors of mortality and hospital admission costs. In this vulnerable population, approximately 1 in 6 patients died in hospital. The variables associated with the highest mortality included patient factors (increasing ASA class, higher Elixhauser Comorbidity Index score, level of care designation excluding ICU admission and CPR), surgical factors (admission to neurosurgical service, occurrence of emergent operation, occurrence of multiple operations) and RRT activation factors (activation for respiratory concerns and occurrence of multiple activations). Important predictors of hospital cost included admitting surgical service, occurrence of multiple surgeries, occurrence of multiple RRT assessments, increasing ASA class and higher Elixhauser Comorbidity Index score.

These findings reflect a vulnerable population with high rates of morbidity and mortality and are consistent with the results of previous studies demonstrating increased odds of death and adverse events among postoperative patients with high ASA scores,26 as well as among all patients (including nonsurgical patients) with recurrent deteriorations.27 The neurosurgical population had the highest adjusted mortality among patients for all the admitting surgical subspecialties; these findings are consistent with this patient group’s previously demonstrated mortality and cost burden.28 Lastly, the high mortality rate associated with patients designated for care levels excluding ICU admission and CPR suggests that the priorities of RRT activation for this group should include improving symptom management and clarifying futility of life support, both of which are highly valuable.

With regard to resource utilization, we found that approximately 75% of total cost was driven by direct costs, naturally related to high patient-specific care needs, such as high-acuity nursing and operating room costs. Repeated utilization of RRT team assessments and surgical team resources was associated with higher costs, as expected. Similarly, a level of care designation precluding aggressive interventions, such as ICU admission or CPR, naturally reduced total costs. The importance of patient factors is particularly noteworthy. This is illustrated by the striking impact of poor preoperative medical fitness (ASA class 5) and high comorbidity burden (Elixhauser Comorbidity Index score ≥ 20), independent of age.

Although postoperative adverse events (AEs) are known to be major drivers of morbidity and hospital admission costs in the surgical patient population,15,29 there is little actual variability in incidence between the hospitals with the highest (“worst”) and lowest (“best”) AE rates.30 However, the mortality rates for these AE rates, otherwise known as failure to rescue, at the worst hospitals were often 2 times higher than the rates at the best hospitals.31 This phenomenon describes institutional differences in safety culture and quality of response to AEs; i.e., the ability to halt the potential transition from initial serious AE to eventually substantial morbidity and death. The RRT plays a crucial role in the “rescue” of these complex postoperative patients with acute deterioration by means of early identification, intervention and potential triage to higher-acuity care.32 Similarly, some institutions use surgical step-down units (SDUs) to provide an intermediate level of care between the ICU and the general ward for patients at higher risk, including higher nurse-to-patient ratios or staffing by nurses with advanced training. There is evidence suggesting that SDUs provide a safe postoperative setting for such patients, resulting in fewer complications, improved pain control and reduced need for full intensive care.33

Previously validated approaches to patient safety improvement, notably the Comprehensive Unit-based Safety Program, emphasize identifying patient safety mistakes, using evidence-based interventions, placing value on the wisdom of front-line staff and empowering those staff to become actively involved in safety initiatives.34–36 Similarly, the specific RRT framework is known to vary widely by institution and is often determined by local needs and resources.3 This study therefore serves to identify several high-risk factors for mortality and increased cost among surgical patients requiring RRT activation and highlights a group of particularly vulnerable, high-cost patients. Although most of the variables identified are not directly modifiable, they present an opportunity to inform institutional protocols targeting high-risk patients who may benefit from earlier RRT involvement and closer follow-up care.

Although our cost analysis highlights predictors of high cost among RRT users, it does not specifically allow for an evaluation of resource utilization in an RRT-based care model compared with a non-RRT–based care model. In other words, the cost savings associated with the reduction of ICU utilization and unanticipated mortality remains unclear. These challenges are similarly faced by the existing literature evaluating the effectiveness of RRT care models, which is based primarily on nonrandomized and observational studies, focusing on clinical outcomes.32

Currently, our institution advocates for and supports liberal use of RRT activation, including dedicated foundational teaching to surgical residents regarding their reasons for activation and potential benefits. In addition, all residents completing critical care rotations will participate in the overnight RRT with oversight from an attending physician. However, we have yet to implement mandatory institutional interventions (such as ICU admission or palliative care discussion) aside from at least 1 follow-up assessment by an attending intensivist.

Future opportunities for further research and understanding of actionable items in this vulnerable population should focus on implementation factors of RRTs, including differences between physician- and nurse-initiated activations and the impact of delayed activation.

Our study has several strengths, including systematic record linkage across a number of high-quality data sets from a multihospital health care network, robust and modern methodologic approaches for prediction modelling and validation, and granular resource utilization data.

### Limitations

Limitations of this study relate primarily to the observational nature of the data set. The identification of several factors highly associated with increased mortality and total hospital costs does not necessarily imply causation or even that modifying such factors would result in improved outcomes. Although our institution does not use formal SDUs, there are slight variations in nurse-to-patient ratios and nursing protocols between and within surgical wards, which we were unable to account for. Although previous literature has noted fewer ICU admissions and decreased length of stay with SDU implementation,33 to our knowledge there is no existing evidence to characterize the association between SDU implementation and RRT utilization.

In addition, we were unable to comment on the cases of RRT activation noncompliance where activation criteria were met but did not result in RRT activation and assessment. Instead, these findings should serve as grounds for hypothesis generation and should prompt institutions to reflect on the specific vulnerabilities of their own RRT system when dealing with surgical patients.

Despite robust attempts to minimize confounding and overfitting, we were unable to control for all factors, including the degree of physiologic derangement as well as day–night differences, especially given the difference in training between the attending intensivist during the day and the resident physician at night.

Although 2 large hospitals contributed to this patient cohort, they exist within a single health care network and are influenced by similar local ICU and surgical practices that may limit generalizability. In addition, while each hospital uses the same RRT activation criteria and institutional database, we were unable to account for the specific patient population mix at each site with separate analyses. Lastly, we were unable to capture data regarding the nature and timing of interventions provided by the RRT, including resuscitation and medications.

## Conclusion

We found that RRT activation among surgical inpatients highlights a particularly vulnerable population at high risk of mortality and should lead to a higher level of scrutiny and diligence. We described several predictors of increased mortality and high resource utilization that may serve to more efficiently identify and tailor RRT care and follow-up as part of future patient safety initiatives.

## Footnotes

*   **Funding:** This study was funded by a grant from the Canadian Institutes of Health Research (MOP-142237) and the Bruyère Research Center for Individualized Health, which is supported by the Bruyère Foundation. This study was also supported by ICES, which is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care (MOHLTC). The opinions, results and conclusions reported in this article are those of the authors and are independent from the funding sources. No endorsement by ICES or the Ontario MOHLTC is intentded or should be inferred.

*   **Competing interests:** Andrew Seely holds patents related to multiorgan variability analysis and is the founder and CEO of Therapeutic Monitoring Systems Inc. No other competing interests were declared.

*   **Contributors:** A. Tran, S. Fernando, D. Kim and D. McIsaac conceived the study. A. Tran, S. Fernando, D. McIsaac, G. Mok, D. Kim and P. Reardon acquired the data, which A. Tran, S. Fernando, D. McIsaac, B. Rochwerg, A. Seely, D. Kubelik, K. Inaba, D. Kim, P. Reardon, J. Shen, P. Tanuseputro, K. Thavorn and K. Kyeremanteng analyzed. A. Tran, S. Fernando, D. McIsaac, A. Seely, D. Kim, P. Reardon, J. Shen, K. Thavorn and K. Kyeremanteng wrote the manuscript, which A. Tran, S. Fernando, D. McIsaac, B. Rochwerg, G. Mok, A. Seely, D. Kubelik, K. Inaba, D. Kim, P. Reardon, P. Tanuseputro and K. Thavorn critically revised. All authors gave final approval of the version to be published.

*   Accepted February 19, 2020.

## References

1.  Fernando SM, Reardon PM, Bagshaw SM, et al.Impact of nighttime rapid response team activation on outcomes of hospitalized patients with acute deterioration.Crit Care 2018;22:67
    
    

2.  Davies O, DeVita MA, Ayinla R, et al.Barriers to activation of the rapid response system.Resuscitation 2014;85:1557–61.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/j.resuscitation.2014.07.013&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=25108061&link_type=MED&atom=%2Fcjs%2F63%2F6%2FE598.atom) 

3.  Winters BD, Weaver SJ, Pfoh ER, et al.Rapid-response systems as a patient safety strategy.Ann Intern Med 2013;158:417–25.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.7326/0003-4819-158-5-201303051-00009&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=23460099&link_type=MED&atom=%2Fcjs%2F63%2F6%2FE598.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000316058600009&link_type=ISI) 

4.  Sarani B, Palilonis E, Sonnad S, et al.Clinical emergencies and outcomes in patients admitted to a surgical versus medical service.Resuscitation 2011;82:415–8.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/j.resuscitation.2010.12.005&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=21242020&link_type=MED&atom=%2Fcjs%2F63%2F6%2FE598.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000289555800010&link_type=ISI) 

5.  Moroseos T, Bidwell K, Rui L, et al.Rapid response team implementation on a burn surgery/acute care ward.J Burn Care Res 2014;35:21–7.
    
    

6.  Jones D, Bates S, Warrillow S, et al.Effect of an education programme on the utilization of a medical emergency in a teaching hospital.Intern Med J 2006;36:231–6.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=16640740&link_type=MED&atom=%2Fcjs%2F63%2F6%2FE598.atom) 

7.  Jones D, Opdam H, Egi Moritoki E, et al.Long-term effect of a medical emergency team on mortality in a teaching hospital.Resuscitation 2007;74:235–41.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/j.resuscitation.2006.12.007&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=17367913&link_type=MED&atom=%2Fcjs%2F63%2F6%2FE598.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000251286100007&link_type=ISI) 

8.  Sarani B, Sonnad S, Bergey M, et al.Resident and RN perceptions of the impact of a medical emergency team on education and patient safety in an academic center.Crit Care Med 2009;37:3091–6.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1097/CCM.0b013e3181b09027&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=19938331&link_type=MED&atom=%2Fcjs%2F63%2F6%2FE598.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000272509800013&link_type=ISI) 

9.  Baxter AD, Cardinal D, Hooper J, et al.Medical emergency teams at the Ottawa Hospital: the first two years.Can J Anaesth 2008;55:223
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1007/BF03021506&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=18378967&link_type=MED&atom=%2Fcjs%2F63%2F6%2FE598.atom) 

10. Oake N, Taljaard M, van Walraven C, et al.The effect of hospital acquired *Clostridium difficile* infection on in-hospital mortality.Arch Intern Med 2010;170:1804–10.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1001/archinternmed.2010.405&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=21059973&link_type=MED&atom=%2Fcjs%2F63%2F6%2FE598.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000283937600012&link_type=ISI) 

11. Downey AW, Quach JL, Haase M, et al.Characteristics and outcomes of patients receiving medical emergency team review for acute change in conscious state or arrhythmias.Crit Care Med 2008;36:477–81.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1097/01.CCM.0000300277.41113.46&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=18091535&link_type=MED&atom=%2Fcjs%2F63%2F6%2FE598.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000252794000015&link_type=ISI) 

12. Vigorito MC, McNicoll L, Adams L, et al.Improving safety culture results in Rhode Island ICUs: lessons learned from the development of action-oriented plans.Jt Comm J Qual Patient Saf 2011;37:509–14.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=22132663&link_type=MED&atom=%2Fcjs%2F63%2F6%2FE598.atom) 

13. Khan RM, Ajuaid M, Ageel H, et al.Introducing the Comprehensive Unit-Based Safety Program for mechanically ventilated patients in Saudi Arabian Intensive Care Units.Ann Thorac Med 2017;12:11–6.
    
    

14. Ronksley PE, McKay JA, Kobewka DM, et al.Patterns of health care use in a high-cost inpatient population in Ottawa, Ontario: a retrospective observational study.CMAJ Open 2015;3:E111–8.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiY21ham8iO3M6NToicmVzaWQiO3M6ODoiMy8xL0UxMTEiO3M6NDoiYXRvbSI7czoxOToiL2Nqcy82My82L0U1OTguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

15. Reardon PM, Fernando SM, van Katwyk S, et al.Characteristics, outcomes and cost patterns of high-cost patients in the intensive care unit.Crit Care Res Pract 2018doi:10.1155/2018/5452683
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1155/2018/5452683&link_type=DOI) 

16. Elixhauser A, Steiner C, Harris DR, et al.Comorbidity measures for use with administrative data.Med Care 1998;36:8–27.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1097/00005650-199801000-00004&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=9431328&link_type=MED&atom=%2Fcjs%2F63%2F6%2FE598.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000071181500004&link_type=ISI) 

17. Daabiss M.American Society of Anaesthesiologists physical status classification.Indian J Anaesth 2011;55:111–5.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.4103/0019-5049.79879&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=21712864&link_type=MED&atom=%2Fcjs%2F63%2F6%2FE598.atom) 

18. Ondeck NT, Bovonratwet P, Ibe IK, et al.Discriminative ability for adverse outcomes after surgical management of hip fractures: a comparison of the Charlson Comorbidity Index, Elixhauser Comorbidity Measure, and Modified Frailty Index.J Orthop Trauma 2018;32:231–7.
    
    

19. Steyerberg EW (2009) Clinical prediction models: a practical approach to development, validation and updating (Springer, Cham (Switzerland)).
    
    

20. Ontario Ministry of Health and Long-Term Care (2017) Ontario Case Costing Initiative (Ontario Ministry of Health and Long-Term Care, Toronto) Available: [www.ontario.ca/data/ontario-case-costing-initiative-occi/](http://www.ontario.ca/data/ontario-case-costing-initiative-occi/). accessed 2019 Oct 15.
    
    

21. McIsaac DI, Abdulla K, Yang H, et al.Association of delay of urgent or emergency surgery with mortality and use of health care resources: a propensity score–matched observational cohort study.CMAJ 2017;189:E905–12.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMToiMTg5LzI3L0U5MDUiO3M6NDoiYXRvbSI7czoxOToiL2Nqcy82My82L0U1OTguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

22. Peduzzi P, Concato J, Kemper E, et al.A simulation study of the number of events per variable in logistic regression analysis.J Clin Epidemiol 1996;49:1373–9.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/S0895-4356(96)00236-3&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=8970487&link_type=MED&atom=%2Fcjs%2F63%2F6%2FE598.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=A1996VY92400008&link_type=ISI) 

23. Moran JL, Solomon PJ, Peisach AR, et al.New models for old questions: generalized linear models for cost prediction.J Eval Clin Pract 2007;13:381–9.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1111/j.1365-2753.2006.00711.x&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=17518803&link_type=MED&atom=%2Fcjs%2F63%2F6%2FE598.atom) 

24. Mihaylova B, Briggs A, O’Hagan A, et al.Review of statistical methods for analysing healthcare resources and costs.Health Econ 2011;20:897–916.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1002/hec.1653&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=20799344&link_type=MED&atom=%2Fcjs%2F63%2F6%2FE598.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000293062400005&link_type=ISI) 

25. Austin PC, Ghali WA, Tu JV.A comparison of several regression models for analysing cost of CABG surgery.Stat Med 2003;22:2799–815.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1002/sim.1442&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=12939787&link_type=MED&atom=%2Fcjs%2F63%2F6%2FE598.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000184846000009&link_type=ISI) 

26. Hopkins TJ, Raghunathan K, Barbeito A, et al.Associations between ASA physical status and postoperative mortality at 48 h: a contemporary dataset analysis compared to a historical cohort.Perioper Med (Lond) 2016;5:29
    
    

27. Fernando SM, Reardon PM, Scales DC, et al.Prevalence, risk factors and clinical consequences of recurrent activation of a rapid response team: a multicenter observational study.J Intensive Care Med 2019;34:782–9.
    
    

28. Fernando SM, Reardon PM, Dowlatshahi D, et al.Outcomes and costs of patients admitted to ICU due to spontaneous intracranial hemorrhage.Crit Care Med 2018;46:e395–403.
    
    

29. Dimick JB, Chen SL, Taheri PA, et al.Hospital costs associated with surgical complications: a report from the private sector national surgical quality improvement program.J Am Coll Surg 2004;199:531–7.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/j.jamcollsurg.2004.05.276&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=15454134&link_type=MED&atom=%2Fcjs%2F63%2F6%2FE598.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000224368400002&link_type=ISI) 

30. Ghaferi AA, Birkmeyer JD, Dimick JB.Variation in hospital mortality associated with inpatient surgery.N Engl J Med 2009;361:1368–75.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1056/NEJMsa0903048&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=19797283&link_type=MED&atom=%2Fcjs%2F63%2F6%2FE598.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000270300000011&link_type=ISI) 

31. Ghaferi AA, Dimick JB.Understanding failure to rescue and improving safety culture.Ann Surg 2015;261:839–40.
    
    

32. Barocas DA, Kulhalli CS, Ehrenfeld JM, et al.Benchmarking the use of a rapid response team by surgical services at a tertiary care hospital.J Am Coll Surg 2014;218:66–72.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcjs%2F63%2F6%2FE598.atom) 

33. Prin M, Wunsch H.The role of stepdown beds in hospital care.Am J Respir Crit Care Med 2014;190:1210–6.
    
    

34. Khan RM, Ajuaid M, Ageel H, et al.Introducing the Comprehensive Unit-Based Safety Program for mechanically ventilated patients in Saudi Arabian intensive care units.Ann Thorac Med 2017;12:11–6.
    
    

35. Timmel J, Kent PS, Holzmueller CG, et al.Impact of the comprehensive unit-based safety program on safety culture in a surgical inpatient unit.Jt Comm J Qual Patient Saf 2010;36:252–60.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=20564886&link_type=MED&atom=%2Fcjs%2F63%2F6%2FE598.atom) 

36. Weaver SJ, Lofthus J, Sawyer M, et al.A collaborative learning network approach to improvement: the CUSP learning network.Jt Comm J Qual Patient Saf 2015;41:147–59.