Summary
Scoping reviews of innovations in health care characterized by large numbers and types of publications present a unique challenge. A novel software application, Synthesis, can efficiently scan the literature to map the evidence and inform practice. We applied Synthesis to the National Surgical Quality Improvement Program (NSQIP), a high-quality database designed to measure risk-adjusted 30-day surgical outcomes for national and international benchmarking. The scoping review describes the breadth of studies in the NSQIP literature. We performed a comprehensive electronic literature search using PubMed, MEDLINE, Web of Knowledge and Scopus to capture all NSQIP articles published between Jan. 1, 2000, and Dec. 31, 2020. Two reviewers independently reviewed articles to determine their relevance using predefined inclusion criteria. We imported references into Synthesis to semiautomate data management. Extracted data included surgical specialty, study type and year of publication. Of the 4661 NSQIP articles included, 3631 (77.9%) were published within the last 5 years. Among NSQIP-related articles, the most common study types were based on outcomes (46.7%) and association (41.7%), and the most common surgical specialties were general surgery and orthopedic surgery, representing 35.7% and 24.0% of the articles, respectively. Synthesis enabled a rapid review of thousands of NSQIP publications. The scoping review provided an overview of the articles in the NSQIP literature and suggested that the NSQIP is increasingly being described in publications of quality and safety in surgery.
Scoping reviews are conducted to identify gaps in current knowledge and to describe opportunities for further research. The magnitude of publications in medicine in the 21st century poses a challenge for literature surveillance. Synthesis is a novel software application that can automate the laborious work of scoping reviews. We sought to show the power of Synthesis as it is applied to quality improvement in surgery.
Surgery is an essential component of health care. About 11% of the overall global burden of disease is treatable by surgery.1 The World Health Organization reported that the crude mortality rate after major surgery is 0.5%–5% and up to 25% of all patients have postoperative complications.2
The American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) is a nationally validated, outcomes-based program with more than 700 participating hospitals from 11 different countries.3 Data in the NSQIP are collected from patient medical charts and include demographic information, medical comorbidities and 30-day morbidity and mortality risk-adjusted outcomes for all major surgical procedures. Data entry is performed by trained qualified personnel using a standard data dictionary and definitions. These data can be risk-adjusted using the ACS surgical risk calculator, a publicly accessible prediction tool that was developed using data from more than 4.3 million operations captured by the NSQIP database. The calculator can predict the likelihood of an outcome or complication based on individual patients’ preoperative data.4,5 Data reported by the NSQIP are also adjusted for case-mix to account for the complexity of operations performed, which allows surgical centres to calibrate their results against each other.3
A growing number of publications are related to the NSQIP. Despite the growing number, there is limited collated information as to the type of research being conducted and the topics of articles being published. Systematic and scoping reviews have reported on specific patient populations, surgical specialties and procedures within the NSQIP literature.6–10 However, little is known about the entire breadth of research being conducted and the types of articles being published.
Synthesis
Synthesis is a Java-based program for literature reference management that was custom developed by D.W.Y.,11 and uses the open-source Apache Lucene’s text search abilities.12
Synthesis is novel because it can identify an entire data set of relevant publications (i.e., NSQIP). Through text-mining algorithms, it can quantify the literature according to user-defined rules. These rules are defined by keywords or phrases, and can be enhanced through Boolean operators such as AND, OR, and NOT, as well as proximity and wild-card modifiers.
Keywords, phrase and derived rules can be searched within an entire data set of imported references in near-real-time similar to the “find” command in commercially available word processors. Synthesis can use a text definition file based on keywords or phrases, Boolean operators, wild cards and proximity searching to tag each reference that meets user-defined criteria.
User-defined rules can be targeted toward specific parts of the publication’s reference, such as the title, abstract, journal name and other available metadata. The full-text publication (PDF) can be used as well (e.g., to search for statistical methods). Synthesis uses an automated computerized approach where the software can iterate through thousands of publications in a consistent manner in a matter of minutes. This allows for rule definitions to be modified and for the literature review to be updated at a future date with minimal effort.
In addition, Synthesis provides standard features for literature review, such as embedded PDF viewing, automatic de-duplication of the references from the various bibliographic databases used and enhanced features such as word clouds and topic clustering (a form of artificial intelligence to automatically detect topics from text documents).
Scoping review
We completed this scoping review based on the framework described by Arksey and O’Malley.13 We identified relevant studies through a query of 4 electronic databases, PubMed, Ovid, MEDLINE, Web of Knowledge and Scopus. We conducted searches on Feb. 11, 2021, and limited them to title and abstract only. We selected relevant studies in the English language published between Jan. 1, 2005, and Dec. 31, 2020. We chose this interval because NSQIP Participant Use Files were available to participants only as of 2005. All studies included “NSQIP OR “National Surgical Quality Improvement Program” in the title or abstract. We imported references into Synthesis and excluded duplicates. We also excluded conferences, commentaries (e.g., letters, corrections, editorials, discussions), books and book chapters, as well as grey literature, given that the number of qualifying publications was adequate to meet the aims of the scoping review. We manually reviewed references that met the screening criteria to eliminate narratives, commentaries or duplicates that were not excluded in the initial screening.
Two reviewers (C.M. and A.D.) independently reviewed the titles and abstracts to determine their relevance to the study. Reviewers were blinded to each others’ selections. Articles were included if they were specific to a statistical analysis of NSQIP data. Disagreements on eligibility were referred to a third reviewer (D.W.Y.) for arbitration.
We imported all articles into Synthesis. We catalogued articles into categories based on words and phrases in the title. We identified words and phrases within titles using Boolean operators, wild cards and proximity searches. These words and phrases were then subjectively grouped into main categories. Statements were created consisting of a string of commonly occurring words, phrases and basic algorithms in an iterative process. We identified broader themes based on the constructed categories, namely surgical specialty and study type. It should be highlighted that articles could be catalogued within more than 1 category. For example, multidisciplinary articles were labelled with each relevant surgical specialty and fell into several categories. Titles for which the algorithm produced no category were labelled as unknown.
We built the initial word cloud for surgical specialty using current procedure terminology codes. We constructed categories of surgical specialties based on surgical specialties common in American and Canadian surgery programs. We defined categories within surgical specialty based on a variety of associated keywords or concepts, mostly related to surgical procedure or anatomy. A list of the main categories and examples of their associated subcategories can be found in Appendix 1, Table 1, available at www.canjsurg.ca/lookup/doi/10.1503/cjs.001722/tab-related-content.
We defined categories of study types using previously published ontology.11 The categories in this study included association, characteristics, estimates, surveillance, risk, utilization, implementation, prediction, methodology and evaluation. We developed additional categories using the previously described approach, including education, economics, outcomes and adverse events. A list of the main categories and examples of their associated subcategories can be found in Appendix 1, Table 2.
We used Synthesis to quantify the frequency of each category within the surgical specialty and study type themes. We identified the total number of NSQIP articles per year between 2005 and 2020 to describe the growth of the NSQIP literature. We created a heat map to visualize and identify patterns of publications of study types by surgical specialty.
Our electronic search produced 15 132 articles. A total of 4661 articles were deemed eligible for this scoping review. A flow diagram can be seen in Figure 1. Interrater reliability for screening titles and abstracts was near perfect, with a Cohen κ of 0.84.
Flow diagram. NSQIP = National Surgical Quality Improvement Program. *Missing abstract, no NSQIP in title or abstract, conference abstracts, erratum, commentary, book or book chapter.
Of 4661 articles, 3631 (77.9%) were published between 2015 and 2020. The growth in NSQIP publications over time is illustrated in Figure 2. The most common surgical specialties were general surgery (n = 1666, 35.7%) and orthopedic surgery (n = 1120, 24.0%). The next most common were surgical oncology (n = 560, 12.0%) and vascular surgery (n = 378, 8.1%). The most common study types were based on outcomes (n = 2179, 46.7%) and association (n = 1944, 41.7%). The proportion of studies by surgical specialty and study type are illustrated in Figure 3.
Distribution of National Surgical Quality Improvement Program (NSQIP) articles by year.
Heat map of distribution of National Surgical Quality Improvement Program (NSQIP) publications by study type and surgical specialty. ANES = anesthesia; CSX = cardiac surgery; ENT = otolaryngology and head and neck surgery; GENSX = general surgery; NEURO = neurosurgery; OBGYN = obstetrics and gynecology; OMF = oro-maxilofacial surgery; ONCO = surgical oncology; ORTHO = orthopedic surgery; PEDS = pediatric surgery; PRSX = plastic surgery; THOR = thoracic surgery; TRANS = transplant surgery; URO = urology; VASC = vascular surgery.
Discussion
This study highlights the usefulness of Synthesis and shows how it can facilitate large scoping reviews by semi-automating data management and analysis. Synthesis quickly and efficiently enabled a scoping review of 4661 relevant NSQIP articles published between 2005 and 2020, an otherwise Herculean task. The topic definitions described in the current study may also be used to perform similar analyses of other surgical databases, such as the National (Nationwide) Inpatient Sample (NIS)14 and National Cancer Database (NCDB).15 The implications of this technology are vast, as it allows investigators the advantage of searching a large database for definitions and cataloguing articles using quantitative text analysis.
We found that more than two-thirds (78%) of NSQIP articles were published within the last 5 years, showing an upward trend and growth in the volume of NSQIP literature. The NSQIP has continued to evolve and similar versions of the database now exist, such as the Trauma Quality Improvement Program and NSQIP-Pediatric. A particular advantage of NSQIP over other databases used to monitor surgical outcomes (such as the NIS or NCDB) is that the NSQIP was specifically designed with the intention to be used for quality improvement and the data are collected from patient medical charts. Data from the NSQIP also comprise all major surgeries, unlike the NCDB, which includes only malignant neoplastic diseases. It is important to consider the purpose behind the creation of a database, as this influences sampling procedures, patient populations and collected variables. Administrative and claims-based databases are limited when used for quality improvement.16
In this scoping review, general surgery and orthopedic surgery were the most published surgical specialties within NSQIP. Other specialties have also been actively involved in quality-improvement initiatives using the NSQIP. Overall, outcomes and associations were the most common study types found in the NSQIP literature. These findings are as expected, since the NSQIP was designed with the intention of exploring associations between different risk factors and adverse events to improve surgical outcomes. We also identified other study types, such as education and economics, showing the unique applications of the NSQIP.
This scoping review describes how the NSQIP has been used across surgical specialties to understand and improve the quality of surgical care, offering insight into the themes of more than 4000 published articles. We also describe a novel approach using text data mining to categorize academic publications. Pre-existing systematic and scoping reviews on the NSQIP have been limited to a narrower focus, such as specific populations, surgical specialties and procedures.6–10 Although this type of review does not capture findings in as much detail as a systematic review, the use of Synthesis allows for interpretation of large volumes of data and has produced a larger review than has generally been feasible allowing for a global picture of the existing NSQIP literature.
In addition, we identified a paucity of articles on implementation, utilization, estimates, education and economics. Although the NSQIP is primarily used to report adverse events and outcomes, future studies should maximize its use by conducting other study types. For example, the NSQIP can be used to evaluate the effects of implementing an intervention or using new techniques or devices, or can help estimate cost-effectiveness.
Despite potential subjectivity, the existing text definitions are easily modifiable. A detailed list of definitions used in this study have been included in Appendix 1, Tables 3 and 4 to indicate our position and any potential bias. Finally, it is important to note that some articles could not be categorized based on title alone. For example, “O’surgery case log data, where art thou?” by Patel and colleagues17 does not identify a surgical specialty or study type and could not be categorized. Therefore, 52 (1.1%) articles remained of unknown surgical specialty.
Conclusion
The current scoping review describes a novel approach using Synthesis, an enhanced software application specialized on text data management and analysis, for categorizing topics of NSQIP articles based on titles of academic publications. We showed how Synthesis can be applied to perform scoping reviews more quickly and efficiently with any desired body of literature.
Footnotes
Presented at the 2021 American College of Surgeons Quality and Safety Virtual Conference, July 2021.
Competing interests: Eric Hyndman is the scientific officer with and owns stock options of Nanostics. Dean Yergens is a cofounder and codeveloper of Synthesis Research, which owns the intellectual property for the Synthesis software application. No other competing interests were declared.
Contributors: All of the authors contributed to the conception and design of the work, drafted the manuscript, revised it critically for important intellectual content, gave final approval of the version to be published and agreed to be accountable for all aspects of the work.
- Accepted November 23, 2022.
This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY-NC-ND 4.0) licence, which permits use, distribution and reproduction in any medium, provided that the original publication is properly cited, the use is noncommercial (i.e., research or educational use), and no modifications or adaptations are made. See: https://creativecommons.org/licenses/by-nc-nd/4.0/
In this issue
Article tools
Related Articles
Cited By...
- No citing articles found.