Abstract
Background: For decades, trauma registries have been the primary source of data for resource allocation, quality improvement efforts and hypothesis-generating research in trauma care. Surprisingly, the quality and completion of data in these registries has rarely been reported. In preparation for a research program on population-based epidemiology of severe trauma, we evaluated the Calgary component of the Alberta Trauma Registry (ATR).
Methods: We identified the ATR records of all adult trauma patients (aged ≥16 yr) admitted to hospitals in the Calgary Health Region (CRH) between April 1, 2001 and March 31, 2002 with severe injuries (Injury Severity Score ≥12). From these registry data, we randomly selected 100 patient records, and we compared 14 fields, sampling parameters from prehospital care to discharge, with information from the hospital chart.
Results: Only 9 of 100 records were found to be incomplete. Of these, none had more than 1 field incomplete. Of the approximately 1400 data fields assessed, only 9 were missing data, resulting in a 99% (1391/1400) completion rate. Of 100 records, 22 were found to have inaccurate data; of these, 18 had 1 incorrect field, 2 had 2 incorrect fields and 2 had 3 incorrect fields. Overall, the ATR is 98% accurate.
Conclusions: The Calgary component of the ATR can be considered accurate and complete. Some of its inaccuracy is attributable to a change in the way time to operating room was recorded. Data from all other fields collected in a standard manner can continue to be used with confidence for administrative and research purposes.
The first computer-based trauma registry was developed in 1969 at Cook County Hospital in Chicago.1,2 In the following years, numerous registries have been created, all defined as a “systematic collection of clearly defined set of health and demographic data for patients with specific health characteristics, held in a central database for a predefined purpose.”3 All registries serve 1 primary function: to assess and improve patient care.2 In addition, registries have been used for many secondary gains, for example, to inform injury prevention initiatives, help in the design of new trauma programs and help investigators understand the medical, economic and social impacts of trauma.2 Recently, there has been a shift in the use of registries to develop and test research hypotheses.
Unfortunately, uncertainty in registry records poses a problem to epidemiological research.4 Registry data can be misleading, because most registries do not include out-of-hospital deaths, essential prehospital information or postdischarge deaths.5 Also, many errors can arise in data coding and in retrieving data from the hospital records.5 Despite these concerns and despite the tremendous importance of trauma registries, accuracy and completeness of these registries has not been widely reported in the literature. In this study, we evaluated the Alberta Trauma Registry (ATR), a publicly accessible data source6 used extensively for quality improvement, administrative purposes and population- based research.
Methods
The ATR
By convention, all 23 trauma centres in Canada submitting data to the National Trauma Registry (including the ATR) have selected Injury Severity Scores (ISSs) of 12 or greater as the principal criterion for registry inclusion. Injuries of this magnitude are considered to be severe and require significant resource deployment. The ATR prospectively collects data on patients sustaining severe injury (ISS ≥ 12) in the province.
Patients entered into the Calgary component of the ATR (mostly through analysis of trauma admissions to the region’s single tertiary care trauma referral centre) with addresses in the borders of the Calgary Health Region are identified; detailed data on the circumstances of their injuries, the extent and nature of their injuries and their outcomes are prospectively extracted from their hospital records. Patients with severe injuries who are inadvertently triaged to and treated at either of the 2 nontrauma referral hospitals in the region are captured through active screening by registry data analysts. This includes evaluating all diagnoses of injury discharges at the 2 nontrauma referral hospitals. Based on diagnosis, if patients are suspected to have an ISS greater or equal to 12, the hospital records are reviewed and patients who meet the ATR criteria are entered into the database. This is a routine activity to ensure all patients meeting ATR criteria are evaluated. This strategy of trauma patient identification, which relies on a streamlined integrated regional trauma referral system to a single trauma centre, limited alternatives for definitive complex trauma care; active surveillance of other major hospitals makes it highly unlikely that significant numbers of severely injured patients in the health region are not identified. All injury admissions in the trauma centre (from daily admission report), all emergency department deaths (from an emergency department logbook) and all trauma team activations (from trauma team activation forms) are entered into the digital innovation collector database. As a final cross-check, discharge lists from health records based on all patients with a mechanism of injury code that falls within inclusion criteria ranges of the ATR are received monthly. This is crossreferenced with the defined population to ensure that all appropriate patients are reviewed.
Study design
The Ethical Review of Clinical Trials and Health Research at the University of Calgary assessed and approved our protocol.
Accuracy was defined as “the extent to which the results of a method agree with an independent external criterion.”4 Accordingly, this study compared specific fields in the ATR to corresponding fields in the patients’ hospital records to determine the concordance of the 2 sources. As detailed by Stone,4 the goal was to measure the level of errors that occur due to incorrect transcription or other recording problems. There was a need to determine the completeness aspect of validity. This was accomplished by measuring the percentage of records that have entries in all the fields indicated. Records with missing data in any of the predetermined fields were considered incomplete. Because the definitions of “accuracy” and “completeness” differ with respect to this study design, we treated them as being mutually exclusive.
Data fields
There are approximately 1300 data fields in the ATR; however, most are not accessed by each patient. Each patient circumstance will determine the number of data fields collected. For example, the number of scene interventions, the methods of transportation, the use of peripheral hospitals and the number of surgical interventions will all expand the number of data fields used by each patient. We identified 14 fields entered for all patients representing injury mechanism, severity, aspects of care and outcome. These fields were used as the parameters to assess accuracy and completeness of the ATR (Box 1) and represent information that should be available on all patients.
Parameters used to assess accuracy and completeness of the ATR
Demographic parameters: |
Mechanism of injury |
GCS |
Primary injury “E code” |
Admission parameters: |
Systolic blood pressure |
Heart rate |
GCS on arrival to hospital |
Duration of stay/discharge parameters: |
Initial time to OR |
OR time |
ICU length of stay |
Hospital length of stay |
Death or discharge |
ATR = Alberta Trauma Registry; GCS = Glasgow Coma Scale; OR = operating room; ICU = intensive care unit.
The 2 fields used to assess demographic data were patient’s age and the location where the trauma occurred. The patient’s age was the age at admission, taken from their birth date. The ATR automatically changes the age of the patient by 1 year if a birthday occurs within the time of hospital admissiona and when the patient data are recorded in the ATR. If this occurs, the individual entering the data manually overrides the information and enters the correct age. Consequently, by assessing this field, we can achieve a better estimate of the level of accuracy in manual data input. The location where trauma occurred identifies the city, town or rural area in which the trauma occurred.
Prehospital data fields included the mechanism of injury, Glasgow Coma Scale (GCS) and the International Classification of Diseases (ICD)-9 primary injury “E code.” The mechanism of injury included blunt injury, penetrating injury, burn or other.
For admission data, systolic blood pressure, heart rate and GCS on arrival to hospital were measured. The systolic blood pressure at admission to the emergency department was measured, and a numerical value within 10 mm Hg above or below the hospital record was considered to be accurate. We used the GCS at the emergency department of the original hospital and on arrival to the tertiary trauma referral centre.
We tested the initial time to the operating room, operating room time, intensive care unit (ICU) length of stay, hospital length of stay and recorded whether the patient died or was discharged to assess accuracy and completeness of the ATR with respect to inhospital trauma care and outcome.
We used an excel spreadsheet with 792 patients admitted during the study period (April 1, 2001 to March 31, 2002) for randomization. We analyzed in detail the first 100 records from the resulting randomized list. According to the ATR, 792 patients qualified for the registry over the fiscal year (April 1, 2001 to March 31, 2002); thus a 1 in 8 sample over 2001–2002 gives an approximate sample size of 99 (n = 99).
Results
We evaluated 100 records, each containing 14 fields, for accuracy and completeness. We mutually exclusively assessed 1400 fields for accuracy and then again for completeness.
From the 100 records sampled, 22 hospital records had inaccurate fields. Of these 22 records, 18 had 1 incorrect field, 2 had 2 incorrect fields and 2 had 3 incorrect fields. From the 22 records, we found a total of 28 incorrect fields. From a total of 1400 fields, 1372 were found to be accurate, resulting in a 98% accuracy level (1372). Of the 28 incorrect fields, 19 fields were operating room time (81% accuracy), 2 were GCS (98% accuracy), 5 were ICU length of stay (95% accuracy) and 2 were hospital length of stay (98% accuracy). Overall, the ATR appears to be 98% accurate.
Of the 100 records sampled in this study, only 9 were found to have incomplete fields. None of these 9 records had more than 1 field missing or incomplete. Because there were 100 records sampled each with 14 fields, a total of 1400 fields were sampled for completeness. Of the 1400 data fields assessed, only 9 were missing data, resulting in a (1391/1400 = 0.99) 99% (99.36%) completion rate.
Discussion
Steps have been taken to assess registries. This has been done in part by measuring the completeness of specific fields. Rodenberg assessed the Florida Trauma System statewide database by determining the completeness of records.7 This was done by checking the patient’s age, mechanisms of injury (blunt v. penetrating), initial BP, initial GCS score, initial respiratory rate (RR) and ISS and subsequently determining what percentage of records had entries in all fields.7 Further evidence of validation can be found in literature evaluating cancer registries. Astrom and colleagues8 assessed record validity by comparing the Swedish Cancer Registry, Cause of Death Registry and hospital records. Also in 2001, the validity of a cancer registry measuring the quality of breast cancer care was assessed by comparing registry data to medical records for hospital-based and ambulatory services (6 fields).9 Lastly, Stone created and tested a methodology for validating data in a registry where both completeness and accuracy were tested on the Glasgow Register of Congenital Malformations in a retrospective fashion.4 As observed in the literature, both accuracy and completeness need to be determined for registry data to be used for research purposes. However, there is another inherent problem in assessing and using registries — definitions. Unfortunately, there is a wide range of definitions used, correctly and incorrectly, ambiguously and unambiguously, when describing the validity or quality of registries. Hence, we adopted the definitions used by Arts and others3 for this investigation. On the basis of an extensive literature review, the authors stated that data quality can be defined as “the totality of features and characteristics of a data set that bear on its ability to satisfy the needs that result from the intended use of the data.”3
For our purposes, quality and validity were used synonymously. Inherent in this definition of validity are 2 features, “data accuracy,” the level of similarity between registered data and “the truth” (a gold standard and a hospital record) and “data completeness,” which Arts and colleagues define as “the extent to which all necessary data that could have been registered have actually been registered.” 3 Unfortunately, although the hospital records are considered the gold standard both in this study and others, we are cognizant that hospital records themselves may not have 100% validity. There is, however, no better alternative. In addition, at our single tertiary care trauma referral centre, a standardized history sheet is used to maintain adequate data acquisition on all trauma patients.
From the literature, it is evident that comparing registry records to hospital medical records is an established method for evaluating accuracy. Also, this study measured the percentage of completeness. This method has been used to evaluate the completeness of cancer registries, birth registries and registries on congenital malformations. Moreover, by measuring demographic, prehospital and admission data, as well as the duration of stay and discharge data, we assessed all aspects of the continuum of care (patient at the scene to discharge from the hospital).
One criticism may be the small sample size in this study. Unfortunately, there is no consensus on an adequate sample size. One study used a sample of approximately 5% to evaluate intercollegiate sports injuries.10 Another study used a 1 in 4 sample, giving a total sample size of 105 (n = 105).4 Because the ATR had 792 patients qualify for the registry over the fiscal year (April 1, 2001 to March 31, 2002), a 1 in 8 sample from 2001–2002 gives an approximate sample size of 99.
This study tested the overall usefulness of the ATR by measuring both the accuracy and the completeness of its records. This is the first example of validity assessment in the context of multisystem trauma (a trauma database).
A 99% field completion rate is reassuring. The 9 missing fields are attributable to simple errors in data entry in the ATR. Two of the missing fields were patient’s age, where the patients’ age changed between the time of the trauma and the time of discharge. In this situation, the change in the age needed to be manually recorded. Likely, the requirement for manual entry in this situation resulted in field incompletion.
Although 22 inaccurate records might seem to indicate a low accuracy level, when considered in the context of the total number of fields evaluated, the registry is 98% accurate. There were minimal errors in the input of data and specific caveats that explain these inaccuracies. First, in regard to the “time in theatre” field during the study period, there was a change in the operating room time being measured (procedure start time v. time in theatre) at the beginning of the fiscal year (April 1, 2001). However, due to the similarity of these times, the observed 81% accuracy level of this field may not have tremendous clinical significance. Second, both inaccuracies in the GCS data occurred in cases where the patients were intubated, and the initial GCS took this into account.
Although the ATR had clear validity, most trauma registries have poor validity.7 In fact, Rodenberg observed completion rates between 19.75% and 31.95%, depending on where information came from (level I, II or III facilities) and clearly stated that, because of completion rate deficits, errors would occur when evaluating the trauma system or assessing outcomes. 7 Consequently, we recommend regularinternal reviews of registries for further quality assurance, particularly those registries used for research purposes. This would be of great benefit because knowledge from these registries is used at the provincial and national level in the assessment of health policy and resource planning.
Conclusion
This formalized assessment demonstrates that the ATR is valid and contains high-quality data. The 100% accuracy seen in such fields as mechanism of injury, injury type, diagnosis and discharge status and the 99% completion of screened fields suggests that the ATR can be used with confidence, for quality improvement, administration and research.
Acknowledgements
Thanks to Avik Dey, Heather Karasch and Salimah Javer for their help with this research study.
Footnotes
Presented at the Annual Scientific Meeting of the Trauma Association of Canada, April 1–3, 2004.
Competing interests: None declared.
Contributors: None declared.
- Accepted October 16, 2006.