Prioritization of patients on scheduled waiting lists: validation of a scoring system for hip and knee arthroplasty
===================================================================================================================

* Barbara L. Conner-Spady
* Gordon Arnett
* John J. McGurran
* Tom W. Noseworthy
* the Steering Committee of the Western Canada Waiting List Project

## Abstract

**Introduction:** The hip and knee replacement priority criteria tool (HKPT) is 1 of 5 tools developed by the Western Canada Waiting List Project for setting priorities among patients awaiting elective procedures. We set out to assess the validity of the HKPT priority criteria score (PCS) and map the maximum acceptable waiting times (MAWTs) for patients to levels of urgency.

**Methods:** Two studies were used to assess convergent and discriminant validity. In study 1, consecutive patients on a waiting list for hip or knee arthroplasty were assessed by orthopedic surgeons from the 4 provinces in Western Canada, using the HKPT and data on patient age, gender, joint site, type of surgery (primary or revision), 2 measures of surgeon-rated patient urgency, and diagnosis. In study 2, 6 patients were videotaped during a consultation interview with the surgeon and were assessed by a group of experts. We measured function with the PCS and the Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC).

**Results:** In study 1, we assessed 394 patients, and in study 2, 19 raters assessed the 6 patients. Correlations between the PCS and other measures of physician-rated urgency were strong, ranging from 0.78 to 0.89. For a subgroup of 60 patients, correlation between the PCS and function as measured with the WOMAC was 0.48, and correlation was greater (0.45–0.56) between items measuring similar constructs (e.g., pain at rest) than those measuring different constructs (0.21–0.40). In study 2, median MAWTs ranged from 4 to 24 weeks for 5 levels of urgency based on PCS percentiles.

**Conclusions:** Results from this study support the validity of the PCS as a measure of surgeon-rated urgency for hip or knee arthroplasty. Evaluative studies are needed to assess the validity and acceptability of the tools and the establishment of MAWTs in clinical practice.

Long waiting lists for joint arthroplasty continue to be a major concern in Canada, with median waiting times from specialist assessment to surgery ranging from approximately 11 to 28 weeks.1–3 Currently, waiting lists are managed by individual orthopedic surgeons, and the order of patients is not rationalized or prioritized within or across physician lists using a transparent and standardized approach.4,5 Most surgeons use broad, ill-defined categories such as urgent, semiurgent and routine to prioritize patients in the queue. However, evidence has shown that order in the queue bears little relation to the severity of patient symptoms such as pain and disability.4

In a recent review of Canadian waiting lists and wait times McDonald and colleagues6 concluded that with some exceptions, “wait lists do not provide a fair and transparent basis for managing patients or allocating resources.” A key recommendation was the development and use of standardized measures, based on clinical urgency and capacity to benefit, to assess patient priority. To address the problems of waiting list management, the Western Canada Waiting List (WCWL) Project developed 5 tools designed to provide an explicit, transparent and fair method for prioritizing patients on waiting lists.7

Priority criteria are currently being used in New Zealand and parts of the United Kingdom, but evidence of their reliability and validity is minimal. 8,9 In this paper we address aspects of construct validity of the hip and knee replacement priority criteria tool (HKPT), designed to rank order patients scheduled for primary or revision hip and knee replacement.10

Traditionally, there are 3 types of validity: construct, content and criterion-related. This concept has evolved into a more unified view with construct validity as the foundation of validity inquiry, a foundation that subsumes construct, content and criterion-related evidence.11–13 Validation on the basis of only a single type of evidence is no longer considered sufficient. Further, the tools themselves are no longer to be validated, but rather the inferences about score meaning or interpretation. Messick13 has defined validity as the process of evaluating the degree to which the empirical evidence and theoretical rationales support interpretations and actions based on a score or other indicator. This broader conceptualization of validity is increasingly used by health services researchers,14 and is adopted as a framework in this paper.

The HKPT comprises 7 criteria developed by a panel of practitioners and researchers.15 A weighted sum results in a priority criteria score (PCS) intended to rank patients in order of urgency. We examine here aspects of the validity of the PCS as a measure of patient urgency. Specifically, our research questions were:

*   What is the congruence between the PCS and other indicators of patient urgency?

*   What are the convergent and discriminant validity characteristics of the PCS in relation to a patient-rated measure of health status?

*   What are physician-rated maximum acceptable waiting times (MAWTs) for different levels of patient urgency?

## Methods

To address these questions, 2 studies were designed. Both have been described in a separate paper on tool development and reliability.10 The first collected data on consecutive patients seen by orthopedic surgeons who completed an assessment of priority criteria for each patient. The second study used data obtained from clinicians who evaluated interviews and videotapes of 6 patients of differing urgency who were on a waiting list for hip and knee replacement during a consultation with the orthopedic surgeon. The findings were analyzed by the 2-tailed *t* test for independent samples. A *p* value of less than 0.05 was considered significant.

### Study 1

From December 1999 to May 2000, data were collected on consecutive patients seen by 17 orthopedic surgeons from across 4 provinces in Western Canada. The HKPT was completed by the surgeon at the patient visit. In addition to measures of patient urgency and health status, data obtained included age, gender, joint site, type of surgery (primary or revision), 2 measures of surgeon-rated patient urgency, and diagnosis. Diagnoses were coded using the ICD-9.

The HKPT comprises 7 criteria, each with 3 to 4 severity levels measuring: pain on motion; pain at rest; ability to walk without significant pain; functional limitations; abnormal findings on physical examination related to the affected joint; potential for progression of the disease documented by radiographic findings; and threat to role and independence. A panel of orthopedic surgeons, researchers and clinicians developed the criteria, which were adapted from the New Zealand clinical priority assessment criteria (CPAC).16 Weights were determined by regression analysis and clinical judgement, and the PCS was calculated by summing the weighted items.

Surgeons were also asked to rate their patients on 2 measures of urgency:

*   A 100-mm visual analogue scale (VAS) with anchors of 0 (not urgent at all) and 100 (extremely urgent: just short of an emergency)

*   A 5-point Likert scale on relative urgency scored from 1 (much less urgent than the average patient) to 5 (much more urgent than the average patient).

A subset of surgeons collected patient data using the Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC),17 a disease-specific questionnaire that consists of 3 subscales measuring: pain, stiffness and physical function. The WOMAC tool is widely used to measure function and symptoms in patients with osteoarthritis of the hip or knee. It was not developed to rank relative urgency for intervention. It consists of 24 items (5 pain, 2 stiffness and 17 physical function) with the degree of severity or difficulty measured on a 5-point Likert scale (0 = none, 1 = mild, 2 = moderate, 3 = severe, 4 = extreme). Subscale scores were derived from the summation of items for each dimension and transformed to a 0–100 scale, with the higher score reflecting worse function.

The congruence between the PCS and other measures of patient urgency were evaluated by correlational analysis. As a guideline to assess the strength of a relationship, an *r* value greater than 0.3 was the minimal correlation considered important.18 It was expected that correlations between surgeon-rated measures of urgency would be strong (≥0.8) and that correlations between surgeon and patient measures would be moderate (at least 0.5). Analysis of convergent and discriminant validity between the PCS and the WOMAC was based on the approach described by Campbell and Fiske.19 We hypothesized that correlations between patient- and surgeon-rated variables measuring similar constructs (e.g., pain at night) would be moderate and positive (convergent validity) and that correlations between variables measuring the same construct would be higher than correlations between variables measuring different constructs (discriminant validity). For example, the correlation between patient- and surgeon-rated pain at night should be higher than the correlation between pain at night and pain on walking. To determine conceptually similar variables, the WOMAC criteria and subscales were matched to the priority scoring criteria.

### Study 2

After informed consent, 6 patients on a waiting list for hip or knee replacement surgery (3 hip, 3 knee), representing a range of urgency, were recruited from participating doctors’ practices and videotaped during a clinical interview with an orthopedic surgeon. The interviews were conducted by 3 surgeons, including one of the authors (G.A.). The surgeon posed questions normally covered in an initial patient history-taking and conducted an examination of the hip or knee. Videotapes also included x-rays of the affected joint and an explanation of the findings. Panel members and their colleagues viewed the videotapes and independently scored each patient using the HKPT. They also rated the urgency for each patient on the 100-mm VAS. Waiting time was defined as the time from consultation with the surgeon to the surgical procedure. MAWT from the physician’s perspective was measured by the question: “In your clinical judgement, what should be the maximum waiting time for this patient?”

Convergent validity was assessed by correlational analysis of the PCS, VAS and MAWT. To establish preliminary MAWTs for different levels of urgency, the distribution of MAWTs was mapped onto 5 groups, based on the PCS expressed in percentiles. Percentiles allow a comparison of relative performance with respect to 2 different variables, in this case, urgency scores and MAWTs.

## Results

### Study 1

The sample comprised 406 patients (240 [59%] female), ranging in age from 17 to 93 years (mean [and standard deviation] 66.8 yr [13.1]). One hundred and ninety-three (48%) were on a waiting list for hip replacement and 213 (52%) for knee replacement. Primary arthroplasty was scheduled in 369 (91%) and revision arthroplasty in 37 (9%). Of patients scheduled for primary arthroplasty, 336 (91%) had a diagnosis of osteoarthritis, 16 (4%) had rheumatoid arthritis, and 17 (5%) had other conditions. The analysis was based on 394 patients for whom complete data for the priority criteria and VAS urgency were available. To determine the effect of volume of patients assessed on urgency ratings, surgeons were grouped into high (> 20) and low (≤20) providers split by the median number of patients assessed. On *t* testing there was no difference in the PCS or VAS urgency between the groups.

The WOMAC was available for a subgroup of 60 patients. There was no significant difference in gender, type of surgical procedure or urgency measure (PCS, VAS, relative urgency) for patients with and without WOMAC data. However, the 60 patients for whom the WOMAC was available were significantly younger than the other patients (mean 61.4 and 67.8 yr respectively).

The mean (and standard deviation [SD]) PCS was 48.39 (20.61) and VAS urgency 59.85 (21.63) (Table 1). Mean WOMAC subscale scores ranged from 58.82 (function) to 64.92 (pain). Analysis using *t* tests for independent samples showed that female patients had a significantly higher mean PCS (50.25) and VAS (61.81) than male patients (45.26 and 56.53 respectively). Patients requiring revision arthroplasty had a significantly higher mean PCS (63.87) and VAS (79.71) than patients scheduled to undergo primary replacement (46.46 and 57.81 respectively).

View this table:
[Table 1](http://canjsurg.ca/content/47/1/39/T1)

Table 1 
Descriptive Statistics for Urgency Measures in Patients Scheduled for Hip Arthroplasty

The multiple correlation (*R*) between the combined 7 priority criteria and the VAS urgency was 0.82, whereas the correlation of the PCS with the 5-point relative urgency scale was 0.78. To compare distributions of the PCS and VAS and identify outliers, PCS and VAS scores were grouped into 5 equal groups based on percentiles (Table 2). For example, patients with a PCS at or below the 20th percentile (i.e., a PCS ≤ 28) would be among the 20% least urgent, whereas those with a PCS above the 80th percentile (PCS > 64) would be among the 20% most urgent. Similarly, patients with a VAS score of 41 or less would be the least urgent and those with a VAS greater than 79 would be the most urgent. Outliers were defined as cases classified in the 20% most urgent group by the VAS and the 20% least urgent by the PCS. Only 1 outlier was identified (PCS 22, VAS 83). This patient was scheduled for revision arthroplasty because of recurrent dislocation and was rated as “much more urgent than the average patient” on the relative urgency scale. No patient was in the 20% most urgent group based on the PCS and in the 20% least urgent based on the VAS.

View this table:
[Table 2](http://canjsurg.ca/content/47/1/39/T2)

Table 2 
Percentile Scores for the Priority Criteria Score (PCS) and Visual Analogue Score (VAS)

Fig. 1 compares the distribution of the PCS and VAS urgency scores across 5 levels of relative urgency compared to the average patient. Only 216 patients were included because the relative urgency question was removed after revision of the HKPT form. Although both scales were positively correlated with relative urgency, the PCS had a greater range of scores than the VAS urgency for all but the middle levels of relative urgency (Fig. 1). Thirty-two (15%) patients were rated as “much more urgent” and 65 (30%) “more urgent” than the average patient. Of those rated as “much more urgent” than the average patient, 27 (84%) were above the 80th percentile for the VAS and 25 (78%) were above the 80th percentile for the PCS.

![FIG. 1](http://canjsurg.ca/https://www.canjsurg.ca/content/cjs/47/1/39/F1.medium.gif)

[FIG. 1](http://canjsurg.ca/content/47/1/39/F1)

FIG. 1 
The distribution of the priority criteria score (white boxes) and visual analogue scale urgency scores (screened boxes) on a scale of 0–100 across 5 levels of relative urgency compared to the average patient. Summary plot is based on the median, quartiles and extreme values. The box represents the interquartile range that contains 50% of values. The whiskers are lines that extend from the box to the highest and lowest values, excluding outliers. The line across the box indicates the median. Circles represent outliers.

Correlations between the PCS and the WOMAC function subscale were 0.48. To assess convergent and discriminant validity, 3 priority criteria were tested against WOMAC criteria and subscales measuring similar constructs: pain on motion, pain at rest and function. Convergent validity was moderate, ranging from 0.45 to 0.56, for similar constructs and was greater than measures comparing different constructs (0.21–0.40) (Table 3).

View this table:
[Table 3](http://canjsurg.ca/content/47/1/39/T3)

Table 3 
Correlation of Comparable WOMAC and Priority Criteria Score Items for a Subgroup of 60 Patients

### Study 2

Nineteen experts (14 orthopedic surgeons, 4 other physicians and 1 physical therapist) rated the 6 patient interviews using the HKPT, the VAS and a MAWT. Correlations between the PCS and VAS urgency rating were high (0.89), whereas MAWTs were moderately correlated with the PCS (–0.67) and the VAS urgency findings (–0.74). When we mapped MAWTs to 5 levels of urgency based on the PCS by grouping patients into quintiles according to the PCS scores (e.g., patients with a PCS > 80 were rated among the 20% most urgent and those with a PCS < 30 were rated among the 20% least urgent), we found that both the mean and median MAWTs decreased with increasing levels of urgency (Table 4). Median MAWTs ranged from 4 weeks (most urgent group) to 24 weeks (least urgent group).

View this table:
[Table 4](http://canjsurg.ca/content/47/1/39/T4)

Table 4 
Maximum Waiting Times in Weeks for Quintiles of Priority Criteria Scores

## Discussion

The HKPT was designed to capture clinical judgement and the complex decision-making process involved in assessing urgency for joint arthroplasty. Although there is no standard against which to test the PCS, one way of measuring convergent validity is to compare the PCS to other methods of physician judgement of urgency (i.e., urgency measured on a VAS and urgency relative to average patients in their practice). In study 1, the correlation between the PCS and relative urgency was strong (0.78) and the 5 priority items were highly correlated with the VAS (*R* = 0.82). Compared with 5 levels of relative urgency, the PCS and VAS discriminated fairly well between extreme levels of urgency, although there was some overlap of the PCS between adjacent levels of relative urgency, particularly in the middle group. For example, the PCS ranged from 7 to 70 for patients rated “about as urgent” as other patients in practice. For cases rated “much more urgent” and “much less urgent,” the range of PCS scores was generally less, with the exception of 1 outlier.

Outliers were defined as patients in the 20% most urgent group as measured by the VAS and also in the 20% least urgent group based on the PCS. An examination of the presence and characteristics of outliers is important to the validity of the PCS as a measure of urgency, particularly where high-stakes decisions are made on the basis of a priority score. For example, in New Zealand, where thresholds for the CPAC determine access to surgery, 20 errors in the conceptualization and measurement of urgency could have significant consequences.

Potential limitations of the VAS urgency and relative urgency measures are the variations in the range of the VAS for different surgeons and in case mix in different surgeons’ practices against which each patient is assessed. Although we found no difference in priority ratings among groups of surgeons based on the number of patients assessed, there is some evidence that perception of priority could be subject to the effects of actual wait time in orthopedic units. In a Swedish study of assessment and prioritization of identical simulated referrals for orthopedic consultation, units with longer waiting times assigned patients a lower priority than other units.21 The level of agreement for indications for joint arthroplasty also varies among physicians. 22–25 Unfortunately, an objective criterion against which to test the PCS as a measure of urgency does not exist. However, the criteria do reflect the basis of physicians’ best judgements of urgency.

In the comparative analysis of the PCS with the WOMAC, which were developed for differing reasons, evidence generally supported the convergent and discriminant validity of the PCS. Similar criteria measuring pain and function in the HKPT and WOMAC were moderately correlated (0.45–0.56) and more strongly related than items measuring different constructs (e.g., pain at rest and pain walking). The overall correlation of the PCS and WOMAC function was moderate (0.48) in contrast to the findings of Derrett and associates,8 who reported low correlations (0.29) between the New Zealand CPAC and a condition-specific tool in patients wait-listed for hip and knee replacement surgery. In addition, they found little relationship between the hip and knee CPAC and patient benefit, as measured by improvement in health-related quality of life.

In study 2, correlations between the PCS and VAS urgency were high, whereas correlations between perceived MAWT and the PCS (–0.67) and VAS (–0.74) were moderately strong. For use in the management of waiting lists, MAWTs need to be established for different levels of urgency. Little work has been done in this area. Clinically reasonable waiting times in Canada have been assessed largely through physician opinion surveys with no allowance for differing levels of urgency. For example, physicians responding to the Fraser Institute surveys estimated clinically reasonable waiting times of 6.5 weeks from the consultation to the surgical procedure.26

Naylor and Williams27 used 4 groups of time frames to determine urgency ratings for a waiting list for hip or knee replacement surgery, 0–4 weeks for the most urgent group and 26–52 weeks for the least urgent. Results from our study provided preliminary median MAWTs for varying levels of urgency, based on the PCS. These ranged from 4 weeks for the most urgent to 24 weeks for the least urgent, highly comparable to the waiting time suggested by Naylor and Williams.

A limitation of our MAWT estimates comes from the use of a limited number of patients, assessed in a simulated clinical situation. It is also important to note that although most of the percentiles for the PCS were similar for both studies, normative scores, such as percentiles, are sample-dependent and should be collected on a representative sample of the population in which the scores will be used. A further limitation is that these are surgeons’, not patients’, views.

The rationale for using priority scoring criteria is to improve fairness, explicitness and transparency, and to provide more consistent access to surgery. However, a criticism has been either weak or no evidence of the validity of techniques to rank patients in order of urgency for various procedures.8,9,20,28–32 Validation is a continuous process of evaluating evidence over time. Accordingly, we acknowledge that the HKPT needs to be tested in a wide range of clinical populations. Our work is continuing on validity testing and the establishment of MAWTs in clinical practice. Prioritizing patients based on clinical urgency rather than simple queuing should result in relative improvement in their clinical outcome after surgery. Evidence of the relationship of the PCS to patient outcomes would provide important support for the validity of the PCS as a prioritization tool. Although evidence suggests that patients who have worse preoperative functional status may have comparatively worse pain and function 1–2 years after arthroplasty,33,34 the impact of waiting on patient outcomes is unclear. 35,36 Further research is needed to understand the relationships and possible interactions between patient urgency, length of waiting time and patient benefit. In addition, the impact of short-term variations in symptoms on patients’ priority rankings is unknown and would be an important aspect to assess for the fair use of the tool. Implementation should involve continuous monitoring and an evaluation of the effects of implementation on patient outcomes, case mix, patterns of resource use, gaming (i.e., playing the system) and impact on the patient–doctor relationship.20,31,37,38

## Conclusions

Our preliminary results show support for the validity of the PCS as a measure of physician-rated urgency. Although only 1 case was identified as an outlier, the implication for implementation of scoring tools is that continuous monitoring and evaluation are needed to determine validity in clinical practice. Results also support the convergent and discriminant validity of the PCS in relation to similar dimensions in the WOMAC.

## Acknowledgements

Members of the Steering Committee of the Western Canada Waiting List Project are as follows: Dr. Tom Noseworthy, Department of Community Health Sciences, University of Calgary (Chair); Dr. Morris L. Barer, Centre for Health Services and Policy Research, and Department of Health Care and Epidemiology, University of British Columbia, Vancouver; Dr. Charlyn Black, Centre for Health Services and Policy Research, University of British Columbia, Vancouver; Ms. Lauren Donnelly, Acute and Emergency Services Branch, Saskatchewan Health, Regina; Dr. David Hadorn, Western Canada Waiting List Project; Dr. Isra Levy, Health Programs, Canadian Medical Association, Ottawa; Mr. Steven Lewis, Access Consulting, Saskatoon; Mr. John McGurran, Western Canada Waiting List Project, and Department of Public Health Sciences, University of Toronto; Dr. Sam Sheps, Department of Health Care and Epidemiology, University of British Columbia, Vancouver; Dr. Mark C. Taylor, Department of Surgery, University of Manitoba, Winnipeg; Mr. Laurie Thompson, Health Services Utilization and Research Commission, Saskatoon; Mr. Darrell Thomson, British Columbia Medical Association, Vancouver; Ms. Barbara Young, Clinical Evaluation Services, Calgary Regional Health Authority, Calgary.

The Western Canada Waiting List Project was supported by a financial contribution from the Health Transition Fund (Health Canada) as Project NA489. The views expressed herein do not necessarily represent the official policy of federal, provincial or territorial governments.

We are indebted to the 19 partner organizations from the 4 western Canadian provinces for their ongoing support throughout the project: British Columbia Medical Association; Capital Health Region (Victoria); Vancouver/Richmond Health Board; British Columbia Ministry of Health; University of British Columbia, Centre for Health Services and Policy Research; Alberta Medical Association; Capital Health Authority (Edmonton); Calgary Regional Health Authority; Alberta Health and Wellness; University of Alberta, Department of Public Health Sciences; Saskatchewan Medical Association; Regina Health District; Saskatoon District Health; Saskatchewan Health; Health Services Utilization and Research Commission; Winnipeg Regional Health Authority; Manitoba Health; Manitoba Centre for Health Policy and Evaluation; Canadian Medical Association.

We wish to acknowledge the members of the hip and knee replacement panel who contributed to the development of the hip and knee replacement surgery priority criteria tool: Dr. Ted Findlay, Dr. Donald Garbuz, Dr. Robert Glasgow, Ms. Karin Greaves, Dr. David Hedden, Dr. Mary Hurlburt, Dr. Bill Johnston, Dr. Stewart McMillan, Dr. Jack Reilly, Dr. Anne Sclater, Dr. Kenneth Skeith and Dr. Lowell van Zuiden. We thank colleagues of the panel members and the patients who participated in the pilot testing and reliability work in Winnipeg, Regina, Saskatoon, Calgary, Edmonton and Vancouver. Finally, we thank Ms. Elaine Dunn and Ms. Anne-Marie Pedersen for their contributions to data collection.

## Footnotes

*   **Reprint requests to:** Dr. Tom W. Noseworthy, Chair, Western Canada Waiting List Project Steering Committee, University of Calgary, Heritage Medical Research Bldg., 3330 Hospital Dr. NW, Calgary AB T2N 4N1; fax 403 210-9378; tnosewor{at}ucalgary.ca

*   **Competing interests:** None declared.

*   Accepted September 5, 2003.

## References

1.  Institute for Clinical Evaluative Services (ICES). Information strategy: urgency rating, waiting list management and patient outcomes monitoring for primary hip/knee joint replacement [technical report]. Toronto: ICES; 2000.
    
    

2.  Alberta Health and Wellness. Hip and knee replacement surgery—volumes and wait times. 2002/2003. Available: [www.health.gov.ab.ca](http://www.health.gov.ab.ca) (accessed 2003 Dec 4).
    
    

3.  British Columbia Ministry of Health. Surgical wait list registry. 2001. Available: [www.swl.hlth.gov.bc.ca/swl](http://www.swl.hlth.gov.bc.ca/swl) (accessed 2003 Aug).
    
    

4.  Williams JI, Llewellyn TH, Arshinoff R, Young N, Naylor CD. The burden of waiting for hip and knee replacements in Ontario. Ontario Hip and Knee Replacement Project Team. J Eval Clin Pract 1997;3:59–68.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1111/j.1365-2753.1997.tb00068.x&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=9238608&link_type=MED&atom=%2Fcjs%2F47%2F1%2F39.atom) 

5.  Coyte PC, Wright JG, Hawker GA, Bombardier C, Dittus RS, Paul JE, et al. Waiting times for knee-replacement surgery in the United States and Ontario. N Engl J Med 1994; 331:1068–71.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1056/NEJM199410203311607&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=8090168&link_type=MED&atom=%2Fcjs%2F47%2F1%2F39.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=A1994PL56000007&link_type=ISI) 

6.  McDonald P, Shortt S, Sanmartin C, Barer M, Lewis S, Sheps S. Waiting lists and waiting times for health care in Canada: More management!! More money?? Ottawa: Health Canada; 1998.
    
    

7.  Noseworthy TW, McGurran JJ, Hadorn DC. Waiting for scheduled services in Canada: development of priority-setting scoring systems. J Eval Clin Pract 2003; 9:23–31.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1046/j.1365-2753.2003.00377.x&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=12558699&link_type=MED&atom=%2Fcjs%2F47%2F1%2F39.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000180655000002&link_type=ISI) 

8.  Derrett S, Devlin N, Hansen P, Herbison P. Prioritizing patients for elective surgery: a prospective study of clinical priority assessment criteria in New Zealand. Int J Technol Assess Health Care 2003;19:91–105.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1017/S0266462303000096&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=12701942&link_type=MED&atom=%2Fcjs%2F47%2F1%2F39.atom) 

9.  Kipping R, Glenn R, McLeod H, Clark J. A review of priority scoring and slot systems for elective surgery. Birmingham (UK): University of Birmingham, Health Services Management Centre; 2002.
    
    

10. Arnett G, Hadorn DC, Steering Committee of the Western Canada Waiting List. Developing priority criteria for hip and knee replacement: results from the Western Canada Waiting List Project. Can J Surg 2003;46:290–6.
    
    

11. 1.  Wainer H, 
    2.  Braun HI
    
    Angoff WH. Validity: an evolving concept. In: Wainer H, Braun HI, editors. Test validity. Hillsdale (NJ): Lawrence Erlbaum; 1988. p. 3–17.
    
    

12. 1.  Linn R
    
    Cronbach LJ. Construct validity after thirty years. In: Linn R, editor. Intelligence: measurement, theory and public policy. Urbana (IL): University of Illinois Press; 1989. p. 147–72.
    
    

13. 1.  Linn R
    
    Messick S. Validity. In: Linn R, editor. Educational measurement. New York: Macmillan; 1989. p. 13–103.
    
    

14. Streiner DL, Norman GR. Health measurement scales. A practical guide to their development and use. 2nd ed. Oxford: Oxford University Press; 1995.
    
    

15. WCWL Western Canada Waiting List Project. From chaos to order — making sense of waiting lists in Canada. Final report of the Western Canada Waiting List Project. March 31, 2001. Available: [www.wcwl.org/library/final\_reports](http://www.wcwl.org/library/final_reports) (accessed 2003 Dec 16)
    
    

16. Hadorn DC, Holmes AC. The New Zealand priority criteria project. Part 1: overview. BMJ 1997;314:131–4.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTQvNzA3NC8xMzEiO3M6NDoiYXRvbSI7czoxNzoiL2Nqcy80Ny8xLzM5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

17. Bellamy N, Buchanan WW, Goldsmith CH, Campbell J, Stitt LW. Validation study of WOMAC: a health status instrument for measuring clinically important patient relevant outcomes to antirheumatic drug therapy in patients with osteoarthritis of the hip or knee. J Rheumatol 1988;15: 1833–40.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=3068365&link_type=MED&atom=%2Fcjs%2F47%2F1%2F39.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=A1988R854800020&link_type=ISI) 

18. Gorsuch RL. Factor analysis. 2nd ed. Hillsdale (NJ): Lawrence Erlbaum; 1983.
    
    

19. Campbell DT, Fiske DW. Convergent and discriminant validation by the multitrait–multimethod matrix. Psychol Bull 1959; 56:81–105.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1037/h0046016&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=13634291&link_type=MED&atom=%2Fcjs%2F47%2F1%2F39.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=A1959WD16300001&link_type=ISI) 

20. Derrett S, Paul C, Herbison P, Williams H. Evaluation of explicit prioritisation for elective surgery: a prospective study. J Health Serv Res Policy 2002; 7Suppl1:14–22.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1258/135581902320176322&link_type=DOI) 

21. Lofvendahl S, Hellberg S, Hanning M. [How was the referral sheet interpreted? Questionnaire on priority assessment of patients referred to orthopedic surgeons showed great differences between reviewers.] Lakartidningen 2002; 99:1931–9.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=12043416&link_type=MED&atom=%2Fcjs%2F47%2F1%2F39.atom) 

22. Mancuso CA, Ranawat CS, Esdaile JM, Johanson NA, Charlson ME. Indications for total hip and total knee arthroplasties. Results of orthopædic surveys. J Arthroplasty 1996;11:34–46.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/S0883-5403(96)80159-8&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=8676117&link_type=MED&atom=%2Fcjs%2F47%2F1%2F39.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=A1996TU51500009&link_type=ISI) 

23. Tierney WM, Fitzgerald JF, Heck DA, Kennedy JM, Katz BP, Melfi CA, et al. Tricompartmental knee replacement. A comparison of orthopædic surgeons’ self reported performance rates with surgical indications, contraindications, and expected outcomes. Knee Replacement Patient Outcomes Research Team. Clin Orthop 1994;305:209–17.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=8050231&link_type=MED&atom=%2Fcjs%2F47%2F1%2F39.atom) 

24. Wright JG, Coyte P, Hawker G, Bombardier C, Cooke D, Heck D, et al. Variation in orthopedic surgeons’ perceptions of the indications for and outcomes of knee replacement. CMAJ 1995;152:687–97.
    
    [Abstract](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czo5OiIxNTIvNS82ODciO3M6NDoiYXRvbSI7czoxNzoiL2Nqcy80Ny8xLzM5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

25. Quintana JM, Arostegui I, Azkarate J, Goenaga JI, Elexpe X, Letona J, et al. Evaluation of explicit criteria for total hip joint replacement. J Clin Epidemiol 2000; 53:1200–8.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/S0895-4356(00)00244-4&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=11146265&link_type=MED&atom=%2Fcjs%2F47%2F1%2F39.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000166010100003&link_type=ISI) 

26. Walker M. Waiting your turn: hospital waiting lists in Canada. 11th ed. Vancouver The Fraser Institute; 2001.
    
    

27. Naylor CD, Williams JI. Primary hip and knee replacement surgery: Ontario criteria for case selection and surgical priority. Qual Health Care 1996;5:20–30.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjY6IjUvMS8yMCI7czo0OiJhdG9tIjtzOjE3OiIvY2pzLzQ3LzEvMzkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

28. Dennett ER, Kipping RR, Parry BR, Windsor J. Priority access criteria for elective cholecystectomy: a comparison of three scoring methods. N Z Med J 1998;111:231–3.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=9695752&link_type=MED&atom=%2Fcjs%2F47%2F1%2F39.atom) 

29. Dennett ER, Parry BR. Generic surgical priority criteria scoring system: the clinical reality. N Z Med J 1998;111:163–6.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=9612483&link_type=MED&atom=%2Fcjs%2F47%2F1%2F39.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000073548300004&link_type=ISI) 

30. Halliwell T. How fair is cataract prioritisation? N Z Med J 1998;111:405–7.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=9830430&link_type=MED&atom=%2Fcjs%2F47%2F1%2F39.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000076817100002&link_type=ISI) 

31. Jackson NW, Doogue MP, Elliott JM. Priority points and cardiac events while waiting for coronary bypass surgery. Heart 1999;81:367–73.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiaGVhcnRqbmwiO3M6NToicmVzaWQiO3M6ODoiODEvNC8zNjciO3M6NDoiYXRvbSI7czoxNzoiL2Nqcy80Ny8xLzM5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

32. Seddon ME, French JK, Amos DJ, Ramanathan K, McLaughlin SC, White HD. Waiting times and prioritization for coronary artery bypass surgery in New Zealand. Heart 1999;81:586–92.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiaGVhcnRqbmwiO3M6NToicmVzaWQiO3M6ODoiODEvNi81ODYiO3M6NDoiYXRvbSI7czoxNzoiL2Nqcy80Ny8xLzM5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

33. Fortin PR, Penrod JR, Clarke AE, St-Pierre Y, Joseph L, Belisle P, et al. Timing of total joint replacement affects clinical outcomes among patients with osteoarthritis of the hip or knee. Arthritis Rheum 2002;46:3327–30.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1002/art.10631&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=12483739&link_type=MED&atom=%2Fcjs%2F47%2F1%2F39.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000179754000026&link_type=ISI) 

34. Holtzman J, Saleh K, Kane R. Effect of baseline functional status and pain on outcomes of total hip arthroplasty. J Bone Joint Surg Am 2002;84:1942–8.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NjoiamJqc2FtIjtzOjU6InJlc2lkIjtzOjEwOiI4NC8xMS8xOTQyIjtzOjQ6ImF0b20iO3M6MTc6Ii9janMvNDcvMS8zOS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

35. Hajat S, Fitzpatrick R, Morris R, Reeves B, Rigge M, Williams O, et al. Does waiting for total hip replacement matter? Prospective cohort study. J Health Serv Res Policy 2002;7:19–25.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1258/1355819021927638&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=11822257&link_type=MED&atom=%2Fcjs%2F47%2F1%2F39.atom) 

36. Mahon JL, Bourne RB, Rorabeck CH, Feeny DH, Stitt L, Webster-Bogaert S. Health-related quality of life and mobility of patients awaiting elective total hip arthroplasty: a prospective study. CMAJ 2002;167:1115–21.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMToiMTY3LzEwLzExMTUiO3M6NDoiYXRvbSI7czoxNzoiL2Nqcy80Ny8xLzM5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

37. Batterbury M. A priority scoring system for cataract surgery in Liverpool: implementation not successful. 1999. Available: [http://bmj.com/cgi/eletters/318/7181/412#2652](http://bmj.com/cgi/eletters/318/7181/412#2652) (accessed 2003 Dec 4).
    
    

38. Edwards RT. Points for pain: waiting list priority scoring systems. BMJ 1999;318: 412–4.
    
    [FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTgvNzE4MS80MTIiO3M6NDoiYXRvbSI7czoxNzoiL2Nqcy80Ny8xLzM5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==)