Development of a cumulative teaching score for tracking surgeon performance in undergraduate medical education
==============================================================================================================

* Christine C. Moon
* Sneha Raju
* George Christakis

## Abstract

**Background:** Surgeon educators are important in undergraduate medical education (UME). However, teaching activities are undervalued and under-recognized compared with research, resulting in poorer quantity and quality of surgeon teaching. The purpose of this study was to investigate teaching roles available to surgeons and the amount of effort involved.

**Methods:** A comprehensive review of all possible roles surgeons may take in UME at our institution was assembled. Delphi committee members were asked to evaluate each teaching role on the amount of effort needed per hour. Results were analyzed using descriptive statistics, and a Cronbach α of 0.60 or higher was the threshold to declare consensus.

**Results:** Twenty-five participants, including physicians, residents and medical students, completed the study. Consensus was reached on the amount of effort needed for each teaching role. These values were used to prototype a cumulative teaching score that can be used to qualitatively quantify surgeon teaching.

**Conclusion:** Surgeon teaching is important in UME, but not tracked and thus not valued. To improve the quantity and quality of surgeon teaching in UME, we need to track, reward and recognize surgeon teaching activities. The “effort score” we developed to objectively and transparently qualify teaching was able to determine the relative effort needed for each teaching activity in UME at the University of Toronto. Combining the effort score and time committed to each teaching activity will produce a cumulative teaching score for each instructor.

Surgeons participate in undergraduate medical education (UME) by lecturing, training students in surgical procedures and skills, observing performance, and providing feedback through evaluating and mentoring students. Surgeon involvement in UME is limited for a number of reasons,1–3 including time constraints because of surgeons’ busy practices as well as inadequacy of formal training in teaching.4–6 However, the most important deterrent for surgeon participation in UME is the lack of departmental and university recognition.7 A recent review paper by the Higher Education Academy in Great Britain provides an excellent summary of the valuation of teaching in UME in universities with a large emphasis on research.8 Although participation in clinical and basic research is highly sought after, and researchers are promoted and rewarded handsomely (both financially and in other forms, such as through promotions) at major Canadian universities, teaching and education in general is not valued at the same level. As teaching is undervalued compared with research activities at universities, the quality and quantity of teaching and participation in education are poorly measured, if at all. Teaching activities are not systematically measured and, thus, have little to no bearing on promotions and hiring. Measuring teaching activities may alter the culture of an academic institution by adding value and thereby increasing interest in teaching. Despite the low value placed on teaching, surgeons do enjoy it.9 Surgeons have many motivations to teach, including sharing their knowledge with students, teaching as a way to keep up with new developments, and enjoying mentoring and recruiting new individuals into the surgical specialties.9

Surgeon participation in UME is very important for various reasons. First, surgeons are needed to teach basic knowledge and surgical skills to all undergraduate students. Spratt and colleagues10 have shown that basic surgical skills require additional emphasis during clerkship. Another study involving recent medical graduates concluded that they greatly valued basic surgical proficiencies even when they were interested in nonsurgical disciplines.6 Trainees also felt that opportunities to acquire these skills were scarce.6 The presence of surgeons in undergraduate education also allows for recruitment of students to surgical subspecialties. Direct contact and positive experiences with faculty during medical school are strong predictors of specialty choice.7,11–15 Studies have shown that increased surgical faculty involvement in early medical education created greater interest in surgical subspecialties.16,17

Potential solutions to increase the valuation of surgical teaching and UME include allocating designated time for teaching, providing administrative support for various education roles, and establishing a system of adequate compensation or recognition for teaching. This study was created in an attempt to increase recognition of surgical teaching at the University of Toronto in order to adequately recognize and reward surgeon educators and increase the number of surgeon participants.

The necessary first step to reach this goal is to establish a method to capture all teaching moments of surgeon educators. Interestingly, there is limited collection of data on surgeon teaching activities in the Department of Surgery at the University of Toronto. Preclerkship courses document their lecturers, and surgery clerkship information is collected by the Department of Surgery, but there is no central repository of information within the faculty to determine who is teaching across UME. Furthermore, there is virtually no documentation on cocurricular teaching by surgeons or on the work they might do on committees or in leadership positions. This is true of most Canadian medical schools.

The present study is, to our knowledge, the first of its kind to document and account for all surgeon-related education activities, including teaching during the preclerkship period; cocurricular teaching activities, such as supervision and teaching at student clubs, in informal teaching moments and during the mentorship and evaluation of students; and in the creation of new educational programs. Although teaching evaluation scores are important, they do not quantify the amount and quality of teaching or other activities, such as mentorship and support of student-led initiatives. Thus, we developed an “effort score” that attempts to qualify each teaching activity in a standardized way. The effort scores, developed by our expert panel using the Delphi method, qualify the teaching that surgeons are doing. By combining the effort score for a type of teaching activity with the time spent on each teaching activity, we are able to calculate a cumulative teaching score that is at once qualitative and quantitative for each member of our faculty for every academic year, and thus better recognize each surgeon’s teaching activities. We hope that this system will serve as the basis for recognizing and rewarding surgeons for scholarly teaching and as a way to incentivize others to participate in teaching. This system could be applied across all medical schools in Canada.

## Methods

### Review of surgical education roles

In the first part of this study, the team undertook a comprehensive review of all possible roles surgeons may take in UME at the University of Toronto, using university online resources and interviews with students, support staff, and physician administrators and instructors. We developed an appendix of roles describing every possible teaching role a surgeon instructor may undertake at the undergraduate level, broken down by year and course. In addition to curricular and formal teaching activities, as many informal and cocurricular teaching activities as possible were documented. Importantly, educational administrative duties were included in this list. Administrative duties are related to teaching because they make education possible; furthermore, having surgeon representation in administrative spaces (e.g., academic committees) is important in the creation and maintenance of good surgical education at the undergraduate level. A condensed list can be found in Appendix 1, available at canjsurg.ca/009516. The types and quality of teaching data already being collected by the university were also determined.

### Delphi methodology

The main component of the study consisted of a 3-step consensus-building approach using the Delphi methodology with a panel of education experts. The Delphi methodology is based on the principle that collective beliefs are more valuable than opinions of a single individual in the generation of knowledge.18 It is a methodology that allows for the systemization of opinions, experience, expertise and critical thinking.19 The purpose was to qualitatively determine through this expert panel the amount of effort needed for each teaching activity.

### Identification and recruitment of education experts in undergraduate medical education

A purposeful sample of 48 participants, including staff physicians, residents, administrative support staff, and undergraduate medical students, was invited to participate. The staff physicians included instructors with a range of teaching expertise and experience in different roles. Teaching activities of committee members varied greatly. Some members participated in 1 or 2 teaching activities, such as giving a lecture once a year, whereas others were more substantially involved, such as running a core preclerkship course. In addition, 3 key administrative support staff, 2 residents with undergraduate teaching experience, and undergraduate students who served on the Department of Surgery Undergraduate Education Committee, were invited to participate.

Participants were invited via email, with the consent form and first round of the electronic survey included in the body of the email. The email was sent from the chair of the Department of Surgery. All surveys were conducted online, with an option for participants to request paper copies of the survey (none chose this option). Surveys were created and stored on SurveyMonkey, with a secure business account. The online survey method rather than a face-to-face consensus-building approach was used from the beginning to minimize any possible effect of power dynamics on Delphi members in the presence of their supervisors or instructors.

### Delphi consensus

Participants were first asked to independently identify teaching roles in UME to ensure as complete an appendix of roles as possible. Roles missing from the appendix were added to the list, and the new, complete list of teaching roles was distributed to the Delphi committee.

Participants were then asked to evaluate each teaching role on the amount of effort needed per hour of the activity. Effort was defined as a combination of the following: expertise required; additional training or learning required; intellectual and physical exertion; and an “aggravation factor” to account for the general feelings of stress, distress, or nuisance associated with different roles. The “effort score” was used in an attempt to qualify the teaching activities with respect to each other on a scale of 1 to 5, where a score of 1 indicated no effort at all and a score of 5 indicated the most effort. Consensus was defined as a Cronbach α value of 0.7 or higher, indicating internal consistency as described previously.20,21 All values were rounded as appropriate to maintain 1 significant digit.

After 2 survey-based rounds, the Delphi committee met in person to review our findings and ensure consensus was reached. This discussion was facilitated by the research assistant, who holds no position of power or authority over any member of the group, in an effort to minimize the influence of powerful individuals on members’ stated opinions.

### Statistical analysis

Results were analyzed using descriptive statistics, and Cronbach α values were used to determine if consensus was reached. The majority of scores showed at least adequate-to-good internal consistency (Cronbach α > 0.7), with most showing excellent internal consistency (Cronbach α value > 0.85).

## Results

Forty staff physicians, 2 support staff, 2 residents, and 4 undergraduate medical students were invited to participate in a Delphi study to determine the levels of effort needed for each teaching activity. Physicians with a range of teaching and administrative experience in UME, both surgeons and others, were invited to participate. In all, 25 participants (20 physicians, 2 residents and 3 undergraduate medical students) completed the study. Characteristics of the committee members can be found in Table 1.

View this table:
[Table 1](http://canjsurg.ca/content/62/3/150/T1)

Table 1 
Delphi committee characteristics

The final appendix of roles contains more than 115 individual teaching opportunities and their descriptions. More than 30 roles were not included in this appendix, either because they were clearly defined and compensated elsewhere (e.g., dean positions) or were outside the scope of a surgeon’s teaching practice (e.g., family medicine–related clerkship activities). Roles fit into 1 of 7 categories: administrative, leadership, preclerkship, clerkship, longitudinal integrated clerkship, co-curricular roles, and informal roles. Certainly, many of the roles were similar and overlapped, and changes in curriculum could change the formal titles of these roles. However, it was important to outline the variety of niches and places surgeons could fit in. Consensus was reached on almost all effort score values, with most teaching activities falling between moderate effort and most effort to teach per hour (scores of 3–5). The exceptions were for a course in the art of medical practice, a first-year course meeting once weekly, and for the surgery rotation for third-year medical students (Cronbach α value of 0.6). Table 2 shows effort scores for each type of teaching activity. Consensus was defined as a Cronbach α ≥ 0.7, indicating internal consistency.20,21

View this table:
[Table 2](http://canjsurg.ca/content/62/3/150/T2)

Table 2 
Cumulative teaching scores per hour of undergraduate teaching activity

## Discussion

Surgeon participation in undergraduate education is limited for a variety of reasons, including a lack of time, knowledge of possible roles and appropriate compensation. Above all, surgeon teaching is not tracked because it is not valued, and is not valued because it is not tracked. There are a number of studies confirming that lack of adequate compensation in the form of career advancement and remuneration is an obstacle in physician participation in medical education.22–32 These concerns hold true for surgeon educators, who are constrained by busy operating room schedules, clinics and on-call days. There is a need to funnel resources to increase and improve surgeon participation in UME.

Based on our expert panel, there was general consensus that a there are 2 independent factors — effort and time — influencing compensation. Effort was defined as a combination of the expertise required, the additional training or learning required, intellectual and physical exertion, and an aggravation factor. Time was defined as the amount of time necessary to participate in any given education role as well as preparation time. Thus, our study attempted to qualify teaching types based on how much effort was needed per hour of teaching.

An effort score was developed to objectively qualify types of teaching, while providing transparency on the scoring system through a Delphi consensus-building process. The effort scores can become a starting point in the development of an appropriate system to recognize surgeon educators who are involved in undergraduate teaching. We found that experts in the field rated various teaching roles at different levels of difficulty. For instance, highly administrative roles, such as committee chairs, directors and curriculum developers, received high effort scores. The reasons for this could include requirements for mind-intensive work, high levels of expertise, and greater overall commitment. On the other hand, lecturers and examination graders received the lowest effort scores. However, as expected, the range of scores was small because all teaching roles require some amount of effort and thought.

When discussed with the expert panel, we discovered 2 main reasons for low effort scores. First, low-scoring teaching activities, such as lecturing, can often be designed once and repeated thereafter by the same lecturer several times over the years. Furthermore, since lecturers are often individuals with several years of experience in the field, there is little background work involved when lecturing. Similarly, for graders, whereas the first few tests may be time-consuming, the rest are relatively unchallenging. Experts defined high-effort roles as “mind intensive,” while low-effort roles tended to be “time intensive.”

The scores compiled here by our Delphi committee can be used to calculate the cumulative teaching score (CTS), a potential scoring system for teachers in undergraduate medical education, not only at the University of Toronto, but elsewhere in Canada. For instance, instructors in a specific third-year course at the University of Toronto were reported to have undergone 6 hours of faculty development and to have facilitated 14 hours of small-group sessions for the course in a single academic year. Using the effect scores found in Table 1, a faculty member whose only teaching was in this course would have a teaching score of 66.6 (Fig. 1). As the CTS is further developed, we hope to add multipliers in order to be able to incorporate student feedback and evaluation scores.

![Fig. 1](http://canjsurg.ca/https://www.canjsurg.ca/content/cjs/62/3/150/F1.medium.gif)

[Fig. 1](http://canjsurg.ca/content/62/3/150/F1)

Fig. 1 
The proposed formula to calculate the cumulative teaching score (CTS). This score will represent a quantitative measure of surgeon involvement in undergraduate medical education and can be further used for compensation purposes. The score is calculated based on 2 measures: the time involved to perform the activity and the effort involved as delineated by the Delphi committee. Above is an example of a sample surgeon who has completed 2 teaching responsibilities over the past year accounting for 20 hours, which gives him a CTS of 66.6.

The effort scores produced here can be used as a starting point in the development of an appropriate system to recognize surgeon educators who are involved in undergraduate teaching. In all, the results show that instructors and other stakeholders, including students themselves, believe teaching to be an activity that takes commitment and investment on the part of the instructor and the institution that houses them.

### Implementation of scores for other faculties of medicine

A school that chooses to adopt a similar equation as part of their undergraduate education, with the hopes of adequately compensating surgeon educators, must implement a sustainable system that is easily integrated into the current administrative structure. A committee should be instituted to ensure that the system is feasible, supported, and seen as valuable by surgical faculty. The potential benefits for surgeons, in terms of compensation, recognition, mentoring and career advancement, should be emphasized to increase buy-in. This committee can also be responsible for updating educational activities in the database as they change; for instance, with the creation of a new teaching activity, this could involve developing an appropriate effort score and matching description for the appendix of roles.

The appendix of roles and the effort scores presented here can be adapted for use at other Canadian universities. It is important to note that while these roles were collected specifically at the University of Toronto, they are in accordance with many teaching roles that can be found at all medical schools across Canada.

The steps to implementing a metric system to measure teaching activities can be complex. The first step should be a comprehensive review of all teaching opportunities available to surgical faculty. Second, the committee would need to decide on “effort” scores based on the difficulty of each teaching role. A more efficient option may be to use the effort scores delineated in the present study, after consensus from key stakeholders. Third, a multiplier reflecting student and faculty-led feedback should be incorporated into the equation. Finally, a Web-based system needs to be created that can be easily used by administrative staff, faculty members and students. A mobile app component may be considered for ease of use by clients. Such a database can be programmed to create faculty profiles for individuals or departments. Furthermore, an appendix of roles such as the one created for this study may be given to incoming faculty as a way of introducing the breadth of roles a surgeon may take and to further encourage surgeon involvement in undergraduate teaching.

### Limitations

This study had certain limitations. First, though we chose a purposeful sample of experts to participate in our Delphi committee, most committee members did not have direct experience with every single teaching role across UME; staff physician educators tended to focus their time in a few chosen teaching activities, residents tended to conduct training in their surgical specialty only, and undergraduate students still had not experienced all of the undergraduate curriculum. This meant that participants were forced to answer “don’t know” in their scoring of the effort of at least some of the teaching activities. de Villiers and colleagues19 also indicate that the Delphi committee should ideally include 5–10 members from each category of members; we were unable to fulfill this requirement owing to limited numbers of educational support staff overall and low interest and engagement in education among residents and medical students.

Furthermore, it must be acknowledged that not every faculty member will approach the same teaching position in the same manner. Thus the effort scores obtained through this study may not exactly match individual instructors’ amount of effort per hour of work. However, we attempted to mitigate these effects in our study by gathering a panel of experts with a wide range of experiences across all teaching positions. They represent many different teaching approaches.

Finally, the study was not prospective. Just as it is almost impossible to know of every single teaching role across UME, it is impossible to predict new teaching positions that may arise, particularly with constant curricular changes. However, no matter the specific curriculum and titles for teaching roles, the general teaching roles remain the same; for instance, lecturers are almost always present in any medical school curriculum, at least to some extent.

## Conclusion

Surgeon teaching is important in UME, but not tracked and thus not valued. In order to improve the quantity and quality of surgeon teaching in UME, we need to track, and subsequently reward and recognize, surgeon teaching activities.

In this study, we were able to determine the relative effort needed for each teaching activity in UME at the University of Toronto. This effort score is an objective qualification by our expert panel of the types of teaching being done by our surgeon faculty. Next steps include beginning to track surgeon teaching data and conducting a qualitative study on the culture around UME among surgeons. Increasing surgeon teaching in UME is important to improve the recruitment and training of students in surgical specialties as well as to further research and teaching by future surgeons.

At the University of Toronto, we are beginning to recognize and reward surgeon educators. In the past several years, the university has begun to make special efforts to hire surgeon educators, including those who have completed teaching or education degrees. On the whole, there is very little literature in Canada, or indeed globally, about the types and quality of surgeon teaching at the undergraduate level. Furthermore, there are few departments that track surgeon teaching activities. We hope that tracking activities will bring to light the hard work of surgeon educators and that tying teaching activities to compensation and recognition will improve the quantity and quality of surgeon teaching in UME.

## Footnotes

*   Presented at the Canadian Undergraduate Surgery Education Committee Conference, November 2015, and the Canadian Conference in Medical Education, April 2016.

*   **Competing interests:** None declared.

*   **Contributors:** C. Moon and G. Christakis designed the study. C. Moon and S. Raju acquired and analyzed the data, which G. Christakis also analyzed. C. Moon and S. Raju wrote the article, which all authors reviewed and approved for publication.

*   Accepted August 28, 2018.

## References

1.  1.Turner S, Diederichs B, de Gara C.Surgeon involvement in pre-clinical medical education: attitudes of directors of education.Can Med Educ J 2012;3:e69–72.
    
    

2.  2.Polk HC Jr..The declining interest in surgical careers, the primary care mirage, and concerns about contemporary undergraduate surgical education.Am J Surg 1999;178:177–9.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/S0002-9610(99)00167-1&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=10527432&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000082978800001&link_type=ISI) 

3.  3.Fuller L, Lin G, Matsui JY, et al.Teaching surgery to medical students: perspectives from our mentees.Bull Am Coll Surg 2008;93:48–53.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=18330343&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 

4.  4.Fry H.Equipping surgeons as educators.ANZ J Surg 2009;79:186–91.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=19317787&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 

5.  5.Reznick RK, Brewer ML, Wesley RM, et al.The practicing doctor’s perspective on the surgical curriculum.Am J Surg 1988;156:38–43.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/S0002-9610(88)80167-3&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=3394891&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 

6.  6.Birch DW, Mavis B.A needs assessment study of undergraduate surgical education.Can J Surg 2006;49:335–40.
    
    

7.  7.Haubert LM, Way D, Dephilip R, et al.Surgeons as medical school educators: an untapped resource.Anat Sci Ed 2011;44182–9.
    
    

8.  8.Fung D, Gordon C (2016) Rewarding educators and education leaders in research-intensive universities (Higher Education Academy).
    
    

9.  9.Collins JP, Smith MJ, Lambert T, et al.Sustaining the surgical educator workforce in Australia and New Zealand.ANZ J Surg 2011;81:411–7.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=22295340&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 

10. 10.Spratt JS, Papp KK.Practicing primary care physicians’ perspectives on the junior surgical clerkship.Am J Surg 1997;173:231–3.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/S0002-9610(97)89598-0&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=9124633&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 

11. 11.Ko CY, Escarce JJ, Baker L, et al.Predictors for medical students entering a general surgery residency: national survey results.Surgery 2004;136:567–72.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/j.surg.2004.05.021&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=15349103&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000223844300011&link_type=ISI) 

12. 12.O’Herrin JK, Lewis BJ, Rikkers LF, et al.Why do students choose careers in surgery?.J Surg Res 2004;119:124–9.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/j.jss.2004.03.009&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=15145693&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000221674500004&link_type=ISI) 

13. 13.Redlich PN, Milkowski T, Bragg D, et al.Multiple variables influence the educational value of surgical clerkship sites.Am J Surg 2006;191:178–82.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=16442942&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 

14. 14.Musunuru S, Lewis B, Rikkers LF, et al.Effective surgical residents strongly influence medical students to pursue surgical careers.J Am Coll Surg 2007;204:164–7.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/j.jamcollsurg.2006.08.029&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=17189125&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000243351200020&link_type=ISI) 

15. 16.Sandquist MK, Way DP, Patterson AF, et al.General surgery versus specialty rotations: a new paradigm in surgery clerkships.J Surg Res 2009;153:152–5.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=18674785&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 

16. 17.Debas HT, Bass BL, Brennan MF, et al.American Surgical Association Blue Ribbon Committee report on surgical education: 2004.Ann Surg 2005;241:1–8.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1097/01.sla.0000167759.15670.14&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=15621984&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000226082600001&link_type=ISI) 

17. 18.1.  Linstone HA, 
    2.  Turoff M
    
    , eds (1975) The Delphi method: techniques and applications (Addison-Wesley Publishing, Boston).
    
    

18. 19.de Villiers MJR, de Villiers PJT, Kent AP.The Delphi technique in health sciences education research.Med Teach 2005;27:639–43.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1080/13611260500069947&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=16332558&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000233852400013&link_type=ISI) 

19. 20.Tavakol M, Dennick R.Making sense of Cronbach’s alpha.Int J Med Educ 2011;2:53–5.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.5116/ijme.4dfb.8dfd&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 

20. 21.Bland JM, Altman DG.Cronbach’s alpha.BMJ 1997;314:572
    
    [FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTQvNzA4MC81NzIiO3M6NDoiYXRvbSI7czoxODoiL2Nqcy82Mi8zLzE1MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

21. 22.Brennand EA, Hanson J, de Gara CJ.Faculty attitudes towards undergraduate medical education.University of Alberta Health Sciences Journal 2006;3:9–15.
    
    

22. 23.Vinson DC, Paden C.The effect of teaching medical students on private practitioners’ workloads.Acad Med 1994;69:237–8.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=8135982&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=A1994NB04800024&link_type=ISI) 

23. 24.Levy BT, Gjerde CL, Albrecht LA.The effects of precepting on and the support desired by community-based preceptors in Iowa.Acad Med 1997;72:382–4.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1097/00001888-199705000-00018&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=9159585&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=A1997WZ79100021&link_type=ISI) 

24. 25.Ullian JA, Shore WB, First LR.What did we learn about the impact on community-based faculty? Recommendations for recruitment, retention, and rewards.Acad Med 2001;76:S78–85.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=11299175&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 

25. 26.Gray J, Fine B.General practitioner teaching in the community: a study of their teaching experience and interest in undergraduate teaching in the future.Br J Gen Pract 1997;47:623–6.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMDoiNDcvNDIzLzYyMyI7czo0OiJhdG9tIjtzOjE4OiIvY2pzLzYyLzMvMTUwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

26. 27.Latessa R, Beaty N, Colvin G, et al.Family medicine community preceptors: Different from other physician specialties?.Fam Med 2008;40:96–101.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=18247174&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 

27. 28.Woolliscroft JO, Harrison RV, Anderson MB.Faculty views of reimbursement changes and clinical training: a survey of award-winning clinical teachers.Teach Learn Med 2002;14:77–86.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=12058550&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 

28. 29.Moorhead R, Brett T, Arnold-Reed D, et al.General practice teachers.Med Teach 2012;34:84–5.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=22250688&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 

29. 30.Vinson DC, Paden C, Devera-Sales A, et al.Teaching medical students in community-based practices: a national survey of generalist physicians.J Fam Pract 1997;45:487–94.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=9420584&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom) 

30. 31.Wilson A, Fraser R, McKinley RK, et al.Undergraduate teaching in the community: Can general practice deliver?.Br J Gen Pract 1996;46:457–60.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMDoiNDYvNDA5LzQ1NyI7czo0OiJhdG9tIjtzOjE4OiIvY2pzLzYyLzMvMTUwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

31. 32.Dodson MC.Motivation and reward factors that affect private physician involvement in an obstetrics and gynecology clerkship.Obstet Gynecol 1998;92:628–33.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=9764642&link_type=MED&atom=%2Fcjs%2F62%2F3%2F150.atom)