Abstract
Objective: To evaluate the feasibility of incorporating hand-held computing technology in a surgical residency program, by means of hand-held devices for surgical procedure logging linked through the Internet to a central database.
Setting: Division of General Surgery, University of Toronto.
Design: A survey of general surgery residents.
Methods: The 69 residents in the general surgery training program received hand-held computers with preinstalled medical programs and a program designed for surgical procedure logging. Procedural data were uploaded via the Internet to a central database. Survey data were collected regarding previous computer use as well as previous procedure logging methods.
Main outcome measure: Utilization of the procedure logging system.
Results: After a 5-month pilot period, 38% of surgical residents were using the procedure-logging program successfully and on a regular basis. Program use was higher among more junior trainees. Analysis of the database provided valuable information on individual trainees, hospital programs and supervising surgeons, data that would assist in program development.
Conclusions: Hand-held devices can be implemented in a large division of general surgery to provide a reference database and a procedure-logging platform. However, user acceptance is not uniform and continued training and support are necessary to increase acceptance. The procedure database provides important information for optimizing trainees’ educational experience.
Computer technology carries significant potential benefits in a variety of health care applications, but the medical field continues to lag behind most information-rich industries in the adoption of these new technologies.1 Obstacles to computerization include accessibility, costs and user knowledge. Since hand-held devices address many of these concerns, they are becoming increasingly popular.2 From a clinical perspective, handheld computers can be used to track patient information, as a medical reference source, and to perform medical calculations. Hand-held computers are now being used in medical education, such as for documentation of procedural reports in a family medicine residency program,3 for procedure and resuscitation tracking in emergency medicine,4–6 and in obstetrics and gynecology,7 and anesthesiology.8 One advantage of documenting procedures on hand-held computers is that data can be entered at the time of the procedure, eliminating potential loss of information and avoiding the need for duplicate entry.3 Although there are many theoretical advantages to this technology, the implementation of hand-held devices in medicine is not widespread. Poor acceptance by health care practitioners may relate to the lack of customized software and poor training in the use of this technology.9
An important component of a surgical residency is the opportunity to fulfil accreditation requirements with respect to performing a minimum number of specific surgical procedures. A surgical procedure log is therefore kept during the residency and submitted to document fulfilment for licensure to practise. Many different methods of logging and storing the data are used. We introduced computerized procedure logging with hand-held computers into a large general surgery training program. This study is the first to evaluate electronic procedure logging in an academic surgical training program. The use of hand-held computers has been described in a surgical group practice focusing on the use of hand-held devices for physician schedules, billing and cost-effectiveness.10 The objective of our procedure logging system was to generate a database to allow the documentation of individual trainees’ procedural experience, and to provide information for the training program director to improve the educational experience. This report documents the feasibility of such an intervention and the benefits and problems encountered in the introduction of this new technology in our training program.
Methods
Setting
The 69 surgical residents (PGY 1–5) in the Division of General Surgery of the Department of Surgery at the University of Toronto were provided with hand-held computers loaded with customized and commercially available software. These trainees rotate through 9 different city hospitals during their training.
Hand-held devices
The Palm IIIxe unit (Palm Inc., Santa Clara, Calif.) was chosen as the hand-held device since many residents were familiar with the Palm operating system. The devices are pocket-sized 8.3 × 2 × 12-cm units, with 8 megabytes of RAM. Information is entered using a stylus on a touch-sensitive screen and can be password protected. Entered data can easily be transferred from the hand-held to a desktop or laptop computer and vice versa using a hard-wired simple synchronization. An Infrared Data Association port (IRDA) allows transfer of information from one device to another.
Software
A Palm-based software program for surgical procedure logging (iqLog; Infiniq, Toronto) was developed, allowing data collection conforming to the requirements of the American Board of Surgery. This program provides easy data entry on the screen of the hand-held device with drop-down lists to enter date, hospital, surgical role, attending surgeon and procedure. Text entry was required only for the optional patient identifier and for additional notes. The entered procedures could be reviewed on the main screen of the Palm iqLog program and could be edited subsequently.
All surgical residents were provided with communication software to be used on their home computer. This allowed the entered procedural data to be transmitted from the Palm unit via the Internet to a central database controlled by the Division of Surgery (Fig. 1). Each trainee was provided with a username and password and was able to view his or her entered procedures on a secure Web site. The program director of the Division of Surgery was able to access a summary of all the surgical trainees using the system and list their logged procedures. These data could be further analyzed according to trainee, supervising surgeon, hospital and level of training.
The transfer of surgical procedures entered on the Palm hand-held device (A) to the computer (B) by hard-wire synchronization, followed by Internet transfer to the central database (C). The database can be visualized and analyzed via secure Internet access.
The procedure-logging software program also includes a one-way messaging system. This allows the program director to enter short messages and transmit these messages in a read-only format to the trainees whenever they perform a synchronization. In addition to the procedure-logging system, applications that provide access to medical information were installed onto the Palm devices. JFile (www.land-j.com) was chosen as a spreadsheet software application as it allows easy conversion of documents from Microsoft Excel. ISilo (www.isilo.com) was chosen as the reader for text references. Surgical reference information, training program objectives, pharmacopoeias, laboratory reference values and hospital handbooks were converted into a Palm format and installed on the hand-held devices. Medical calculators were also installed. These databases could be updated, edited and transferred to the individual Palm units through synchronization operations.
The surgical residents were assisted with the installation of the medical pharmacopoeia “Epocrates” (www.epocrates.com) as well as the shared-access “WeSync” calendar (www.wesync.com). “WeSync” provides a shared read-only calendar published by the Division of Surgery. Information for this calendar is entered by a surgical administrator and updated when necessary.
Training
To increase the acceptability of the hand-held devices and optimize their benefits to residents, all residents participated in a 2 to 3-hour training program. All trainees were provided with a user-friendly customized handbook about basic Palm usage and an outline of the software applications installed onto their hand-held devices. Ten training sessions were held during a 6-week period with an average attendance of 7 surgical residents. The teaching was organized in a Power-point presentation using a Palm emulator for a direct and interactive presentation. The teaching sessions covered basic Palm functions, setting up desktop software, the surgical procedure-logging software, accessing medical information and using “WeSync.” Two instructors were available to assist inexperienced users. After the first training session, feedback and suggestions were evaluated to assist in course development and to improve and enhance the training sessions for future participants. All 69 surgical trainees were provided with pocket cards that included software tips and instructions as well as a support telephone number and email. For immediate personal support, 2 support-team members were available by pagers.
Data security
The collection and storage of patient information on hand-held devices gives rise to concerns regarding patient confidentiality.11 Although the risk is no higher than with standard procedure logs where patient stickers are often used, trainees were advised to limit identifying patient information. Entry in the patient identifier field was optional, and largely for trainees’ personal reference use. They were advised to avoid names and preferably enter only an initial or the first digits of the hospital number. Data transfer to the central database incorporated 128-bit encryption. The software program allows the option to delete the data stored on the hand-held device after the synchronization to the central database. The presence of patient identifiers was evaluated by review of the database.
Baseline survey
An anonymous survey was distributed to all surgical residents during the training sessions to evaluate education, computer knowledge, previous experience with hand-held devices and previous technique of procedure logging before implementation of the hand-held logging system.
Main outcomes
Outcome measures were aimed at evaluating the feasibility, acceptance, benefits and disadvantages of this computerized procedure-logging system. The quantitative outcome measured was the utilization of the system by tracking access to the central database. Qualitative data were obtained regarding concerns, complaints and comments by the surgical trainees during the implementation of this system. The requirement for support services by telephone and email was tracked.
Results
Baseline survey
The survey was completed by all 69 surgical residents to evaluate their previous computer knowledge and experience with hand-held devices. Although 12% of subjects did not have a home computer, all surgical trainees used and received email. Other common uses for computers include statistical analysis (91%), research searches to support studies or publications (88%), reading electronic journals (58%) and preparing teaching material (45%).
Before our study, 31 surgical residents (45%) had been using handheld devices for an average of 14 months (range from 2–48 mo). All but 2 of these trainees were using Palm operating system devices. The most popular uses were for addresses (97%), scheduling (90%) and personal to-do lists (87%). Only 4 (13%) of the 31 residents used their handheld device for procedure logging. The majority kept track of their procedures by collecting patient stickers, taking notes on paper, in a logbook or on their desktop computer. Difficulties with procedure logging were experienced by 14 trainees (20%) and included losing patient data and procedural details.
Implementation
Following the implementation and training period, a subgroup of trainees rapidly adopted this technology and became regular users of the procedure-logging program. However, subsequent user acceptance was slow, and 5 months after the system was implemented only 26 residents (38%) were using it successfully and regularly (Fig. 2). Review of the nonusers revealed that a significant proportion were in nonprocedural rotations, as in research or in the intensive care unit. Analysis by year of training demonstrated that the system was more widely used by junior trainees, likely because they had more to gain from the data accumulation. Of the residents in the first to third postgraduate years, 17 (53%) were using the surgical procedure log on a regular basis whereas 9 (24%) fourth and fifth year residents were logging their procedures through the hand-held device.
Use of the procedure-logging system for the first 5 months after implementation. Solid bars represent postgraduate years 4 and 5, shaded bars represent years 1 to 3.
Database analysis
Procedure-logging information was available from 26 trainees during this 5-month period. Although this may not be representative of the whole training program, a large volume of data was generated. In all, 1465 procedures were logged, for an average of 56.3 per trainee. More than 99% of data entries were complete, including at least the resident identifier, surgical role, hospital, supervising surgeon and procedure. Greater than 99% of residents complied with the request to avoid patient-identifying information. The procedures were analyzed by resident (Table 1), by supervising surgeon (Table 2) and by hospital (Table 3), providing valuable data for management of the educational program. The database allowed reports to be generated for individual trainees, comparisons among trainees and comparisons among supervising surgeons and hospitals.
Example of an Analysis of Procedures Possible From the Centralized Procedure-Logging Database With Use of the Hand-Held Palm Device: by Resident
Example of an Analysis of Procedures Possible From the Centralized Procedure-Logging Database With Use of the Hand-Held Palm Device: by Supervising Surgeon
Example of an Analysis Possible From the Centralized Procedure-Logging Database With Use of the Hand-Held Palm Device: by Hospital
Qualitative feedback
The active support service provided to trainees (by email, telephone or pager) allowed data collection on problems encountered, concerns expressed and general satisfaction. During the implementation phase an average of 25 support calls or emails were received per month. The major concerns involved setting up the communication software on desktop computers, suggestions for additions to the software program and requests for improved access to information via the Internet site. A small number of trainees unfamiliar with hand-held computers required additional training.
Discussion
We introduced hand-held devices into a university general surgery training program to facilitate the logging of surgical procedures. This involved the development of customized software, a hand-held computer-training program for surgical trainees and ongoing technical support. Although significant merit was noted, a number of pitfalls and concerns were identified. Previous studies have described electronic hand-held procedure-logging systems in family practice, obstetrics and gynecology, in emergency medicine and in anesthesia.3–8 These reports provided little data on user acceptance of these systems, and of problems encountered. The system introduced in the Division of General Surgery differs from previously reported logging systems in that it allows data transfer via the Internet to a remote data warehouse. This was necessitated by the large training program and diverse geographical distribution of trainees. Data collection was also aimed not only at generating individual trainee reports but also at providing information of value for program development.
The Palm training sessions were rated highly, particularly by first-time users. The emphasis was placed on basic Palm use and so for several residents who were already familiar with hand-held devices, the provided information seemed redundant. Feedback was obtained and training sessions were modified according to users’ needs.
Tracking of access to the central database showed that after implementation of the surgical procedure log, there was an initial enthusiasm with a subgroup of early adopters to the system and subsequently a slower increase in the number of users. By 5 months, only 38% of trainees were actively uploading data. We found that more first to third-year residents were regular users than fourth and fifth-year residents. This might be attributed to a number of the factors. Many fourth-year residents were in their research rotation during the study period and were not doing surgical procedures at the time. Residents in their fifth and final year of training were to complete their training shortly and already had an established logging system. Junior trainees, who were more likely to benefit from a comprehensive personal procedure log, were more compliant. Improved training, support and software functionality may improve user compliance.
During the first weeks after the implementation, the inability to keep a hard copy of the entered procedures was a concern of the residents. After the establishment of a Web site with the option to view and print the logged procedures, user acceptance improved. One of the most common technical problems was the set-up of the desktop communication software on the home computer. This was resolved through telephone and email support. Other support issues were generally minor and attributable to lack of familiarity with the hand-held platform. No major hardware concerns were noted.
A potential problem identified was that 8 (12%) residents did not have a home personal computer and 5 (7%) residents used a Macintosh system, which was not compatible with the customized software. To address this, we have set up a variety of hospital computers with the communication software to allow every resident the opportunity to have access to a computer.
All data entered were self-reported by the trainees, as in conventional procedure-logging systems. No attempt was made to evaluate the accuracy of the data entered, since the goal of this project was to evaluate the feasibility of implementing this mobile computerized system for surgical procedure logging. However, this technology does allow the possibility in the future of introducing data confirmation by supervisors using passwords or signature recognition.
The data collected by this system were used to generate reports for individual trainees and to provide invaluable data to study and improve the training program. There is significant potential for this information to benefit the education process by the ability to identify the educational institutions that provide the best teaching experience.
Conclusions and considerations
Hand-held devices can be implemented in a large general surgical training program to provide a procedure-logging platform. Although such a system appears extremely attractive and acceptable, the introduction of new technology is clearly associated with a degree of resistance. Constant technical support and problem-solving is necessary to optimize user acceptance. The long-term acceptance of this innovation remains to be seen.
Although our study did not directly address this issue, the potential value of a large database to both individual trainees and to program development is enormous. The electronic data collection at the point-of-care provides immediate, up-todate information formatted for analysis, with no additional data entry or manipulation required. This provides the program director access to the individual trainee’s progress as well as to program-wide analyses. Due to the Internet-based nature of the technology, the possibility exists of broadening this system to other training programs. However, it is important to consider that interpretation of any data relies on a high level of compliance with data entry, and the accuracy of procedural information recorded. Future evaluation of this system will have to address issues of data accuracy and validity.
Acknowledgements
We acknowledge the assistance provided by Palm Canada Inc., and Infiniq, a division of Blue Oaks Software, Mississauga, Ont.
- Accepted May 6, 2002.
In this issue
Article tools
Related Articles
Cited By...
- No citing articles found.