Summary
Qualitative research contributes to the medical literature through the observation, description and interpretation of theories about social interactions and individual experiences as they occur in their natural setting. This type of research has the potential to enhance the understanding of surgeons’ and patients’ preferences, attitudes and beliefs, as well as assess how these may change with time. To date, there is no widely accepted standard for the methodological assessment of qualitative research. Despite ongoing debate, this article seeks to familiarize surgeons with the basic techniques for the critical appraisal of qualitative studies in the surgical literature.
The randomized controlled trial (RCT) and the systematic review of RCTs, both forms of quantitative research, represent the gold standard in clinical research.1 While quantitative research seeks to establish conclusions through causal determination, predictions and statistical analysis, it is limited in its examination of perspectives, attitudes and beliefs of individual participants in favour of objective, numerical data.1,2
Unlike quantitative research, qualitative research contributes to the literature through the observation, description and interpretation of theories about social interactions and individual experiences as they occur in their natural setting.3 This type of research has the potential to enhance the understanding of surgeons’ and patients’ preferences, attitudes and beliefs, as well as assess how these may change with time.
While qualitative research methods have been well documented since 1985, there is growing recognition that this study design is well suited for the surgical literature.4 In 2016, Maragh-Bass and colleagues5 performed an analysis on publication trends in surgical literature that showed that qualitative surgical research has gained in popularity, representing more than half of all articles published in 32 surgical journals since 2011. Recent examples of qualitative studies within the surgical literature include the role of salespeople in the surgical suite,6 the perspectives of orthopedic surgeons on patient candidacy for total joint arthroplasty7 and the postsurgical barriers to exercise in the bariatric patient.8 Despite the popularity of qualitative research, such studies are often considered low-level evidence as they are routinely likened to case reports, expert opinions or anecdotal findings owing to a lack of familiarity with its methods.1,5,9 This thinking is ultimately misleading, as there is ample evidence within the literature to suggest that qualitative research has a useful role to play in the surgical domain.5
To date, there is no widely accepted standard for the methodological assessment of qualitative research.10 Despite ongoing debate, this article seeks to familiarize surgeons with the basic techniques for the critical appraisal of qualitative studies in the surgical literature.
Clinical scenario
A 57-year-old construction worker, who had a total knee replacement (TKR) by another surgeon at a peripheral hospital, has not been able to return to work after 6 months. You, an orthopedic surgeon, and your senior orthopedic resident review the operative record as well as the pre- and postsurgery radiographs; you cannot find anything wrong with the previously performed surgery. Your resident informs you that the patient was upset that he was on the waiting list for 2 years and was in constant pain while he continued working the year before surgery. His surgeon sent him for 7 physiotherapy sessions, after which the patient was told he was fit to return to work 6 weeks after surgery. You and your resident decide to investigate if there is any research that delves into the issue why some patients are unable to return to work after seemingly well-performed TKRs; you plan to present this case and your findings at the next orthopedic surgery grand rounds.
Literature search
As the research question explores an underlying social phenomenon — the factors influencing a patient’s decision to return to work following TKR — the ideal article type would be a qualitative study. You use MEDLINE to perform a literature search.11
Deriving keywords from the research question, you use the medical subject heading (MeSH) “qualitative research,” along with the search terms “total knee replacement” AND “return to work”; this search yields 2 qualitative studies.12,13
One of these articles is a systematic review assessing the influence of patient factors on employment following hip and knee replacement.12 Although this article references the topic of interest, it is not specific to TKR and does not cite the qualitative literature. As the data are presented from quantitative sources only, the study does not provide information regarding the patient perspective, experiences and social interactions associated with return to work following TKR. The second article is a qualitative study by Bardgett and colleagues13 published in 2016. You determine this is the only article to address the question posed in the clinical scenario. You decide to critically appraise this article.
A synopsis and the characteristics of the study by Bardgett and colleagues13 can be found in Table 1 and Table 2, respectively.
Are the results valid?
Creswell14 described 9 common characteristics of qualitative research:
Natural setting — data are collected directly from participants in the setting where they experience the phenomenon.
Researcher as key instrument — researchers collect data through direct observation, interviews and the examination of documents as opposed to questionnaires or alternative instrumentation.
Multiple sources of data — interviews, observations, and available documentation are used to collect data.
Inductive data analysis — data are organized from simple to more abstract concepts to develop conclusions and themes.
Participants’ meaning — meaning is derived from participants directly without involvement of study researchers.
Emergent design — the research process may be subject to change once data extraction begins and new information is obtained.
Theoretical lens — the study and its conclusions are viewed through the appropriate social, political, or historical context.
Interpretive inquiry — researchers and participants interpret the data based on their own personal background and prior understanding.
Holistic account — researchers use the data to identify and describe the “big picture” of the topic under investigation. 2,14
For qualitative research to be relevant to the practising surgeon, it must provide an understanding of a social phenomenon that has previously gone unrecognized or offer new insight into an already familiar area of social interaction.10
Although personal opinions and hypotheses have their own place in the literature, qualitative research represents a structured methodological approach to further one’s understanding of a social interaction or personal experience as well as develop theories regarding its cause and overall relevance.1
In qualitative research, validity (as it refers to the idea of discovering truth) is not a single, fixed, or universal concept. 15 As a result, qualitative researchers have replaced the word “validity” in favour of more appropriate terminology, such as credibility, rigour and trustworthiness.15,16
Was the formulation of the research question clearly described?
Just as in quantitative research, the research question in qualitative research tells the reader what is going to be discovered, generated, or explored.17 Often, this research question is divided into two types: the central question, which is the most general question to ask, and the subquestions, which subdivide the central question into specific questions.17 Often in qualitative research the question will look at the “how” or “what” in order to describe or understand a situation or process.17
Bardgett and colleagues13 provide a brief background into previous research in the area, stating that they focused only on the quantitative side of returning to work.13 The authors explain that a qualitative approach to the research question is needed to obtain information directly from the patients based on their attitudes and perspectives.
The central research question for the appraised article is why certain factors may influence a patient’s return to work.13 The subquestion was to look at the potential deficiencies in the delivery of care directly pertinent to return to work.13 The central and subquestions indicate that the aim of study was to gain a greater insight into the factors influencing return to work from the patient’s perspective.
Was the rationale for participant selection and observation sufficiently explained?
Qualitative studies do not have a predetermined sample size or a statistical method to assess the appropriateness of the chosen sample size.10,18 To address this, participants are selected to meet particular criteria — a process referred to as “purposive sampling.”10,18 The criteria for participant selection may evolve during the course of data extraction and analysis to include typical, unusual or important cases to explore new themes and perspectives as they emerge. To account for this, relevant information, such as religion, socioeconomic status and profession, should be acknowledged by the researcher in order to establish the appropriate context for the sample’s perspectives.19
Bardgett and colleagues13 explain that participants were taken from a cohort of 50 patients, all under the age of 60 years at the time of surgery; the patients were originally recruited from a postal questionnaire study. From those 50 patients, 37 who were employed before surgery consented to participate. Purposive sampling was then used to select 10 patients who represented “a range of characteristics known to influence rates of return to work,” including age, sex and type of employment.13 Although the authors explain why the chosen patients were used for this study, they do not indicate why a sample size of 10 patients was used.
Were data collection methods and instruments adequately explained?
As previously stated, in order to present a thorough assessment, researchers should use multiple sources; typically, qualitative studies use 3 basic techniques for data collection:
Observation — the direct or indirect surveillance of study participants10,20
Participant interviews — structured, semistructured, or unstructured discussions to enable participants to describe a phenomenon as they experience it21
Document analysis — the direct examination of relevant information linked to the topic under investigation (i.e., medical charts, operative reports).10,22
It has been suggested that in the case of an interview, the characteristics of the interviewer be given to the reader, along with any relationship that the interviewer may have with the participants.23 This information allows the reader to conclude how the interviewer could possibly influence the responses of the participants and, furthermore, how they may lead the interview.23
Bardgett and colleagues13 used a semistructured patient interview focusing on the preoperative and early postoperative phases of the patients’ journey (see Box 1 for interview questions). The authors justify this data collection method, stating that patients were able to discuss topics that were directly relevant to them, even those not previously considered by the research team. The interviews had the following limitations. First, the time at which the interview took place following surgery, which ranged from 8–35 months, introduced the possibility of recall bias; therefore, responses and recollections from the participants may have been affected. Second, the authors did not define the criteria for “early postoperative phase”; therefore, the reader is unaware of specific timelines. Finally, the reader is not given adequate information regarding the interviewer. Bardgett and colleagues13 explain that it was a research-trained physiotherapist; however, it is unknown whether this physiotherapist was a third-party individual or if he or she was treating the patients.
Semistructured interview guide from the study by Bardgett and colleagues13
Introduction
Could you start by explaining to me what your job involves?
How did your arthritis affect you/your work?
What has happened since your operation?
Was the experience after the operation what you were expecting?
What was the involvement of your employer?
Is there anything that has helped you or would have helped you to return to work more easily?
What influenced the decision to return to work at that time?
Close
In summary, Bardgett and colleagues13 used only one form of data collection, did not disclose the characteristics of the interviewer, and did not state if there was a prior relationship between the interviewer and the participants. This could introduce some inconsistency into the data and ultimately affect how the reader interprets the results.
Was the data extraction of sufficient detail and scope?
Tong and colleagues23 explain that the following information should be disclosed to readers regarding data extraction: the number of coders, a description of how the themes were derived, a mention of any software that was used for the extraction of data and a mention of whether the interviewees were able to review their transcripts.
The study by Bardgett and colleagues13 states that one author took the audio recordings and “transcribed at verbatim.” It was stated that after the first coder listened repeatedly to the recordings, a second coder revisited them in order to verify the initial codes and themes found by the first reviewer. Therefore, the article did list how many coders were involved in data extraction. In regard to how the themes were derived, Bardgett and colleagues13 stated that a thematic approach was used to identify common patterns among the interviews. The authors state that this type of analysis allowed for the identification, analysis and reporting of patterns found across the data set, while working with themes that were identified a priori.13 Although not explicitly stated, the authors did say that each transcript was coded by hand; therefore, one could assume that no software was used during the process. Finally, as there was no mention, one can assume that the interviewees did not review their transcripts.
Were the method and credibility of data synthesis, interpretation and presentation sufficiently explained?
A variety of analytical approaches may be used to interpret qualitative studies, including ethnography, grounded theory, phenomenological analysis and content analysis. Each method, although unique, ultimately attempts to develop a theory or narrative from the qualitative data.10 Although the details of each analytical method are beyond the scope of this article, in this section we review the general features of data analysis that are relevant to most methods — triangulation and data saturation.
In the context of qualitative analysis, triangulation refers to the act of verifying outcomes through multiple sources of information. Denzin24 and Patton25 identified 4 types of triangulation:
Method — the use of multiple forms of data collection (i.e., interviews, direct observation and field notes) to gather information about the same phenomenon26
Investigator — the use of two or more researchers to compare several different observations and conclusions24,26
Theory — the use of existing social science theories to substantiate or refute findings of a qualitative study10,26
Data source — using data from different participants (i.e., individuals, groups, or communities) in order to compare and contrast multiple perspectives and validate data.26
The theory behind data saturation is fundamental to qualitative research. Specifically, it refers to the point in which new data fit into an identified theory without the need for revision.10 Although there is no single approach to reaching data saturation, researchers have agreed on the following principles to identify data saturation: no new data, no new themes, no new coding, and the ability to replicate the study.27,28 Failure to reach data saturation suggests that not all relevant themes or perspectives may be represented in study outcomes, ultimately jeopardizing the credibility of the study’s results.
Although Bardgett and colleagues13 do not cite triangulation directly, they do use a form of investigator triangulation: they used multiple researchers to compare observations and conclusions, thereby strengthening the reader’s confidence that the data were interpreted and coded appropriately and instilling confidence in the results.
In regard to data saturation, Bardgett and colleagues13 state that the aim of the study was not to reach data saturation, but to “identify important key themes.” Although the authors give an explanation, the failure to use data saturation as an end point suggests that all relevant themes may not be adequately accounted for in the data and, therefore, that the credibility of these findings may be questioned.
What are the results?
Qualitative research relies on sufficient detail on the part of the study authors to elicit a clear picture of the perspectives and experiences of study participants.10 To do this, researchers may opt to supplement their analysis with excerpts from interview transcripts, observation notes and relevant documentation. Use of excerpts allows the reader to judge for themselves whether the results accurately reflect the data and assess whether the data clearly supports the study’s conclusions.10 Conversely, if the study conclusions are not represented by the data excerpts, the reader may doubt the interpretation skills of the study author or the methods for analysis.
Bardgett and colleagues13 identified three themes that influenced a patient’s return to work following TKR (Table 1). Each theme references direct excerpts from interview transcripts to substantiate the authors’ interpretations and conclusions.
Were participants’ characteristics clearly presented?
In qualitative research, a description of the basic demographic characteristics of study participants is necessary in order to properly interpret the findings of the research.23 As the act of being observed and/or reordered will undoubtedly affect study participants, it is important to know the characteristics of both the participants and the interviewer so that the reader may understand how the two groups may influence each other, ultimately affecting the study results.23
With regard to demographic information, Bardgett and colleagues13 supply the reader with information on sex, age at the time of surgery and type of employment for each participant (Table 2). This information allows the reader to interpret the results in the context of the patients’ age and employment background, all of which have been recognized as important factors influencing return to work.13
Were study-derived constructs and their credibility adequately explained?
“Constructs” refer to mental abstractions that attempt to convey meaning about a particular topic in only a few words. They provide a shared meaning that allows the study authors to communicate ideas clearly and precisely to their audiences.29 For example, the term “ageism” is a construct for prejudice or discrimination on the basis of a person’s age. As constructs are not directly observable — we cannot directly observe ageism even though we may associate ageism with different signs or actions — clear definitions are required to ensure conceptual clarity and good qualitative research.29
It has been suggested that direct quotations may be used to define and support the use of constructs in the qualitative literature.23 The incorporation of direct quotations from participants is said to add “transparency and trustworthiness” to both the raw data and to the interpretation of the findings.23 Quotations allow the reader to assess how the data (in the form of quotations) relate to the conclusions made by the study authors.
In the article assessed in our clinical scenario,13 direct quotations from participants are used to define and support each construct claimed by the authors. The themes they identified are also clearly stated, and the quotations used are well integrated with additional information supplied to the reader.13 This article also lists the study number of the respective participant after each quotation, as suggested by Tong and colleagues.23 The reader is able to see a clear association between the author’s interpretation and the interview transcript excerpt, ultimately strengthening the study findings and supporting the author’s analysis.
Were study constructs differentiated from pre-existing constructs?
As previously stated, for qualitative research to be relevant to the practising surgeon, it must provide an understanding of a social phenomenon that has previously gone unrecognized or offer new insight into an already familiar area of social interaction.10 Although empirically developed constructs do not need to agree with existing theories or beliefs, it is helpful to the reader if the authors relate these constructs to the prevailing knowledge and the existing literature. 10
Bardgett and colleagues13 effectively relate their derived themes to pre-existing constructs defined in the literature. For example, when referencing “delays in surgical intervention” as a construct affecting return to work following TKR, Bardgett and colleagues13 express how this theme relates to the “physician–patient relationship” previously established in the literature. By differentiating from preexisting constructs, authors are able to expand on theories already established in the literature.
Was the transferability/generalizability of the results discussed?
Within the qualitative literature, the term “transferability” is synonymous with external validity and refers to the application of the study findings beyond the setting in which the study was conducted.9 Given the contextual nature of qualitative research insofar that it records social interactions and personal experiences within a given setting, the reader must carefully assess the transferability of study results to other sociocultural settings.19
There were many limitations identified that may affect the transferability of the study results in the article by Bardgett and colleagues.13 First, as their study was based out of a large teaching hospital, the findings may be applicable only to severe cases requiring interventions at an academic as opposed to a community hospital or clinic. In addition, all study participants were white British citizens younger than 60 years who were working at the time of surgery. This raises the question as to whether the results could be generalized to less severe cases requiring TKR and to patients of different ethnic backgrounds, ages and employment status, which may have associated factors influencing return to work.
Were the data reported according to accepted guidelines?
The Standards for Reporting Qualitative Research (SRQR) developed by O’Brien and colleagues make up a 21-item checklist endorsed by the EQUATOR Network.3,30 The SRQR functions to improve the transparency of qualitative research by establishing a list of recommendations that should be represented in all qualitative studies.3 These standards were developed with the intent to assist readers, editors and reviewers to assess qualitative study methods and the application of its results.
Applying the SRQR to the study by Bardgett and colleagues13 shows that 20 of the 21 items are represented; “researcher characteristics and reflexivity” is the only item not reported. The article by Bardgett and colleagues has good overall adherence to the SRQR reporting standards and therefore shows sufficient transparency.
Will the results help me care for my patient?
Are the results of this study applicable to my situation?
As previously stated, there were limitations identified in the study by Bardgett and colleagues13 that may affect the transferability of data to real-world patients. It is the responsibility of individual physicians to consider how their patients may be similar to the patients referenced in the study population.
Does the study present a compelling theory?
For qualitative research to be useful, it must be believable. As a result, the utility of a qualitative study depends on the narrative and arguments it presents. To assess this, Elder and Miller10,31 proposed the following characteristics to be evaluated by the reader: parsimony (the use of minimal assumptions to explain the data), consistency (whether the study conforms with the existing literature or presents reasons for its disagreement), fertility (whether the study presents areas for future research) and clarity (whether the study’s narrative is free of redundancy, ambiguity, or contradiction).31 Simply, the reader must ask themselves whether the study makes sense and whether the account is compelling.31
The article by Bardgett and colleagues13 demonstrates consistency, fertility and clarity. Specifically, it describes its role within the existing literature, suggests areas for future research and presents a clear and concise narrative that is substantiated with patient transcriptions. Conversely, the study uses multiple unsupported assumptions to justify patient selection and therefore does not demonstrate appropriate parsimony. For example, patients younger than 60 years and working at the time of surgery may not be representative of the population seeking to return to work following TKR.
Resolution of the scenario
At the next orthopedic surgery grand rounds, you present this case and your findings to your colleagues. You suggest that although the qualitative study by Bardgett and colleagues13 identifies three key themes that may influence this patient’s experience of return to work following TKR, including delays in surgical intervention, limited and inconsistent advice among health care providers to optimise return to work and the provision of rehabilitation to optimise recovery and return to work, you admit that further analysis may be required to refine and/or revise these theories, given that the study authors did not reach the point of data saturation and investigated only a small sample at a single site. You suggest that you and your colleagues initiate a national qualitative study to assess patient factors influencing return to work following TKR.
Conclusion
The purpose of qualitative research is to better understand the social interactions and perspectives of individuals within a given setting. As no widely accepted standard for appraisal exists, readers should use this guideline (Box 2) not as a finite checklist, but rather as a guide to familiarize themselves with qualitative studies. As the critical appraisal of qualitative research may differ according to the specific methodology used, readers should reference a text specific to the qualitative study method, just as Bardgett and colleagues13 cited the article by Braun and Clarke32 as the basis for their thematic analysis, when seeking a more thorough appraisal of the literature.
Guidelines for how to assess a qualitative research article
Are the results valid?
Was the research question formulation clearly described?
Was the rationale for participant selection and observation sufficiently explained?
Were data collection methods and instruments adequately explained?
Was the data extraction of sufficient detail and scope?
Were the method and credibility of data synthesis, interpretation and presentation sufficiently explained?
What are the results?
Were participants’ characteristics clearly presented?
Were study-derived constructs and their credibility adequately explained?
Were study constructs differentiated from pre-existing constructs?
Was the transferability/generalizability of the results discussed?
Were the data reported according to accepted guidelines?
Will the results help me in caring for my patient?
Are the results of this study applicable to my setting?
Does the study present a compelling theory?
Footnotes
Competing interests: None declared.
Contributors: All authors contributed substantially to the conception, writing and revision of this article and approved the final version for publication.
- Accepted October 20, 2017.