Using Qualitative Methods to Assess the Measurement Property of a New HIV Disability Questionnaire

Abstract

The purpose of this article is to describe our experience using a qualitative team approach and predetermined theoretical framework to assess sensibility of a newly developed HIV disability questionnaire. Two interviewers conducted structured qualitative interviews with 22 adults living with HIV, asking participants how well the questionnaire characterized the disability they experienced living with HIV. Data collection and analysis occurred over six stages with four analysts who met throughout. Strengths of our approach included the ability to assess the sensibility of the questionnaire from the perspective of adults living with HIV, collect and analyze data across multiple sites, establish a systematic team analytical process, and enhance rigour through multiple coding, team reflexivity, and interviewer and analyst triangulation. Challenges included increased resources required to coordinate and implement this approach, differential recruitment rates, initial divergent analytical styles, and the potential to miss emerging codes given the structured nature of the analysis. This article offers a methodological process for researchers to use a qualitative team approach with directed content analysis to assess the sensibility of a new health status questionnaire.

The Context: Assessing the Sensibility of a New Questionnaire

With longer survival, people living with HIV are facing increased health-related consequences of HIV, its associated treatments, and concurrent health conditions (Blanch et al., 2002; Gaidhane et al., 2008; Henderson, Safa, Easterbrook, & Hotopf, 2005; Kiser & Pronovost, 2009; Palella et al., 1998; Rusch et al., 2004a, 2004b; Weiss, Osorio, Ryan, Marcus, & Fishbein, 2010; Willard et al., 2009). Together, the physical, mental, emotional, and social health-related challenges experienced by adults living with HIV can be conceptualized as disability. Adults living with HIV defined disability as symptoms and impairments, difficulties carrying out day-to-day activities, challenges to social inclusion, and uncertainty or worrying about the future that can fluctuate on a daily basis and over the entire course of living with HIV (O'Brien, Bayoumi, Strike, Young, & Davis, 2008). Measuring disability is important for determining its prevalence and impact and for identifying possible interventions to reduce or prevent disablement.

As part of a larger research study, we set out to assess the measurement properties of a newly developed HIV Disability Questionnaire (HDQ) designed to describe the presence, severity, and episodic nature of disability experienced by adults living with HIV. Measurement properties are characteristics of an instrument that should be evaluated prior to use with a given population. Sensibility is often first assessed to determine whether an instrument is meaningful to respondents. This is defined as “a mixture of ordinary common sense plus a reasonable knowledge of pathophysiology and clinical reality” (Feinstein, 1987, p. 144). The concept of sensibility was derived from Feinstein's work on clinimetrics, a discipline dedicated to the quality of clinical measurement (Feinstein, 1987). According to Feinstein, components of sensibility include the following: purpose and framework, overt format, face and content validity, and ease of usage (Feinstein, 1987). Purpose and framework addresses issues of clinical function, justification, and applicability of the questionnaire. For example, it addresses questions such as: What is this instrument supposed to do? Why is this instrument needed? In what situation is this instrument applied? Overt format addresses the way in which the instrument is presented. Face validity assesses whether the questions asked are consistent with the purpose of the instrument, appraises whether questions are asked clearly and to the right person, and evaluates whether the instrument as a whole is coherent (e.g., Do the questions make sense?). Content validity refers to the suitability in the selection and aggregation of components, including omission of important items, inclusion of unsuitable items, suitability of response options, and quality of the data that are collected. Ease of usage assesses ease of administration and use of the instrument from the perspective of participants and researchers, including elements of time, risk, and efforts involved in organizing data to show results of the instrument. Because no mathematical formulae for assessing sensibility exist, evaluation is often based on the judgment of clinicians and individuals for whom the questionnaire was developed (Feinstein, 1987). Hence, qualitative methods are ideal to assess the sensibility of new measures. We used Feinstein's theory as a framework to conduct structured qualitative interviews with adults living with HIV and clinicians who work in HIV care in order to assess the sensibility of the HDQ.

Qualitative Content Analysis

Numerous approaches to qualitative data analysis exist. Content analysis is one analytical technique commonly used to analyze text data (Cavanagh, 1997). The goal of content analysis is to provide knowledge and understanding about a phenomenon (Downe-Wamboldt, 1992). Qualitative content analysis may be used to systematically classify large amounts of text data into categories (Weber, 1990). Classification of data is done through the process of coding and identifying themes or patterns (Hsieh & Shannon, 2005). Hsieh and Shannon (2005) identified three approaches to qualitative content analysis that differ based on how initial codes are developed. Conventional content analysis is used in studies aimed to describe a phenomenon where codes are derived from the data. Directed content analysis involves a deductive and structured approach where an existing framework or theory informs the initial coding scheme. The coding scheme consists of pre-existing conceptual categories that are applied to the data. Summative content analysis involves quantifying words or content within the data to explore the extent to which content is used (Hsieh & Shannon, 2005).

In particular, directed content analysis can be useful for researchers conducting qualitative studies in teams with multiple members involved in data collection and analysis. Benefits of team approaches to qualitative research can include breadth of expertise, perspective, and insight among members, although challenges can include increased demands for coordination and communication and conflicting roles and analytical interpretations (Fernald & Duclos, 2005; Hall, Long, Bermbach, Jordan, & Patterson, 2005). In situations where data collection occurs across different sites, several researchers may be involved in recruiting participants, conducting interviews or focus groups, and engaging in data analysis. Because the quality of information obtained can be dependent on the interviewer, and the analysis influenced by a researcher's position and assumptions, a study with multiple researchers may result in variable data collected and diverse analytical interpretations (Patton, 2002). While some might consider this a source of bias, it may provide an opportunity for triangulation, reflexivity, and multiple coding that can heighten study rigour, conceptual analysis, and interpretation (Barbour, 2001; Barry, Britten, Barber, Bradley, & Stevenson, 1999). Hence, the structured nature of directed content analysis can offer a systematic and organized approach for teams conducting qualitative research.

Directed content analytical techniques were an ideal qualitative approach to assess the sensibility of our new questionnaire for adults living with HIV. This approach allowed us to apply Feinstein's components of sensibility as initial codes for analyzing the interview data (Feinstein, 1987). Our study took place across two geographical sites to obtain perspectives from both large and smaller urban cities. Hence, we pursued a team approach to carrying out data collection and directed content analysis in order to assess the sensibility of the questionnaire. To our knowledge, this article is the first to describe using a qualitative team approach to assess the measurement properties of a questionnaire.

Purpose and Objectives

The purpose of this article is to describe our experience using a qualitative team approach to assess the sensibility of the newly developed HIV Disability Questionnaire (HDQ). Specific objectives are to (1) present our team approach to data collection and qualitative directed content analysis (methods) and (2) discuss the strengths and challenges associated with this approach (results and discussion). Rather than present the sensibility assessment findings of the HDQ, which are published elsewhere (O'Brien et al., 2013), our primary focus is to describe our process of using a qualitative team approach for directed content analysis of interview data. By providing a detailed account of our methodological process, other researchers will be better informed to consider this approach when assessing the sensibility of a newly developed questionnaire.

Methods

We recruited participants using maximum variation sampling, a sampling mode of choice in order to “document unique variations that have emerged in adapting to different conditions” (Lincoln & Guba, 1985, p. 200). The aim is not to focus on similarities that can develop into generalizations, but to detail the specific variations that give the context its unique flavour (Lincoln & Guba, 1985). We recruited adults over the age of 18 living with HIV who self-identified as having experienced at least one health-related challenge attributed to their illness. Participants were from acute care hospitals, a specialty care hospital, and a number of local AIDS service organizations across two cities in Ontario, Canada. We specifically targeted our recruitment to maximize variation across sites to obtain representation from participants living in both large and smaller urban cities and to assess HDQ sensibility across varying geographical contexts. We obtained written informed consent from each participant. This research was approved by Research Ethics Boards at McMaster University, Hamilton, Ontario, and St. Michael's Hospital, Toronto, Ontario, Canada.

Data Collection

Two interviewers in different sites administered the HDQ and conducted structured qualitative interviews with 22 adults living with HIV (11 in each city). Interviews were held in hospital clinics and AIDS service organizations. Interviews ceased at the point when we collectively determined no new codes were emerging from the ongoing interviews and responses were becoming repetitive with previous interviews.

The HDQ is a 70-item self-administered questionnaire comprised of four domains that represent the dimensions of disability: symptoms and impairments (35 items), uncertainty (15 items), difficulties carrying out day-to-day activities (8 items), and challenges to social inclusion (11 items). Each item possesses a 7-point ordinal response scale asking how the individual is feeling that day (disability presence and severity), and a nominal response scale asking whether the health-related challenge fluctuated (either worsened or improved) within the past week (episodic nature of disability). Example items from each of the four domains include: “I feel too tired to do my usual activities” (symptoms and impairments); “I worry about my future health living with HIV” (uncertainty); “I have difficulty walking” (difficulties with day-to-day activities); and “My ability to work or attend school is limited because of my illness” (challenges to social inclusion). The final item asks the respondent to classify his or her health on the day of HDQ completion as either a “good day” or “bad day” living with HIV. A cover sheet includes the definition of disability, an explanation of the purpose of the HDQ, an explanation of how responses will be used, and instructions for how to complete the questionnaire (O'Brien, Bayoumi, King, Alexander, & Solomon, manuscript submitted for publication).

Interview guide

The two interviewers used an interview guide to facilitate consistency in their overview of the study and interview approach with participants (see Appendix for Interview Guide). The interview guide was developed by the lead author using Feinstein's theory of sensibility, and then it was reviewed by the other members of the research team prior to implementation.

Interviewers introduced the HDQ to participants as a new HIV-specific questionnaire developed to describe the presence, severity, and episodic nature of disability (or health challenges) that people might experience living with HIV. Interviewers indicated that they were meeting with participants, who included people living with HIV, and clinicians who work in HIV care to obtain feedback on the HDQ. Participants were provided with an overview of the definition of disability. Interviewers then asked participants questions to (a) determine how well the questionnaire characterized disability and (b) identify ways to refine the questionnaire to better capture their disability experience. Interviewers specifically asked participants about the way in which the HDQ was structured (e.g., adequacy of items and response options, readability, clarity of instructions, and overall format), whether the questionnaire adequately characterized their disability experience, and whether it captured the potential episodic nature of disability over time (see Appendix). Interviews ranged from 30 minutes to 1 hour and 20 minutes in duration. Interviews were digitally recorded and transcribed verbatim by a transcriptionist. All transcripts were audio-checked by each interviewer for accuracy. Participant names or any other information that could identify participants was deleted from the interview transcripts. All information remained strictly confidential and available only to study investigators and research staff.

Directed Content Analysis

We conducted a constant comparative analysis where data collection and directed content analysis occurred simultaneously. We used Feinstein's theory of sensibility to develop a preliminary coding scheme for analyzing the interviews (Feinstein, 1987). The coding scheme included categories related to face and content validity of the questionnaire (items and response options), item generation (missing items that should be added), item reduction (repetitive or redundant items), ease of usage (literacy level, feasibility, format, and length of time to complete), episodic nature of disability, title of the questionnaire, and overall recommendations for revisions. Categories in the preliminary coding scheme, specifically those related to face and content validity, included four additional subcategories, one for each domain of the HDQ. The original coding scheme is available upon request. Data management was facilitated using NVivo7 qualitative software.

Four authors (two primary analysts and two secondary analysts) met formally on three occasions to discuss analysis of the interviews, assess consistency in coding, revise the interview guide, and refine the coding scheme. The two interviewers (TB and MS), who were also the primary analysts, communicated informally throughout data collection and analysis. We refer to the interviewers and primary analysts as Analyst A and B, and the secondary analysts as Analyst C and D. Our data collection and analysis occurred in six stages (Figure 1).

Figure 1.

Overview of the qualitative team approach to assessing the sensibility of the HIV Disability Questionnaire.

Stage 1

All four analysts coded the transcripts from the first four interviews (two from each city) using the preliminary coding scheme developed from Feinstein's theory of sensibility. They met to discuss overall impressions of the interviews and revise the coding scheme to ensure consensus in coding the elements of sensibility (Figure 1).

Stage 2

All four analysts recoded the first four interview transcripts plus six new interview transcripts (three from each city) using the updated coding scheme for a total of 10 interviews. The analysts met on a second occasion to formally discuss the coding of these transcripts and their analytical interpretations and to finalize the coding scheme for use with the remaining transcripts. At this time, a few minor revisions were made to the interview guide based on interpretations and team discussion, which included adding a question that asked participants about their overall impressions after completing the HDQ, specifically whether the HDQ evoked any discomfort; adding a question about the optimal time frame in which to ask about the episodic nature of disability in the HDQ; and adding a probing question about whether the term “disability” should be removed from the questionnaire title.

Stage 3

Twelve additional interviews with adults living with HIV were conducted (six in each city) for a total of 22 interviews. The primary analysts (A and B) each coded the transcripts from the 11 interviews he/she had completed using the final coding scheme. Primary analysts were in frequent communication throughout via telephone and Internet to discuss overall impressions of the interviews, clarify queries related to coding, and ensure consistency in applying the codes. These communications also allowed the analysts to cross-check and validate any discrepancies or uncertainties in the coding across the two cities.

Stage 4

Using the qualitative software, coding reports were generated for each city. The reports were merged based on coding category into a single dataset that included all coded data from both cities (Figure 1). The merged dataset included 10 main coding categories (A to J), each containing data from all 22 interviews (Figure 2).

Figure 2.

Ten main category codes in the finalized sensibility coding scheme.

Stage 5

The dataset was randomly divided into two sets of five categories so that each primary analyst received all data for five of the 10 coding categories. Each primary analyst cross-checked the interview transcripts and summarized the data within his or her assigned five categories. This ensured that half of the coded data by each primary analyst was cross-checked by the other primary analyst.

Stage 6

The team met for a third time to discuss the completed directed content analysis for the 22 interview transcripts across coding categories. We discussed our overall interpretations of the findings related to the sensibility of the HDQ and recommendations for future HDQ revision. During this meeting we reflected on the strengths and challenges associated with our collaborative team approach.

Results and Discussion

Our qualitative team approach to data collection and directed content analysis was novel for assessing sensibility of a new HIV disability questionnaire. Using qualitative methods allowed us to explore sensibility directly from the perspective of adults living with HIV. In addition to comments on face and content validity, format, and ease of usage, participants provided insightful recommendations for future HDQ revision to increase its applicability and feasibility for use in research and clinical settings. Results of the sensibility assessment are published elsewhere (O'Brien et al., 2013). Our focus in this article is to describe the strengths and challenges of our qualitative team-based approach.

Strengths of the Qualitative Team Approach to Directed Content Analysis

Qualitative research has a role in developing and assessing properties of a new instrument (Morse, 2010). Guidelines on the development of patient-reported measures highlight the importance of qualitative research that engages the target population in generating items and evaluating item relevance, wording, readability, recall period, response options, format, instructions, and respondent burden of a new questionnaire (U.S. Department of Health and Human Services Food and Drug Administration, 2006). Clinimetric approaches that include patients and clinicians in instrument development and property assessment can be useful for establishing measures of complex multi-dimensional health constructs (such as disability), enhancing the content validity, interpretability, feasibility, and relevance of an instrument to the intended target population (Marx, Bombardier, Hogg-Johnson, & Wright, 1999; Terwee et al., 2007).

The structured and systematic nature of directed content analysis was an ideal analytical approach to use with a team of researchers. This methodological approach has advantages for researchers planning analysis of qualitative interview or focus group data with a predetermined theoretical framework, such as sensibility.

First, having two researchers conduct the interviews facilitated data collection across multiple geographic sites. This reduced travel costs and time that would be required for one interviewer to travel between cities to recruit participants and conduct the sensibility interviews. Although our decision to have two researchers collect data in the different sites was a pragmatic one, it led to other unanticipated strengths associated with a team approach to data collection and directed content analysis.

Second, a qualitative study with two or more interviewers has the potential to decrease bias that may occur from a single person collecting all the data, a process known as interviewer triangulation (Patton, 2002). Despite our structured interview guide, each interviewer approached the interviews with his/her own prior assumptions and experiences. The interviewers differed in characteristics such as gender, age, and previous HIV research experience. As a result, the two interviewers used different interview techniques, probing questions, and interpretations during the interview, which allowed us to collect a more diverse range of responses. Bringing together researchers with different expertise, assumptions, and knowledge bases can heighten the credibility of data collected and quality of the research (Hall et al., 2005; Opie, 1997). Ongoing communication between the interviewers throughout the data collection enabled the team to reflect on their prior assumptions and experiences and address any inconsistencies as the data were collected and analyzed. Additionally, interviewers were able to share tips on interview facilitation, debrief about challenging interviews, discuss overall impressions from the interviews, and help clarify the meaning of certain responses and how they related to the overall sensibility assessment of the HDQ.

Third, having a team of analysts independently analyze, cross-check, and compare findings as a larger group enabled multiple coding, analyst triangulation, and researcher reflexivity (Barbour, 2001; Mays & Pope, 2000; Patton, 2002). All four analysts coded the first 10 transcripts and met formally on three occasions to discuss their interpretations. Two primary analysts reviewed the coding of all 22 interviews for five of the 10 main coding categories and cross-checked their coding according to the final coding scheme. Coding agreement between analysts was not essential; rather, group discussion led to heightened data interrogation and added thoroughness to the analysis (Barry et al., 1999; Berends & Johnston, 2005). Group discussion can result in team reflexivity where members share their experiences, assumptions, and interpretations, dually enhancing intellectual rigour as well as team morale (Barry et al., 1999). This team-based reflexivity is termed working “separately together,” where individuals work as a team to advance from separate perspectives to a common and collective understanding of a given concept (Siltanen, Willis, & Scobie, 2008). Our approach enabled team members to share individual impressions of the interviews, cross-check the coding of the data, and collectively refine the interview guide and coding scheme. Our experience aligns with evidence that team approaches to qualitative research can enhance methodological rigour of design, analysis, and interpretation (Barry et al., 1999; Berends & Johnston, 2005). Compared with individuals working alone, teams can better establish a standardized coding process with enhanced accuracy (Delaney & Ames, 1993), incorporate different perspectives (Liggett, Glesne, Johnston, Hasazi, & Schattman, 1994), and bring the analysis to a richer level of conceptual thinking and interpretation (Barry et al., 1999; Oleson, Droes, Hatton, Chico, & Schatzman, 1994).

Fourth, Feinstein's theory of sensibility provided a strong foundation from which to successfully implement our team approach to data collection and analysis. The framework provided a foundation for our interview guide and coding scheme—six of the category codes in our coding scheme (purpose, face validity, content validity, ease of usage, response sets, and format) were derived from the framework whereas the remaining four category codes (overall impressions of the HDQ, episodic component, title, and additional emerging themes for consideration) were added specifically to further inform the refinement of the HDQ. Using this framework enabled us to comprehensively assess the sensibility of the HDQ (O'Brien et al., 2013).

Finally, with our team of analysts, it was imperative to establish a systematic process for data management and coding at the outset of the study. As a result, we collectively developed the coding scheme at our first meeting after all four analysts coded the first four interview transcripts. After the first 10 interview transcripts were coded by all four analysts and the final coding scheme was established, the two primary analysts (and interviewers) proceeded more independently to code the remaining 11 transcripts, communicating informally as needed throughout. Establishing a systematic approach to data collection, management, and coding early on in the research process is important, particularly for multi-site studies with large teams (Hall et al., 2005; Liggett et al., 1994).

Challenges with the Qualitative Team Approach to Directed Content Analysis

While our qualitative team approach successfully achieved the sensibility assessment of the HDQ, it was not without challenges. First, increased time, effort, and costs were required to coordinate and implement the team approach to qualitative interviews and directed content analysis. The team approach is time-consuming because all researchers are engaged in all aspects of the research process (Siltanen et al., 2008). Having four researchers engaged in coding lengthened our timelines for analysis. We scheduled formal meetings in addition to informal communication to allow us to check for any inconsistencies in our interview approach and analysis and to make decisions on revising the coding scheme prior to moving forward. We believe the extra time invested was offset by the increased methodological rigour, which had the potential to increase the credibility of the results (Barry et al., 1999; Fernald & Duclos, 2005). Our study occurred across two sites with four researchers; however, the team approach may be more challenging for larger studies across multiple sites with more researchers (Guest & MacQueen, 2007). Researchers will need to determine the feasibility of using our qualitative team analytical approach in relation to the number of geographic sites, sample size, and number of individuals on the research team (Fernald & Duclos, 2005).

Second, our recruitment and interview completion occurred at different rates across the sites. One analyst had completed and coded more interviews than the other analyst and was left waiting for recruitment at the other site to reach the same sample size of participants. In our view, this was unavoidable because recruitment typically occurs at different rates in and across sites. We scheduled our meetings accordingly and did not find that the difference in recruitment rates hindered the progress of the study.

Third, team members initially approached this study with disparate analytical styles and expectations. Our first team meeting was essential to reinforcing the study purpose, determining a systematic process for data management, analysis, and interpretation, and establishing methods for communication throughout the study (Hall et al., 2005). Subsequent team meetings allowed us to periodically evaluate our team process, review timelines, roles, and responsibilities, and share overall analytical interpretations (Hall et al., 2005).

Fourth, applying a structured approach to data collection and directed content analysis may have resulted in missing emerging aspects of sensibility related to the HDQ. This can occur with any structured type of qualitative analysis and might not be unique to our approach. Researchers should remain open to the possibility of new categories not initially captured in the coding scheme that may emerge from the content analysis. Despite the structure of directed content analysis, any text that cannot be categorized with an initial coding scheme should be given a new code (Hsieh & Shannon, 2005). We had a category “emerging themes for consideration” in our coding scheme to ensure we captured any new emerging themes that related to our questionnaire assessment. Ongoing communication and documentation between the primary analysts allowed important emerging themes to be incorporated into the coding framework throughout (Fernald & Duclos, 2005).

Finally, while our methodological approach worked well for our study objective, directed content analysis is only recommended for deductive approaches to qualitative analysis, where an existing theoretical framework informs the development of a structured interview guide and the subsequent coding scheme (Hsieh & Shannon, 2005). Although we provided an example used in a measurement study assessing sensibility, directed content analysis may be used by researchers engaging in other types of qualitative studies with a predetermined theoretical framework.

Conclusions

The HIV Disability Questionnaire is the first HIV-specific disability questionnaire developed and assessed from the perspective of people living with HIV. Qualitative methods, and specifically a team approach to data collection and directed content analysis, were ideal for assessing the sensibility of a newly developed questionnaire. Strengths of our approach included the ability to assess sensibility directly from the perspective of adults living with HIV, collect and analyze data across multiple sites, establish a systematic team analytical process, and enhance rigour of data analysis and interpretation through multiple coding, team reflexivity, and interviewer and analyst triangulation. Challenges included the increased time and resources required to coordinate and implement this approach, differential recruitment rates, initial divergent analytical styles, and the potential to miss emerging codes because of the structured and deductive nature of the analysis.

This methodological process provides an opportunity to bridge the quantitative fields of measurement with qualitative methodology, specifically in terms of outlining the important role of qualitative methods in assessing the measurement property of a new health status instrument. Researchers developing new measures may use similar qualitative team approaches to assess sensibility of new and existing health status questionnaires. This methodology also highlights the role of individuals who will complete the questionnaire in the future, in our case people living with HIV, participating in the assessment of measurement properties of health status instruments.