Designing and Using Surveys in Nursing Research: A Contemporary Discussion

Abstract

This commentary summarizes the contemporary design and use of surveys or questionnaires in nursing science, particularly in light of recent reporting guidelines to standardize and improve the quality of survey studies in healthcare research. The benefits, risks, and limitations of these types of data collection tools are also briefly discussed.

The use of research questionnaires or surveys in nursing is a long standing tradition, dating back to the 1960s (Logan, 1966) and 1970s (Oberst, 1978), when the scientific discipline emerged. This type of tool enables nursing researchers to gather primary data from a specific population, whether it is patients, carers, nurses, or other stakeholders to address gaps in the existing evidence base of a particular clinical, pedagogical, or policy area. However, the recent creation of a checklist for reporting survey studies called CROSS: Consensus-Based Checklist for Reporting of Survey Studies, hints at problems in their design, development, administration, and reporting (Sharma et al., 2021). This commentary discussion focuses on the process of developing, validating, and administering surveys in nursing research and some ways to strengthen this methodological approach.

Ideally, surveys should be constructed to gather the minimum amount of information from respondents to provide good quality data about a problem or phenomenon. Gathering large amounts of unnecessary data may complicate a survey, leading to a low response rate and weak findings. Therefore, time and expertise is needed when designing research surveys (Robb & Shellenbarger, 2020). Firstly, existing evidence should be reviewed to identify if an existing survey could be utilized or refined. The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN), and associated database and critical appraisal checklist, could be employed to examine the psychometric properties of an established tool and its methodological quality before use (Mokkink et al., 2016). For instance, Charette et al. (2020) followed the COSMIN approach when conducting a systematic review of the psychometric properties of scales that assessed new nurses clinical competence.

If a new instrument needs to be developed, then reviewing relevant literature could help inform what should be measured for example, what people know, think, feel, or do, along with guiding the content of specific survey questions (Polit & Beck, 2020). Other techniques can be employed to create questions including a Delphi study to gather expert opinion (Bender et al., 2018), or focus groups with patients, clinicians, educators, students, or policy makers (Tajik et al., 2010). Decisions about the style of survey questions also needs consideration as each brings advantages and disadvantages. An open question gives a respondent free reign with their answer which could uncover fresh insights on a topic. However, it may contribute to respondent fatigue if too many are asked, and the data can be time consuming to analyze (O’Cathain & Thomas, 2004).

How each question is worded is also important to avoid leading, composite, or presumptive questions, ones that are vague, overly lengthy and complex, or include double negatives, jargon, or terminology unfamiliar to the reader, so that what is being asked and answered is clear and consistent. The sequence of questions should also be logical, opening with more general non-threatening questions, followed by more specific ones that can be grouped or filtered accordingly, and closing with socio-demographic variables and a thank you (Boynton & Greenhalgh, 2004). Closed fixed choice questions can be formulated in a number of ways including checklists, frequency or Likert-type scales, Guttman or cumulative scales, Thurstone scales, and rankings, which vary in their content, structure, and layout, and require either a dichotomous or multiple-choice response. The sensitivity of any measurement scale is important to ensure it accurately represents the respondents answer and reduces the risk of bias (Polit, 2014). Hence, piloting a draft survey with a small sample of intended respondents can help identify problems with ambiguous content, the format of questions, or confusing instructions or layouts.

The validity and reliability of a survey instrument should also be established to demonstrate the questions are worded appropriately and illicit accurate answers. Validity is about accuracy, in terms of how well a survey measures what it is supposed to. It can be assessed in three ways: (1) face or content validity, (2) construct validity, and (3) criterion validity (Rattray & Jones, 2007). Content validity looks at comprehensiveness, and whether questions adequately measure all aspects of a subject matter. For example, Devriendt et al. (2012) examined the content validity of the Safety Attitudes Questionnaire through expert clinician review and using the content validity index and a modified kappa index. Construct validity focuses on whether the concept(s) that underpin the questions in a survey correspond with contemporary theory and scientific knowledge. For instance, McSherry et al. (2002) employed factor analysis to determine construct validity for a Spirituality and Spiritual Care Rating Scale. Numerous research studies are often required to evaluate and refine the construct validity of a survey instrument to ensure it is robust. Some go further and investigate both convergent and discriminate validity, the two sub-types of construct validity (Hallett et al., 2018; Zhao et al., 2020)

Criterion validity refers to how much the scores in a survey measure agree with a gold-standard that is considered an ideal measure of the constructs or variables. This approach is not always feasible, if there are no reliable measures for independent comparison (Polit & Beck, 2020). It can be done in two ways, the first by calculating a correlation coefficient which tests the strength of the association (not agreement) between the results from a survey and an external independent measure. Secondly, sensitivity and specificity can be calculated, although there is usually a trade-off between the two (Groves, 2009). Sensitivity measures the ability of a survey to detect all instances of its subject matter (true positives), while specificity measures the ability of a survey to discriminate all instances of its subject matter from those which are not related (true negatives). Both false negatives and false positive errors may occur, so the nature of the research and survey instrument should guide which type of error should be minimized as much as possible (Dillman, 2014). The two types of criterion validity, concurrent and predictive validity, can also be measured. Concurrent validity compares survey questions or items to a related validated measure, both of which are assessed at the same time, whereas predictive validity compares survey items against some criterion measure at a later time (Kim & Abraham, 2016). While validity testing can be time consuming, expensive, and require a significant amount of statistical expertise, it is a robust way to develop and improve the psychometric properties of surveys.

The other major concept used to evaluate the quality of surveys is reliability which focuses on the consistency of a survey and its items, to ensure it would give the same results if repeated under the same conditions (Rattray & Jones, 2007). The three kinds of reliability testing are: (1) test retest, (2) inter-rater, and (3) internal consistency. Test retest looks at consistency of a measure across time and whether survey results from the same person were the same on at least two or more occasions. This can be measured using a number of statistical techniques such as the intraclass correlation coefficient and Wilcoxon signed rank test (Lovén Wickman et al., 2019). Inter-rater reliability examines the consistency of a measure across raters or observers to determine if a person scores items in a survey in the same way multiple times. Cohen’s kappa (Dancey et al., 2012) and the intra-class correlation coefficient (Ryu et al., 2013) are common statistical measures for this. Finally, internal consistency is how consistently respondents’ answer items in a survey, if pairs of questions measuring the same concept are asked in different ways which can be calculated using Chronbach’s alpha (Paans et al., 2010). Although reliability may be established and the survey results reproducible, this does not mean they are valid and may be incorrect unless the instruments’ validity is also determined.

Once a survey is designed, it then needs to be administered to the appropriate population, once ethical approval is granted. Self-completing surveys are the most common as they can quickly and easily be given to a large population using online, electronic or paper methods which are affordable options. An interview administered survey is an alternative, where questions are answered in the presence of a researcher, if sensitive topics need to be discussed, if vulnerable populations need to be reached, or if a survey is long and complex. A Cochrane review of interventions for administering postal and electronic questionnaires reported several strategies such as utilizing stamped addressed envelopes, financial incentives, and personalized communications, were effective in increasing the response rate (Edwards et al., 2009), as low response rates can negatively impact the results of survey studies. After data is gathered, verified, cleaned, and anonymized, it needs to be coded using suitable analyses. Epi Info™ (https://www.cdc.gov/epiinfo/index.html) is a popular tool for entering and storing survey data before exporting it to a statistical analysis package (Kebede et al., 2017).

Finally, surveys are frequently published in scientific nursing journals. However, Sharma et al. (2021) highlighted the substantial variability and inconsistency in how research surveys are reported which can weaken the evidence base on a topic. They emphasized that despite two guidelines for reporting non-web and web-based surveys, SURGE (Grimshaw, 2014) and CHERRIES (Eysenbach, 2004), improvements in the reporting of survey research have not materialized and these tools have limitations. Hence, a new comprehensive survey reporting guideline called CROSS was developed to enhance the transparency and replicability of survey based health research (Sharma et al., 2021). This new guideline should be used in nursing to enable survey studies to be better designed, conducted, and reported. By undertaking rigorous, high-quality surveys, researchers can advance nursing science, strengthen the evidence base on which nurses practice, and help make a positive impact on patient care and health service delivery.