Future Research Considerations for Mixed-Mode Surveys

Most surveys conducted today use more than one mode of outreach and/or data collection to make contact, gain cooperation, and/or collect responses. The use of mixed-mode surveys has continued to grow, fueled by the overall proliferation of surveys, the rise of web surveys and other digital technologies, people’s communication and mode preferences and tools to restrict how they can be contacted, the COVID-19 pandemic (that especially impacted in person outreach and data collection), declining survey response rates, and rising data collection costs. As a result of this continued surge in mixed-mode surveys, the literature and research on how to best design, conduct, and analyze mixed-mode surveys has also continued to grow as shown by books including Internet, phone, mail and mixed mode surveys: The tailored design method (Dillman et al. 2014) and Mixed-mode official surveys: design and analysis (Schouten et al. 2022) and a special issue of the Journal of Survey Statistics and Methodology “Recent Innovations and Advances in Mixed-Mode Surveys” (S. Coffey, Maslovskaya, and McPhee 2024), as well as the increase in articles on mixed-mode surveys in the Journal of Official Statistics.

Nonetheless, there are still numerous challenges with mixed-mode surveys and critical areas that need further research to enable additional efficiencies and innovation. The following sections present key areas for future research and development of mixed-mode surveys related to recruitment and outreach strategies, adaptive and responsive designs, survey system considerations, measurement and mode effects, adjustment methods to address selection and measurement effects, and the emergence of new technologies and modes. Taken together these present a comprehensive research agenda for mixed-mode surveys across the survey lifecycle from design and data collection to analysis and estimation.

Recruitment and outreach approaches that use multiple modes prompted many of the early mixed-mode designs. Substantial research has shown how using multiple modes of contact, and in particular alternating between different modes, can help improve response rates and often reduce nonresponse bias by bringing in respondents with one mode that may not have responded to another (Dillman et al. 2014). Using multiple modes or different modes for some sample members, also helps respond to people’s mode preferences (Smyth et al. 2014) thus increasing their likelihood of responding and potentially the quality of the data they provide. Since there are a variety of ways to mix modes and new modes continue to emerge, additional research is needed on how to best sequence modes as part of a recruitment strategy, and how that might vary for different populations and depending on what contact information may be available.

In these efforts, it is important to focus on experimentation and testing of overall survey designs, where the full set of survey procedures and contacts are considered in combination. Since survey elements and procedures interact and work together to encourage people to respond and reduce total survey error (Dillman et al. 2014), it is beneficial to understand how all the design elements operate in combination, and not just focus on individual procedures without understanding how they relate to other features. Often, research focuses on testing specific design elements, but that can lead to differential results across experimental studies or when implementing the individual features within a broader survey design, likely due to how these features interact with other aspects of the design. More testing and experimentation are needed on overall survey designs, like these studies that experimentally compare various types of mixed-mode designs (Olson and Smyth 2022; Wells et al. 2024). As our survey designs have become even more complex, it is essential to evaluate how aspects of the recruitment and data collection strategy work together in combination.

More and more mixed-mode surveys will continue to move to adaptive and responsive designs in the next decade. Early research focused on using frame data to tailor initial design approaches for different groups but was often limited by the availability of frame data, as well as outcome data to train and test the models (Jackson et al. 2023), and thus resulted in varied results. Linking other data sources to survey frames will continue to be important for expanding available contact information to enable various modes of outreach and increasing sample member characteristics available for tailoring features of the design. These approaches show promise for longitudinal designs where often comprehensive data about key demographic and other characteristics, as well as information on effort and outcomes, are available from prior waves (Watson and Cernat 2023).

Identifying the best design for a survey involves carefully considering how the design impacts quality and costs. Olson et al. (2021) have focused on better understanding the cost of mixed-mode surveys, using cost indicators that can be compared across modes and ideally across organizations. More recent approaches are focused on balancing quality and cost by using adaptive and responsive designs to make decisions at the case-level by analyzing impacts to measures like bias and variance and estimated data collection costs to identify cases that have a low propensity to respond, high potential data collection costs, and have minimal impact to bias and variance (Coffey and Elliott 2024; Wagner et al. 2024). Furthermore, Coffey, Damineni, et al. (2024) provide a framework for incorporating adaptive survey design approaches to non-survey data, such as administrative data, that will be important as our mixed-mode surveys evolve further into data systems that include survey data along with non-survey sources including device data, biomeasures, administrative data, and other sources. The future of this work is critical to developing effective mixed-mode designs that meet quality and cost needs, where the optimal effort is expended for the quality desired.

A related important area for further innovation in mixed-mode surveys are the survey data systems needed for tracking cases and outcomes across modes. Substantial progress has been made in this area, but more effort is needed to enable seamless management of cases across modes and detailed understanding of contact attempts and outcomes for all modes. Paradata, questionnaire data, and metadata need to be collected, tracked, and harmonized across modes. Processes including cleaning rules, adjustments, and weighting should be performed on a regular cadence during data collection, often daily or weekly, rather than waiting until the end of key phases or overall data collection. Data science techniques can further enable the automated testing and processing of data to ensure each step, rule, and model are deployed and working as expected. Real time harmonized data across modes is critical to enable the more dynamic adaptive and responsive designs mentioned earlier and ultimately effective innovations and improvement in our mixed-mode survey designs.

One of the principal aspects of mixed-mode surveys that needs further research is measurement differences and mode effects. Measurement errors, related to mode, can occur because of differences in how respondents provide answers in different modes. There is a growing understanding of how questions are perceived differently across modes, how that can impact the question-answer process, and that the presence or absence of an interviewer can have significant impacts on how people answer certain questions (Dillman et al. 2014; Schouten et al. 2022). Furthermore, there is research on how questions may be constructed differently across modes, and how these (even small) differences can influence how people respond (Dillman and Christian 2005). In addition, there have been studies that illuminate how specific question topics or designs are more likely to lead to mode effects (Christian et al. 2007; Schork et al. 2021; Smyth et al. 2008; Sparkman et al. 2024), but more research is needed in this area, including device effects within web surveys where respondents use a variety of smartphones, tablets, laptops, or other computers for completing surveys.

Furthermore, although some of the differences in how people respond across modes are known, further investigation is needed on how to minimize or correct for them. This entails two broad areas of research around questionnaire design and adjustment methods for addressing mode effects. Dillman et al. (2014) discuss the importance of designing questionnaires consistently across modes, including using the same question wording and response options, while still leveraging mode-specific features to improve measurement within specific modes (e.g., automated skip patterns and fills in web surveys that cannot be included in paper surveys). However, more exploration is needed especially on impacts for different question topics and how to handle more complex questions (e.g., industry and occupation, activity diaries, and employment or education history). Similarly, additional research is needed on how to potentially correct for mode effects, building on the early work in this area (Kolenikov and Kennedy 2014).

A critical area for future research is on adjustment methods and techniques for correcting selection and measurement effects in mixed-mode surveys. One of the biggest challenges with these techniques is developing statistical models that can properly correct for selection and measurement effects, even when these are often confounded (where some respondents answer via one mode and others via another mode). Investigation continues to grow in this area (Sakshaug et al. 2022; Schouten et al. 2022; Yu et al. 2024), however these often rely on benchmarks from other surveys or bridge surveys conducted as part of transitioning modes that require substantial cost and suffer from limitations or errors themselves. Additional investment in large-scale data collections that can be used to develop and test models for multiple surveys and types of estimates is ideal to help improve efficiency and make these types of approaches more accessible. Additionally, considerably more research is needed on techniques that could be applied without benchmarks, bridge surveys, or other additional data collection. Relatedly, many of the current techniques require optimizing for a single or a few key variables, whereas surveys produce numerous estimates of interest, and adjustment or analytic methods that are estimate specific can be very difficult for data users to incorporate in their analyses. Exploration of approaches that can be applied more broadly across multiple survey estimates, rather than being variable specific, and across various types of analytic approaches (means, proportions, model-based coefficients) will be key to the future of mixed-mode data analysis and usage.

New technologies and modes continue to emerge that have the potential to improve our mixed-mode surveys but require research on their effectiveness and how best to integrate them with other modes before being fully incorporated into our survey designs. For example, the past few years have seen increased research on texting in surveys, especially on the use of text messages for outreach and to encourage completion of web surveys (Cabrera-Álvarez and Lynn 2024; L. Christian 2024; L. M. Christian et al. 2024) but also as a data collection tool, especially for simple requests that can benefit from the speed of text-based data collection (Spiegelman et al. 2024). Relatedly, there has been an expansion of research around multi-device surveys and how smartphones can enable various types of contact (phone, text, email) and data collection approaches (web, phone, text, passively tracking behavior, etc.), but also introduce potential additional measurement considerations.

Video interviewing is a new method mostly introduced during the COVID-19 pandemic where interviewers conduct surveys live with respondents using video conferencing or other similar tools. Early research suggests that the quality of data from video interviewers can be comparable to that from in-person surveys but also are susceptible to interviewer effects and social desirability bias like other interviewer-administered modes (Endres et al. 2023; Hansen et al. 2024; West et al. 2022). In addition, there are design and operational considerations that need to be considered when conducting video interviews (Schober et al. 2020) and barriers and challenges to getting people to schedule and complete video interviews (Hillygus 2024) that need further research.

The more recent advent of artificial intelligence (AI) has generated research using large language models (LLMs) in web-based surveys for motivation, prompting, or other uses, including incorporating human-like interviewers (Wuttke et al. 2024) or more passive AI-powered chatbots (Barari et al. 2024). Substantial research is still needed on the potential implications for the respondent experience, completion rates, measurement, and data quality among others. As these and other technologies and modes emerge, additional research will be needed before integrating them into our mixed-mode survey designs.

Although the survey landscape is facing many challenges, these problems also provide exciting opportunities for research and innovation on continued improvements to our mixed-mode surveys and approaches. The research areas covered here should help survey practitioners consider some of the key issues facing mixed-mode surveys and how they can conduct and share research within these areas to improve our knowledge and ultimately the data we rely on for official statistics.