Sage Journals: Discover world-class research

Abstract

Panel attrition poses major threats to the survey quality of panel studies. Many features have been introduced to keep panel attrition as low as possible. Based on a random sample of refugees, a highly mobile population, we investigate whether using a mobile phone application improves address quality and response behavior. Various features, including geo-tracking, collecting email addresses and adress changes, are tested. Additionally, we investigate respondent and interviewer effects on the consent to download the app and sharing GPS geo-positions. Our findings show that neither geo-tracking nor the provision of email addresses nor the collection of address changes through the app improves address quality substantially. We further show that interviewers play an important role in convincing the respondents to install and use the app, whereas respondent characteristics are largely insignificant. Our findings provide new insights into the usability of mobile phone applications and help determine whether they are a worthwhile tool to decrease panel attrition.

Keywords

panel app panel attrition interviewer effects consent survey quality refugee

Panel surveys are one of the most valuable data sources in the social sciences. Compared to cross-sectional studies, they provide a stronger basis for causal inference, as data analysis can be based on within-variation at the respondent or household level—thereby controlling for unobserved time-constant heterogeneity (Brüderl & Ludwig, 2015). However, panel surveys also face specific threats to data quality, one of which is panel attrition. Panel attrition describes the fact that some sample elements drop out of the panel over time, for instance, because they are unwilling to further participate in the study or because the field agency loses track of respondents when they move. We use a panel study on refugees, an extremely mobile population, as a case study to identify whether a mobile phone application can reduce panel attrition and in turn increase data quality.

One group that is prone to cause panel attrition in particular are recent immigrants. Recent immigrants move more frequently due to job search or leaving preliminary living arrangements. As immigration in the past decades has increased drastically (due to accelerated globalization of markets and humanitarian crises), many large-scale and renowned panel studies have incorporated targeted migration and minority samples in their surveys over the last couple of years. For example, in 2016, the German Socio-Economic Panel (SOEP), an annually conducted nationwide panel study (Goebel et al., 2019), implemented a refugee-specific enlargement sample, focusing on asylum seekers and refugees who have arrived in Germany since 2013 (Kühne et al., 2019, further prominent examples are the Panel Study on Income Dynamics, and the UK Household Longitudinal Study).

As refugees comprise a population that is exceptionally mobile due to official regulations and allocation procedures, panel attrition and address quality were expected to be a serious issue. Thus, in addition to SOEP standard panel care measures, a mobile app was developed in order to improve address quality, maintain interest in the study, and, in turn, decrease panel mortality. The app incorporates five features:

Allowing respondents to inform about changes to their address;

Collecting respondents’ email addresses;

Geo-tracking of respondents’ devices;

Implementing short questionnaires; and

Providing news on the situation of refugees in Europe.

The article at hand aims to analyze

whether the usage of a mobile phone app decreases panel attrition,

which factors affect the consent to using a mobile phone app,

and which role the interviewers play in convincing participants to use an app.

With these research objectives, we build on and extend earlier research that primarily focuses on consent and data quality regarding mobile phone apps in survey research (Keusch, Leonard, et al., 2019; Sugie, 2018). In order to answer the above-mentioned questions, first, we describe the self-selection of respondents into downloading the app along with the permission to activate geo-tracking and the usage of email addresses for recontacting purposes. Second, we explore interviewer effects regarding consent and app usage. Finally, we assess address quality for the second wave and analyze whether the app and its features reduce the number of unknown whereabouts and increase the willingness to participate in the second wave of the Institute for Employment Research (IAB)–Federal Office for Migration and Refugees (BAMF)–SOEP Survey of Refugees.

Literature Review

In this section, we give an overview of the different strands of literature this article makes a contribution to. We begin with panel attrition, followed by recent developments in the usage of mobile phones in survey research and consent to linking survey and digital trace data. In the end, we will discuss research concerning interviewer effects.

Panel Attrition

Panel attrition describes the phenomenon that some respondents or households drop out of the panel over time. In many existing large-scale panel surveys, panel participation rates are rather high when compared to initial first wave or cross-sectional study response rates (RR). For instance, in the SOEP, the participation rate is above 90% from Wave 3 onward (Kroh et al., 2018). The same holds true for the UK Understanding Society longitudinal study (Lynn et al., 2012), the Michigan Panel Study of Income Dynamics (Sastry et al., 2018), the Swiss Household Panel (Lipps, 2007), and the HILDA Survey (Watson & Wooden, 2006). While attrition rates remain fairly low in many existing panel surveys, their cumulative impact can result in a serious loss in sample size over time. For instance, a 10% attrition rate (i.e., a 90% participation rate) decreases an initial Wave 1 sample size of 5,000 households down to 3,281 in Wave 5 (−34%).

Generally, panel attrition is caused by three main factors: (1) missing/invalid information needed to recontact respondents such as addresses or telephone numbers, (2) individual loss of interest in the study and, in turn, refusing to participate in the next wave or in general, and (3) death and morbidity.

Research shows that the reasons underlying panel attrition are not fully random and some subpopulations are more likely to drop out of the panel. Factor 1, changes in information needed to recontact panelists, is most frequent in highly mobile populations, that is, participants who move often and across large distances (e.g., Trappmann et al., 2015). This holds true not only for younger people but also especially for refugees as they represent a rather mobile population (Buber-Ennser et al., 2016; Kühne et al., 2019). With respect to Factor 2, a loss in interest and, thus, refusal, there is indication that recent migrants and refugees are more likely to refuse further participation in the panel as they experience a high respondent burden due to language and cultural issues (e.g., Siegers et al., 2019). Finally, regarding Factor 3 (and fairly trivial), older panelists are much more likely to drop out of the panel due to their demise (for a general overview of the reasons underlying panel attrition, also see Schoeni et al., 2013).

Nonrandom panel attrition poses major problems for maintaining quality in panel surveys (Lugtig, 2014). Not only does panel attrition decrease the sample size, and thus, statistical power, but it also introduces potential nonresponse bias into estimates if those who drop out of the panel systematically differ from those who further participate. Although there are a vast number of post hoc measures to account for nonrandom panel attrition by means of weighting adjustment (e.g., Rendtel & Harms, 2009), such means are not a magic bullet (e.g., Vandecasteele & Debels, 2007). In particular, they reach their limits if panel attrition is constantly high, sample sizes become too small, or some groups of the target population drop out (almost) completely. Therefore, field agencies have introduced several measures to keep attrition as low as possible: There is evidence that sending out letters and postcards or even providing monetary incentives are helpful in this regard (Castiglioni et al., 2008; Laurie, 2008; Schoeni et al., 2013). Additionally, in face-to-face surveys, interviewer continuity is proven to maintain high participation rates (Campanelli & O’Muircheartaigh, 1999; Lynn et al., 2014; Vassallo et al., 2015) and response quality (Kühne, 2018). However, most of these measures seek to keep interest in the study high, and the improved address quality only appears as an accidental side effect. High address quality includes up-to-date information on residential addresses and only a small number of unknown whereabouts or nonexisting addresses. Especially in the case of (face-to-face) panel surveys, maintaining address quality is a crucial aspect in order to recontact households and respondents. Ongoing panel studies show that address quality is a smaller problem than respondent motivation (Couper & Ofstedal, 2009; Siegers et al., 2019). Nevertheless, research suggests that this not necessarily holds true for mobile target populations such as immigrants (Couper & Ofstedal, 2009). In contrary to native-born respondents, immigrants tend to move back and forth between their country of origin and other third countries. Recent immigrants, on the other hand, are more likely to move within one country for job-seeking purposes or leave the country when their residence permits expire.

In addition to maintaining large sample sizes, there is a second reason why knowing the actual place of residence is important for panel studies. In order to correct for panel attrition by means of nonresponse weighting, field agencies need to differentiate between quality neutral dropouts (no longer part of the target population) and dropouts that are still eligible to participate. In some national surveys, people who leave the country are considered to be data quality neutral dropouts, as they are no longer part of the underlying target population. Thus, in order to estimate unbiased nonresponse weights, knowledge about residency is important.

As panel attrition poses a severe problem to survey quality, survey providers constantly look for ways to decrease attrition, one of which can be the use of mobile phone applications with the purpose of keeping in touch with the panelists.

Usage of Mobile Phones in Survey Research

Since the 1990s, digitalization of survey practice is increasing (Groves, 2011). While self-administered web surveys on a computer can already be considered state of the art, mobile phones are still not widely used (e.g., de Bruijne & Wijnant, 2014b; Poggio et al., 2015, p. 5; Stier et al., 2019). Smartphones have a high potential utility in different aspects. For example, smartphones can deliver small questionnaires between the main waves of panel studies, or they replace computer-assisted and pen-and-paper interviews in general (Keusch, Leonard, et al., 2019; Sugie, 2018). Furthermore, they are able to gather digital trace data about survey participants. Most smartphones provide hardware that allows, among other features, voice identification and geo- and eye-tracking (Höhne & Schlosser, 2019; Lane et al., 2010). However, substantial research and the development of theoretical frameworks on how survey data and digital trace data can be integrated sustainably have only recently gained the attention of survey methodologists, and thus, research on this matter is comparably scarce (for an overview, see Stier et al., 2019).

Despite its great potential, evidence regarding the utility of smartphones and apps in social science research is mixed and many challenges remain. To begin with, presumably not all members of a target population own a smartphone (Keusch, Bähr et al., 2020). Therefore, field institutes need to find solutions that avoid underrepresenting substantial parts of the target population such as the elderly or poor (Fuchs & Busse, 2009; Hoogeveen et al., 2014). In order to tackle this challenge, Hoogeveen et al. (2014) suggest that the field institutes should, if necessary, supply mobile phones and solar chargers (to remain independent from energy sources). Additionally, from a technical perspective, mobile phone applications for the purpose of survey research need to be designed in a way that they run on a vast array of operating systems (Dalmasso et al., 2013).

Generally speaking, the evidence regarding data quality of surveys conducted via smartphones is quite mixed. Compared to standard web surveys conducted on and designed for a computer, some scholars report that surveys designed for mobile phones reach a similar quality level (Ballivian et al., 2015; Dillon, 2012; Topoel & Lugtig, 2014; Wells et al., 2013). Others report substantial problems regarding data quality including poor RR and the detail and length in open-answer response formats (de Bruijne & Wijnant, 2013).

Additionally, lower completion rates are shown to be a problem (Mavletova, 2013), along with social desirability (Mavletova & Couper, 2013) and the interview length (Couper & Peterson, 2016). Studies assessing the reasons for lower quality come to the conclusion that the individual characteristics of the respondents are not the only factor, but rather the deviating presentation of a questionnaire on a mobile phone, such as smaller pictures and truncated scales, shows an impact (Couper & Peterson, 2016; Peytchev & Hill, 2010). Dillon (2012), on the other hand, finds only small differences between smartphones and other modes of data collection.

Additionally to using mobile phone applications for interviews, smartphones are also used to implement geo-tracking of individuals, as the majority of current devices are equipped with GPS sensing (for a discussion see Bähr et al., 2020). Besides gathering information on individual movement, researchers are increasingly able to use geo-spatial data to learn about, for instance, the respondents’ place of work, or their leisure activities (Colak et al., 2016).

However, employing such technology raises the question of selectivity. For example, in a study by Toepoel & Lugtig (2014), only about a quarter of all participants agreed to apply geo-tracking (see also Kreuter et al., 2018, who report similar consent rates). Furthermore, younger participants, people with high income, and respondents living in small households were more inclined to use such an application (Topoel & Lugtig, 2014, p. 551). This is in line with findings of de Bruijne and Wijnant (2014a), who report that besides the young, especially progressive thinking respondents use mobile phones to answer web surveys.

One crucial aspect of selectivity is the informed consent to linking survey data to additional data sources. The effect of consent to linkage has thus been the subject of many analyses in the past to determine the loss in survey quality when linking survey data to additional data sources.

Consent to Linking Survey Data to Additional Data Sources

A considerable amount of literature has been published on consent to linking survey data to additional data sources. Most of this work focuses on record linkage, meaning the connection between surveys and administrative records. In our study, we focus on consent to linking survey to mobile phone data (Keusch, Struminskaya, et al. (2019) provide a comprehensive overview of the current research.) In most countries, using a mobile phone or an app to gather additional information requires informed consent from the respondents and their agreement to current data protection regulations (e.g., Kreuter et al., 2018). Past research on informed consent in surveys indicates that receiving consent to data linkage, in general, and to geo-tracking, in particular, is by no means trivial, introducing a vast array of potential sources of bias and errors (see Couper et al., 2017, for an overview).

Previous research shows that similarly to consent to administrative data, also consent to use a mobile phone app for data collection is selective. Moreover, consent rates vary from study to study. For example, Couper and colleagues (2017), in their review, found that consent rates range between 5% and 67%. Similarly, Jäckle and colleagues (2017) report selectivity regarding smartphone usage within the Understanding Society Innovation Panel. For the purpose of their study, an application used the camera, measured physical movements, and had access to GPS sensing. Results show that not only the consent in downloading the app but also the consent to the various features was selective (Wenz et al., 2019). Similar results are presented based on the Dutch Longitudinal Internet studies for the Social Sciences panel (Elevelt et al., 2019) and based on the German PASS, a panel study on Labor Market and Social Security (Kreuter et al., 2018).

Drawing on experiences from consent to record linkage, some efforts have been made in order to reduce errors introduced by data linkage. Sakshaug et al. (2013) propose placing consent questions at the beginning of the interview. Additionally, respondents are more likely to give consent when they are promised a reward such as a shorter interview (Sakshaug & Kreuter, 2014). A further feature that proves to increase consent rates is confidentiality. If the field agency is perceived as trustworthy and respondents believe that the data is handled with care, they are more inclined to agree to linkage (Singer, 1993). On a similar note, scholars report that respondents who have a high sense of privacy tend to refuse consent more often (Fobia et al., 2019). For example, respondents who do not want to report their income are likely not to give consent to data linkage (Sakshaug et al., 2012; Sala et al., 2012). Additionally, if respondents believe that there is a greater benefit resulting from linkage, consent rates increase as well (Dunn et al., 2004; Fobia et al., 2019; Kahnemann & Tversky, 1979, 1984; Kreuter et al., 2015).

Interviewer Effects in Survey Data

Despite their comparatively high costs compared to other modes of data collection such as web surveys, social scientists still rely on interviewers. Interviewers contribute to data quality by maintaining high participation rates, handling complex survey instruments, and providing assistance and clarifications to respondents (Fowler & Mangione, 1990). However, interviewers can also introduce interviewer effects in the data (see Schaeffer et al., 2010; West & Blom, 2017, for an overview of the topic). These interviewer effects introduce variance and/or bias into survey estimates, thus threatening the reliability and validity of survey measures.

Turning to panel participation and attrition, research shows that interviewers vary in their ability to motivate respondents into further participation in a panel (Haunberger, 2010; Pickery et al., 2001; Vassallo et al., 2015). So far, however, existing research cannot identify a robust set of interviewer characteristics associated with higher participation rates. One exception seems to be interviewer experience, with more experienced interviewers obtaining higher (re-)participation rates (Durrant et al., 2010; Hox & de Leeuw, 2002).

Moreover, interviewers also vary in their ability to obtain consent to (record) linkage requests (Eisnecker & Kroh, 2017; Korbmacher & Schröder, 2013; Sakshaug et al., 2017, 2012; Sala et al., 2012). Sakshaug et al. (2017) use an experimental approach to show that consent rates are generally higher in interviewer-administered surveys (face-to-face, 93.9%) compared to those that are self-administered (mail/web, 53.9%). Are certain interviewer characteristics associated with higher consent rates? Sala et al. (2012) indicate that the interviewer’s sociodemographic characteristics, personality traits, and attitudes to persuading respondents do not affect consent. Only the interviewer’s experience within a survey wave has a significant negative effect on consent rates (which is in line with Korbmacher & Schröder, 2013). Similarly, Sakshaug et al. (2012) also show little evidence for systematic interviewer effects. Moreover, they do not indicate that matching respondents and interviewers based on demographic characteristics increases consent. Korbmacher and Schröder (2013) find a U-shaped interviewer age effect on consent rates (with a turning point at about 55 years).

Regardless of interviewer characteristics associated with consent rate, the literature points to substantial interviewer variance in consent rates (Korbmacher & Schröder, 2013; Mostafa & Wiggins, 2017; Sakshaug et al., 2012). For instance, Korbmacher and Schröder (2013) observe interviewer intraclass correlation levels on the consent decision of over 40% even after controlling for respondent- and interviewer-level characteristics.

Thus far, the literature lacks empirical evidence related to interviewer effects on the respondents’ decision whether to download a mobile app, provide an email address, or consent to geo-tracking. However, given the research on the interviewers’ influence on respondents’ consent to record linkage, expecting interviewer effects seems plausible.

In summary, the literature shows that panel attrition is a severe problem. Mobile phone applications might help decrease panel attrition because they have considerable potential in gathering additional data on survey respondents (e.g., GPS locations). Moreover, mobile phone applications are able to provide respondents with information and questionnaires. Although previous research has shown that consent to using mobile phone apps is selective, there is little evidence on the reasoning behind this selectivity (an exception is the recent publication by Keusch & Bähr et al., 2020). Previous research on face-to-face interviews and interviewer effects gives rise to the hypothesis that interviewers play a crucial role. Additionally, to date, most survey apps have been used for analytical purposes. However, it remains unclear whether these apps are also capable to improve survey quality by decreasing panel attrition. We contribute to the current literature on the usage of smartphones in surveys in two ways: First, we assess aspects of self-selection into using the app. Second, we analyze whether such an app is capable of reducing panel attrition.

The IAB-BAMF-SOEP Survey of Refugees

In order to answer our research questions, we employ data from the IAB-BAMF-SOEP Survey of Refugees (Kühne et al., 2019). The IAB-BAMF-SOEP Survey of Refugees is a random sample and panel study of asylum seekers and refugees (called anchor persons) and their households who migrated to Germany between 2013 and 2016. The sample is drawn from the Central Register of Foreigners (AZR) in Germany (von Gostomski & Pupeter, 2008) and is the result of a joint project between the German IAB, the research centre of the German BAMF, and the German SOEP. The AZR comprises all people without a German passport who live in Germany for longer than 3 months and stores information about their residence permit. Hence, the IAB-BAMF-SOEP Survey of Refugees is a unique source of data that allow inference on the 2013–2016 cohort of refugees in Germany. The first wave RR (RR2 according to American Association for Public Opinion Research [AAPOR] standard definition) of this survey is approximately 50% (Kühne et al., 2019).¹ We work with the first and second waves (2016 and 2017), comprising 4,465 individuals living in 3,273 households for the first wave and 3,343 individuals living in 2,303 households for the second wave (RR2 = 69%).² Fieldwork for the first wave took place from July to November 2016, and fieldwork for the second wave was from September 2017 to March 2018 (see Table A1 in the Online Appendix for detailed sample characteristics in terms of, e.g., age, gender, and education).

Design of and Consent to Using the App

Each anchor person was asked for consent to download and use the mobile phone application at the end of the personal questionnaire in the first wave (N = 3,273). Due to the random sampling of anchor respondents, this approach subsequently leads to a random allocation of requests within households. The respondents were asked and guided to install the app right away in order to ensure that participants do not forget about the app. The consent to the various features however did not necessarily take place during the interview so that respondents would not feel pushed. The request for geo-tracking and providing an email address for recontacting purposes was implemented separately within the app and thus not part of the initial consent request (for the wording, see the Online Appendix. Figure A1 in the Online Appendix additionally shows a picture of the app design). All aspects of the application (including the informed consent) were in line with the General Data Protection Regulation standards at that time in Germany (2016).

Ultimately, the application comprised five main features:

Geo-tracking of respondents’ devices;

Collecting respondents’ email addresses;

Allowing respondents to inform about any address changes;

Answering short questionnaires (rewarded with €20); and

Providing news on the situation of refugees in Europe.

Geo-tracking and the provision of email addresses were supposed to facilitate the tracking of respondents in the event that they moved and did not actively provide their new address. Address changes and the provision of email addresses could be introduced into the app through a separate feature that was available at all times. The GPS tracking allowed to allocate the respondent in the case that a participant could not be located during fieldwork for the second wave. Implementing short questionnaires and offering information on the situation of refugees in Europe sought to keep interest in the study high. From July 2016 to May 2017 (between Wave 1 and Wave 2 interviewing), a total of 27 different news articles and four small questionnaires were provided through the app (see Table A2 in the Online Appendix for the headlines of the news articles. The questionnaire is also shown in the Online Appendix). The small questionnaires and the news outlets were not presented at once but implemented one after the other in between the fieldwork of Wave 1 and Wave 2.

Table 1 indicates that roughly 40% of the respondents agreed to download the app. However, only 35% explicitly declined consent, whereas a quarter of the respondents either did not understand the consent question, had no smartphone, or encountered technical issues.

Table 1.

Consent to Download Application.

Outcome	% (n)
Consent	39.1 (1,280)
No consent	35.2 (1,152)
Did not understand the question	12.2 (400)
No smartphone	10.3 (337)
Unclear/no further information	3.2 (104)
Total	100 (3,273)

Furthermore, as Figure 1 indicates, only 60% of those who gave consent actually installed the app on their smartphone (thus, overall, 23% of the total sample). In addition, of those who installed the app, only 57% provided their email address, 65% allowed geo-tracking, and 57% provided their address. Interestingly though, of those 485 who allowed geo-tracking, only 231 (48%, not displayed as a table) actually activated their GPS signal between panel waves so that the field agency could use the provided information. Therefore, the consent rate that actually translates into GPS tracing is substantially lower. Furthermore, 51% of those who consented but did not install the app encountered software problems during the download. Reports from the field reveal that most technical problems occurred because the software could not be installed due to outdated or unfitting hardware. In cases where technical problems were only minor or a matter of comprehension, the interviewer helped in resolving issues (n = 33). Despite consenting, the remaining 49% did not download the app without providing further information on the underlying reason.

Figure 1.

Installation of the app and consent to different features.

Translation of Field Instruments and Interviewer Allocation

All field material (including the app) was translated from German to English, Standard Arabic, Farsi/Dari, Pashto, Kurdish-Kurmanji, and Urdu (see Table A3 in the Online Appendix for the distribution in the sample and how many respondents could choose their mother tongue). All field material was first translated to English. Either English or German (depending on the availability of translators) was then the basis for the translation into the other languages. Two translators working separately conducted a one-way translation and then produced a reconciled final version.

At the beginning of each interview, respondents had to choose a language that could not be changed during the course of the interview.The mode of the interview was either Computer Assisted Personal Interview (CAPI) or (Audio) Computer Assisted Self-Interview [(A)CASI], defaulting to CAPI. However, in the event that the interviewer did not speak the chosen language and the interviewee did not speak German, the respondent could fill out the questionnaire himself/herself. Additionally, audio recordings of each question were provided to address other language barriers, like illiteracy. In this event, the interviewers did not leave, rather they provided general guidance (see Jacobsen, 2018, for more details).

As all field material was available in the most important languages and self-interviewing was an option, the field institute did not rely on interviewers capable of speaking the most common mother tongues. The most important interviewer selection criterion, according to the field agency, was that the interviewers were experienced in working as interviewers prior to this study. Additionally, all interviewers underwent a thorough training and preparation, especially in regard to cultural and refugee-specific particularities.

A total of 95 interviewers interviewed 3,273 households. On average, each interviewer interviewed 34 households (median: 24). About 65% of the interviewers were male. The mean age of the interviewers was 50 (median: 52). About 70% of the interviewers had no prior experience as interviewers for the fieldwork agency. On average, the interviewer’s experience amounted to 1.8 years.

Estimation of Self-Selection into Downloading the App and Its Utility Regarding Panel Attrition

Our analysis consists of two steps. First, we aim at estimating a potential self-selection of respondents into (a) consent to download the app, (b) consent to geo-tracking, and (c) provision of an email address. Second, we estimate whether using the app and its various features improves (a) address quality and (b) participation in the next wave. For all models, we rely on hierarchical, multilevel logistic regression analysis with respondents clustered in interviewers. In each case, the dependent variable represents a dummy taking the value 1 if a respondent consents $(y_{i} \in {0, 1})$ , allows geo-tracking, participates, etc., and the value 0 otherwise. For instance, within the multilevel logistic regression model, the probability of a respondent i, nested in interviewer j, to download the app is a logistic function:

Pr (y_{i} | x_{1 i}, \dots, x_{k i}) = \frac{exp (β_{0} + β_{1} x_{1 i j} + \dots + β_{k} x_{k i j} + ς_{j})}{1 + exp (β_{0} + β_{1} x_{1 i j} + \dots + β_{k} x_{k i j} + ς_{j})},

where $x_{1 i j}, \dots, x_{k 1 j}$ denote the predictor variables. The parameters $β_{0}, \dots, β_{k}$ are the regression coefficients and have to be estimated. $ς_{j}$ denotes the Level 2 residuals.

Estimation of Interviewer Effects

In many surveys, including the SOEP, interviewers are not randomly allocated to respondents or households, but rather to geographic areas, the primary sampling units. Often, only a single interviewer is working in a given area, and interviewers are allocated to one area only. In these cases, estimated interviewer effects are likely to be confounded with area effects. Answers observed by a single interviewer may be more homogeneous not because of the interviewers’ (biasing) effects on responses but due to the homogeneity of individuals living in the same geographic area. However, research shows that interviewer effects are usually larger than area effects (e.g., Schnell & Kreuter, 2005). In order to minimize potential confounding with area effects, we include a variety of interviewer- and area-level fixed effects in order to mitigate endogeneity of area/interviewer selection effects (as in Korbmacher & Schröder, 2013; Kühne, 2020; West et al., 2013). A full list of all interviewer-level and area-level variables used as controls in the models is presented in the Online Appendix in Table A4.

While biasing effects of interviewer characteristics are captured in regression coefficients, the interviewers’ contribution to the variance of a survey measure—the interviewer variance—is usually quantified by estimating the intra-interviewer correlation coefficient (Kish’s $ρ_{int}$ ), representing the share of variance that is due to the interviewer (Kish, 1962). It is derived as the share of interviewer-level variance compared to the total variance that is decomposed into interviewer-specific variance $(s_{u_{j}}^{2})$ and the individual respondent residual variance $(s_{e_{i j}}^{2})$ :

ρ_{int} = \frac{s_{u_{j}}^{2}}{s_{u_{j}}^{2} + s_{e_{i j}}^{2}} .

Explanatory and Control Variables

Since the usage of an app in a survey of recent immigrants is rather new, we assess the self-selection with an explorative strategy regarding demographic and immigrant-specific variables, language proficiency, and respondent burden. As demographic characteristics, we use gender, age, employment status, education, and country of origin. In order to estimate language effects, we further include German proficiency and whether the questionnaire was available in the respondents’ mother tongue. We assume that being proficient in the official language in a foreign country increases trust. Third, we test for immigrant-specific features such as residence status (asylum seekers might not want to allow tracking in case of a rejection of their asylum application) and the year of immigration. We chose the year of immigration because we assume that newly arrived refugees might feel more obligated to participate. In order to observe the effect of respondent burden, we also include the total length of the interview in the model and the number of other individuals in the household, which are part of the target population (since this would increase the time, the interviewer spends in the household). Additionally, we insert a variable counting the amount of unanswered questions in the survey prior to the consent questions. For a detailed description of all variables on individual level, see Table A5 in the Online Appendix.

In addition to the explanatory variables at respondent and household level, a variety of controls are included in all models. At the geographic area level, we include information about the county (e.g., unemployment rate, share of university students), the municipality (e.g., size, share of population 65+), and the neighborhood (e.g., type of house, purchasing power index; see Table A4 in the Online Appendix for more information about area controls). At the interviewer level, we control for gender and age (see Table A5 in the Online Appendix at the bottom). Additionally, for models analyzing recontacting and reparticipation, we test whether the interviewer changed between the waves in 2016 and 2017.

Key explanatory variables for the models testing address quality and participation in the next wave are four dummy variables (1 = yes, 0 = otherwise), reflecting the usage of GPS, providing their email address, successful download of the app (to test the effect of short questionnaires and news outlets), and providing info about address changes through the app. The explanatory variables of the previous models serve as additional control variables.

Results

Table 2 displays three different models estimating (1) consent to download the app, (2) consent to provide geo-tracking, and (3) consent to provide an email address. In the first model, we see that the respondent-level variables play an important part in explaining consent to download the app. Regarding sociodemographics, the model shows that men and young respondents have a higher chance of consenting to download the app. However, education, the country of origin, and employment status are not correlated. Regarding language barriers, the estimation indicates that chances to consent increase when the survey language matches the mother tongue and when respondents are proficient in German (compared to a total lack of language proficiency). Regarding migrant-specific variables, the table shows that respondents who just recently migrated have a substantial and significantly higher chance of consenting. Further, the type of residence permit is not crucial in this regard. Finally, if the survey took more time, people had a higher chance of complying and the bigger the household, the higher the chance to consent as well. Surprisingly, the number of unanswered questions during the interview is not correlated with giving consent. We expected otherwise as missing values are often the consequence of high respondent burden, sensitivity, or low comprehension; all of which are factors we assumed to be correlated with giving consent.

Table 2.

Explaining Consent to Download the App, Geo-Tracking, and Providing an Email Address.

Variable (Consent)	Consent to Download		Geo-Tracking		Email
Variable (Consent)	Odds Ratio	[95% CI]	Odds Ratio	[95% CI]	Odds Ratio	[95% CI]
Gender
Female (ref.)	—	—	—	—	—	—
Male	1.53**	[1.17, 2.01]	0.87	[0.43, 1.78]	1.42	[0.77, 2.61]
Age
Old (ref.)	—	—	—	—	—	—
T(w)eens	1.84***	[1.40, 2.42]	0.56	[0.30, 1.07]	1.10	[0.63, 1.92]
Language proficiency
Good (ref.)	—	—	—	—	—	—
Intermediate	0.88	[0.64, 1.23]	1.40	[0.69, 2.85]	0.50*	[0.26, 0.95]
Poorly	0.63*	[0.44, 0.91]	2.26	[0.95, 5.34]	0.59	[0.28, 1.25]
Legal status
Other (ref.)	—	—	—	—	—	—
Residence permit	0.94	[0.70, 1.27]	2.40*	[1.14, 5.08]	0.52*	[0.27, 0.97]
Mother tongue available as survey language?
No match (ref.)	—	—	—	—	—	—
Match language	2.14***	[1.59, 2.88]	1.26	[0.60, 2.63]	1.74	[0.88, 3.42]
Duration of interview
Short (ref.)	—	—	—	—	—	—
Medium	2.81***	[2.00, 3.94]	1.48	[0.56, 3.91]	0.51	[0.20, 1.28]
Long	4.16***	[2.81, 6.14]	0.99	[0.34, 2.82]	0.53	[0.19, 1.44]
ISCED (condensed)
<Primary (ref.)	—	—	—	—	—	—
Primary education	1.31	[0.71, 2.42]	0.29	[0.04, 2.32]	2.39	[0.45, 12.60]
Secondaray education	1.89*	[1.03, 3.48]	0.29	[0.04, 2.23]	6.97*	[1.32, 36.74]
Bachelor’s+	2.44**	[1.28, 4.66]	0.16	[0.02, 1.26]	12.43**	[2.26, 68.32]
Employment status
Unemployed (ref.)	—	—	—	—	—	—
Employed	0.82	[0.57, 1.19]	1.15	[0.52, 2.52]	0.85	[0.41, 1.77]
Year of immigration
Before 2014 (ref.)	—	—	—	—	—	—
2014	2.01**	[1.32, 3.06]	1.82	[0.67, 4.94]	1.30	[0.50, 3.40]
2015	2.12***	[1.40, 3.22]	3.45*	[1.24, 9.62]	1.11	[0.43, 2.88]
2016	4.19***	[1.89, 9.29]	0.99	[0.16, 6.04]	0.84	[0.17, 4.12]
Household size
One person (ref.)	—	—	—	—	—	—
Two persons	1.54**	[1.15, 2.08]	0.97	[0.49, 1.91]	1.17	[0.63, 2.19]
Three persons+	1.56	[0.84, 2.91]	1.50	[0.33, 6.92]	0.29	[0.07, 1.19]
Country of origin
Syria (ref.)	—	—	—	—	—	—
Afghanistan	0.48**	[0.31, 0.74]	0.35	[0.11, 1.10]	0.98	[0.37, 2.59]
Iraq	0.99	[0.65, 1.50]	0.66	[0.25, 1.74]	1.17	[0.48, 2.82]
Eritrea	1.00	[0.54, 1.85]	3.33	[0.42, 26.20]	1.71	[0.45, 6.57]
Balkan	1.11	[0.57, 2.17]	1.42	[0.24, 8.44]	1.11	[0.22, 5.66]
Iran/Pakistan	1.10	[0.58, 2.11]	0.58	[0.13, 2.73]	0.36	[0.09, 1.42]
Other	1.05	[0.67, 1.64]	1.45	[0.50, 4.16]	1.45	[0.57, 3.72]
Amount of unanswered questions
No missing	—	—	—	—	—	—
<10 Questions	0.94	[0.69, 1.29]	0.80	[0.39, 1.67]	1.40	[0.75, 2.62]
>10 Questions	0.83	[0.59, 1.14]	1.29	[0.57, 2.88]	1.84	[0.93, 3.66]
Interviewer gender
Male (ref.)	—	—	—	—	—	—
Female	0.95	[0.43, 2.13]	0.24	[0.05, 1.24]	0.51	[0.14, 1.93]
Interviewer age	1.05**	[1.02, 1.08]	1.01	[0.95, 1.07]	0.99	[0.95, 1.04]
Constant	0.02		32.95		3.58
n _respondents	2,209		688		688
n _interviewers	85		66		66
ICC/Kish’s $ρ_{int}$	0.44		0.70		0.58

Note. CI = confidence interval; ICC = intraclass correlation coefficient; ISCED = International Standard Classification of Education.

*p < .05. **p < .01. ***p < .001.

Looking at the correlates regarding the consent to geo-tracking and the provision of email addresses, two effects stand out. It is striking that respondents with a residence permit have a significantly higher chance of giving consent to geo-tracking and providing an email address. It is plausible that asylum seekers without a residence permit and people with a suspension of deportation avoid being tracked in case of a negative decision on their asylum claim. Additionally, respondents with German language proficiency also have a higher chance to allow geo-tracking.

Turning to (biasing) effects of interviewer gender and age, we see that older interviewers are more likely to obtain consent to download the app. As age correlates with interviewing experience, interviewer work experience may also positively influence download success. Analyzing the interviewer variance in each model (intraclass correlation coefficient, $ρ_{int}$ ), we see indication of large interviewer effects. In Model 1, the share of variance in downloading behavior that is associated with the interviewer level is estimated at 0.44 (or 44%). In Model 2, relating to consent to provide geo-information, the interviewer-level variance is estimated at 70%. Finally, in Model 3 (providing an email address), 58% of the variance is estimated at the interviewer level. In all three models, these large estimates reflect that successfully obtaining consent to download the app, accessing geo-information, and receiving email addresses are largely dependent upon the interviewer and their behavior.

The large interviewer effects may in principle be due to a relatively small group of interviewers that achieve extraordinary low or high consent rates. Figure 2 reports the interviewer-specific rates (percentages) for consent to download the app (left), consent to providing geo-information (middle), and providing an email address (right).

Figure 2.

Interviewer-level rates in consent to download the app (left), providing geo-information (middle), and providing an email address (right).

As can be seen, there is a large variation in success rates across interviewers. While some interviewers achieve very low (down to 0%) and very high (up to 100%) consent rates, the majority of interviewers lie in between. Thus, the observed interviewer effects cannot be explained by only few interviewers performing very well or very badly.

Effects on Recontacting Rates and Participation

The Wave 2 reparticipation rate amounts to roughly 69% (not displayed as a table), which is substantially lower compared to other SOEP subsamples (Siegers et al., 2019). Additionally, address quality is also worse. More than 4% (4.2%) of the addresses were not traceable (usually over 99% of the SOEP households’ addresses are known in a consecutive wave; Siegers et al., 2019, p. 58). We assume that this difference is due to the specificities of refugees who during their first years move frequently due to allocation procedures and preliminary housing arrangements.

Table 3 displays the effect of allowing geo-tracking, the provision of email addresses, installing the application, and information regarding address changes on address quality (by means of a successful contact) in the second wave (Model 1) and participation when the household could be contacted (Model 2). The results indicate that respondents who allowed geo-tracking, provided their email address, or used the app to provide information about address changes did not have a higher chance of being recontacted by the field institute.³ Additionally, in the event that the members of a household were contacted in the second wave, their motivation to participate did not increase when the respondents had downloaded the app in general.

Table 3.

Effects on Recontacting Rates and Participation.

Variable (Consent)	Recontact (y/n)		Participation (y/n)
Variable (Consent)	Odds Ratio	[95% CI]	Odds Ratio	[95% CI]
Allowed geo-tracking?
No (ref.)	—	—	—	—
Yes	1.24	[0.37, 4.18]	1.19	[0.77, 1.83]
Providing email address?
No (ref.)	—	—	—	—
Yes	1.29	[0.51, 3.28]	0.95	[0.64, 1.39]
Installed app?
No (ref.)	—	—	—	—
Yes	0.65	[0.31, 1.37]	1.05	[0.75, 1.49]
Address provided?
No (ref.)	—	—	—	—
Yes	1.94	[0.70, 5.40]	1.05	[0.71, 1.56]
Gender
Female (ref.)	—	—	—	—
Male	0.73	[0.45, 1.21]	1.17	[0.97, 1.42]
Age
Old (ref.)	—	—	—	—
T(w)eens	0.67	[0.42, 1.05]	0.82*	[0.68, 0.99]
Language proficiency
Good (ref.)	—	—	—	—
Intermediate	1.10	[0.58, 2.06]	1.06	[0.83, 1.37]
Poorly	1.10	[0.57, 2.12]	1.01	[0.77, 1.31]
Legal status
Other (ref.)	—	—	—	—
Residence permit	1.43	[0.85, 2.42]	1.25*	[1.01, 1.53]
Mother tongue available as survey language?
No match (ref.)	—	—	—	—
Match language	1.09	[0.63, 1.89]	0.91	[0.73, 1.13]
Grouped duration of interview
Short (ref.)	—	—	—	—
Medium	1.32	[0.77, 2.27]	1.11	[0.89, 1.39]
Long	1.19	[0.68, 2.09]	1.03	[0.81, 1.32]
ISCED (condensed)
<Primary (ref.)	—	—	—	—
Primary education	0.30	[0.09, 1.07]	1.20	[0.85, 1.70]
Secondary education	0.35	[0.10, 1.22]	1.37	[0.97, 1.95]
Bachelor’s+	0.47	[0.12, 1.87]	1.32	[0.89, 1.95]
Employment status
Unemployed (ref.)	—	—	—	—
Employed	0.93	[0.47, 1.84]	0.91	[0.70, 1.19]
Year of immigration
Before 2014 (ref.)	—	—	—	—
2014	1.22	[0.56, 2.62]	1.16	[0.87, 1.55]
2015	0.75	[0.36, 1.56]	1.17	[0.88, 1.56]
2016	0.48	[0.15, 1.55]	1.00	[0.60, 1.67]
Household size
One person (ref.)	—	—	—	—
Two persons	1.37	[0.80, 2.37]	1.89***	[1.52, 2.35]
Three persons+	1.46	[0.41, 5.21]	1.34	[0.86, 2.11]
Country of origin
Syria (ref.)	—	—	—	—
Afghanistan	1.23	[0.56, 2.69]	0.92	[0.68, 1.24]
Iraq	1.18	[0.58, 2.42]	0.84	[0.63, 1.12]
Eritrea	1.43	[0.43, 4.76]	1.09	[0.72, 1.65]
Balkan	0.26**	[0.10, 0.65]	0.21***	[0.13, 0.33]
Iran/Pakistan	0.47	[0.16, 1.38]	0.60*	[0.37, 0.98]
Other	0.67	[0.31, 1.45]	0.62**	[0.45, 0.84]
Amount of unanswered questions
No missings	—	—	—	—
<10 Questions	0.97	[0.55, 1.71]	0.92	[0.73, 1.16]
>10 Questions	0.88	[0.50, 1.54]	0.81	[0.65, 1.03]
Interviewer gender
Male (ref.)	—	—	—	—
Female	0.93	[0.56, 1.55]	1.05	[0.80, 1.36]
Interviewer age	1.01	[0.99, 1.03]	1.00	[0.99, 1.01]
Interviewer change
No (ref.)	—	—	—	—
Yes	0.41**	[0.22, 0.77]	0.61***	[0.49, 0.77]
Constant	13.24		1.27
n _respondents	2,977		2,872
n _interviewers	86		86
ICC/Kish’s $ρ_{int}$	0.04		0.04

Note. CI = confidence interval; ICC = intraclass correlation coefficient; ISCED = International Standard Classification of Education.

*p < .05. **p < .01. ***p < .001.

Given these results, we have serious doubts that the usage of this mobile phone app and its various features improves address quality and increases participation rates in upcoming panel waves.

Turning to interviewer effects and compared to the results in the previous section, the share of variance estimated to be due to the interviewer level is much smaller ( $ρ_{int} = 0.04$ in Model 1 and $ρ_{int} = 0.04$ in Model 2). In other words, after controlling for respondent-, area-, and interviewer-level covariates, the variation in contact and response behavior is largely associated with the individual (or household) level not the interviewer level.

Discussion

By estimating multilevel logistic regression models with respondents clustered in interviewers, we show that providing an app with geo-tracking, which collects email addresses and offers a way to inform about address changes within the app, does not significantly improve address quality nor the respondents’ motivation to participate in the second wave of a refugee panel study. Additionally, we provide indication that the provision of short questionnaires and news outlets does not increase participation rates among contacted households. We therefore conclude that using a panel app is not necessarily an advantage for decreasing panel attrition in contrast to “classical” measures such as sending out postcards or providing monetary compensation (see, e.g., Schoeni et al., 2013). Regarding the consent to use a mobile phone application, we provide indication that interviewers play a substantial role in convincing respondents to comply. Additionally, individual characteristics such as age, gender, and language correlates also influence consent.

Address Quality and Respondent Motivation

Admittedly, the absence of an effect of geo-tracking on address quality is somewhat surprising, as geo-tracking, in theory, provides detailed information on the respondent’s abode. We assume that most participants switched off their GPS hardware at some point as it decreases battery runtime, or respondents just forget about the app and its purpose. Additionally, GPS coordinates can only be a rough estimation of the actual residence (e.g., if someone moved abroad permanently). Our results therefore confirm what has already been known: GPS coordinates provided without further information are insufficient when it comes to identifying people’s homes. However, if software is available, GPS tracking data might be used for testing geographical eligibility.

The absence of an effect of the provision of email addresses and changing addresses is equally surprising. After consulting the field agency, we have three potential explanations for the absence of an effect. First, the provision of address changes was imprecise; second, the new address had already been outdated during the actual fieldwork; and third, if respondents moved frequently, they forgot to inform about multiple address changes.

Regarding the implementation of short questionnaires and news outlets, we assume that the missing effect can be explained by referring to respondent burden. Some respondents may experience the questionnaires and news outlets provided via the app as somewhat burdensome or just boring. These rather negative experiences with the survey project may impede a higher motivation to participate in the next wave. Additionally, some of the news outlet might reflect a negative connotation toward refugees (see Online Appendix), resulting in hostile feelings. If this is the case, the absence of an effect is less surprising and appears in a more plausible light.

Moreover, some respondents could have replaced their smartphones, or given them to friends or family members, thus rendering the app redundant. Future projects should keep this in mind when implementing a smartphone application. Therefore, monitoring the app activity in future projects would be most valuable in order to observe whether the application is used in general and by the intended respondents.

Given these somewhat disappointing results, we raise the question whether the ineffectiveness is due to the feature itself or rather its implementation. For example, the application itself did not produce constant reminders to provide address changes or to keep the GPS tracking switched on, which could have lead to outdated addresses. Additionally, most of the questionnaires were unintentionally designed like a test about the knowledge on German society and could have raised some concerns regarding their utility and usage. In addition, the questionnaire in the app deviates from the SOEP questionnaire, which may result in some worries about the actual purpose of the app. Given that refugees are particularly vulnerable in terms of their legal status, this might have posed a problem. For future research on this matter, we therefore conclude that the development of such an app should keep the particularities of the target population in mind and should avoid that the design unintentionally decreases the app’s utility.

Consent to Using the Application

Regarding the analysis of consent, what stands out are the large interviewer effects. Our results strongly indicate that interviewers play a crucial role in achieving high consent rates regarding the usage of a panel app. What are possible explanations for these strong effects? First, interviewers differ in their skills, and more experienced interviewers may be more successful in obtaining consent as they have had time to improve their interviewing and communication routines.⁴ Second, interviewers differ in how they communicate, manage to establish a relationship with respondents, and create a pleasant interview situation (“rapport”)—all of which likely affect the willingness of respondents to consent to download an app or agree to share sensitive GPS information. More detailed (qualitative) data about the interview and social interaction are needed in future studies in order to analyze this aspect. Third, the interviewers’ own opinion toward the app and consent requests likely affect their motivation while providing information about and asking for consent. In this regard, for instance, Sakshaug et al. (2013) show that interviewers who would not personally agree to the request they are supposed to ask ultimately obtain lower consent rates. While the observed interviewer effects in principle may also in part be due to area (sample point) effects, controlling for a large number of area and respondent-level characteristics minimizes potential confounding. Moreover, our results match the existing literature on interviewer effects in consent to record linkage requests, showing high interviewer variability in consent rates (Sakshaug et al., 2013, Sala et al., 2012).

The strong language effects on consent are a plausible result. Respondents have a higher chance to consent to download the app when they are proficient in German and the survey is conducted in their mother tongue. Thus, the provision of the mother tongue seems crucial. Using the mother tongue increases the understanding of the consent question and, thus, builds trust, subsequently increasing consent rates. Therefore, if recent migrants are a substantial part of the target population, consent questions need to be translated.

Further, crucial factors are immigrant-specific variables. These are not necessarily transferable to surveys on nonmigrants but are also important to mention. Consent rates for geo-tracking were especially low for refugees with a pending residence permit or with a suspension from deportation. Respondents with a pending asylum case or a suspension of deportation may be more skeptical toward being easily tracked as they may face deportation. However, those who migrated to Germany more recently are more likely to download the app. We assume that this is because they feel obliged to participate either because they were granted protection (as a sign for being grateful) or because they think that their cooperation in this regard poses an advantage for future residence issues (however, as the SOEP questionnaire clearly points out that the interview is independent from their asylum claim, this seems rather unlikely).

Our findings have substantial implications for future research and survey practice:

In light of our study, it is at least questionable whether the costs of developing a mobile phone application and its utility are balanced. As classic panel care measures (interviewer continuity and monetary incentives) are historically proven to work, it might be better to use conservative methods in this respect, especially given limited budgets.

The design of mobile apps needs to keep the particularities of the target population in mind.

In line with previous research on interviewer effects, we strongly recommend training interviewers accordingly in order to achieve high consent rates. Relying on interviewers provides the opportunity to give additional information to the consent question, and interviewers can answer any remaining questions or concerns expressed by the interviewees.

Our analyses suggest that using the mother tongue of respondents is crucial in order to receive consent to data linkage in surveys of recent immigrants.

In contrast to earlier studies, our data do not indicate that smartphone-based data collection can be applied easily to the refugee population in Germany. Around 10% of the respondents did not own a smartphone; 21% encountered software problems. Having the field agency provide smartphones might be an effective (albeit very expensive) way to overcome these technological hurdles and coverage issues.

Supplemental Material

Supplemental Material, sj-do-1-ssc-10.1177_0894439320985250 - Using a Mobile App When Surveying Highly Mobile Populations: Panel Attrition, Consent, and Interviewer Effects in a Survey of Refugees

Supplemental Material, sj-do-1-ssc-10.1177_0894439320985250 for Using a Mobile App When Surveying Highly Mobile Populations: Panel Attrition, Consent, and Interviewer Effects in a Survey of Refugees by Jannes Jacobsen and Simon Kühne in Social Science Computer Review

Supplemental Material

Supplemental Material, sj-docx-1-ssc-10.1177_0894439320985250 - Using a Mobile App When Surveying Highly Mobile Populations: Panel Attrition, Consent, and Interviewer Effects in a Survey of Refugees

Supplemental Material, sj-docx-1-ssc-10.1177_0894439320985250 for Using a Mobile App When Surveying Highly Mobile Populations: Panel Attrition, Consent, and Interviewer Effects in a Survey of Refugees by Jannes Jacobsen and Simon Kühne in Social Science Computer Review

Footnotes

Data Availability

All data are publicly available for research purposes through the research data center of the German SOEP at the German Institute for Economic Research ().

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This work was partially funded by the German Federal Ministry of Education and Research (BMBF) under Grant 01UW1603AX.

Software Information

All work was carried out using Stata Version 14. The DoFile is provided as a Supplementary file. The mobile phone application was produced for the Apple and Android app store but is not available for download anymore.

Supplemental Material

Supplemental material for this article is available online.

Notes

References

AAPOR (2016). Standard definitions. Final Dispositions of Case Codes and Outcome Rates for Surveys. Revised 2016. https://www.aapor.org/AAPOR_Main/media/publications/Standard-Definitions20169theditionfinal.pdf

Bähr

Haas

G.-C.

Keusch

Kreuter

Trappmann

(2020). Missing data and other measurement quality issues in mobile geolocation sensor data. Social Science Computer Review. Advance online publication. https://doi.org/10.1177/0894439320944118

Ballivian

Azevedo

J. P.

Durbin

(2015). Using mobile phones for high-frequency data collection. In Tonninelli

Pinter

de Pedraza

(Eds.), Mobile research methods: Opportunities and challenges of mobile, research methodologies (pp. 21–40). Ubiquity Press.

Brüderl

Ludwig

(2015). Fixed-effects panel regression. In Best

Wolf

(Eds.), The Sage handbook of regression analysis and causal inference (pp. 327–357). Sage.

Buber-Ennser

Kohlenberge

Rengs

Al Zalak

Goujon

Striessnig

Potančoková

Gisser

Testa

M. R.

Lutz

(2016). Human capital, values, and attitudes of persons seeking refuge in Austria in 2015. PLOS ONE 11(9). https://doi.org/10.1371/journal.pone.0163481

Campanelli

O’Muircheartaigh

(1999). Interviewers, interviewer continuity, and panel survey nonresponse. Quality & Quantity, 33(1), 59–76. https://doi.org/10.1023/A:1004357711258

Castiglioni

Pforr

Krieger

(2008). The effect of incentives on response rates and panel attrition: Results of a controlled experiment. Survey Research Methods, 2(3), 151–158. https://doi.org/10.18148/srm/2008.v2i3.599

Colak

Alexander

L. P.

Alvim

B. G.

Mehndiratta

S. R.

González

M. C.

(2016). Analyzing cell phone location data for urban travel: Current methods, limitations, and opportunities. Transportation Research Record: Journal of the Transportation Research Board, 2526(1), 126–135. https://doi.org/10.3141/2526-14

Couper

M. P.

Antoun

Mavletova

(2017). Total survey error in practice. In Biemer

P. P.

de Leeuw

Eckman

Edwards

Lyberg

L. E.

Tucker

N. C.

West

B. T.

(Eds.), Mobile web surveys: A total survey error perspective (pp. 133–154). Wiley.

10.

Couper

M. P.

Ofstedal

M. B.

(2009). Keeping in contact with mobile sample members. In Lynn

(Ed.), Methodology of longitudinal surveys (pp. 183–203). John Wiley.

11.

Couper

M. P.

Peterson

G. J.

(2016). Why do web surveys take longer on smartphones? Social Science Computer Review, 35(3), 357–377. https://doi.org/10.1177/0894439316629932

12.

Dalmasso

Datta

S. K.

Bonnet

Nikaein

Antipolis

(2013). Survey, comparison and evaluation of cross platform mobile application development tools [Paper presentation]. 2015 IEEE 9th International Conference on Research Challenges in Information Science (RCIS), Athens, Greece.

13.

de Bruijne

Wijnant

(2013). Comparing survey results obtained via mobile devices and computers: An experiment with a mobile web survey on a heterogeneous group of mobile devices versus a computer-assisted web survey. Social Science Computer Review, 31(4), 482–504. https://doi.org/10.1177/0894439313483976

14.

de Bruijne

Wijnant

(2014a). Improving response rates and questionnaire design for mobile web surveys. Public Opinion Quarterly, 78(4), 951–962. https://doi.org/10.1093/poq/nfu046

15.

de Bruijne

Wijnant

(2014b). Mobile response in web panels. Social Science Computer Review, 32(6), 728–742. https://doi.org/10.1177/0894439314525918

16.

Dillon

(2012). Using mobile phones to collect panel data in developing countries. Journal of International Development, 24(4), 518–527. https://doi.org/10.1002/jid.1771

17.

Dunn

K. M.

Jordan

Lacey

R. J.

Shapley

Jinks

(2004). Patterns of consent in epidemiologic research: Evidence from over 25,000 responders. American Journal of Epidemiology, 159(11), 1087–1094. https://doi.org/10.1093/aje/kwh141

18.

Durrant

G. B.

Groves

R. M.

Staetsky

Steele

(2010). Effects of interviewer attitudes and behaviors on refusal in household surveys. Public Opinion Quarterly, 74(1), 1–36.

19.

Eisnecker

P. S.

Kroh

(2017). The informed consent to record linkage in panel studies: Optimal starting wave, consent refusals, and subsequent panel attrition. Public Opinion Quarterly, 81(1), 131–143. https://doi.org/10.1093/poq/nfw052

20.

Elevelt

Lugtig

Topoel

(2019). Doing a time use survey on smartphones only: What factors predict nonresponse at different stages of the survey process? Survey Research Methods, 13(2), 195–213. https://doi.org/10.18148/srm/2019.v13i2.7385

21.

Fobia

A. C.

Holzberg

Eggleston

Childs

J. H.

Marlar

Morales

(2019). Attitudes towards data linkage for evidence-based policymaking. Public Opinion Quarterly, 83(1), 264–279. https://doi.org/10.1093/poq/nfz008

22.

Fowler

F. J.

Mangione

T. W.

(1990). Standardizes survey interviewing: Minimizing interviewer-related error. Sage.

23.

Fuchs

Busse

(2009). The coverage bias of mobile web surveys across European countries. International Journal of Internet Science, 4(1), 21–33.

24.

Goebel

Grabka

M. M.

Liebig

Kroh

Richter

Schröder

Schupp

(2019). The German Socio-Economic Panel Study (SOEP). Journal of Economic and Statistics, 239(2), 345–360. https://doi.org/10.1515/jbnst-2018-0022

25.

Groves

R. M.

(2011). Three eras of survey research. Public Opinion Quarterly, 75(5), 861–871. https://doi.org/10.1093/poq/nfr057

26.

Haunberger

(2010). The effects of interviewer, respondent and area characteristics on cooperation in panel surveys: A multilevel approach. Quality & Quantity, 44(5), 957–969. https://doi.org/10.1007/s11135-009-9248-5

27.

Heckman

J. J.

(1976). The common structure of statistical models of truncation, sample selection and limited dependent variables and a simple estimator for such models. Annals of Economic and Social Measurement, 5(4), 475–492.

28.

Heckman

J. J.

(1979). Sample selection bias as a specification error. Econometrica, 47(1), 153–161. https://doi.org/10.2307/1912352

29.

Höhne

J. K.

Schlosser

(2019). Survey Motion: What can we learn from sensor data about respondents’ completion and response behavior in mobile web surveys? International journal of Social Research Methodology, 22(4), 379–391. https://doi.org/10.1080/13645579.2018.1550279

30.

Hoogeveen

Croke

Dabalen

Demombynes

Giugale

(2014). Collecting high frequency panel data in Africa using mobile phone interviews. Canadian Journal of Development Studies, 35(1), 186–207. https://doi.org/10.1080/02255189.2014.876390

31.

Hox

de Leeuw

(2002). The influence of interviewers’ attitude and behavior on household survey nonresponse: An international comparison. In Groves

R. M.

Dillman

D. A.

Eltinge

J. L.

Little

R. J. A.

(Eds.), Survey nonresponse (pp. 103–120). Wiley.

32.

Jäckle

Burton

Couper

M. P.

Lessof

(2017). Participation in a mobile app survey to collect expenditure data as part of a large-scale probability household panel: Response rates and response biases (Understanding Society Working Paper Series 09). https://ideas.repec.org/p/ese/ukhlsp/2017-09.html

33.

Jacobsen

(2018). Language barriers during the fieldwork of the IAB-BAMF-SOEP Survey of Refugees in Germany. In Behr

(Ed.), Surveying the migrant population: Consideration of linguistic and cultural issues (pp. 75–84). GESIS.

34.

Kahnemann

Tversky

(1979). Prospect theory: An analysis of decisions under risk. Econometrica, 47(2), 263–292. https://doi.org/10.2307/1914185

35.

Kahnemann

Tversky

(1984). Choices, values, and frames. American Psychologist, 39(4), 341–350. https://doi.org/ 10.1037/0003-066X.39.4.341

36.

Keusch

Bähr

Haas

G.-C.

Kreuter

Trappmann

(2020). Coverage error in data collection combining mobile surveys with passive measurement using apps. Sociological Methods & Research. Advance online publication. https://doi.org/10.1177/0049124120914924

37.

Keusch

Leonard

M. M.

Sajons

Steiner

(2019). Using smartphone technology for research on refugees: Evidence from Germany. Sociological Methods & Research . Advance online publication. https://doi.org/10.1177/0049124119852377

38.

Keusch

Struminskaya

Antoun

Couper

M. P.

Kreuter

(2019). Willingness to participate in passive mobile data collection. Public Opinion Quarterly, 83(S1), 210–235. https://doi.org/10.1093/poq/nfz007

39.

Kish

(1962). Studies of interviewer variance for attitudinal variables. Journal of the American Statistical Association, 57(297), 92–115.

40.

Korbmacher

J. M.

Schröder

(2013). Consent when linking survey data with administrative records: The role of the interviewer. Survey Research Methods, 7(2), 115–131. https://doi.org/10.18148/srm/2013.v7i2.5067

41.

Kreuter

Haas

G.-C.

Keusch

Bähr

Trappmann

(2018). Collecting survey and smartphone sensor data with an app: Opportunities and challenges around privacy and informed consent. Social Science Computer Review . Advance online publication. https://doi.org/10.1177/0894439318816389

42.

Kreuter

Sakshaug

J. W.

Tourangeau

(2015). The framing of the record linkage consent question. International Journal of Public Opinion Research, 28(1), 142–152. https://doi.org/10.1093/ijpor/edv006

43.

Kroh

Kühne

Siegers

Belcheva

(2018). SOEP-Core: Documentation of sample sizes and panel attrition (1984 until 2016) (SOEP Survey Papers 480: Series C—Data Documentations). https://www.diw.de/documents/publikationen/73/diw_01.c.579464.de/diw_ssp0480.pdf

44.

Kühne

(2018). From strangers to acquaintances? Interviewer continuity and socially desirable responses in panel surveys. Survey Research Methods, 12(2), 121–146. https://doi.org/10.18148/srm/2018.v12i2.7299

45.

Kühne

(2020, June 15). Interpersonal perceptions and interviewer effects in face-to-face surveys. Sociological Methods & Research. Advance online publication. https://doi.org/10.1177/0049124120926215

46.

Kühne

Jacobsen

Kroh

(2019). Sampling in times of high immigration: The survey process of the IAB-BAMF-SOEP Survey of Refugees. Survey Methods: Insights from the Field. https://doi.org/10.13094/SMIF-2019-00005

47.

Lane

N. D.

Miluzzo

Peebles

Choudhury

Campbell

A. T.

(2010). A survey of mobile phone sensing. IEEE Communications Magazine, 48(9), 140–150. https://doi.org/10.1109/MCOM.2010.5560598

48.

Laurie

(2008). Handbook of longitudinal research: Design, measurement, and analysis. In Menard

(Ed.), Minimizing panel attrition (pp. 167–184). Elsevier.

49.

Lipps

(2007). Attrition in the Swiss Household Panel. Methods, Data, Analyses, 1(1), 45–68.

50.

Lugtig

(2014). Panel attrition: Separating stayers, fast attriters, gradual attriters, and lurkers. Sociological Methods & Research, 43(4), 699–723. https://doi.org/10.1177/0049124113520305

51.

Lynn

Burton

Kaminska

Knies

Nandi

(2012). An initial look at non-response and attrition in Understanding Society (Understanding Society Working Paper Series 2). https://www.iser.essex.ac.uk/research/publications/working-papers/understanding-society/2012-02

52.

Lynn

Kaminska

Goldstein

(2014). Panel attrition: How important is interviewer continuity? Journal of Official Statistics, 30(3), 443–457. https://doi.org/10.2478/jos-2014-0028

53.

Mavletova

(2013). Data quality in PC and mobile web surveys. Social Science Computer Review, 31(6), 725–743. https://doi.org/10.1177/0894439313485201

54.

Mavletova

Couper

M. P.

(2013). Sensitive topics in PC web and mobile web surveys: Is there a difference? Survey Research Methods, 7(3), 191–205. https://doi.org/10.18148/srm/2013.v7i3.5458

55.

Mostafa

Wiggins

R. D.

(2017). What influences respondents to behave consistently when asked to consent to health record linkage on repeat occasions? International Journal of Social Research Methodology, 21, 119–134.

56.

Peytchev

Hill

C. A.

(2010). Experiments in mobile web survey design: Similarities to other modes and unique considerations. Social Science Computer Review, 28(3), 319–335. https://doi.org/10.1177/0894439309353037

57.

Pickery

Loosveldt

Carton

(2001). The effects of interviewer and respondent characteristics on response behavior in panel surveys: A multilevel approach. Sociological Methods & Research, 29(4), 509–523. https://doi.org/10.1177/0049124101029004004

58.

Poggio

Bosnjak

Weyandt

(2015). Survey participation via mobile devices in a probability-based online-panel: Prevalence, determinants, and implications for nonresponse. Survey Practice, 8(1), 1–7. https://doi.org/10.29115/SP-2015-0002

59.

Puhani

(2000). The Heckman correction for sample selection and its critique. Journal of Economic Surveys, 14(1), 53–68. https://doi.org/10.1111/1467-6419.00104

60.

Rendtel

Harms

(2009). Methodology of longitudinal surveys. In Lynn

(Ed.), Weighting and calibration for household panels (pp. 265–286). John Wiley.

61.

Sakshaug

J. W.

Couper

M. P.

Ofstedal

M. B.

Weir

D. R.

(2012). Linking survey and administrative records: Mechanisms of consent. Sociological Methods & Research, 41(4), 535–569. https://doi.org/10.1177/0049124112460381

62.

Sakshaug

J. W.

Hülle

Schmucker

Liebig

(2017). Exploring the effects of interviewer- and self-administered survey modes on record linkage consent rates and bias. Survey Research Methods, 11(2), 171–188. https://doi.org/10.18148/srm/2017.v11i2.7158

63.

Sakshaug

J. W.

Kreuter

(2014). The effect of benefit wording on consent to link survey and administrative records in a web survey. Public Opinion Quarterly, 78(1), 166–176. https://doi.org/10.1093/poq/nfu001

64.

Sakshaug

J. W.

Tutz

Kreuter

(2013). Placement, wording, and interviewers: identifying correlates of consent to link survey and administrative data. Survey Research Methods, 7(2), 133–144. https://doi.org/10.18148/srm/2013.v7i2.5395

65.

Sala

Burton

Knies

(2012). Correlates of obtaining informed consent to data linkage: Respondent, interview, and interviewer characteristics. Sociological Methods & Research, 41(4), 414–439. https://doi.org/10.1177/0049124112457330

66.

Sastry

Fomby

McGonagle

(2018). Using the Panel Study of Income Dynamics (PSID) to conduct life course health development analysis. In Halfon

Forrest

Lerner

Faustmann

(Eds.), Handbook of Life Course Health Development (pp. 579–599). Springer.

67.

Schaeffer

N. C.

Dykema

Maynard

D. W.

(2010). Interviewers and interviewing. In Marsden

P. V.

Wright

J. D.

(Eds.), Handbook of survey research (2nd ed.). Emerald Group Publishing.

68.

Schnell

Kreuter

(2005). Separating interviewer and sampling-point effects. Journal of Official Statistics, 21(3), 389–410.

69.

Schoeni

R. F.

Stafford

McGonagle

K. A.

Andreski

(2013). Response rates in national panel surveys. The Annals of the American Academy of Political and Social Science, 645(1), 60–87. https://doi.org/ 10.1177/0002716212456363

70.

Siegers

Belcheva

Silbermann

(2019). SOEP-Core V34—Documentation of samples sizes and panel attrition in the German Socio-Economic Panel (SOEP) (1984 until 2017) (SOEP Survey Papers 606: Series C—Data Documentations (datendokumentationen)). http://hdl.handle.net/10419/195331

71.

Singer

(1993). Informed consent and survey response: A summary of the empirical literature. Journal of Official Statistics, 9(2), 361–375.

72.

Stier

Breuer

Siegers

Thorson

(2019). Integrating survey data and digital trace data: Key issues in developing an emerging field. Social Science Computer Review.Advance online publication. https://doi.org/10.1177/0894439319843669

73.

Sugie

N. F.

(2018). Utilizing smartphones to study disadvantaged and hard-to-reach groups. Sociological Methods & Research, 47(3), 458–491. https://doi.org/10.1177/0049124115626176

74.

Topoel

Lugtig

(2014). What happens if you offer a mobile option to your web panel? Evidence from a probability-based panel of internet users. Social Science Computer Review, 32(4), 544–560. https://doi.org/10.1177/0894439313510482

75.

Trappmann

Gramlich

Mosthaf

(2015). The effect of events between waves on panel attrition. Survey Research Methods, 9(1), 31–43. https://doi.org/10.18148/srm/2015.v9i1.5849

76.

Vandecasteele

Debels

(2007). Attrition in panel data: The effectiveness of weighting. European Sociological Review, 23(1), 81–97. https://doi.org/10.1093/esr/jcl021

77.

Vassallo

Durrant

G. B.

Smith

P. W. F.

Goldstein

(2015). Interviewer effects on non-response propensity in longitudinal surveys: A multilevel modelling approach. Journal of the Royal Statistical Society: Series A (Statistics in Society), 178(1), 83–99. https://doi.org/10.1111/rssa.12049

78.

von Gostomski

C. B.

Pupeter

(2008). Random sampling based on the Central Register for Foreigners: experiences from representative survey “Selected groups of migrants in Germany 2006/ 2007” (Ram). Methods, Data, Analyses, 2(2), 149–177.

79.

Watson

Wooden

(2006). Modelling longitudinal survey response: The experience of the HILDA survey [Paper presentation]. ACSPRI Social Science Methodology Conference, University of Sydney, Sydney, Australia.

80.

Wells

Bailey

J. T.

Link

M. W.

(2013). Comparison of smartphone and online computer survey administration. Social Science Computer Review, 32(2), 238–255. https://doi.org/10.1177/0894439313505829

81.

Wenz

Jäckle

Couper

M. P.

(2019). Willingness to use mobile technologies for data collection in a probability household panel. Survey Research Methods, 13(1), 1–22. https://doi.org/10.18148/srm/2019.v1i1.7298

82.

West

B. T.

Blom

A. G.

(2017). Explaining interviewer effects: A research synthesis. Journal of Survey Statistics and Methodology, 5(2), 175–211. https://doi.org/10.1093/jssam/smw024

83.

West

B. T.

Kreuter

Jaenichen

(2013). “Interviewer” effects in face-to-face surveys: A function of sampling, measurement error, or nonresponse? Journal of Official Statistics, 29(2), 277–297. https://doi.org/10.2478/jos-2013-0023

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

0.18 MB

Using a Mobile App When Surveying Highly Mobile Populations: Panel Attrition,Consent,and Interviewer Effects in a Survey of Refugees

Abstract

Keywords

Literature Review

Panel Attrition

Usage of Mobile Phones in Survey Research

Consent to Linking Survey Data to Additional Data Sources

Interviewer Effects in Survey Data

The IAB-BAMF-SOEP Survey of Refugees

Design of and Consent to Using the App

Translation of Field Instruments and Interviewer Allocation

Estimation of Self-Selection into Downloading the App and Its Utility Regarding Panel Attrition

Estimation of Interviewer Effects

Explanatory and Control Variables

Results

Effects on Recontacting Rates and Participation

Discussion

Address Quality and Respondent Motivation

Consent to Using the Application

Supplemental Material

Supplemental Material, sj-do-1-ssc-10.1177_0894439320985250 - Using a Mobile App When Surveying Highly Mobile Populations: Panel Attrition, Consent, and Interviewer Effects in a Survey of Refugees

Supplemental Material

Supplemental Material, sj-docx-1-ssc-10.1177_0894439320985250 - Using a Mobile App When Surveying Highly Mobile Populations: Panel Attrition, Consent, and Interviewer Effects in a Survey of Refugees

Footnotes

Data Availability

Declaration of Conflicting Interests

Funding

Software Information

Supplemental Material

Notes

References

Supplementary Material