Abstract
This article analyzes whether response patterns in surveys differ between the general population, regular immigrants, and recent refugees. Analyses show that the address quality of refugees contacted in the first wave of a panel study is worse than that of the general population, but of a similar quality to that of other recent immigrants. Once contacted, people in refugee households are more willing than others to participate in the first wave. In subsequent waves, this pattern changes. Address quality remains relatively low, and the motivation to participate deteriorates and is worse in comparison with other populations. However, Cox regression models of individual response behaviour reveal that this is mostly a composition effect. When socio-demographic and interviewer characteristics are taken into account, refugees have a lower risk of attrition than other immigrants, but they have a similar risk as the general population. This article provides important insights for the implementation of research about recent immigrants and refugees into ongoing panel studies.
Introduction
In countries where administrative individual-level data are sparsely available for public scientific research, social reporting relies on household panel studies. Prominent countries in this regard (with their respective studies listed between parentheses) are Germany (Socio-Economic Panel [SOEP]), the United Kingdom (United Kingdom Household Longitudinal Study (UKHLS)), and Switzerland (Swiss Household Panel Study [SHP]). To fulfill their purpose—i.e., to track intra-individual change and to allow inferences about a changing population over time—such studies must constantly observe changes in their underlying target population. A rapid influx of immigrants can cause such a change in the underlying population and thus jeopardize the generalizability of these important studies. A prominent and very recent example is the 2015–16 inflow of refugees to Europe, and to Germany in particular. During these years, around 1.3 million applications for asylum were made in Germany alone (BAMF 2018:30). We assume that many of those who have sought protection will remain in the long term. Therefore, to be able to adequately observe the population in Germany, German general population surveys must include these groups.
Understanding the response behavior of refugees is vital for determining if specific tools and measures are necessary to encourage the participation of refugees in studies. This research is crucial to maintain high data quality and minimize bias when integrating migrant enlargement samples into existing panel studies, as panel attrition can reduce efficiency and introduce bias in survey estimates. This article aims to fill this research gap by comparing the response behavior of recent refugees (immigrated between 2013 and 2016), other immigrants (immigrated since 1995 or direct descendants of immigrants), and the native-born ethnic majority.
More specifically, we will analyze and compare contactability (i.e., can households be contacted) and the motivation of respondents (i.e., whether contacted households participated). First, we will focus on first-wave household nonresponse. Second, we will compare household response rates of consecutive panel waves. Third, by means of Cox regression models, we will analyze the attrition over time of individuals in households that have been contacted once.
The SOEP study is an excellent case study with which to pursue this research. To meet the aforementioned challenge of adequately reflecting social change, the SOEP has been updated several times since 1984 with additional samples, including samples of research recent about migrants (Goebel et al. 2019). Specifically, in 2016, the IAB-BAMF-SOEP Survey of Refugees was launched and integrated into the SOEP to consider acknowledge the large-scale immigration of refugees to Germany. This makes it possible to compare the development of the different samples and to work out the special features involved when establishing a panel study of refugees in Germany.
Expected Challenges in Implementing a Panel Study on Refugees in Germany
Although panel studies have considerable benefits because they allow for multiple observations over time (Giesselmann and Windzio 2014), there are two specific threats to survey quality: address quality and respondent motivation. Both aspects can lead to panel attrition, meaning that respondents cease to participate in the survey or do not participate in the first place. Especially if attrition is not random, the risk of nonresponse bias increases (Groves and Lyberg 2010) and thus threatens the explanatory power of the panel. Address quality is mostly influenced by the up-to-dateness of the sampling frame and the respondents’ spatial mobility. A decrease in motivation is often due to the posing of many questions that are especially complex and therefore difficult to answer, to boring content, and/or to limited time resources on the part of the participants (Lorenc et al. 2013).
Although weighting adjustment can correct for some bias that is due to panel attrition, it reaches its limit when panel attrition is too high and some groups attrite almost completely (Roßmann and Gummer 2016; Vandecasteele and Debels 2007). Established panel studies usually experience panel attrition of 5%–10% between panel waves (for SOEP, see Siegers, Steinhauer, and Dührsen 2021; for UKHLS, see Kantar 2021; for PSID, see Heeringa, Chang, and Johnson 2018). Although this rate of attrition seems low, over time this attrition rate can produce a substantial decrease of sample size. For instance, after three years, a 10% panel attrition in a sample size of N = 5,000 results in a sample size of N = 3,645. Therefore, keeping respondents’ address quality and motivation high is key to maintaining a high survey quality.
Challenges to Contactability in a Survey of Refugees
Refugees can be considered a so-called hard-to-reach population. This means that, in comparison with other societal groups, it is more difficult to access this group to conduct interviews (Tourangeau 2014).
One reason for this is the comparatively high spatial mobility of refugees, which means that addresses are often not (or no longer) valid. Refugees in Germany are not initially free to choose their place of residence; instead, they are assigned to a specific federal state (Bundesland) and then to a specific municipality (see Tanis 2022). They are obliged to live in a reception center for up to 18 months. Thereafter, refugees are distributed within the federal state. This is followed by a transition to the private housing market.
This means that in the first few years after arrival, refugees often stay in one place for a longer period of time only after several stages, which leads to frequent address changes between panel waves. Therefore, analyses of the relocation behavior of officially recognized refugees show that these refugees are significantly more spatially mobile than other groups of migrants or the native-born ethnic majority (Tanis 2022; Weber 2022:22).
Challenges to Motivation in a Multi-wave Survey of Refugees
We assume that recently immigrated refugees feel obliged to participate in a study. On the one hand, this is due to their feeling that they want to give something back for the protection they have received. On the other hand, it is due to uncertainty about the extent to which non-participation could possibly have negative consequences, especially since the BAMF, which is also responsible for the asylum procedure in Germany, was identifiable as one of the institutions conducting the study. This is likely to play a role especially among refugees who are unfamiliar with scientific studies and/or have fled authoritarian regimes.
In the following waves of the panel study, however, the willingness to participate may decrease. On the one hand, the feeling of having to participate in such a study could diminish as familiarity with the customs in the host country increases. Especially those who have had an asylum application rejected might no longer feel obliged to participate. In addition, respondents are now familiar with the response burden of the previous wave (Holbrook et al. 2007). This burden should be particularly high in the first waves due to language difficulties (Jacobsen 2022), especially when questionnaires and accompanying documents are not available in the mother tongues of refugees and an exchange with the interviewer is possible only to a limited extent due to the language barriers (Laganà et al. 2013; Lipps and Ochsner 2018).
We anticipate that the willingness of refugees to participate is initially high, then drops and stabilizes in the further course at a level similar to that of other survey groups.
The IAB-BAMF-SOEP Survey of Refugees
The IAB-BAMF-SOEP Survey of Refugees is a random sample from the Central Register of Foreigners (Ausländerzentralregister (AZR), Babka von Gostomski and Pupeter 2008). The study started in 2016 with two samples (referred to as M3 and M4 within the SOEP) from a population that consisted of persons who came to Germany in the period extending from January 2013 to January 2016 (inclusive) and applied for asylum, as well as of the household members of those persons (Kroh et al. 2017). M4 differed from M3 in that refugee families were oversampled. At the start of 2017, a third sample, M5, was drawn and the population of the study was expanded by including persons who had entered Germany in the period extending from February 1, 2016, to December 31, 2016.
The research group applied a multi-stage, stratified, disproportionate sampling design. Primary sampling units (PSU) were clusters of local immigration offices that administered the addresses of the target population. Individuals listed in the AZR were the secondary sampling units (SSU). Every household member of a sampled respondent is included in the study. The research group stratified the target population by Bundesländer (federal states) and by rural/urban regions. Moreover, people older than 30 who had already been granted protection (asylum or refugee status), people who came from countries with a high likelihood of receiving protection status, and women were all assigned a higher sampling probability.
The first wave of the survey went into the field in 2016, and to date—the second half of 2022—six survey waves have been successfully carried out along with an additional refreshment as well as an enlargement sample (referred to as M6) in 2020. M6 is not part of this study, as only one wave of M6 has been released to date.
The default method of data collection was computer-assisted personal interviews (CAPI). All field material was available in German, English, Pashto, Urdu, Kurmanji, Farsi/Dari, and standard Arabic. The target persons were contacted in writing at the beginning of the field phase and informed about the study and the visit of an interviewer. The interviewer then made personal contact with the household to be interviewed.
At the beginning of each interview, each respondent had to choose a language, which was then presented side-by-side on the CAPI screen with the German version. If respondents were illiterate, audio recordings for each question were available as well (until the second wave). If the interviewer did not speak the respondent’s language and the respondent did not speak German, then self-interviewing was an option, although this method was not frequently used (Jacobsen 2018). More information on fieldwork, first-wave response rates, and panel attrition can be found in Jacobsen et al. (2019), Kroh et al. (2017), and Siegers et al. (2021).
Method
Sample
To explore differences in response rates and response behaviour between different samples of the SOEP, we worked with all SOEP sub-samples and with respective households and individuals for which information on the gross sample was available. The SOEP does not provide information on the first-wave gross samples for sub-samples that entered the SOEP between 1984 and 1998. We restricted our analyses to sub-samples that were part of the SOEP from 1998 to 2019. This left us with sub-samples E, F, G, H, I, J, K, L, M1, M2, M3, M4, M5, N, and O (N, household = 107,753). Therefore, we cannot make any statements about the oldest sub-samples A and C. Additionally, we cannot compare our results with earlier migrant samples such as B and D.
To ensure comparability, we restricted our analyses to households that were interviewed by means of CAPI (N, household = 104,476). For attrition over time, we restricted the analyses to households and individuals that participated in at least two waves and applied listwise deletion due to item nonresponse. This left us with an analytical sample at the household level of N = 96,250 distinct households for first-wave analysis, N = 43,876 distinct households for the analysis of consecutive waves, and N = 61,331 distinct individuals for the analysis of individual dropout risk over time. To ensure that the listwise deletion due to item nonresponse (N, person = 1163/1.9% of the sample) did not introduce bias, we estimated a logistic regression analysis with 1 = “deleted,” 0 = “included in the analysis” as the dependent variable. The explaining variables for the main analyses (see below) served as explaining variables. The results did not provide indication of systematic missingness.
Analytical Strategy
Using frequency statistics, we first compared the first-wave household nonresponse between core samples (defined below), new-immigrant samples, and refugee samples. Here, conducting more complex analyses was impossible as no information about non-respondents was available for the first wave. Second, by pooling all SOEP waves, and by employing linear probability models, we estimated the likelihood of a household to be successfully contacted and motivated in subsequent panel waves. We chose a linear probability model in contrast to a logistic regression analysis because such a model requires fewer assumptions, and estimates are less affected by omitted variables and can be compared across models (Mood 2009).
However, it is likely that the motivation to participate is dependent not only on household features but also, and more so, on individual characteristics. Additionally, we anticipate that only a fraction of a household attrites (partial unit nonresponse). If at least one household member remains in the panel, this would not decrease the household response rate as such, but it would still mitigate the explanatory power of the panel study, as the total number of interviewed individuals would decrease. Therefore, as a third step, by means of Cox regression, we analyzed the individual attrition risk over time of those who had been contacted once and compared core, new-immigrant, and refugee samples. Also, to identify refugee-specific determinants of attrition risk at the individual level, we additionally specified one Cox regression model that included refugee-specific variables. In the analysis at the individual level, we clustered standard errors at the household level.
Variables
For all analyses, we differentiated the dependent variables of contactability (i.e., could households be successfully contacted) and the motivation of respondents (i.e., whether successfully contacted households participated).
The key independent variable in all employed models was constructed by grouping the households and individuals according to the sub-samples of the SOEP. We constructed three groups: (1) Core samples: In line with the naming of the SOEP group, the samples E, F, G, H, I, J, K, L, N, and O are called core samples (2) New-immigrant samples: M1, M2 (3) Refugee samples: M3, M4, M5
For the analyses of consecutive panel waves at the household level, we controlled for: • Size of household (household size increases the time that an interviewer is at one place), • Duration of participation in the SOEP (years since the first interview) and duration squared (we anticipate that with ongoing panel participation the likelihood of dropping out decreases), • Survey year (to account for period effects that might interfere with the fieldwork), and • Federal state (to account for regional differences that could interfere with fieldwork, such as regional holidays, weather, infrastructure).
For the analyses of individual attrition risk, we tested: • Age and age squared in years (as younger respondents are more likely to attrite), • Gender (as females might be more affected by male interviewers), • Employment status: working (including full-time, part-time, and marginal employment, vocational training, internship, military service, etc.) and non-working (as working respondents are harder to contact and motivate due to time restrictions), • Educational qualification based on ISCED (primary or lower, secondary, and tertiary) (as taking part in complex surveys could be easier for highly educated), • Terciles of log household income (as high-income households are harder to motivate), • Number of respondents living in a household (as interview duration correlates positively with household size), • Household type (living alone vs. multi-person household) (as household size increases the time an interviewer is at place), • Federal state (see above), and • Year of the first interview in SOEP (see above).
For the refugee-specific models, we also included the respondents’ self-assessed German-language skills (speaking: good to very good, so-so, very poor to not at all), type of accommodation (shared accommodation or private), and current residency status or residency status prior to attrition (in the asylum procedure, protected status granted, suspension of deportation, other).
Interviewers play a central role both in establishing contact on site and in motivating the participants (Kühne and Kroh 2021). For the refugee sample, the SOEP group recruited interviewers who had prior knowledge of Arabic and Farsi. These interviewers were younger and less experienced than those covering the other sub-samples. To account for this, we will additionally control for interviewer characteristics in the analyses (age, gender, years of work, previous contact with the household).
Results
First-wave Response—Household Level
First-wave Contactability and Motivation across SOEP Sub-samples.
Source: SOEP v.36. DOI: 10.5684/soep.core.v36eu, note: contactability is defined as the share of accessible addresses relative to all addresses that went into field; motivation is defined as the share of successful household interviews relative to all accessible households.
The motivation (successful interview after successful contact) of the refugees in the first wave was significantly higher (see Table 1), and the differences are striking. Whereas for the refugee samples we find that 64% of all contacted households participated, only about 30% of core or new-immigrant households could be motivated to participate.
Response in Consecutive Waves—Household Level
Contact and Participation of Households in Consecutive Waves by Sample, Linear Probability Models.
Source: SOEP v.36. DOI: 10.5684/soep.core.v36eu.
Models 1 and 2 in Table 2 indicate that in subsequent waves, households in the refugee samples have a lower likelihood of being contacted compared to households in the new-immigrant samples and in the core samples. The likelihood of a successful contact for households from the core samples is 5% points higher than the likelihood for households from the refugee samples, and 4% points higher than the likelihood for households from the new-immigrant samples. This finding is independent of the control variables used, such as the survey year, the household size, or the number of years since the SOEP first contacted a household. When these control variables are taken into account, the difference is reduced by just 1% point to 4 and 3% points respectively.
Additionally, households of the refugee survey that had been contacted only once have a lower motivation than households of the core, but they have a similarly low motivation compared with households of the new-immigrant samples (models 3 and 4). The likelihood of participating in the survey is 19% points higher among households from the core than among refugee households. When the control variables are considered, this difference is almost halved to 10% points, but remains substantial.
Response in Consecutive Waves—Individual Level
Figure 1 displays a Kaplan–Meier survival curve for individual respondents of the core population, the new-immigrant samples, and the refugee samples. A clear pattern is displayed: the steepest curve is found with respondents from the refugee samples, followed by new immigrants, while the core population shows the flattest curve. Interestingly, though, the refugee sample appears to experience especially high attrition in the first two waves. Later, attrition rates seem to normalize to a certain extent, and the attrition rate after wave two only slightly exceeds the attrition rate of new immigrants and the core population. Motivation of individuals over time by sample. Source: SOEP v.36. DOI: 10.5684/soep.core.v36eu.
Relationships between the Risk of Attrition and the Characteristics of Respondents and Interviewers, Cox Regression, all Samples.
Source: SOEP v.36. DOI: 10.5684/soep.core.v36eu. Additional controls are: survey year, federal state, and first interview in SOEP.
Relationships between the Risk of Attrition and the Characteristics of Respondents and Interviewers; Cox Regression, Refugees Only.
Source: SOEP v.36. DOI: 10.5684/soep.core.v36eu. Additional controls are: survey year, federal state, and first interview in SOEP.
Regarding the specific characteristics of refugees, the risk of attrition is highest among those still in the asylum procedure. Also, those refugees who are still living in shared accommodation have a higher risk of attrition than refugees who are already living in private accommodation. Finally, it becomes clear that, as expected, the participants’ German-language skills played an important role: The poorer their language skills, the higher their risk of attrition.
Discussion and Conclusion
In summary, analyzing the different SOEP samples has produced at least three new and important insights, which we discuss in more detail below: (1) Samples of recent immigrants show a higher share of invalid addresses than the general population samples. (2) Refugees have a similar motivation to participate in a panel study in comparison to the general population. (3) Refugees who are already granted asylum or refugee status and who live in private accommodation show a higher motivation to participate.
Our analysis reveals that the contactability of the first wave of SOEP refugee samples (M3 to M5) is worse compared to the core samples, but similar in quality to the more recent SOEP migrant sub-samples (M1 and M2). We believe there are two main reasons for this difference. First, the M1 to M5 samples relied on address-based register data, which means specific addresses were selected for contact by interviewers. This sampling strategy differs from the strategies used for other samples in the SOEP, such as random walks, where interviewers have more flexibility to choose a different respondent if the initially selected address is invalid. Therefore, the low address quality is not necessarily due to respondents’ characteristics, but rather reflects different sampling frames and approaches. Second, we attribute the low address quality to the high mobility of recent immigrants during their early days in Germany, as mentioned earlier.
In later waves, the pattern changes. While address quality remains low, the willingness to participate declines in comparison to previous waves. The likelihood of attrition among contacted households is higher compared to other SOEP sub-samples. The persistently low contactability of households can now be attributed solely to high mobility, as the addresses of these households have been verified in the first wave and changes can now only be due to movements. This discovery suggests that refugee households not only exhibit significant mobility in the immediate settlement period but also maintain this mobility for a longer duration than initially anticipated. Notably, in subsequent waves, contacted refugee households demonstrate lower motivation to participate than before.
The Cox regression results indicate that the household-level findings could be composition effects. The lower motivation of refugees to participate after wave 1, compared to both the newer migration samples and the core population, can be attributed to the socio-demographic composition of the different samples. However, when considering the socio-demographic characteristics of participants and interviewers, refugees, even after the first wave, exhibit a motivation to participate that is as high as respondents from the core population, and even higher than respondents in the recent-immigrant samples.
We see this as our most striking finding since, to date, only descriptive statistics have been analyzed, and these indicate that refugees’ motivation to participate in a survey after wave 1 is particularly low. We now provide evidence that this could be a composition effect.
Our study also provides one insight that might counteract unwanted attrition: an insight into interviewer characteristics. We find, first, that with refugees there are fewer cases of attrition when there is a female interviewer and, second, that interviewer stability decreases the risk of panel attrition. This second finding, especially, hints at the fact that building trust with this population is key to initiate a stable panel.
Two limitations should be noted, though. First, it would have been an advantage to test different modes for panel retention, such as mobile phone applications, post cards, or email addresses. Unfortunately, the SOEP does not provide any info on its procedures at the household level. Second, it would have been advantageous to have more information about the interviewer, as our results, once again, stress the importance of the interviewer’s function.
To conclude, our analyses have important implications for panel studies that aim to include recent refugees: (1) When sampling immigrants based on a address register, we propose anticipating a large proportion of nonresponses due to invalid addresses. Practitioners should therefore anticipate resources for tracing hard-to-reach households. (2) Oversampling refugees who have already been granted proper protection reduces the rate of attrition and, (3) When CAPI is used, as here, the selection of interviewers plays an important role. Experienced female interviewers achieve the best results. However, it is most important not to change interviewers between waves. By maintaining the same interviewer, a trusting relationship can be built up with the participants, which is important for their long-term participation.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Correction (January 2024):
This article has been updated with two additions to the reference list since its original publication.