Contemporary Challenges in School Recruitment for Criminological Survey Research: Lessons From the International Self-Report Delinquency Study in England,Germany,the Netherlands,and the United States

Abstract

Several multiwave cross-national surveys have experienced drops in school participation for youth health and risk behavior (HRB) surveys in Western European countries. This article considers explanations for the challenge in recruiting schools for surveys in England, Germany, the Netherlands, and the United States and the most important lessons learned during school recruitment for the third wave of the International Self-Report Delinquency Study in these four countries. Comparing school response rates for international academic surveys with those focused on HRB, schools have been increasingly less likely to participate in HRB surveys over the past two decades. However, considerable variation within and across surveys and countries suggests there are numerous influences on school recruitment, and there may be facilitators on which researchers could capitalize. We conclude that when planning future school-based HRB surveys, researchers should consider multiple strategies to engage schools from the outset, tailored to regional and national settings.

Keywords

school recruitment cross-national research youth health youth risk behavior school response rates

Introduction

Adolescence is a period of extensive physical and social change characterized by increased risk behaviors and peer pressure, and diminishing parental influence; choices made during adolescence can affect young people’s health and well-being in later life (Kleinert, 2007; Viner et al., 2012). Schools are an attractive and cost-effective setting for conducting research on youth delinquent behavior and other youth risk behaviors (Claudio & Stingone, 2008; Marshall, 2010). Compulsory schooling up to age 16 in most countries ensures the presence of young people of nearly all socioeconomic and cultural backgrounds, including groups that are often underrepresented in research (Bjarnason, 1995; Smit, de Zwart, Spruit, Monshouwer, & van Ameijden, 2002). With high individual response rates common in school surveys (Smit et al., 2002), this method can achieve representative samples that allow generalization to the larger school-age population. Moreover, particularly for sensitive research topics, such as delinquent behavior, sexuality, or drug use, adolescents associate school surveys with more privacy, anonymity, and confidentiality than surveys administered at home (Gfroerer, 1985; Michaud, Delbos-Piot, & Narring, 1998; Smit et al., 2002). They also appear to be more effective in eliciting accurate prevalence rates when seeking self-reported engagement in activities such as illicit drug use (cf. Fendrich & Johnson, 2001).

School surveys are frequently used in large multiwave cross-national studies on youth health and risk behavior (HRB), such as the International Self-Report Delinquency Study (ISRD; Junger-Tas et al., 2012), the Health Behavior in School-Aged Children study (HBSC; Currie, Nic Gabhainn, & Godeau, 2009), and the European School Survey Project on Alcohol and Other Drugs (ESPAD; Hibell et al., 2012). Starting in the 1980s and 1990s, these studies used school surveys to chart trends in adolescent behavior and development within and across countries over time, test the universality of developmental and behavioral theories, and explore the effects of different policy approaches and social interventions for preventing health problems and delinquent behaviors (Junger-Tas et al., 2012; Roberts et al., 2009). High-quality cross-national comparisons require standardized research designs and methods and high individual and school response rates to ensure reliable and representative data. A sample unrepresentative of the school-age population could result in biased prevalence rates (Micklewright et al., 2012).

As coordinators of the third wave of the ISRD (ISRD-3) in England and the Netherlands, an international school-based survey on youth delinquent offending among 11- to 16-year-old high school students, we experienced serious challenges in recruiting schools, as did our German and U.S. colleagues. Several cross-national studies have referred to the challenges in school recruitment for international surveys (e.g., Kraus & Hibell, 2014; Marshall, 2010; Meinck, Cortes, & Tieck, 2017; Roberts et al., 2009; Sturgis, Smith, & Hughes, 2006). However, empirical research on developments in school recruitment is limited. A review of school surveys by England’s Department for Education between 1995 and 2004 revealed a 2% annual decline in school response (Sturgis et al., 2006). A more recent study by Meinck et al. (2017) found that between 2006 and 2016 around 17% of countries failed to meet the minimum participation rates at school level set by the International Association for the Evaluation of Educational Achievement (IEA). However, neither study provides unequivocal evidence for a general downward trend in school response for these four countries. Given these observations, the research question addressed in this article is “How can challenges in school recruitment for ISRD-3 in England, Germany, the Netherlands, and the United States be explained and what lessons can be learned?” Here, we aim to help researchers anticipate and prepare for recruitment challenges in future school surveys such as the planned ISRD-4.

The article proceeds as follows: In the section “Factors Affecting School Recruitment and Participation in Cross-National Surveys,” we provide an overview of factors found in academic literature associated with school recruitment and participation in cross-national surveys; the section “Method” sets out the method; the section “How Could Challenges in School Recruitment for ISRD-3 Be Explained?” presents our analysis of explanations regarding challenges experienced in school recruitment in cross-national surveys; and the section “What Could Be Learned From ISRD-3 School Recruitment in England, Germany, the Netherlands, and the United States?” offers an overview of the most important lessons learned in ISRD-3 for England, Germany, the Netherlands, and the United States. We end with some recommendations and considerations for researchers considering school-based research.

Factors Affecting School Recruitment and Participation in Cross-National Surveys

Drawing from literature on survey response in school-based research and wider survey methodology, we identified three types of factors that can affect school recruitment for cross-national surveys, namely, contextual, study-related, and survey implementation factors.

Contextual factors entail a range of factors that are not within, or only slightly within, researchers’ scope of control but could negatively affect school recruitment. First, and understandably, research is not a high priority for teachers and school leaders who are being confronted with more urgent daily matters, such as teaching activities, administration, and managing the demands of local or central government or school boards (Lamb, Puskar, & Tusaie-Mumford, 2001; Sturgis et al., 2006). Increasing demands over the years from performance management and educational reforms has only added to the pressure on schools and teachers, reducing the capacity for noncompulsory activities such as research (Hargreaves, Lieberman, Fullan, & Hopkins, 2014). Second, a general and growing problem in social research thought to contribute to lower response rates is oversurveying (Baruch & Holtom, 2008; Fulton, 2018; Marshall, 2010). The more requests that schools receive for research access, the more likely they are to be selective or to refuse participation completely. Third, the increasing demands of ethical guidelines for research with children and young people with respect to privacy and the need for active parental consent may inhibit the scope for research in some countries. Additional effort and resources required to achieve higher consent rates could be off-putting for schools (Esbensen, Melde, Taylor, & Peterson, 2008; Shaw, Cross, Thomas, & Zubrick, 2015).

Study-related factors inherent to the study’s topic and design could be influential in several ways. First, the topic and its perceived sensitivity could influence head teachers’ appraisal of the study’s value and importance. Studies on delinquent and other risk behaviors do not contribute directly to schools “core business” of academic education and may be considered less important and of less direct value to the school (Bonell et al., 2014; Vanderlinde & Van Braak, 2010). Cross-national studies may appear even further removed from the classroom and regional/national education policies as they generate theoretical knowledge and generalized trends, and survey questions tend to be a balance of individual countries’ interests and those of the survey’s developers (Roberts et al., 2009; Vanderlinde & Van Braak, 2010). Particularly with sensitive topics, head teachers may also fear negative reactions from parents and students.

Second, studies are constrained by their budget, and resources inevitably define to some extent options to improve school recruitment, for instance, team capacity, publicity, and incentives for participation for schools and students. Other study-related factors include frequency of the survey over years and sampling strategy (e.g., Sturgis et al., 2006; Testa & Coleman, 2006; White, 2012). For example, school preparedness to participate may be reduced when frequency of repeats is high and the timing of sampling can be planned for times in the school year that permit subsequent replacement sampling.

Third, schools’ preparedness to take part will also be conditioned by the stance of key stakeholders to whom they may be accountable. In some countries, schools are highly responsive to the requirements of government departments of education (who are very likely to recommend or require participation in the Programme for International Student Assessment [PISA], for example, which assesses academic performance) and to other state bodies who may support surveys on HRB (cf. Marshall, 2010). In other countries, schools have considerable autonomy, especially in relation to surveys on HRB, although head teachers may be responsive to other stakeholders such as local and national police. As a general rule, strong stakeholder support is likely—but not guaranteed—to help secure school agreement; and where government bodies actually fund such surveys, the pressure on schools to comply is that much stronger.

Survey implementation factors are the factors that lie most within researchers’ control as these factors relate to country-specific and local knowledge about gatekeepers, organizational structures, and formal and informal opportunities available to promote research in schools and increase chances of participation. The academic literature identifies two main hurdles in getting schools to take part: simply gaining access to those who can agree to their school’s participation and then actually securing their agreement (Rice, Bunker, Kang, Howell, & Weaver, 2007). The literature identifies various strategies for getting both access and agreement: endorsement by significant stakeholders; determining the best order of contact where several people’s agreement is required; identifying a teacher or other person with an interest in the topic to act as “champion” for the survey; clear communication that stresses practical support and assistance, emphasizing mutual benefits and importance of the study; and offering incentives to schools and students (Baruch & Holtom, 2008; Brown, Mounts, Lamborn, & Steinberg, 1993; Lamb et al., 2001; Sturgis et al., 2006; Testa & Coleman, 2006; White, 2012)

Method

Selection of Surveys

As mentioned above, the trigger for preparing this article was the difficulties that the authors experienced in securing acceptable school response rates in ISRD-3. This prompted us to examine the experience of other researchers mounting international schools-based survey in the countries with which we were most familiar and on which we had most data: Germany, The Netherlands, England, and the United States. Our choice of countries was thus pragmatic. We identified five cross-national school-based surveys on adolescents; three of these were HRB surveys and two were surveys of academic performance. An overview of the surveys is presented in Table 1:

Table 1.

Survey Characteristics.

Main topic	Health and risk behavior			Academic performance
	ESPAD	HBSC	ISRD	PISA	TIMSS
	Alcohol and drug use	Health, well-being, and health behaviors including sexual behavior	Victimization and self-reported youth delinquent behavior	Science, mathematics, reading, problem solving, and financial literacy skills	Mathematics and science skills
First wave	1995	1982	1990	2000	1995
Institution	N/A	WHO	N/A	OECD	IEA
N countries	>40	42	35	>80	>60
Frequency	4 years	4 years	Fluctuating	3 years	4 years
N waves	6	9 (5 covered in analysis)	3 (2 covered)	6	6
Target group	Students turning 16 in the year of data collection	11-, 13-, and 15-year-old students	Grades 7, 8, 9/national equivalent (aged 12-16 years)	15-year-old students	Eighth grade (aged 13 years)
Sample	National	National	City-based (2 cities)	National	National
Sampling units	Schools, classes	Schools, classes	Schools, classes	Schools, students	Schools, classes
Minimum sample size per country	2,800 students	4,500 students (1,500/age group)	1,800 students (300 per grade/city)	5,250 students (35-42 students per school for at least 150 schools)	4,000 students and 150 schools
Target school response rate	N/A	N/A	N/A	>85%	>85%

Note. ESPAD = European School Survey Project on Alcohol and Other Drugs; HBSC = Health Behaviour in School-Aged Children study; ISRD = International Self-Report Delinquency Study; PISA = Programme for International Student Assessment; TIMSS = Trends in International Mathematics and Science Study; WHO = World Health Organization; OECD = Organisation for Economic Co-operation and Development; IEA = International Association for the Evaluation of Educational Achievement.

The academic performance surveys, PISA and Trends in International Mathematics and Science Study (TIMSS), were generally larger than the HRB surveys with at least 60 participating countries, with high targets set for school response rates (>85%). HBSC and ISRD both aimed for a spread of grades and age groups, whereas the other three studies limited their scope to one particular age group or grade. ISRD had some notable differences to the other surveys—it had a city-based rather than a national sampling design, fewer waves, and waves that occurred at fluctuating intervals.

Data Collection and Analysis

School response rates for the past two decades (1995-2015) were retrieved from international, country, and technical reports and other sources of information, such as open access databases and articles (see Appendix A for file with sources per survey). Response rates for the United Kingdom or Great Britain were used if rates specifically for England were not available.

Studies often use replacement to deal with schools’ refusal to participate in a survey—where each refusing school is replaced by a matched school from a secondary list. We used before replacement rates rather than after replacement rates in surveys where replacement was used, to provide rates that were genuinely comparable across the five surveys. Whenever possible, we calculated rates by dividing the unweighted number of participating schools before replacement by the unweighted total number of all eligible schools in the original sample, including schools that could not be contacted and before addition of replacement schools or extra samples. For most studies, we were able to obtain a list of comparable response rates. Only two response rates that were judged unreliable—HBSC USA 2010 and HBSC ENG 2014—were omitted from the regression analysis.

We first considered trends across countries and studies and then ran separate linear and quadratic regressions with Stata 14.2 SE to identify general trends in both HRB surveys and surveys of academic performance, while accounting for study and country. Predictive average margins were obtained through Stata’s “margins” command to assess differences in trends between HRB and academic surveys. The R-package “ggplot2” was used to create plots. The regression analyses were carried out using an aggregated data set for the countries under investigation.

Our examination of other contextual, study-related, and survey implementation factors is based on a narrative review of technical reports, research diaries, and contact logs with additional information from the coordinators of ISRD-3 in England, the Netherlands, Germany, and the United States. Based on the review of literature and information from the four countries, we identified the most important problems in school recruitment and strategies implemented by coordinators of the countries to tackle problems they encountered as well as observed effects (or lack thereof).

How Could Challenges in School Recruitment for ISRD-3 Be Explained?

As a first step in exploring challenges in school recruitment in ISRD-3 in England, Germany, the Netherlands, and the United States, we examined whether similar challenges were evident in the other four surveys.

Examining Trends in School Recruitment

First, an important observation is that no clear evidence exists for a general downward trend across all surveys and countries (see Figures 1 and 2). Another observation is that surveys of academic performance generally report higher response rates than HRB surveys, so there is a clear difference between HRB surveys and surveys of academic performance.

Figure 1.

Trends school response rates for Germany, The Netherlands, England/United Kingdom, and the United States across three Health Risk Behavior (HRB) surveys before replacement.

Figure 2.

Trends school response rates for Germany, The Netherlands, England/United Kingdom, and the United States across two surveys of academic performance before replacement.

Trends across surveys for individual countries were inconclusive. For the Netherlands, there were signs of a downward trend across two HRB surveys and one academic survey, respectively, PISA, ESPAD, HBSC, and Dutch ISRD rates were generally low. It was not possible to assess trends for TIMSS as the Netherlands did not participate beyond 2003. There were also signs of a downward trend for HRB surveys in Germany though excellent rates for PISA. The results for England and the United States were mixed: England had upward trends for academic surveys and varying response rates for HRB surveys, and the United States had relatively high rates for academic surveys and mixed rates for HRB surveys.¹

Regression analysis provided a somewhat nuanced picture: Where response rates were generally higher for surveys of academic performance than for HRB surveys; in later years, a downward trend in school response rates appeared to have set in for both academic and HRB surveys (see Figure 3; regression table included in Appendix B). Despite a negative trend for academic surveys, predictions are still relatively positive with an average predicted response rate between 52% and 93% after 2020. For HRB surveys, predictions are more negative with predicted rates between 0% and 40%, suggesting that over the past two decades, schools have been increasingly less willing to participate in surveys, and particularly those focused on HRB.

Figure 3.

Average predicted mean response rate with 95% CIs for Surveys of academic performance (N = 40) and HRB surveys (N = 37) controlled for country and study differences.

These findings provide evidence that challenges with school recruitment in ISRD-3 fit in a wider context of decreasing response rates. However, considerable variation still exists within and across surveys and countries, which suggests that other, possibly modifiable, factors also affect school recruitment.

Factors Influencing School Recruitment

To better understand variation in response rates, we looked at contextual, study-related, and survey implementation factors that appeared to be influential during school recruitment for ISRD-3 in England, Germany, the Netherlands, and the United States. We found support for all three types of factors affecting school recruitment.

Contextual factors were conspicuous: Oversurveying was clearly an issue for school recruitment in ISRD-3 and was also mentioned in several HBSC and ESPAD reports. In the Netherlands, where most nonparticipating schools (90%) gave reasons for refusal, 62% said that the volume and nature of research requests were barriers to participation. One Dutch school had recorded all requests during one school year, totaling over 100. Pressures of school daily life also appeared to be an important reason for refusal. Schools were not able to fit the survey into their teaching activities, participation was thought to interfere with exam preparation, or other organizational issues prevented schools from participating, such as staff changes or moving to a new school building. In combination, these factors had led some schools or school boards in the United States and the Netherlands to adopt a “no-research” policy. For the United States in particular, the twin requirements of active parental consent and district-level permission (additional to school agreement) proved to be severe barriers to participation. While these contextual factors may clearly depress response rates, it is not at all clear that they will systematically bias the sample in ways that are related to the topic under investigation.

Besides contextual factors, study-related factors, such as topic, study design, and resources, also affected school recruitment. The generally higher response rates for surveys of academic performance suggest that these academic surveys may be better received by schools than HRB surveys. Furthermore, as discussed above, head teachers are likely to be responsive to pressure from key stakeholders, such as central and local government education departments, especially when these are committing funding to the survey in question. Responses from refusing schools in ISRD-3 confirmed that some schools refused because youth delinquency was thought to be irrelevant and not an immediate and pressing issue at their school. Communicating the international scope of the research deterred some schools from participating as they intuitively associated the magnitude of the scope with a heavy time investment. Topic sensitivity was also a reason for refusal. Schools were anxious that such a survey would frighten or upset students. In the United States, schools also worried about parents’ reactions to a survey on “delinquency” and possible resulting negative publicity. In the Netherlands, some schools with a high-risk student population were offended by the focus of ISRD-3 (and similar studies) on the problems faced by their students rather than their positive achievements. Nonetheless, some schools explicitly participated because they thought the survey was relevant to their high-risk population or was of interest as related topics would be covered in the school curriculum.

Two other study-related factors, study design and resources, appeared to be of influence in several ways. The two surveys of academic performance with higher response rates were also larger surveys that served a national and international benchmarking function. National education and school systems in several countries attached considerable importance to the surveys (Breakspear, 2012; Grek, 2009); consequently, surveys of academic performance were more likely than HRB surveys to have a high public profile and to be well known among educators, translating into pressure on schools to participate.

The benchmarking function of surveys of academic performance also means that they have probably been better resourced than HRB surveys. Certainly, in the four countries covered by this article, ISRD-3 was conducted with modest budgets, leaving little flexibility to be responsive to school demands for participation in the survey. In both England and the Netherlands, a single researcher was responsible for recruitment and data collection. This proved an insufficient level of staffing to manage ongoing contact with a large number of schools and to arrange and conduct data collection. Recruitment efforts were improved by the addition of two research assistants in both England and the Netherlands, but, in hindsight, the scale of the recruitment task was severely underestimated.

Like the contextual factors discussed above, most of these survey-related factors are unlikely to be related to the topic under investigation and should not result in sample bias in ways that challenge the validity of the findings. The exception to this is where schools are anxious about the risks of stigmatization arising from participation in a survey about youth crime. This is clearly an issue that warrants close attention. If there is a systemic tendency for schools in high-crime areas to refuse to take part, this could skew results significantly. However, we do not have any firm evidence that this was a problem for ISRD-3.

Regarding survey implementation factors, gaining access to schools and being able to discuss the survey with head teachers or other teachers directly or on the phone was routinely difficult. Nonresponse, rather than a clear “no,” meant that numerous repeated contacts were made by phone and email—probably exacerbating survey fatigue on part of the schools. Even with a named contact person and positive school response, it was difficult to establish contact. Endorsement from national, federal, and local ministerial departments was one way in which researchers in all four countries tried to get in contact with schools and communicate the value of the study. With the exception of Germany, formal endorsement from these authorities—if obtained—did not help to persuade schools. In England and the Netherlands, accessing schools through intermediary organizations was a more successful approach (see the section “Use a social networking approach”). Both England and the United States experimented with financial incentives for schools and students to persuade schools for participation, but these incentives proved largely unsuccessful. It should be noted, though, that for some of the schools in the poorer neighborhoods in the United States, the financial incentive (in the form of a US$5 or US$10 gift card to the local Dunkin’ Donuts) appeared to be a factor in the schools’ willingness to participate. Our judgment is that these factors associated with survey implementation are unlikely to be systematically related to the topic of the survey and thus are unlikely to result in sample bias.

What Could Be Learned From ISRD-3 School Recruitment in England, Germany, the Netherlands, and the United States?

This article has identified a range of factors that can influence school response rates. In this section, we reflect on our experience of ISRD-3 to identify and summarize those factors that seem to have had a positive effect on response rates in the four countries under examination.

Create a Clear School Recruitment and Communication Plan in Advance

School recruitment has become more challenging over the years; researchers need to seriously consider how school recruitment could be facilitated, including available resources, sufficient staffing to meet school demands, and communication with schools. A well-orchestrated plan at the start, sufficient resources to put it into effect, and a coordinated and dedicated team of researchers in each country should maximize response rates from the outset. Elements of this plan will need to vary from country to country, exploiting what is found to work locally.

Communication with schools and school interests should not be overlooked. The U.S. team, for instance, created a study website, made a short video explaining the study, and organized an information event with school counselors. In the Netherlands, researchers replaced a rather long formal letter with a brief message clearly stating benefits and limited time investment to schools. This shorter message appeared to facilitate recruitment when combined with other strategies as it helped to “pitch” the survey to school staff. Both the United States and the Dutch team offered schools (and school districts in the United States) the possibility to receive an individualized report with results. In the Netherlands, this report included results that could be of interest to schools—such as levels of school and teacher bonding and school disorganization—but would not harm student anonymity. In the United States, schools were only provided with an individualized report if at least 100 students participated in the survey; no findings with regard to delinquency or parental use of violence were included. The U.S. team attached a sample report to each invitation (after summer 2016) as an incentive for schools to take part. Generally, reports were appreciated, even if they were not the central motivating factor for most schools to participate.

Use a Social Networking Approach

As suggested by Baruch and Holtom (2008), the best way to get beyond gatekeepers is a social networking approach: making contact through an intermediary organization or person already known and respected by the school. The English team was able to obtain the support of local police in one city, where individual officers with personal connections to schools facilitated contact between researchers and school, resulting in higher response rates (26% vs. 16% for the city without police endorsement). The Netherlands team also used a multilayered social networking approach, contacting schools through a range of organizations, such as school social work, a market research organization, and a teacher association. This approach helped to recruit about half of all participating schools, including schools that had refused earlier. A pitfall of this approach, however, was that it could be tricky to keep track of school recruitment when contacts were mediated through these organizations.

In line with a social networking approach, researchers with local knowledge could be helpful in building a rapport with schools, as well as in practically arranging school visits. In England and Germany, additional researchers based locally in the cities were recruited to assist with data collection and this worked well. In the Netherlands, being a relatively small country, no locally embedded researchers were used.

Speak the Language of Schools and Connect to School Interests

Researchers can choose to “sell” a survey to schools either by being upfront about the sensitive topic or by describing the survey in more general HRB terms, referencing other topics covered in the survey. In England, the first strategy was, in a sense, automatically chosen by seeking endorsement of local police forces. After several refusals, the Netherlands, Germany, and the United States decided to experiment informally with more neutral titles for the project—using terms such as “safety” or “security” or national equivalents with less negative connotations than “crime” and “victimization.” In many cases, rephrasing seemed to help at least to get “a foot in the door” and provided opportunities to discuss study content and possible participation with schools. This strategy avoided deterring those schools that considered youth delinquent behavior irrelevant or were anxious about the risks of negative publicity associated with a survey of delinquent behavior.

Other HRB surveys also show diverging strategies. While the full title of ESPAD—European School Survey Project on Alcohol and Other Drugs—is clearly upfront about its content, the title, Health Behavior in School-Aged Children (HBSC), employs the broader concept of health behavior while also capturing more sensitive topics, such as alcohol and drug use and risky sexual behavior. ESPAD and ISRD generally have lower response rates than HBSC. It, therefore, makes sense to consider the title and content of the survey from schools’ perspectives at the time of proposal and consider the likely impact on schools’ initial preparedness to consider participating.

More than just rephrasing project titles, it is important to really consider school interests and how schools could benefit from these studies from the outset. One possibility—where international surveys permit this level of flexibility—would be to add additional question modules of interest to schools, school boards, or school districts in a particular country, for instance, by co-designing a school-related module with schools or offering schools a choice from a selection of different modules to tailor the survey to their individual school context. Different options have different implications that should be considered carefully, such as questionnaire length and the required level and type of reporting back to schools, school boards, or school districts. However, it is important for researchers to really consider and incorporate school interests in their study design.

Discussion and Conclusion

For decades, large multiwave cross-national HRB surveys have been important sources of information on trends in adolescent HRB. They have proved valuable both for criminological theory and for public policy on youth crime prevention. Recent challenges in recruiting schools in mainly developed countries may represent a significant threat to the continuation of such surveys. This study has examined the nature of school nonresponse in four industrialized Western countries, reviewing trends in school response and explanations for challenges in school recruitment for three HRB and two surveys of academic performance to find out whether challenges were common to school surveys in general or were restricted to specific surveys, study types, and/or countries.

Our analysis suggests divergent trends for different types of survey in these four countries: Surveys focusing on academic performance have secured relatively good response rates, though there is a slight downward trend in recent years. By contrast, those that investigate HRB have increasingly struggled. We believe these divergent trends are best understood as a result of three intertwined factors. First, and most important, in line with previous research (e.g., Marshall, 2010; Sturgis et al., 2006), our study showed that school surveys have become an immensely popular method of data collection and this popularity has created serious problems of survey fatigue. Second, schools in developed countries have tended to be evermore exposed to managerialist pressures to perform, which has led to an increasingly sharp focus on these activities that are measured by performance indicators. Third, the combination of survey fatigue and managerialist targets have led schools to limit the number of surveys to which they will agree, and, in rationing research access, they are more likely—if agreeing at all—to prefer those surveys that focus on the academic achievement and could help to improve performance. Stricter ethical guidelines for surveys of children and young people, particularly when active consent is required, may create an additional barrier to research on sensitive topics, such as drug use or sexual health, as schools need to play a greater role in the consent process with parents, requiring more demand on their time and resources. This was certainly the case for the U.S. research team for ISRD-3. The interplay of these factors has, paradoxically, threatened the viability of school surveys specifically in Western European countries and in the United States—all of which have long traditions of social research with schools and students. Some non–Western European countries (e.g., India and Czech Republic) also had significant problems with gaining cooperation from the schools but, overall, it seems that low levels of school access is most typical of Western European countries (see Table 2; Enzmann et al., 2017).

School surveys that focus on academic achievement will probably continue to secure access to schools due to their topic salience, though, even they are not immune to survey fatigue. But what of the more socially relevant HRB surveys, such as ISRD, ESPAD, and HBSC? Can the developed—and relatively overresearched—countries that we have examined in this article hope to complete another round of in school surveys successfully? Despite downward trends in school response rates, much variation existed within and between countries and studies, implying that other factors, some of which are within researchers’ control, also affect school recruitment. In line with previous research (e.g., Lamb et al., 2001; Testa & Coleman, 2006; White, 2012), we suggested some aspects of school recruitment that should be considered thoughtfully in advance (see section “How Could Challenges in School Recruitment for ISRD-3 Be Explained?”) and provided some valuable lessons learned from ISRD-3 (see section “What Could Be Learned From ISRD-3 School Recruitment in England, Germany, the Netherlands, and the United States?”). Researchers should anticipate refusal, but a well-orchestrated plan at the start, sufficient resources to put it into effect, and a coordinated and dedicated team of researchers in each country should suffice to maximize response rates from the outset. Elements of this plan will need to vary from country to country, exploiting what is found to work locally.

The extensive autonomy that schools have in deciding on participation in research, combined with growing demands placed on schools, has made schools decisive actors whose interests have to be addressed in the process. An important challenge for cross-national multiwave research will be to initially establish what schools would value from the survey and find ways to accommodate these interests. Finally, with declining response rates, it becomes increasingly more important for researchers to collect information on characteristics of refusing and nonresponding schools—school size, level, proportion of low socioeconomic status (SES) students—to gain insight on selectivity of nonresponse. Using a simulation study on school refusal for ESPAD in Germany, however, Thrul, Pabst, and Kraus (2016) also found that refusal at the school level does not immensely affect the validity of estimated prevalence rates, which indicates that even some degree of selectivity in nonresponse and a smaller sample could be sufficient to obtain valid results.

We should draw attention to the limitations of this study. First, the study did not include school response rates from national academic and HRB surveys, which might have provided more robust evidence of trends in individual countries. However, we considered trends from international surveys most relevant to estimating likely rates for ISRD-4 given their broader policy focus and less tangible relevance to national school context. Second, our narrative review of factors affecting response rates may have missed barriers or facilitators that a more systematic review would have identified. However, we have provided an overview of factors that were prevalent for ISRD-3, which can contribute to a wider body of evidence on survey implementation in schools.

A further limitation relates to the fact that all the surveys considered in this article were carried out before the implementation of the European Union’s General Data Protection Regulation (GDPR), which came into force in 2018. A key requirement here is that whenever (with some exceptions) any organization collects personal data, they are required to secure the positive or active consent of those providing data. Children are deemed able to provide informed consent from the age of 16, with scope for some country-level variation (e.g., children aged 13 years plus can be offered online services under U.K. derogations). It is not yet clear what impact GDPR will have on the sorts of survey that we have considered. Although none of the surveys collected names, date of birth, or addresses, it could be argued that when children report on their age, sex, and grade, and also on individual experiences such as victimization, the data set could be construed as personal data, as the identities of at least some individuals could be inferred by a motivated intruder. Although researchers can take appropriate measures to address the risks of identification—for example, secure storage, data encryption, and restricted access—GDPR is likely to foster a much more cautious climate about the handling of data, especially when these are provided by children. Schools might insist upon positive consent from both parents and children aged below 16 years, exponentially increasing the difficulty of mounting school surveys. Clearly, GDPR poses a set of issues for those carrying out school-based surveys that need to be watched closely, as guidance and case law evolve.

This article has focused on the many obstacles that may confront school surveys. We feel that in the interests of balance, we should remind readers of the importance of these school surveys and of the benefits inherent in the methodology. School surveys, especially when conducted online, result in high individual (pupil) response rates and provide guarantees of privacy and anonymity. Where the topic is on delinquency and victimization, and on attitudes to crime and justice, the findings are of considerable relevance to policy and practice. And even where school response rates are low, these do not necessarily imply sample bias (as we have discussed above), and where school response rates are well under 50%, it should be remembered that such data sets can still be used for theory testing even if their point-estimates of prevalence and incidence may have to be interpreted with care.

Footnotes

Appendix

Appendix B

Linear and Quadratic Regression to Test Trends in School Response for Academic and HRB Surveys.

	HRB		Academic
	Linear	Quadratic	Linear	Quadratic
	b (SE)	b (SE)	b (SE)	b (SE)
Constant	86.48*** (10.28)	79.13*** (10.92)	78.32*** (5.09)	68.43*** (6.62)
Year	−1.50** (0.50)	0.08 (1.62)	1.38*** (0.32)	4.04** (1.27)
Year²		−0.07 (0.07)		−0.12* (0.05)
Germany	Ref.	Ref.	Ref.	Ref.
The Netherlands	−11.08 (7.28)	−10.68 (6.75)	−29.19*** (7.24)	−30.21*** (5.93)
England	−25.11* (9.54)	−23.86** (7.44)	−22.88*** (4.42)	−24.53*** (3.90)
United States	−17.61* (8.04)	−16.32* (6.94)	−24.44*** (4.68)	−26.09*** (3.90)
PISA			Ref.	Ref.
TIMSS			−0.41(4.34)	3.01 (4.73)
ESPAD	Ref.	Ref.
HBSC	−13.86** (4.75)	−14.52* (5.41)
ISRD	−27.70** (7.85)	−27.02*** (6.81)
R ²	.60	.62	.56	.62
Adjusted R²	.52	.52	.50	.55
N	37	37	40	40

Note. HRB = health and risk behavior; PISA = Programme for International Student Assessment; TIMSS = Trends in International Mathematics and Science Study; ESPAD = European School Survey Project on Alcohol and Other Drugs; HBSC = Health Behavior in School-Aged Children study; ISRD = International Self-Report Delinquency Study.

p < .05. **p < .01. ***p < .001.

Acknowledgements

We would like to thank Diego Farren and Katharina Neissl for providing technical reports, contact logs and other relevant information respectively for Germany and the US, and for their contribution to early drafts. We would also like to thank the reviewers and guest editors for their helpful comments.

Authors’ Note

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This paper was conceived from cross-country discussions that took place during the implementation of the International Self-Report Delinquence study (ISRD3) and draws on documentation from the research process. ISRD3 was funded by the Netherlands Organisation for Scientific Research (NWO 464-13-115) and the Economic and Social Research Council (ES/L016656/1) under an Open Research Area grant.

Notes

Author Biographies

Renske S. van der Gaag is PhD researcher in the Department of Political Science and Public Administration, Faculty of Social Sciences, at the Vrije Universiteit Amsterdam and senior researcher at the Amsterdam Research Institute for Societal Innovation at the Amsterdam University of Applied Sciences.

Lauren Herlitz is a PhD student in the Department of Public Health, Environments and Society at the London School of Hygiene & Tropical Medicine, and a research assistant at the Institute for Health and Human Development at the University of East London.

Mike Hough is emeritus professor in the School of Law, Birkbeck, University of London, and previously Director of the Institute for Criminal Policy Research.

References

Baruch

Holtom

B. C.

(2008). Survey response rate levels and trends in organizational research. Human Relations, 61, 1139-1160.

Bjarnason

(1995). Administration mode bias in a school survey on alcohol, tobacco and illicit drug use. Addiction, 90, 555-559. doi:10.1080/09652149542238

Bonell

Humphrey

Fletcher

Moore

Anderson

Campbell

(2014). Why schools should promote students’ health and wellbeing. BMJ, 348(7958), g3078.

Breakspear

(2012). The policy impact of PISA: An exploration of the normative effects of international benchmarking in school system performance (OECD education working papers, Vol. 71). OECD. Retrieved from https://ideas.repec.org/p/oec/eduaab/71-en.html

Brown

B. B.

Mounts

Lamborn

S. D.

Steinberg

(1993). Parenting practices and peer group affiliation in adolescence. Child Development, 64, 467-482.

Claudio

Stingone

J. A.

(2008). Improving sampling and response rates in children’s health research through participatory methods. Journal of School Health, 78, 445-451.

Currie

Nic Gabhainn

Godeau

(2009). The Health Behaviour in School-aged Children: WHO collaborative cross-national (HBSC) study: Origins, concept, history and development 1982-2008. International Journal of Public Health, 54(Suppl. 2), 131-139.

Enzmann

Kivivuori

Marshall

I. H.

Steketee

Hough

Killias

(2017). A global perspective on young people as offenders and victims: First results from the ISRD3 study. Cham, Switzerland: Springer.

Esbensen

F.-A.

Melde

Taylor

T. J.

Peterson

(2008). Active parental consent in school-based research: How much is enough and how do we get it? Evaluation Review, 32, 335-362.

10.

Fendrich

Johnson

T. P.

(2001). Examining prevalence differences in three national surveys of youth: Impact of consent procedures, mode, and editing rules. Journal of Drug Issues, 31, 615-642.

11.

Fulton

B. R.

(2018). Organizations and survey research: Implementing response enhancing strategies and conducting nonresponse analyses. Sociological Methods & Research, 47, 240-276.

12.

Gfroerer

(1985). Influence of privacy on self-reported drug use by youths. In Rouse

B. A.

Kozel

N. J.

Richards

L. G.

(Eds.), Self-report methods of estimating drug use (pp. 22-30). Washington, DC: U.S. Government Printing Office.

13.

Grek

(2009). Governing by numbers: The PISA “effect” in Europe. Journal of Education Policy, 24, 23-37.

14.

Hargreaves

Lieberman

Fullan

Hopkins

D. W.

(Eds.). (2014). International handbook of educational change: Part Two (Vol. 5). New York, NY: Springer.

15.

Hibell

Guttormsson

Ahlström

Balakireva

Bjarnason

Kokkevi

Kraus

(2012). The 2011 ESPAD report: Substance use among students in 36 European countries. Stockholm, Sweden: Modintryckoffset AB.

16.

Junger-Tas

Marshall

I. H.

Enzmann

Killias

Steketee

Gruszczynska

(2012). The many faces of youth crime. New York, NY: Springer.

17.

Kleinert

(2007). Adolescent health: An opportunity not to be missed. The Lancet, 369, 1057-1058.

18.

Kraus

Hibell

(2014). Whence and whither: Strengths and future challenges of ESPAD. Nordic Studies on Alcohol and Drugs, 31, 319-322.

19.

Lamb

Puskar

K. R.

Tusaie-Mumford

(2001). Adolescent research recruitment issues and strategies: Application in a rural school setting. Journal of Pediatric Nursing, 16,43-52.

20.

Marshall

I. H.

(2010). “Pourquoi pas?” versus “absolutely not!” Cross-national differences in access to schools and pupils for survey research. European Journal on Criminal Policy and Research, 16, 89-109.

21.

Meinck

Cortes

Tieck

(2017). Evaluating the risk of nonresponse bias in educational large-scale assessments with school nonresponse questionnaires: A theoretical study. Large-Scale Assessments in Education, 5(1), Article 3.

22.

Michaud

P.-A.

Delbos-Piot

Narring

(1998). Silent dropouts in health surveys: Are nonrespondent absent teenagers different from those who participate in school-based health surveys? Journal of Adolescent Health, 22, 326-333.

23.

Micklewright

Schnepf

S. V.

Skinner

(2012). Non-response biases in surveys of schoolchildren: the case of the English Programme for International Student Assessment (PISA) samples. Journal of the Royal Statistical Society: Series A: Statistics in Society, 175, 915-938.

24.

Rice

Bunker

K. D.

Kang

D.-H.

Howell

C. C.

Weaver

(2007). Accessing and recruiting children for research in schools. Western Journal of Nursing Research, 29, 501-514.

25.

Roberts

Freeman

Samdal

Schnohr

C. W.

de Looze

M. E.

Nic Gabhainn

. . . Rasmussen

(2009). The Health Behavior in School-Aged Children (HBSC) study: Methodological developments and current tensions. International Journal of Public Health, 54, 140-150.

26.

Shaw

Cross

Thomas

L. T.

Zubrick

S. R.

(2015). Bias in student survey findings from active parental consent procedures. British Educational Research Journal, 41, 229-243.

27.

Smit

de Zwart

Spruit

Monshouwer

van Ameijden

(2002). Monitoring substance use in adolescents: School survey or household survey? Drugs: Education, Prevention and Policy, 9, 267-274.

28.

Sturgis

Smith

Hughes

(2006). A study of suitable methods for raising response rates in school surveys. Nottingham, England: DfES.

29.

Testa

A. C.

Coleman

L. M.

(2006). Accessing research participants in schools: A case study of a UK adolescent sexual health survey. Health Education Research, 21, 518-526.

30.

Thrul

Pabst

Kraus

(2016). The impact of school nonresponse on substance use prevalence estimates—Germany as a case study. International Journal of Drug Policy, 27, 164-172.

31.

Vanderlinde

Van Braak

(2010). The gap between educational research and practice: Views of teachers, school leaders, intermediaries and researchers. British Educational Research Journal, 36, 299-316.

32.

Viner

R. M.

Ozer

E. M.

Denny

Marmot

Resnick

M. D.

Fatusi

Currie

(2012). Adolescence and the social determinants of health. The Lancet, 379, 1641-1652.

33.

White

L. S.

(2012). Conducting school-based research: Lessons learned from an after-school intervention study. Journal of Pediatric Health Care, 26, 75-79.