Abstract
This is the protocol for a Campbell evidence and gap map. The objective of this map is to systematically synthesize evidence on primary interventions for preventing cognitive and behavioral radicalization published between 2000 and 2025. It will categorize interventions by target population, intervention type, and outcomes. The EGM will indicate the quality of available evidence, identifying clusters and gaps to inform future research priorities, resource allocation, and Preventing and Countering Violent Extremism (P/CVE) programming.
Keywords
Background
Radicalization leading to violent extremism poses a significant and evolving challenge globally, prompting the development of diverse preventive strategies. Within the broader field of Preventing and Countering Violent Extremism (P/CVE), primary prevention interventions, those designed to stop individuals from engaging with extremist ideologies or behaviors in the first place, are considered crucial. This emphasis on primary prevention reflects a public health approach that prioritizes upstream efforts to prevent problems before they fully manifest.
The importance of such primary prevention lies in its proactive stance, aiming to address underlying vulnerabilities and build resilience across populations before extremist narratives take root or individuals embark on a path toward radicalization. By focusing on upstream factors such as enhancing critical thinking, fostering social cohesion, promoting positive identities, and mitigating psycho-social risks, these interventions seek to create an environment less conducive to extremism. Consequently, they aim to reduce the pool of individuals susceptible to radicalizing influences by bolstering protective factors and reducing risk factors at individual, community, and sometimes societal levels, thereby preventing the initial adoption of extremist worldviews or engagement in related harmful behaviors.
Despite the recognized importance of these primary interventions, the evidence base for their effectiveness and application is often dispersed across various sources, frequently lacks systematic synthesis, and many initiatives remain under-evaluated. This fragmentation significantly hinders the ability of policymakers, practitioners, and researchers to make evidence-informed decisions regarding program design, resource allocation, and priorities for future research.
To address this critical need, this protocol outlines the development of an Evidence and Gap Map (EGM). The EGM will systematically identify, characterize, and visually map the existing research landscape on primary interventions targeting cognitive and behavioral radicalization. By doing so, it will provide a comprehensive overview of available evidence, highlight areas where robust evaluations are lacking, and identify critical gaps where further research is most needed, thereby guiding efforts to build a stronger evidence base for effective primary prevention. The following sections will first detail the specific problem of radicalization that these interventions seek to address.
The Problem, Condition, or Issue: Understanding Radicalization
Navigating the concepts central to this EGM requires acknowledging a foundational challenge in the field: the terms ‘radicalization,’ ‘extremism,’ ‘violent extremism,’ and ‘terrorism’ are frequently used, often interchangeably or in deeply interconnected ways, despite having distinct conceptual nuances (Borum, 2011a; Schmid, 2013). While distinct, these concepts usually exist on a spectrum or represent escalating stages, where radicalization (the dynamic process of adopting increasingly extreme views and/or justifications for action) can sometimes lead to extremism (the state of holding views or endorsing behaviors that deviate significantly from societal norms) and, in some instances, to violent extremism or terrorism (the use or advocacy of violence to achieve particular ends). This lack of consistent definition and the tendency toward conflation present ongoing difficulties for research, policy, and practice.
The primary prevention interventions mapped in this EGM are designed to address the complex phenomenon of radicalization. Radicalization is widely understood as a progressive process where individuals or groups adopt increasingly extreme political, social, or religious beliefs and aspirations. However, it remains a contested concept with definitions and interpretations varying across diverse ideological, social, and political contexts (Borum, 2011a; Kundnani, 2012; McCauley & Moskalenko, 2008; Silva, 2018; Stampnitzky, 2013). This definitional ambiguity has implications for the framing of counter-terrorism measures and the creation of ‘suspect communities,’ which can marginalize specific groups based on ethnicity, religion, or political beliefs (Bal & Bos, 2017; Travalingo & Moon, 2022).
Several influential definitions provide insight into the core elements of radicalization. McCauley and Moskalenko (2008) define radicalization as a progressive process in which individuals or groups adopt increasingly extreme ideals, rejecting the status quo and advocating violence as a means of societal change. The United Nations (2016) similarly characterizes it as adopting extreme ideals that either reject the status quo or justify violence. Rizwan et al. (2021) describe it as a “battle of ideas” against societal norms, while Jugović and Živaljević (2021) frame it as a political, psychological, and social process within group dynamics. While definitions often emphasize the adoption of extreme beliefs justifying violence, they frequently overlook broader social and political dynamics, such as marginalization and power relations (Smith et al., 2019). Research increasingly recognizes that radicalization is not solely an individual psychological shift but is deeply influenced by group dynamics and societal contexts that shape collective identities and norms (Kundnani, 2012; Travalingo & Moon, 2022).
Schmid (2013) distinguishes between cognitive (adopting extremist beliefs), emotional (intense feelings motivating extremism), and behavioral (actions supporting or committing violence) radicalization. This tripartite model aligns with research highlighting the interplay between psychological and social factors in radicalization (Stekelenburg, 2014; Trip et al., 2019). While Borum (2011a) cautions against conflating ideological radicalization with terrorism, Stekelenburg (2014) emphasizes its emergence within broader intergroup tensions. Collectively, these perspectives underscore that radicalization is not a linear, isolated cognitive shift, but a complex dynamic where individual psychological vulnerabilities intersect with and are amplified by broader social and intergroup tensions.
Radicalization is often conceptualized as a continuum, moving from initial ideological commitment and social alienation toward potential engagement in disruptive or violent actions (Della Porta, 2013; McCauley & Moskalenko, 2017). There is, however, ongoing debate regarding whether non-violent extremism should be considered part of this continuum or treated as a distinct phenomenon (McCauley & Moskalenko, 2017; Moghaddam, 2005; Neumann, 2013). This continuum can involve various stages, including ideological commitment, social alienation (McCauley & Moskalenko, 2017), propaganda and recruitment (Conway et al., 2019; Weimann, 2015), civic disruption (Della Porta, 2013), targeted violence, and terrorism (Schmid, 2013). The involvement of individuals in extremist networks (Sageman, 2004) and the subversion of institutional norms (Borum, 2011a) are also key outcomes of the radicalization process, though their relationship to violent extremism is complex. Additionally, gender influences how individuals engage with and contribute to extremism (Rothermel, 2020).
Over the past two decades, radicalization has become a growing global concern, influenced by a complex interplay of socio-political and economic factors. Research has documented a rise in various forms of extremist activity, including increases in extremist groups, the proliferation of online extremist content, and related violent incidents (BDI, 2024; HOPE not hate, 2024; SPLC, 2022). Scholars have identified the expansion of social media as a significant facilitator for the dissemination of radical narratives and recruitment (McAlexander et al., 2021; Puigvert et al., 2020). The events of September 11, 2001, and the subsequent ‘War on Terror’ significantly shaped contemporary counterterrorism (CT) and countering violent extremism (CVE) approaches. However, some of these measures have been criticized for inadvertently fueling grievances among marginalized communities, potentially contributing to cycles of radicalization (Ebbrecht, 2022; Holla, 2020; Kupper & Meloy, 2021; Piazza, 2011). Radicalization processes, which can lead to various forms of extremist thought and behavior, are often shaped by, or expressed through, a broad spectrum of extremist ideologies. These include, but are not limited to, far-right, far-left, ethno-nationalist, and religiously-inspired extremism, each with distinct narratives and objectives, despite the historically disproportionate focus of counter-extremism efforts on particular ideological threats over others (Kundnani, 2012; Schmid, 2013; Heath-Kelly, 2013). It is also recognized that more idiosyncratic grievances may drive some pathways to extremist violence or less formally structured belief systems. Factors such as demographic shifts and economic decline have been linked by some research to the rise of right-wing extremism in Western societies, potentially through impacts on feelings of disenfranchisement (Bolet, 2020; Gidron & Mijs, 2019; McAlexander et al., 2021; Piazza, 2016; Rousseau et al., 2021).
The impacts of radicalization are felt at both macro and micro levels. Societally, it can contribute to fragmentation, increased violence, and the erosion of social cohesion (Brauner, 2023; Gelfand et al., 2013; Herrera & Kydd, 2022; Puigvert et al., 2020; Tesfai et al., 2023). Individually, radicalization can involve profound psychological changes, often stemming from feelings of alienation or injustice (Holla, 2020; Rousseau et al., 2021). This transformation can, in some cases, lead to engagement in violent extremism, posing risks to public safety (McAlexander et al., 2021). Additionally, the process can also have detrimental effects on mental health as individuals grapple with the moral implications of their actions and the potential for social ostracization (Horgan & Braddock, 2010; Miller et al., 2024).
Addressing radicalization effectively presents numerous challenges. Beyond the lack of consensus on definitions, the field itself faces debates regarding its status and the efficacy of counter-radicalization strategies, many of which have been criticized for overlooking root causes such as social inequality and political disenfranchisement (Shortland et al., 2017; Uscinski et al., 2021). The role of social media further complicates efforts to counter extremist content (Clever et al., 2023; Jackson, 2019; Kenyon et al., 2022). Moreover, the stigmatization and marginalization of certain communities, such as Muslims and racial and ethnic minority communities, exacerbated by discriminatory policies or negative media portrayals, can hinder prevention efforts and increase vulnerability (Abbas, 2005; Rashid & Riaz, 2022; Rieger et al., 2019). The global nature of radicalization also necessitates coordinated, multidisciplinary strategies that can navigate differing political contexts and definitional challenges (Gaikwad et al., 2021; Ismail, 2022; Youngblood, 2020).
Acknowledging that radicalization is a complex process with diverse manifestations, not all of which involve violence (McCauley & Moskalenko, 2017; Neumann, 2013), for this Evidence and Gap Map (EGM), we developed the following working definition that synthesizes key insights from the literature: Radicalization is the process by which individuals or groups undergo a transformation in their beliefs, often involving extreme political, social, or religious views. This process typically includes cognitive and social changes that lead to behaviors justifying or supporting violence, non-violent extremism, or other harmful actions to achieve ideological, political, or religious goals. Radicalization can be driven by a complex interplay of psychological, social, and contextual factors, including grievances, perceived injustice, and a desire for belonging.
Cognitive and Behavioral Dimensions of Radicalization
Radicalization is often described as a multifaceted process rather than a singular event (McCauley & Moskalenko, 2017; Moghaddam, 2005). Contemporary research (e.g., Borum, 2011b; Horgan & Braddock, 2010) increasingly highlights the dynamic interplay of cognitive, emotional, and behavioral factors in this process, although disentangling these elements for analysis and intervention remains a key challenge in the field (Doosje et al., 2016; Kruglanski et al., 2014). This EGM focuses on the cognitive and behavioral dimensions, recognizing their importance in the literature on radicalization processes (Schmid, 2013; Trip et al., 2019) and their relevance to primary interventions. While we acknowledge the significant influence of emotional and social factors (Bhui et al., 2014; Ellis et al., 2021), these are not the primary focus of this map.
We recognize that cognitive and behavioral radicalization are often seen as sequential stages, with changes in thinking occurring before behavioral actions. However, in primary prevention, addressing behaviors does not mean waiting until violence happens. Instead, it focuses on targeting early indicators like social withdrawal or seeking importance through negative peer groups and offering healthier behavioral options. Engaging individuals in pro-social activities such as civic participation, sports, or community leadership helps meet psychological needs for belonging and purpose, which might otherwise lead them toward extremism (Kruglanski, Bélanger, & Gunaratna, 2014; Stephens et al., 2021). Therefore, this EGM aims to identify interventions that strengthen both behavioral and cognitive resilience.
Cognitive Radicalization
Cognitive radicalization involves internalizing extremist beliefs and narratives. This process often begins with the adoption of ideologies that demonize out-groups, promote conspiracy theories, or glorify violence as a means to achieve ideological goals (Bartlett & Miller, 2012; Borum, 2011b; Moghaddam, 2005). Individuals may develop attitudes that condone or encourage violence against perceived enemies, often rooted in dehumanization and a belief in the necessity of violence to defend the in-group or cause (Hogg, 2014; Moghaddam, 2005). This is frequently accompanied by distorted perceptions of reality, including a biased worldview, selective interpretation of information, and an echo chamber effect that reinforces extremist beliefs and rejects contradictory evidence (Borum, 2011b). Furthermore, an “us vs. them” mentality emerges, fostering a substantial in-group/out-group distinction, often accompanied by a sense of victimhood and the belief that violence against the out-group is justified (Hogg, 2014; Moghaddam, 2005). Therefore, we use the following operationalization: Cognitive Radicalization refers to the internalization of extremist beliefs, ideologies, and narratives that may eventually justify violence or harmful actions. This cognitive process entails profound shifts in an individual’s thinking, attitudes, and perceptions, resulting in the adoption of increasingly extreme views. It represents the psychological mechanisms that contribute to the formation of extremist identities, which can then drive individuals toward violent extremism or terrorism.
Behavioral Radicalization
Behavioral radicalization encompasses observable actions indicating a progression toward extremism, ranging from passive support to active violence. These behaviors often reflect internal cognitive processes but can further influence belief systems (Moghaddam, 2005). Key indicators include consuming extremist propaganda, which reinforces radical ideologies and normalizes violence (Klausen, 2015; Silke, 2008); engaging in online hate speech, which allows for the free expression and normalization of extremist views (Awan & Zempi, 2015; Klausen, 2015); joining extremist groups, which provides social reinforcement and strengthens commitment to extremist ideologies (Sageman, 2004); and participating in activities that prepare for violence, such as paramilitary training or acquiring weapons (Awan & Zempi, 2015). The interplay between cognitive and behavioral dimensions is well-established, and studies (e.g., Bhui et al., 2014) indicate that while cognitive and behavioral dimensions are intertwined, emotions like anger, injustice, alienation, or fear also play a significant role in driving individuals toward radical behaviors. While this behavioral spectrum encompasses violent acts, this EGM focuses on the primary prevention of these outcomes by targeting their early precursors (e.g., social isolation) and promoting protective behavioral alternatives (e.g., civic engagement). Therefore, we use the following operationalization: Behavioral Radicalization refers to the observable actions and behaviors that signal a progression toward extremism. These behaviors can vary widely, from passive support to active participation in violent acts. Notably, the behavioral dimension of radicalization often reflects the internal cognitive process but can also influence or catalyze further shifts in belief systems.
The Intervention
Preventing and Countering Violent Extremism (P/CVE)
Preventing and Countering Violent Extremism (P/CVE) is a strategic response that addresses the root causes of radicalization to mitigate its societal impacts. The P/CVE framework, along with the concept of ‘radicalization’ it seeks to address, is subject to ongoing debate and criticism within academia and civil society. Concerns have been raised regarding the potential for securitization, the stigmatization of certain communities, and the oversimplification of complex socio-political issues that may contribute to extremism (Kundnani, 2012; Heath-Kelly, 2013; Ragazzi, 2017). Indeed, as noted in Understanding Radicalization, an overemphasis in policy or research on any single ideological manifestation can contribute to these concerns by leading to an incomplete picture of radicalization dynamics and potentially inequitable application of P/CVE measures.
Unlike the focus on individual radicalization processes, P/CVE encompasses a broader range of interventions aimed at preventing various violent outcomes, from non-violent extremism to terrorism and building community resilience (Stevens, 2009; Ucko, 2018; Weine et al., 2013). Emerging in response to the early 21st-century rise in terrorism, P/CVE’s comprehensive framework includes community engagement, education, policy development, and socio-economic initiatives to foster resilience to extremist ideologies (Ucko, 2018).
P/CVE interventions are proactive, targeting the socio-economic, political, and psychological drivers of radicalization, with an emphasis on early intervention and community involvement (Fransen, 2023; Kolbe, 2019). Key features of P/CVE include a focus on prevention, addressing conditions like social exclusion and marginalization (Dalgaard-Nielsen, 2010); a multi-dimensional approach, targeting individual, community, and societal factors (Silke, 2018); and context-specific interventions tailored to local realities (Mattsson, 2019; Neumann, 2016). Core components of P/CVE include community engagement through grassroots organizations (Fransen, 2023); educational initiatives promoting critical thinking and media literacy (Weine & Kansal, 2019); socio-economic development to address economic disparities and marginalization (“Community-Based Preventive Measures,” 2018); counter-narratives to challenge extremist ideologies (Busher et al., 2023); gender-sensitive approaches (Rothermel, 2020); collaborative stakeholder engagement across government and civil society (Abbas, 2018); and robust evaluation and adaptation to ensure effectiveness (Baruch et al., 2018).
Defining Primary Interventions
This EGM focuses specifically on primary interventions for preventing cognitive and behavioral radicalization. These interventions are understood within a broader public health model (see The Public Health Model and Appendix 2 for further details on prevention levels), which conceptualizes prevention across multiple levels (e.g., primordial, primary, secondary, and tertiary) to address health and social problems proactively (Caplan, 1964; Gordon, 1983; Mrazek & Haggerty, 1994). For this EGM, we have developed the following definition of primary interventions, which aligns with the primary level of this public health continuum: Primary interventions within a public health framework are proactive strategies implemented before the onset of the problem or any clear initial symptoms/signs (in this case, radicalization or violent extremist behaviors) to reduce risk factors, enhance protective factors, and build resilience at both the individual and community levels, preventing the initial adoption of extremist ideologies. They encompass two subcategories within this primary prevention level:
Why It Is Important to Develop the EGM
The fragmented research landscape and the urgent need for evidence-informed prevention strategies underscore the importance of an EGM on primary interventions for preventing cognitive and behavioral radicalization. Many primary interventions, especially community-based ones, lack rigorous impact evaluations (Bressan et al., 2024; Neyroud et al., 2023), leaving significant areas of ‘evidence of absence,’ areas where evaluated evidence is sparse or non-existent despite potential policy interest. The absence of a robust, evidence-based foundation in P/CVE raises not only questions of effectiveness but also the risk of iatrogenic effects. Interventions relying on ill-defined markers of risk may inadvertently cause harm, including stigmatization of targeted communities, erosion of trust in institutions, and the reinforcement of grievances (Khalil, 2019; Mastroe & Szmania, 2016; Scarcella et al., 2016; Vidino & Hughes, 2015).
This EGM will identify interventions with varying levels of evaluation evidence, including those with rigorous impact evaluations, outcome evaluations, process evaluations, internal evaluations, and those primarily documented through practitioner reporting, highlighting gaps and directing future research toward under-evaluated areas (Snilstveit et al., 2016; Sydes et al., 2023). By systematically searching for and mapping available evaluative information, the EGM will illuminate ‘evidence of absence’, pinpointing specific intervention types, target populations, or outcomes for which no, or very limited, evaluated evidence currently exists despite potential policy or practitioner interest. Identifying these areas of absence is as important as identifying areas of evidence concentration for guiding future research and investment. This approach enables researchers, policymakers, and practitioners to gain a comprehensive overview of what works, identify knowledge gaps, and prioritize future research, especially regarding interventions lacking rigorous evaluation or evidence-based selection of risk and protective factors (Neyroud et al., 2023; Wolfowicz et al., 2021).
The EGM categorizes interventions by target population, type, and geographical context, which will inform more targeted research and resource allocation (Neyroud et al., 2023; Snilstveit et al., 2016; Williams et al., 2015). Moreover, the user-friendly visual map enhances accessibility for practitioners, improving practice by translating research into actionable insights (Cochrane, 2022a; 2022b). This EGM will pave the way for future systematic reviews and meta-analyses by facilitating knowledge synthesis, leading to more definitive conclusions about intervention effectiveness and informing evidence-based decision-making. The EGM methodology is particularly well-suited to this field due to its ability to map diverse evidence, highlight gaps, accommodate context-specific interventions, and enhance accessibility for practitioners and policymakers (Cochrane, 2022b; Snilstveit et al., 2016; Wolfowicz et al., 2021).
While several existing EGMs and systematic reviews are valuable contributions to the field of P/CVE, they do not fully address this EGM’s specific focus on primary prevention interventions targeting cognitive and behavioral radicalization. For instance, Sydes et al.’s (2023) EGM, while comprehensive in its coverage of criminal justice interventions, primarily focuses on secondary and tertiary prevention efforts aimed at individuals already engaged in extremist behaviors or incarcerated. This leaves a significant gap in our understanding of the evidence base for primary prevention interventions, despite their widespread use in national and local P/CVE programming.
Similarly, Zych & Nasaescu’s (2022) and Sarma et al.’s (2022) systematic reviews, while addressing family-related factors, focuses on interventions for families of individuals already exhibiting signs of radicalization rather than primary prevention strategies. Del Pino‐Brunet et al. (2021) review examines a broad range of interventions, including secondary and tertiary approaches, and focuses on migrant populations, limiting its generalizability to other at-risk groups. Bressan et al.’s (2024) systematic review of the effectiveness of P/CVE interventions also includes a mix of primary, secondary, and tertiary interventions, making it difficult to isolate the evidence related explicitly to primary prevention.
Rizwan et al. (2021) offer a critical analysis of de-radicalization interventions, which are tertiary and thus outside the scope of this EGM. Several reviews have focused on specific types of interventions—such as counter-narratives (Carthy et al., 2020; Treacy et al., 2024), educational programs (Davies, 2018), or realist-informed mechanisms (Gielen, 2017), but these are typically constrained to particular modalities or theoretical approaches rather than mapping the full breadth of primary prevention strategies. Pistone et al.’s (2019) scoping review maps broad P/CVE interventions but does not focus exclusively on primary prevention or distinguish between cognitive and behavioral outcomes, and excludes relevant literature beyond terrorism-focused interventions. Additionally, Windisch et al.’s (2022) systematic review of online interventions focuses on reducing hate speech and cyber hate, which, while related, does not encompass the full spectrum of online primary prevention efforts targeting cognitive and behavioral radicalization. While Wolfowicz et al. (2021) provide a systematic review of risk and protective factors, their work does not map the interventions, nor is there a focus on primary interventions.
Other reviews have made valuable contributions in this space, but remain limited in scope. Brouillette-Alarie et al. (2022) conducted a systematic review on the outcomes of primary and secondary prevention programs, offering insights into effectiveness measures, but without mapping the breadth of intervention types. They also make the critical observation that primary prevention can be either universal or selective in scope. This distinction is essential when applying the public health model to P/CVE, where many interventions target groups considered vulnerable but not showing distinct signs of radicalization. Wolfowicz et al. (2022) reviewed 26 primary prevention studies and classified the targeted risk and protective factors, but their review did not differentiate by program delivery context or outcome type. At the level of individual studies, Hirschfield et al. (2012) conducted an early process evaluation of UK youth-focused primary interventions, identifying mechanisms of change but not systematically categorizing outcomes or evaluating effectiveness. Collectively, this existing literature, while important, does not offer a comprehensive global mapping of primary prevention interventions by outcome, delivery modality, or population group.
This fragmented landscape, with its emphasis on interventions after radicalization has commenced, and the lack of synthesis focused solely on primary prevention, underscores the need for a dedicated EGM that systematically maps primary prevention interventions and their outcomes, focusing specifically on the cognitive and behavioral dimensions of radicalization before engagement with extremist ideologies.
Objectives
This EGM aims to systematically map the evidence on primary interventions for preventing cognitive and behavioral radicalization from 2000 to 2025. The EGM will categorize interventions by target population, intervention type, and outcomes measured. The key objectives of this EGM are: A. To identify and characterize primary interventions designed to prevent cognitive and behavioral radicalization, by mapping interventions documented with varying forms and levels of evaluative information ranging from impact evaluations, outcome evaluations, and process evaluations, to internal reports and practitioner reporting that offers insights into intervention design, implementation processes, and observed outcomes or lessons learned. This characterization will encompass the target population, intervention type, setting, theoretical framework, intended outcomes, implementation characteristics (e.g., duration, intensity, delivery mode), and outcome measures used (regardless of reported effectiveness). B. To identify and characterize gaps in the evidence base, specifically related to primary interventions designed to prevent cognitive and behavioral radicalization. This characterization of gaps will involve the following: (a) Identifying interventions with limited or no published evaluations, including those documented primarily through internal assessment, program descriptions, or practitioner insights (gray literature). (b) Identifying interventions where the measured outcomes do not adequately capture the intended impact on cognitive and behavioral dimensions of radicalization (e.g., relying solely on general well-being measures without assessing specific attitudes or beliefs related to extremism). (c) Identifying interventions with limited data on implementation characteristics (e.g., fidelity, dosage, adaptation), contextual factors (e.g., setting, community characteristics), stakeholder engagement, and unintended consequences. (d) Highlighting gaps in the types of interventions, target populations, and geographical contexts that have been evaluated, and describing broad observable patterns in the evidence base, such as general temporal trends in evaluation publication, if apparent.
Methods
Evidence and Gap Map: Definition and Purpose
This project employs an EGM methodology to systematically organize and present the available research evidence concerning primary cognitive and behavioral radicalization interventions. EGMs offer a distinct approach to synthesizing evidence, moving beyond traditional systematic reviews to provide a broader perspective on what research exists and where knowledge gaps remain (Snilstveit et al., 2016). They achieve this by systematically mapping evidence across various dimensions, including intervention types, target populations, and outcomes. A key feature of EGMs is their ability to convey this mapped evidence landscape through visual and often interactive representations, thereby enhancing accessibility and utility for stakeholders.
This allows for a high-level overview of existing research, facilitating the identification of evidence clusters and areas requiring further investigation, and, evidence of absence. Snilstveit et al. (2016) highlight that EGMs are particularly useful for presenting complex information in an accessible format, often using a matrix structure to display the relationship between interventions and outcomes. While rigorous search methods, similar to those used in systematic reviews, are employed to identify relevant literature (Snilstveit et al., 2016), the EGM’s purpose is not to provide definitive answers about intervention effectiveness but to map the terrain of existing research. This makes EGMs particularly valuable for complex social interventions like radicalization prevention, where the evidence base may be fragmented, and diverse interventions address varied aspects of the problem.
This specific EGM aims to map the evidence landscape for the period 2000–2025, with a focus on primary interventions. It will include both primary studies and systematic reviews as sources of evidence, offering a comprehensive overview of the available data that encompasses interventions targeting both cognitive and behavioral dimensions of radicalization. By highlighting the distribution of research across these dimensions, the EGM will: • • •
In line with the emphasis on using EGMs as public goods, this project aims to create an accessible and user-friendly resource that empowers stakeholders to make informed decisions about radicalization prevention strategies and research priorities.
Framework Development and Scope
The framework for this Evidence and Gap Map (EGM) was developed through a multi-stage process, incorporating input from various sources:
This iterative process ensured that the framework is grounded in both academic literature and insights from policy development, as well as relevant programmatic experience, reflecting current policy priorities and stakeholder needs. While radicalization and efforts to prevent it are global phenomena, this initial EGM will, in practice, likely yield a preponderance of evidence originating from, or pertaining to, interventions in what is often termed the ‘Global North’ (e.g., North America, Europe, Australia, New Zealand). This anticipated outcome is primarily due to considerations of feasibility related to the review team’s primary language expertise (English) and the greater accessibility of published and gray literature on evaluated P/CVE programs from these regions within the project’s timeframe and resources. Our search strategy, while designed to be broad, is therefore likely to identify a larger volume of studies from these more extensively documented regions. We acknowledge this as a practical limitation and recognize the rich and diverse experiences and interventions in the ‘Global South,’ which warrant dedicated future research and mapping efforts.
This Evidence and Gap Map (EGM) aims to systematically map the available evidence on primary interventions designed to prevent radicalization, focusing on the period from 2000 to 2025. We define primary interventions as proactive strategies implemented before the onset of radicalization or violent extremist behaviors. These strategies aim to reduce risk factors, enhance protective factors, and build resilience at both the individual and community levels, preventing the initial adoption of extremist ideologies. This includes both: • •
Interventions focusing on secondary (e.g., counseling for at-risk youth showing early warning signs) or tertiary (e.g., disengagement and rehabilitation programs) prevention are excluded from this analysis.
The EGM will categorize interventions based on their primary approach (see Types of Intervention for detailed descriptions and examples of each intervention type): • Community-Based Interventions • Educational Interventions • Mentorship Programs • Family-Based Interventions • Sports-Based Interventions • Online Interventions • Psychosocial Interventions • Arts and Culture-Based Interventions • Public Awareness Campaigns • Other (to be specified, with a concise description of the intervention type)
The EGM will also categorize interventions by their target population (see Types of Population for detailed descriptions): • General Populations • At-Risk Individuals and Groups (who meet the definition of ‘at-risk’ for selective primary prevention outlined in Types of Population) • Family Members and Peer Networks • Digital Communities • Locations and Environments • Other (to be specified, with a concise description of the population type)
The EGM will map these intervention types and target populations against a range of outcome domains, including resilience and protective factors, attitudinal and behavioral outcomes, community-level outcomes, institutional outcomes, psychosocial outcomes, and unintended consequences (see Types of Outcome Measures for details).
The primary organizing structure of the EGM will be a matrix, with intervention types as rows and outcome domains as columns. The map will visually represent the distribution of evidence, highlighting areas of concentration and gaps in the research.
While we acknowledge the theoretical distinction between cognitive and behavioral dimensions of radicalization, we recognize that, in practice, primary prevention interventions may not always explicitly or exclusively target one dimension over the other. Many interventions aim to address both cognitive and behavioral aspects, sometimes simultaneously or with an understanding that influencing cognitions (e.g., extremist attitudes) is intended, even if indirectly, to affect behaviors (e.g., susceptibility to mobilization).
Therefore, to accurately reflect the potential multifaceted nature and the stated objectives or scope of the interventions themselves, rather than forcing a single primary focus, our coding instrument (Appendix 5) includes separate items to assess whether an intervention is described as addressing cognitive aspects, behavioral aspects, or both. This approach allows for a more nuanced mapping, enabling interventions to be coded as primarily targeting one dimension, both, or having an unclear primary dimensional focus based on the available documentation.
Stakeholder Engagement
Initial consultations were held with representatives from the Department of Homeland Security and Public Safety Canada to establish the objectives and scope of this EGM. This discussion identified key themes, outcome categories, and areas requiring further exploration. As part of our stakeholder engagement strategy, we will convene an Advisory Group of experts and practitioners from diverse fields relevant to P/CVE. This group will include representatives from government agencies, academic institutions, non-governmental organizations, and international partners.
We aim to ensure that the Advisory Group represents a diverse range of perspectives and expertise. Invitations will be extended to stakeholders from key regions globally, including North America (e.g., United States, Canada), Europe (e.g., United Kingdom, Norway, Netherlands), Oceania (e.g., Australia, New Zealand), as well as actively seeking representation from various countries and experts across Asia, Africa, and Latin America. Researchers and practitioners with specific expertise in P/CVE strategies, radicalization prevention, and community resilience will also be engaged. This diverse membership will provide critical insights into global practices and help identify cross-contextual lessons. The Advisory Group will contribute in several key ways:
To promote transparency, the final report will include a list of participating organizations and experts, acknowledging their invaluable contributions to the project. This inclusive approach ensures that the EGM is methodologically robust and reflects the practical realities and priorities of those engaged in P/CVE work worldwide.
Conceptual Framework
This EGM is grounded in a public health approach to preventing and countering violent extremism (P/CVE). This approach emphasizes proactive, multi-level interventions that address the root causes of radicalization (CDC, 2024; DHS, 2024; Jugl et al., 2020). Our conceptual framework draws upon and integrates several key models and approaches to guide the EGM’s scope, the categorization of interventions, and the understanding of potential outcome pathways. These include:
The Public Health Model
As detailed in Defining Primary Interventions, the public health model provides a framework for understanding radicalization as a preventable phenomenon similar to other public health challenges like disease or injury. This model emphasizes a multi-tiered prevention framework, examples and further details of which are provided in Appendix 2. These levels include, as adapted from the Department of Homeland Security (DHS, 2024) and Gordon (1983): • • • •
Risk and Protective Factors
This paradigm is central to the conceptual framework underpinning primary prevention interventions. It posits that an individual’s vulnerability to radicalization is influenced by a complex interplay of risk factors (variables that increase the likelihood of radicalization) and protective factors (variables that decrease the likelihood or buffer against risk). These factors can operate at multiple levels of the social-ecological model, including individual, interpersonal, community, and societal levels (Borum, 2011b; DHS, 2010). Primary prevention interventions, within this paradigm, are designed to mitigate risk factors and bolster protective factors. Examples of risk factors include social isolation, exposure to extremist narratives, and experiences of discrimination. Examples of protective factors include strong social support networks, critical thinking skills, and a positive sense of identity and belonging.
These factors are theorized to shape both cognitive outcomes (e.g., extremist attitudes, beliefs, and worldview) and behavioral outcomes (e.g., participation in extremist activity, violence, or criminal mobilization). For instance, reduced social isolation or enhanced critical thinking may disrupt cognitive susceptibility to extremist narratives, thereby reducing the likelihood of behavioral engagement.
This framework assumes that reducing risk factors and strengthening protective factors—whether cumulatively, independently, or interactively—contributes to the prevention of radicalization. Interventions may produce change directly at the behavioral level or indirectly through cognitive shifts (e.g., reducing attitudinal support for violence), which in turn decrease behavioral risk.
These relationships are visually summarized in the logic model (see Figure 1), which illustrates how primary prevention interventions aim to reduce radicalization by modifying risk and protective factors at multiple levels of the social ecology. Logic Model Situating Primary Prevention Within the Broader Frameworks of Public Health and Socio-Ecology
Social-Ecological Model
This model (Bronfenbrenner, 1979; McLeroy et al., 1988) complements the public health and risk/protective factor approaches by emphasizing the multi-level influences on individual behavior. It recognizes that individuals are embedded within nested systems of influence, which need to be addressed holistically to be effective (see Appendix 2). These levels include: • • • •
Primary prevention interventions can target any or all of these levels.
Logic Model
This logic model draws on and extends the structure proposed by Brouillette-Alarie et al. (2022) in their systematic review of tertiary prevention outcomes in the field of violent radicalization. Figure 1 illustrates how primary prevention interventions, both universal and selective, are situated within a broader public health continuum. These interventions target modifiable risk and protective factors across societal, community, and individual levels, and operate through mechanisms such as trust-building, identity affirmation, and critical media literacy. These mechanisms are theorized to produce cognitive outcomes (e.g., improved resistance to extremist messaging) and behavioral outcomes (e.g., reduced susceptibility to participation). While secondary and tertiary prevention pathways are shown for conceptual framing, they fall outside the scope of this review. The examples presented in the figure are illustrative only; a more detailed articulation of interventions, populations, and outcome measures is provided in Conceptual Framework and Dimensions.
Dimensions
Types of Study Design
This EGM will include a range of quantitative study designs, prioritizing the inclusion of diverse methodologies while assessing their quality to illuminate evidence gaps. While randomized controlled trials (RCTs) are widely regarded as the gold standard for causal inference, practical and ethical constraints often limit their use in interventions addressing violent extremism and radicalization. Therefore, this review will incorporate robust quasi-experimental designs and other comparative approaches. Study designs that will be eligible for this EGM include • • ○ Matched control group designs (with or without baseline measures), utilizing propensity score or statistical matching. ○ Difference-in-difference (DiD) analyses comparing pre- and post-intervention data across treatment and control groups. ○ Interrupted time series designs with pre- and post-intervention measures, both short (<25 observations) and long (≥25 observations). ○ Regression discontinuity designs, leveraging intervention eligibility thresholds. ○ Non-randomized comparison groups with contextually valid control groups and multivariate controls. • • • •
Studies with methodological limitations will not be excluded solely on the basis of their design. Instead, their quality will be assessed according to Tools for Assessing Risk of Bias/Study Quality, and findings will be documented to provide a comprehensive understanding of the evidence landscape, highlighting both strengths and weaknesses in the existing research. Coding study quality and other characteristics allows this EGM to identify where evidence is strong and limited by design or practical constraints.
For practitioner reports or gray literature (captured under ‘Observational Outcome Evaluations’ or ‘Qualitative Studies’) to be included, they must go beyond purely descriptive accounts of the intervention. They must contain at least some documented observation, reflection, or assessment of the intervention’s processes, perceived effects on participants (related to cognitive or behavioral aspects of radicalization), or lessons learned concerning preventing radicalization. Studies meeting inclusion criteria but lacking published results will be identified and documented as ongoing research, providing a roadmap for future evidence updates.
This EGM will include studies published or reporting on interventions conducted from January 2000 to July 2025. The decision to select this timeframe, particularly encompassing the period after the early 2000s, reflects a strategic effort to map evidence from an era of significant evolution and transformation in P/CVE approaches. While earlier interventions existed before 2000, the subsequent two decades have seen notable shifts in the nature of extremist threats, the development of intervention strategies, and the methodologies used for their evaluation. Key considerations for this timeframe include: • • • • •
Types of Intervention
Our EGM will capture primary interventions aimed at preventing radicalization, violent extremism, and related harmful behaviors. For an intervention to be included, the study or program documentation must explicitly state that the intervention is intended to prevent radicalization, violent extremism, or related outcomes, or it must clearly articulate how the intervention targets known risk factors for radicalization or aims to bolster protective factors specifically in the context of preventing extremism. Thus, interventions with broad social or educational aims (e.g., general youth employment or standard digital literacy) will only be eligible if their documentation explicitly connects them to P/CVE goals or relevant risk/protective factors for radicalization.
For this EGM, primary interventions are defined as proactive efforts designed to address the factors contributing to radicalization before individuals develop the extremist cognitions (e.g., adopting extremist worldviews) or engage in the related harmful behaviors (e.g., extremist expressions or actions), as detailed in Cognitive and Behavioral Radicalization, that characterize established radicalization. These interventions target underlying vulnerabilities, build resilience, and reduce susceptibility to extremist influences across various ideologies, including political, social, and religious.
Eligible interventions must focus on primary prevention interventions. We will categorize interventions based on their primary approach, although we acknowledge that some interventions may incorporate elements of multiple approaches. The following categories will be used, with examples provided for each. This list is intended to be as comprehensive as possible within the scope of primary prevention, but we acknowledge that new or hybrid approaches may emerge. We will remain open to including interventions that meet the definition of primary prevention, even if they don’t fit neatly into these categories.
Many primary interventions target risk factors comprehensively, but some are designed to disrupt specific cognitive or behavioral pathways. For instance, Educational Interventions, like media literacy, mainly aim to reduce cognitive vulnerability to extremist narratives. Meanwhile, Sports-Based and Community-Based Interventions focus on behavioral radicalization by promoting alternative behaviors. These programs are based on the idea that involving individuals in positive, status-enhancing activities can decrease behavioral risk factors such as isolation and grievance, which may lead to mobilization (UNODC, 2020). By differentiating these approaches, the EGM will help determine if the evidence supports cognitive-focused strategies more than behavioral ones or if they are often combined. ○ Examples: Interfaith dialogue initiatives, community festivals celebrating diversity, youth leadership programs promoting civic engagement, and community-based conflict resolution workshops. ○ Examples: Curriculum units on critical media analysis, classroom discussions on controversial issues and diverse viewpoints, programs promoting intercultural understanding and respect. ○ Examples: Structured mentorship programs connecting youth with adult mentors who provide guidance, support, and positive social connections. ○ Examples: Parenting workshops focusing on positive communication, conflict resolution, and building strong family bonds; family-based programs promoting cultural understanding and integration. ○ Examples: Initiatives that use sports to teach teamwork. ○ Examples: Public awareness campaigns promoting digital literacy and critical thinking skills; online resources promoting tolerance and understanding; platforms fostering positive online communities and counter-narratives (focused on prevention with the goal of inoculating them against extremist messaging or building general resilience, not countering existing extremist views through debunking specific extremist viewpoints held by individuals already showing early signs of radicalization). ○ Examples: Resilience training interventions for youth, workshops on stress management and coping skills, interventions promoting positive self-esteem and identity development, and social skills training programs. ○ Examples: Community theatre projects addressing social issues, music and art workshops promoting cross-cultural understanding, storytelling initiatives sharing diverse perspectives. ○ Examples: Campaigns promoting media literacy and critical thinking, campaigns highlighting the dangers of extremism and promoting positive alternatives, campaigns fostering interfaith dialogue and understanding.
Eligible comparison groups for inclusion will include: No intervention Standard practices (“business as usual”) Alternative interventions Waitlist control
Types of Population
The following populations will be included in this EGM to capture a broad range of actors and contexts involved in the primary prevention of radicalization and violent extremism: • • • • •
Types of Outcome Measures
Primary Prevention Outcome Domains and Measures
Types of Location/Situation
Our search strategy does not impose a priori geographical restrictions, as we aim to identify relevant literature globally. However, as detailed in Framework Development and Scope, practical considerations related to scope, resources, language capabilities, and database coverage lead us to anticipate that our search may yield a higher concentration of studies from regions with more extensively documented P/CVE research (often termed the ‘Global North’), due to factors such as publication language and database indexing. Notwithstanding this anticipated distribution, the review will consider eligible studies from any region. To ensure transparency, the EGM will code and report the geographical location of all included studies. This will allow us to present the evidence base’s geographic distribution and highlight any resulting imbalances or gaps.
Types of Settings
Intervention settings will not be restricted. Eligible studies may focus on schools, communities, workplaces, online environments, or other contexts where primary prevention occurs.
Search Methods and Sources
Academic Database Searches
We will conduct systematic searches across academic databases and gray literature sources (detailed below and in Searching Other Sources) to identify studies evaluating primary interventions aimed at preventing cognitive and behavioral radicalization. Our search strategy (see Facets 1 & 2) is designed to locate evaluations of eligible primary prevention interventions as defined by our inclusion criteria (see Dimensions). A sample search strategy for APA PsycINFO is provided in Appendix 1. The full search strategies for all databases will be available in the final report.
Core Academic Databases
The search strategy is structured around two primary facets to ensure both precision and high sensitivity. Facet 1 identifies relevant concepts of radicalization and extremism using a two-part proximity search, linking root terms for extremism with qualifier terms for violence. Facet 2 is a comprehensive set of terms for interventions, composed of three distinct sub-components (2a, 2b, and 2c) designed to capture a wide range of relevant studies. These sub-components are combined using the ‘OR’ operator to form the complete intervention facet. The final search logic combines the two main facets using the ‘AND’ operator: Facet 1 AND (Facet 2a OR Facet 2b OR Facet 2c). This structure ensures that retrieved studies address the core problem of radicalization (Facet 1) while also describing a relevant intervention or programmatic activity (Facet 2).
We do not include a separate facet for general “primary prevention” terms (e.g., “early intervention”) because these terms are often used inconsistently in the literature and would retrieve a large number of irrelevant results. Instead, we rely on our clearly defined inclusion/exclusion criteria and rigorous screening process to identify studies that meet our definition of primary prevention. Across all searched academic databases, at a minimum, we will search the title, abstract, author-supplied keywords, and subject heading/indexing term fields, where available and appropriate for the specific database interface (see Appendix 1).
Facet 1: Radicalization and Violent Extremism Concepts
This facet uses a two-part proximity search to identify documents where concepts related to extremist ideologies, groups, or behaviors are discussed in the context of conflict or violence.
(Part A) Root Terms for Extremism: (“alt right” OR “alt-right” OR antifascis* OR “anti fascis*” OR “anti-fascis*” OR “anti-government” OR “anti-immigra*” OR “conspiracy theor*” OR ecoextremis* OR “eco extremis*” OR “eco-extremis*” OR ecoterror* OR “eco terror*” OR “eco-terror*” OR extremis* OR fascism* OR fascist* OR “far right” OR “far-right” OR “hate crime*” OR “hate group*” OR “identitarian*” OR incel* OR “involuntary celiba*” OR islamism* OR islamist* OR jihadi* OR manosphere* OR militia* OR neonazi* OR “neo nazi*” OR “neo-nazi*” OR “political violence” OR prejudice* OR radicali* OR salafi* OR “single issue” OR “single-issue” OR “sovereign citizen*” OR “white nationalis*” OR “white supremac*” OR xenophobi*) [PROXIMITY OPERATOR, e.g., NEAR/12 or ADJ12]
(Part B) Qualifier Terms for Conflict and Violence: (“conflict*” OR “extreme*” OR “extremis*” OR “radical*” OR “terror*” OR “violen*”)
Facet 2: Intervention and Program Concepts
This facet combines three distinct sets of terms with an “OR” operator to create a comprehensive net for capturing relevant intervention studies.
Facet 2a: Specific Intervention Types and Concepts
This list identifies specific, named intervention modalities and key concepts central to primary prevention programs. (“arts-based” OR “awareness campaign*” OR “awareness raising” OR “build* resilien” OR civics OR “community building” OR “community empowerment” OR “community engagement” OR “community intervention*” OR “community program*” OR “community resilience” OR “community theater*” OR “community theatre*” OR “contact intervention*” OR “coping skills” OR “counter extremis*” OR “counter-extremis*” OR counternarrative* OR “counter narrative*” OR “counter-narrative*” OR “counter radicali*” OR “counter-radicali*” OR “counter violen*” OR “counter-violen*” OR “critical consumption” OR “critical thinking” OR “culture-based” OR curricul* OR “digital literacy” OR education* OR “emotional regulation” OR “emotion regulation” OR “engag* communit*” OR “family communication*” OR “family focus*” OR “family-focus*” OR “family resilience” OR “family support*” OR “health communication*” OR “identity development” OR inclusive OR “increas* engagement*” OR inoculat* OR “interfaith dialogue*” OR “inter-faith dialogue*” OR learn OR learns OR learning OR literacy OR “media campaign*” OR “media intervention*” OR “media literacy” OR mentor* OR “online campaign*” OR “online intervention*” OR “online safety” OR “peer support*” OR “positive online communities” OR “role model*” OR “positive psychology” OR outreach OR “physical* activ*” OR “public awareness” OR “rais* awareness” OR “self-esteem” OR “social cohesion” OR “social inclusion” OR “social marketing” OR “social support*” OR “sport based” OR “sports-based” OR storytelling OR “story telling” OR “story-telling” OR “stress management” OR “teach* tolerance” OR teamwork OR “team work” OR “team-work” OR “think critical*” OR “tolerance teach*” OR “youth development” OR “youth empowerment”)
Facet 2b: General Prevention Domains Combined With Programmatic Terms
This component uses a proximity search to link broad prevention-related domains with general terms for interventions or programs.
Prevention Domains: (“art” OR “arts” OR “artist*” OR “behavio*” OR “cognitive” OR “families” OR “family” OR “music*” OR “narrative” OR “parent*” OR “prevent*” OR “psychosocial” OR “psycho-social” OR “resilience” OR “school*” OR “social skills” OR “sport*”) [PROXIMITY OPERATOR, e.g., NEAR/4 or ADJ4]
Programmatic Terms: (“interven*” OR “initiative*” OR “program*” OR “scheme*” OR “service*” OR “strateg*” OR “therap*” OR “train*” OR “treat*”)
Facet 2c: General Programmatic Terms Combined With Conflict/Violence Context
This final component uses a proximity search to identify general programmatic activities discussed in the context of extremism or violence, acting as a broad safety net.
Programmatic Actions
(“action plan*” OR activit* OR advoca* OR campaign* OR communica* OR deliver* OR educat* OR effort OR efforts OR framework* OR guideline* OR initiative* OR interven* OR measure OR measures OR mechanism* OR messaging OR outreach* OR pilot* OR prevent* OR program* OR project* OR protocol* OR resource* OR respond* OR response* OR scheme* OR seminar* OR service* OR strateg* OR treat* OR toolkit* OR “tool kit*” OR “tool-kit*” OR train* OR workshop*) [PROXIMITY OPERATOR, e.g., NEAR/12 or ADJ12]
Qualifier Terms for Conflict and Violence: (“conflict*” OR “extreme*” OR “extremis*” OR “radical*” OR “terror*” OR “violen*”)
The development of our search strategies involved an iterative refinement process. We tested initial conceptual searches in key databases (e.g., PsycINFO), reviewing titles and abstracts of the first 200 results to assess relevance. Synonyms were added, overly broad terms removed, or Boolean operators (AND, OR, NOT), truncation (*), and proximity operators (NEAR, ADJ) were adjusted as needed to improve the balance between sensitivity (finding all relevant studies) and specificity (excluding irrelevant studies). This iterative process was repeated, testing variations of the search string, until a satisfactory balance was achieved, as judged by the retrieval of a high proportion of known relevant studies and a manageable number of irrelevant studies. This core search logic was then adapted to the specific syntax and controlled vocabulary of each database to produce the search results provided in Appendix 1.
Handsearching
To supplement our electronic database searches and identify recent studies that may not yet be fully indexed, we will conduct targeted handsearching of selected key journals. Based on their high relevance to primary interventions for preventing cognitive and behavioral radicalization, the following journals have been prioritized for handsearching: Terrorism and Political Violence, Studies in Conflict & Terrorism. Behavioral Sciences of Terrorism and Political Aggression. Journal for Deradicalization. Perspectives on Terrorism. Criminology & Public Policy. Journal of Experimental Criminology. Journal of Quantitative Criminology. Critical Studies on Terrorism. International Journal of Conflict and Violence. Campbell Systematic Reviews. Aggression and Violent Behavior. Psychology of Violence.
For each of these selected journals, we will manually scan the table of contents and abstracts of all issues published online within the 24 months preceding the final execution date of our primary database searches. Any articles identified as potentially relevant through this process will be retrieved for full-text screening according to the eligibility criteria outlined in Screening and Study Selection.
Searching Other Resources
The search strategy will extend beyond academic databases to encompass other relevant resources, ensuring a comprehensive and well-rounded review. These supplementary searches aim to identify gray literature, practitioner networks, and other direct outreach opportunities that may need to be captured in traditional academic channels. Each resource will be carefully selected for its relevance to the review’s scope and the quality of the evaluated interventions.
Grey Literature Sources
We will search the following gray literature databases and repositories using search terms and phrases relevant to the primary prevention of cognitive and behavioral radicalization. While the search terms may be adapted to the specific search functionalities of each source (e.g., using simplified terms or keyword searches on websites), the same rigorous inclusion/exclusion criteria used for academic databases will be applied to all gray literature sources. We recognize that gray literature may not consistently be formatted with clear titles and abstracts in the same way as academic publications. Therefore, the screening process will involve careful review of the full text (or relevant sections of documents) to determine eligibility. We will document the search strategies used for each gray literature source, including any adaptations made, and the rationale for these adaptations. This will ensure transparency and replicability, even with the inherent challenges of searching gray literature: OpenGrey National Criminal Justice Reference Service (NCJRS) CORDIS Open Science Framework (OSF)
We will use simplified search terms and phrases appropriate for each source, focusing on key concepts related to the primary prevention of radicalization. We recognize that these primary gray literature sources predominantly cover North American and European outputs. Comprehensive identification and retrieval of gray literature from diverse Global South contexts, often in local languages and less centralized repositories, presents significant logistical and resource challenges and is a practical limitation of this EGM. We will also attempt to capture relevant reports through our searches of international organization websites that operate globally.
Government, Professional, and Practitioner Networks
Government and International Organization Websites.
Government and International Organizations Gray Literature
Think Tanks, Research Centers, and Evidence Portals
Think Tanks, Research Centers, and Evidence Portals Gray Literature
Practitioner Networks
We will actively engage with relevant practitioner networks to identify potentially unpublished evaluations, program descriptions, best practice guides, and other materials that may not be publicly available. This engagement will involve: Identifying Key Networks: We will identify key practitioner networks operating in the fields of countering violent extremism, radicalization prevention, community resilience, and related areas. This will include professional organizations, advocacy groups, and informal networks of practitioners. Contacting Networks: We will contact these networks through email, online forums, conference attendance, and other appropriate channels. Requesting Information: We will clearly explain the purpose of our EGM and request information on any relevant, unpublished materials they may possess or be aware of, particularly evaluations of primary prevention interventions. We will emphasize that we are interested in both formal evaluations and less formal program descriptions or practitioner insights. Data Protection and Ethics: We will ensure that all interactions with practitioner networks adhere to ethical guidelines and data protection regulations. We will only request and use information that can be shared ethically and with appropriate permissions.
Specific networks to be contacted include: CPN-PREV (https://cpnprev.ca/) Eradicate Hate - Prevention Practitioners Network (https://eradicatehatesummit.org/prevention-practitioners-network)
Citation Searching (Backward and Forward)
To further enhance the comprehensiveness of our search and identify additional relevant literature, we will use citation searching, which includes both backward and forward methods. Backward citation searching (also known as reference list harvesting) involves manually reviewing the reference lists of all studies included in the EGM after full-text screening. This process aims to identify potentially relevant older primary studies or systematic reviews that the included studies have cited.
Complementing this, forward citation searching will be conducted using Google Scholar’s “Cited by” feature for all included studies and any highly relevant systematic reviews. This forward approach helps to locate newer studies that have cited these key documents, thereby capturing subsequent research in the field.
All potentially relevant citations identified through either backward or forward citation searching will undergo the same rigorous screening and eligibility assessment outlined in Screening and Study Selection. This dual citation searching strategy is designed to maximize the identification of relevant evidence.
Direct Outreach
Direct outreach will involve contacting relevant organizations, experts, and practitioners directly to gather information on studies, reports, and programs that may not be readily accessible through databases or networks. This will involve contacting governmental bodies, local organizations, and international entities focused on countering radicalization, as well as field experts, for the collection of unpublished data and insights into current and ongoing intervention efforts that may be critical to the review. This includes making targeted efforts to identify and engage with regional organizations or networks in the Global South that are recognized for their work in peacebuilding or P/CVE-related fields, to gather information on relevant studies and programs. We will prioritize studies, reports, and programs that are publicly available or can be obtained through reasonable requests to the producing organizations. This ensures that our findings are, in principle, discoverable and verifiable by other researchers.
Analysis and Presentation
Methods for Mapping and Visualization
To create the EGM, we will utilize EPPI-Reviewer, a systematic review software developed by the EPPI-Centre. This tool allows for the effective organization, coding, and synthesis of data for research projects. Its mapping functions will represent the evidence landscape, highlighting areas of concentration and gaps within the reviewed literature (Eppi-Centre, 2019).
The EGM will structure evidence on primary interventions as a matrix, with interventions as rows and outcomes as columns. Each cell in the matrix will contain a bubble representing the number of studies reporting on that specific intervention-outcome combination. The size of the bubble will be proportional to the number of studies. Larger bubbles will indicate more studies. The color of the bubble will represent the overall risk of bias/quality assessment for those studies, using a traffic light system: green for low risk of bias, yellow for some concerns, and red for high risk of bias. Where studies within a single cell vary in their risk of bias, these will be disaggregated into multiple smaller bubbles, each reflecting the number of studies within a specific risk of bias category, rather than a single, aggregated bubble. This structure is designed to identify areas of strength and gaps in the evidence base for primary interventions.
Rows: Intervention Categories
Interventions will be categorized to reflect the diversity of approaches within primary interventions: A. i. Government agencies (e.g., schools, public health services, municipal programs) ii. Community-based organizations (e.g., local NGOs, religious institutions, youth centers) iii. Multi-agency collaborations (e.g., partnerships between schools and community groups) B. i. Education-Based Initiatives: Programs promoting critical thinking, digital literacy, and inclusive curriculum design. ii. Community Engagement: Efforts to foster dialogue, cultural exchange, and community building. iii. Youth Development: Skills training, recreational programs, and mentorship aimed at empowering young people. iv. Socioeconomic Support: Initiatives addressing vulnerabilities like unemployment, housing insecurity, or social exclusion. v. Media and Digital Interventions: Campaigns promoting positive narratives and countering extremist propaganda online.
Columns: Outcome Categories
Outcomes are categorized to capture the direct and indirect impacts of primary interventions: Resilience and Protective Factors: For example, empathy, critical thinking, social cohesion at the individual/group level, or self-esteem. Attitudinal Outcomes: Reflects shifts in beliefs, tolerance, or attitudes toward diversity, inclusion, and coexistence. Behavioral Outcomes: Includes reductions in participation in extremist activities, engagement with hate groups, or sharing extremist content. Community-Level Outcomes: Measures improvements in social cohesion, trust, and resilience to divisive ideologies. Unintended or Adverse Outcomes: Considering stigmatization, backlash, and increased polarization.
The EGM will incorporate filters for study characteristics, including geographic region, target population (e.g., age, gender), and study quality. The map will enable stakeholders to explore evidence tailored to their needs by providing a flexible and user-friendly interface. The primary output of this EGM is a visual map and descriptive summary of the existing evidence landscape; our interpretation will focus on: Describing the distribution of available evidence across intervention types, populations, settings, and outcome domains. Identifying ‘evidence clusters’ where multiple studies (particularly those with more rigorous designs and lower risk of bias) exist for specific intervention-outcome linkages. Highlighting ‘evidence gaps’ where few or no studies, or only studies with high risk of bias, exist for specific intervention-outcome linkages, target populations, or settings. Discussing broad observable patterns in the evidence base, such as trends in study designs, geographical distribution of research, or common outcome measures used. These interpretations will be presented narratively, supported by descriptive statistics from the map (e.g., counts of studies), to inform future research priorities, policy considerations, and practice.
Data on implementation characteristics (e.g., intervention setting, delivery format, intensity, duration) and contextual factors (e.g., specific target population sub-groups, geographical context) extracted from included studies will be reported in several ways. Key characteristics will be available as filters in the interactive EGM (as detailed in Filters for Presentation), allowing users to explore evidence for specific contexts. These factors will also be summarized in the ‘Table of Included Studies’ and discussed narratively when describing evidence clusters and gaps, particularly to explore potential sources of heterogeneity or to identify under-researched implementation approaches or contexts.
When interpreting areas of sparse evidence or apparent ‘evidence gaps,’ we will acknowledge the potential influence of publication and reporting biases, recognizing that the absence of published evidence for certain intervention-outcome linkages does not necessarily equate to evidence of no effect.
Report Structure
The report will be organized into the following sections: (1) Executive Summary, (2) Background, (3) Objectives, (4) Methods, (5) Results, and (6) Discussion: (1) (2) (3) (4) (5) (6)
Filters for Presentation
The interactive Evidence Gap Map (EGM) will be displayed as a matrix, with rows representing intervention characteristics (e.g., intervention agency, intervention type) and columns representing cognitive and behavioral radicalization outcomes. The interactive EGM will include the following key filters to allow users to explore the evidence based on specific characteristics: ○ Individuals or groups at risk of cognitive and behavioral radicalization ○ Youth in vulnerable communities or at-risk populations ○ Families and caregivers of at-risk individuals Other filters may be included if feasible and if they provide additional meaningful insights, but they will be considered secondary to the key filters listed above.
In maps that include systematic reviews, we will use color coding to differentiate between reviews that have included primary-level studies, including mixed-level studies, and those with no included studies (Snilstveit et al., 2016).
Dependency
The unit of analysis for this review will be the study, rather than the specific publication or document. We will manage dependencies by adopting the methodology described by Sydes et al. (2023). Specifically, in cases of ‘many-to-one’ dependency (where multiple publications report on the same underlying study), we will link all related records to a single study ID. We will then extract and consolidate data from across all available reports to code the study characteristics, outcomes, and risk of bias. Conversely, for ‘one-to-many’ dependency (where a single document reports on multiple distinct studies), we will treat each study as an independent unit of analysis.
Regarding systematic reviews, we will employ a dual approach to avoid double-counting while capturing the full breadth of evidence. First, we will harvest eligible primary studies contained within systematic reviews and map them as individual studies. Second, the systematic reviews themselves will be included as a distinct evidence type, viewable via a ‘Study Type’ toggle filter, but excluded from the primary study counts in the main intervention-outcome matrix.
If a study directly compares two or more eligible interventions (head-to-head), it may appear in multiple columns on the map. We will flag such studies to ensure that potential double counting is clearly identified. Similarly, if a systematic review covers multiple intervention-outcome combinations, it will be flagged to prevent visual inflation of evidence.
Data Collection and Analysis
Screening and Study Selection
Our screening approach will ensure consistent application of eligibility criteria across all data sources. While documents from some sources may already have been partially vetted, all will undergo rigorous screening at each stage, including title, abstract, and full-text levels.
Screening and Retrieval
Records from electronic searches, practitioner networks, and gray literature will be imported into DistillerSR, a systematic review software with integrated Large Language Model (LLM) capabilities. We will document the flow of records through the screening process in a PRISMA flow diagram and provide a table of full-text records excluded at the eligibility stage with reasons. The screening process will proceed along the following steps: (1) Import and deduplication: DistillerSR automatically flags potential duplicate records. Human reviewers will then validate these suggestions, confirming or refuting the duplicate status, and overseeing the removal of confirmed duplicates. (2) AI-Assisted prioritization: DistillerSR’s integrated LLM will analyze the title and abstract of each unique record. Based on this analysis, and informed by the pre-defined inclusion/exclusion criteria and training data (see 4.8.1. B), DistillerSR will prioritize records by their likelihood of meeting inclusion criteria and highlight specific words, phrases, or sentences within the title and abstract that contribute to the relevance assessment. (3) Primary title and abstract screening review and validation: A trained human reviewer (Reviewer 1) will independently assess every (100%) title and abstract, making an initial inclusion, exclusion, or ‘unclear’ decision. While informed by DistillerSR’s LLM prioritization and highlighting, the decision at this stage rests solely with Reviewer 1. This screening will apply to both primary studies and systematic reviews. (a) Title/abstract eligibility: During their independent review, Reviewer 1 will apply the following initial eligibility criteria. If the reviewer suspects that a record may be a duplicate of another record, they will flag it for further review in Step 6. (i) The document pertains to primary interventions addressing cognitive or behavioral radicalization (as defined in Understanding Radicalization). (ii) The document contains indicators of quantitative or qualitative evaluation (e.g., mentions of methods such as surveys, interviews, trials, statistical analysis, outcome measurement, or reported findings/results sections). (iii) Either the title and abstract are available in English, or the document is in a language for which we have access to reliable translation and validation resources (as described in the section on the use of Large Language Models) (b) Title/abstract exclusion: Documents that do not meet these criteria, as determined by Reviewer 1, will be excluded. Records with ambiguous titles or abstracts will be marked as ‘unclear’. (4) Secondary title and abstract review and quality control: Following Reviewer 1’s initial screen, a second trained human reviewer (Reviewer 2) will conduct the following quality control checks to enhance rigor and minimize missed studies: (a) Check on prioritized records: Reviewer 2 will independently screen the records with the highest relevance scores (top 20% as ranked by DistillerSR) to ensure high-priority studies are not missed. This review by Reviewer 2 will occur regardless of Reviewer 1’s initial decision on these prioritized records. (b) Check for false negatives: Reviewer 2 will independently screen a random sample of at least 20% of the records that Reviewer 1 marked for exclusion and which fell outside the check on prioritized records (4a). (c) Review of ‘unclear’ records: All records marked as ‘unclear’ by Reviewer 1 will be automatically passed to Reviewer 2 for a second independent assessment. (5) Full-text retrieval, review, and validation: Records deemed eligible for full-text retrieval following the completion of both primary (Step 3) and secondary (Step 4) title and abstract screening, including resolution of any discrepancies (Step 7), will be flagged for full-text retrieval. This will involve utilizing university library subscriptions, inter-library loan services, internet searches (e.g., Google Scholar), and direct requests to authors if necessary. (a) Human full review and validation: Once the full text is obtained, both Reviewer 1 and Reviewer 2 will independently assess every (100%) full-text document. (b) Full eligibility: Reviewers will apply the following full eligibility criteria. If a reviewer suspects that a record may be a duplicate of another record, they will flag it for further review in Step 7. (i) The document focuses on primary interventions addressing cognitive and behavioral radicalization (as defined in Understanding Radicalization). (ii) The document reports findings based on quantitative outcome data related to primary interventions addressing cognitive and behavioral radicalization, using one of the eligible study designs (as defined in Types of Study Design); or provides qualitative insights into the implementation, context, or participant experiences of primary interventions. For systematic reviews, this means the review must include studies with either eligible quantitative or qualitative data. (iii) The study was conducted between January 2000 and July 2025. (iv) The full text is available in English, or the document is in a language for which we have access to reliable translation and validation resources (as described in the section on the use of Large Language Models) (v) For systematic reviews, reviewers will confirm that the review focuses on primary interventions, includes quantitative data, and meets the other eligibility criteria. If a systematic review consists of a mix of primary, secondary, and tertiary interventions, but also includes primary interventions, it will be included, and the relevant primary-level data will be extracted. (c) Final exclusion: Documents that do not meet all of the full-text criteria will be excluded. Reasons for exclusion will be documented. (6) Review of flagged duplicates: After all screening stages (both title/abstract and full-text), all records flagged as potential duplicates will be manually reviewed and compared to determine if the flagged records are duplicates definitively. Confirmed duplicates will be removed. (7) Discrepancy Resolution: For both the title/abstract and full-text screening stages, any disagreements on inclusion/exclusion arising between Reviewer 1 and Reviewer 2 will be resolved through discussion and consensus. If consensus cannot be reached, a third, senior reviewer will mediate the final decision. Studies will only be included in the EGM if both reviewers agree on inclusion after the consensus process, or if a mediated decision results in inclusion. Reasons for exclusion at the full-text stage will be documented.
Inter-rater Reliability and Training for Screening
All reviewers (Reviewer 1 and Reviewer 2, and any mediators) involved in the screening process will undergo training on the inclusion/exclusion criteria (detailed in steps 3a and 5b of Screening and Study Selection) and the multi-stage screening procedures. The training will incorporate a two-stage pilot screening process. First, reviewers will independently apply the title/abstract criteria to a pilot set of 50 records. Following this, they will independently apply the full-text criteria to a pilot set of 20 full-text articles. These pilot sets will include a mix of records pre-identified as likely relevant, likely irrelevant, and ambiguous, allowing for calibration on difficult cases. After each pilot screening stage, reviewers will meet to discuss their decisions, identify any discrepancies, resolve them by consensus (with mediation if needed), and refine their understanding and application of the criteria. This iterative process will continue until a high level of agreement (target Kappa >0.80 on a calibration set) is achieved, ensuring a consistent understanding and application before full-scale screening begins.
For title/abstract screening, inter-rater reliability will be monitored through the discrepancy resolution process (Step 7 of Screening and Study Selection). All disagreements between Reviewer 1 and Reviewer 2 arising from the secondary title and abstract review, as well as quality control checks (Steps 4a, 4b, and 4c), will be documented and discussed. While formal ongoing Kappa calculations for all components of the secondary title and abstract review might be complex due to their targeted nature, the frequency and nature of disagreements will be tracked as an indicator of consistency. If systematic disagreements emerge, further calibration sessions will be held.
For full-text screening, given that all full-text articles will be independently screened by two reviewers (as per Step 5 of Screening and Study Selection), inter-rater reliability will be formally assessed. Cohen’s Kappa statistic will be calculated on batches of full-text screening decisions (after the first 50 full texts reviewed by both, then periodically). A target Kappa value of at least 0.80 will be sought.
If Kappa falls below 0.80 at any stage requiring formal calculation, or if systematic disagreements persist, further training, clarification of criteria, and discussion will be undertaken to address inconsistencies. The entire review team may refine the inclusion/exclusion criteria themselves if persistent, significant discrepancies indicate ambiguity in the criteria. All discrepancy resolutions will be documented.
Handling Ambiguous Cases and Missing Information During Screening
At the title and abstract stage, as per Step 3b of Screening and Study Selection, if Reviewer 1 finds a title or abstract ambiguous regarding its eligibility, it will be marked as ‘unclear.’ All such records will then undergo a second independent assessment by Reviewer 2 (Step 4c of Screening and Study Selection). Disagreements between Reviewer 1 (initial assessment if not ‘unclear’) and Reviewer 2 regarding these or any other title and abstract records from the secondary review process will be resolved according to the discrepancy resolution process (Step 7 of Screening and Study Selection).
Where full-text documents are unavailable for records deemed potentially eligible after title and abstract screening, but the title and abstract suggest relevance, these records will be classified as “Studies Awaiting Classification.” Efforts to retrieve these full texts will be documented.
If a retrieved full text is available, but information critical for determining final eligibility (as per criteria in Step 5b of Screening and Study Selection) is unclear or missing, the two reviewers (as per Step 5a of Screening and Study Selection) will first attempt to resolve the ambiguity through discussion and re-examination of the study text. If consensus on eligibility cannot be reached or critical information remains missing, we will attempt to contact the corresponding authors of the primary study for clarification (e.g., via email, allowing a reasonable response time). If the information required for an eligibility decision remains unavailable or fatally ambiguous after these efforts, the study will be excluded, and the reason documented. Studies that are ultimately included but have missing information relevant for later data extraction (rather than eligibility) will be handled as described in Data Extraction and Management.
Data Extraction and Management
Once studies are deemed eligible for inclusion in the review, documents will be coded using a combination of Google Gemini and manual coding by trained human reviewers. A standardized coding companion (see Appendix 5) detailing the specific coding categories, providing clear definitions and examples, and outlining specific instructions for Google Gemini, will be developed and used to ensure consistency. The coding framework will capture data across the following domains: • • • •
Data Extraction and Coding
(1) Independent Coding: The data extraction process will involve two parallel, independent coding streams on all eligible records: (a) Human Coding: A trained human coder (Coder 1) will independently review the full text of each included study and code all relevant data according to the standardized coding companion. This coding will be performed without reference to Google Gemini’s output. (b) LLM-Assisted Coding: The full text of each included study will be input into Google Gemini. Using the coding companion as a guide, specific prompts will be crafted for each coding category to instruct Gemini to extract the relevant information. See examples of prompts below. These prompts will be refined during pilot testing and the complete set included as supplementary material in the EGM report: (i) Participants: “Extract the following information about the study participants...” (ii) Intervention: “Identify and describe the intervention, including...” (iii) Outcomes: “Identify all outcome measures related to...” (iv) Research Methodology: “Extract the following information about the study design...” (2) Initial Data Consolidation (Coder 1): After both the independent human coding and the LLM-assisted coding are complete for a given study, Coder 1 will review both sets of coded data alongside the original study text. Coder 1 will: (a) Compare the human-coded data, the LLM-extracted data, and the original study text. (b) Validate the accuracy and completeness of both sets of data. Identify any discrepancies, errors, or omissions in either the human coding or the LLM output. (c) Integrate the information from both coding streams, resolving any discrepancies and creating a single, preliminary consolidated dataset for that study. This dataset represents Coder 1’s best judgment, informed by both their coding and Gemini’s output. (3) Double-Coding: A second trained human coder (Coder 2) will independently code a random sample of at least 20% of the included studies. This double-coding will be blind (i.e., Coder 2 will not see Coder 1’s work, Gemini’s output, or the preliminary consolidated dataset). Coder 2 will follow the same process as described in Step 1a (independent human coding). (4) Final Data Consolidation and Discrepancy Resolution: For the double-coded studies (20% sample): (a) The preliminary consolidated dataset (created by Coder 1 in Step 2) will be compared to the independently coded data from Coder 2. (b) Any discrepancies between Coder 1’s consolidated dataset and Coder 2’s independent coding will be identified. (c) Coder 1 and Coder 2 will discuss these discrepancies and attempt to reach consensus through discussion and re-examination of the original study text. (d) If consensus cannot be reached, a third reviewer will mediate and make the final decision. (e) The final, resolved coding for the double-coded studies will be incorporated into the overall dataset. (5) Data Entry: The final, validated, and consolidated coded data (for all studies) will be entered into a structured database (e.g., a spreadsheet or a dedicated data extraction tool) for analysis.
Inter-Rater Reliability and Training for Extracting and Coding
All reviewers will undergo training on data extraction, the coding companion, and the use of Google Gemini for assisted coding. This training will cover the entire data extraction process, including independent human coding, LLM-assisted coding, validation of LLM output, and the creation of a consolidated dataset. The training will include a detailed explanation of each coding category within the framework, providing clear definitions, examples, and decision rules. Reviewers will also receive specific instruction on how to use Google Gemini to assist with coding, including crafting effective prompts and critically evaluating and validating Gemini’s output. They will be explicitly instructed that initial human coding should be performed independently of Gemini’s output, and that the goal is to create a consolidated dataset leveraging the strengths of both approaches.
Before full-scale data extraction begins, all coders will participate in a pilot coding exercise. They will independently code the same ten eligible studies, following the complete data extraction process described in Data Extraction and Management. This means each coder will perform manual coding, use Google Gemini to assist with coding, validate Gemini’s output, and create a preliminary consolidated dataset for each of the ten pilot studies. The pilot coding exercise will also involve pilot testing and refining the Google Gemini prompts. After each coder has completed both manual and LLM-assisted coding for the pilot studies, the results will be compared. The accuracy and completeness of the LLM output will be assessed by comparing it to the manual coding and the original study text. Any discrepancies or omissions will be used to refine the prompts, making them more specific and targeted. This iterative process of prompt refinement will continue until a satisfactory level of accuracy and completeness is achieved. This will be judged by consistent agreement between the LLM output, the manual coding, and the original study text. If, despite these efforts (training, discussion, mediation), significant discrepancies persist (e.g., kappa consistently below 0.6), the coding companion, the LLM prompts, or both, will be revisited and refined by the entire review team. This may involve clarifying definitions, providing more detailed examples, or adjusting the prompts to improve their accuracy. The pilot coding process may be repeated with the revised materials if necessary.
To ensure ongoing inter-rater reliability throughout the data extraction process, the coding team will hold regular meetings to discuss any challenging cases, ambiguities, or emerging issues. Inter-rater agreement (kappa) for these double-coded studies (comparing the second coder’s independent coding to the final consolidated data from the first coder) will be calculated after the pilot coding and periodically throughout the data extraction process. A target kappa value of at least 0.8, indicating substantial agreement, will be sought. If the kappa value for data extraction falls below 0.8, the following steps will be taken: (1) Coders will meet to discuss discrepancies and identify the source of disagreement; (2) The coding companion and data extraction procedures will be reviewed and clarified if necessary; (3) Additional training will be provided to the coders; (4) A new set of pilot records will be coded; (5) Kappa will be recalculated. If kappa remains below 0.8 after these steps, the data extraction process for problematic items or studies will be repeated by a new pair of coders.
Any disagreements between reviewers at any stage of the screening process will be resolved through discussion and consensus. If consensus cannot be reached, a third reviewer, who is a senior member of the research team with expertise in systematic review methodology, will be consulted to make the final decision.
Tools for Assessing Risk of Bias/Study Quality
The quality and risk of bias of included studies will be assessed using different tools, depending on the study type. We will independently assess the methodological quality of all included systematic reviews using the AMSTAR 2 tool (Shea et al., 2017). AMSTAR 2 is a validated instrument specifically designed to evaluate the rigor and transparency of the systematic review process. It assesses aspects such as the clarity of the research question, the comprehensiveness of the search strategy, the appropriateness of the study selection and data extraction methods, and the risk of bias assessment within the review.
For primary quantitative studies, we will use the Cochrane Risk of Bias tool 2.0 (RoB 2) (Sterne et al., 2019) to assess the risk of bias in randomized controlled trials (RCTs). RoB 2 evaluates bias arising from the randomization process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of the reported result.
For quasi-experimental and other non-randomized quantitative studies, we will use the Risk Of Bias In Non-randomized Studies—of Interventions (ROBINS-I) tool (Sterne et al., 2016). ROBINS-I assesses bias due to confounding, participant selection, intervention classification, deviations from intended interventions, missing data, outcome measurement, and the selection of the reported result.
For qualitative studies, we will use the appropriate Joanna Briggs Institute (JBI) critical appraisal tool for qualitative research to assess methodological quality.
For all study types, two trained coders will independently assess the risk of bias and quality. Any disagreements between coders will be resolved through discussion or consultation with a third reviewer. The results of the risk of bias/quality assessments will be incorporated into the EGM visualization. Studies will be categorized as ‘low risk of bias,’ ‘some concerns,’ or ‘high risk of bias’ (or equivalent categories based on the specific tool used). This information will be represented visually, for example, by using different colors or shapes for the bubbles representing each study on the map. This will allow users to quickly assess the overall quality of the evidence for each intervention-outcome combination. In addition to the visual representation, the results of the risk of bias/quality assessments will be summarized narratively in the final report. We will discuss the overall quality of the evidence base, highlighting any common methodological limitations or strengths across the included studies. This will inform our interpretation of the findings and our recommendations for future research.
Use of Large Language Models (LLMs)
Large Language Models (LLMs) offer potential advantages for streamlining specific tasks within systematic reviews and Evidence Gap Maps (EGMs). In this EGM, LLMs will be utilized for translation, initial screening, and coding while maintaining rigorous human oversight for all critical appraisal and decision-making, in line with the recommendations for responsible AI use in evidence synthesis outlined by Thomas et al. (2024). We recognize the nascent nature of LLM applications in systematic reviews and EGMs and acknowledge the potential for biases and limitations, as highlighted by (Lieberum et al., 2024). Throughout this EGM, LLMs are used as assistive tools to augment, not replace, rigorous human review and judgment. Trained human reviewers will make all final decisions regarding study inclusion, data extraction, and coding.
DeepL Translator will be used to translate potentially relevant non-English language materials (titles, abstracts, and if necessary, full texts) into English (Junczys-Dowmunt et al., 2018). We will then make every effort to obtain expert validation of the translated full texts. This will involve, first, requesting an existing English version from the study authors; and second, if an English version is not available, requesting that the authors validate the accuracy of the DeepL translation. If author validation is not possible, we will seek collaboration with bilingual researchers with relevant expertise. Studies for which we cannot obtain reliable validation will be excluded. We recognize that this process, while aiming for inclusivity, is resource-intensive and the feasibility of robustly translating and validating a potentially large volume of diverse-language literature represents a practical limitation on the EGM’s comprehensive global coverage.
DistillerSR, an LLM-powered screening tool, will help prioritize articles based on their relevance to the inclusion/exclusion criteria, increasing the efficiency of human reviewers. As detailed in Screening and Retrieval (Step 2), DistillerSR will be used to prioritize records for title/abstract screening by assigning a relevance score and highlighting key passages. However, human reviewers will independently assess every title and abstract, regardless of the LLM’s assessment.
Google Gemini, an advanced LLM, will assist with coding included studies, specifically for extracting data related to intervention characteristics (e.g., type, target population, setting), outcomes, and study design. This approach is chosen to enhance efficiency and to explore the potential for LLMs to identify relevant information that human coders might miss. However, the core methodology remains based on independent human coding. As detailed in Data Extraction and Management, all included studies will be independently coded by a human reviewer without reference to the LLM’s output. Subsequently, the LLM-assisted coding and the independent human coding will be compared and integrated to create a final, consolidated dataset. This approach leverages the strengths of both human and AI coding.
We acknowledge the potential for LLMs to introduce bias or misinterpret nuanced information (Lieberum et al., 2024). While the use of LLMs for data extraction is an emerging area, advanced models like Google Gemini can achieve high accuracy rates, particularly when combined with human oversight and validation. Our methodology, with its emphasis on independent human coding, 100% human review of LLM output, double-coding for inter-rater reliability, and rigorous pilot testing, is designed to maximize the benefits of LLMs while mitigating potential risks, even when applied to the diverse range of study designs (including quasi-experimental studies) expected in this field. We will conduct further analysis to compare the accuracy of manual coding and LLM-assisted coding, with reference to the different outcomes coded across the study designs.
Our use of LLMs is not intended to replace traditional human-led review methods. Rather, it is a strategic approach to enhance the efficiency, transparency, and exploratory potential of the review, while maintaining the highest standards of methodological rigor and human oversight. The core methodology – independent human screening, independent human coding, double-coding for reliability, and rigorous quality checks – remains fundamentally the same as in a traditional review. The LLMs are tools to assist with these processes, not to automate them entirely.
Supplemental Material
Supplemental Material - Primary Interventions for Preventing Cognitive and Behavioral Radicalization: An Evidence and Gap Map
Supplemental Material for Primary Interventions for Preventing Cognitive and Behavioral Radicalization: An Evidence and Gap Map by Kieren Aris, Meriem Rebbani and Joshua Freilich in Campbell Systematic Reviews
Footnotes
Acknowledgments
The DHS Science and Technology Directorate supported this review. We thank our advisory group and reviewers for their considered feedback.
Author Contributions
• Content: Kieren Aris, Meriem Rebbani
• EGM methods: Kieren Aris, Joshua Freilich
• Statistical analysis: Kieren Aris
• Information retrieval: Kieren Aris, Douglas Salzwedel
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: External Sources: DHS Science and Technology Directorate (DHS02-Y203). Internal sources: No sources of support were provided.
Declaration of Conflicting Interests
The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The authors have prior experience conducting research and working in the field of P/CVE. Kieren Aris has worked on developing, implementing, and evaluating primary prevention interventions for various governmental and non-governmental organizations. Joshua D. Freilich and Meriem Rebbani have researched and published on P/CVE-related topics. This experience informs the design and implementation of this Evidence and Gap Map. However, none of the authors has any direct financial interests or personal relationships that would represent a conflict of interest in conducting or reporting this research.
Plans for Updating the EGM
Kieren Aris will be responsible for updating this review, which is anticipated to occur every three years.
Supplemental Material
Supplemental material for this article is available online.