AIMING FOR SUCCESS

Abstract

Despite the popularity of the Evidence-Based Policy Making paradigm, scholarly evidence often fails to have an impact in emotional or value-laden policy debates. Consequently, changes to Canada’s gun control laws in recent years have often failed to incorporate scholarly research. This is problematic given that the forces of path dependence impose costs on policy makers who seek to reverse established policies, even if they are dysfunctional. This article lays the theoretical foundations for a Firearms Policy Evaluation Framework, which can be used by scholars, policy makers, advocates, and the public to conduct preliminary evaluations of proposed firearms policies before they become law. The utility of the framework is then demonstrated with an evaluation of the 2020 assault-style weapons ban in Canada, which includes a systematic scoping review of the literature on the impact of assault-weapons bans.

Judging from the rhetoric emerging from Canada's heated gun debate, there seems to be a bipartisan appetite for evidence-based policy making. The website of the Canadian Doctors for Protection from Guns, a pro-control group, calls for evidence-based public health solutions to gun violence. ¹ Similarly, the Canadian Coalition for Firearm Rights (CCFR) often decry the politicization of the gun debate and call for solutions based on evidence. ² Despite both sides expressing their desire for evidence-based policy solutions to the problem of gun violence, each group still holds different views on exactly what that entails. This phenomenon is not unique to debates over gun control. Key policy actors often mobilize the rhetorical power of the Evidence-Based Policy Making (EBPM) paradigm within their policy narratives when engaging in debates on topics ranging from the obesity debate (Boswell 2014; Botterill and Hindmoor 2012) to criminal justice (Freiberg and Carson 2010).

Despite the lip service paid to the value of evidence, evidence-based approaches seem to struggle to influence policy, especially when applied to emotional and value-laden issues. Further, policy makers, advocates, stakeholders, and the public often lack the resources needed to make proper use of evidence. Starting from the premise that evidence-based policy making will never be perfect, but that scholars can contribute knowledge to important policy areas and work toward a more democratic and participatory discourse on complex issues, this article begins to lay out an evaluation tool for assessing proposed firearms policies: The Firearms Policy Evaluation Framework (FPEF).

It is difficult to evaluate the success of gun control policies in a complex social world. Oftentimes, evaluations of policies take decades, by which time it is hard to reverse policies that have been proven ineffective or costly. The use of analytical tools like the FPEF can therefore help interested parties to conduct preliminary evaluations of proposed policies before they become law or are implemented. This will not solve contentious policy debates. However, such analytical tools can provide policy scholars and advocates with a tool to clarify key assumptions behind policies, put these assumptions into conversation with the scholarly literature, and work toward a more deliberative democratic discourse. They can also provide non-experts with a tool to better evaluate the claims made by their elected representatives and speak truth to power.

I first outline the literature on EBPM and explain the main principles of the FPEF. I then demonstrate the utility of the framework by analyzing the Canadian government's 2020 ban on assault-style weapons, drawing on court and government documents, speeches, press releases, and a systematic scoping review of the scholarly literature. While the chosen case study for this article is Canadian, the FPEF is designed to be as parsimonious as possible, allowing it to be applied to other jurisdictions, including the acrimonious American gun debate.

Evidence-Based Policy Making (EBPM) and Policy Evaluation

Since the 1990s, the concept of evidence-based policy making has enjoyed renewed support, both in policy and scholarly communities and among members of the public (Freiberg and Carson 2010; Head 2010; Nutley, Davies, and Walter 2003). In Canada, the concept was adopted as a mantra by the Liberal Party in the 2015 election, which sought to frame the incumbent Conservative Party as anti-science. Evidence-based policy is a popular buzzword, given “the attractive underlying notion that important decisions should be made objectively, using the best available information, rather than being unduly influenced by ideology, prejudice, or whims” (G. W. Richards 2017, 165). EBPM is also a guiding assumption of the evaluation literature, as evaluators seek to influence policy and programs using the best available evidence (Weiss et al. 2008; Pawson, Wong, and Owen 2011).

EBPM has great promise and can be a useful tool for holding powerful policy makers to account when they place interests and ideology ahead of the evidence, but the movement does have limitations. Scholars studying the EBPM movement have focused their critiques on two principal areas: the practical problems involved in implementing EBPM, and the philosophical foundations of the paradigm.

The ideals of the EBPM paradigm have faced a number of practical issues when implemented, like the differences between policy makers and researchers with regard to what constitutes evidence (Cairney and Oliver 2016; Head 2010; Weiss et al. 2008), problematic hierarchies of evidence (Head 2008; Nutley, Davies, and Walter 2003), the instrumental use of evidence by self-interested and boundedly rational policy makers (G.W. Richards 2017; Weiss et al. 2008), institutional limitations to the implementation of EBPM, the inherent ambiguity of evidence (Pawson, Wong, and Owen 2011), and the lack of resources that policy makers have to analyze available evidence (Howlett 2009, 154).

Other critiques leveled against the paradigm have been philosophical. Scholars have suggested that EBPM relies on outdated assumptions regarding human rationality. This critique does have some merit. Policy scholars since Simon (1950) have recognized the limits of comprehensive rationality, and instead operated with the understanding that policy makers act within the limits of bounded rationality. Critics have questioned the basic premise of EBPM, that “flaws in the policy process can be overcome through accessing and utilising the best available evidence” (Botterill and Hindmoor 2012, 368).

Portraying the policy process in this way is problematic, given the influence that factors other than expert-generated evidence, like interests, values, and emotion, have on both evidence-gathering and the policy process. Empirical research often contains hidden values and assumptions built into models. Further, values and emotion impact every step in the policy process, from agenda setting to identifying and collecting evidence, and then putting that evidence to use. In debates over highly emotional issues, evidence often plays a secondary role in policy makers’ decision making and the formation of public opinion (Freiberg and Carson 2010).

This is especially true for issues that involve risk-management, given that our perception of risk can be heavily influenced by our culture, values, and emotions (Bédard 2017). When people disapprove of an activity they are more likely to overestimate the danger of that activity, and vice versa (Freiberg and Carson 2010; Kahan and Braman 2003), leading to sometimes disproportionate policy responses. Further, the more people fear something happening, the more likely they are to think that it is likely to happen, despite evidence to the contrary. This helps to explain why statistically rare, yet emotionally disturbing events, like terror attacks, plane crashes, or random mass killings capture the public imagination and impact policy (Freiberg and Carson 2010). Simply put, evidence is less likely to have an impact in polarized debates with large value-gaps between competing positions, especially when public safety is in play (Head 2010, 80–81).

Having acknowledged the problems with the EBPM paradigm, we are left with the question of how the concept can be salvaged. Rather than succumb to relativism, or pursue an unrealistic standard, scholars have instead found a middle ground where EBPM can help to serve as a counterbalance against political considerations and interests and can thus promote the public good (R. Richards 2017). We should see the use of evidence as part of a deliberative democratic process. Evidence may not always meet the gold standard of clinical studies (Howlett 2009), and will not directly lead us to a single right answer but we can develop better evidence that will lead us to better answers (Boswell 2014).

Given the fact that evidence is less likely to have an impact on emotional and value-laden debates, and that policy makers often lack the analytical abilities or motivation to properly assess the available evidence, there is an impetus for policy scholars to wade into these messy policy areas and develop theoretical tools to help politicians, advocates, public servants, and members of the public navigate the available evidence. It will not always be able to reconcile conflicting evidence or the competing values involved in these debates, but by making their assumptions explicit, can help the policy makers and the public make more informed choices, a key goal of evaluation research (King and Alkin 2019).

Firearms policy is once such debate. Often dismissed as a uniquely American problem, the gun debate is happening around the world, from the Czech Republic (Zeldin 2017) to Brazil (John 2019), to Israel (Liebermann 2018), Switzerland (Bachmann 2019), and most recently again in the United States following the May 24, 2022 elementary school massacre in Uvalde, Texas. In Canada, there has been an active policy debate over gun control since the 1970s, which was reinvigorated by the election of the Liberal party in 2015. The government has pursued several changes to Canada's already robust gun laws. Despite this, there is a serious absence of academic discussion on firearms policy in Canada (Ferguson and Koziarski 2019). Firearm policies enacted by the government directly impact Canada's 2.2 million licensed firearms owners, and policy failures can have indirect consequences on the wider Canadian public through increases in gun violence.

Firearms policy, sometimes referred to as firearms-control policy, gun control, or gun safety, is a polarized, emotional, and value-laden issue area, with significant disagreement between stakeholders and major expert groups studying the problem, like political scientists, criminologists, economists, and public health researchers (Berg, Lott, and Mauser 2019). The heated nature of the gun debate means that gun control is not simply a rational or technical debate about which policies are most likely to reduce crime. Rather, firearms policies can serve other purposes for policy makers. A significant literature from the United States demonstrates that support for gun control can often be a way for political elites to demonstrate antagonism toward groups in society that they do not like; for example, rural people (Kleck 2001). That is, like other issues subsumed into the “culture wars,” support for gun control policies may be a way for certain groups to enact their values and express their disapproval of others in society (Blanco 2016; Campbell 2019; Kleck 1996; Kleck, Gertz, and Bratton 2009).

Further, debates over gun control are often used by political parties to play wedge politics in Canada. For example, Dumouchel, Ouellet, and Giasson (2021) found that the Conservative Party of Canada under Stephen Harper effectively used the issue of the long-gun registry to win key ridings in Canada in 2011. The general challenges of applying EBPM, especially in policy areas so laden with emotion, values, and division, are particularly evident in firearms policy. This provides a special impetus for scholars to work toward the development and refinement of an evaluation framework for the assessment of firearm policies that can put the available evidence in conversation with claims made by self-interested policy makers.

Evaluating Proposed Firearms Policies

With the exception of the United States, with its unique constitutional amendment, once put in place, firearm policies are rarely reversed. This is caused by path dependence (Fioretos, Falleti, and Sheingate 2018; Hall and Taylor 1996), where movements in one policy direction accumulate momentum and make a major reversal of policy difficult. This is especially the case with firearms policy in Canada, given the reluctance of politicians to be seen as undermining public safety by rolling back policies. As a result, even dysfunctional, costly, and ineffective policies are rarely reversed.

The sole case in recent Canadian history of a firearms policy being reversed was the 1993 long-gun registry, which was dismantled a decade later due to multi-billion dollar cost overruns and a failure to link the registry to any reduction in gun crime (CBC News 2004; Stenning 2003; Parker et al. 2017). Given that firearms policies are rarely reversed following their introduction, this creates an important impetus for scholars of evaluation to give policy makers, civil servants, advocacy groups, and the public tools to assess the possible implications of a proposed policy. This framework should take into consideration several factors, which I propose here (Figure 1).

Figure 1.

Firearms policy evaluation framework (FPEF).

Problem Representation and Policy Goals

The first area where a proposed firearms policy can be evaluated is in its problem representation or problem definition. A key contribution of critical approaches to evaluation have been to shed light on the how problem definition can be heavily influenced by values, interests, and power. It is important for evaluators to be aware of how policy makers represent problems when proposing policies (Archibald 2020). For policies to be effective they must be tackling real problems, rather than perceived problems, in order to avoid Type III error, or solving the wrong problem (Mitroff and Featheringham 1974). It is important to ask tough questions about the representation of a given problem. This includes examining competing or alternative representations of the problem, unpacking the logical assumptions that support this representation, and evaluating the competing evidence that supports the differing views of the problem (Archibald 2020).

After determining the validity of the problem representation, it is also important to evaluate the aim of a policy. Generally, firearms policies aim to achieve one of two goals: prevention or enforcement. The former are laws that use gun control to prevent future homicides, accidents, and suicides by limiting guns to a segment of desirable people while keeping them out of the hands of undesirable people. In some polities, the list of desirable people includes only police officers and the military. In others, citizens able to demonstrate competency and good character can be licensed to own them. Prevention laws also mandate things like mandatory firearms education, ensure that guns are stored safely to limit unauthorized access, and set parameters on which firearms and accessories are too dangerous for the public to own. The second type of firearm policies are those that govern enforcement. These laws make it easier for law enforcement to solve crimes committed with firearms, or to enforce firearm prohibitions. Examples of these laws include things like serial number requirements, red flag laws, or firearms registration.

Logic, Evidence, and Harm

Having assessed the validity of a proposed policy's problem representation and aim, there are a series of tests that can then be applied to evaluate the “causal theory” of the policy (Sabatier 1986). The first is the application of logic models. Though it may seem self-evident that the internal logic of a policy should be evaluated, policy makers face a variety of pressures that can lead to logical inconsistencies in policy. The evaluation literature makes extensive use of logic models for evaluation policies and programs (Jones et al. 2020; Kaplan and Garrett 2005; McLaughlin and Jordan 1999; Renger 2006; Renger and Titcomb 2002). Logic models are useful for clarifying the underlying implicit assumptions that go into a policy or program (McLaughlin and Jordan 1999).

I propose applying the ATM model (Renger 2006; Renger and Titcomb 2002), which evaluates policies by examining the antecedent conditions, or the why behind the policy, how the policy targets these conditions, and how these changes can be measured. The ATM model is useful because of its focus on making explicit the “underlying rationale” of a policy by representing them visually, in a way that is easy to unpack and grasp (Renger and Titcomb 2002). Oftentimes, simply making explicit the logical assumptions built into a policy can help expose logical flaws.

Logic models are generally used to assess programs or policies that are already in place. Evaluating proposed policies would involve zeroing in on the connection between the first two parts of this model, the aim of the policy and the tools it proposes to achieve this aim, given that it is not possible to measure the success of a policy before it is implemented. Researchers can use draft texts of the proposed policy, speeches, or news releases from key policy makers, and expert evidence to assess the antecedent conditions and targets of the policy. Further, representing the logical assumptions visually is a useful tool for making the evaluation more accessible to the general public (Jones et al. 2020).

Given that policies can often be motivated by political interests, there is value in putting these policies into conversation with the available scholarly evidence. In a perfect world, policies would be justified based on available evidence from other jurisdictions. First, the logical assumptions articulated during the logic test should have empirical support. Further, there should be evidence that a similar policy has worked elsewhere. This could be provided using a scoping review of the available literature. Most firearms policies used globally today, like licensing, registration, or background checks, have been around for decades. If a policy is effective, policy makers should be able to provide evidence that demonstrates that a policy has had a positive impact in another jurisdiction.

Finally, the potential harm that a policy may cause should be weighed against the potential benefits it will provide. While potential benefits can be estimated using evidence from other jurisdictions, potential harms are more difficult to elucidate and quantify. Still, it is important to consider possible harms that could result from a proposed firearm policy and weigh them against evidence of likely benefits. Harms can be measured both by looking at quantitative data on costs, as well as using qualitative data like claims made by key stakeholders. While these stakeholder claims do not necessarily represent “cold hard facts,” clarifying claims of harm allows us to place them into discussion with the policy rationales proposed by policy makers.

The harms considered could be to the public, to the government itself, or to key stakeholders. Gun control policies that relax controls too much can cause harm to the public by allowing firearms to be easily acquired by those with malicious intentions. Further, inefficient policies like the long-gun registry do harm to the government's pocketbook through cost overruns, and to the legitimacy of future gun control efforts.

What is often forgotten or minimized in the scholarly conversation on guns is the harm that policies can do to key stakeholders, like sports shooters, hunters, and firearms collectors. Ultimately, gun control involves the exchange of liberty for security. Individuals sacrifice the liberty of unlimited access to lethal technology, in exchange for the security that gun control provides. Yet this exchange of security is not evenly distributed across society. Non-gun owners give up little when new gun control measures are put in and thus may see little reason to restrain the use of these policies. This is what Luban (2005) dubs the Mel Brooks Fallacy, based on a famous quote by the acclaimed Jewish-American director: “tragedy is when I break a fingernail, and comedy is when you fall down a manhole and die.” This logical fallacy occurs because “the supposed ‘trade-offs’ between security and rights is too easy as long as it's a trade-off of your rights for my security” (Luban 2005, 243). It is easy for those who do not own guns to support further gun control, as it has no impact on their lives. But for gun owners, the potential costs can be significant.

This is especially true for people living in remote areas, like some of Canada's indigenous peoples, who use firearms to manage the danger posed by wild predators and to procure food. It is certainly true for other Canadians as well, for whom gun ownership can be an important part of their culture or a serious leisure activity (Littlefield and Ozanne 2011; Murray et al. 2016; Olmsted 1990; Schwartz 2021). These harms can include the financial costs of lost property, the personal and emotional losses of losing a treasured possession or one's serious leisure pursuit, and the potential legal liability of running afoul, intentionally, or accidentally, of this often-complex legal area.

Having outlined some of the basic considerations that should be addressed by the FPEF, I will now unpack the example of the Liberal ban on “assault-style” firearms to demonstrate how these considerations can be applied to an empirical case study.

SOR/98-462 and Bill C-21 – Ban on “Assault-Style” Firearms

Firearms in Canada are much more tightly controlled than in the United States. Handguns have been regulated in Canada since the 1890s (Brown 2012). Long-gun regulation in Canada came about the late 1970s, when Bill C-51 created a licensing regime, mandated background checks, and prohibited ownership of fully automatic weapons. These laws were further tightened in 1991 with the creation of Bill C-17, which introduced a requirement for license applicants to undergo a mandatory firearms safety course, provide a sign-off from their spouse or intimate partner on their application. The bill also introduced a 28-day minimum waiting period for license processing, ³ created safe storage requirements for firearms, and limited magazine capacity for semi-automatic rifles to 5-rounds (Brown 2012; Fleming, McLean, and Tatalovich 2018; Mauser and Buckner 1997).

The Liberal government under Prime Minister Trudeau, who took power in 2015, has put forward several changes to the country's gun control laws. The Liberal government first announced its intention to pursue a ban on “military-style assault rifles” in the fall of 2019, with the Prime Minister Justin Trudeau making a speech and posting on his social media account: “Thoughts and prayers are not enough. We will ban all military-style assault rifles, give municipalities the ability to restrict or ban handguns, and strengthen gun control.” ⁴

Following the announcement, the government launched a public consultation which included in-person meetings, written submissions, and an online survey, which was completed by over 133,000 Canadians. When asked if more should be done to limit access to assault weapons, only 21 percent of respondents said yes, while 77 percent said no (“Engagement Summary Report - Reducing Violent Crime: A Dialogue on Handguns and Assault-Style Firearms” 2019).

Despite the lukewarm results of the public consultation, the government used the agenda window provided the mass shooting in Nova Scotia in the spring of 2020 to move ahead with the policy. Rather than affecting the ban through legislation the government proceeded to enact it by Order in Council, a legal instrument that allows the executive to make changes to laws without going through standard parliamentary scrutiny. Almost instantly, 1,500 models of firearms were banned in Canada, including the infamous AR-15 (Tasker 2020) (Figure 2).

Figure 2.

Evaluating the May 2020 assault-style weapons ban.

Problem Representation and Policy Aim

The government identified two principal problems that this policy was intended to solve: gun-violence and violence against women (Maclean’s 2020). The problem definition built into this policy can thus be identified as: “legally owned assault-style weapons are being used to kill Canadians, especially Canadian women.” Alternative representations of the problem advanced by stakeholders can be articulated as “Firearm homicides, including mass shootings, are most often carried out with illegally acquired firearms, usually sourced from outside of Canada. Assault-style firearms are an arbitrary categorization with little practical meaning.”

First, it is important to assess the extent to which mass shootings are a significant problem in Canada. While these events are tragic, command significant media attention, and cause public anxiety, understanding gun violence through the prism of mass shootings obscures more than it clarifies. Between 1998 and 2015, the death rate from mass shootings in Canada per 100,000 people was 0.012, compared to 0.133 in the United States. Canada's overall gun death rate is eight times lower than that of the United States. Firearms-related crime in Canada has been trending downward since the 1990s, and the small rise since 2013 has largely been driven by gang-related crime in Canada's big cities (Beattie, David, and Roy 2018; “Firearm-Related Violent Crime in Canada” 2018).

The central problem with the government's problem representation was in the definition of an assault-style firearm. The government did not explain why it selected the firearms that were added to the ban, or why it considered them to be “military-grade” or “assault-style,” suggesting that these terms may be more framing technique than technical designation. The government seems to have wrangled with this definition for some time. When the policy was originally proposed it was announced as a ban on “military-style assault weapons.” Following their public consultation when the issue of the unclear definition was raised, the government then adopted the phrase “assault-style firearm.”

The ban continued the longstanding Canadian policy of prohibiting firearms by make and model, rather than by objective characteristics. Rather than ban semi-automatic firearms entirely, the government selected models of firearms that it considered to be more dangerous, without providing any criteria for how these firearms were selected. Meanwhile, other commonly owned semi-automatic rifles, like the Russian SKS or Israeli Tavor, were not banned. No publicly available framework for how these decisions were made was released.

While in theory having a list of banned firearms would make it easy for individuals to understand if the firearm that they owned was now prohibited, two factors complicate this. First, the use of the term “variant” in the legislation leads to confusion on which firearms are considered “variants” of others. For example, the bill bans variants of the AR-15 but does not propose a strict definition of what that means; that is what conditions need to be satisfied for a firearm to be considered a variant of an AR-15. The AR-15 was invented in the late 1950s, and its patent has long expired. Firearm engineers have copied many of the popular aspects of the firearm platform's design, leading to massive confusion over what is and is not an AR-15 variant.

Second, police and the industry generally rely on the RCMP Firearms Reference Table (FRT) to determine whether or not a firearm is legal, but the FRT is not legally binding, and in some cases, an individual may not end up knowing if the firearm they own is still legal or not until they are standing in front of a judge (Canadian Coalition for Firearm Rights et al. v. Attorney General. Cross-Examination of Murray Smith 2020; Notice of Motion: Canadian Coalition for Firearm Rights et al. v. Attorney General of Canada 2020).

This example illustrates the importance of making explicit a government's problem definition when analyzing a policy. Often, looking at the problem representation can expose immediate flaws in the policy, in this case the inconsistent way the government has defined the object they are seeking to ban. If defining assault-style weapons is so complicated, it raises questions regarding why the government considers these firearms to be more dangerous than others, especially other semi-automatic firearms. The aim of the policy, according to the government, was preventing or reducing public mass shootings and violence against women by banning these assault-style weapons. Using the tests of logic, evidence, and harm, we can evaluate whether the policy is likely to succeed in achieving this.

Logic, Evidence, and Harm

At first glance, there is a logical connection between the problems the government was trying to solve, and the tools proposed to solve that problem. Putting this into conversation with the available evidence, however, raises some concerns.

The argument that banning assault-style weapons will reduce mass public shootings is premised on the idea that legally owned assault-style weapons are (a) the “weapon of choice” for mass shooters, and (b) that they are intrinsically more dangerous than other firearms. In his speech announcing the ban, the Prime Minister cited a list of mass shootings that had occurred in Canada since the 1990s to support the first assertion. This was rhetorically powerful but closer examination of the nine examples the Prime Minister referenced demonstrates that these crimes were carried out with handguns as well as long guns, and usually involve firearms or accessories that are sourced illegally from the United States. Only two of the nine shootings listed involved a licensed gun owner using only a legally purchased and unmodified assault-style weapon. ⁵ For example, the Nova Scotia shooting, which the government used as an immediate justification for the ban, was largely conducted using firearms smuggled into from the United States, and the estate of a close friend, as the shooter did not have a firearms license and could not legally purchase them in Canada (Russell 2020).

This argument also raises the issue of whether banning these firearms would lead to a reduction in mass shootings, or simply push would-be mass killers to use firearms like handguns or rifles not impacted by the ban, or source their assault-style firearms from the United States. The record of past mass shootings mentioned above casts aspersions on the likelihood that the ban would have a major effect on statistically rare mass shootings.

The argument that assault-style weapons are more dangerous than other firearms rests on two premises. First, that these firearms can fire more rounds in a shorter amount of time because of their semi-automatic action, where a single depression of the trigger fires a single round, and second, because they accept large-capacity magazines. The problem with this logic is that large-capacity magazines were banned in Canada in the 1990s with Bill C-17. Any semi-automatic firearm in Canada is limited to using magazines that hold five cartridges or less. A potential mass shooter could not simply walk to a gun store and purchase a large-capacity magazine. They would need to modify or acquire one illegally. Further, the list of firearms the government chose to classify does not include all semi-automatic firearms capable of accepting detachable magazines. Therefore, the ban is either incomplete, or redundant.

With regard to empirical support for the policy, it is somewhat telling that the government did not produce or refer to a single piece of research in the announcement of the ban. Instead, the government's defence of the ban largely consisted of listing past mass shootings and repeating that the ban is popular with the public. When pressed to provide the evidence used to justify the policy by a Federal Judge as part of a lawsuit against the government, the defence invoked Section 39 of the Canada Evidence Act, which allows them to block access to the documents used to justify a cabinet decision (Naumetz 2021). The empirical support for this policy is further limited by the fact that Statistics Canada does not keep track of the percentage of firearms used in homicides that qualify as “assault-style,” given the inconsistent way the term has been defined.

How have assault-weapons bans fared in other jurisdictions? To assess this, I employed a systematic scoping review as laid out by Arksey and O’Malley (2005). This method was used by Ferguson and Koziarski (2019) to conduct a systematic review of the literature on firearms policy in Canada. This method involves identifying a research question, identifying relevant studies and selecting studies based on objective criteria, organizing the data, and summarizing the results (Arksey and O’Malley 2005). The research question I employed was: What is known from the existing literature about the effectiveness of assault-weapons bans? To search for articles, I used my university library's search database, the ProQuest Database, as well as Google Scholar. I conducted key word searches including: “assault weapon,” “assault rifle,” and “assault weapon ban.” Relevant articles were also selected from the bibliographies of articles that I drew from.

Articles were downloaded and coded by field (economics, criminology, public health, political science), methodology, and case-study jurisdiction. In order to be selected for inclusions, the article needed to be (1) peer-reviewed and from a scholarly source, (2) analyze empirical data directly or review past studies, (3) be published between January 1, 2000 and November 1, 2021. Overall, 12 articles were identified that fit these inclusion criteria, across the fields of economics, criminology, and public health. All of these articles focused on the United States. Some looked at state-level assault weapons bans, while others examined the 1994 Federal Assault Weapons ban brought in by the Clinton administration. The small number of sources turned up by the review was unsurprising given the paucity of research on the topic of assault weapons was noted by several authors.

The results of the literature are challenging to aggregate, given that different authors measured assault-weapons bans at different levels (federal vs. state) and on different dependent variables. This partially has to do with disagreements over the placement of the goalposts for measuring the impact of gun control measures, like assault-weapons bans. Gius (2014, 266) argues that the impact of these laws should be measured on the firearm-related murder rate since it “is the crime rate most affected by gun control measures, and hence is the most appropriate crime rate to use in an analysis of the effectiveness of gun control measures.” Others examined the impact of assault-weapons bans on the number of fatal mass shootings, the number of people killed in mass shootings, or the number of people injured in mass shootings.

Two additional articles focused on the impact of the Federal Assault Weapon ban on the number of assault weapons recovered by police, or on pricing in the criminal market. Another was an older review of available evidence on the topic.

Overall, the evidence on the effectiveness of assault-weapons bans in the existing literature is mixed. Two articles provided evidence that supported an assault-weapons ban. DiMaggio et al. (2019) found that the 1994 Federal Assault Weapons ban in the United States may have been associated with lower instances of fatalities in mass shootings. Further, Post et al. (2021) used regression discontinuity analysis to estimate that the ban prevented 11 mass shootings during the decade it was in place. The limitations of these studies, however, were that neither were able to separate the impact of Large-Capacity Magazine (LCM) bans from a ban on assault-weapons. Further, DiMaggio and others’ study relied on data collected from the media, which likely overreported the number of mass shootings involving assault-style weapons given their sensational nature. This is reflected in the overrepresentation of assault rifles in their dataset (85.8 percent of firearms) compared to the other authors estimates, which were closer to 20–58 percent (Koper 2020; Lemieux 2014).

The results of five articles did not support an assault-weapons ban. Gius (2014) found no statistically significant relationship between state-level and federal-level assault-weapons bans and reduction in the overall gun-related murder rate. Lemieux (2014), also looking at state-level bans, found no significant relationship between the use of assault weapons and the number of victims, deaths, or injuries in mass shootings in their database. They also found that the Federal Assault Weapons ban had no statistically significant impact on the number of mass shootings. Koper and Roth (2001) found no indication that the ban had an impact on gun violence in the United States, though they noted the temporal limitations of their study, which was conducted only a few years after. Webster et al. (2020) also found no evidence that state-level assault-weapons bans were associated with the overall number of mass shootings. Finally, Kalesan et al. (2016, 1,852) found that state-level assault weapons bans “associated with an increase in the risk of firearm-related deaths,” providing an important illustration of the difference between correlation an causation.

Two articles by the same author provided a mixed assessment. While Gius (2015) found that state-level assault weapons bans may have a small negative impact on mass shooting fatalities and school shooting fatalities (Gius 2018), he is careful to acknowledge the limitations of drawing conclusions from these tentative results, noting that because both mass shootings and school shootings make up a small proportion of gun violence in the United States, “even if a certain type of gun control measure was found to completely eliminate mass shootings (which assault weapons bans do not), the overall murder rate would decline by a very small amount” (Gius 2015, 283–84). Similarly, given the small sample size of school shooting incidents, Gius (2018, 320) notes that “it is unclear if gun control is the most appropriate policy to use to reduce the incidence and severity of school shootings.”

Two broad themes emerge from this review. First, the evidence on the effectiveness of assault weapons bans is mixed, though the balance of the evidence points towards their ineffectiveness. Second, if the bans do impact crime, it is likely the large-capacity magazine limit that is doing most of the heavy lifting. This was noted by several scholars (Koper 2020; Koper and Roth 2001; Lemieux 2014; Webster et al. 2020). For example, Webster et al. (2020, 181) the only study to separate LCM bans from assault-weapons bans, noted that bans on LCMs were associated with “a 48 percent lower risk of fatal mass shooting.”

The implications of this for the Trudeau government's assault-style weapons ban demonstrate that the ban is unlikely to have an impact. Canada has had a ban on large-capacity magazines since 1991 (G. A. Mauser and Buckner 1997). It is worth noting that the Canadian ban limits magazine capacity to five cartridges, half of the level of the 1994 American Federal Assault Weapons Ban and most state-level bans, which limit magazines to ten rounds.

It is also possible that a ban on assault-style weapons could reduce the overall number of guns in Canada and thus reduce gun violence. Evidence from jurisdictions like Australia, however, show that these bans do not necessarily reduce the overall number of guns in the country, as gun owners usually replace banned guns with other models. For example, more than 25 years after Australia's ban, the rate of guns owned per 100 Australians has risen by 1.7 percent. Similarly, the Assault Weapons ban in the United States did not dampen American's enthusiasm for purchasing firearms. In 1994, when the ban was passed, there were 192 million firearms in private hands (Cook and Ludwig 1997). By the year 2000, that number had increased to over 250 million, and by 2004 when the ban expired, was over 275 million (Ingraham 2018).

In addition to tackling mass shootings, the Canadian government tried to frame the assault-style weapons ban as a strategy to reduce intimate partner violence in Canada. In a speech given by Deputy Prime Minister Chrystia Freeland, she stated that these assault-style firearms posed a disproportionate risk to women and girls in Canada (“Deputy Prime Minister Remarks on the Ban on Assault-Style Firearms” 2020). The argument that these firearms are particularly dangerous to women seems once again more framing technique than reality. According to the most recent data released by Statistics Canada, homicide victims in Canada are overwhelmingly male (74 percent), and firearms are rarely used in instances of domestic homicide in Canada, at a rate of 1 per million spouses, which has dropped steadily over the past three decades (Beattie, David, and Roy 2018; Statscan 2018). Statistics presented by the government as part of a Gender-Based Analysis of the 2021 Budget counteract the government's own argument. Firearms were present ⁶ in 0.62 percent of cases of police-reported intimate-partner violence in 2019, suggesting that firearms availability is not a major factor in gender-based violence in Canada (“Budget 2021” 2021). It is also unclear why the assault-style firearms identified by the government pose a particular risk to women compared to other firearms that would still be available.

In assessing the harm that the ban will cause, it is important to look at both quantifiable and unquantifiable harms that a policy can cause to various stakeholders and the government. It is always challenging to balance harm and benefits when evaluating a proposed policy. All policies involve trade-offs. There is value in documenting and clarifying these harms, so that they can be considered against the policy's benefits.

When considering the assault-style weapons ban, the first tangible harm to be considered is the cost to the public. The former Minister of Public Safety Bill Blair has estimated that the cost of the buyback program would be $600 million, while independent experts have estimated the cost would end up being between $1.64 and $4.92 billion (G. Mauser 2020), a fairly considerable public investment.

It is also important to consider the harm done to stakeholder communities, in this case Canadian gun owners and businesses. The ban opens gun owners up to significant legal liability, given the confusion mentioned above regarding which firearms are included in the prohibition. For example, the Royal Canadian Mounted Police has been continuing to add models to the banned list without notifying owners.

The unquantifiable harms caused by the ban are more difficult to assess, especially given the government did not enact the ban through legislation, which would have given stakeholders an opportunity to have their voices heard in the process. There is evidence, however, that the ban will have a significant impact on many of Canada's 2.2 million gun owners. First, the ban involves the confiscation, euphemized as a buy-back, of legally owned property from Canadian citizens, something that should be considered with caution by governments in a liberal democratic society. Sacrificing this principle could harm the legitimacy of the state in the eyes of a significant subset of Canadians. Second, the ban limits the ability of gun owners to participate in hunting and sports shooting, two activities which have a long historical legacy in Canada. Research conducted by the Ontario Federation of Anglers and Hunters indicates that the majority of the firearms implicated in the ban were widely used for hunting (DeMille 2021). For example, the Ruger Mini-14, a popular model of varmint rifles for farmers and hunters, was banned. Many of the firearms included in the prohibition have been commonly available in Canada for over 50 years.

The prohibition also limits the ability of gun owners to participate in many modern shooting sports. While traditional shooting sports, like bench-rest target shooting, involve firing at targets from a fixed position, modern shooting sports like Three-Gun, IPSC, Dynamic Shooting Sports, and others have evolved to incorporate more athleticism as well as modern semi-automatic firearms into the competition. These competitions became popular in the 1970s and are some of the fastest growing shooting sports. They are widely practiced in Canada, the United States, and in many European countries. The “assault-style” weapons ban severely impedes the ability of Canadian athletes to take part in some of these competitions. For example, Ryan Steacy, a Canadian competitive sports shooter, has joined a lawsuit against the government for the ban, noting that it would destroy his athletic career (Notice of Motion: Canadian Coalition for Firearm Rights et al. v. Attorney General of Canada 2020). Hunting and participation in the shooting sports are serious leisure activities and participants devote significant amounts of time, resources, and money to these pursuits (Littlefield and Ozanne 2011; Martin et al. 2014; Schwartz 2021).

As this example shows, the FPEF is useful for making explicit the problem definition and policy aim of proposed firearm policies and critically evaluate the logic, empirical support, and potential harm of these policies. I have demonstrated that the Assault-Style Weapons ban put into place by the Trudeau government faces issues of misalignment between the proposed aims of the policy and its solutions, a lack of empirical support, and has the potential to cause harm to key stakeholders, including the government.

Conclusion

In this article, I proposed the construction of a Firearms Policy Evaluation Framework (FPEF) and laid out some of the considerations that should be included in this framework. The FPEF is designed to be used by scholars, policy makers, advocates, and public to evaluate proposed firearm policies. This is important given that once firearms policies are put into place, they can be difficult to reverse. The framework balances the multiple considerations that go into making social regulatory policy: logic, scholarly evidence, stakeholder needs, and the values and norms of liberal democratic societies. While the framework may not always provide a clear policy solution, it is a useful tool to outline competing problem definitions, make explicit the logical foundations and empirical support for a policy, and consider potential harms to segments of the population. The FPEF provides a simple and visual representation that makes scholarly evidence and expertise available and accessible to a wider audience, contributing to a more robust deliberative democratic discourse.

The potential utility of the tool was demonstrated with an analysis of the Canadian government's recent ban on assault-style firearms. This analysis included a systematic scoping review of the empirical literature on assault-weapons. The FPEF clarified some of the issues with the Liberal government's ban, including an unclear problem representation and policy aim, an absent connection between the proposed problem and solution, the lack of empirical evidence available to support such a ban, and the harm the ban would cause to stakeholders and the government.

Gun control is a useful tool for policy makers, and Canadians benefit from the country's robust gun control infrastructure but, like any policy tool, can hit a point of diminishing returns. The obsession with the Canadian public on statistically rare mass shootings further obscures the everyday realities of gun violence in Canada, which often impact racialized and indigenous populations most heavily (Department of Justice 2019; Yousif 2020). Policy makers should consider alternatives to highly visible firearms policies, such as evidence-based violence interruption programs, which have proven in preliminary analyses to be effective at reducing the most common forms of gun violence (Brantingham et al. 2018; Park et al. 2021; Picard-Fritsche and Cerniglia 2013; Whitehill et al. 2013).

While this article has sketched an initial design for the FPEF, more work is needed to develop the tool into a more sophisticated instrument and to create other policy making tools to close the gap between popular debates on policy making and empirical research. The purpose of EBPM approaches is not necessarily to overcome politics, or to fulfill some sort of mythical promise of value free policy. Tools like the FPEF can, however, help to promote dialogue between interest parties by making explicit the logical assumptions behind policies and putting these policies into conversation with the available evidence in ways that policy makers often seek to avoid.

Future research should focus on further refining the FPEF and applying it beyond Canada. It could be particularly useful in the American context, where the gap between the two sides in the gun debate is much wider than in Canada. Though the legislative and institutional context is different in the United States, where firearms policy is primarily made at the state level, and constitutional considerations weight more heavily than in the Canadian context, the basic considerations at play when making firearms policy are the same.