Sage Journals: Discover world-class research

Abstract

Research on knowledge mobilization in policy making has been largely focused on identifying relevant factors having an effect on the uptake of evidence by actors and organizations. However, evidence on the magnitude of those effects remains limited and existing methods allowing for this have been scarcely used in this field. In this article, we first provide a rationale for greater investigation of substantive effect sizes, using methods such as mediation analysis and conditional probabilities. Using cross-sectional data from Québec (Canada) government policy analysts, we test an absorptive capacity model and describe direct, specific indirect, and total effects estimated from a path analysis. The results show that some factors have considerable effects, such as physical access and individual field of training, whereas some mediated relations are worth considering. Finally, we discuss some practical implications with regard to policy making and policy analysis but also the methodological standards of empirical research in this field.

Keywords

policy analysis path analysis knowledge mobilization evidence-informed policy making probabilities

Introduction

The field of research on knowledge mobilization is in full swing and has speedily developed over the past years. Although a plethora of studies have been published on various aspects of the phenomenon—mostly theoretical and empirical investigations of “facilitators and barriers” of knowledge mobilization—it is fair to say that the evidence accumulated remains limited in some respect. Some systematic reviews of literature (e.g., Innvaer, Vist, Trommald, & Oxman, 2002; Lavis et al., 2005; Oliver, Innvaer, Lorenc, Woodman, & Thomas, 2014; Orton, Lloyd-Williams, Taylor-Robinson, O’Flaherty, & Capewell, 2011) and some literature syntheses on the matter (e.g., Contandriopoulos, Lemire, Denis, & Tremblay, 2010; Mitton, Adair, McKenzie, Patten, & Perry, 2007) have been published, but it has been noted that this field of research nonetheless largely rests on actors’ perceptions (Oliver et al., 2014, p. 13), their understanding of the policy and academic context, their organization’s decision-making processes, and their judgment about the likely importance/impact of those factors:

There are diverse methods for identifying potential barriers including qualitative approaches (individual interviews, focus groups), surveys and direct observation. However, there are no standard approaches available yet. As a result, those involved with knowledge translation activities need to use their judgment about how best to elicit barriers given their understanding of the context and potential barriers and resources available to them. (Grimshaw, Eccles, Lavis, Hill, & Squires, 2012, p. 5)

If this type of method indeed allows researchers to identify potentially important factors (as is the case in studies reported in existing systematic reviews cited above), the fact remains that relatively few studies focus on empirically assessing their relative practical importance. As noted in Oliver et al.’s (2014) systematic review, “this paper has only counted the frequencies with which factors are mentioned without any weighting. Without more research, it is difficult to say what impact different factors might have” (p. 12).

Although it is recommended that “we should in general be more concerned with estimating the sizes or magnitudes of effects (i.e., effect sizes) than with the outcome of statistical tests” (Kline, 2011, p. 13), such recommendations are still rarely taken seriously and applied in social science empirical research. By focusing on conventional test statistics, p values, and regression coefficients, researchers convey only part of the statistical information provided by their data and model. Assessing the magnitude of effects is here seen as a tool that allows discrimination between statistically significant but substantially unimportant effects (i.e., nil or almost nil) and substantially significant effects. In the context of research on knowledge mobilization, this distinction is crucial.

Furthermore, an argument is often made with regard to the apparent complexity of knowledge mobilization–related phenomena (see, for instance, Best & Holmes, 2010; Sanderson, 2009). As suggested by Black (2001), “The implicit assumption of a linear relation between research evidence and policy needs to be replaced with a more interactive model” (p. 275). Willing to avoid simplistic conceptualizations, models are put forth with the intention to reflect the complex conceptual dimensions of the phenomenon, but, in return, appropriate methods to test these models are lagging. This is manifest in the capacity to model mediated relations between explanatory factors of knowledge mobilization.

One potentially relevant tool for this purpose is structural equation modeling (SEM), which allows modeling complex relations by including indirect effect and mediating variables. In other words, it allows the investigation of possible mechanisms operating in a statistical model. Such methods, long known in the social sciences, have been scarcely used in research on knowledge mobilization (see notably, Cummings, Estabrooks, Midodzi, Wallin, & Hayduk, 2007; Wellstead, Stedman, & Howlett, 2011), and where they have been used, issues of interpretation sometimes arise. The problem here is not that the data presented are not instructive or valid, but rather that the communication strategy does not cash out the full implications of the identified relations nor does it allow to qualify the intensity of the relation in a suitable fashion. This is an issue for most results reported from quantitative studies and is perhaps even more salient when performing SEM as the complexity is increased. Strategies need to be devised to provide informative and substantial interpretation of regression coefficients, on an intelligible scale that would be intuitively understood and would reflect some uncertainty by reporting a confidence interval. We here argue that statistical tools can help to simplify the interpretation of regression results and can be put to use in complex models such as SEM. This can be seen as a gain for applied researchers using similar methods, but perhaps even more for non-academic research users interested in consulting and using research results.

Therefore, the main research objectives guiding this article are the following: (a) to identify the correlates of research evidence consultation by policy analysts and the mediated relations between those factors, and (b) to describe, using statistical simulation tools, the magnitude of those effects on the main outcomes. Using cross-sectional data collected among Québec (Canada) government policy analysts, we test a complex model derived from the absorptive capacity model using path analysis and describe the predicted probabilities of explanatory factors of interest.

Extant Literature

Knowledge Mobilization Theories and Assumptions

Knowledge utilization has been conceptualized in a number of ways—most notably as various dimensions (e.g., Weiss, 1979) or various stages (e.g., Knott & Wildavsky, 1980; Rich, 1997; Rich & Oh, 2000), then operationalized into scales and indexes (e.g., Cherney, Povey, Head, Boreham, & Ferguson, 2012; Landry, Amara, & Lamari, 2001). However, although these strategies have had some success, issues of precision remain. Following Estabrooks et al. (2011), we argue that the conceptual confusion commonly found in the field can be, at least in part, resulting from the relative absence of reliable measures and indicators of the phenomenon of interest. That is why we will privilege here a more restrictive understanding of the phenomenon, thus lending itself to unambiguous and empirically relevant measurements.

We here focus specifically on the consultation of scientific evidence by policy analysts, as potential users, while recognizing that diffusion and impact of research are logically distinct phenomena (Rich & Oh, 2000). In a similar vein, potential users of research evidence can be identified and defined rather broadly,¹ and we are here focusing on civil servants categorized as policy analysts located in government departments.

Policy analysis, defined as the analytical and advice-giving tasks performed by policy analysts² to inform the public policy-making process (Colebatch, 2006; Lomas & Brown, 2009; Mayer, van Daalen, & Bots, 2004), has been recently highlighted for its relevance to evidence-informed policy making (Howlett & Newman, 2010), mainly for three reasons. First, given the professional tasks of these individuals, they can be considered as potential entry points for research evidence and possible channels of communication (Cohn, 2007; Howlett, 2009a; Nutley, Walter, & Davies, 2007; Ouimet et al., 2010; Sundell, Wärngård, & Head, 2013). To a certain extent, these professionals can be considered to act as informal knowledge brokers in that they absorb, interpret, and communicate external knowledge (Howlett, 2011). Second, it has been noted that to this day, relatively few studies focus on policy analysts compared with other public administration positions (such as managers or directors—e.g., Howlett & Walker, 2012; Jennings & Hall, 2011): “empirical data on just about every aspect of policy analysis in government are lacking” (Howlett, 2009a, p. 2). And finally, the quality and nature of their professional tasks are intrinsically related to the overall policy analytical capacity of the public administration, the latter being instrumental for evidence-informed policy making to succeed (Howlett, 2009b).

Knowledge mobilization, as an umbrella concept, is seen here as encompassing multiple related phenomena (such as knowledge transfer, knowledge translation, knowledge use, etc.).³ As such, it can also be decomposed along specific dimensions (Green, Ottoson, Garcia, & Hiatt, 2009), namely, the source of research evidence, the content, medium of communication, user’s attributes, and context. What specific theories of knowledge utilization do is depict relations between these dimensions, while highlighting specific factors of interest.⁴ In this respect, three main theoretical approaches, or families of theories, exist (Landry, Lamari, & Amara, 2003; Rich & Oh, 2000): (a) theories dealing with actors’ rationality (and its limitations), (b) theories focusing on organizational interests, and (c) theories highlighting communication and social relations among actors. Although they can be considered as alternative accounts, they can in return be considered complementary theories that can, and to some extent need to be, synthesized.

Theories of rational choice, and perhaps even more in the case of limited rationality theories (Bendor, 2010; Rich & Oh, 2000), focus partly on the individual’s cognitive capacity when processing information. For instance, the ability to cognitively access and interpret scientific results (provided in part by one’s academic training) matters greatly with respect to knowledge mobilization. Also, as limited rationality theories tell us, individuals are bound by structural and organizational factors worth taking into consideration when attempting to provide a specific behavioral account. Therefore, the organizational resources (or lack thereof—physical, technological, human, etc.) in a given policy context are thought to facilitate (or impede) knowledge mobilization (Ellen, Lavis, Ouimet, Grimshaw, & Bédard, 2011; Léon, Ouimet, Lavis, Grimshaw, & Gagnon, 2013; Oliver et al., 2014; Orton et al., 2011). And finally, it is assumed that greater relations with actors within and/or outside the policy context (e.g., academics) can facilitate and stimulate knowledge exchange and mobilization (Innvaer et al., 2002; Lavis et al., 2005; Oliver et al., 2014; Orton et al., 2011). As summarized in Oliver et al.’s systematic review (2014),

Organisational factors, including availability and access to research were considered to be important influences on whether evidence was used in policy, and the quality of the relationship and collaboration between researchers and policymakers to be the single most mentioned facilitator. (p. 11)

We argue that these kinds of factors need to be assessed more thoroughly in their relation to knowledge mobilization as we have little indication of their empirical importance or of possible mediated relations between them.

Absorptive Capacity Model and Hypotheses

The absorptive capacity model summarizes most of the theoretical components surveyed above. Although this framework⁵ was initially developed in the field of management to model features of private entities (i.e., firms), insights from this theoretical perspective can be extended to public organizations (Ouimet, Landry, Ziam, & Bédard, 2009). In this context, the individual absorptive capacity mainly refers to four components that are thought to operate sequentially: the ability to recognize the value of the knowledge to acquire, the acquisition of external knowledge, the utilization of knowledge, and, finally, its application. In other words, if we only focus on the first two components, being able to recognize the value of research and thus have a positive view of its relevance would influence future consultation.

The absorptive capacity as a whole is itself thought to be function of various variables such as prior knowledge, social interactions, and so on. Beyond the individual level, the absorptive capacity is considered to depend on organizational features such as the research knowledge infrastructures available that would allow one to actually obtain research evidence from outside sources such as academic journals. As one can start to appreciate, the absorptive capacity model does not limit itself in describing direct relations between its components and takes on board the assumption that some variables (such as the ability to recognize the value of external knowledge and social interactions) can mediate the relations. Put differently, one could assume under the absorptive capacity model that the likelihood of an individual engaging in a relation or communication with an academic might be mediated by the very fact that this prospective knowledge user has an academic background allowing him to see the policy relevance and usefulness in requesting scientific information from an academic. What we intend by drawing on this model is to reexamine factors that have been highlighted as important in the literature and to assess possible mediated relations using quantitative techniques to estimate the magnitude of effects in that context.

From this model, specific hypotheses can be derived and will be put to the test using the data and methods described below. These hypotheses are formulated for both expected direct effects (i.e., the shortest relation between two variables) and indirect effects (i.e., relations between variables that follow a path through a third variable).

Among the factors affecting the components of the absorptive capacity of the individual is prior knowledge. With some filiation with limited rationality theories, we can expect that research utilization will depend on individual cognitive factors such as academic training. Consequently,

Hypothesis 1: The level of academic training, as well as the disciplinary field of training, has a direct effect on the probability of using research.

Because it is assumed in the model that prior knowledge has an effect on both social interactions and the recognition of the potential value of research, it is possible to derive two hypotheses concerning indirect effects. In both cases, some form of familiarity with academic research principles and practices can act on the level of trust one puts into academics themselves and the knowledge they produce. Therefore, it is plausible that a training background in research can lead to greater interactions with academics and an overall positive attitude toward academic research:

Hypothesis 1.1: The level of academic training, as well as the disciplinary field of training, has an indirect effect on the probability of using research, through an effect on the probability of interactions with academic researchers.

Hypothesis 1.2: The level of academic training, as well as the disciplinary field of training, has an indirect effect on the probability of using research, through an effect on the probability of recognizing the potential value of research.

Here, we can derive a third hypothesis by combining the previous two hypotheses:

Hypothesis 1.3: The level of academic training, as well as the disciplinary field of training, has an indirect effect on the probability of using research, through a successive effect on the probability of interactions with academic researchers and of recognizing the potential value of research.

In this model, the attitude an individual displays toward research (i.e., its perceived relevance, quality, and usefulness) is likely to act as a predictive factor of utilization. As described in the hypotheses above, one’s attitude toward research is bound to be shaped by prior knowledge and experience and can in turn influence interactions with researchers and research utilization. Beyond this, it is plausible that one’s attitude also has a direct effect on research utilization. We can therefore formulate the following hypothesis:

Hypothesis 2: Recognizing the potential value of research has a direct effect on the probability of using research.

Third, the presence (or absence) of organizational resources allowing some form of access to external knowledge is a factor that is generally taken for granted⁶ and therefore is rarely put to empirical testing. It nonetheless appears fundamental in facilitating research utilization. What is more, the level of physical access to external knowledge is likely to vary across policy sectors, as demonstrated by Léon et al. (2013) where substantial variations in access to key scientific journals were found among Canadian Health Ministries. We can thus formulate the following theoretical expectations regarding access infrastructures:

Hypothesis 3: Access to infrastructures permitting research acquisition has a direct effect on the probability of using research.

Although it may appear rather straightforward that physical access to research would be considered a precondition to actual use, physical access to research can also have potential indirect effects, as it is plausible that accessing research might be a means of researcher identification. Through database searches and the retrieval of published literature, one can identify experts in a given field with whom it may appear relevant to engage with. Therefore,

Hypothesis 3.1: Access to infrastructures permitting research acquisition has an indirect effect on the probability of interactions with academic researchers.

Hypothesis 3.2: Access to infrastructures permitting research acquisition has an indirect effect on the probability of using research, through an effect on the probability of recognizing the potential value of research.

Then again, we can derive a third indirect hypothesis by the combination of the previous two hypotheses:

Hypothesis 3.3: Access to infrastructures permitting research acquisition has an indirect effect on the probability of using research, through a successive effect on the probability of interactions with academic researchers and recognizing the potential value of research.

While capturing in part the variations in knowledge infrastructures across the public administration, as well as hiring and training policies, we can argue that levels of consultation of research evidence might vary across specific policy sectors. And although it was not explicitly described as part of the original absorptive capacity framework, it nonetheless appears that the professional context of an individual is an important factor. Therefore,

Hypothesis 4: Working in a given policy sector (such as health and social services), compared with other sectors, has a direct effect on the probability of using research.

Finally, as signaled by Todorova and Durisin (2007), social interactions occupy a central role in the absorptive capacity framework: “Social integration mechanisms, which build connectedness and shared meanings, influence all processes of knowledge absorption” (p. 781, emphasis added). Therefore, we can expect to find the following relation:

Hypothesis 5: Interacting with academic researchers has a direct effect on the probability of using research.

As it was signaled that this factor could affect every component of the model, notably the recognition of the potential value of research, it can be expected that interactions will act as a mediating factor. Although we argued above that prior knowledge can affect the perceived relevance of academic research, it is reasonable to expect that engaging with academics can also contribute to one’s displaying of a positive attitude toward academic research. Therefore,

Hypothesis 5.1: Interacting with academic researchers has an indirect effect on the probability of using research, through an effect on the probability of recognizing the potential value of research.

Combining these hypotheses gives the specific model that has been tested in this article. In the next section, we describe the data and analytical strategy used.

Data and Method

Data and Variables

The data used in this article comes from a telephone survey conducted among Québec (Canada) public administration professional employees.⁷ More specifically, the survey targeted 14 professional groups of civil servants without formal decision-making powers but who nonetheless performed analytical and advice-giving functions. A stratified random sample selection procedure was adopted, and at the end of the data collection phase, 1,617 individuals located in 18 of the 22 ministries had completed the survey, with a net response rate of 62.48%.⁸ The questions asked covered a 12 months period (as opposed to longer reference periods in other surveys, such as 5 years) so as to assure precision. Given the sampling procedure used and the large number of observations, the data can be considered representative of the sampled population of analysts in Québec public administration.

The model described below is built from three outcome (endogenous) variables taken from the survey: The main outcome variable, research utilization, was measured by asking respondents whether they had consulted academic research over the past 12 months within their work context (yes or no), interactions with academic researchers was measured by asking respondents whether they had contacted at least one academic researcher over the last year (yes/no), and attitude toward research was measured by asking respondents to identify and rank which information/knowledge-producing groups (i.e., public civil servants, journalists, university professors/researchers, private-sector consultants, non-university/think tank researchers) produced the most useful information/knowledge, based on the professional experience of the past 12 months.

The main explanatory factors considered here are prior knowledge, captured by a first variable describing the type of training, measured by asking the respondent to identify the type of degree they earned, and by a second variable describing the disciplinary field of training, measured by asking the respondents to identify in which field(s) their training took place; and physical access to research, captured by a variable describing the respondent’s reported access to electronic bibliographic databases from their workstation. As controls, the following variables were included: gender, age, and administrative domain

Analytical Strategy

The analytical strategy applied in this article is inferential and essentially aims at identifying the correlates of knowledge utilization by professional civil servants and at describing the complex mediation relations between those factors and the magnitude of these relations.

Path analysis (as part of the SEM family) was used. Recall that, contrary to conventional regression (linear or not), SEMs allow us to test models accounting for greater complexity by the identification of direct and specific indirect relations. SEMs are mainly promoted due to their ability to model relation between latent variables and multiple indicators, through factorial analysis. Although it may seem preferable, a priori, to build theoretical constructs from multiple latent variables, it appears that it is not a necessary condition to perform valid analyses.⁹ For instance, Kline (2011) suggests that solely using observed variables is admissible: “There are also times when there is just a single observed measure of each construct, and [path analysis] is a single-indicator technique” (p. 103). In other words, path analysis in such a context would be based only on manifest, directly observed variables (MacKinnon, 2008). In this respect, the quality of the analysis is intimately related to the quality of the variables mobilized, but depends most of all on the quality of the model: “one indicator is sufficient if it is properly causally specified” (Hayduk et al., 2007, p. 284; see also Hayduk & Littvay, 2012). In this case, as in many others, parsimony should be privileged.

Path Model

The path model, derived from relevant theories and hypotheses described above, can be summed up by the following three equations:

y_{1} = β_{4} y_{2} + β_{5} y_{3} + γ_{6} x_{1} + γ_{7} x_{2} + γ_{8} x_{3} + γ_{9} x_{4} + γ_{10} x_{5} + γ_{11} x_{6} + γ_{12} x_{7} + γ_{13 - 27} x_{8 - 22} + γ_{29 - 32} x_{24 - 27} + ζ_{1},

y_{2} = γ_{33} x_{1} + γ_{34} x_{2} + γ_{35} x_{3} + γ_{36} x_{4} + γ_{37} x_{5} + γ_{38} x_{6} + γ_{39} x_{7} + γ_{40 - 54} x_{8 - 22} + γ_{55} x_{23} + γ_{56 - 59} x_{24 - 27} + ζ_{2},

y_{3} = β_{60} y_{2} + γ_{61} x_{1} + γ_{62} x_{2} + γ_{63} x_{3} + γ_{64} x_{4} + γ_{65} x_{5} + γ_{66} x_{6} + γ_{67} x_{7} + γ_{68 - 82} x_{8 - 22} + γ_{83} x_{23} + γ_{84 - 87} x_{24 - 27} + ζ_{3},

where, $y_{n}$ corresponds to endogenous variables, each with its own sub-equation, $β_{n} y_{n}$ represents the estimated parameters between two endogenous variables, $γ_{n} x_{n}$ describes parameters estimated for exogenous variables, and $ζ_{n}$ is an error term included in each equation. This model is a saturated and recursive one.¹⁰ It basically means that every possible relation in the model is estimated, and as a consequence, it is a model with zero degrees of freedom.

The justification for specifying a saturated path model comes from the fact that, contrary to what one might intuitively think, postulating a model with fewer relations is equal to imposing supplementary assumptions about the theoretical model. As Reichardt puts it (2002),

Overidentifying restrictions are never correct. The fewer incorrect restrictions that are imposed, the less biased the estimates of the path coefficients are likely to be, ceteris paribus. Because they have no overidentifying restrictions, just-identified models are to be preferred to overidentified models to the extent that minimizing bias is a priority. (p. 313)

In our case, we lack any plausible justification to withdraw a theoretical relation in the model. This is a contentious argument as most SEM researchers tend to privilege over-identified models because these allow overall model evaluation through exact fit tests and various adjustment indices.¹¹ As suggested by Bowen and Guo (2012),

It should be noted that the magnitude of factor loadings and/or structural paths does not have to be large in order for a model to have good fit. A model may have good fit and yet contain paths that are nonsignificant or smaller than expected. (p. 149)

What then is a well-fitting model if the individual relations assessed have little or no magnitude?

This disjunction between model fit and individual relations is due to the nature of the tests performed (i.e., chi-square and fit indices), which, more or less, only reflects the disparity between the data covariance matrix (Sigma) and the theory-implied matrix (Sigma $Θ$ ). In other words, a proper fit is not a sufficient condition to corroborate the verisimilitude of a model (Hayduk & Pazderka-Robinson, 2007; Kline, 2011; Tomarken & Waller, 2003; Waller & Meehl, 2002). This is where estimating the predicted probabilities and the marginal effects of variables of interest come in handy.

Predicted Probabilities

The estimation of the model described above generates results of interest (direct, indirect, and total effects—standardized or not), but our main focus lies in the estimation of the magnitude of those effects on an intelligible scale. To this end, we use statistical simulation to convert the estimated parameter of effects into probabilities and express them along with a confidence interval. The estimation of those predicted probabilities follows a counterfactual perspective in that we compare an observed state with a hypothetical one—in a fashion akin to experimental epistemology (Reiss, 2011). This perspective is particularly well-advised when using dichotomous variables (as in our case), as suggested by Valeri and VanderWeele (2013): “The counterfactual approach . . . helps in understanding mediation with binary outcomes and binary mediators” (p. 6).

More specifically, the estimation proceeds by comparing the predicted probabilities of the outcome variable (y) when a given variable (x) shifts in value (i.e., typically, from 0 to 1, for a binary variable) while holding other variables constant. In this case, the exogenous variables remaining constant are left to their observed values (rather than to their mean or any other value—mean, mode, etc.). Therefore, the values used to estimate the effect are the same values each individual takes on each variable. The strategy involves three steps.

First, the model described above was estimated¹² using Mplus 6 (Muthén & Muthén, 1998-2010) and the outputs (mainly, the parameters and the variance–covariance matrix) were then adapted to be read on Stata 12 (StataCorp, 2011).¹³ Second, a normal distribution (n = 1,000) is simulated for each parameter of the model (i.e., the parameter itself becomes the mean of that new distribution) so as to allow for a confidence interval. Finally, the probit equations for each model are solved: The outcome of the equations where the variable of interest is at its maximum (1) is subtracted to the outcome of the equations where the variable of interest is at its minimum (0). This is computed by simulating sequentially, through a loop and for each individual in the data set ( $x_{i - n}$ ). A normal distribution of parameters is obtained for individual i simulating that every individual has the same values on every variable, and then an average effect is obtained for this specific individual by estimating the subtraction described above. The same goes for $x_{2}$ to $x_{n}$ , and in the end, the reported effect is an average of those averaged effects, now described within a lower and upper confidence interval (95%). In all these stages, the exogenous variables remain constant at their observed values, and the results are expressed following a conversion into probabilities.¹⁴ Therefore, the estimated outcome can be interpreted as the effect, in percentage points, of the explanatory factor of interest on the probability of occurrence of the outcome, bound within a confidence interval. This strategy was repeated for every exogenous variable of interest (i.e., every factor found to be statistically significant in the model).

Results

Descriptive Statistics

First of all, it was found that 78.45% of the respondents answered positively when asked whether they had consulted at least one scientific article over the past 12 months. As was expected, there are some variations in answers when compared across policy sectors. For instance, the Ministry of Sustainable Development, Environment, and Parks, the Ministry of Agriculture, Fisheries, and Foods, and the Ministry of Health and Social Services are the administrative domains where consultation rates were the highest with, respectively, 91.43%, 87.22%, and 87% of respondents claiming that they have consulted research evidence over the past 12 months. Comparatively, the Ministry of Justice, the Ministry of Family and Seniors, and Horizontal ministries (i.e., Executive Council and Treasury Board) were the administrative domains where the rates were the lowest: Respondents claimed to have consulted research evidence with proportions of 56.1%, 58.06%, and 63.01%, respectively.

When looking at the other endogenous variables of our model, we found that 53.93% of respondents had not contacted an academic researcher over the past 12 months and most of those who did contacted researchers in (exclusively) natural sciences and engineering (17.5%). With regard to the attitude toward research (i.e., recognizing its potential value), it was found that 53.91% ranked university professors or researchers at the top of the list (first or second) when comparing five information and knowledge producers (i.e., public civil servants, non-academic think tanks, journalists, private-sector consultants).

Path Model

The estimated paths¹⁵ are represented in Figure 1, as we see the relations of interest between the exogenous variables and the three endogenous variables considered that were described theoretically in the structural equations above. Recall that this is a saturated model, and as such, it does not show any degree of freedom, which means that the global model fit cannot be assessed. Nonetheless, and in line with our research objective, it remains instructive to scrutinize the individual relations of the model and their magnitude.

Figure 1.

Model, estimated relations, and coefficients.

A certain number of variables significantly affect the principal endogenous variable—research utilization—in a direct fashion. More specifically, seven of them describe either a positive (+) or a negative (−) relation: Interactions with academic researchers (+), attitude toward academic research (+), holding a professional master and/or doctoral degree (+), holding a research master and/or doctoral degree (+), academic training in social sciences (−), academic training in administrative sciences (−), and physical access to bibliographic databases (+). In the same way, three variables directly affect the interactions with academic researchers variable, that is, holding a research master and/or doctoral degree (+), academic training in administrative sciences (−), and physical access to bibliographic databases (+). Also, we denote three variables, which were significantly associated in a direct fashion with the attitude toward academic research variable: Interactions with academic researchers (+), holding a professional master and/or doctoral degree (+), and academic training in administrative sciences (−).¹⁶

In this model, some specific indirect relations (i.e., relations mediated by one or multiple factors) appear as being significant when in relation with research use, through a path via the interaction with academic researchers variable. There are three of those: Holding a research master and/or doctoral degree (+), academic training in administrative sciences (−), and physical access to bibliographic databases (+). Also, it was found that two indirect relations allow some influence on research consultation, via the attitude toward research variable. These two variables are the following: Holding a professional master and/or doctoral degree (+) and physical access to bibliographic databases (+). Finally, and almost by way of deduction (or perhaps extension), we note that some indirect relations are significant when influencing research consultation through paths describing successive influence on the interaction with academic researchers and attitude toward academic research variables. The three variables forming these chains of influence are the following: Holding a research master and/or doctoral degree (+), academic training in administrative sciences (−), and physical access to bibliographic databases (+).

Predicted Probabilities

Applying the method described earlier on, the effects of variables that have been found statistically significant have been estimated. The results of those estimations are presented in Table 1.

Table 1.

Predicted Probabilities for Direct, Indirect, and Total Effects Using Observed Baseline Values.

Specific indirect effects	Pr = (x_l − x₀)	95% CI
Professional MA/PhD → Interactions → Consultation	.008	[−.003, .023]
Professional MA/PhD → Attitude → Consultation	.013	[.002, .027]
Professional MA/PhD → Interactions → Attitude → Consultation	.012	[.000, .028]
Direct effect	.072	[.017, .137]
Total effect	.106	[.047 .178]

Research MA/PhD → Interactions → Consultation	.026	[.011, .047]
Research MA/PhD → Attitude → Consultation	.007	[−.001, .019]
Research MA/PhD → Interactions → Attitude → Consultation	.011	[−.003, .028]
Direct effect	.073	[.024, .133]
Total effect	.117	[.061, .186]

Administrative sciences → Interactions → Consultation	−.043	[−.093, −.012]
Administrative sciences → Attitude → Consultation	−.022	[−.059, −.000]
Administrative sciences → Interactions → Attitude → Consultation	−.003	[−.049, .043]
Direct effect	−.208	[−.442, −.010]
Total effect	−.276	[−.515, −.057]

Physical access → Interactions → Consultation	.025	[.010, .044]
Physical access → Attitude → Consultation	.004	[−.004, .015]
Physical access → Interactions → Attitude → Consultation	.008	[−.008, .027]
Direct effect	.152	[.088, .225]
Total effect	.189	[.115, .273]

Note. The shaded lines represent the relations that are detected as statistically non-significant (at the 0.05 threshold) when estimating the model. They are nonetheless included in the computation as they theoretically contribute to the total effect. CI = confidence interval.

First of all, it can be seen that specific relations can be identified as being significant or insignificant by looking at the scope of the confidence interval: An effect covering 0 is potentially nil and is likely to be insignificant. Relations that have been found insignificant earlier describe a potentially nil effect, but nonetheless, some relations describe a potentially nil effect while they had been found statistically significant in the basic model. That is the case for the specific indirect relations sequentially going through both mediators (i.e., interactions and attitude) that starts from both postgraduate training in research and physical access to bibliographic databases.

As described in Table 1, the single most important effect found in this study, interpretable in percentage points,¹⁷ is attributable to training in administrative sciences (−27.6%), when compared with training in social sciences. However, the intervals are rather wide in this case, thus limiting the precision of the estimate. Let us note that this variable also has a sizable negative effect on research utilization (−4.3%) through an effect on the probability of interactions with academic researchers. The variable describing the physical access to research also has a considerable effect on research utilization as its total effect is of 18.9% (which corresponds to a direct effect of 15.2% and to an indirect specific effect of 2.5% mediated by the interactions with academic researchers variable). With regard to the postgraduate training in research variable, when compared with bachelor’s degree, we note an increase of 11.7% on the probability of consulting research. This total effect can be decomposed into a direct effect of 7.3% and a specific indirect effect of 2.6% mediated by the interactions with academic researchers variable. Finally, the postgraduate professional training variable, when compared with a bachelor’s degree, describes the least important effect as we note a total effect of 10.6% on the probability of using research. Put differently, this variable generates a direct effect of 7.2% and has a specific indirect effect of 1.3% on utilization, mediated by the attitude toward research variable.¹⁸ In return, notice that the interval bounds comprise values close to 0.

We can see through these analyses that the specific indirect effects remain somewhat weak (perhaps with the exception of the training in administrative sciences variable, mediated through the interactions variable). Even in the case of statistically significant relation, the confidence interval bounds tend to include values leaning toward 0, thus increasing the likelihood of a potential nil effect. As a corollary, we note that none of the variables generated specific indirect effect that were all significant: In all cases, only one out of three relations appear significant (as one can tell either by the p value in the model or the confidence intervals describing the distribution of the postestimations).

Discussion

Theoretical and Policy Implications

Based on the results just described, we can affirm that our hypotheses regarding direct relations have been largely corroborated, with some nuances with regard to the indirect relations expected. First of all, it was found that academic training at the postgraduate level (either professional or research-oriented), when compared with an undergraduate level’s training, is a significant correlate of research utilization. In contrast, there is a difference between a postgraduate training in research (i.e., research-oriented master thesis and PhD), associated with a direct and positive relation to the probability of interactions with researchers, and a professional postgraduate training (i.e., master with essays and internships), rather associated with a positive attitude toward research. With regard to the field of training, administrative sciences (compared with health sciences) were the only field where a significant (and negative) relation was found.

These sets of hypotheses thus corroborate the existing assumptions according to which prior experience in research can act as a predisposition to research utilization (Hanney, Gonzalez-Block, Buxton, & Kogan, 2003; Jewell & Bero, 2008; Landry et al., 2003; Lavis et al., 2005; Orton et al., 2011). In this respect, our conclusions tend to support views expressed elsewhere:

Research skills are . . . crucial in generating, assessing and analyzing evidence. . . . Such skills can, for instance, help to expose the meaning and limitations of evidence from both formal research and other sources. In addition, a detailed understanding of research methodology is needed for effective relationships with external research contractors, in relation to complex data analysis and other issues. (Nocon, 2009, p. 96)

This last remark suggests that the interpretation of results and their import into a given policy context are rarely straightforward (Greener & Greve, 2013; Moat, Lavis, & Abelson, 2013). In a similar fashion, our results support the idea that policy analysis, to be properly conducted, requires a set of specific skills (Howlett & Lindquist, 2004; Kothari, MacLean, & Edwards, 2009; Oliver et al., 2014). What is manifest here is that research abilities are crucial for policy actors to fully adopt and implement the principles of evidence-informed policy making (Bédard & Ouimet, 2012; Howlett, 2009b). Without such skills, multiple consequences are foreseeable, such as the inability to recognize and interpret limitations and potential biases and/or erroneous application of research results (Deeming, 2013; Jewell & Bero, 2008).

Our second hypothesis was also corroborated as it was found that displaying a positive attitude toward the potential value of research was identified as increasing the probability of research utilization. Nonetheless, this variable, when considered a mediator, contributed rather weakly to an explanation of research utilization, as most indirect relations mediated by this variable were found insignificant. Therefore, our results, without completely falsifying this hypothesis, suggest a nuanced interpretation of this otherwise importantly regarded factor in the literature (see notably, Landry et al., 2001; Lavis et al., 2005; Mitton et al., 2007; Orton et al., 2011).

Similar to the relations pertaining to the field of training variable, the estimated relations from the physical access variable corroborate our hypotheses regarding the direct relation and the indirect relations, with the exception of the relation mediated by the attitudinal variable. This organizational factor, which used to be somewhat ignored in previous systematic reviews (see, for instance, Innvaer et al., 2002; Lavis et al., 2005), is considered to be a crucial one in explaining research utilization (Oliver et al., 2014). Our results corroborate this idea, as it was the most important factor in terms of magnitude in our study.

The fourth hypothesis concerned the expected variations in research utilization across policy sectors (Davies, Nutley, & Smith, 2000; Howlett & Lindquist, 2004) and was also corroborated. Without detailing all the estimated comparisons, let us note that when compared with the Ministry of Justice, some ministries (for instance, Health and Social Services, Municipal Affairs, Regions and Land Occupancy, Agriculture, Fisheries, and Food, etc.¹⁹) had positive effect on research utilization, whereas others (for instance, the Immigration and Cultural Communities, etc.²⁰) had a positive effect on interactions with researchers. Finally, some ministries (such as Transportation²¹) instead had a negative effect on displaying a positive attitude toward research. Overall, sectors were a considerable source of variation, often having a rather sizable effect in percentage points.

Finally, the fifth set of hypotheses, which described the direct relation between the interaction variable and the utilization of research, as well as the indirect relation between them mediated by the attitude variable, was found to be significant. Recall that the argument underlying the interaction variable suggests a certain number of virtues to this explanatory factor:

Interaction can also promote receptivity of the research by policy makers through early and repeated engagement of them with the research findings. Interactions might help research users to better understand research findings and facilitate information exchange between the two groups about professional norms and expectations. (Kothari, Birch, & Charles, 2005, p. 119)

In this respect, our results suggest that interactions perform these tasks, in part through direct effects, but also in part through indirect effect—the latter not being recognized formally in the extant literature. One of the distinctive features of our results is that while the interaction variable was found to be significant, it was also found to be instrumental in obtaining statistically significant indirect effects. Interactions with academic researchers are generally presented as a substantive explanatory factor—although its proper empirical impact is seldom demonstrated (see notably, Almeida & Báscolo, 2006; Contandriopoulos et al., 2010; Green et al., 2009; Innvaer et al., 2002; Lavis et al., 2005; Mitton et al., 2007)—and it is implied that a direct effect is at work. In contrast, our analyses allow us to support this claim, but also demonstrate that its effect is in fact greater than expected once we take into account its potential indirect effects.

At this stage, we can formulate four general observations with regard to the theoretical implications of our results. First of all, the absorptive capacity framework was largely corroborated. Although the global model has not been tested per se, its corroboration is evidenced by the examination of its lower-order components (i.e., the specific relations constituting the model—Bowen & Guo, 2012). What we have shown is consistent with the main approach focusing on interactions but also that more complex relations are at work between these factors.

Second, our analyses show that beyond individual relations, an explanation of research mobilization appears to be built on a conjunction of factors. In this respect, it would have been possible to estimate the probability of utilization as a function of a combination of factors, rather than isolating a single one each time while maintaining the other factors constant. These possible combinations—being quite numerous, and partly arbitrary—have been ignored for parsimony’s sake. Nonetheless, we have noted that cumulated specific indirect effects represent a substantial part of the effect emanating from a given factor. It could be expected to observe similar patterns in different contexts.

This combination of factors could be heuristically represented using Mackie’s (1965) description of INUS conditions as being “an insufficient but necessary part of a condition which is itself unnecessary but sufficient for the result” (p. 245). In this respect, it would be interesting that actors involved in the implementation of interventions aiming at increasing the mobilization of scientific knowledge in the policy process consider similar combination of factors. As it was shown here, certain variables can be considered as being anterior to a given phenomenon. For instance, implementing an intervention aiming at increasing the level of interaction with researchers is likely to be limited in a context where cognitive factors are not taken into account. In other words, some relations appear to have sequential features.

Third, the mediation analysis performed in this study demonstrates, as could have been expected, that a single factor can have multiple effects. This observation is partly related to our first one regarding the complexity at work between the explanatory factors. There is a need here for a greater recognition of this in the theoretical literature but perhaps more importantly in the analytical and empirical efforts devoted to reflect this complexity. To the extent that “contributions to the effect can come from different sources and via different pathways” (Cartwright & Hardie, 2012, p. 24, emphasis added), it is important to take these into account when setting interventions to increase research mobilization. Recall that it would be possible to dismiss some possible explanatory factors as having no direct effect while some sizable indirect might be at work. In other words, our understanding of the extant literature lead us to observe a gap between theories, the research objectives, and the methodological apparatus deployed accordingly.

Finally, although our results can be confidently generalized to our study population, our research design does not warrant further generalizations. Policy analysts may tend to share similar attributes and tasks, they nonetheless operate in highly varied administrative contexts (e.g., resources, networks, decision-making processes, etc.), thus limiting comparability. Furthermore, little data are available to systematically compare those policy analysts across contexts (for related surveys, see, for instance, Howlett, 2009a; Howlett & Wellstead, 2011, 2012; Wellstead & Stedman, 2010; Wellstead, Stedman, & Lindquist, 2009).

Implications for Researchers

Beyond the theoretical and empirical aspect of our work, three methodological features need to be discussed. First of all, our analytical strategy is in line with further attention to more complex models integrating mediated relations allowing the exploration of causal chains and causal mechanisms. Methodological refinement, when applied to substantial research questions, can be highly valuable, especially in a field that can hardly be said to have reached maturity. Moreover, beyond the theoretical motivation to pursue these, it has been argued that these can in fact be relevant from a policy point of view (Pearl, 2001), as policy interventions can plausibly have indirect effects and take place in a set of complex relations. Perhaps more difficult to interpret, complex models nonetheless appear to be more realistic when used to described social and policy contexts.

Second, remaining committed to simple regression analysis, in our case, would have underestimated the empirical importance of some factors as demonstrated above. Although most indirect effects remained relatively modest, they nonetheless appeared rather sizable when considered conjointly (for instance, when considering the difference between the total and the direct effect²²). We argue, after many others (Ferguson, 2009; Fritz, Scherndl, & Kühberger, 2013; Hanmer & Kalkan, 2013; Herron, 1999; King, Tomz, & Wittenberg, 2000; Preacher & Kelley, 2011; Wood, 2013; etc.), that there is a real need to go beyond basic statistics and to report information more intelligible and substantial metrics than mere coefficients, p values, and statistical thresholds. When relevant to do so, the benefit from this methodological shift is (at least) twofold: Significant and substantive effects (as opposed to significant but unsubstantive ones) can more easily be detected, and research results can be interpreted and communicated in a more intelligible way to both researchers and policy makers. As the field of research on mobilization has been concerned with formats of presentation of results (Lavis et al., 2005; Lavis, Permanand, Oxman, Lewin, & Fretheim, 2009), here is one aspect that should require greater attention and interest.

Third, although not discussed at length in the present article, the analytical strategy to compute quantities of interest needs to be carefully chosen. As suggested by Hanmer and Kalkan (2013), the outcome of the postestimation is likely to be sensitive to the computational strategy. In our case, when isolating an effect of interest, the remaining variables were set to be constant at the observed values, although they could have been set to any other value. The rationale for doing for was to remain as close as possible to the actual values of the data set to ensure greater realism. However, it is frequent to find in the empirical research literature that these values have been set to be constant to their mean, their mode, an average case or else, and that it is not necessarily relevant to do so (for instance, setting a dichotomous variable to its mean). It has been argued elsewhere that one should be careful when selecting imputed values in the computation of predicted probabilities so as to avoid extreme or unrealistic counterfactual scenarios can bias the estimation (King & Zeng, 2006) and we hereby reiterate these warnings. More generally, the idea here is that results tend to be sensitive to the methodological and analytical choices one makes, and in this respect, greater rigor and transparency would be most welcome.

Footnotes

Acknowledgements

This article has gone through multiple iterations, and the author therefore thanks Mathieu Ouimet and Julian Reiss for useful thoughts and suggestions in the process.

Author’s Note

This research was completed while holding a postdoctoral fellowship from the Social Sciences and Humanities Research Council of Canada (SSHRC).

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research and/or authorship of this article.

Notes

Author Biography

Pierre-Olivier Bédard is currently a postdoctoral research fellow at Durham University (UK). He holds a PhD in political science from Laval University (Canada).

References

Almeida

Báscolo

(2006). Use of research results in policy decision-making, formulation, and implementation: A review of the literature. Cadernos de Saúde Pública, 22, 7-19.

Bédard

P.-O.

Ouimet

(2012). Cognizance and consultation of randomized controlled trials among ministerial policy analysts. Review of Policy Research, 29, 625-644.

Bendor

(2010). Bounded rationality and politics. Berkeley: University of California Press.

Best

Holmes

(2010). Systems thinking, knowledge and action: Towards better models and methods. Evidence & Policy, 6, 145-159.

Black

(2001). Evidence based policy: Proceed with care. Commentary: Research must be taken seriously. British Medical Journal, 323, 275-279.

Bowen

N. K.

Guo

(2012). Structural equation models. New York, NY: Oxford University Press.

Canadian Institute of Health Research. (2013). Retrieved from http://www.cihr-irsc.gc.ca/f/29418.html

Cartwright

Hardie

(2012). Evidence-based policy: A practical guide to doing it better. New York, NY: Oxford University Press.

Cherney

Povey

Head

Boreham

Ferguson

(2012). What influences the utilisation of educational research by policy-makers and practitioners? The perspectives of academic educational researchers. International Journal of Educational Research, 56, 23-34.

10.

Cohen

W. M.

Levinthal

D. A.

(1990). Absorptive capacity: A new perspective on learning and innovation. Administrative Science Quarterly, 35, 128-152.

11.

Cohn

(2007). Academics and public policy: Informing policy analysis and policymaking. In Dobuzinskis

Howlett

Laycock

(Eds.), Policy analysis in Canada: The state of the art (pp. 574-597). Toronto, Ontario, Canada: University of Toronto Press.

12.

Colebatch

(2006). What work makes policy? Policy Sciences, 39, 309-321.

13.

Contandriopoulos

Lemire

Denis

J.-L.

Tremblay

(2010). Knowledge exchange processes in organizations and policy arenas: A narrative systematic review of the literature. Milbank Quarterly, 88, 444-483.

14.

Cummings

G. G.

Estabrooks

C. A.

Midodzi

W. K.

Wallin

Hayduk

(2007). Influence of organizational characteristics and context on research utilization. Nursing Research, 56(4 Suppl.), S24-S39.

15.

Davies

H. T. O.

Nutley

Smith

P. C.

(2000). What works? Evidence-based policy and practice in public services. Bristol, UK: Policy Press.

16.

Deeming

(2013). Trials and tribulations: The “Use” (and “Misuse”) of evidence in public policy. Social Policy & Administration, 47, 359-381.

17.

Durning

Osuna

(1994). Policy analysts’ roles and value orientations: An empirical investigation using Q methodology. Journal of Policy Analysis and Management, 13, 629-657.

18.

Ellen

M. E.

Lavis

J. N.

Ouimet

Grimshaw

Bédard

P.-O.

(2011). Determining research knowledge infrastructure for healthcare systems: A qualitative study. Implementation Science, 6, Article 60.

19.

Estabrooks

C. A.

Squires

J. E.

Strandberg

Nilsson-Kajermo

Scott

S. D.

Profetto-McGrath

. . . Wallin

(2011). Towards better measures of research utilization: A collaborative study in Canada and Sweden. Journal of Advanced Nursing, 67, 1705-1718.

20.

Ferguson

C. J.

(2009). An effect size primer: A guide for clinicians and researchers. Professional Psychology: Research and Practice, 40, 532-538.

21.

Fritz

Scherndl

Kühberger

(2013). A comprehensive review of reporting practices in psychological journals: Are effect sizes really enough? Theory & Psychology, 23, 98-122.

22.

Green

L. W.

Ottoson

J. M.

Garcia

Hiatt

(2009). Diffusion theory and knowledge dissemination, utilization, and integration in public health. Annual Review of Public Health, 30, 151-174.

23.

Greener

Greve

(2013). Introduction: Evidence and evaluation in social policy. Social Policy & Administration, 47, 355-358.

24.

Grimshaw

Eccles

Lavis

J. N.

Hill

S. J.

Squires

J. E.

(2012). Knowledge translation of research findings. Implementation Science, 7, Article 50.

25.

Hanmer

M. J.

Kalkan

K. O.

(2013). Behind the curve: Clarifying the best approach to calculating predicted probabilities and marginal effects from limited dependent variable models. American Journal of Political Science, 57, 263-277.

26.

Hanney

Gonzalez-Block

Buxton

Kogan

(2003). The utilisation of health research in policy-making: Concepts, examples and methods of assessment. Health Research Policy and Systems, 1, Article 2.

27.

Hayduk

Littvay

(2012). Should researchers use single indicators, best indicators, or multiple indicators in structural equation models? BMC Medical Research Methodology, 12, Article 159.

28.

Hayduk

Pazderka-Robinson

(2007). Fighting to understand the world causally: Three battles connected to the causal implications of structural equation models. In Outhwaite

Turner

(Eds.), The SAGE handbook of social science methodology (pp. 147-171). London, England: SAGE.

29.

Hayduk

Robinson

H. P.

Cummings

G. G.

Boadu

Verbeek

E. L.

Perks

T. A.

(2007). The weird world, and equally weird measurement models: Reactive indicators and the validity revolution. Structural Equation Modeling, 14, 280-310.

30.

Herron

M. C.

(1999). Postestimation uncertainty in limited dependent variable models. Political Analysis, 8, 83-98.

31.

Howlett

(2009a). Policy advice in multi-level governance systems: Sub-national policy analysts and analysis. International Review of Public Administration, 3(3), 1-16.

32.

Howlett

(2009b). Policy analytical capacity and evidence-based policy-making: Lessons from Canada. Canadian Public Administration, 52, 153-175.

33.

Howlett

(2011). Public managers as the missing variable in policy studies: An empirical investigation using Canadian data. Review of Policy Research, 28, 247-263.

34.

Howlett

Lindquist

(2004). Policy analysis and governance: Analytical and policy styles in Canada. Journal of Comparative Policy Analysis: Research and Practice, 6, 225-249.

35.

Howlett

Newman

(2010). Policy analysis and policy work in federal systems: Policy advice and its contribution to evidence-based policy-making in multi-level governance systems. Policy and Society, 29, 123-136.

36.

Howlett

Walker

R. M.

(2012). Public managers in the policy process: More evidence on the missing variable? Policy Studies Journal, 40, 211-233.

37.

Howlett

Wellstead

A. M.

(2011). Policy analysts in the bureaucracy revisited: The nature of professional policy work in contemporary government. Politics & Policy, 39, 613-633.

38.

Howlett

Wellstead

A. M.

(2012). Professional policy work in federal states: Institutional autonomy and Canadian policy analysis. Canadian Public Administration, 55, 53-68.

39.

Innvaer

Vist

Trommald

Oxman

(2002). Health policy-makers’ perceptions of their use of evidence: A systematic review. Journal of Health Services Research & Policy, 7, 239-244.

40.

Jennings

E. T.

Jr. Hall

J. L.

(2011). Evidence-based practice and the use of information in state agency decision making. Journal of Public Administration Research and Theory, 22, 245-266.

41.

Jewell

C. J.

Bero

L. A.

(2008). “Developing good taste in evidence”: Facilitators of and hindrances to evidence-informed health policymaking in State Government. Milbank Quarterly, 86, 177-208.

42.

King

Tomz

Wittenberg

(2000). Making the most of statistical analyses: Improving interpretation and presentation. American Journal of Political Science, 44, 341-355.

43.

King

Zeng

(2006). The dangers of extreme counterfactuals. Political Analysis, 14, 131-159.

44.

Kline

R. B.

(2011). Principles and practice of structural equation modeling (3rd ed.). New York, NY: Guilford Press.

45.

Knott

Wildavsky

(1980). If dissemination is the solution, what is the problem? Science Communication, 1, 537-578.

46.

Kothari

Birch

Charles

(2005). Interaction and research utilization in health policies and programs: Does it work? Health Policy, 71, 117-125.

47.

Kothari

MacLean

Edwards

(2009). Increasing capacity for knowledge translation: Understanding how some researchers engage policy makers. Evidence & Policy, 5, 33-51.

48.

Landry

Amara

Lamari

(2001). Utilization of social science research knowledge in Canada. Research Policy, 30, 333-349.

49.

Landry

Lamari

Amara

(2003). The extent and determinants of the utilization of university research in government agencies. Public Administration Review, 63, 192-205.

50.

Lavis

J. N.

Davies

Oxman

Denis

J.-L.

Golden-Biddle

Ferlie

(2005). Towards systematic reviews that inform health care management and policy-making. Journal of Health Services Research & Policy, 10, 35-48.

51.

Lavis

J. N.

Permanand

Oxman

A. D.

Lewin

Fretheim

(2009). SUPPORT Tools for evidence-informed health Policymaking (STP) 13: Preparing and using policy briefs to support evidence-informed policymaking. Health Research Policy and Systems, 7(Suppl. 1), Article S13.

52.

Léon

Ouimet

Lavis

Grimshaw

J. M.

Gagnon

M.-P.

(2013). Assessing availability of scientific journals, databases, and health library services in Canadian Health Ministries: A cross-sectional study. Implementation Science, 8, Article 34.

53.

Lomas

Brown

A. D.

(2009). Research and advice giving: A functional view of evidence-informed policy advice in a Canadian Ministry of Health. Milbank Quarterly, 87, 903-926.

54.

MacCallum

R. C.

Austin

J. T.

(2000). Applications of Structural Equation Modeling in psychological research. Annual Review of Psychology, 51, 201-226.

55.

Mackie

J. L.

(1965). Causes and conditions. American Philosophical Quarterly, 2, 245-264.

56.

MacKinnon

(2008). An introduction to mediation analysis. New York, NY: Taylor & Francis.

57.

Mayer

van Daalen

Bots

P. W. G.

(2004). Perspectives on policy analysis: A framework for understanding and design. International Journal of Technology, Policy and Management, 4, 169-191.

58.

Mitton

Adair

C. E.

McKenzie

Patten

S. B.

Perry

B. W.

(2007). Knowledge transfer and exchange: Review and synthesis of the literature. Milbank Quarterly, 85, 729-768.

59.

Moat

K. A.

Lavis

J. N.

Abelson

(2013). How contexts and issues influence the use of policy-relevant research syntheses: A critical interpretive synthesis. Milbank Quarterly, 91, 604-648.

60.

Muthén

L. K.

Muthén

B. O.

(1998-2010). Mplus user’s guide (7th ed.). Los Angeles, CA: Author.

61.

Noblet

J. P.

Simon

Parent

(2011). Absorptive capacity: A proposed operationalization. Knowledge Management Research & Practice, 9, 367-377.

62.

Nocon

(2009). Using evidence to inform policy in a non-governmental public body. Evidence & Policy, 5, 85-98.

63.

Nutley

S. M.

Walter

Davies

H. T.

(2007). Using evidence: How research can inform public services. Bristol, UK: Policy Press.

64.

Oliver

Innvaer

Lorenc

Woodman

Thomas

(2014). A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Services Research, 14, Article 2.

65.

Orton

Lloyd-Williams

Taylor-Robinson

O’Flaherty

Capewell

(2011). The use of research evidence in public health decision making processes: Systematic review. PLoS ONE, 6(7), e21704.

66.

Ouimet

Bédard

P.-O.

Turgeon

Lavis

J. N.

Gélineau

Gagnon

Dallaire

(2010). Correlates of consulting research evidence among policy analysts in ministries: A cross sectional study. Evidence & Policy, 6, 433-460.

67.

Ouimet

Landry

Ziam

Bédard

P.-O.

(2009). The absorption of research knowledge by public civil servants. Evidence & Policy, 5, 331-350.

68.

Pearl

(2001). Direct and indirect effects. In Proceedings of the Seventeenth conference on Uncertainty in artificial intelligence (pp. 411-420). San Francisco, CA: Morgan Kaufmann.

69.

Preacher

Kelley

(2011). Effect size measures for mediation models: Quantitative strategies for communicating indirect effects. Psychological Methods, 16, 93-115.

70.

Reichardt

C. S.

(2002). The priority of just-identified, recursive models. Psychological Methods, 7, 307-315.

71.

Reiss

(2011). A plea for (good) simulations: Nudging economics toward an experimental science. Simulation & Gaming, 42, 243-264.

72.

Rich

R. F.

(1997). Measuring knowledge utilization: Processes and outcomes. Knowledge and Policy: The International Journal of Knowledge Transfer and Utilization, 10(3), 11-24.

73.

Rich

R. F.

C. H.

(2000). Rationality and use of information in policy decisions: A search for alternatives. Science Communication, 22, 173-211.

74.

Sanderson

(2009). Intelligent policy making for a complex world: Pragmatism, evidence and learning. Political Studies, 57, 699-719.

75.

StataCorp. (2011). Stata statistical software: Release 12. College Station, TX: Author.

76.

Sundell

Wärngård

Head

B. W.

(2013). How do government agencies use evidence? [Report]. Stockholm, Sweden: National Board of Health and Welfare.

77.

Thompson

G. N.

Estabrooks

C. A.

Degner

L. F.

(2006). Clarifying the concepts in knowledge transfer: A literature review. Journal of Advanced Nursing, 53, 691-701.

78.

Todorova

Durisin

(2007). Absorptive capacity: Valuing a reconceptualization. Academy of Management Review, 32, 774-786.

79.

Tomarken

A. J.

Waller

N. G.

(2003). Potential problems with “well fitting” models. Journal of Abnormal Psychology, 112, 578-598.

80.

Valeri

VanderWeele

T. J.

(2013). Mediation analysis allowing for exposure-mediator interactions and causal interpretation: Theoretical assumptions and implementation with SAS and SPSS macros. Psychological Methods, 18, 137-150.

81.

Waller

N. G.

Meehl

P. E.

(2002). Risky tests, verisimilitude, and path analysis. Psychological Methods, 7, 323-337.

82.

Weiss

C. H.

(1979). The many meanings of research utilization. Public Administration Review, 39, 426-431.

83.

Wellstead

A. M.

Stedman

R. C.

Howlett

(2011). Policy analytical capacity in changing governance contexts: A structural equation model (SEM) study of contemporary Canadian policy work. Public Policy and Administration, 26, 353-373.

84.

Wellstead

A. M.

Stedman

R. C.

Lindquist

E. A.

(2009). The nature of regional policy work in Canada’s federal public service. Canadian Political Science Review, 3, 34-56.

85.

Wellstead

Stedman

(2010). Policy capacity and incapacity in Canada’s federal government. Public Management Review, 12, 893-910.

86.

Wood

(2013). Making statistical methods more useful: Some suggestions from a case study. SAGE Open, 3(1). Retrieved from http://sgo.sagepub.com/content/3/1/2158244013476873

87.

Zahra

S. A.

George

(2002). Absorptive capacity: A review, reconceptualization, and extension. Academy of Management Review, 27, 185-203.

The Mobilization of Scientific Evidence by Public Policy Analysts

Abstract

Keywords

Introduction

Extant Literature

Knowledge Mobilization Theories and Assumptions

Absorptive Capacity Model and Hypotheses

Data and Method

Data and Variables

Analytical Strategy

Path Model

Predicted Probabilities

Results

Descriptive Statistics

Path Model

Predicted Probabilities

Discussion

Theoretical and Policy Implications

Implications for Researchers

Footnotes

Acknowledgements

Author’s Note

Declaration of Conflicting Interests

Funding

Notes

Author Biography

References