Abstract
In recent years, measuring devices and movements, such as the social investment perspective, have gained traction, allowing organisations to show the impact and social value of their work. A common characteristic is the persuasive rhetoric of how evaluation, quantification and measurement can improve human service organisations. The aim of this article is to unfold the argumentation for social investment fund (SIF) evaluation in human service organisations in three Swedish municipalities, exposing the logics according to which such proceedings are justified and legitimised in contemporary Swedish welfare state. The article draws on qualitative data from interviews with managers, evaluators and social investment staff, and evaluation reports from three Swedish municipalities. Using modality theory, according to which argumentation and reason are performed, findings reveal that SIF evaluation is promoted as a scientific construct with the possibility of producing certainty in the form of evidence regarding what works in a linear knowledge transfer model. Also, municipalities are portrayed as morally responsible for conducting SIF evaluations, emphasising that despite methodological challenges, pursuing such evaluations remains worthwhile. In this sense, SIF evaluation promotes an argument centred on two mutually reinforcing themes: scientific legitimacy and societal legitimacy.
Introduction
In recent years, several measuring devices and movements like social impact measurement (Rawhouser et al., 2019), social finance, impact investing (Chiapello and Knoll, 2020) and social return of investment (Corvo et al., 2022) have gained traction, allowing organisations to show the impact and social value of their work. Among these, the social investment perspective rests on policies that both invest in human capital development and help make efficient use of human capital while fostering greater social inclusion. Social policies are productive, essential to economic development and employment growth (Morel et al., 2012). Summative evaluations of financial and social outcomes are at the heart of these pursuits, are driven by accountability and transparency regarding how programmes are managed and shed light on the extent to which investments have made social impacts on complex social problems (Vo et al., 2016).
In Sweden, the social investment idea was translated into municipal social investment funds (SIFs) in the early 2000s. Formally, these were a response to deficiencies in service design, delivery and accountability and meant to address the need for innovative organisational solutions to proactively tackle entrenched social issues (cf. Jonsson and Johansson, 2018). Projects within these funds adopt a pre-emptive perspective on societal challenges, aiming to prevent social problems before they emerge. Evaluation is fundamental to SIF projects, demonstrating success and what ‘works’; i.e. reducing future costs. As such, there is a close relationship between evaluation results and costs which contributes to assessing social interventions based on their demonstrable payoff rather than other non-economic or non-measurable values. SIFs are therefore also related to the ideas of evidence-based policymaking and the ‘what works movement’ (Boaz et al., 2019). In fact, a recent research report on Nordic social investment projects argues that ‘the rise of social investment projects has nurtured and expanded an evaluation culture within the social policy area’ (Højbjerg Jacobsen and Schindler Rangvid, 2023: 8), resonating with an observed ‘output culture’ within the Swedish public sector (Fred and Mukhtar-Landgren, 2021).
A common characteristic of these interrelated ideas is the persuasive and seductive rhetoric regarding how evaluation, quantification and measurement can rescue human service organisations from the dark world of non-evidenced-based practices (cf. Ekeland et al., 2019; Jacobsson and Meeuwisse, 2020: 281). Such rhetoric can result in the use of symbolic evaluation, such as what occurs when the evaluation practices projected onto the organisational environment do not correspond to its operational practices (Boswell, 2012). Different sociological literatures have studied valuation and evaluation as fundamental social processes (Lamont, 2012) and their relationship to societal orders of justification (Boltanski and Thévenot, 2006), representing level-spanning institutional logics (Thornton et al., 2012). However, the legitimising qualities of evaluation arguments, particularly regarding social investment, have received less attention. This focus may clarify how SIF evaluation has promoted the measurement of costs associated with preventive welfare work and effective cost-reduction strategies (Nordesjö, 2021). Additionally, it highlights the flexibility of SIF evaluation in addressing different audiences (Nordesjö, 2024). A close examination of evaluation arguments may also enrich the literature on valuation and evaluation by using concepts of modality to explore how different arguments legitimize SIF evaluation.
The aim is to unfold the argumentation for SIF evaluation in human service organisations in three Swedish municipalities, exposing the logics according to which such proceedings are justified and legitimised in the contemporary Swedish welfare state. Investigating how local implementing actors argue for SIF evaluation can contribute to understanding the role of evaluation in social investment practices and how evaluation builds legitimacy through argumentation in human service organisations. Such organisations seek legitimacy by aligning themselves to often inconsistent and contradictory social norms and values (Hasenfeld, 2010). In this setting, evaluation may have a legitimising role for generating secure knowledge about so-called ‘wicked problems’, which are difficult to define, are constantly changing, involve actors with conflicting values and can harbour disagreement on what constitutes having solved the problem (see Rittel and Webber, 1973). The study draws on qualitative data from interviews with managers, evaluators and social investment staff, and evaluation reports in three Swedish municipalities, which have carried out social investment projects and evaluations. Argumentation is studied in the form of modal logics, according to which argumentation and reason are performed (Doležel, 1976, 1998).
Setting and methods
In Sweden, social investment funds are organised on a local level within one of the 290 municipalities. SALAR (The Swedish Association of Local Authorities and Regions) defines social investment as ‘a limited investment which, in relation to the usual working methods, is expected to give a better outcome for the target group and at the same time lead to reduced socio-economic costs in the long run’ (UPH, 2017: 6). These are innovative, temporary and collaborative projects, evidence- or knowledge-based, kept separate from day-to-day public administration, and possible to monitor and assess from a societal as well as a financial perspective (UPH, 2015). The overall theme has been to act today to lower costs tomorrow. Evaluation is integrated within the selection, implementation, calculation and decision-making of SIF interventions. Its role is to assess outcomes and costs resulting from evidence- or knowledge-based interventions and link them to various cost scenarios, thus serving as a foundation for decision-making for managers or politicians.
Interviews were carried out in three Swedish municipalities (pop. 70,000–150,000) who had carried out, and were currently working on, SIF projects. The funds were similarly organised in the municipalities where financial resources were allocated to a fund from which specialised departments (e.g. social services) could apply for resources for a project. Social investment staff process the applications and the municipality's elected politicians decide on what project to initiate. SIFs can also include a refund system that forces projects to refund an amount corresponding to the reduction of the social problem it has achieved. The idea is that since the social problem is reduced through social interventions, resources should be paid back to the fund that other projects can then apply for.
The three municipalities implemented similar evaluation arrangements for their projects. This may come from the fact that they all participated in a social investment network facilitated by SALAR. This consisted of a monitoring system comprised of indicators for activities and social and economic outcomes which tracks the development towards project goals. In addition, an outcome evaluation is carried out after the projects, preferably with control groups, but more often through quasi-experimental designs where results are compared to other similar target groups. Pre–post analyses are also used. Some evaluations are supplemented by qualitative assessments authored by involved actors. Formative evaluations may be carried out to supplement the outcome evaluations. Finally, cost calculations such as cost–benefit analyses are performed to assess the financial gains of the achieved outcomes. Evaluation reports draw on all these evaluation activities and provide information for decision-makers such as department managers and the municipal executive committee (Table 1).
Data collection.
Sixteen interviews were carried out as part of a multisite study of the implementation of social investment initiatives in Sweden during 2017–2021. Municipalities were approached through a SALAR network on social investment and interviewees were contacted. SI staff work at the central municipal level and facilitate SI fund projects in different departments, handle applications, design monitoring systems and may perform evaluations. Department managers are project owners and are financially responsible for the projects. Project managers work with the interventions. One external evaluator was interviewed. Interviews were semi structured along the two themes of governance through SI and evaluation of SI funds.
Municipality A had more interviewees due to its larger SIF organisation and staff. A's influence is balanced by focusing on common themes across all three municipalities, and ensuring each municipality is represented in the findings. Main interview themes focused on the governance and evaluation of SIF evaluation. Interviews were conducted in person and on telephone and lasted 45–120 minutes. They were recorded and transcribed verbatim. Information about the study and the interview was sent in advance, and verbal consent was received from all interviewees. Sixteen evaluations from completed projects in the three municipalities supplemented the interviews.
The interviews were analysed following Dolezel's (1976; 1998) modality theory. The underlying idea is that we structure reality of communication in a way that provides access to our judgments about and perspectives on said reality. Thus, we can access the logic according to which the study participants legitimise and justify social investment evaluation. According to modality theory, all sense-making comes about according to certain narrative setups and logical traits. When we reason in an alethic logic, we express what is possible and necessary; and when we reason in a deontic logic, we speak about what is permitted, forbidden, or obligatory. An axiological logic refers to what is morally good, bad, or indifferent, and an epistemic logic to what is known, unknown, or believed. Distinguishing between the different modal logics gives access to the specific justifications and rationales of social investment evaluation, assisting the analysis in discerning the sense-making of action; exposing, for example, the reasoning behind organisational proceedings. For example, are the arguments qualified by referring to an axiological logic with references to how social investments improve the lives of citizens, or to an epistemic logic in which effects brings us closer to the truth of ‘what works’? In this way, modal logic theory can show the meaning-based characteristics in a broader narrative of social investment and show general patterns in which the actors order the world when they speak about it.
While modal theory was developed in philosophy and literature studies, it is also used in the social sciences; for example when analysing the communication of knowledge and science to describe how identified logics of relevance and credibility express fundamental modal logics (see e.g. Hellman and Room, 2015). Similarly, the interviews and evaluations in this analysis are viewed as communication texts. Certainly, other established sociological lenses such as justification theory (Boltanski and Thévenot, 2006) and the institutional logics perspective (e.g. Thornton et al., 2012) relate valuation and evaluation to sources for legitimacy (Lamont, 2012). In relation to these literatures, modal theory does not identify the content in arguments such as certain sources, orders and categories of justification but draws attention to how basic logics of expressions qualify the truth of a judgement. This situates the modalities closer to the narrative structure of the observed arguments. It makes it possible to use the modalities as analytical tools to unfold arguments in a larger SIF evaluation theme regardless of their content and source of justification and legitimacy.
Consequently, after familiarisation with the data, the four logics were used to code the different arguments for SIF evaluation in interviews and evaluation. For example, arguments related to the potentials and possibilities of SIF evaluation were coded as alethic logic, while statements related to rules and requirements were coded as a deontic logic. In a second step, relationships between the logics were analysed to understand whether a certain logic regularly appeared together with another. Going back and forth between these two steps, two larger themes emerged. The first theme consisted of an alethic and epistemic logic (making certainty possible through SIF evaluation), while the other consisted of a deontic and axiological logic (being morally responsible and obliged to prevent social problems through SIF evaluation). In this way, the deductive coding process resulted in two core arguments, exposing the logics according to which SIF evaluation is justified and legitimised.
Results
The argumentation for SIF evaluation is presented in two themes.
Making certainty possible through SIF evaluation
The first theme represents the overall argument that SIF evaluation contributes to the possible realisation of secure knowledge regarding what social interventions work. The theme is based on two intertwined modal logics. The alethic logic is the first of these, in which we find the possibilities of SIF evaluation. Interviewees argue that SIF evaluation makes it possible to measure outcomes and in turn know which interventions work. There's a lot that is done without necessarily having an evidence basis. Instead, one has to try. But that's why it's all the more important…that's why monitoring and evaluation are crucial, so that we can at least form an understanding of what works and what doesn't. (SIF staff, B) We must be able to work with social investment in the same way as we work with other investments; we must be able to allocate funds to do something now that will have an impact in the longer term. (Manager, A)
The described argumentation can also be found in actual evaluations which are generally quasi-experimental outcome evaluations based on monitoring data. Most of these evaluations contain summaries where outcomes are described and related to costs, making it possible for decision-makers to act on research evidence: The project is not economically efficient, but in the long run, there are indications that positive municipal and socio-economic effects will arise, although not to the extent that the initiative can be economically justified. (Evaluation A1, 2017: 3)
Thus, an alethic logic is prominent in both interviews and evaluations and actors have faith in the possibilities of social investment. It is necessary to measure, observe and evaluate to gain knowledge and make sense of the world. In this sense, an alethic logic is intertwined with an epistemic logic, where actors’ arguments refer to what is known, unknown and believed about SIF evaluation. The intertwinement is due to the case of SIF evaluation itself being promoted by national and implementing actors as a scientific construct, and also understood by interviewees as such. It is therefore expected that interviewees’ argumentation would draw on epistemic logic to explain the possibilities of SIF evaluation: The systematics, the monitoring, the activities we have carried out. Even monitoring of activities has been done, the question is what it has led to, for whom. That is really the key question. (SI staff, A)
In broader terms, epistemic logic functions similarly to a linear knowledge transfer model, where the creation of the product of ‘evidence’ through evaluation and research is distributed to decision-makers for practical and immediate application (see Boaz and Nutley, 2019). As such, it embodies a scientific view of evaluation, grounded in the rationalistic belief that goals (politics and decision-making) and means (social science and evaluation) can be distinctly separated (Furubo, 2019). Some arguments show how such a linear knowledge transfer model also benefits social work professionals: ‘proving that social work pays off is very important for projects and social workers; demonstrating that it works both for others and for oneself (SI staff B)’. In this sense, evidence from SIF evaluation benefits social work professionals not by supporting their professional practice with practice-based knowledge, but rather by functioning as a legitimising and verifying source of knowledge which can give social workers confidence and certainty in interventions.
In sum, the intertwinement of an alethic and epistemic logic generates an argument in which evaluation as a scientific activity has the possibility to produce positive evidence on what works in a linear knowledge transfer model. This argumentation has trickled down to professionals, where a professional practice can benefit from knowing which interventions work. The argumentation is justified by giving insight into broader societal values for the target group and the economy.
Being morally responsible and obliged to prevent social problems through SIF evaluation
The second theme expresses how one is morally obligated to perform SIF evaluation to work towards certain values. Similar to the first theme, there is a relationship between two logics. First, axiological logic shows what is good, bad, or indifferent about SIF evaluation. What is evaluation's underlying theory of social change, and what are its criteria used to measure success (Shadish et al., 1991)? Interviewees’ answers broadly fit into the same argumentation. Knowing what interventions are effective can prevent social problems and social exclusion in the future: The main purpose of the fund is to reduce future social exclusion. It is to measure and try to see whether we achieve outcomes in these projects. That's the main purpose. (I: To investigate whether it has occurred or not?) Yes, and of course, you don't reduce social exclusion in three years, but you can see if it is heading in the right direction. Can we see outcomes in areas…people talk about linkage…moving backward…reduced social exclusion…what contributes to that when you're an adult, like having a job, for example…or being integrated into society…so the next step is to determine what is a measure of being closer to employment. Or being closer to being integrated into society. (SIF staff, B) The fundamental idea that we should work preventively is unavoidable. In Sweden, we are very good at ‘extinguising fires’ /…/ what does crime cost, what does it cost when children do not go to school, what does smoking cost, what does whatever cost…when we looked at it in 2013, you see that the lake is perhaps 10,000 times more than our municipal budgets, I think it was 550 billion (SEK) that municipalities had in their budget. It is impossible to estimate how much money we spend when it's already burning. And I have asked if we can allocate 0.5% of our 6.5 billion for prevention. (SIF staff, C)
Relating costs to outcomes and measurement accordingly makes SIF evaluation an instrument that can help prevent costly social problems. While SIF evaluation aspires to be morally good by preventing social problems, evaluation criteria are descriptive quantified outcomes that are tied to standardised costs, conveying impartiality and objectivity (Porter, 2003). SIF evaluation is therefore not only scientifically but also morally sound.
However, SIF evaluation is not only a moral act, but an obligation. Deontic logic refers to proscriptive or prescriptive norms for which actions are prohibited, obligatory, or permitted for a narrative to unfold. Ultimately, SIF evaluation is obligatory for knowing what is certain and how to act: But to know what needs to be changed, one must know what works and is effective. If you can't demonstrate actual results, you can't say what works. Therefore, we have standardised tests that cannot be questioned. Not like grades, where it's teachers’ assessments. (Project leader, B)
Deontic logic is also identified through formal and informal norms of how SIF evaluation is supposed to be carried out. Among interviewees, it is generally argued that the success of SIF evaluation is built on there being a monitoring system in place which can work as a baseline for outcome evaluation. However, there are very different ideas about what the requirements are in the three municipalities. The most extensive evaluation system can be found in A, where evaluation is integrated into all parts of project application and implementation. In B, ‘there should be measurements before starting the project to establish a baseline…and then regularly to track progress.’ (SI staff, B). In C, evaluation activities are less formalised and decided within each project. But regardless of municipality, both interviewees and evaluations emphasise the difficulties in carrying out evaluations that abides by the intended requirements of SIF evaluation. For example, interventions should be evidence-based as far as possible, although finding such interventions has been difficult. Instead, ‘proven experience is sufficient, or finding other municipalities and organisations that have done something that has proven to work in various ways’ (Manager, A). Also, measuring outcomes is associated with methodological difficulties in all municipalities such as a lack of data, and attribution issues. In A, the quantitative measures provided ‘have been difficult to interpret or have shown results that are challenging to directly link to the team's efforts’ (Evaluation A7, 2017: 20). In C, evaluators state that ‘we never managed to access the necessary data required to perform our socioeconomic calculations’ (Evaluation C1, 2018: 3). For these reasons, other forms of evaluation, such as formative qualitative evaluation in A, are also carried out to supplement SIF evaluation and gain other forms of knowledge. However, although SIF evaluation evidently has methodological challenges and may render ambiguous or invalid results in need of additional supporting evaluation, deontic logic stands strong. For example, SIF staff in B highlight all the municipal discussions on outcome evaluation and the work done on monitoring and argue that ‘in spite of all its (of SIF evaluation) weaknesses and problems, that's still what will endure’. The quote illustrates that SIF evaluation is too morally important to give up on and its requirements should be held intact to be able to show the value of preventive work.
In sum, the second theme expresses how municipalities are morally responsible for performing SIF evaluation. Although the obligations of SIF evaluation pose methodological challenges, formal and informal norms are still worth striving for. Taken together, the two logics are independently insufficient and are part of a two-sided argumentation where they support and lend legitimacy to each other.
Discussion
This article set out to unfold the argumentation of SIF evaluation in human service organisations in three Swedish municipalities. The study is limited in that few interviewees have been included, and that no politicians, professionals, or clients have been interviewed, which could have contributed to a more nuanced and extensive narrative. Findings consist of two themes in the argumentation on SIF evaluation. First, SIF evaluation is promoted as a scientific construct with the possibility of producing certainty in the form of evidence on what works in a linear knowledge transfer model. Second, municipalities are morally responsible for performing SIF evaluation and despite methodological challenges, formal and informal SIF norms are still worth striving for.
The use of modality theory has exposed how underlying logics are intertwined and are significant for the role of argumentation in evaluation. The argumentation in the first theme can be understood from a realist ontological standpoint aligned with modern expectations of evaluation as an objective and impartial truth-telling activity. In this way, it resonates with other contemporary ideas meant to improve human service organisations through measurement and evaluation. Closely related in terms of the concepts used is evidence-based policy and practice, whose rhetoric has been described as ‘seductive’ and as promoting cookbook practices and ignoring the complexities of social work (Ekeland et al., 2019). Indeed, just as the ‘brilliant danger of an ‘evidence-based’ slogan’ (Tomkins and Bristow, 2023), the logic of SIF evaluation appears obvious and directed at common sense. Who does not want certain evidence on what works rather than ‘gut feeling’ to guide our decisions (Barfoed and Jacobsson, 2012)? The SIF evaluation perspective therefore ties itself to a EBP discourse, which in Sweden has been implemented as a state-governed top-down model of knowledge utilisation (Jacobsson and Meeuwisse, 2020). This is also supported by the absence of practice-based knowledge in SIF evaluation, and the view that SIF evaluation benefits social work professionals primarily in a legitimising and verifying way. This is paradoxical, since the original conception of evidence-based practice is built also on professional practice-based knowledge and clients’ experiences (see e.g. Bergmark and Lundström, 2006).
While the first theme represents an instrumental idea of how evaluation can produce useful results, the second theme reveals a societal purpose of SIF evaluation and can be understood as legitimising the first theme. Thus, SIF evaluation resonates with the broad spectrum of social impact measurement approaches and a ‘caring’ capitalism (Barman, 2016); both in terms of accountability, where the aim is to investigate if investments have contributed to social impact, and to contribute to decision-making, but also in terms of the ambition to address complex social issues and problems by studying impact (cf. Vo et al., 2016). Municipalities in this study have argued that they are morally responsible for performing SIF evaluation, since it can contribute to solving contemporary social problems in a cost-efficient manner. This is important in human service organisations, which constantly need to seek legitimacy and source resources from their surroundings by aligning themselves to often inconsistent and contradictory societal norms and values. Additionally, these organisations are inherently moral, where work is also driven by moral values of social and societal improvement (Hasenfeld, 2010). The value of the second theme becomes evident considering the methodological challenges reported in SIF evaluation and a way of reinforcing and connecting additional values to an assumed value-neutral evaluation logic. The argumentation of the first theme is not enough to legitimise the use of SIF evaluation in the municipalities by itself. Referring to social and societal values also give stakeholders rhetorical resources when discussing cost-efficient interventions by showing the (economical) value of preventive social work.
In conclusion, this article demonstrates how SIF evaluation promotes an argument based on two mutually reinforcing themes. The first theme relies on the second for societal legitimacy, while the second theme depends on the first for scientific legitimacy. While this investigation has illuminated the argumentative aspects of evaluation; future research could turn to other literatures, such as the institutional logics perspective (Thornton et al., 2012), to examine how these arguments span organisational levels and relate to different sources of legitimacy. Studying the legitimation process among social workers involved in measurement practices could provide insights into whether the argumentation is primarily a managerial and administrative phenomenon or if it is enacted at a professional level. This would reveal tensions and implications of the social investment argumentation for social work professionalism and client work.
Footnotes
Acknowledgements
The author would like to thank the social investment research group for valuable comments on early drafts. A special thanks to Matilda Hellman for discussions on argumentation.
Funding
The author disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Swedish Research Council for Health, Working Life and Welfare under Grant 2017-02151.
