Street-level bureaucrats’ attitudes towards clients in discretionary decision-making: Evidence from the Norwegian labour and welfare administration

Abstract

It is commonly assumed that the attitudes of street level bureaucrats (SLBs) towards benefit recipients influence the discretionary decisions they make. Although discretion in the policy process has been a contentious issue, the impact of bureaucrats’ attitudes has received limited scholarly attention. This study investigates the impact of attitudes on discretionary decision-making and whether the impact of attitudes is conditioned by the SLBs’ perception of recipients’ conduct, when sanctioning non-compliance of requirements among social assistance recipients. The study employs survey data from 925 bureaucrats within the Norwegian Labour and Welfare Administration, measuring four attitude components: cognitive, positive affective, negative affective, and behavioral attitudes. A vignette experiment allows assessing how sensitive discretionary decision-making is to SLBs’ perceptions of recipient conduct. The findings indicate that cognitive attitudes significantly impact discretionary decision-making and are sensitive to SLBs’ perceptions of the recipients’ personal responsibility. Prejudiced cognitive attitudes (viewing recipients as dishonest, manipulative, unpredictable, or stubborn) are associated with a greater tendency to attribute non-compliance to a lack of willingness, irrespective of external circumstances. The study contributes to a deeper understanding of discretionary decision-making and sheds light on the complex dynamics between SLBs’ attitudes and their perceptions of client attributes, offering new insight for public policy and administration.

Keywords

Behavioural public administration decisionmaking expertise public administration professionalism/professions policy implementation

Introduction

Street-level bureaucrats (SLBs) are public employees who interact directly with citizens and have substantial discretion in the execution of their work. The combination of a wide discretionary space and relative autonomy from organisational authority implies that they form public policy, and that their individual actions add up to agency behaviour (Lipsky, 2010: 13). This study investigates whether the attitudes of SLBs influence their use of discretion, which might have a wide-reaching impact on the citizens’ lives (Maynard-Moody and Musheno, 2003).

The extant scholarship has shown that the SLBs’ attitudes are a key part of understanding bureaucratic decision-making (Baviskar and Winter, 2017; Oberfield, 2014). However, the impact of attitudes may be shaped and mitigated by existing structural conditions, such as workload and resources (Lipsky, 2010). Multiple scholars have emphasized the influences of these structural determinants on SLBs’ behaviour and discretionary decision-making (Brodkin, 2011; Soss et al., 2011; Van Berkel et al., 2017). This perspective emphasizes that various contexts, such as the policy context, governance context, and organizational contexts, are structuring street-level practice. The constraints imposed by these contexts may matter, but how, or whether they fully mitigate the influence of attitudes in the SLB’s discretionary decisions, is an empirical question.

Laws, regulations, and circulars are often ambiguous and need to be interpreted by SLBs’. Since individuals may interpret both laws and structural conditions differently, this might lead to various responses and behaviours. Previous research has also pointed out that SLBs’ tend to categorize benefit recipients as, for example: “worthy,” “deserving,” “motivated” and “competent” (Guul et al., 2021; Maynard-Moody and Musheno, 2003). This may mean that the daily interactions with clients lead SLBs to evaluate the clients’ behaviour in various ways.

While the role of attitudes in decision-making has benefitted from considerable scholarly attention, the conceptualization, and implications of SLBs’ attitudes towards clients has received limited attention. Attitudes are understood as a mindset or a tendency to behave in a certain manner, due to the individual’s experiences and temperament (Pickens, 2005). We employ an instrument, developed for measuring SLBs’ attitudes toward clients, which includes a cognitive component, two affective components, and a behavioural attitude component (Keulemans and Van de Walle, 2018). As this instrument is developed for Dutch tax bureaucrats, we first examine its validity for SLBs in the welfare administration. By doing this we also test the researchers’ ambition to have developed an instrument that is more generally able to catch SLBs’ attitudes towards clients.

The knowledge of discretionary decisions-making in the frontline of the welfare state, is generally deficient (Hupe and Hill, 2016). The “activation turn” has increased the space of discretion in the implementation of welfare policy, which so far has been poorly explored (Brodkin, 2013; Van Berkel, 2020). The trend towards increased use of activity requirements for recipients makes it important to study SLBs’ assessments and decisions regarding those who do not comply with their obligations (Van Berkel et al., 2017; Watts and Fitzpatrick, 2018).

This study adds to existing knowledge about discretionary decision-making in two ways: First, we assess the role of SLBs’ agency and attitudes while accounting for important structural factors. To achieve this empirically, we employ strict office level fixed-effect estimations which allow us to focus on within-office variation. Second, we move beyond merely measuring and describing how attitudes towards clients vary between different SLBs to also examine the importance of such attitudinal differences for discretionary decision-making. Using a survey experiment, we examine how SLBs’ attitudes interact with their categorization of clients when they make decisions. We also explore whether the significance of attitudes towards recipients seems to be moderated by the SLBs’ perception of recipients’ conduct when sanctioning is imposed in the Norwegian Labour and Welfare Administration (NLWA).

The paper is structured as follows. In the next section, we discuss discretionary decision-making and the role of attitudes toward clients. Then some core characteristics of the legal context of the Norwegian social assistance scheme are presented before we introduce the data and research design. We then present the results of the analysis of survey data, containing a vignette experiment, from 925 SLBs working in 107 frontline NLWA offices. The bureaucrats were asked whether they would sanction this fictive recipient. We conclude with a discussion of the findings and their implication for theory and practice.

Discretionary decision-making and the role of SLB’s attitudes

Discretionary decision-making

SLBs have some freedom to exercise professional judgement delegated by a higher public authority. The discretionary space is surrounded by standards and legal rules and depends on a cognitive activity carried out by actors. SLBs should act in accordance with their best judgement supported by good reasons, within the boundaries set by legal rules and standards (Molander et al., 2012). Scrutinizing how this freedom is used by SLBs is important in policy analysis (Buffat et al., 2015).

However, SLBs are not only “state agents.” As “mediators” between the state and the street they interact face-to-face with citizens (Zacka, 2017) and are expected to be both “state agents” and “citizen agents” (Maynard-Moody and Musheno, 2003). As state agents, SLBs’ behaviour is guided by laws and regulations, while as “citizen agents” their perception of recipients’ conduct and deservingness are important for the decisions they make. SLBs are expected be able to interpret political signals, laws, and regulations while being sensitive to citizens’ needs, behaviours, and situations.

Additionally, SLBs are actors with certain educational backgrounds, gender and preferences, which may shape their attitudes. Given that their decisions involve uncertainty, both regarding the motives of users, the legitimacy of these motives, and in some cases the interpretation of the law, the SLBs’ attitudes may be of particular importance. Attitudes influence how we process information, and specifically SLB’s attitudes towards clients may bias public service delivery (Keiser, 2010; Maio et al., 2018). It is often argued that the attitudes bureaucrats have inform the course of action that they take (Maynard-Moody and Musheno, 2003).

Conditionality expands the discretionary space

The overall aim of activation policies is to increase the employability of persons outside of the labour market. More emphasis is placed on linking mandatory activity requirements to the receipt of benefits, which should help to qualify the recipient for paid work. Behavioral conditions require clients to participate in activation measures such as job-search activities or training. When clients fail to comply, the benefit might be reduced or withdrawn (Watts and Fitzpatrick, 2018). The recognition that recipients meet the necessary requirements are an intrinsic part of the decisions made by SLBs. This extends the discretionary space where bureaucrats must navigate both the legislation, the behaviors of recipients and the reasons for non-compliance.

However, the policy context, governance context, and organizational contexts, may limit the discretionary space and standardize street-level practice. A core issue in the study of behavioral conditionality has been to what degree imposing sanctions can be perceived as “organized practice” (Caswell and Høybye-Mortensen, 2015; Soss et al., 2011). The argument has been that sanctioning must be understood as more than a response to the clients’ behaviors and characteristics, and more than an individual action taken by a SLB. Due to the constraints imposed by the institutional context, the SLB cannot treat clients as they wish. The discretionary space is influenced in a relatively uniform manner by the pressure from management, working systems characterized by low funding, performance management, close monitoring (Brodkin, 2011). Standardizing conditions like these, imply that the bureaucrats’ attitudes to a limited degree would influence the decisions (McGann et al., 2019).

We recognize that structural conditions constrain SLBs’ discretionary decisions, and do not take a position on which conditions best explain the variations in discretionary decisions. Rather, the aim is to study the role of individual agency in discretionary decision-making. For this reason, the intention is to keep these conditions as constant as possible, to examine whether the SLBs’ attitudes towards clients too, may have significance for their propensity to sanction non-compliance of activity requirements.

SLBs’ attitudes and discretionary decision-making

Attitudes are understood as mindsets or tendencies to behave in certain manners, due to individuals’ experiences and temperaments. It is widely acknowledged that attitudes consist of three components: affect, cognition, and behavior (Pickens, 2005). Attitudinal models of decision-making suggest that attitudes may play a crucial role in decision making. Such attitudinal models have a long-standing tradition in studies of judicial decision-making (Segal et al., 2017; Segal and Spaeth, 2002), yet remain underexplored in social policy research. Although different attitudinal models exist and some focus on whether and how attitudes are moderated by ideology (Epstein and Knight, 2013), the core addition to the structural perspectives discussed above, is the assumption that attitudes may influence decision-making processes also after accounting for structural, or organizational factors (Lax, 2007).

The core argument is that attitudes influence how SLBs process information. It is relatively well-established that the attitudes of bureaucrats inform the course of action they take (Maynard-Moody and Musheno, 2003); that bureaucrats can differ in the categorizations they employ (Keulemans and Van de Walle, 2018); and that they are able to switch between these attitudes (dispositions) on a case-to-case basis and even within a case (Zacka, 2017). Attitudes may additionally influence the bureaucrat’s sensitivity to the information presented by benefit recipients (Zacka, 2017) and their decision-making. For example, a client perceived as in need may trigger the bureaucrat to put in an extra effort in favor of the client (Kroeger, 1975; Maynard-Moody and Musheno, 2003). Moreover, the bureaucrats’ reactions are dependent on their assessment of the information presented by the clients (Keulemans and Van de Walle, 2018; Raaphorst and Van de Walle, 2018). Hence, the first empirical expectation is: SLBs’ attitudes are systematically related to their propensity to sanction non-compliant recipients, even when accounting for contextual factors.

However, within psychology, decision-making has mostly been regarded as a form of cognition (Eiser and Van der Pligt, 2015). As we are employing a model which differentiates between cognitive, affective, and behavioral attitudes, we expect that it is primarily the cognitive attitudes component that is systematically linked to the SLB’s willingness to sanction. The cognitive attitudes dimension refers to the beliefs and attributes that bureaucrats associate with benefit recipients (e.g. “they are dishonest”), as opposed to the affective attitudes bureaucrats feel when interacting which recipients (e.g. “they make me feel upset”) or how they interact with the recipients (e.g. “I help recipients”). As people are more likely to recall information from memory that is congruent with their current feelings (Isen et al., 1978), we expect that bureaucrats who view as recipient as manipulative, or dishonest are more likely to sanction non-compliant recipients.

Interactions between SLBs’ attitudes and perceptions of recipients

However, not all information brought forth by clients may trigger the same attitudes or responses on behalf of the bureaucrats. Some facts may be inconsequential, while other types of facts brought forth by the clients may impress some, but not all bureaucrats (Goodsell, 1981; Kroeger, 1975; Zacka, 2017). Hence, an underlying implication of the studies in the previous sections is that only some types of attitudes may be consequential to the decision-making process. This point is illustrated by several existing studies. These studies show how SLBs experience encounters with clients and how certain information provided by the clients may take precedence in their decisions, as illustrated by the meticulous work of Maynard-Moody and Musheno (2003). The seminal studies of Kroeger (1975) and Stone (1981) discuss how bureaucrats either use their discretionary space to the benefit of the recipient, when they are positive towards the recipient, and how they are more likely to be punitive when they have a negative attitude towards the recipient.

In social cognition theory, the concept of heuristics has been understood as a combination of fast information-processing mode and a frugal decision rule, where judgements follow from a limited set of cues. Heuristics can be interpreted as “judgmental shortcuts,” that allow people to form opinion in the absence of substantive knowledge. In the process of determining what case or recipient one is facing, mental shortcuts seem to be an important element (Kahneman, 2017). The importance of heuristics in political attitude formation has been shown, particularly in how deservingness heuristics strongly influence judgements about help giving (Aarøe and Petersen, 2014; Oorschot, 2000; Petersen et al., 2011). A recipient perceived as in need may trigger the SLB to put in an extra effort in favor of the recipient (Kroeger, 1975; Maynard-Moody and Musheno, 2003).

To investigate the role of the interaction between SLBs’ attitudes and their perception of recipients for discretionary decision-making, is demanding. A recent survey experiment has demonstrated that SLBs’ propensity to sanctioning recipients is extremely sensitive to their perception of personal responsibility. When non-compliance with activity requirements is perceived as an expression of lack of will (choice), compared to lack of ability (circumstances), the propensity to sanction more than doubles (Torsvik et al., 2022).

In turn, these assessments of individual clients may fuel and inform their overall attitude towards all recipients they meet (Oberfield, 2014). While considerable attention has been devoted to how SLBs categorize benefit recipients during decision making processes, the interaction between this categorization and attitudes in decision-making, have received less attention. We intend to examine whether more general cognitive skepticism towards social assistance recipients to the same extent leads to sanctioning when they meet an “will-not recipient” compared to an “cannot-recipient.” The second empirical expectation is: The SLBs’ attitudes are sensitive to their perception of the recipients’ personal responsibility when sanctioning non-compliance of activity requirements.

Social assistance in Norway

Social assistance is a means-tested financial support that covers basic subsistence costs for those who are unable to provide for themselves or have exhausted the rights to unemployment benefits (Gough et al., 1997). In Norway, social assistance is regulated in the “Social Services Act” (SSA). Since the early 1990s the municipalities have had the opportunity to make social assistance conditional on participation in activation programs (cf. SSA of 1991, § 20). In 2017, when amendment §20a was added, municipalities were instructed to implement mandatory activation for social assistance claimants under 30 years of age.

However, conditionality is mitigated by the clause: ‘unless weighty reasons speak against it’ (SSA of 1991, §20a). The government circular for this act emphasizes that activity requirements must not be ‘disproportionately burdensome for the recipient or limit his or her freedom of action or choice in an unreasonable way’ (NAV, 2018). If the recipient does not comply with the activity requirement, bureaucrats can impose a sanction in the form of a reduction in the benefits. However, SLBs are instructed to first assess whether the recipient’s non-compliance was caused by a lack of ability or will.

Mitigation clauses might narrow or widen the discretionary space (Eleveld, 2017). Discretionary clauses mean that SLBs “can,” but do not have to enforce a sanction, while “good reason clauses” refers to whether the recipient has good reason for non-compliance. Compared to other countries in Europe, the Norwegian Social Services Act has more mitigation clauses (Eleveld, 2017), which instruct SLBs to consider many aspects of the non-compliance and therefore contribute to expand the discretionary space.

This means that SLBs in Norway are legally expected to make assessments both on appropriate activity requirements and whether non-compliance should be sanctioned. Previous studies from Norway indicated that managers and bureaucrats use their discretionary power in the implementation of conditionality and sanctions (Gjersøe et al., 2020; Sadeghi and Terum, 2020).

Data and estimation strategy

We employ survey data collected from SLBs in the NLWA. The questionnaire was completed by 1646 employees (a 51% response rate) from 107 frontline offices in 2018. The questionnaire was web-based and included items on attitudes toward clients, bureaucrats’ educational background, gender, workload, work experience in NLWA, and vignettes describing non-compliant social assistance recipients. The vignettes were only answered by those who handle social assistance cases on a regular basis, N = 920, which is the sample for this study. This ensures the SLB’s familiarity with the specific legislative frameworks. To avoid bias and priming effects due to previous questions, the vignettes were placed in the first part of the questionnaire. After having read the randomly assigned vignette, which vary the characteristics of the recipients and the degree of non-compliance (randomization test supplemental table: A3), the SLB are asked to decide on whether they would sanction the specific fictive recipients and whether they consider that reasons for non-compliance the client’s personal responsibility.

Attitudes towards clients

The general attitudes are captured using the instruments developed by Keulemans and Van de Walle (2018) for tax bureaucrats in the Netherlands. It consist of three theoretical components. Firstly, a cognitive attitudes component capturing beliefs and attributes that bureaucrats associate with clients. Secondly, two affective attitude components: the positive and negative affective attitude components capturing the general evaluation of recipients and the bureaucrat’s emotional responses towards recipients. Lastly, the behavioral attitude component capturing the past behaviors of the SLB, including their fairness and overall approach towards clients. Within this operationalization, attitudes are defined as: “their summary evaluation of clients along a dimension ranging from positive to negative that is based on the bureaucrats’ cognitive, affective and behavioural information on clients” (Keulemans and Van de Walle, 2018: 88). This operationalization is in line with the one employed by the present study.

As the context of SLB’s in the NLWA is different for that of tax bureaucrats, we first replicate their multi-component model using the same indicator battery and software, on the full sample of the survey (N = 1042). The Norwegian questionnaire included a translation of the parsimonious factor solution presented by Keulemans and Van de Walle (2018: 96), however taxpayers was changed to “users” to better reflect the context in which NLWA SLB’s work in. All items are measured on a seven-point Likert scales ranging from ‘never’ (0) to ‘always’ (6).

In terms of interpreting the results, high scores on either the negative affective attitudes or cognitive attitudes are interpreted as indicative of an overall negative attitude towards clients, while high scores on positive affective or behavioral attitudes are considered indicative of generally positive attitudes towards clients. This study elaborates on Keulemans and Van de Walle (2018) by exploring the significance of attitudes for discretionary decision-making using vignettes.

Vignettes

The main dependent variable in this study is whether the bureaucrats sanction recipients, who for some various reasons, are non-compliant. Each of the texts has been piloted and validated to reflect real cases as closely as possible and was developed in cooperation with experienced SLBs in the NLWA. The vignette text was then standardized, with the only thing varying being the explanation for why Daniel did not participate in the program.

“Daniel Eriksen is 21 years old and has not completed secondary school. He often stays up late at night playing games. Daniel says it’s hard to find interesting jobs that are not too far away. He has only applied for a few jobs and has never been called in for an interview. A month ago, he applied for social assistance, which he was approved for on the condition of participating in a job search course. The first two weeks he attended eight of 10 classes on time, and the other days he did not arrive until 11 o’clock (AM).

A) Daniel has contacted you and expressed that the course is boring and that he is not learning anything new. He therefore thinks it is a waste of time.

Today, you were notified that Daniel has arrived late too many times in the past week and that he has stopped attending at all on Wednesdays.”

In this “will-not vignette” (A), the text in italics indicates that Daniel found the program boring and futile. The aim was to communicate that Daniel was is in control of his decision to not participate.

In the “cannot vignette” (B), the text was the same, apart from the explanation for non-compliance, which was:

B) Daniel has contacted you and says he finds it hard to attend class because he is afraid to be around people he does not know. Problems associated with social anxiety were also the subject of an earlier conversation you had with him.

To validate whether the SLB’s perceived the vignettes as described above, we replicated the analyses from Torsvik et al. (2022). In other words, bureaucrats rated the vignette they received on a 7-point Likert scale ranging from ‘lack of will’ to ‘lack of capacity.’ The analyses (available upon request) indicate that SLB’s who read vignettes (a) (program as boring and futile) were more likely to attribute non-compliance to a lack of will. Conversely, those who read vignettes (b) (social anxiety) mainly categorized his non-compliance as a lack of ability. These results confirm the results presented by Torsvik et al. (2022) (supplemental table A4).

Control variables

The regression analyses control for gender (female = 0, male = 1), completed higher education in the field of social work (coded 1, 0 otherwise), number of users they are responsible for (continuous), and whether they have been employed in the NWLA for more than 5 years (coded 1, 0 otherwise). These variables may influence the bureaucrats’ views and ways they apply extant regulations (Assadi and Lundin, 2018).

Estimation strategy

To explore the relationship between the SLBs’ attitudes towards recipients and sanctioning decisions, we estimate hierarchical linear probability models and present the coefficients, bordered by their 95% confidence interval, as the willingness to sanction is a dichotomous variable. Standard errors are clustered at the office-level, which aids in accounting for unobserved heterogeneity between offices and therefore corrects for the possible confounding effects of office size, culture, and office-level effects. To ease interpretation the results from the coefficients of interest are presented in figures (regression tables supplemental A1 and A2). Although it is not possible to randomly vary the SLBs’ attitudes, our estimation strategy allows us to identify the degree to which attitudes are influenced by potential non-compliance and trigger sanctioning. One of the advantages of using nearly identical fictive recipients with randomly varied characteristics is that avoid any selection issues due to either: that bureaucrats most often work with a certain type of clients, or that certain types of clients are allocated to some bureaucrats. This is especially important, as one of the main arguments in the attitude’s literature, is that attitudes are formed also based on previous encounters yet have a general effect.

To account for differences between offices and within-office variation between SLB, we estimate two different types of hierarchical models. We use office-level fixed effects models, which control for all differences - whether they’re observed or not, as long as they stay constant within offices. These models allow us to better isolate the role of SLBs’ attitudes on sanctioning.

Results

The results first present the replication of the multicomponent measurements of attitudes towards clients in a Norwegian context. Thereafter, the results of (a) whether bureaucrats’ attitudes are systematically related to their propensity to sanction non-compliant recipients, and (b) the interaction between SLBs’ attitudes and the perception of the recipients’ personal responsibility, when sanctioning non-compliance of activity requirements.

Replication of the measurement instruments

One of the aims of this study was to replicate the attitude model developed and tested by Keulemans and Van de Walle (2018). Based on the psychological multicomponent model of attitude, they developed a measurement instrument for assessing SLBs’ attitudes towards clients. The instrument was tested and validated through two surveys conducted among Dutch street-level tax bureaucrats. Their study identified four distinct components, measured by 17 items (table 1): the cognitive attitude, the positive affective attitude, the negative affective attitude, and the behavioral attitude component. The instrument in this validation study demonstrated both factorial and construct validity (Keulemans and Van de Walle, 2018).

Table 1.

Summary of exploratory factor analysis results.

	Rotated factor loadings (N = 1042)
	Factor 1	Factor 2	Factor 3	Factor 4
Cognitive attitudes
Clients are manipulative	0.65
Clients are dishonest	0.62
Clients are hostile	0.48
Clients are unpredictable	0.56
Clients are stubborn	0.57
Positive affective attitudes
Clients make me feel inspired		0.73
Clients make me feel active		0.72
Clients make me feel determined*		0.16
Clients make me feel alert*		0.39
Negative affective
Clients make me feel upset			0.68
Clients make me feel afraid			0.66
Clients make me feel nervous			0.73
Clients make me feel insecure			0.59
Clients make me feel uncomfortable*			0.28
Behavioral attitudes
I explain things to clients				0.71
I make clients feel at ease				0.56
I help clients				0.70
Eigenvalues	2.23	1.53	2.40	1.68
% variance	16.05	8.54	16.05	11.22
Cronbach’s α	0.74	0.77	0.78	0.76

Note: Items marked with * are excluded from the index calculations.

Following Keulemans and Van de Walle (2018), we used principal axis factoring with oblique rotation (direct oblimin) as the extraction method. Table 1 shows the results. The overall results indicate that a four-factor solution appropriately describes the data, which is in line with their findings. Three items were excluded: “make me feel alert,” “make me feel determined,” and “make me feel uncomfortable.” This decision was made for the following reasons: they had cross-loadings greater than 0.3, factor loadings of 0.4 or under and had low communalities. The grounds for item exclusion are in line with those presented by Keulemans and Van de Walle (2018) and DeVellis (2016). The parsimonious factor solution differs for two of the attitude components: positive affective attitudes and negative affective attitudes, from the parsimonious solution presented by Keulemans and Van de Walle (2018).

We then calculated additive indexes, where item scores added and subsequently averaged for each respondent. Table 2 presents mean, standard deviation minimum and maximum values. The correlations between the four attitude scales replicate those found by Keulemans and Van de Walle (2018) to a large extent. As shown in Table 3, the subscales are distinctive (i.e. no multicollinearity). We note that except for behavioral and cognitive attitudes, all other correlations reach statistical significance. The correlations between the positive and negative attitude subscales are significant. These two subscales cannot be fully described as unipolar in the sample used in this study, marking a difference from Keulemans and Van de Walle’s (2018) study. The magnitude of the correlation is low (0.1) and therefore of limited substantial importance. The Chronbach’s α scores were respectable for all the subscales and range between 0.74 and 0.78 (DeVellis, 2016), which indicated that no items could be removed to increase the reliability of the scales.

Table 2.

Descriptive measures of indices calculated for the four attitude subscales.

	Mean	Std.Dev.	Min.	Max
Cognitive	1.75	0.63	0	4.8
Positive affective	3.29	1.1	0	6
Negative affective	0.95	0.49	0	4.25
Behavioral	4.85	0.78	2.3	6

Table 3.

Intercorrelations between the attitude subscales. Pearson correlation (2-tailed sig. test).

	Cognitive	Positive affective	Negative affective	Behavioral
Cognitive	1
Positive affective	−0.130	1
Positive affective	0.000
Negative affective	0.371	−0.119	1
Negative affective	0.000	0.001
Behavioral	−0.057	0.329	−0.175	1
Behavioral	0.060	0.000	0.000

Additional tests were performed to assess the validity of these results. The hypothesized structure presented in Table 1 was tested with confirmatory factor analysis. The results indicate that while there is not a perfect fit between the model and the data (χ² = 241.53, df = 71, p = .000), the goodness-of-fit indices such as the CMIN/DF of 3.4 and the GFI of 0.96 were considered satisfactory. The RMSEA of 0.045 also indicates a reasonable fit of the model (for further details see DeVellis, 2016). The overall goodness-of-fit is similar to that presented by Keulemans and Van de Walle (2018). In sum, these results show that this instrument is also replicable in the case of NLWA.

Descriptive statistics

Table 4 shows the descriptive statistics of the pooled sample, for the cases where the fictive clients will not and cannot comply with the requirements for social assistance. Overall, 80% of the SLBs sanction clients who will not comply, yet only 37% of the SLBs sanction clients who cannot comply. The 43% difference between these will not and cannot comply is statistically significant (p < .001). The scores for each of the general attitude components are the same between SLBs who read the will not and cannot vignettes. Yet, we have a very considerable share of missingness on these variables (between 27 and 29%), which is due to SLBs not answering on these items. While individual variables do not have a high share of missing values on individual items, collectively they do. This is probably due to the idiosyncratic occurrence of respondents answering one variable but not the other, and vice versa. The 14-items covering the attitudes instrument were placed in the latter part of the questionnaire and we cannot rule out attrition due to questionnaire length. We also re-estimated the models withing the multiple imputation procedure from Stata17, under the missing at random assumption which yielded substantially equivalent results (available upon request).

Table 4.

Descriptive statistics for main variables for the pooled sample and by reason for non-compliance.

	Overall, N = 925^a	Will not, N = 466^a	Cannot, N = 459^a	p-value^b of difference between “will not” and “cannot”
Sanctions	59%	80%	37%	<.001
Cognitive attitudes	1.82 (0.64)	1.81 (0.59)	1.83 (0.68)	0.8
(Missing)	271	136	135
Positive affective	3.31 (1.12)	3.30 (1.07)	3.31 (1.16)	0.8
(Missing)	252	127	125
Negative affective	0.95 (0.47)	1.01 (0.47)	0.98 (0.47)	0.4
(Missing)	251	126	125
Behavioral attitudes	4.83 (0.77)	4.78 (0.78)	4.89 (0.75)	0.072
(Missing)	251	127	124
Social work education	35%	33%	36%	0.4

^a%; Mean (SD).

^bPearson’s Chi-squared test; Wilcoxon rank sum test.

Regression results

Attitudes and sanctioning

Figure 1 shows that respondent who are deemed as unable to (cannot) comply with the requirements are less likely to be sanctioned. In the vignette where Daniel has social anxiety the probability of sanctioning is considerably lower (around 42% lower likelihood of sanctioning in cannot cases compared to will not cases). This finding is in line with the findings presented by Torsvik et al. (2022).

Figure 1.

LPM results on the clients’ reasons for non-compliance and the caseworkers’ attitudes with office level FE and full set of controls on willingness to sanction across all vignettes. 95% CI.

These results also show that only the cognitive attitudes component is significantly related to sanctioning. In other words, neither positive affective, negative affective, nor behavioral attitudes are related to the caseworkers’ general willingness to sanction non-compliant social assistance clients. Higher scores on the cognitive attitude component (who answer that users are manipulative, dishonest, unpredictable, stubborn), are linked with a higher probably of sanctioning across the pooled cases.

We note that the associations been gender, number of users, years of experience in NAV and social work background are not significantly linked with sanctioning. Although we use slightly different operationalizations and regression estimators than Torsvik et al. (2022: 85), this finding replicates their results and is in line with their conclusions.

These findings offer some support to the first empirical expectation, which stipulated a systematic link between attitudes towards clients and sanctioning, also when accounting for structural factors. However, the findings indicate that solely the SLBs’ cognitive attitudes component is systematically related to their propensity to sanction non-compliant recipients, even when we account for both individual (gender, education, caseload) and office-level fixed effects. Adding the fixed effect at office level increase the explained variance (R²) in the models from 22.3% to 38.3%, highlighting the importance of accounting for structural factors.

Attitudes, perceptions, and sanctioning

Our second empirical expectation stipulates that there is an interaction between the SLBs’ cognitive attitudes and their perception of the recipients’ personal responsibility, here proxied by the reason for non-compliance (will not/cannot). We, therefore, include an interaction term to account for this.

The interaction is presented in Figure 2, which show that the SLB’s cognitive attitudes (users are manipulative, dishonest, unpredictable, stubborn) are impacted differently when they read a will not vignette, compared to when they read a cannot vignette. As in the previous model, the chance of sanctioning a will not vignette is generally higher than for a cannot vignette. In Table 4, we have shown that almost everyone (80%) who received the will not vignette sanctioned, while in the cannot vignette only 37% sanctioned. For the last vignette this may imply that there was greater scope for the bureaucrats’ attitudes to influence the decision.

Figure 2.

Interaction between cognitive attitudes and reason for non-compliance.

Cognitive attitudes are correlated with the likelihood of sanctioning cannot fictive recipients, that is recipients who indicate that they have social anxiety. For instance, bureaucrats who score 1.75 (average score in the pooled sample) on cognitive attitudes have a predicted probability of sanctioning cannot cases of ∼36%, while the predicted probability for sanctioning such cases for bureaucrats who score 2.4 (one standard deviation above the mean) on the cognitive attitudes scale is 47%. Hence, an increase of one standard deviation of the cognitive attitude variable is linked with an ∼11% increase in predicted probability of sanctioning cannot benefit recipients.

Despite increased uncertainty at the higher end of the cognitive scale, we note that caseworkers who score higher (from 3.25 and above) are as likely to sanction clients who cannot comply with the demands as who will not. Since we have a relatively small sample size and considerable uncertainty for values of 4 or higher on the cognitive attitudes scale, we cannot exclude that these results are simply a reflection of small sample sizes and high uncertainty.

One question that arises from these analyses is whether SLBs who score higher on the cognitive attitude component are also more inclined to interpret the “cannot vignette” as a “will not vignette”? To investigate this, we make use of the variable where the SLBs place the vignettes on a scale 7-point scale ranging from ‘lack of will’ to ‘lack of capacity’ as our dependent variable. The results are presented in Figure 3.

Figure 3.

LMP models of attitudes, with controls on whether the non-compliance in the vignette is due to a lack of capacity (low) or a lack of will (high). Office FE and robust SE.

SLBs who score high on the cognitive attitude component (view users as manipulative, dishonest, hostile, unpredictable, and stubborn), are more likely to attribute the reason for non-compliance to a lack of will, also when controlled whether the vignette included information on whether the course was boring and futile, or the fictive recipient had social anxiety. This confirms that SLBs who score higher on the cognitive attitude component are more likely to interpret non-compliance as the recipient’s personal responsibility. In the encounter with the cannot vignette, the prejudiced is thus more inclined to interpret this as a will not case, and consequently also more inclined to sanction this vignette.

Discussion and conclusion

In the policy process, discretion has been seen as a “necessary evil” (Hupe and Hill, 2021). It is a prerequisite for being sensitive to the specific situation, but it can also contribute to undermining the implementation of the government’s ambitions. SLBs’ attitudes towards clients are crucial in both roles as “citizen agents,” and “state agents,” as they might threat access to equal treatment. Investigating how SLBs’ attitudes towards clients affect their discretionary decision-making is demanding. In studies based on qualitative data and design the importance of attitudes towards clients has previously been salient, but less salient in survey studies of discretionary decision-making. The primary contribution of this study is twofold. Firstly, we test an instrument is developed to measure SLBs’ attitudes towards clients. While this instrument has previously been tested on tax bureaucrats, it is indented to be valid also for other types of SLBs. Secondly, we explore whether there is a causal relationship between SLBs’ attitude towards clients and discretionary decision-making.

The measurement instrument

We are testing a multicomponent instrument, which intends to measure attitudes towards clients for Dutch tax bureaucrats (Keulemans and Van de Walle, 2018). However, we note that the parsimonious factor solution differs for three items and the correlations between the positive and negative attitude subscales are significant in our sample. Despite this difference, these general attitudes instrument allows us to differentiate between: cognitive attitudes (beliefs and attributes that bureaucrats associate with clients), two affective attitude components (which inform the general evaluation of recipients and the bureaucrat’s emotional responses towards recipients) and behavioral attitudes (informs of the past behaviors of the SLB).

Our findings indicate that the instrument functions reasonably well for the SLBs in the NLWA. However, it remains plagued by the same limitations as the original study (i.e., adequate but not perfect model fit and potential social desirability bias). Social desirability bias is a concern especially for the cognitive attitude component, where SLBs might be unwilling to admit that they find clients manipulative or dishonest, hence leading us to underestimate the importance of this dimension. Due to the norms associated with the role, this tension might perhaps be experienced stronger by the SLBs in the welfare sector, compared to the tax administration. In sum, the multicomponent instrument of attitude towards clients, demonstrates applicability across different types of street-level bureaucrats and across national contexts.

Interaction between attitudes and discretionary decision-making

Most previous studies have not been able to distinguish between attitudes towards clients and other related attitudes. By making use of an instrument of attitudes towards clients in an exploration of SLBs’ sanctioning behaviour, the study contributes to more nuanced knowledge of discretionary decision-making.

Our findings indicate that attitudes indeed play a significant role in the discretionary decision-making process of SLBs when it comes to sanctioning recipients, even after considering structural conditions. From a theoretical standpoint, we proposed that under certain conditions, attitudes may influence sanctioning practice, but this relationship may be mitigated by the context in which bureaucrats work (Brodkin, 2011; Soss et al., 2011). Contextual factors were accounted by employing rigid office-level fixed effects (i.e., focusing on the within-office variation in attitudes and sanctioning). Accounting for contextual factors is important, both theoretically and empirically: the scholarship has pointed out that increased performance monitoring and other new public management types of reforms have are likely to standardize practices and in turn decision making. Our findings suggest that this might be the case for social assistance in Norway: clients who cannot comply with the activation requirements are on average less likely to be sanctioned, irrespective of the SLB’s attitudes.

It is important to note that not all types of attitudes towards clients demonstrated a relationship with the propensity to impose sanctions. We do not find statistically significant associations between the affective attitudes, or behavioral attitude components and sanctioning. However, cognitive attitudes towards clients were associated with SLBs’ discretionary decisions regarding sanctions. SLBs who exhibited unfavourable cognitive attitudes towards clients (measured as bureaucrats who consider that recipients are likely to be manipulative, dishonest, hostile, unpredictable, and stubborn) were more inclined to impose sanctions for non-compliance with activity requirements.

Previous studies showed that SLBs use of discretionary space is based on their categorization of clients (Jilke and Tummers, 2018). This categorization follows from an interpretation of a limited set of cues, which are informed by general attitude towards clients (Keulemans and Van de Walle, 2018). This study illustrates the importance of the interaction between attitudes and categorizations of clients. Torsvik et al. (2022) showed that SLBs’ propensity to sanctioning recipients is extremely sensitive to their perception of the client’s personal responsibility. We additionally show an interaction between SLBs’ attitudes towards clients and perception of personal responsibility when SLBs are imposing sanctions. In line with previous research, we show that SLBs with a condemnatory moralistic view of clients are more inclined to respond unfavourably. Yet, prejudiced SLBs are more inclined to impose sanctions because they more often tend to categorize the reason for non-compliance as lack of will. This emphasises how crucial SLBs’ perception of personal responsibility is and illustrates how the distinction between “cannot” and “will not” contributes to determine the outcome of discretionary decision-making.

We do not discount that education within social work is important. Bureaucrats with a social work education are significantly less likely to attribute lack of participation to lack of capacity (cannot) rather than a lack of will (will not). This finding can be interpreted through the lens of the specific moral framework inherent in social work education (Hasenfeld, 1999), which posits that clients inherently possess good intentions but may lack the necessary abilities to meet activity requirements. Bureaucrats with a higher number of users are more likely to attribute non-compliance to a lack of will (Figure 3). Although this association does not reach statistical significance in Figure 1, it shows a similar tendency – a high number of users is associated with a higher likelihood of sanctioning. As such it cannot be excluded that high caseloads may lead to ‘client-overload’ and potentially more cynical caseworkers.

Overall, our findings revealed a complex interplay between SLBs’ cognitive attitudes toward recipients, their categorisations of recipients, and their propensity to impose sanctions for non-compliance. The article also contributes to the growing literature which demonstrates how the perceptions of client attributes affect discretionary decision-making (Fekjær et al., 2023).

Limitations

This paper builds on data which are developed and collected for a different purpose. We argue that reusing existing data is important, as it both an effective way to replicate extant research and potentially uncover new mechanisms. Nevertheless, cognitive attitudes toward clients were measured using items that primarily reflect negative perceptions. While these items effectively capture a particular dimension of cognitive attitudes, they present a limitation by focusing predominantly on distrust and suspicion. This one-sided emphasis restricts the scale’s ability to represent the full spectrum of caseworkers’ cognitive evaluations of clients. Our results do not reveal any differences for the positively worded items from the positive affective and behavioral attitudes. This limitation also serves as feedback for the original developers of the instrument (Keulemans and Van de Walle, 2018), whose scale may benefit from further refinement to encompass a broader range of cognitive attitudes. We hope that future research will refine these attitude scales for bureaucrats.

The findings presented here are instrument, sample and context specific. Although the design offers a state-of-the-art way of controlling for selection (i.e. bureaucrats working with the same type of client, which then shapes their attitudes and sanctioning practice), in our design the attributes of the clients are randomly allocated and not the attitudes of the bureaucrats. These findings should not be interpreted as a definite proof of a causal relationship between general attitudes and willingness to sanction. Furthermore, since the vignette employed in this study was implemented considerable progress has been made regarding the design of studies employing vignettes (Druckman 2022; Harrits and Møller, 2021). One such critique of importance is that the current study aims to explain differences between vignettes without having a baseline and that despite the randomization (which functioned well in the present study, supplemental A3), study participants added their own reference frames when interpreting the vignettes beyond included stimuli (Harrits and Møller, 2021). While pure controls do not exist (Druckman, 2022), we do acknowledge that comparison between the different treatments can be problematic. While this remains an important caveat for the current study, a mitigating factor which strengthens our interpretation of the results is that study participants did classify the four different stimuli in accordance with was expected. Lastly, the sample employed in our study is not large and the survey has some missingness. Hence, we cannot exclude that some of the non-significant associations we find are due to a lack of statistical power.

Implications

Since discretionary decision-making has a pivotal role in public administration, the findings in this study are of importance to policymakers at all levels. To ensure that the rule of law is implemented, it is crucial to develop mechanisms that can mitigate the role of individual attitudes towards clients and improve the quality of discretionary reasoning. One alternative might be to offer supportive measures to the bureaucrat, such as various kinds of decision support. Another alternative might be to structure the decision situation so that bureaucrats must weigh reasons for and against a decision collectively or by making the bureaucrats anticipate that they would have to justify their decisions. The latter could happen by regularly selecting a random sample of decisions for revision in an internal forum. The assumption is that the possibility of being exposed to review, would lead decision-makers to reason as if they had to justify their actions to an audience (Molander et al., 2012).

It is important to acknowledge that such mechanisms have some limits. Keulemans and Van de Walle (2020) show that work groups have a limited impact on the individual’s client-attitude. They suggest that careful recruitment of new collaborators is important, in addition to designing formal work structures that mitigate the role of individual attitudes towards clients. We note that some of these strategies are at least partly at odds with the theoretical frameworks discussed in the beginning, and that more research is needed on effective ways to transfer these findings into concrete policy implications.

Supplemental Material

Supplemental Material - Street-level bureaucrats’ attitudes towards clients in discretionary decision-making: Evidence from the Norwegian labour and welfare administration

Supplemental Material for Street-level bureaucrats’ attitudes towards clients in discretionary decision-making: Evidence from the Norwegian labour and welfare administration by Andreea Ioana Alecu, Talieh Sadeghi and Lars Inge Terum in Journal of Public Policy and Administration.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Andreea Ioana Alecu

Supplemental Material

Supplemental material for this article is available online.

References

Aarøe

Petersen

(2014) Crowding out culture: scandinavians and Americans agree on social welfare in the face of deservingness cues. The Journal of Politics 76(3): 684–697.

Assadi

Lundin

(2018) Street-level bureaucrats, rule-following and tenure: how assessment tools are used at the front line of the public sector. Public Administration 96(1): 154–170.

Baviskar

Winter

(2017) Street-level bureaucrats as individual policymakers: the relationship between attitudes and coping behavior toward vulnerable children and youth. International Public Management Journal 20(2): 316–353.

Brodkin

(2011) Policy work: street-level organizations under new managerialism. Journal of Public Administration Research and Theory 21(suppl_2): i253–i277.

Brodkin

(2013) Street-level organizations and the welfare state. In: Work and the Welfare State: Street-Level Organizations and Workfare Politics. Washington, DC: Georgetown University Press, 17–34.

Buffat

Hill

Hupe

(2015) Understanding Street-Level Bureaucracy. Bristol, UK: Policy Press.

Caswell

Høybye-Mortensen

(2015) Responses from the frontline: how organisations and street-level bureaucrats deal with economic sanctions. European Journal of Social Security 17(1): 31–51.

DeVellis

(2016) Scale Development: Theory and Applications. Thousand Oaks, CA: Sage Publications.

Druckman

(2022) Experimental Thinking. Cambridge, UK: Cambridge University Press.

10.

Eiser

van der Pligt

(2015) Attitudes and Decisions. London, UK: Psychology Press.

11.

Eleveld

(2017) The sanctions mitigation paradox in welfare to work benefit schemes. Comparative Labor Law and Policy Journal 39: 449.

12.

Epstein

Knight

(2013) Reconsidering judicial preferences. Annual Review of Political Science 16: 11–31.

13.

Fekjær

Øverbye

Terum

(2023) Who deserves to be sanctioned? A vignette experiment of ethnic discrimination among street-level bureaucrats. Acta Sociologica 67: 00016993231156486.

14.

Gjersøe

Leseth

Vilhena

(2020) Frontline implementation of welfare conditionality in Norway: a maternalistic practice. Social Policy and Administration 54(3): 491–504.

15.

Goodsell

(1981) Looking once again at human service bureaucracy. The Journal of Politics 43(3): 763–778.

16.

Gough

Bradshaw

Ditch

, et al. (1997) Social assistance in OECD countries. Journal of European Social Policy 7(1): 17–43.

17.

Guul

Pedersen

Petersen

NBG

(2021) Creaming among caseworkers: effects of client competence and client motivation on caseworkers’ willingness to help. Public Administration Review 81(1): 12–22.

18.

Harrits

Møller

MØ

(2021) Qualitative vignette experiments: a mixed methods design. Journal of Mixed Methods Research 15(4): 526–545.

19.

Hasenfeld

(1999) Social services and welfare-to-work. Administration in Social Work 23(3): 185–199.

20.

Hupe

Hill

(2016) ‘And the rest is implementation.’ Comparing approaches to what happens in policy processes beyond great expectations. Public Policy and Administration 31(2): 103–121.

21.

Hupe

Hill

(2021) Implementing Public Policy: An Introduction to the Study of Operational Governance. Thousand Oaks, CA: Sage Publications Ltd, 1–100.

22.

Isen

Shalker

Clark

, et al. (1978) Affect, accessibility of material in memory, and behavior: a cognitive loop? Journal of Personality and Social Psychology 36(1): 1–12.

23.

Jilke

Tummers

(2018) Which clients are deserving of help? A theoretical model and experimental test. Journal of Public Administration Research and Theory 28(2): 226–238.

24.

Kahneman

(2017) Thinking, fast and slow. London: Penguin Books.

25.

Keiser

(2010) Understanding street-level bureaucrats’ decision making: determining eligibility in the social security disability program. Public Administration Review 70(2): 247–257.

26.

Keulemans

Van de Walle

(2018) Understanding street-level bureaucrats’ attitude towards clients: towards a measurement instrument. Public Policy and Administration 35: 84–113.

27.

Keulemans

Van de Walle

(2020) Street-level bureaucrats’ attitude toward clients: a study of work group influence in the Dutch and Belgian tax administration. Public Performance and Management Review 43(2): 334–362.

28.

Kroeger

(1975) Bureaucracy, social exchange, and benefits received in a public assistance agency. Social Problems 23(2): 182–196.

29.

Lax

(2007) Constructing legal rules on appellate courts. American Political Science Review 101(3): 591–604.

30.

Lipsky

(2010) Street-Level Bureaucracy: Dilemmas of the Individual in Public Service. Manhattan, NY: Russell Sage Foundation.

31.

Maio

Haddock

Verplanken

(2018) The Psychology of Attitudes and Attitude Change. Washington, DC: Sage Publications Limited.

32.

Maynard-Moody

Musheno

(2003) Cops, Teachers, Counselors: Stories from the Front Lines of Public Service. Ann Arbor, MI: University of Michigan Press.

33.

McGann

Nguyen

Considine

(2019) Welfare Conditionality and Blaming the Unemployed. Administration & Society. Los Angeles, CA: Sage Publications Sage, 1–29.

34.

Molander

Grimen

Eriksen

(2012) Professional discretion and accountability in the welfare state. Journal of Applied Philosophy 29(3): 214–230.

35.

NAV (2018) Rundskriv til Lov om sosiale tjenester i NAV, R35–00 [Circular to the social services act in NAV]. Arbeids & Velferdsetaten (NAV).

36.

Oberfield

(2014) Becoming Bureaucrats: Socialization at the Front Lines of Government Service. Philadelphia, PA: University of Pennsylvania Press.

37.

Oorschot

(2000) Who should get what, and why? On deservingness criteria and the conditionality of solidarity among the public. Policy & Politics 28(1): 33–48.

38.

Petersen

Slothuus

Stubager

, et al. (2011) Deservingness versus values in public opinion on welfare: the automaticity of the deservingness heuristic. European Journal of Political Research 50(1): 24–52.

39.

Pickens

(2005) Attitudes and perceptions. Organizational Behavior in Health Care 4(7): 43–76.

40.

Raaphorst

Van de Walle

(2018) A signaling perspective on bureaucratic encounters: how public officials interpret signals and cues. Social Policy and Administration 52(7): 1367–1378.

41.

Sadeghi

Terum

(2020) Frontline managers’ perceptions and justifications of behavioural conditionality. Social Policy and Administration 54(2): 219–235.

42.

Segal

Spaeth

(2002) The Supreme Court and the Attitudinal Model Revisited. Cambridge, UK: Cambridge University Press.

43.

Segal

Champlin

Howard

, et al. (2017) The attitudinal model. In: Routledge Handbook of Judicial Behavior. London, UK: Routledge, 17–33.

44.

Soss

Fording

Schram

, et al. (2011) Disciplining the Poor: Neoliberal Paternalism and the Persistent Power of Race. Chicago, IL: University of Chicago Press.

45.

Stone

(1981) Attitudinal tendencies among officials. In: The Public Encounter: Where State and Citizen Meet. Bloomington, IN: Indiana University Press, 43–68.

46.

Torsvik

Molander

Terum

(2022) The will to sanction: how sensitive are caseworkers to recipients’ responsibility when imposing sanctions on non-compliance in a welfare-to-work programme? International Journal of Social Welfare 31(1): 77–85.

47.

van Berkel

(2020) Making welfare conditional: a street-level perspective. Social Policy and Administration 54(2): 191–204.

48.

Van Berkel

Caswell

Kupka

, et al. (2017) Frontline Delivery of Welfare-To-Work Policies in Europe: Activating the Unemployed. Abingdon, UK: Taylor & Francis.

49.

Watts

Fitzpatrick

(2018) Welfare Conditionality. London, UK: Routledge.

50.

Zacka

(2017) When the State Meets the Street: Public Service and Moral Agency. Cambridge, MA: Harvard University Press.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.75 MB