Not so different in present attitudes and behaviour,but expected future membership: A technical replication of a study of party youth in six European democracies

Abstract

Based on the statistical analysis of an original survey of young party members from six European democracies, a study concluded that three types of young members differed systematically regarding their membership objectives, activism, efficacy and perceptions of the party and self-perceived political future. We performed a technical replication of the original study, correcting four deficiencies, which led us to a different conclusion. First, we discuss substantive significance in addition to statistical significance. Second, we ran significance tests on all comparisons instead of limiting them to an arbitrary subset. Third, we performed pairwise comparisons between the three types of members instead of using pooled groups. Fourth, we avoided the inflation of the type-I error rate due to multiple testing by using the Bonferroni–Holm correction. We found that most of the differences between the types lacked substantive significance, and that statistical significance only coherently distinguished the types of members in their future membership, but not in their present behaviour and attitudes.

The study of party youth in Europe

In a study of young party members, Bruter and Harrison made a laudable effort to understand their motivations and behaviour by running an original online survey of 2919 members aged 18–25 belonging to 15 parties in six European democracies. The questionnaire comprised 13 topics covering multiple items that asked about the respondent’s socio-economic characteristics and their perceptions, preferences, behaviour and expected future involvement in the party (Bruter and Harrison, 2009: 1286–1287). Bruter and Harrison first hypothesized that young party members could be primarily designated as being either morally, socially or professionally minded. Morally oriented members serve a moral duty by getting engaged in a party; professionally oriented members are focused on office and think and behave like political leaders; socially oriented members use their party as a source of social interaction and stimulation (Bruter and Harrison, 2009: 1264–1266); an exploratory factor analysis confirmed the hypothesis about the presence of three types of members (2009: 1271).

Building on this finding, Bruter and Harrison (2009) tested four hypotheses on how the three member types differ in terms of i) membership objectives, ii) levels and forms of activism, iii) efficacy and democratic perceptions, and iv) self-perceived political future and membership in the party. ¹ Each of the four hypotheses is a compound or composite hypothesis covering two to six theoretical expectations, yielding 19 expectations in total (Bruter and Harrison, 2009: 1267; see Tables A.3–A.6 in the online appendix). We have reproduced the hypothesis on member attitudes as an example of a composite hypothesis. The numbers in brackets highlight the individual expectations:

Professional-minded members are the most efficacious [1], the most positive about the party’s organization [2], and the most likely to think of politics as a profession [3]. Social-minded members are the most critical [4] and least efficacious [5]; moral-minded members are least likely to think of politics as a profession [6]. (Bruter and Harrison, 2009: 1267)

Each individual expectation was tested by running an ‘ANOVA test of difference’ between the mean responses of party members belonging to different groups (Bruter and Harrison, 2009: 1279). Some expectations were tested with one item, while others were linked to up to six items (see Tables A.3–A.6 in the online appendix). For example, the expectation that socially minded members are most critical means that this member type would show the lowest mean response across six items measuring different attitudes towards their own party (Bruter and Harrison, 2009: 1279–1281). Bruter and Harrison concluded that the results confirmed the hypotheses, and stated that ‘Each type of young party member is characterized by different patterns of perceptions, attitudes, activities, preferences, and hopes and expectations about their own future’ (Bruter and Harrison, 2009: 1284).

In this paper, we point to four deficiencies in the original empirical analysis and remedy them in a technical replication. Our results qualify the original conclusion. Young party members only show statistically significant differences in their long-run expectations and display an incoherent picture when it comes to present perceptions, attitudes and activities. We further show that most differences in mean responses are marginal and lack substantive significance.

Four problems with the analysis

We admire the effort that Bruter and Harrison put into collecting the data to answer a relevant research question. However, we argue that the analysis suffers from four problems. The shortcomings we have corrected are, first, a neglect of substantive significance. Since the number of observations – about 3000 – is large, it is particularly important to consider substantive significance because statistical significance should not be difficult to achieve. Second, some comparisons of means were excluded from significance testing for no obvious reason. For the 22 items that we reanalyse below, the original analysis contains significance tests for only 11 of them.

Third, the test of differences between the three types of young members assessed whether one group ‘differ[s] … significantly from the rest of the sample’ (Bruter and Harrison, 2009: 1279). The comparison of a group’s mean to the pooled mean response of the two other groups is inconsistent with the valuable idea of distinguishing three groups of young members. A test of one group against the pooled mean response of the other two groups does not allow one to say that the three groups differ from each other, because this is not what is being tested. One can easily conceive of constellations in which a test of one group against two pooled groups masks interesting findings. It is possible that one group differs from the pooled group, but not from one or even both groups if we run separate significance tests. Similarly, one group of young members might not differ from a pooled sample, but differ from one or both of the other two groups in pairwise comparisons. ²

The fourth problem is significance testing without taking into account that multiple comparisons inflate the probability of making a type-I error (Abdi, 2010: 573). Multiple comparisons require controlling the family-wise error rate (FWER) by using an appropriate error correction technique. In the original study, multiple testing was an issue because 12 tests related to both the objectives of young members and the expected future involvement in the party; 21 tests were tied to both activism and democratic perceptions. ³ Figure 1 shows how the FWER depends on the number of tests. The uncorrected FWER increases to 0.46 for 12 tests and to 0.66 for 21 tests, demonstrating that the original analysis ran a considerably increased risk of falsely rejecting the null hypothesis of no significant mean differences.

Figure 1.

Family-wise error rate conditional on number of tests.

A technical replication and its results

We performed a technical replication that remedied the four deficiencies of the original analysis. Since the original data is lost (see Note 2), we had to rely on the summary statistics reported in the article – mean, standard deviation and number of responses per type – to replicate the analysis of the four hypotheses. First, our replication ran analyses on all 22 items instead of a subset of them. Second, we aligned design and theory by comparing one group of young members with another group instead of a pooled sample, yielding 66 tests in total because we had to perform three pairwise comparisons per item. ⁴ Third, for each hypothesis we established a FWER of 0.05 by using the Bonferroni–Holm (1979) error correction, which is widely applied and has more power than the Bonferroni correction (Abdi, 2010). Fourth, we discuss the substantive significance of the findings and how large the mean differences were.

We re-evaluated the hypotheses by cross-tabulating the aggregate results of the significance tests with and without error correction. ⁵ Table 1 aggregates the results across all hypotheses. Table 2 summarizes the results separately for each hypothesis.

Table 1.

Aggregate results for 22 items leading to 66 tests.

		Significant without error correction
		No	Yes
Significant with error correction	No	21	11
	Yes	-	34
αFWER = 0.05

alpha_FWER: family-wise error rate per hypothesis.

Table 2.

Disaggregated results for 22 items leading to 66 tests.

		Party objectives		Activism
		Significant without error correction		Significant without error correction
		No	Yes	No	Yes
Significant with error correction	No	4	3	7	6
	Yes	-	5	-	8
		Democratic perceptions		Future membership
		Significant without error correction		Significant without error correction
		No	Yes	No	Yes
Significant with error correction	No	8	2	2	0
	Yes	-	11	-	10
αFWER = 0.05

alpha_FWER: family-wise error rate per hypothesis.

Twenty-one tests out of 66 did not allow us to reject the null of no mean difference without error correction. This number increased to 32 when we controlled for multiple testing, and means we failed to reject the null hypothesis almost as many times as we rejected it. The aggregate perspective showed that the three types of young members can hardly be considered to ‘have their own “story”’ (Bruter and Harrison, 2009: 1284). There are some differences between young party members, but only half of their story is unique while the other half of the story is shared.

This conclusion was supported by a disaggregated analysis because none of the four hypotheses was fully confirmed. We failed to reject the null hypothesis of no mean difference in the majority of tests for three hypotheses: seven out of 12 for party objectives, 13 out of 21 for party activism and 10 out of 21 for democratic perceptions. The only dimension for which we could reject the null hypothesis most of the time concerned expected future involvement, displaying 10 significant comparisons out of 12. This dimension was also the only one for which error correction did not affect the results.

When we further disaggregated the perspective and looked at individual comparisons between groups (Tables A.3–A.6 in the online appendix), we found that less than one-third of all comparisons allowed us to reject the null hypothesis for both pairwise comparisons implied by a specific expectation in a hypothesis (10 items out of 36). ⁶ For 16 items, one test was significant and the second one not, and 10 items failed to reject the null hypothesis for both comparisons.

In the last step of our reanalysis, we focused on the substantive significance of the mean differences. This is based on the premise that an assessment of statistical and substantive significance should go hand in hand. Figure 2 plots the mean differences separately for each hypothesis and pairwise comparison of types of members. On a five-point scale, we considered differences of less than 10% of the scale to be of little significance (equivalent to 0.5 points). ⁷ Of all 66 comparisons, only 5% involved a difference in means of 0.5 or more. Even if we reduce the definition of substantive significance to 5% of the 0–4 scale (0.25 points), only 25% of all comparisons reach this threshold. Most of the mean comparisons therefore lacked substantive significance. ⁸

Figure 2.

Absolute mean differences per hypothesis.

Conclusion

We identified four deficiencies in the statistical analysis of an original survey of young party members and corrected them in a technical replication. In contrast to the original conclusion that ‘Each type of young party member is … different’ (Bruter and Harrison, 2009: 1284), our analysis suggests that the picture is much more nuanced. We argue that young party members do not display relevant differences in their membership objectives, levels of activism or democratic perceptions, because half of the comparisons were not significant and most of them lacked substantive significance. The only statistically significant mean differences we found related to the party members’ self-perceived political future. Most of these differences, however, also displayed low substantive significance.

The replication results provide crucial insights into ‘who exactly young party members are, what they seek, and how they perceive their involvement and future role’ (Bruter and Harrison, 2009: 1260). Our findings indicate that the differences in future aspirations are not strongly grounded in present differences in young party members’ attitudes and behaviour. Given that professionally minded, socially minded and morally minded members do not differ coherently in their current opinions and behaviour, it is puzzling that they believe their future engagement in the party and politics more generally will be different. We cannot explore the reasons for this discrepancy, but consider this an insight worth exploring in follow-up research on the party youth.

Supplemental Material

Appendix – Supplemental material for Not so different in present attitudes and behaviour, but expected future membership: A technical replication of a study of party youth in six European democracies

Supplemental material, Appendix for Not so different in present attitudes and behaviour, but expected future membership: A technical replication of a study of party youth in six European democracies by Lion Behrens and Ingo Rohlfing in Research & Politics