Entry Characteristics and Participation in a Peer Learning Program as Predictors of First-Year Students’ Achievement,Retention,and Degree Completion

Abstract

Success in the first year of higher education is important for students’ retention beyond their first year and for completion of their undergraduate degree. Institutions therefore typically front-load resources and interventions in the first year. One such intervention is the Peer Assisted Study Sessions (PASS) program. This program is known in the United States as Supplemental Instruction. It provides first-year students with an opportunity to learn study skills in the context of a particular unit of study (course/module). In this article, we consider the relationship between students’ prior academic achievement and participation in the PASS program, as well as the impact of participation on first-year students’ first-year grade point average, retention, and degree completion. The findings suggest that PASS does not just attract academically high-achieving students and that participation in it contributes to students’ academic achievement in their first year, retention beyond the first year, and completion of an undergraduate degree.

Keywords

Supplemental Instruction Peer Assisted Study Sessions retention degree completion

The importance of the first year in higher education has long been recognized as being vital for students’ ultimate success in attaining degree completion (Kift, Nelson, & Clarke, 2010; Krause, Hartley, James, & McInnis, 2005; Nelson, Clarke, Kift, & Creagh, 2011; Tinto, 2007; Tinto & Pusser, 2006). Many factors contribute to retention and attrition during or after the first year. These factors include issues both social and academic. For students, an academic transition is often marred by difficulty in understanding what is expected of them and what the differences are between high school and university (van der Meer, 2012; van der Meer, Jansen, & Toorenbeek, 2010). Although these differences may vary as a function of different school systems, university systems, entrance selection, or student characteristics, there is a general appreciation that most students have to orient themselves to a new environment (Jansen & Bruinsma, 2005; Jansen & van der Meer, 2007a, 2007b; van der Meer & Jansen, 2008). This is evidenced by the great variety of social and academic orientation and intervention programs that operate in most parts of the world.

Although there is a great variety of these programs, some have migrated across the world and been implemented in different countries (sometimes with minor variations) irrespective of differences in school and university systems.

Supplemental Instruction

Supplemental instruction (SI) is one of the more widely known programs that is implemented in many universities in the United States. The popularity of which is due to its proven contribution to student achievement (Arendale, 1994; Martin & Arendale, 1992a, 1992b, 1992c, 1992d, 1994). The SI program—developed in the early 1970s in the United States to combat the attrition of first-year students—was initially focused on the attrition of African American students in the health sciences (Blanc, DeBuhr, & Martin, 1983; Arendale, 1997; Martin, 1983; Martin & Arendale, 1992a, 1992d). The essence of the program is the facilitation of study sessions by upper-year students in units of study with high failure rates. These study sessions are not aimed a reteaching of content but helping first-year students to become familiar with the university learning environment. This is done through the development of study skills in the context of a particular discipline and the unit of study to which SI is attached. As the student facilitators (or SI leaders) have recently completed the same unit of study, they have a good appreciation of the challenges that students may face with the academic content as well as what it is like to be a first-year student. SI leaders typically receive substantial training to ensure that they use collaborative and active learning approaches to support students in developing new skills and habits. A major focus of the training is to ensure that facilitators refrain from reteaching content. SI facilitators as well as SI program coordinators are trained drawing on material provided by U.S. and regional SI centers. This is aimed at ensuring a degree of “fidelity” with regard to how SI is implemented. Furthermore, SI facilitators receive regular monitoring visits to ensure quality and fidelity to the program ethos and design.

SI is a voluntary program to which students typically sign up within the first 2 weeks of a semester. Marketing of the program to students typically involves emphasizing the voluntary and mainstream nature of the program that benefits all students and presenting evidence that participation typically results in better academic outcomes. As all students are encouraged to sign up in the first 2 weeks and to attend all sessions, the program is not perceived to be remedial and to serve only less capable students (at the point when they realize that they are struggling).

The theoretical framework of the SI program was conceptualized by its developers as being the opposite of a reactive “medical model” fix-it support program, whereby only failing or struggling students partake in the “intervention.” SI is a proactive nonremedial universal nonstigmatizing social and academic induction-focused program (Arendale, 1993). The emphasis is on difficult courses, not failing students; in other words, it is aimed at all students enrolled in difficult courses. Its success has been attributed to this focus on its being nonremedial and its being student led; students appreciate the absence of a power differential and the ease of being able to engage with the peer facilitator (van der Meer & Scott, 2009). Arendale (1993) provided a good overview of the theoretical foundations of SI. In surveying key aspect of the SI program, he identified the range of theoretical ideas that have informed the overall theoretical framework of the program, including constructivism (drawing on Piaget and Vygotsky), Dale’s “cone of experience,” Tinto’s model of students’ social and academic integration, and theories related to students’ development of metacognitive and collaborative learning capabilities.

Apart from benefiting academic results, it also supports students’ transition and integration into academia and development of social capital (Holland, 2010; Toews & Yazedjian, 2007). We define “social capital” as a complex of interinterpersonal networks that provide students with friendships, peer support, and information that support the development of productive information- and help-seeking behavior. We concur with Holland (2010) that intentional development of social capital is important to support the academic achievement and retention of students from ethnic minorities and low socioeconomic backgrounds and those with English as an additional language.

Before funding this program across the country, the U.S. Department of Education subjected the program to rigorous evaluation regarding its effectiveness in achieving its intended goals. This resulted in a validation of the claims that SI is effective in achieving higher academic results and pass rates irrespective of prior ability and ethnicity (Arendale, 1993, 1994). The International Center for Supplemental Instruction was set up at the University of Kansas–Missouri, where the program originated. This center plays a national role in the accreditation and training of SI coordinators and programs, as well as an international role in accrediting other national centers, as in Australasia.

Whereas the program was originally developed in the United States, the massification of higher education in the 1980s and 1990s across the Western world prompted an interest in other countries for programs that would enhance students’ transition into higher education. Massification generally resulted in more diverse student populations entering higher education, such as students from minority ethnic backgrounds and low socioeconomic backgrounds (Scott, 1995). As many students from these groups were the first in their family to go to university, they had greater challenges in making sense of what was expected of them and how to succeed. This resulted in higher attrition rates for these groups. As a consequence of universities across the world looking for programs to support diverse students and reduce attrition rates, from the mid-1990s the SI program started to be introduced in the United Kingdom, Sweden, South Africa, Australia, and New Zealand. The name “Supplemental Instruction” was considered confusing and not self-explanatory in countries other than the United States. Hence, the name Peer Assisted Study Sessions (PASS) program was coined. In Australia, PASS programs are now operating in 38 of the 40 universities. A center for PASS at the University of Wollongong was accredited by The International Center for Supplemental Instruction to organize accreditation and training for the Australasian and Pacific region. Since about 2005, a number of New Zealand universities started to explore PASS. The university in this study started its pilot in 2008 and is now providing PASS in a range of subjects. Development of the program is ongoing.

SI/PASS Research

Despite its longevity and the amount of research into this program, doubts have been expressed about the effectiveness of the intervention for all students. The authors of a recent systematic review of SI/PASS (Dawson, van der Meer, Skalicky, & Cowley, 2014) highlighted the various reasons for such skepticism. One of these is the issue of self-selection. As the essence of the program is its voluntary nature, the assumption is often made that motivated and academically able students are more likely to enroll than students who are academically less able, who might more benefit from participation.

As a consequence of critique regarding self-selection by more able students, a number of researchers into the effectiveness of PASS have sought to report on or control for high school achievement results or entry scores (Dawson et al., 2014). These results and measures, of course, vary from country to country. Most of the current research comes from the United States. To our knowledge, no study has yet sought to include New Zealand high school achievement data. This study therefore aims to contribute to the research into SI/PASS effectiveness by seeking to control for prior academic performance by using New Zealand high school achievement data. In New Zealand, students’ achievement in their final year of high school determines their direct entry into universities and, when successful, results in the New Zealand Certificate of Educational Achievement (NCEA) university entrance qualification. Throughout their last year, students accumulate credits through internal assessment tasks as well as exams (these tasks/exams are referred to as NCEA Achievement Standards).

SI/PASS research in the university of this study commenced shortly after the pilot. PASS was introduced at this university in 2008 in the School of Business. As students who were enrolled in a bachelor of commerce degree were required to complete a range of compulsory units of study, these units were considered high stakes for progression into the degree; some had high failure rates and repeat numbers. Since 2009, PASS was also introduced in a number of other compulsory high-stakes health science units of study. Not all first-year students during that period therefore had equal access to PASS as the program was in development.

Focus of this Study

To develop a greater understanding of the relationship between students’ participation in PASS and their prior academic performance, as well as their performance at university as a consequence of PASS participation, we used data from three first-year entry cohorts (2009–2011). The rationale for these three cohorts was that they would also allow us to look at degree completion figures within 6 years after students started their first year. To meaningfully consider completion data, we chose a period of 6 years so that part-time students could be included as well as students who took double degrees or studied medicine.

To contribute to the extant evidence base of SI/PASS programs across the globe in terms of controlling for self-selection, this study is based on a quasi-experimental design, with high school academic performance as proxy for academic ability. Controlling for students’ academic performance at point of entry to the university, however, does not account for self-selection based on motivation. Some studies attempted to address this as well (Dawson et al., 2014). Regarding the evolutionary stage of the program in our institution, our present concern is to provide evidence that ensures that the SI/PASS is seen as a program that serves students with a range of backgrounds and therefore deserves ongoing funding. However, we anticipate addressing this in future studies in our institution as well so that we can attract an even wider range of students into the program.

Apart from analyses for the three whole cohorts of commencing first-year students, separate analyses were performed for three different groups identified as equity groups—namely, Māori, Pacific Island, and students who do not live in a residential college. With regard to Māori and Pacific Island students, the New Zealand Ministry of Education (2014) set goals for their “participating and achieving at all levels on a par with other students in tertiary education.” This is the rationale to include in our analyses Māori and Pacific Island students’ first-year achievements in relation to the PASS program and establish whether the program contributed to their success in the first year to a similar degree or better than that for the whole cohort. With regard to the third group, in this university, between 60% and 75% of first-year students live in a residential college for their first year, which provides a living/learning environment with academic support. As such, these students receive more support than students who do not live in a residential college. For that reason, we analyzed whether the PASS program contributed to first-year achievement of nonresidential college students to the same level or better as the overall cohort.

Although the focus is on the relationship among SI/PASS participation, prior academic achievement, and academic outcomes, we are aware that other factors might influence students’ academic success and/or PASS participation. As this is not an experimental study, to explore other possible factors, we sought to include in the various regression analyses variables that could affect the outcome variables. We were limited in this by the available data that we could access. From the student administration database, we extracted various demographic variables, such as gender, age, and English as an additional language. We also accessed the data of the New Zealand schools that students attended, including type (single sex or coeducational), proxy indicator for socioeconomic status (decile of New Zealand schools), and size of the school.

Apart from addressing the issue of self-selection, another focus of the study was to assess the impact of PASS participation on students’ overall academic success, which we sought to measure in terms of PASS attendees’ academic performance in the first year (relative to other students), their retention beyond first year, and their degree completion. Concerning assessment of the impact of SI/PASS participation on first-year grade point average (GPA), for this study, we are using the total number of sessions attended by students in their first year of studies across multiple units of study/modules. In most research, the effect of PASS participation is measured by assessing the relationship between number of sessions attended and final mark for a particular unit of study. This “dose-response effect” typically shows a strong relationship (Dawson et al., 2014). Most students in this study (75%) enrolled in PASS for only one of their units and 16% for two units. Measuring the impact of the attendance level, not just on the unit of study concerned, but also on the whole of students’ first-year performance is our contribution to the research into this area, as this has not been measured in many studies. This means that it cannot be expected that the impact will be as strong as when assessed in the context of just the unit of study for which the students were enrolled. However, the purpose of SI/PASS programs is not just about absolute academic performance (as measured in marks/grades) but also about support for first-year students’ academic and social integration. This integration is considered vital for student retention and consequent degree completion, as argued by Tinto in his seminal work in the 1970s and many others since him (Braxton, Milem, & Sullivan, 2000; Draper, 2002; McGinty, 1990; Synco, 2012; Tinto, 1975, 2000, 2006; Tinto & Goodsell, 1994; Tinto & Pusser, 2006). Therefore, we consider even a modest effect of SI/PASS on first-year academic performance, in combination with an impact on retention and degree completion, to be a good overall measure of the impact of the SI/PASS program on overall students’ academic success.

In summary, the key questions that we sought to answer were as follows:

Does students’ prior academic achievement affect their decision to participate in the SI/PASS program as well as their performance in the first year?

To what degree do SI/PASS participation and prior academic achievement have an effect on overall student success, as measured by the first-year GPA, retention beyond the first year, and degree completion within 6 years?

To what degree did participation in PASS in the first year contribute to the first-year academic achievement of three equity groups relevant for the context of the university of this study: Māori students, Pacific Island students, and students who do not live in one of the residential colleges?

Even though this study reports on findings from a New Zealand university, its findings have relevance to SI/PASS programs elsewhere in the world. All universities that have implemented SI/PASS programs are linked to regional SI/PASS centers and, through those, to the U.S. International Center for Supplemental Instruction. Consequently, there is a great degree of uniformity in the application of the program across the globe. All SI/PASS coordinators receive training through those centers and ensure that the fundamental principles of the programs, as described, are adhered to.

Method

Data were extracted from the university’s student administration database for all students who enrolled for the first time at the university in the 2009–2011 period. Students were deselected for further analysis if they were not enrolled in an undergraduate bachelor program, if they had an admission status of “graduate entry,” if they entered with transfer credits from another university, or if their previous occupation was self-reported as “university student.” Also, returning students who may have repeated their first year were excluded. Deselecting these students ensured that those who were included were “true” first-year undergraduate students.

In a separate exercise, NCEA data for the 2009–2011 period were extracted for those students whose National Student Numbers matched those for first-year students for that period. The separate records for each NCEA Achievement Standard were reformatted into single records for each student enrolled at our university. The credits for each standard were multiplied by a weighting factor based on attainment: 0 for not achieved, 2 for achieved, 3 for merit 3, and 4 for excellence. All weighted credits were then summed. This measure combined quantity and attainment level in the “weighted NCEA” score, which was used in our analyses. This data file was then matched and merged with the student administration data file.

Not all students enter New Zealand universities through the NCEA pathway. Other pathways include International Baccalaureate, Cambridge, and a range of other overseas qualifications. Furthermore, New Zealand students >20 years old can be admitted under the Special Admission pathway. For most analyses, students other than those coming through NCEA were excluded. For some analyses, we did not use the weighted NCEA score but created dummy variables for different entry pathways. The use of these dummy variables allowed us to use the greater data set for some analyses.

To identify and control for other school-related characteristics, such as the type of school and decile rating (which is considered a proxy for the socioeconomic background of students and schools), we matched our data with school information data from the New Zealand Ministry of Education as recorded in 2008.

Last, attendance data collected from the PASS coordinator for each participant were merged into this file. Rather than just using a binary indicator of PASS/not PASS, the number of PASS sessions that students had attended was used; typically, students were able to attend up to 12 sessions for each subject and in some cases 13 sessions, where an extra session was added directly prior to exams. If students attended PASS for more than one unit of study, the attendance numbers were summed. For students who did not attend any sessions, this field was set to zero. For some analyses, such as regression analyses, we used the total number of PASS sessions attended (including zero). However, in an analysis of variance (ANOVA) to determine if attending PASS resulted in cumulative benefits, we banded the number of PASS sessions into 5 bands: 0, 1–3, 4–7, 8–13, and >13 sessions attended. These bands are based on data that have been gathered over the years at our institution as well as in other SI/PASS research (Dawson et al., 2014) that suggest that students who attend more sessions perform better. These bands have been reported to reflect more or less the different patterns of attendance (in one unit of study)— from minimal to regular and nearly all available sessions. However, they are not absolute and serve only the purpose of making the results easier to interpret.

We considered the impact of PASS attendance and NCEA on completion of at least one degree within 6 years in two ways: (a) through a simple comparison in percentages for PASS attendance and completion percentages and (b) through a logistic regression, including, as the binary variable, PASS attendance and NCEA with completion of at least one program within 6 years. As we wanted to include students who may have studied part-time, took a double degree, or studied medicine, we considered 6 years a good period. Although some students were still studying without having completed a degree after 6 years because of their part-time status, (n = 231, 2% of total cohort), we decided to include them because we wanted the cohort to be as complete as possible. The results are therefore not to be read as absolute figures.

Statistical analyses based on SPSS 22 (IBM, Chicago, IL) were then performed on the data, including descriptive analysis, ANOVA, multiple regression, and binary logistic regression. For all regression analyses in this study, we used the adjusted R² measure to mitigate any overfitting of predictors; only these are reported.

Findings

Data Sources

Table 1 shows the results of the process of data extraction and matching of the different data sources.

Table 1

Number of Students in the Different Data Sources by Year

	2009	2010	2011	Total
First year at this university	4,502	4,658	4,143	13,303
Undergraduate students who have not studied at university before	3,768	4,030	3,588	11,386
Students who were admitted through NCEA university entrance	2,753	2,957	2,809	8519
NCEA data available for students	2,716	2,890	2,609	8215
GPA Year 1 data available	3,573	4,021	3,583	11,177
NCEA data and first-year GPA available	2,710	2,885	2,609	8204
School data available	2,679	2,840	2,568	8087

Note. NCEA = New Zealand Certificate of Educational Achievement; GPA = grade point average.

Table 2 shows the number and percentages of the different groups of interest for this study.

Table 2

Number (Percentages) of Students in Different Groups

	2009	2010	2011	Total
Students who attended PASS (>0 sessions)	280 (7.4)	352 (8.7)	536 (14.9)	1,168
Students not in a residential college	1,461 (38.8)	1,558 (38.7)	1,116 (31.1)	4,135
Māori students	406 (10.8)	442 (11)	405 (11.3)	1,253
Pacific Island students	110 (2.9)	132 (3.3)	111 (3.1)	353
Students not in a residential college who attended PASS^a	51 (3.5)	96 (6.2)	167 (15)	314^b
Māori students who attended PASS^a	19 (4.7)	37 (8.4)	77 (19)	133^c
Pacific Island students who attended PASS^a	2 (1.8)	14 (10.6)	20 (18)	36^d

Percentage within group. ^b8% vs. 12% college students. ^c11% vs. 11% non-Māori. ^d10% vs. 11% non–Pacific Island.

Impact of Prior Education on PASS Participation

To ascertain whether students who entered university with “better” qualification or higher scores were more likely to attend PASS sessions, we performed the following analyses: (a) an ANOVA with first-year students’ weighted NCEA credits as an independent variable and with the number of PASS sessions attended (banded in different ranges) as the dependent variable and (b) multiple regression with total number of PASS sessions attended as the outcome variable and with a number of possible predictors related to students’ former schools (i.e., before attending our university), their demographics, and their weighted NCEA scores.

There were some differences among attendance groups, F(4, 8210) = 4.20, p = .002 (see Table 3). However, Bonferroni post hoc analysis indicated that the only significant difference (p = .03) was between the group who did not attend PASS (M = 233, SD = 98.16) and the group who attended 4 to 7 sessions (M = 212, SD = 97.36). Furthermore, nonparticipants had on average higher NCEA scores than those who did attend PASS sessions.

Table 3

Mean Total Weighted NCEA Scores by PASS Sessions Attended

Sessions	Score	n	SD
None	232.76	7,304	98.16
1–3	219.83	252	79.37
4–7	211.67	193	79.36
8–13	222.02	329	72.24
>13	236.69	137	85.44
Total	231.50	8,215	96.20

Note. NCEA = New Zealand Certificate of Educational Achievement; PASS = Peer Assisted Study Sessions.

To further test the relationship between NCEA score and PASS participation, we looked at (a) the difference in the distribution of the weighted NCEA scores between those who attended PASS and those who did not and (b) the linearity between the weighted NCEA scores and PASS sessions attended. As Figure 1 shows, there was no linearity, and the distribution between attendees and nonattendees was reasonably normal (slightly more skewed toward higher scores for nonattendees). In summary, those students who attended PASS were not just the academically more able students.

Figure 1.

Relationship between New Zealand Certificate of Educational Achievement scores and Peer Assisted Study Sessions (PASS) attendance: Test of linearity through a scatterplot and comparison of distribution of scores between PASS attendees and nonattendees.

The multiple regression with PASS attendance as the outcome variable (including zero) clearly suggests that a very small proportion of variance in attendance is explained by the entered predictors, R² = .003. As can be seen in Table 4, very few predictors reached significance, including the weighted NCEA score, p = .07.

Table 4

Predictors for Attending PASS Sessions

	Unstandardized coefficients		Standardized coefficients, beta	t	p
	B	SE	Standardized coefficients, beta	t	p
(Constant)	3.16	1.63		1.94	.05
Decile band	0.07	0.07	0.01	1.01	.31
No. of enrolments of last high school	−0.05	0.01	−0.05	−4.65	.00
Gender of school (single sex, coed)	0.03	0.05	0.01	0.57	.57
Language (English or not)	0.17	0.16	0.01	1.04	.30
Gender	0.01	0.11	0.00	0.06	.95
Total weighted NCEA score	0.00	0.00	−0.02	−1.79	.07
Direct from NZ school	−0.29	0.18	−0.03	−1.64	.10
Overseas before university	0.09	0.24	0.00	0.36	.72
Age in Year 1	−0.11	0.08	−0.02	−1.39	.17
Disability declared	−0.10	0.22	−0.01	−0.46	.64
Living in residential college	0.28	0.10	0.03	2.81	.01

Note. R² = .003, F = 3.41, p < .001. PASS = Peer Assisted Study Sessions; NCEA = New Zealand Certificate of Educational Achievement; NZ = New Zealand.

In conclusion, from the data available at our institution for the years under investigation, it cannot be concluded that there any clear predictors for participation in PASS, including students’ prior academic ability as measured by their NCEA scores.

Prior Achievement, PASS, and First-Year GPA

We also sought to answer what impact prior educational achievement (NCEA scores) and PASS participation had on the GPA (on a 9-point scale) in students’ first year. We assessed this through correlation, ANOVA, and regression.

There was a clear relationship between the first-year GPA and both the NCEA score and number of PASS sessions attended. However, there was no significant correlation between NCEA score and PASS sessions attended, r(8213) = –.016, p = .149 (Table 5). The strong correlation between NCEA and first-year GPA may be interpreted to mean that performance in high school, as measured by the NCEA assessment framework, does matter for achievement in the first year at university; in other words, it does seem to have predictive validity.

Table 5

Correlations Among NCEA Score, PASS Sessions Attended, and GPA Year 1

	NCEA Score	PASS Sessions
PASS sessions	−.016	—
GPA Year 1	.676^**	.062^**

Note. NCEA = New Zealand Certificate of Educational Achievement; PASS = Peer Assisted Study Sessions; GPA = grade point average.

p < .01.

ANOVA showed a relationship between the number of sessions attended and a higher average GPA, F(4, 11172) = 11.95, p < .001 (see Table 6). The results by bands suggest that attending just a few sessions (one to three) in one unit of study does not have a big impact on the overall GPA, but attendance of more than four sessions does. As mentioned, 75% of students in this study were enrolled in PASS for one unit of study only and a further 16% for two units of study. A typical workload of first-year students in our institution is seven units of study per year. Considering this, the relatively modest relationship between GPA and PASS attendance could be considered an indication that PASS attendance may have some impact on the overall academic achievement in the first year through its function in students’ academic and social integration objective of PASS.

Table 6

GPA Year 1 by PASS Sessions Attended

Sessions	M	n	SD
None	4.21	10,009	2.34
1–3	4.07	338	2.04
4–7	4.30	265	1.80
8–13	4.46	402	1.78
>13	5.39	163	1.94
Total	4.24	11,177	2.30

Note. GPA = grade point average; PASS = Peer Assisted Study Sessions.

Table 7 shows the results of the multiple regression with the GPA for the first year as an outcome and a range of predictors that may contribute to success or confound other variables.

Table 7

Predictors of First-Year GPA

	Unstandardized coefficients		Standardized coefficients, beta	t	p
	B	SE	Standardized coefficients, beta	t	p
(Constant)	−3.00	0.75		−4.01	.00
Decile band	0.18	0.03	0.05	5.56	.00
Size last high school	0.01	0.01	0.01	1.49	.14
Gender of school	0.03	0.02	0.02	1.44	.15
Language (English or not)	0.51	0.08	0.06	6.82	.00
Gender	0.10	0.05	0.02	1.95	.05
Total weighted NCEA score	0.02	0.00	0.69	81.16	.00
Direct from NZ school	−0.37	0.08	−0.06	−4.60	.00
Overseas before university	0.05	0.11	0.00	0.41	.68
Age in Year 1	0.12	0.04	0.04	3.49	.00
Disability declared	−0.17	0.10	−0.01	−1.66	.10
Living in residential college	0.20	0.05	0.04	4.37	.00
Pass sessions attended	0.05	0.01	0.07	8.84	.00

Note. R² = .48. GPA = grade point average; NCEA = New Zealand Certificate of Educational Achievement; NZ = New Zealand.

The results suggest that this model predicts a good proportion of variance, R² = 0.48, F(14, 8061) = 532.44, p < .001. The weighted NCEA score reached the highest beta value, followed by PASS attendance. These results suggest that for every 24 PASS sessions that students attend (i.e., if they attend 12 sessions of PASS for two units of study), it is predicted that their GPA for the whole year will on average increase by just over one whole letter grade. For some students, that could mean going from a fail to a C– or from a C+ to a B–.

To make the results easier to interpret, we chose to include the actual PASS sessions attended as a predictor rather than attended sessions banded in groups (as for the ANOVA as shown in Table 6). However, for comparison, we did run a regression using the PASS bands as well. The results of the two regression analyses indicated that both models explained 48% of the variance (R² = .48); for the model with total sessions, F(12, 8063) = 615.64, p < .0001; and for the model with PASS bands, F(12, 8063) = 613.27. PASS totals significantly predicted the first-year GPA (β = .07, p < .001), as did PASS bands (β = .06, p < .001). The nonstandardized coefficients were B = .05 (β = .07) for actual PASS totals and B = .17 (β = .06) for actual PASS bands. In other words, 20 sessions predict an increase of 1 grade letter; a jump of 5 bands (>13 sessions), an increase of 1.02 grade letter.

Table 8 provides the data from the regression performed separately for the three equity groups: Māori students, Pacific Island students, and students who do not live in a residential college. For all three groups, PASS participation suggests a greater impact on GPA, especially for Pacific Island students. However, it needs to be noted that the number of students involved are relatively fewer.

Table 8

Regression Results (GPA Year 1) for Selected Equity Groups

	Māori		Pacific Island		College
	B	Beta	B	Beta	B	Beta
(Constant)	−6.32		−12.63		−5.39
Decile band	0.34	0.11^**	0.34	0.13^*	0.25	0.06^**
Size last high school	0.04	0.04	0.05	0.06	−0.01	−0.01
Gender of school	0.10	0.04	0.16	0.07	0.01	0.01
Language (English or not)	0.09	0.01	−0.05	−0.01	0.54	0.06^**
Māori					−0.22	−0.03
Pacific Island students	−0.55	−0.11			−0.19	−0.02
Gender	0.26	0.06^*	0.66	0.16^*	0.26	0.05^*
Total weighted NCEA score	0.02	0.63^**	0.02	0.69^**	0.02	0.67^**
Direct from NZ school	−0.05	−0.01	0.30	0.06	−0.19	−0.04
Overseas before university	0.31	0.02	−0.11	−0.01	−0.14	−0.01
Age in Year 1	0.26	0.11^*	0.50	0.22^**	0.23	0.12^**
Disability declared	−0.60	−0.06^*	−0.23	−0.02	−0.19	−0.02
Living in residential college	0.20	0.04	−0.16	−0.04
Pass sessions attended	0.07	0.11^**	0.13	0.16^**	0.07	0.10^**
R ²		0.48		0.45		0.43
F		60.03		19.47		466.78

Note. GPA = grade point average; NCEA = New Zealand Certificate of Educational Achievement; NZ = New Zealand.

p < .05. ^**p < .01.

Progression Beyond the First Year

We considered the impact of PASS attendance and NCEA in two ways—through a simple comparison in percentages of students who were retained in Year 2 and had participated in PASS and through an ANOVA with those who were retained in Year 2 as the independent variable and with weighted NCEA score as the dependent variable. As can be seen in Table 9, percentage-wise, more students who participated in PASS returned after Year 1.

Table 9

Retention Beyond Year 1 by PASS Sessions Attended

Sessions	Returned after Year 1	Left after Year 1 and did not return	Left and returned after Year 2	Total
None
n	8,589	1,505	124	10,218
%	84.10	14.70	1.20	100.00
1–3
n	334	2	2	338
%	98.80	0.60	0.60	100.00
4–7
n	265	0	0	265
%	100.00	0.00	0.00	100.00
8–13
n	400	1	1	402
%	99.50	0.20	0.20	100.00
>13
n	162	0	1	163
%	99.40	0.00	0.60	100.00
Total
n	9750	1508	128	11,386
%	85.60	13.20	1.10	100.00

Note. PASS = Peer Assisted Study Sessions.

ANOVA revealed that students who had an average higher weighted NCEA score seemed more likely to return, F(1, 11175) = 230.69 p < .001. We then performed a logistic regression with returning in Year 2 as a binary variable with retuning as a reference. Table 10 shows that the odds ratio Exp(B) suggests that a single session of PASS attendance is associated with an a threefold increase in students’ odds of returning in year 2; in other words, students are approximately three times more likely to return with each PASS session. For the first-year GPA, a 1-point increase in GPA on the 9-point scale resulted in a 50% increase in a student’s odds of returning. Performance on NCEA, as we saw earlier, significantly predicts the first-year GPA but does not add anything further when controlled for by GPA and is not significant, p = .11. Living in a residential college increases the odds of returning to the second year by 40%. The pseudo R² (Cox and Snell, Nagelkerke) statistics suggests that the variance explained by the model is between 10% and 19%. The significance of the Hosmer and Lemeshow goodness-of-fitness test was <0.05, which typically is considered indicative of a bad fit for the model. However, this may be a function of the sample size.

Table 10

Predictors of Retention in Year 2

	B	SE	Wald	p	Exp(B)
Pass attendance	1.16	0.27	18.49	.00	3.19
Living in a residential college	−0.35	0.08	21.32	.00	0.70
Total weighted NCEA	0.00	0.00	2.55	.11	1.00
School decile 4-7 (medium)	0.18	0.07	5.78	.02	1.19
GPA Year 1	0.43	0.02	338.30	.00	1.53
Constant	0.63	0.14	21.28	.00	1.88

Note. df = 1.00 for each row. School decile is a proxy for social economic status (decile 1-3 = low, decile 4-7 = medium, decile 8-10 = high). NCEA = New Zealand Certificate of Educational Achievement; GPA = grade point average.

Completion of a Degree

Table 11 shows that we can see again a relationship between PASS and completion, with more students who attended PASS completing a program within 6 years. For students who attended four or more sessions, this was in excess of 90%, as opposed to 68% for those students who did not attend any PASS sessions.

Table 11

Completion Rates by PASS Sessions Attended

Sessions	Programs completed within 6 years		Total
Sessions	None	≥1	Total
None
n	3,265	6,953	10,218
%	32.00	68.00	100.00
1–3
n	55	283	338
%	16.30	83.70	100.00
4–7
n	23	242	265
%	8.70	91.30	100.00
8–13
n	30	372	402
%	7.50	92.50	100.00
>13
n	8	155	163
%	4.90	95.10	100.00
Total
n	3,381	8,005	11,386
%	29.70	70.30	100.00

The logistic regression results in Table 12 show that PASS, alongside first-year GPA scores, still has a positive effect on course completion—specifically, a 20% increase in students’ odds of completion with every PASS session attended in their first year. The pseudo R² statistics (Cox and Snell, Nagelkerke) suggests that the variance explained by the model is between 20% and 29%.

Table 12

Predictors of Completion Within 6 Years

	B	SE	Wald	p	Exp(B)
Pass attendance	0.19	0.02	95.61	.00	1.21
Living in a residential college	−0.39	0.06	39.50	.00	0.68
Total weighted NCEA	0.00	0.00	0.46	.50	1.00
School decile 1-3 (low)	0.62	0.27	5.45	.02	1.86
School decile 4-7 (medium)	0.53	0.24	5.02	.03	1.70
School decile 8-10 (high)	0.50	0.24	4.43	.04	1.64
GPA Year 1	0.47	0.02	672.68	.00	1.59

Discussion

The first question that we sought to answer was whether a student’s prior academic achievement has an impact on her or his decision to participate in the PASS program? The results suggest that NCEA predicts students’ academic success in their first year, which in turn contributes to their chances progressing to their second year and completing their degree. Although NCEA predicts performance in Year 1, the data suggest that it is not only the academically “better” students who participate in PASS.

With regard to the second key question—to what degree do PASS participation and prior academic achievement have an impact on students’ academic success, as measured through the first-year GPA, first-year retention, and degree completion within 6 years—the data clearly suggest that PASS does seem to contribute to students’ success over and above their academic ability at point of entry. With regard to the third question, analyzed separately for the three equity groups (Māori students, Pacific Island students, and students who do not live in residential colleges), these findings hold as well. Our findings therefore seem to confirm U.S. and other international research findings that SI (the U.S. PASS equivalent) works for all students, irrespective of prior ability and ethnicity (see, e.g., Blanc et al., 1983; Bowles & Jones, 2004; Bowles, McCoy, & Bates, 2008; Couchman, 2008; Dawson et al., 2014; Gattis, 2000; Rath, Peterfreund, Xenos, Bayliss, & Carnal, 2007).

It could be argued that the beta values for the impact of PASS on the first year GPA are not that large. However, it needs to be remembered that in the years under investigation, PASS was available to only a limited number of units of study. Results, though, are encouraging. Hypothetically, if students attend 10 of 12 PASS sessions for four of their seven units of study during their first year, their GPA is predicted to increase by two letter grades. In other words, for some students who enter university with low NCEA entry scores, PASS might be the difference between passing some of their units or not. For the three equity groups, the benefits of PASS appear to be larger. Although the absolute numbers of participants in these three groups are smaller, the relative percentage of participants is similar. Further research of other years may shed more light on whether this is a recurrent theme. If so, this may suggest that PASS may be particularly suitable for addressing some of the differences in achievement between these three equity groups and the nonequity group. As articulated in other articles, it is important to identify what particular interventions reflect culturally responsive approaches and can be therefore inferred to support achievement for all students (Kokaua, Sopoaga, Zaharic, & van der Meer, 2014; Sopoaga & van der Meer, 2012; Sopoaga et al., 2013; van der Meer, 2011; van der Meer & Scott, 2013; van der Meer, Scott, & Neha, 2010). We understand “culturally responsive approaches” in this context to mean approaches that seek to reduce the discontinuity between students’ cultural background and teaching and support practices in the higher education environment. Some specific elements of the PASS program, such as its more interdependent collaborative focus, have been considered to resonate more with Māori and Pacific Island students (see, e.g., van der Meer & Scott, 2013).

Studies such as these are especially relevant in these times when many universities face a range of complex challenges. Most universities are currently operating in an environment of fiscal constraints. This means that allocation of scarce resources is of ongoing concern for university management. Where support is provided to enhance students’ academic achievement and the chance of progression and completion, it is important that there be sufficient evidence to suggest that this support is effective in achieving its objectives. This then is one of the reasons for this project: to provide evidence-based research to support the continuation and possible expansion of one particular program—PASS. We argue that a similar evidence-based approach for other existing learning support programs might be equally worthwhile. The more effective that programs are at enhancing students’ retention, completion, and academic success, the better it will be for institutions’ fiscal positions, reputations, and the well-being of their students and their future lives.

However, this does not have to mean that the outcome of this research has to result in the disestablishment of programs that may not currently be as effective as they could be. The research could inform a process of ongoing improvement and evaluation. Furthermore, it is important to broaden the research focus to assess other outcomes. For example, SI/PASS programs have major benefits for first-year students’ social integration by the virtue of students being able to connect with other students in an informal and student-centered environment. This connectedness with other students can contribute to students’ overall well-being, especially their mental well-being (Burton, Hess, & Becker, 2012; Larcombe et al., 2016). There are many other possible benefits as well—for example, students’ sense of belonging to the institution, culturally and otherwise (Dawson et al., 2014).

The benefits of programs such as SI/PASS extend to the second- or third-year students who act as PASS leaders for the study sessions. There is a growing worldwide interest in the benefits of students’ extracurricular activity in leadership positions (Grande & Srinivas, 2001; Higher Education Research Institute, 1996; Keup, 2010; Komives, 2006; Skalicky & Caney, 2010; van der Meer, Skalicky, & Rogan, 2016).

Limitations of this Study

Achieving a robust level of data integrity was a considerable challenge, given the changeover between a disestablished and new student administration system over the period of this study. We believe that after considerable efforts and data checking, we achieved a reasonably good level of data integrity, but we do not exclude the possibility that some individual data records may not be entirely correct, because of some rare combination of exceptions. We are confident, however, that these are very few and, considering the sample size, might have a very small effect, if at all.

Footnotes

Authors

JACQUES VAN DER MEER is the associate dean (academic and research) at the College of Education of the University of Otago. His research interests relate to peer learning, student retention, and achievement.

ROB WASS is a lecturer in the Higher Education Development Centre at the University of Otago. His research interests relate to constructivist approaches to learning, critical thinking, and Vygotsky’s concept of zone of proximal development.

STEPHEN SCOTT is the director of First-Year Experience at the University of Otago. His research interests relate to the first-year student experience and Māori achievement.

JESSE KOKAUA is a researcher in the Pacific Island Research and Student Support Research Unit at the University of Otago. He evaluates the programs provided by the unit and undertakes research topics about the factors that contribute to increased academic achievement by Pacific students.

References

Arendale

(1993). Foundation and theoretical framework for Supplemental Instruction. In Martin

D. C.

Arendale

D. R.

(Eds.), Supplemental Instruction: Improving first-year student success in high-risk courses (2nd ed., pp. 19–26). Columbia, SC: National Resource Center for the Freshman Year Experience and Students in Transition.

Arendale

(1994). Understanding the Supplemental Instruction (SI) model. New Directions for Teaching and Learning, 60(4), 11–22.

Arendale

(1997). Review of research concerning the effectiveness of SI from the University of Missouri–Kansas City and other institutions from across the United States.” In Mioduski

Enright

(Eds.), Proceedings of the 17th and 18th Annual Institutes for Learning Assistance Professionals: 1996 and 1997 (pp. 1–25). Tucson, AZ: University of Arizona University Learning Center.

Blanc

DeBuhr

Martin

(1983). Breaking the attrition cycle: The effects of Supplemental Instruction on undergraduate performance and attrition. The Journal of Higher Education, 54(1), 80–90.

Bowles

Jones

(2004). The effect of Supplemental Instruction on retention: A bivariate probit model. Journal of College Student Retention: Research, Theory & Practice, 5(4), 6.

Bowles

McCoy

Bates

(2008). The effect of Supplemental Instruction on timely graduation. College Student Journal, 42(3), 853–859.

Braxton

Milem

Sullivan

(2000). The influence of active learning on the college student departure process: Toward a revision of Tinto’s theory. Journal of Higher Education, 71, 569–590.

Burton

Hess

Becker

(2012). College students’ reasons for concealing suicidal ideation. Journal of College Student Psychotherapy, 26(2), 83–98.

Couchman

(2008). Who am I now? Accommodating new higher education diversity in Supplemental Instruction. Australasian Journal of Peer Learning, 1(1), 10.

10.

Dawson

van der Meer

Skalicky

Cowley

(2014). On the effectiveness of Supplemental Instruction: A systematic review of Supplemental Instruction and peer-assisted study sessions literature between 2001 and 2010. Review of Educational Research, 84(4), 609–639.

11.

Draper

S. W.

(2002). Tinto’s model of student retention. Retrieved from http://www.psy.gla.ac.uk/~steve/localed/tinto.html

12.

Gattis

K. W.

(2000). Long-term knowledge gains due to Supplemental Instruction in college chemistry courses. Journal of Research and Development in Education, 33(2), 118–126.

13.

Grande

Srinivas

(2001). Student leadership and activism for social change in the US. Education for Health: Change in Learning & Practice, 14(2), 198–206.

14.

Higher Education Research Institute. (1996). A social change model of leadership development: Guidebook version 3. College Park, MD: National Clearinghouse for Leadership Programs.

15.

Holland

N. E.

(2010). Postsecondary education preparation of traditionally underrepresented college students: A social capital perspective. Journal of Diversity in Higher Education, 3(2), 111.

16.

Jansen

Bruinsma

(2005). Explaining achievement in higher education. Educational Research and Evaluation, 11(3), 235–252.

17.

Jansen

van der Meer

(2007a, July). Feeling prepared for university? Perceived preparedness and expectations of prospective students. Paper presented at the 10th First-Year in Higher Education Pacific-Rim Conference: Regenerate, Engage, Experiment, Brisbane, Australia.

18.

Jansen

van der Meer

(2007b). First-year students’ expectations and perceptions of readiness before they start university. In Proceedings of the 30th Annual HERDSA Conference: “Enhancing Higher Education: Theory and Scholarship” (pp. 275–284). Hammondville, Australia: Higher Education Research and Development Society of Australasia.

19.

Keup

(2010, October). National context and institutional practice: Findings from a national survey of peer leadership experiences andoutcomes. Presentation at the National Resource Center for the First-Year Experience and Students in Transition’s Institute on Peer Educators, Indianapolis, IN.

20.

Kift

Nelson

Clarke

(2010). Transition pedagogy: A third generation approach to FYE— case study of policy and practice for the higher education sector. The International Journal of the First Year in Higher Education, 1(1), 1–20.

21.

Kokaua

Sopoaga

Zaharic

van der Meer

(2014). The development of a pre-enrolment screening tool to inform targeted support services in the first year in health sciences. The International Journal of the First Year in Higher Education, 5(1), 55.

22.

Komives

(2006). Handbook for student leadership programs. College Park, MD: National Clearinghouse for Leadership Programs.

23.

Krause

Hartley

James

McInnis

(2005). The first year experience in Australian universities: Findings from a decade of national studies. Canberra, Australia: Australian Government Publishing Service.

24.

Larcombe

Finch

Sore

Murray

C. M.

Kentish

Mulder

R. A.

. . . Williams

D. A.

(2016). Prevalence and socio-demographic correlates of psychological distress among students at an Australian university. Studies in Higher Education, 41(6), 1074–1091.

25.

Martin

(1983). Supplemental Instruction: A model for student academic support. Kansas City: University of Missouri Student Learning Center.

26.

Martin

Arendale

(1992a). Foundation and theoretical framework for Supplemental Instruction. In Martin

D. C.

Arendale

D. R.

(Eds.), Supplemental Instruction: Improving first-year student success in high-risk courses (2nd ed., pp. 41–50). Columbia, SC: National Resource Center for the Freshman Year Experience and Students in Transition.

27.

Martin

Arendale

(1992b). Review of research on Supplemental Instruction. In Martin

D. C.

Arendale

28.

Martin

Arendale

(1992c). Supplemental Instruction in the first college year. In Martin

D. C.

Arendale

(Eds.), Supplemental Instruction: Improving first-year student success in high-risk courses (2nd ed., pp. 11–18). Columbia, SC: National Resource Center for the Freshman Year Experience and Students in Transition.

29.

Martin

Arendale

(Eds.). (1992d). Supplemental Instruction: Improving first-year student success in high-risk courses (2nd ed., Vol. 7). Columbia, SC: National Resource Center for the Freshman Year Experience and Students in Transition.

30.

Martin

Arendale

(Eds.). (1994). Supplemental Instruction: Increasing achievement and retention. New Directions for Teaching and Learning No. 60. San Francisco, CA: Jossey-Bass.

31.

McGinty

D. A.

(1990). A path analysis of the effects of multiple programs on student persistence: dormitory residence, orientation, tutoring, Supplemental Instruction. Dissertation Abstracts International, 51(02), 368A.

32.

Ministry of Education. (2014). Tertiary education strategy 2014–2019. Retrieved from http://www.education.govt.nz/assets/Documents/Further-education/Tertiary-Education-Strategy.pdf

33.

Nelson

Clarke

Kift

Creagh

(2011). Trends in policies, programs and practices in the Australasian first year experience literature 2000–2010. Brisbane, Australia: Queensland University of Technology.

34.

Rath

K. A.

Peterfreund

A. R.

Xenos

S. P.

Bayliss

Carnal

(2007). Supplemental Instruction in Introductory Biology I: Enhancing the performance and retention of underrepresented minority students. CBE–Life Sciences Education, 6(3), 13.

35.

Scott

(1995). The meanings of mass higher education. Buckingham, UK: SRHE.

36.

Skalicky

Caney

(2010). PASS student leader and mentor roles: A tertiary leadership pathway. Australasian Journal of Peer Learning, 3(1), 13.

37.

Sopoaga

van der Meer

(2012). Investigating factors that influence success of Pacific students in first-year health sciences at university in New Zealand. The New Zealand Medical Journal, 125(1352), 28.

38.

Sopoaga

Zaharic

Kokaua

Ekeroma

Murray

van der Meer

(2013). Pacific students undertaking the first year of health sciences at the University of Otago, and factors associated with academic performance. The New Zealand Medical Journal (Online), 126(1384).

39.

Synco

T. M.

(2012). Background or experience? Using logistic regression to predict college retention. (Dissertation). University of Alabama at Birmingham.

40.

Tinto

(1975). Dropout from higher education: A theoretical synthesis of recent research. Review of Educational Research, 45(1), 89.

41.

Tinto

(2000). Taking student retention seriously: Rethinking the first year of college. NACADA Journal, 19(2), 5–10.

42.

Tinto

(2006, June). Taking student retention seriously. Keynote presentation and paper at Maricopa Community College District, Tempe, AZ.

43.

Tinto

(2007). Taking student retention seriously. Syracuse, NY: Syracuse University.

44.

Tinto

Goodsell

(1994). Freshman interest groups and the first-year experience: Constructing student communities in a large university. Journal of the Freshman Year Experience, 6(1), 7–28.

45.

Tinto

Pusser

(2006). Moving from theory to action: Building a model of institutional action for student success. Washington, DC: National Postsecondary Education Cooperative.

46.

Toews

M. L.

Yazedjian

(2007). College adjustment among freshmen: Predictors for White and Hispanic males and females. College Student Journal, 41(4), 891.

47.

van der Meer

. (2011). Māori and Pasifika students’ academic engagement: What can institutions learn from the AUSSE data? In Radloff

(Ed.), Student engagement in New Zealand universities (pp. 1–10). Melbourne, Australia: Australian Council of Educational Research.

48.

van der Meer

. (2012). Students’ note-taking challenges in the twenty-first century: Considerations for teachers and academic staff developers. Teaching in Higher Education, 17, 13–23.

49.

van der Meer

Jansen

. (2008, August). An international comparative study on first-year students’ expectations and self-perceived readiness. Paper presented at the Polishing the Silver: The 30th EAIR Annual Forum, Copenhagen, Denmark.

50.

van der Meer

Jansen

Toorenbeek

. (2010). “It’s almost a mindset that teachers need to change”: First-year students’ need to be inducted into time management. Studies in Higher Education, 35, 777–792.

51.

van der Meer

Scott

. (2009). Students’ experiences and perceptions of peer assisted study sessions: Towards ongoing improvement. Australasian Journal of Peer Learning, 2(1), 19.

52.

van der Meer

Scott

. (2013). Including everyone: A peer learning program that works for under-represented minorities? The International Journal of the First Year in Higher Education, 4(1), 85–94.

53.

van der Meer

Scott

Neha

. (2010). Retention of first-year Māori students at university Mai Review, 2010(2), 1–14.

54.

van der Meer

Skalicky

Rogan

. (2016). “I didn’t just want a degree”: Students’ perception about benefits from participation in student leadership programmes. Under review.