Applying the Rasch Growth Model for the Evaluation of Achievement Trajectories

Abstract

Considerable interest lies in the growth in educational achievement that occurs over the course of a child’s schooling. This paper demonstrates a simple but effective approach for the comparison of growth rates, drawing on a method first proposed some 80 years ago and applying it to data from the Australian National Assessment Program. The methodology involves the derivation of a ‘meta-metre’ – a quantitative mode of variation in growth – which permits comparison between groups defined by time-invariant characteristics. Emphasis is placed upon the novel characteristics of the method and the valuable information it can provide. Unlike complex modelling procedures, the approach provides a parsimonious model of growth suited to comparisons between groups.

Keywords

Growth psychometrics logistic models Rasch longitudinal NAPLAN measurement

Introduction

Recent government, public policy and academic recommendations have highlighted the need to measure student achievement over the course of schooling, rather than concentrating on a specific timepoint (e.g. Department of Education and Training, 2018; Goss et al., 2018; Masters, 2020; McGaw et al., 2020).

Within the Australian educational landscape, the focus of school-age reporting of achievement has traditionally centred on performance at a single timepoint as measured by the National Assessment Program – Literacy and Numeracy (NAPLAN) (Department of Education and Training, 2018). While such approaches provide meaningful information on educational achievement relative to the population, they fail to address variation in the level of growth for students with differing levels of academic aptitude. They also mask the extent to which student proficiency increases relative to previous performance, particularly for students entering school with low levels of literacy and numeracy. Interest thus lies in a shift away from static measures to a focus on achievement over the course of a child’s schooling. Such views are highlighted in several recent reports (e.g. Goss et al., 2018; Masters, 2020; McGaw et al., 2020) and emphasise a need to focus on variation in levels of achievement in early years of schooling and the subsequent progress that takes place over time – features, respectively, referred to as a student’s ‘initial status’ and ‘growth rate’.

One area of interest raised by Masters (2020) is variability in growth between students from different backgrounds, with the possibility of increasing equity and inclusivity through targeted interventions. To implement such recommendations, whether through policy or pedagogy, methods appropriate for analysing growth between groups are required. Growth modelling approaches that accommodate varying starting points and trajectories, such as the Rasch Growth Model (RGM), are a logical consideration.

To effectively measure the rate of growth across schooling, appropriate methods are required. Growth-oriented methodological approaches accommodate varying starting points and trajectories but are often complex in their specification and interpretation (Curran et al., 2010), requiring specialist statistical software. An alternative methodology, first proposed some 80 years ago, involves the derivation of ‘salient features of growth’ (Rao, 1958, p.1). This paper demonstrates how the application of this approach to a set of longitudinal student assessment data can provide statistics that lend themselves to a presentation of easily comparable linearised results. The approach provides valuable information regarding the overall trajectories of groups of students and demonstrates utility in the evaluation of achievement over time. The visual representation of the model results, as well as their summative properties, makes the RGM particularly useful for communication of information regarding student growth.

The RGM proposes a mechanism for evaluating differences in growth across a variety of phenomena (Olsen, 2003). One application for which this approach may be appropriate is in the modelling of achievement trajectories for school-age children, an area for which Rasch’s probabilistic models are most commonly associated (e.g. Rasch, 1961). Since children’s literacy and numeracy abilities vary when they enter schooling and as a result of learning through schooling, it is feasible to measure and evaluate differences in these two features. To do so, one typically seeks to manifest a construct of interest through a set of assessment items (i.e. to elicit a behaviour which demonstrates the psychological phenomena, such as a performance of reading ability demonstrated through a reading test) and subsequently models the variation in change over time by applying a set of methodological approaches commonly referred to as growth modelling (Williamson, 2016). In doing so, the trajectories of achievement for individuals and groups may be compared.

The following paragraphs provide a summary of the interdisciplinary features of the RGM, its relationship to educational assessment and measurement, and its embedding within Rasch’s more widely recognised work. This approach is then described and applied, with the statistical and descriptive features of the model demonstrated using examples from NAPLAN.

The Rasch growth model

The RGM was first proposed by Georg Rasch in 1940 and subsequently articulated in a series of lectures presented at the 1951 meeting of the International Statistical Institute (Olsen, 2003). In both instances, Rasch’s focus was on the physiological growth of animals and an attempt to derive a ‘simple elementary growth law… [with] time expressed in the physiologically adequate unit of time’ (Olsen, 2003, p.65). Rasch later applied the same methodology in the analysis of economic data, at which time he emphasised the key statistical features associated with the model (Rasch, 1972). In each application of the RGM, irrespective of the domain to which it was applied, the core feature lay in the capacity to derive the primary features of growth for the purpose of efficient comparisons between groups (Rao, 1958).

The RGM approach attempts to derive an estimated rate of growth for each individual, proportional to the increase over time for the population (Rao, 1958). Underlying this is an assumption that observations represent functional changes alongside a continuous variable, time. Methods of this type provide a parsimonious summary of individual differences (McArdle & Nesselroade, 2003), albeit with known limitations in measurement at the individual level relative to more complex methodologies. The RGM aims to identify the principal sources of variation in growth, conveniently expressed as a set of derived variables. Such an approach is consistent with time-ordered analyses that incorporate and acknowledge the role of both individual and group-level differences (Duncan & Duncan, 1995). This has relevance within the context of evaluating differences in developmental trajectories, a point highlighted by Meredith and Tisak (1990) in their call for wider recognition of such procedures by both biological and behavioural researchers.

The Rasch measurement model

The Rasch Measurement Model (RMM) serves a comparative function in which data derived from a testing instrument (e.g. an assessment) can be compared to expectations set under fundamental principles of measurement (Andrich, 2004). These fundamental principles relate to the specific structure of relations between attributes and those relations being wholly attributable to the measurement act itself (i.e. not derived from other measures). Such properties permit a comparison in the degree of differences between two measures (i.e. additivity), but not in the ratio of them (i.e. multiplicativity). In this way, such relations may be considered similar to what Stevens (1946) described as ‘interval scale measurement’. Such features are common to the physical sciences (e.g. temperature as measured in Fahrenheit or Centigrade) and readily permit analysis using linear statistics.

The RMM imposes a priori restrictions on both the model and parameters used to account for the observed structure of data (Andrich & Marais, 2019). In this way, a data set – the numerical summarisation of the qualitative aspects of a testing instrument – can be said to meet the requirements for measurement when it conforms to the structure specified by the RMM, thus permitting quantitative conclusions (Duncan, 1984). It can be argued that the methodology applies an approach consistent with that espoused by Kuhn (1977), in which the merit of the procedure lies not in appraising the appropriateness of a set of models to fit the data, but instead assessing whether observed data suitably represent features expected under fundamental measurement through evaluation of conformity with a pre-specified model.

In the dichotomous RMM, the probability that a person will respond correctly to an item is dictated by the interaction between the person taking the test and the item used to measure the underlying construct. The person’s estimated level of ability determines the likelihood they will respond correctly to the item, given the item’s level of difficulty. This is achieved by calculating the difference between ability and difficulty under certain algebraic constraints. Such an estimate of ability can be conceptualised as the individual’s location along a trait continuum, varying according to their capacity.

A test conforming to the RMM can be used to ascertain an estimate of ability on a construct that is independent of the specific set of items used to make that assessment. This fundamental feature of the RMM – known as specific objectivity – states that the comparison of two people should be independent of the items used to assess them, and similarly that the comparison of two items should be independent of the individuals used for the comparison (Rasch, 1977). The algebraic separation of person and item parameters underlies this notion, ensuring that each parameter can be eliminated in its counterpart’s estimation. Interestingly, this same feature – consistent with requirements for objective measurement – is present within the RGM (Olsen, 2003).

Measuring growth in educational achievement

The RMM can be implemented to evaluate whether quantities under investigation conform to fundamental properties of measurement when growth trajectories are estimated (Williamson, 2017). By applying the RGM to valid and reliable measures of educational achievement that meet the requirements of the RMM, an attempt can be made to derive a set of summative parameters that characterise the growth trajectories of both individuals and groups. This is undertaken under the assumption that performances observed from psychometrically-sound measures of educational achievement represent functional changes associated with skill acquisition through structured learning (i.e. time spent at school). Such an assertion is consistent with developmental theories that posit an asymptotic decrease in the rate of educational achievement over time (Francis et al., 1996). However, like all approaches that are guided by substantive theory, proposed models require subjection to empirical evaluation through data analysis (Williamson, 2016).

Aim, objective and research question

This article addresses the following question: Can Rasch’s Growth Model be used to measure differences in achievement trajectories by representing growth as a function of time? Meaningful application of the RGM will be demonstrated through the use of an example comparing the initial status and growth rates of students in separate Australian states and territories for the reading domain of NAPLAN (i.e. the domain with the highest latent correlation between the NAPLAN assessment domains). Emphasis is placed upon the valuable information that can be derived about achievement trajectories using this model, as demonstrated by visualisation and comparison of growth parameters that are likely to be interpretable by a range of audiences.

Methodology

Data sources

NAPLAN provides annual, point-in-time information regarding student achievement across four domains – reading, numeracy, conventions of language and writing – at the level of the student, school, states and territories (referred to collectively as jurisdictions), and Australia as a whole. NAPLAN assessments are completed by students in Grades 3, 5, 7 and 9. The information collected as part of the program facilitates the monitoring and reporting of performance of specific groups and across all six states and two territories. While NAPLAN assesses performance across both primary and secondary education, the focus currently lies on reporting levels of achievement based on observations at a single timepoint, with limited supplementary reporting at two timepoints – between grades three and five and between grades seven and nine (Australian Curriculum, Assessment and Reporting Authority [ACARA], 2013). By adopting a longitudinal approach to data analysis that incorporates individual-level data measured across all timepoints, a more cogent understanding of educational achievement trajectories in each jurisdiction can be developed. Authorisation for additional analysis and research using this data was provided through formal ACARA channels.

Measures

The measures of interest – student achievement in reading, which typically demonstrates the highest latent correlation with other NAPLAN domains and similarly correlates with other standardised reading assessments, such as PISA (see e.g. Lumsden et al., 2015) – took the form grade-specific weighted-likelihood estimates (Warm, 1989) of reading ability measured in logits, derived using the RMM, as applied across multiple years of NAPLAN assessment (ACARA, 2020). This fulfilled the requirement stipulated by Williamson (2017) that fundamental properties of measurement be attributable to base quantities when growth trajectories are estimated. These estimates were equated onto the NAPLAN reporting scale, which became the unit of analysis. The NAPLAN reporting scale spans across all tested grades (i.e. Grades 3, 5, 7 and 9) and is standardised using a mean of 500 and standard deviation of 100, with scores ranging from approximately 0 to 1000 (ACARA, 2020). This process places reported results across assessment years (e.g. 2013, 2015, 2017, 2019) on the same scale, thus permitting comparability.

Data preparation and processing

Prior to model implementation, data appropriate for longitudinal analysis was sourced and prepared. Separate data sets containing matched ‘gain’ data¹ from the 2013, 2015, 2017 and 2019 NAPLAN assessments were provided to the author for the purpose of data linkage and analysis. As current reporting and analysis techniques do not necessitate the linkage of student data across NAPLAN assessments, a considerable degree of variability exists in the consistency and use of common student identifiers. As a result, a small selection of the large number of variables pertaining to each student were used in the matching process. This was undertaken with the view to maximise the likelihood of a correct correspondence in assessment information included across timepoints.

Data was first partitioned such that only cases from the appropriate grade and year were retained (i.e. only Grade 3 data was retained from the 2013 dataset, only Grade 5 data was retained from the 2015 dataset, etc.). The following variables were subsequently used in the matching process:

- Student Identifier;²

- Date of Birth;

- School Identifier associated with the 2017 (Grade 7) and 2019 (Grade 9) dataset;³

- Previous NAPLAN Reading score.

As the RGM requires complete data across all timepoints of interest, cases with missing data were removed. There was considerable variability in the number students with complete data across all four timepoints in each jurisdiction. To address these imbalances and potential privacy-related issues, a senate-weighted, random sampling procedure was then undertaken in which 1000 students from each jurisdiction with complete data were selected. Due to issues associated with data access and its linkage, data from one jurisdiction was excluded, resulting in a total of 7000 students being included in the analysis sample.

A single time-invariant variable was selected for further analysis – jurisdiction. Jurisdictional data was re-categorised using an anonymised, numerical identifier. Comparisons were subsequently made of descriptive statistics for each of the variables relative to the 2019 ‘gain’ data set, with approximate equivalence found between the two datasets. Computation of the level of negative ‘gain’ – in which student progress decreases over the measures of interest – was also undertaken. During this process, distribution checks for concordance with assumptions of analysis of variance (ANOVA) – including normality of distributions – were undertaken.

Analytic methods

The RGM proposes the existence of a quantitative mode of variation in growth, referred to as the ‘meta-metre’, that is common to all individuals, providing relevant information for the comparison of average growth curves (Rao, 1958). In deriving this ‘age transforming function’ as Rasch first described it (Olsen, 2003), we assert that the rate of growth of an individual is directly proportional to the meta-metre, thus allowing comparisons characterised by linear relationships (Rao, 1958). Such relationships can be expressed in terms of a single growth parameter, retaining dynamic consistency, whereby individual and group-level differences can be expressed by the same mathematical function (Keats, 1980). In this way, while there exists a trajectory for the population, each individual retains an estimate that varies relative to the group as a whole (McNeish & Matta, 2018). These variables are defined by the data, not a priori, allowing for the modelling of individual growth trajectories.

In the context of educational achievement, the RGM asserts that

Y_{n t} = a_{n} + b_{n} τ (t)

(1)

where

Y_{n t}

is domain-specific achievement for student n at timepoint t;

$a_{n}$ is the overall achievement over the time period of interest;

$b_{n}$ is the rate of growth in achievement for student n; and

$τ (t)$ is the time transforming function, referred to by Rasch as the meta-metre (Olsen, 2003).

This can be conceptualised as a single structural equation expressing growth rate, with coefficients $a_{n}$ and $b_{n}$ specifying the growth rate parameters (Stone, 2020). Unlike behavioural models which utilise multiple structural equations – typically requiring specialised software and knowledge for their specification and interpretation (Curran et al., 2010) – the RGM models the group-level effect on the dependent variable, given the meta-metre (Stone, 2020). Through this approach, the RGM asserts that the derived meta-metre should be equivalent for all individuals, allowing one to generalise growth in academic achievement and thus meaningfully represent deviations from it (Stone, 2020). Estimation of the meta-metre and of the parameter estimates used for comparing relative rates of growth are presented in Appendix 1. These estimates permit the comparison of individuals and groups of interest. The independent separation of these parameters, which incorporate the repeated measurements, allows the use of univariate procedures for significance testing. Independent samples t-test, one-way ANOVA and Tukey’s honestly significant difference (HSD) post hoc test were used to reveal the significance and degree of differences between parameter estimates between jurisdictions. These procedures provided tests of the null hypothesis that rates of growth and their initial status were the same (Greenland et al., 2016).

The linear relationship between growth and time, conditional on the meta-metre, allowed the representation of growth to be presented as straight lines on a plot. These straight lines are easily interpretable as continuous changes, serving useful purposes in the identification of trends (Peebles & Ali, 2015). This approach avoids the issue of non-developmental behaviour (i.e. negative growth) characterised in quadratic representations (Williamson, 2016). As per the recommendations outlined in Nese et al., (2013), sets of growth plots, which permit more readily detected differences in growth, were subsequently used to present variation between groups.

Differences in achievement may also be represented as a function of time, providing another effective and interpretable way to visualise growth trajectories (Singer & Willett, 2003). A feature consistent with the RGM is the capacity to transform time such that – by a common transformation – individual growth curves are linearised, with slight variations attributable to error (Rao, 1958). This can be achieved by taking each timepoint to its natural logarithm (i.e. the logarithm of each timepoint taken to the base of the constant e). To paraphrase Rasch, this process – utilising the meta-metre – allows one to measure time in a particular way, allowing a uniform description of growth curves for all individuals considered (Olsen, 2003).

Results

Estimates of initial achievement ${\hat{A}}_{n}$ and rate of growth in achievement ${\hat{B}}_{n}$ were calculated for all individuals within the sample, as well as for the overall sample and each jurisdiction. These calculations were performed using SPSS (IBM Corp., 2019), however, any non-specialist statistical software may be utilised.

As outlined in the Analytic Methods section, the estimated normalised relative rate of growth for the overall sample had a mean of 1.0 (SD = 0.45) and the estimated normalised achievement over time (i.e. initial status) had a mean of 0.0 (SD = 251.41), consistent with expectations. Overall grade means evidenced an asymptotic decrease in the rate of growth over time (i.e. a decrease in the level of growth occurring between grades over time), prototypical of educational achievement trajectories, as shown in Table 1.

Table 1.

Grade means for each jurisdiction in NAPLAN Scale Scores.

Jurisdiction	Grade 3 Mean	Grade 5 Mean	Grade 7 Mean	Grade 9 Mean
1	419.30	501.06	549.37	594.89
2	441.70	517.22	562.39	596.01
3	419.76	498.11	543.97	572.01
4	417.20	493.41	547.16	579.77
5	418.03	505.50	548.43	579.89
6	377.81	467.18	516.44	551.93
7	448.10	529.48	570.25	601.79
ALL	420.27	501.71	548.29	582.33

Differences in jurisdictional growth

Comparison of jurisdictional growth parameters

Grade means for each jurisdiction evidenced a similar asymptotic decrease in the rate of growth over time as that evidenced overall, as shown in Table 1. The normalised relative growth rate and initial status estimates for jurisdictions are presented in Table 2. These provide an indication of the comparability of jurisdictions in their level of growth and the level from which growth commences, which were subsequently tested for significance.

Table 2.

Comparison of growth rate and initial status for jurisdictions.

Jurisdiction	Growth Rate (SD)	Initial Status (SD)
1	1.05 (0.44)	−22.64 (251.41)
2	0.94 (0.44)	45.11 (239.40)
3	0.95 (0.46)	20.52 (259.34)
4	0.99 (0.43)	3.45 (242.03)
5	1.02 (0.42)	−12.22 (233.11)
6	1.08 (0.51)	−77.04 (271.88)
7	0.96 (0.43)	42.83 (239.39)

Noting the linearisation of growth trajectories and independent separation of parameters, one-way ANOVA was used to ascertain whether observed differences in jurisdictional growth rates and initial status were statistically significant. There was a statistically significant difference observed between jurisdictional growth rates (F(6,6993) = 14.16, p < .05). Tukey’s HSD post hoc test was applied to reveal the jurisdictions to which the statistically significant differences in growth rate applied and their degree. As shown in Table 3, 11 of the 21 paired-comparisons between jurisdictions showed significant differences in growth rate, the largest being 0.14 units.

Table 3.

Paired-comparisons in jurisdictional growth rate as determined by Tukey’s HSD.

	1	2	3	4	5	6	7
1	—
2	0.11^a	—
3	0.10^a	−0.01	—
4	0.06^a	−0.04	−0.04	—
5	0.03	−0.08^a	−0.07^a	−0.04	—
6	−0.03	−0.14^a	−0.13^a	−0.10^a	−0.06	—
7	0.09^a	−0.02	−0.01	0.02	0.06^a	0.12^a	—

^aMean difference is significant at the 0.05 level.

There was also a statistically significant difference observed between the initial status’ of jurisdictions (F(6,6993) = 29.43, p < .05). Tukey’s HSD post hoc test was applied to reveal the jurisdictions to which the statistically significant differences in initial status applied and their degree. As shown in Table 4, 13 of the 21 paired-comparisons between jurisdictions showed significant differences in initial status, the largest being 122.15 units.

Table 4.

Paired-comparisons in initial status between jurisdictions as determined by Tukey’s HSD.

	1	2	3	4	5	6	7
1	—
2	−67.75^a	—
3	−43.15^a	24.59	—
4	−26.09	41.66^a	17.07	—
5	−10.42	57.32^a	32.73	15.67	—
6	54.40^a	122.15^a	97.56^a	80.49^a	64.82^a	—
7	−65.47^a	2.28	−22.31	−39.38^a	−55.05^a	−119.87^a	—

^aMean difference is significant at the 0.05 level.

Visual representation of differences between jurisdictions

It is conceivable to represent differences in means simply by regressing the grade means for each jurisdiction on the overall sample means, as displayed in Figure 1. These correspond to the values obtained from Table 1.

Figure 1.

Grade means of each jurisdiction plotted against the overall sample means.

Alternatively, the linear relationship between achievement and time, conditional on the meta-metre, permits a representation of growth for separate groups as straight lines. Figure 2, Figure 3, Figure 4 and Figure 5 show the comparison of grade means for select pairs of jurisdictions, plotted against the transformed means. The line representing growth for each jurisdiction can be characterised by two parameters, identified within the equation on each of the plots, which are consistent with the growth rate and initial status estimates displayed in Table 2 with their corresponding x-axes obtained as the transformation of the sample means obtained from Table 1. By presenting the comparison of a pair of jurisdictions in each plot, the individual trajectories of growth for each jurisdiction may be compared.

Figure 2.

Grade means of jurisdictions 1 and 2 plotted against the transformed means.

Figure 3.

Grade means of jurisdictions 1 and 6 plotted against the transformed means.

Figure 4.

Grade means of jurisdictions 2 and 3 plotted against the transformed means.

Figure 5.

Grade means of jurisdictions 6 and 7 plotted against the transformed means.

Growth trajectories can similarly be represented as a function of time. Attempting to characterise the grade means of jurisdictions from Table 1 as function of linear time (i.e. across the four timepoints) resulted in poor model fit, evidenced by large residuals and shown in Figure 6. An alternative approach is to fit a quadratic model representing a curvilinear functional form, thereby minimising residuals and increasing the variance explained by the model due to the monotonically decreasing rate of growth. This representation is shown in Figure 7.

Figure 6.

Grade means of each jurisdiction plotted against time, assuming linear growth.

Figure 7.

Grade means of each jurisdiction plotted against time, assuming curvilinear (quadratic) growth.

Discussion

Characterising growth in two parameters

By applying the RGM to a longitudinal data set containing individual measures of educational achievement in the form of NAPLAN scale scores and using time-invariant variables pertaining to student jurisdiction, it was possible to derive a pair of parameters by which growth rates and initial status could be compared – the growth rate and initial status. These parameters are single-value summaries of the quality of the trajectories, allowing for a relatively straight-forward approach to comparison and interpretation. Utilising the independent separation of these parameters, univariate tests of significance and multiple comparisons (i.e. post hoc tests) were applied to determine the degree to which differences were statistically significant. These two parameter estimates were subsequently used to visually represent the variation in growth trajectories between each jurisdiction.

The degree of variability in the initial status and growth of jurisdictions was considerable. By combining the use of statistical methods and visual representation, it was possible to observe the degree to which differences exist simply by comparing the linear trajectories. Results, presented graphically, visually emphasised areas of variability while also demonstrating qualities associated with the RGM. For instance, Figure 2 clearly portrays that while jurisdiction 2 starts with a higher rate of mean achievement as reflected in the grade three mean, the rate at which achievement grows in jurisdiction 1 was considerably greater. Confidence in this assertion was provided via the use of ANOVA. Such findings can be contrasted against those shown in Figure 3, in which the rates of growth between jurisdiction 1 and 6 do not differ (i.e. no statistically significant difference in growth rate was found) as exemplified by the parallel trajectories. Such descriptive measures provide a visually alluring method that can be used to effectively communicate key observations to a range of stakeholders. Furthermore, while descriptive in their presentation, the results point to areas for further investigation using both qualitative and quantitative methods.

It is noted that there exists a wide variety of current growth models that incorporate individual variability and permit the modelling of complex developmental theories (Duncan & Duncan, 1995), including those embedded within an IRT framework (e.g. von Davier et al., 2011). Such approaches often draw on the traditions of hierarchical linear modelling (HLM; Raudenbush & Bryk, 2002) and structural equation modelling (SEM; Meredith & Tisak, 1990). While the RGM serves as an effective, niche method for summarising and describing results – providing what Goss et al. (2018) describe as a need for approaches that accommodate non-linear rates of growth for the purpose of comparing relative student progress – additional value could be sought through the comparison of group estimates derived under different methodologies. In the case of the RGM, the linearisation of growth rates via the instantiation of the meta-metre, conforming to properties consistent with objective measurement, provides a benefit that may not be realised in alternative approaches.

Noting the use of the meta-metre in deriving comparative estimates, a focus for further research lies in the impact of violations of this time transforming function, and the conditions under which these are likely. While Rao (1958) describes a process in which the existence of a common transformation can be empirically tested, the degree to which such violations may impact parameter interpretation has not been investigated. Due to the novelty of these methods, further inquiry into such conditions would be an appropriate next step in evaluating the application of the RGM. Future applications would also benefit from exploration into the error terms associated with the RGM parameter estimates. Citing Brody (1993), Stone (2020) highlights that the approach used in the model essentially averages over incidentals (i.e. individual variation and error) with the view to overwhelm individual levels of variation and move to a superordinate level of aggregation. While this may provide appropriate summative properties, the implications of doing so – as well as the consequences of the incorporation of measurement error into specific measurement types such as weighted-likelihood estimates (von Davier, Gonzales, & Mislevy, 2011) – requires evaluation.

Limitations of the RGM

While the RGM provides an effective method through which group growth can be described and compared, the model itself is not explanatory. The purpose of the model is largely descriptive; therefore, no attempt is made to provide clarity on the multitude of possible factors contributing to observed differences. While providing an effective method for reporting and lending itself to use as a preliminary step in research activities, the testing of complex theories is reserved for alternative models. As broad sets of such methodological approaches exist, it is recommended that further research compare estimates derived from such models against those of the RGM, while also considering the properties associated with their outcomes.

A further limitation of the RGM is the requirement for complete longitudinal data. While the model retains dynamic consistency (Keats, 1980) as a series of differential equations modelling rate of change as a function of the state of a variable, the incorporation of individual differences in the estimates of group-level summative parameters results in a requirement for non-missing data. One outcome of this is the possibility of correlations between rates of attrition and other variables of interest. While the current application pertains to a census-based assessment with very high participation rates (i.e. greater than 95%; ACARA, 2019), the possibility remains that high rates of withdrawal or attrition could feasibly give rise to systematic biases within sub-groups that may be explored, thus limiting the interpretability of results.

Similarly, while the present analysis was undertaken predominantly for demonstrative purposes, it should be noted that it was applied to a limited data set. While efforts were made to retain consistency with the original data through a comparison of the distribution of variables of interest, the analysis and results may include sampling biases that obscure true results. For example, as no data was available one jurisdiction, the present derivations are based on a sub-set of the total population. Equally, there were data linkage challenges that resulted in a loss of cases through the matching of students across each NAPLAN cycle – 2013, 2015, 2017 and 2019. To resolve this, the introduction of universal student identifiers for the purposes of data linkage would likely be required, particularly if such approaches were to be implemented at scale. While recommendations put to government have advocated for the implementation of these to serve the needs of longitudinal approaches (e.g. Department of Education and Training, 2018), such decisions may require further ethical examination prior to implementation (Arnold, 2013).

Conclusion

This research project investigated the application of the RGM for evaluating differences between jurisdictions. This came in response to calls for reporting of progress in student achievement throughout the course of schooling. It utilised the summative properties of the RGM and its capacity to represent growth in a manner interpretable to a range of audiences (e.g. educational researchers, statisticians and policy makers), to evaluate variation in NAPLAN reading achievement between jurisdictions.

Drawing on the desire to measure growth over the course of schooling, this process of presenting results visually, supported by significance testing, permits an ease of use for both government and academic audiences.

Despite the limitations associated with the uniqueness of this under-utilised method, the present investigation has provided support for the effectiveness and efficiency of the approach to growth modelling first proposed by Rasch. It has demonstrated that the RGM allows for the efficient comparison of jurisdictions in a manner that is both statistically rigorous and accessible. Such an approach provides both a novel and effective means by which growth may be reported and an avenue for further investigation into group-level differences via qualitative and quantitative research.

Footnotes

Acknowledgements

The idea for the application of the Rasch Growth Model to National Assessment Program – Literacy and Numeracy (NAPLAN) data took root following a presentation to the Australian Curriculum, Assessment and Reporting Authority (ACARA) by Professor David Andrich in July 2020. This presentation discussed the measurement of growth by the Danish Mathematician Georg Rasch. Rasch had applied his approach to growth of physiological variables, such as weight, which have a natural origin. Andrich showed how Rasch’s approach could be adapted to measured variables with an arbitrary origin, such as those derived using the measurement model carrying his name. Correspondence between the author and Andrich continued throughout the period of the development of the manuscript, with particular assistance provided in the derivation of the ‘meta-metre’. I extend my thanks and acknowledgement to David for his support and encouragement.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Leigh Cameron Patterson

Notes

Appendix

It is possible to estimate a meta-metre for the comparison of rates of growth (Andrich et al., 2022) by taking the mean measurements at each timepoint t (2)

(\sum_{n = 1}^{N} Y_{n t}) / N = (\sum_{n = 1}^{N} a_{n t}) / N + [(\sum_{n = 1}^{N} b_{n t}) / N] τ (t)

which can be expressed as (3)

{\bar{Y}}_{. t} = {\bar{a}}_{.} + {\bar{b}}_{.} τ (t)

for the overall sample, and (3a)

{\bar{Y}}_{. a t} = {\bar{a}}_{. a} + {\bar{b}}_{. a} τ (t)

(3b)

{\bar{Y}}_{. b t} = {\bar{a}}_{. b} + {\bar{b}}_{. b} τ (t)

for any specific sub-groups (e.g. a and b) of interest, allowing free estimation of the meta-metre

τ (t)

{\bar{Y}}_{. t} = {\bar{a}}_{.} + {\bar{b}}_{.} τ (t)

{\bar{Y}}_{. t} - {\bar{a}}_{.} = {\bar{b}}_{.} τ (t)

(4)

\hat{τ} (t) = \frac{{\bar{Y}}_{. t} - {\bar{a}}_{.}}{{\bar{b}}_{.}}

by substituting

\hat{τ} (t)

from (4) into (1), as

Y_{n t} = {\hat{a}}_{n} + {\hat{b}}_{n} \hat{τ} (t) = {\hat{a}}_{n} + {\hat{b}}_{n} [\frac{{\bar{Y}}_{. t} - {\bar{a}}_{.}}{{\bar{b}}_{.}}] = {\hat{a}}_{n} + ({\hat{b}}_{n} / {\bar{b}}_{.}) ({\bar{Y}}_{. t}) - ({\hat{b}}_{n} / {\bar{b}}_{.}) ({\bar{a}}_{.})

(5)

= {\hat{a}}_{n} - ({\hat{b}}_{n} / {\bar{b}}_{.}) ({\bar{a}}_{.}) + ({\hat{b}}_{n} / {\bar{b}}_{.}) ({\bar{Y}}_{. t})

Importantly, this can be expressed as (6)

Y_{n t} = {\hat{A}}_{n} + {\hat{B}}_{n} ({\bar{Y}}_{. t})

where (7)

{\hat{A}}_{n} = {\hat{a}}_{n} - ({\hat{b}}_{n} / \hat{{\bar{b}}_{.}}) (\hat{{\bar{a_{.}}}_{.}})

is the estimated normalised, relative achievement over time (i.e. initial status); and, (8)

{\hat{B}}_{n} = ({\hat{b}}_{n} / \hat{{\bar{b}}_{.}})

is the estimated normalised, relative rate of growth. This being due to (9)

{\bar{A}}_{.} = [\sum_{n = 1}^{N} A_{n}] / N = ([\sum_{n = 1}^{N} a_{n}] / N) - 1_{.} ({\bar{a}}_{.}) = {\bar{a}}_{.} - {\bar{a}}_{.} = 0

(10)

\hat{{\bar{B}}_{.}} = [\sum_{n = 1}^{N} \hat{{\bar{B}}_{n}}] / N = ([\sum_{n = 1}^{N} b_{n}] / N) {/ \bar{b}}_{.} = {\bar{b}}_{.} / {\bar{b}}_{.} = 1

It is noteworthy that it is feasible to estimate ${\hat{A}}_{n}$ and ${\hat{B}}_{n}$ either by regressing $Y_{n t}$ on $(\hat{Y_{. t}})$ , in which ${\hat{A}}_{n}$ and ${\hat{B}}_{n}$ are dependent, or by estimating ${\hat{B}}_{n}$ independently of ${\hat{A}}_{n}$ and then estimating ${\hat{A}}_{n}$ as expressed by (11)

y_{n t} = Y_{n t} - Y_{n (t - 1)} = A_{n} + B_{n} {\bar{Y}}_{. t} - {[A}_{n} + B_{n} {\bar{Y}}_{. (t - 1)}], t = 1,2,3 \dots T

And the estimate

{\hat{y}}_{n t}

given by

{\hat{y}}_{n t} = {\hat{A}}_{n} + {\hat{B}}_{n} {\bar{Y}}_{. t} - [{\hat{A}}_{n} - {\hat{B}}_{n} {\bar{Y}}_{. (t - 1)}], t = 1,2,3 \dots T = {\hat{B}}_{n} {\bar{Y}}_{. t} - {\hat{B}}_{n} {\bar{Y}}_{. (t - 1)} = {\hat{B}}_{n} ({\bar{Y}}_{. t} - {\bar{Y}}_{. (t - 1)})

(12)

= {\hat{B}}_{n} {\bar{Y}}_{. t}

where (13)

{\hat{B}}_{n} = \sum_{t = 1}^{T} y_{n t} {\bar{y}}_{. t} / \sum_{t = 1}^{T} {\bar{y}}_{. t}^{2}

(14)

{\hat{A}}_{n} = {\bar{Y}}_{n .} - \hat{B}_{n} \bar{Y_{. .}}

References

Andrich

(2004). Controversy and the Rasch model: A characteristic of incompatible paradigms? Medical Care, 42(1 Suppl), I7-I16. https://doi.org/10.1097/01.mlr.0000103528.48582.7c

Andrich

Marais

(2019). A course in Rasch measurement theory: Measuring in the educational, social and health sciences. Springer. https://doi.org/10.1007/978-981-13-7496-8

Andrich

Marais

Sappl

(2022). Adapting Rasch measurement to the study of growth: A meta metre for some intelligence and attainment tests. Springer Nature. [In press]

Arnold

(2013). National student identification (but not quite yet)? Privacy Law Bulletin, 9(1), 111.

Australian Curriculum, Assessment and Reporting Authority . (2013). My School technical report: My School student gain. https://docs.acara.edu.au/resources/My_School_Student_Gain_Technical_Information.pdf

Australian Curriculum, Assessment and Reporting Authority . (2019). NAPLAN Achievement in reading, writing, language conventions and numeracy: National report for 2019. https://nap.edu.au/docs/default-source/default-document-library/2019-naplan-national-report.pdf

Australian Curriculum, Assessment and Reporting Authority . (2020). NAPLAN 2019 technical report. https://nap.edu.au/docs/default-source/resources/naplan-2019_technical-report_final.pdf

Brody

T. A.

(1993). The philosophy behind physics. In DeLaPena

Hodgson

(Eds), . Springer. https://doi.org/10.1007/978-3-642-78978-6

Curran

P. J.

Obeidat

Losardo

(2010). Twelve frequently asked questions about growth curve modeling. Journal of Cognition and Development: Official Journal of the Cognitive Development Society, 11(2), 121-136. https://doi.org/10.1080/15248371003699969

10.

Department of Education and Training . (2018). Through growth to achievement: Report of the review to achieve educational excellence in Australian schools. Australian Government. https://www.education.gov.au/quality-schools-package/resources/through-growth-achievement-report-review-achieve-educational-excellence-australian-schools

11.

Duncan

O. D.

(1984). Rasch measurement: Further examples and discussion. In Turner

C. F.

Martin

(Eds.), Surveying subjective phenomena (2, pp. 367-401). Russell Sage Foundation.

12.

Duncan

T. E.

Duncan

S. C.

(1995). Modeling the processes of development via latent variable growth curve methodology. Structural Equation Modeling: A Multidisciplinary Journal, 2(3), 187-213. https://doi.org/10.1080/10705519509540009

13.

Francis

D. J.

Shaywitz

S. E.

Stuebing

K. K.

Shaywitz

B. A.

Fletcher

J. M.

(1996). Developmental lag versus deficit models of reading disability: A longitudinal, individual growth curves analysis. Journal of Educational Psychology, 88(1), 3-17. https://doi.org/10.1037/0022-0663.88.1.3

14.

Goss

Sonnemann

Emslie

(2018). Measuring student progress: A state-by-state report card. Grattan Institute. https://grattan.edu.au/wp-content/uploads/2018/10/Mapping_Student_Progress.pdf

15.

Greenland

Senn

S. J.

Rothman

K. J.

Carlin

J. B.

Poole

Goodman

S. N.

Altman

D. G.

(2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337-350. https://doi.org/10.1007/s10654-016-0149-3

16.

IBM Corp . (2019). IBM SPSS statistics for windows, version 26.0

17.

Keats

J. A.

(1980). Comparing latent trait with classical measurement models. In Spearitt

(Ed.), Improvement of measurement in education and psychology: Contributions of latent trait theories (pp. 17-31). Australian Council for Educational Research

18.

Kuhn

T. S.

(1977). The function of measurement in modern physical science. In Kuhn

T. S.

(1977), The essential tension: Selected studies in scientific tradition and change (pp. 161–190). The University of Chicago Press.

19.

Lumsden

Semo

Blomberg

Lim

(2015). Linking NAPLAN scores to the longitudinal surveys of Australian youth. NCVER. https://www.ncver.edu.au/research-and-statistics/publications/all-publications/linking-naplan-scores-to-the-longitudinal-surveys-of-australian-youth

20.

Masters

G. N.

(2020). Nurturing wonder and igniting passion: Designs for a new school curriculum: NSW Curriculum review. NSW Education Standards Authority. https://research.acer.edu.au/nswcurriculumreview/6/

21.

McArdle

J. J.

Nesselroade

J. R.

(2003). Growth curve analysis in contemporary psychological research. In B Weiner

(Ed.), Handbook of psychology (pp. 447-480). Wiley. https://doi.org/10.1002/0471264385.wei0218

22.

McGaw

Louden

Wyatt-Smith

(2020). NAPLAN review final report. NSW Department of Education. https://naplanreview.com.au/pdfs/2020_NAPLAN_review_final_report.pdf

23.

McNeish

Matta

(2018). Differentiating between mixed-effects and latent-curve approaches to growth modeling. Behavior Research Methods, 50(4), 1398-1414. https://doi.org/10.3758/s13428-017-0976-5

24.

Meredith

Tisak

(1990). Latent curve analysis. Psychometrika, 55(1), 107-122. https://doi.org/10.1007/BF02294746

25.

Nese

J. F.

Lai

C. F.

Anderson

(2013). A primer on longitudinal data analysis in education (Technical Report No. 1320). Behavioral Research and Teaching. https://files.eric.ed.gov/fulltext/ED545257.pdf

26.

Olsen

L. W.

(2003). Essays on Georg Rasch and his contributions to statistics. Københavns Universitet, Økonomisk Institut.

27.

Peebles

Ali

(2015). Expert interpretation of bar and line graphs: The role of graphicacy in reducing the effect of graph format. Frontiers in Psychology, 6(1), 1673. https://doi.org/10.3389/fpsyg.2015.01673

28.

Rao

C. R.

(1958). Some statistical methods for comparison of growth curves. Biometrics, 14(1), 1-17. https://doi.org/10.2307/2527726

29.

Rasch

(1961). On general laws and the meaning of measurement in psychology. In Neyman

(Ed.), Proceedings of the fourth Berkeley symposium on mathematical statistics and probability: Volume IV: Contributions to biology and problems of medicine (pp. 321-333). University of California Press.

30.

Rasch

(1972-2010). Objectivity in social sciences – a method problem. Retirement lecture, 9 March 1972. Rasch Measurement Transactions, 24(1), 1252-1272.

31.

Rasch

(1977). On specific objectivity. An attempt at formalizing the request for generality and validity of scientific statements. Danish Yearbook of Philosophy, 14(1), 58-94. https://doi.org/10.1163/24689300-01401006

32.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods (1). SAGE.

33.

Singer

J. D.

Willett

J. B.

(2003). Applied longitudinal data analysis: Modeling change and event occurrence. Oxford University Press. https://doi.org/10.1093/acprof:oso/9780195152968.001.0001

34.

Stevens

S. S.

(1946). On the theory of scales of measurement. Science, 103(2684), 677-680. https://doi.org/10.1126/science.103.2684.677

35.

Stone

M. H.

(2020). Rasch's logistic model applied to growth. Journal of Applied Measurement, 21(1), 1-16.

36.

von Davier

Carstensen

C. H.

(2011). Measuring growth in a longitudinal large-scale assessment with a general latent variable model. Psychometrika, 76(2), 318-336. https://doi.org/10.1007/s11336-011-9202-z

37.

Warm

T. A.

(1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54(3), 427-450. https://doi.org/10.1007/BF02294627

38.

Williamson

G. L.

(2016). Novel interpretations of academic growth. Journal of Applied Educational and Policy Research, 2(2), 15-35. https://journals.charlotte.edu/jaepr/article/view/509

39.

Williamson

G. L.

(2017). Measuring” academic growth: Where does measurement reside? A commentary on measurement: A very short introduction by David J. Hand. Measurement: Interdisciplinary Research and Perspectives, 15(1), 19-22. https://doi.org/10.1080/15366367.2017.1344520