Abstract
Developmental invariance is important for making valid inferences about child development from longitudinal data; however, it is rarely tested. We evaluated developmental and gender invariance for one of the most widely used measures of child mental health: the parent-reported Strengths and Difficulties Questionnaire (SDQ). Using data from the large U.K. population-representative Millennium Cohort Study (N = 10,207; with data at ages 3, 5, 7, 11, 14, and 17 years), we tested configural, metric, scalar, and residual invariance in emotional problems, conduct problems, hyperactivity/inattention, prosociality, and peer problems. We found that the SDQ showed poor fit at age 3 in both males and females and at age 17 in males; however, it fit reasonably well and its scores were measurement invariant up to the residual level across gender at ages 5, 7, 11, and 14 years. Scores were also longitudinally measurement invariant across this age range up to the partial residual level. Results suggest that the parent-reported SDQ can be used to estimate developmental trajectories of emotional problems, conduct problems, hyperactivity/inattention, prosociality, and peer problems and their gender differences across the age range 5 to 14 years using a latent model. Developmental differences outside of this range may; however, partly reflect measurement differences.
Keywords
The majority of mental health problems and related psychosocial impairments begin before adulthood (Davies, 2013). From a prevention perspective there is, therefore, considerable interest in tracking developmental trajectories of mental health from the earliest stages of life in order to characterize normative versus nonnormative trajectories and to establish the risk and protective factors that distinguish them (Parkes et al., 2016; Patalay et al., 2017). A core challenge in these efforts relates to measuring symptoms and impairments in a comparable manner across different developmental stages (Knight & Zerr, 2010; Murray, Obsuth, et al., 2017). Given that gender differences are evident in most common mental health issues both in terms of their levels and developmental trajectories, there is also a need to ensure comparability of scores across gender (Booth & Murray, 2018). The issue of longitudinal measurement invariance has received relatively little attention, especially in combination with gender differences; however, valid inferences regarding development from longitudinal studies critically depend on it. In this study, we, therefore, examine the longitudinal and gender measurement invariance of one of the most widely used measure of child psychopathology: the Strengths and Difficulties Questionnaire (SDQ; Goodman, 1997) in a large nationally representative sample of children in the U.K. measured from age 3 to age 17.
Longitudinal studies of child and adolescent development represent an invaluable resource for illuminating developmental trajectories of mental health issues from the very beginning of life (e.g., Niarchou et al., 2015). When analyzed with methods such as growth curve or growth mixture models, longitudinal data provide critical information about normative symptom trajectories, variation around normative trajectories, and predictors and outcomes of those trajectories (Murray, Eisner, et al., 2017; Parkes et al., 2016; Patalay et al., 2017; Speyer et al., 2020). When analyzed using methods such as (random intercept) cross-lagged panel models or autoregressive latent trajectory models with structured residuals, they provide information about the developmental relations between symptoms and factors such as peer and academic problems, parent conflict, and symptoms of other disorders (Besemer et al., 2016; Murray, Eisner, et al., 2019; van Lier et al., 2012; Wertz et al., 2015).
However, valid inferences from these methods depend on measuring symptoms in a comparable manner over development (Edwards & Wirth, 2012). In fact, given the widespread biological and social–environmental changes that occur over and between childhood and adolescence, it is quite plausible that the meaning and manifestation of symptoms could differ across developmental stages (e.g., Rapee et al., 2019). For example, it has been suggested that hyperactivity symptoms may have a different manifestation over development, with overt behaviors (e.g., difficulty sitting down for a prolonged period of time) giving way to internal feelings of restlessness at later stages of development (Weyandt et al., 2003). Similarly, whereas physical aggression is relatively common and quite normal in the preschool years, it becomes a marker of much more severe problems by the time of late adolescence (Nærde et al., 2014; Tremblay, 2000).
Similar considerations apply to gender differences. Males and females have previously shown differences in levels and developmental trajectories of many mental health problems, with later onset and emotional problems tending to be more common in females and earlier onset and neurodevelopmental and behavioral issues tending to be more common in males (e.g., Martel, 2013; Rutter et al., 2003). Whenever there are differences in the prevalence of mental health issues across males and females there are concerns that symptoms may be more difficult to detect in the gender with the lower prevalence because informants are less attuned to symptoms in these cases. More than this, however, there is also evidence that the manifestation of mental health issues can differ across males and females. In attention deficit hyperactivity disorder, for example, it has been proposed that females may show profiles that are more characterized by inattention (as opposed to hyperactivity) and emotional symptoms as compared to males (Gershon & Gershon, 2002; Williamson & Johnston, 2015). Similarly, in relation to conduct problems, physical forms of aggression tend to be relatively more common in males than females, compared with social aggression, which shows little to no gender difference (Archer, 2004).
These developmental and gender differences are likely to be manifested as violations of measurement invariance over time, or “longitudinal measurement invariance’ and across gender or ‘gender measurement invariance’ (Edwards & Wirth, 2012). Measurement invariance refers to the distribution of item scores given the level of an underlying latent trait being independent of the measurement occasion (for longitudinal invariance) or group (e.g., gender invariance; Millsap, 2012). That is, after taking into account changes in latent trait levels across time/group, a person’s scores should not depend on whether they came from Wave 1, 2, 3, and so on, or were male or female. Full measurement invariance is difficult to test because of the need to demonstrate that the entire distribution of scores is the same across time/group conditional on latent trait level (see Molenaar & Borsboom, 2013, for a discussion). However, a weaker version: factorial invariance is readily testable within a confirmatory factor analysis (CFA) framework and is often used to test whether a measure shows any evidence of differential functioning across time or group (Liu et al., 2017; Van de Schoot et al., 2012). In a CFA framework, observed variables (e.g., questionnaire items) are defined as indicators of underlying latent factors, such as “internalising problems” or “externalizing problems” (Millsap & Yun-Tein, 2004). Observed indicators are linked to the latent factors by factor loadings that represent the strength of association between indicators and factors. In addition, for the ordinal items which are commonly used in questionnaires (e.g., Likert-type items), thresholds are modelled and are the point on the latent factor scale that individuals transition from scoring in one category (e.g., “strongly disagree”) to the next category (e.g., “disagree”; Millsap & Yun-Tein, 2004).
Different levels of invariance are required for different kinds of inferences from longitudinal data using ordinal indicators (Liu et al., 2017). To compare variances and regression paths involving latent factors over time/group (e.g., to test whether a predictive path is the same over different stages of development) requires “metric invariance,” which is when the factor loadings are equal across time/group. To compare factor means over time/group (e.g., in a growth curve model), “scalar invariance” is required, which is when both loadings and thresholds are equal across time/group (Liu et al., 2017; Liu & West, 2018). If this does not hold then true change over time may be masked or overstated. To compare means over time based on observed scores (as opposed to based on latent factor means) requires “residual invariance,” where residual variances are also equal over time/group (Liu et al., 2017).
Where measurement invariance does not hold at a given level, it is often possible to obtain a partially invariant model which is sufficient to make variance/covariance/mean comparisons over time based on the latent factors, provided that the noninvariance is appropriately modelled (Edwards & Wirth, 2012; Pokropek et al., 2019). The more items that show invariance the better, however, simulation studies suggest that up to 80% of the items can be noninvariant without necessarily invalidating inferences made regarding structural parameters (Pokropek et al., 2019).
Despite its importance, relatively limited attention has been paid to the issue of longitudinal invariance in child and adolescent mental health. Only a relatively small number of studies have tested the longitudinal invariance of mental health instruments used in child and adolescent studies (Croft et al., 2015; Leopold et al., 2016; Mathyssek et al., 2013; Motl et al., 2005; Murray, Obsuth, et al., 2017; Sosu & Schmidt, 2017; Sterba et al., 2010; Verhoeven et al., 2013). One of the most commonly used omnibus measures of child psychopathology is the SDQ (Goodman, 1997). It is widely used both in longitudinal studies and in clinical and educational settings, where it informs the provision of services and is used to evaluate the effects of interventions. In some countries, such as Scotland and the Netherlands, it is employed to track the impact of child-related policies (Sosu & Schmidt, 2017). It measures child psychopathology in five dimensions: hyperactivity/inattention, conduct problems, emotional problems, peer problems, and prosociality. It is available in self-, teacher-, and parent-report versions and is recommended for use across an age range of 3 to 16 years.
The SDQ has undergone extensive psychometric evaluation with numerous studies reporting on its factor structure, reliability, convergent validity, content validity, discriminative validity, and interrater agreement (see review by Kersten et al., 2016). Some studies have also examined its measurement invariance across factors such as gender, informant, and country (Bøe et al., 2016; Ortuno-Sierra et al., 2015; Rogge et al., 2018). However, despite its popularity in longitudinal studies, we could identify only two studies that evaluated the longitudinal measurement invariance of the English language SDQ (Croft et al., 2015; Sosu & Schmidt, 2017). Croft et al. (2015) examined the longitudinal variance of the parent-reported version of the SDQ in the Millennium Cohort Study (MCS; Connelly & Platt, 2014) across ages 3, 5, and 7 years. They found that both metric and scalar measurement invariance held for the conduct problems, hyperactivity/inattention and prosocial behavior subscales; however, only metric measurement invariance held for the emotional and peer problems scales. More recently, Sosu and Schmidt (2017) examined the longitudinal measurement invariance of the parent-reported SDQ in the Growing Up in Scotland study (https://growingupinscotland.org.uk/) across ages 4, 5, and 6 years. They found that metric and scalar measurement invariance was supported for all five subscales over time. No study has yet examined the measurement invariance of the SDQ from childhood into and across adolescence. This is an important gap because substantial physical, cognitive, social, and biological changes occur between childhood and adolescence and within adolescence itself that could undermine the developmental comparability of measures of psychopathology (Rapee et al., 2019). Indeed, in the transition to adolescence the risk for the onset of several disorders increases, especially emotional disorders (Copeland et al., 2014; Roza et al., 2003). Furthermore, despite the fact that gender differences in developmental trajectories of mental health problems are also commonly of interest (Cleverley et al., 2012; Dekker et al., 2007; Murray, Booth, et al., 2019; Salk et al., 2016), no study has yet examined developmental measurement invariance alongside gender measurement invariance for the SDQ scores. Among the few studies that have examined gender measurement invariance in versions of the SDQ, some have identified violations of measurement invariance (Bøe et al., 2016; van de Looij-Jansen et al., 2011); however, it is not clear how these violations interact with developmental stage as longitudinal and gender measurement invariance are yet to be tested together.
It is not, therefore, clear whether gender differences in developmental trajectories and their relations across dimensions and to risk factors and outcomes can be validly compared and/or whether males and females can be validly combined into a single sample (see, e.g., Meredith & Teresi, 2006). Here, we address these gaps by evaluating the gender and longitudinal invariance of the parent-reported SDQ across ages 3, 5, 7, 11, 14, and 17 years using the large, population-representative MCS.
Method
Participants
Participants (N = 10,207) were children/adolescents from U.K.-based the MCS (Connelly & Platt, 2014) who participated up to age 17 (the initial sample was 18,818). The MCS is a longitudinal study that has tracked the family and broader social lives of children born at the beginning of the 21st century (2000-2002). Families were sampled using a stratified sampling procedure and included participants from all four nations of the United Kingdom. The sampling frame was families with a 9-month-old child and eligible for the (universal) child benefit at the time of the first sweep (measurement wave). They were identified based on Department for Social Security (now Department for Work and Pensions) Child Benefit register records. Sensitive cases (including for example where children had died or been taken into local care or where a family were being investigated for benefit fraud) were excluded. A further exclusion criterion was that if a family had already participated in the Department for Work and Pension’s Family and Children Survey. A small number of families (56) were added who were not initially identified via the Child Benefit register but who could be identified via health visitors. Oversampling of ethnic minority groups and disadvantaged families was built into the survey design with design weights used to correct for this oversampling.
Stratification variables and attrition weights that include an adjustment for baseline study design are included as part of the data release to account for the complex sampling design. This allows parameter estimates to be corrected for nonrandom sampling and attrition and for standard errors to be corrected for clustering. We used a pseudo-maximum likelihood estimation technique to make these adjustments. Participants were sampled from a population that was born over a 16-month period, which allowed season effects to be taken into account whilst also making the fieldwork more feasible. The England and Wales samples were born between 1st September 2000 and 31st August 2001; the Scotland and Northern Ireland samples were born between 24th November, 2000, and 11th January, 2002. MCS is fully documented at: https://ukdataservice.ac.uk where data can also be downloaded. Seven sweeps of data are currently available, at ages 9 months, 3, 5, 7, 11, 14, and 17 years with parent-reported SDQ data available at Sweeps 2 to 6 (ages 3-17 years).
Sample demographic information is provided in Table 1. These are based on unweighted data from children with SDQ scores at age 17.
Sample Demographic Information.
Note. These are unweighted and based on the sample of participants with Strengths and Difficulties Questionnaire data up to age 17.
Measures
Parent-Reported Strengths and Difficulties Questionnaire
The parent-reported SDQ measures five dimensions of child psychosocial functioning in five subscales: conduct problems, hyperactivity/inattention, emotional problems, peer problems, and prosocial behavior. Each subscale has five items. The conduct problem items refer to: often having temper tantrums; generally being obedient; often fighting with or bullying other children; often lying or cheating; and stealing from home, school, or elsewhere. The hyperactivity/inattention items refer to: being restless, overactive, being unable to stay still for long; constantly fidgeting or squirming; being easily distracted; thinking before acting; and seeing tasks through to their end. The emotional problems items refer to: often complaining of headaches, stomach-aches, or sickness; having many worries; being often unhappy, down-hearted, or tearful; being nervous or clingy in new situations; and having many fears, being easily scared. The prosociality items refer to: being considerate of others’ feelings; sharing readily with other children; being helpful if someone is hurt, upset, or feeling ill; being kind to younger children; and often volunteering to help others. The peer problems items refer to: being rather solitary and tending to play alone; having at least one good friend; being picked on or bullied by other children; and getting on better with adults than other children. The SDQ version administered at age 3 was adapted slightly to improve its age-appropriateness. Specifically, in the conduct problems subscale: argumentative with adults and can be spiteful were used instead of often lies or cheats, and steals from home, school, or elsewhere; and in the hyperactivity/inattention subscale: can stop and think before acting was used instead of thinks things out before acting. Responses are recorded on a 3-point scale from not true to certainly true. Respondents could also select a “can’t say” or “not applicable” option.
As noted earlier, there is an extensive literature on the psychometric properties of the SDQ. Most studies support the structural and convergent validity of the five-dimensional model implied by the design of the scale. Internal consistency values of the five subscales based on Cronbach’s α have sometimes been noted to be weak; however, in the current study all internal consistency values based on McDonald’s (1999) omega calculated using polychoric item correlations to account for their ordered-categorical response format were good, with only one subscale (peer problems at Sweep 2) showing values <.70. For emotional problems at Sweeps 2 to 7 (age 3 to 17) omega values were as follows: .76, .78, .80, .83, .83, and .83. For conduct problems they were as follows: .79, .76, .81, .84, .85, and .80. For hyperactivity/inattention problems they were as follows: .80, .84, .86, .86, .85, and .81. For prosociality they were as follows: .77, .80, .83, .82, .86, and .82. For peer problems they were as follows: .65, .70, .75, .80, .78, and .72.
Longitudinal and Gender Measurement Invariance
Prior to conducting measurement invariance analyses, CFAs were fit for each gender individually at each time point. For these and all subsequent analyses, we used a five-dimensional model of the SDQ for our analyses, where each subscale (hyperactivity/inattention, conduct problems, emotional problems, peer problems, and prosociality) mapped to a latent factor. This structure was preferred from among at least 12 different factor structures that have been proposed for the SDQ (Gomez & Stavropoulos, 2019). One of the main alternatives to the five-dimensional model that has been proposed is a three-dimensional model with latent factors for externalizing problems, internalizing problems, and prosociality; however, this has received less empirical support (e.g., Croft et al., 2015). Bifactor and second-order models of the SDQ have also been tested and have often shown good fit; however, de la Cruz et al. (2018) directly compared these models to a five-dimensional model and found the latter to be better fitting. Finally, a model similar to the five-dimensional model but with an additional methods factor (“positive construal”) has been proposed (Palmieri & Smith, 2007). Gomez and Stavropoulos (2019) found that this model fit better than a five-dimensional model in their Malaysian sample; however, other studies have failed to find support for this model over the original five-dimensional model (Sanne et al., 2009). On balance, therefore, the five-dimensional model was preferred to other structures that have been proposed because it reflects the theoretical structure for the SDQ and has generally been supported by factor analytic work, including in direct comparisons with alternative structures (e.g., Bøe et al., 2016; de la Cruz et al., 2018; Gomez & Stavropoulos, 2019).
Cross-group (gender) and longitudinal measurement invariance were tested in sequence. We began by testing invariance across gender in each time point. A “forwards” method beginning with the least constrained model and successively adding more constraints was used (see, e.g., Kim & Willson, 2014). Within each time point we began by fitting a configural model (based on the five-dimensional model discussed above) in which the pattern of item loadings was the same across all time but the magnitude of loadings (and thresholds) were allowed to vary across group.
For scaling and identification purposes, the means and variances of all latent factors in one gender (females served as the reference group) were fixed to 0 and 1, respectively, and the loading of one item per latent factor was fixed equal across group, one threshold per item was fixed equal across group, and the second threshold for the marker variable (“reference indicator”) was fixed equal across group. The reference indicator (i.e., the item on which these equality constraints were imposed) was chosen based on an inspection of item contents to determine which were most likely to show invariance over both gender and time. For the hyperactivity/inattention factor restless, overactive and cannot stay still long was used as the reference indicator; for conduct problems generally obedient was used; for emotional problems often seems worried was used; for prosociality often volunteers to help others was used and for peer problems ‘generally liked by other children’ was used. Theta parameterization was used to facilitate testing of residual invariance. As a further model identification constraint in the configural model, item residual variances were fixed to 1 in the reference group (Millsap & Yun-Tein, 2004).
If the configural model showed acceptable fit, we proceeded to test metric invariance by imposing cross-group equality constraints on the factor loadings, that is, the loading for each item was fixed equal to the loadings for the corresponding item across males and females. There are no universally agreed on criteria for establishing invariance within categorical invariance contexts, therefore, we report both (scaled) chi-square difference tests and comparisons of comparative fit index (CFI), root mean square error of approximation (RMSEA) and standardized root mean square residual (SRMR). The disadvantage of the chi-square different tests is that they are liable to be sensitive to trivial misspecifications in the current context given the large sample size (Yuan & Chan, 2016); however, the disadvantage of RMSEA, CFI, and SRMR comparisons is that they are less well-validated for categorical indicators, with no previous study yet having evaluated their performance for the particular combination of number of factors, indicators, group-by-time point number, and sample size for ordinal indicators (Svetina et al., 2020). RMSEA, CFI, and SRMR comparisons served as our primary guide to determining invariance given the very large sample size. We used the criteria of Chen (2007), which suggests that metric invariance holds if CFI decreases by no more than .010; if RMSEA increases by no more than .015, and if SRMR increases by no more than .030. We note that other studies have suggested alternative (both more and less strict) criteria (e.g., Finch & French, 2018; Rutkowski & Svetina, 2017; Svetina & Rutkowski, 2017); however, we selected Chen’s (2007) criteria because we judged them to be well-suited to the goal of identifying measurement invariance violations where they may have a nontrivial biasing effect on multigroup/longitudinal models. Chen’s (2007) criteria were also used for scalar and strict invariance. If metric invariance did not hold, modification indices and expected parameter changes were used to guide the release of constraints to achieve a partially invariant model.
If a (partially) metric invariant model was achieved, we proceeded to test scalar invariance. Here we fixed all item thresholds to be equal across group and followed the same logic for determining invariance as outlined in relation to metric invariance. Chen's (2007) criteria for scalar invariance is that it holds if CFI decreases by no more than .010, if RMSEA increases by no more than .015, and SRMR increases by no more than .010 (Chen, 2007). Scalar invariance constraints were not imposed on any items that did not show metric invariance. If the initial set of scalar invariance constraints resulted in a substantial deterioration in fit according to Chen’s (2007) criteria, we iteratively removed constraints, guided by modification indices and expected parameter changes in order to try and identify a partially scalar invariant model.
Finally, if a partially scalar invariant model could be achieved, we proceeded to test residual invariance. Chen’s (2007) criteria are that residual invariance holds if CFI decreases by no more than .010, RMSEA decreases by no more than .015, and SRMR decreases by no more than .010 with the addition of cross-group equality constraints on residual variances.
After testing cross-group invariance, we proceeded to test longitudinal invariance, using methods based on Liu et al. (2017). Here we also used a “forwards” approach. Identification constraints in the configural model were analogous to the gender invariance model (the mean and variance of the factors at the first time point were fixed and one threshold for each item was fixed, together with loading and threshold equality constraints over time). However, it was also necessary to constrain all residual variances as well as scale factors in the configural model to achieve convergence for the baseline configural model. As such the models testing metric and scalar constraints included residual invariance constraints too so that an appropriate comparison against the configural model could be made. As a final step, we thus compared a model with metric, scalar, and residual constraints to a model with only metric and scalar constraints to assess the tenability of the residual variance constraints. These latter models were compared using a chi-square difference test and based on CFI, RMSEA, and SRMR differences. We also considered information from expected parameter changes and modification indices from the more constrained model altogether to check no substantive noninvariance had been missed.
Where adaptations to the items administered at age 3 were required to improve age-appropriateness at this earlier developmental stage, these items were paired with their corresponding item administered at later ages “as if” they were the same item, allowing their equivalence over time to be addressed as an empirical question.
Models were fit in Mplus 8.4 (Muthén & Muthén, 2015) using weighted least squares means and variances adjusted estimation. Analyses were weighted according to the longitudinal weight for Sweep 7 with stratification into the nine strata of MCS and clustering within households also accounted for by use of a sandwich estimator. The Sweep 7 longitudinal weight includes both design and nonresponse weighting and thus account for nonrandom attrition under the assumption of missingness at random (Rubin, 1976)
Results
Model fits for all age-by-gender subsamples are provided in Table 2. Links to the full model outputs are provided in this same table. Fit was good according to RMSEA at all ages but only poor-to-acceptable by other fit criteria. Fit was poorest at age 3 but tended to be better at older ages, except in the age 17 male group where fit was again poor in the male group. Given the poor fit in these three (time-by-gender) groups we did not include them in further examinations of cross-group or longitudinal invariance and concluded on the basis of these initial single-group CFAs that configural invariance did not hold for these groups.
Fits for Single-Group CFA Models for Males and Females.
Note. Boldface denotes fit judged insufficient to justify inclusion in further invariance analyses. CFI = comparative fit index; TLI = Tucker–Lewis index; RMSEA = root mean square error of approximation; SRMR = standardized root mean square residual
Gender Invariance
Fits for the gender invariance analyses are provided in Table 3. At age 5, there were significant chi-square difference tests at all stages of invariance testing; however, CFI, RMSEA, and SRMR all either improved or declined within acceptable limits. Furthermore, modification indices and expected parameter changes did not tend to suggest any release of constraints that would lead to substantial improvements in fit. We, therefore, concluded that gender invariance up to the residual level held at age 5.
Fits for Gender Invariance Tests.
Note. Metric invariance criteria were that it holds if comparative fit index (CFI) decreases by no more than .010, if root mean square error of approximation (RMSEA) increases by no more than .015, and if standardized root mean square residual (SRMR) increases by no more than .030; scalar invariance criteria were that it holds if CFI decreases by no more than .010, if RMSEA increases by no more than .015, and SRMR increases by no more than .010; residual invariance criteria were that it holds if CFI decrease by no more than .010, RMSEA decreases by no more than .015 and SRMR decreases by no more than .010.
At age 7, a nonsignificant chi-square difference test, an improvement in CFI and no change in RMSEA and SRMR across the configural and metric models suggested that metric invariance held. The chi-square difference tests were significant at the scalar and residual stage; however, CFI, RMSEA, and SRMR changes were within defined limits. Indeed, CFI and RMSEA improved between the scalar and residual models. On balance we concluded that invariance held up to the residual stage across gender at age 7.
At age 11, the chi-square difference test was significant at p < .05 but not p < .001 and CFI and RMSEA improved with the addition of both metric and scalar invariance constraints (SRMR was unchanged), suggesting, on balance, that scalar gender invariance held at this age. The addition of residual constraints yielded a nonsignificant chi-square difference test, an improvement in CFI and RMSEA, and a slight deterioration in SRMR that was within predefined limits. This suggested that at age 11 gender invariance held up to the residual level.
At age 14, there was a significant chi-square difference test at the metric level accompanied by a slight improvement in CFI and RMSEA, and a slight deterioration in SRMR; however, the latter was within predefined limits. With the addition of scalar constraints, a significant chi-square difference test was accompanied by a further improvement in CFI and RMSEA and no change in SRMR, which on balance suggested that scalar invariance held. Finally, the addition of residual constraints yielded a nonsignificant chi-square difference test, improvements in CFI and RMSEA and no change in SRMR, suggesting that gender invariance at age 14 held to the residual level.
Longitudinal Invariance
Given our finding that gender invariance held up to the residual level for ages 5 up to 14 years, we proceeded to test longitudinal invariance across this age range only. Fits for each model tests are provided in Table 4. As noted above, our baseline model included residual invariance constraints but loading and threshold constraints were only the minimum required for identification. This baseline model fit well. The addition of metric constraints to this model was associated with a significant chi-square difference test; however, CFI and RMSEA both improved and SRMR decreased only slightly and well within predefined limits. The addition of scalar constraints to this model was associated with a significant chi-square difference test; however, CFI and SRMR both increased only minimally and within the predefined limits and RMSEA remained unchanged. This model now represented a fully constrained model with loadings, thresholds, and residual variances all constrained to equality over time. To explore the residual invariance assumption, this fully constrained model was compared with a model where residual variances were free to vary across time. The fully constrained model had the same CFI and RMSEA fit as the more relaxed model but a slightly higher SRMR value. The magnitude of the SRMR difference was .02, which is larger than the recommended threshold of .01 for the scalar versus residual invariance model comparison according to the criteria of (Chen, 2007). We, therefore, examined modification indices and expected parameter changes in the fully constrained model to identify possible invariance constraint misspecifications. These suggested the removal of the constraint on the residual variance of item “complains of headaches, stomach aches, sickness” at Sweep 5 (age 11). After the removal of this constraint the SRMR difference against the relevant comparison model was within predefined limits and it was concluded that partial residual invariance held.
Longitudinal Invariance Model Fits.
Note. Metric invariance criteria were that it holds if comparative fit index (CFI) decreases by no more than .010, if root mean square error of approximation (RMSEA) increases by no more than .015, and if standardized root mean square residual (SRMR) increases by no more than .030; scalar invariance criteria were that it holds if CFI decreases by no more than .010, if RMSEA increases by no more than .015, and SRMR increases by no more than .010; residual invariance criteria were that it holds if CFI decrease by no more than .010, RMSEA decreases by no more than .015 and SRMR decreases by no more than .010. df = degrees of freedom.
In these models, residual invariance was assumed to facilitate estimation. bConstraints removed on the residual variance of item “complains of headaches, stomach aches, sickness” at Sweep 5 (age 11).
Discussion
In this study, we tested the longitudinal and gender measurement invariance of a popular measure of child mental health: the parent-reported SDQ (Goodman, 1997) over ages 3, 5, 7, 11, 14, and 17 years in the MCS. Though seldom-tested, longitudinal invariance is critical to many of the inferences drawn from developmental data, with metric invariance required to compare latent variances and covariances over development, scalar invariance required to compare latent means over development, for example, in the analysis of developmental trajectories, and residual invariance required to compare levels over time/group based on observed scores (Edwards & Wirth, 2012). Given that the SDQ data is commonly analyzed within MCS (e.g., Ahn et al., 2018; Carson et al., 2013; Heikkilä et al., 2011; Hesketh et al., 2016; Hope et al., 2014; Noonan et al., 2018; Patalay & Fitzsimons, 2016) and the SDQ is among the most popular omnibus measures of psychopathology in general, our study adds important evidence on the appropriateness of the SDQ as a measure of mental health across childhood and adolescence.
Our results demonstrated residual gender and longitudinal invariance for the parent-reported version of the SDQ in the MCS across ages 5 to 14 years up to the partial residual level (with only one residual invariance constraint required to be freed). Thus, it is likely to be valid to compare variances, covariances and means for each of the parent-reported SDQ constructs over this range of development using mixed gender samples or comparing genders using a latent variable model. Within a (random intercepts) cross-lagged panel model, for example, it may be of interest to evaluate whether the relations between emotional and behavioral problems remain constant over development to illuminate the evolution of their comorbidity. For example, two competing hypotheses: differentiation and dynamic mutualism have been proposed to characterize the changing relation between emotional and behavioral problems over time (Murray et al., 2016). The former proposes that the two symptom domains become less correlated over development while the latter proposes that they becoming increasingly correlated with time. Without metric invariance; however, it is not clear whether changes in their correlation merely reflect changes in their measurement over time. Equally, to identify how symptom levels change across different developmental stages, growth curve models can be used to examine the developmental trajectories of symptoms. However, changes in mean symptom levels over time are potentially confounded with changes in item thresholds over time, making scalar invariance important to demonstrate as a prerequisite to examining developmental trajectories.
Our results, however, suggested that the five-dimensional model that described the covariance in items in both males and females at ages 5 to 14 years did not well-describe their covariance at age 3 or in males at age 17. CFA models fit poorly in these groups, thus suggesting a lack of even configural invariance. Caution is needed when conducting comparisons using the SDQ involving these groups as differences could be attributable to differences in the meaning, manifestation and/or reporting of symptoms. It is possible, for example, that psychopathology is less differentiated at age 3 and thus a smaller number of dimensions is required to describe their covariance. Similarly, by age 17, parental monitoring/supervision has generally declined to a point where peer and self-reports may provide more reliable accounts of psychopathology, especially for males for whom the decline in parental monitoring and associated knowledge of their child in adolescence tends to be more pronounced (Laird, Pettit, Bates, et al., 2003; Laird, Pettit, Dodge, et al., 2003). Further research using, for example, cognitive interviewing and comparisons of parent-reports with other forms of assessment (e.g., observations by trained observers and clinical interviews) could help illuminate the sources of the difference.
Testing longitudinal measurement invariance should be a critical step in the development and validation of instruments used in developmental science (Edwards & Wirth, 2009) as it provides insights into which items may not be comparable over time and the implications of this for the interpretation of those scores at different ages. However, achieving longitudinal invariance must also be weighed against ensuring that an instrument remains appropriate at all ages and is capable of capturing age-specific manifestations in mental health issues. In the SDQ, for example, several items were adapted for administration at age 3 to better capture attention deficit hyperactivity disorder and conduct problems manifestations at this age. As a result, full longitudinal invariance is not necessarily the most appropriate goal in selecting items for a scale to be used across age. Rather, a majority of developmentally invariant items with a small number of noninvariant or age-specific items may provide a better compromise in terms of these competing consideration, given that partial invariance can often be sufficient (Pokropek et al., 2019). Despite the considerable attention paid to the validity and reliability of the SDQ (e.g., see Kersten et al., 2016, for a review), only two studies have examined its longitudinal invariance in the English language version and thus far and these did not extend beyond age 7 nor consider its interaction with gender invariance. Our study thus adds evidence for the comparability of scores from age 5 into adolescence and across males and females.
Our finding of scalar invariance of the parent-reported SDQ across ages 5 to 14 is somewhat more favorable than the findings observed for omnibus child psychosocial functioning instruments in the small number of studies that have examined developmental invariance. For example, in a previous study of the Social Behavior Questionnaire, which shares origins with the SDQ, the anxiety subscale showed substantive departures from invariance (Murray, Obsuth, et al., 2017). Of note, a previous study using data partially overlapping with the current study suggested that the SDQ did not show developmental invariance across ages for all constructs either (Croft et al., 2015). Specifically, they found loading and threshold invariance across ages 3, 5, and 7 years for the conduct problems, prosociality, and hyperactivity/inattention factors but loading invariance only for the peer problems, and emotional problems factors. Our analysis used the same method but a slightly different sample (participants taking part in the MCS up to age 17), an increased number of waves of data, and conducted single-group CFAs for each gender/age combination and gender invariance analyses for each age prior to testing longitudinal invariance analyses. Based on our preliminary analyses, we did not include the age 3 data in our longitudinal invariance analyses because fit was poor for this group in single factor CFAs. The difference between our and this previous study illustrates the subjectivity of decision points in invariance testing and the sensitivity of findings to these decisions. Guiding these decision points and drawing optimal conclusions under conditions where different reasonable decisions lead to different findings is an area where there is a need for future research in invariance testing. This could be supported by simulation studies that cover a greater range of invariance contexts such as the current multigroup by longitudinal context (Kim & Willson, 2014) and/or the application of multiverse analysis frameworks (Steegen et al., 2016) to invariance testing. For example, future simulation studies covering the multigroup longitudinal context would be helpful to shed further light on which fit indices and thresholds are most effective for balancing Type 1 and Type 2 errors.
Limitations and Future Directions
It is important to note the limitations of the current study. First, as noted above standards used to assess invariance are subjective and depend on the level of noninvariance judged to be substantively important. In this study, we used Chen’s (2007) criteria with the aim of striking a balance between detecting levels of noninvariance that, if left unmodeled, would be likely to substantively change conclusions in longitudinal models and between detecting large numbers of incidences of trivial violations. A further issue is that demonstrating measurement invariance does not guarantee that the scores from a measure have the same meaning across development, that is, that they show measurement equivalence over development (Meredith & Teresi, 2006). In fact, there have been examples where measurement invariance in scores has accompanied a shift in an underlying cognitive process (Widaman et al., 1992), illustrating that measurement invariance analyses may not be sensitive to important qualitative shifts over development. Other forms of evidence such as qualitative interviews (e.g., cognitive interviews; Collins, 2003) with informants rating young people at different stages of development could be used to illuminate whether and in what way informants understand and score symptoms for different ages.
Future studies should aim to examine the longitudinal invariance of other commonly used measures of child and adolescent psychosocial functioning across their recommended age ranges. Testing longitudinal measurement invariance remains relatively rare but should be a prerequisite to making strong inferences about developmental patterns from models that involve a comparison of variances, covariances, or means over time. Similarly, greater attention could be paid to developmental invariance during measure development (e.g., Sass, 2011). This would help ensure that at least a core subset of items (sufficient to achieve partial measurement invariance) show invariance over time and/or to provide insights into why particular items do not and the implications for the interpretation of scores at different ages (e.g., Edwards & Wirth, 2009). Such items could avoid referring to developmentally specific behaviors and contexts and focus on symptoms that are likely to be relevant irrespective of developmental stage. However, it is important not to select only developmentally invariant items during measure development as this would risk missing key age-specific manifestations of a construct. In some cases, the manifestations of a construct are so different across development that it is difficult to specify common items that can be used across development (Knight & Zerr, 2010). In these cases, longitudinal invariance analyses will be of limited utility.
Future studies should also examine the developmental invariance of the SDQ in other longitudinal datasets to evaluate the generalizability of the findings. Given that the SDQ is used across the world in a large number of different language versions, and in parent-, teacher, and self-report versions it would, similarly, be of considerable interest to further evaluate its invariance across other key categories in interaction with developmental invariance such as country (Ortuño-Sierra et al., 2015; Stevanovic et al., 2017) and informant (Rogge et al., 2018). One previous study suggested, for example, that measurement invariance of the SDQ did not hold across for 11/25 items across five European nations, which complicates cross-country comparisons of child development. Little is currently known the extent to which the SDQ yields comparable scores across diverse country contexts more broadly (Stevanovic et al., 2017). Future measurement invariance analyses can build further evidence surrounding the interpretation and comparability of item scores as administered in different formats and contexts. However, they can also potentially provide substantive insights into how psychopathology at different developmental stages is viewed by different observers and in different cultural contexts (Meredith & Teresi, 2006).
Conclusions
The parent-reported SDQ shows configural, metric and scalar gender and longitudinal invariance over ages 5, 7, 11, and 14 years (but not 3 and 17) across all its subscales in the MCS. This supports the use of the scale to compare variances and covariances and to examine developmental trajectories in emotional problems, conduct problems, hyperactivity/inattention, prosociality, and peer problems across childhood and adolescence.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
