Sage Journals: Discover world-class research

Abstract

Existing research shows that children’s responses to rewards and punishments are essential for understanding attention-deficit/hyperactivity disorder, oppositional defiant disorder, conduct disorder, and callous-unemotional traits. The present study developed the Contingency Response Rating Scale (CRRS) to fulfill the need for a reliable and valid measure of children’s contingency response style that is brief, easy to use in applied settings, and provides additional information to existing clinical measures. We examined the psychometric properties of the CRRS in a sample of 196 children (ages 5–12), most of whom were referred to evaluate attention and behavior problems in an outpatient clinic. Using principal axis factoring, we identified five factors: (a) punishment ineffectiveness, (b) reward ineffectiveness, (c) punishment dysregulation, (d) reward dysregulation, and (e) contingency insensitivity. The subscales based on these factors showed acceptable test–retest and internal consistency reliability, and scale intercorrelations varied from low to moderate. The subscales also captured significant variance not explained by child or parent demographics and were associated with measures of psychopathology and impairment. The results provide preliminary evidence that the CRRS may be a helpful tool for assessing reward and punishment sensitivity in children with attention and behavior problems.

Keywords

reward punishment rating scale callous-unemotional conduct disorder oppositional defiant disorder ADHD

Measuring how children respond to reward and punishment, or contingency response style (CRS), is a crucial part of understanding attention-deficit/hyperactivity disorder (ADHD), oppositional defiant disorder (ODD), conduct disorder (CD), and callous-unemotional (CU) traits, which are externalizing forms of youth psychopathology. CRS is a pivotal construct in externalizing disorders; three critical factors underscore its importance. First, reward and punishment sensitivity is central to etiological theories of these disorders (Blair, Leibenluft, & Pine, 2014; Dadds & Salmon, 2003; Sonuga-Barke et al., 2016). In this study, these terms specifically refer to children’s typical and primarily observable responses when anticipating or receiving rewards and punishments. This application differs from the usage by Gray and McNaughton (2003) or Carver and White (1994), who associate reward and punishment sensitivity with broader personality traits governed by the Behavioral Activation System and the Behavioral Inhibition System, respectively. Second, research studies consistently demonstrate that children with these disorders differ from typically developing children on measures of reward and punishment sensitivity. While findings are mixed, some evidence shows that children with ADHD are more sensitive to reward and punishment (Fosco et al., 2015; Furukawa et al., 2019; Luman et al., 2010), whereas children with ODD or CD are less sensitive to reward and punishment (Blair, White, et al., 2014; Matthys et al., 2012, 2013). However, punishment insensitivity, especially relative to reward, may be specifically associated with CU traits (Byrd et al., 2014; Frick & Marsee, 2018). Third, all empirically supported psychosocial treatments for ADHD, ODD, or CD incorporate behavioral therapy techniques that are grounded in the effective use of reward and punishment strategies (Eyberg et al., 2008; Pelham & Fabiano, 2008). It is evident that CRS is an important theoretical, empirical, and clinical construct.

In light of its significance, it is surprising that the diagnosis¹ of ADHD, ODD, CD, or CU fails to consider children’s CRS systematically. For instance, the symptoms of these conditions, as specified in the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (5th ed.; DSM-5; American Psychiatric Association [APA], 2013), do not mention reward or punishment. The same is true of other widely used measures of these conditions, such as the Disruptive Behavior Disorders Rating Scale (DBDRS; Fosco et al., 2023; Pelham, Gnagy, et al., 1992) and the Inventory for Callous-Unemotional Traits (ICU; Frick, 2004). Consequently, individual differences in CRS are not systematically incorporated into case formulations and treatment plans.

An essential reason contributing to the discrepancy between the theoretical and empirical importance of CRS and its lack of routine integration into clinical practice may be the absence of a psychometrically sound, empirically supported measure of CRS that is brief, easy to administer in clinical settings, and provides complementary information to existing clinical assessment batteries. Most studies of CRS have used laboratory-based behavioral tasks, such as the Monetary Incentive Delay Task (Byrd et al., 2021), the Probabilistic Response Reversal Task (Budhani & Blair, 2005), and the Guessing Reward Task (Waxmonsky et al., 2022). These tasks are effective but costly, time-consuming, and not easily integrated into routine clinical assessments or small-scale research studies. In contrast to this approach, rating scales are the cornerstone of empirically supported assessment of attention and behavior problems in children, as they are inexpensive, quick, easy to administer and score, and collect information from multiple informants, such as mother, father, and teacher (McMahon & Frick, 2005; Pelham et al., 2005; Ray & Frick, 2020). Accordingly, developing a psychometrically sound, empirically supported rating scale that measures CRS may facilitate the integration of CRS into routine clinical evaluations of children. The development of such a measure could significantly enhance the diagnosis and treatment of children with ADHD, ODD, CD, or CU by guiding the selection and tailoring of evidence-based interventions and by tracking the trajectory and effectiveness of treatment.

The Sensitivity to Punishment and Sensitivity to Reward (SPSR) is an existing rating scale measure of reward and punishment sensitivity in youth (Colder et al., 2011; Colder & O’Connor, 2004). The SPSR is based on Gray’s reinforcement sensitivity theory (Gray, 1987) and consists of 33 items that measure seven primary factors: fear/shyness, anxiety, conflict avoidance, sensory reward, drive, response to social approval, and impulsivity/fun-seeking. The fear/shyness, anxiety, and conflict avoidance scores measure punishment sensitivity, while the remaining factors measure reward sensitivity. The SPSR has several strengths, including strong theoretical underpinnings, supported psychometric properties, and widespread use (e.g., Becker et al., 2013; Fosco et al., 2015; Luman et al., 2012; van Langen et al., 2021). Moreover, one study suggests that parent ratings on the SPSR impulsive/fun-seeking scale predict behavioral treatment response in children with ADHD (van Langen et al., 2021), although another study that used teacher ratings on the SPSR did not find a significant association (Staff et al., 2021). However, the SPSR also has limitations. First, many items on the SPSR assess contingency response indirectly. For instance, the item “In unfamiliar tasks, your child worries about failure” is used to measure punishment sensitivity but could also be interpreted as a measure of anxiety. Second, scores from the SPSR seem to overlap with diagnostic measures. For example, the fear/stress and anxiety scales may capture anxiety symptomatology, and the impulsivity/fun-seeking scale may capture ADHD symptomatology. When used clinically or in psychopathology research, a more optimal measure of reward and punishment sensitivity would be distinct from anxiety, ADHD, and other forms of psychopathology to minimize redundant assessment.

Given these limitations, creating a new measure of children’s CRS is necessary. Such a measure could be valuable for mental health professionals, teachers, and other individuals involved in children’s care and could serve as a valuable tool for both clinical and research purposes. We developed the Contingency Response Rating Scale (CRRS) to address this gap. In conceptualizing the CRRS, we posited that there are three key dimensions of how children respond to contingencies that are important for clinicians to consider when assessing and treating youth:

Effectiveness—how well contingencies work;

Dysregulation—whether contingencies are associated with a loss of self-regulation;

Insensitivity—whether the child seems indifferent to contingencies.

We also posited that these three dimensions may be relevant to both reward and punishment. That is, it may be helpful to evaluate reward effectiveness separately from punishment effectiveness, reward dysregulation separately from punishment dysregulation, and reward insensitivity separately from punishment insensitivity. We also hypothesized that reward dysregulation may occur while children are anticipating a reward or after receiving a reward, but this distinction is not relevant for punishment, which parents typically use reactively. For instance, a review of antecedent strategies for improving children’s compliance found numerous studies that used reward-based procedures but none that used anticipated punishment as an antecedent technique (Radley & Dart, 2016). Conceptually, these dimensions of contingency response seem at least partly distinct. For example, a child may show high dysregulation (e.g., get excessively angry) in response to punishment, but punishment may still be effective for that child. Importantly, it is unclear whether these dimensions can be empirically distinguished using parent ratings and, if so, whether these distinctions are clinically meaningful.

In summary, developing a standardized assessment tool for the complex and multifaceted nature of children’s CRS is crucial. Considering its intended use as a complementary measure to the clinical assessment of attention and behavior problems in youth, the measure should be (a) easy to use; (b) applicable to clinical, educational, and other applied settings; and (c) provide non-redundant information relative to measures of attention and behavior problems, as well as impairment. The present study aimed to develop and conduct a first evaluation of the psychometric properties of the CRRS, designed to assess various dimensions of children’s responses to rewards and punishments, encompassing effectiveness, dysregulation, and insensitivity. It was hypothesized that exploratory factor analysis (EFA) would support a seven-factor structure for the CRRS, each factor representing a distinct dimension: effectiveness, dysregulation, and insensitivity, assessed separately for reward and punishment, differentiating reward dysregulation into anticipation of reward and receipt of reward, while maintaining a unified measure of punishment dysregulation. In addition, scores derived from the factors would exhibit satisfactory test–retest and internal consistency reliabilities and significant but moderate inter-scale correlations. Finally, scores would demonstrate significant associations with measures of ADHD, ODD, CD, CU, and impairment in both correlational and regression analyses.

Method

Participants

The sample comprised 196 elementary children between the ages of 5 and 12 years (M = 8.78, SD = 1.63) recruited from a large metropolitan area in the southeastern United States. The majority of participants were seeking treatment for attention and behavior problems (n = 179; 91.3%), boys (n = 157; 80.1%), White race (n = 147; 79.0%), and of Hispanic ethnicity (n = 126, 66.7%). Full-scale IQ scores, assessed using the Wechsler Abbreviated Scale of Intelligence–Second Edition (Wechsler, 2011), ranged from 71 to 149 (M = 102, SD = 14), and academic achievement scores, assessed using the Wechsler Individual Achievement Test–Third Edition, ranged from 53 to 156 (Reading: M = 104, SD = 15; Writing: M = 104, SD = 17; Math: M = 103, SD = 17). The median family income was $70,000, ranging from $ 0 to over $200,000. The primary caregivers who rated the children, henceforth referred to as parents, were predominantly females (n = 176, 89.8%) and biological parents (n = 168; 85.7%).

Measures

Contingency Response Rating Scale

The development of the CRRS began with identifying potentially distinct dimensions of children’s CRSs, focusing on dimensions potentially relevant to developing and evaluating psychosocial treatments for children with externalizing disorders. These dimensions were identified through a review of relevant literature and discussions among the senior authors. Seven potential dimensions were hypothesized:

Punishment Effectiveness—How well punishment influences the child’s behavior and whether disciplinary strategies decrease undesired behaviors.

Reward Effectiveness—How well rewards and positive incentives motivate the child and whether rewards produce desired behaviors or changes.

Punishment Dysregulation—The tendency to exhibit dysregulated emotional and behavioral responses following punishment, including anger, frustration, aggression, and escalating negative behaviors.

Reward Anticipation Dysregulation—The tendency to become overly fixated on an upcoming reward, exhibiting overfocus and perseverated thoughts about a potential reward.

Reward Receipt Dysregulation—The tendency to exhibit dysregulated emotional and behavioral responses upon receiving a reward, including excessive silliness and overexcitement.

Punishment Insensitivity—Diminished affective and behavioral response to punishment, including a lack of concern, indifference, or a blunted emotional reaction to disciplinary consequences.

Reward Insensitivity—Diminished affective and behavioral response to rewards, including a lack of interest in positive incentives and a blunted emotional response to their potential attainment.

After identifying these dimensions, items were generated to measure each dimension through brainstorming and discussion, drawing heavily on the authors’ clinical and research expertise. Symptoms and measures of youth psychopathology were reviewed, and CRRS items were carefully crafted to minimize conceptual overlap with existing measures. Three to four items were generated for each hypothesized dimension, resulting in 25 items. Positive and negative item phrasing was used to capture potential nuances in responses (Bansal et al., 2022; Ray et al., 2016; Weijters et al., 2010; Willoughby et al., 2015). The 25 items were rated by parents using a Likert-type response format that ranged from 0 (strongly disagree) to 4 (strongly agree). A 5-point Likert-type scale was used because it approximates a continuous scale (Bollen & Barb, 1981; Srinivasan & Basu, 1989), reduces response time, cognitive effort, and cognitive bias (Chen et al., 2015; Weijters et al., 2010), allows for the incorporation of commonly used response anchors (Chyung et al., 2017), and is widely used in assessment measures. The order of items was randomized. The CRRS is shown in the Appendix.

Disruptive Behavior Disorders Rating Scale

The DBDRS is a 45-item measure that assesses symptoms of ADHD, ODD, and CD using a Likert-type response format ranging from 0 (not at all) to 3 (very much) (Pelham, Gnagy, et al., 1992). Parents of 186 participants and teachers of 168 participants rated children using the DBDRS. Mean scores were calculated to derive the following scores: ADHD (Parent: M = 1.78, SD = 0.72, α = .94; Teacher: M = 1.72, SD = 0.83, α = .96), ODD (Parent: M = 1.32, SD = 0.75, α = .90; Teacher: M = 1.12, SD = 0.94, α = .94), and CD (Parent: M = 0.24, SD = 0.28, α = .82; Teacher: M = 0.38, SD = 0.55, α = .70). The psychometric properties of these scores, including factor structure, reliability, and validity, have been supported (Fosco et al., 2023; Pelham, Evans, et al., 1992; Pelham, Gnagy, et al., 1992; Wright et al., 2007).

Inventory of Callous-Unemotional Traits

The ICU is a 24-item rating scale that assesses CU traits using a Likert-type response format ranging from 0 (not at all true) to 3 (definitely true) (Frick, 2004). Parents of 192 participants and teachers of 159 participants rated children on the ICU. Total scores were calculated after reverse-scoring relevant items (Parent: M = 26.31, SD = 11.87, α = .90; Teacher: M = 32.63, SD = 13.53, α = .91). The psychometric properties of the ICU, including factor structure, reliability, and validity, have been supported (Bansal et al., 2022; Kimonis et al., 2008; Ray & Frick, 2020; Willoughby et al., 2015).

Impairment Rating Scale

The Impairment Rating Scale (IRS) consists of seven items for the teacher and nine for the parent versions. Items are designed to assess a child’s current functioning and need for treatment or additional services across developmentally important domains (Fabiano et al., 2006). Items are rated using a visual analog scale ranging from 0 (no problem; definitely does not need treatment or extra service) to 6 (extreme problem; definitely does need treatment or extra services).² Parents of 189 participants and teachers of 158 participants rated children on the IRS. Mean impairment scores were calculated for parent ratings (M = 3.92, SD = 1.62, α = .91) and teacher ratings (M = 3.82, SD = 1.60, α = .86). The psychometric properties of the IRS, including reliability and validity, have been supported (Fabiano et al., 2006; Waschbusch et al., 2004; Waschbusch & Willoughby, 2008).

Procedure

Prior to participation, parents provided written consent and children provided assent. The study adhered to the principles of the Declaration of Helsinki and was approved by the Florida International University Institutional Review Board (IRB Approval # 082710-00). We report how we determined our sample size, all data exclusions, all manipulations, and all measures in the study. The sample size was determined by including data from consecutive evaluations of children 5 to 12 years old who were in projects conducted by a treatment research program over 2 years and provided informed consent and assent to participate. Only measures pertinent to the study’s aims were included in this study, but other measures were also collected. Data are available upon request from the corresponding author.

Data were gathered during a baseline assessment of a summer research and treatment program. Parents and teachers were instructed to rate the child’s unmedicated behavior when relevant. To assess the test–retest reliability of the new measure, parents of a random sample of 30 participants (15%) completed the ratings twice: at baseline and at retest approximately 1 month later (M = 5.1 weeks, SD = 1.7). Retest ratings were conducted over the phone by a research assistant who read each item to the parent and recorded their responses. The same informant provided ratings at each point in time.

Participants were categorized into groups based on their diagnoses of ADHD, conduct problems (CP; i.e., ODD or CD), and CU for one analysis. Clinicians established diagnoses of ADHD, ODD, and CD using criteria in the Diagnostic and Statistical Manual of Mental Disorders (4th ed., text rev.; DSM-IV-TR; APA, 2000), informed by parent and teacher ratings on the DBDRS and the IRS, and parent report on the Disruptive Behavior Disorders Interview (Babinksi & Waschbusch, 2022; Pelham, 1998). The diagnostic distribution was as follows: ADHD-only (n = 39, 19.9%), CP-only (n = 2, 1.0%), and CP/ADHD (n = 138, 70.4%). Among the 140 children with CP, 82 had ODD-only, one had CD-only, and 57 had both ODD and CD. CU traits were assessed by combining parent and teacher ratings on the ICU, and participants in the upper 20% percentile (score of 46 or higher) were classified as having CU. Based on these procedures, participants were categorized into four groups: (a) Controls (n = 17, 8.9%), did not meet the criteria for ADHD, CP, or CU; (b) ADHD-only (n = 37, 19.3%), met the criteria for ADHD but not CP or CU; (c) CP/ADHD (n = 98, 51.0%), met the criteria for CP and ADHD but not CU; and (d) CPCU/ADHD (n = 40, 20.8%), met the criteria for CP, ADHD, and CU. Four participants were excluded because they did not meet the criteria for any group, including two with CP-only and two with ADHD/CU.

Analytic Plan

Six sets of analyses were conducted. First, item-level descriptive statistics, Bartlett’s sphericity test, and the Kaiser–Meyer–Olkin (KMO) statistic were computed to assess the data’s appropriateness for factor analysis. Second, the factor structure of the CRRS was examined by performing an EFA following the procedures outlined by Watkins (2018). Parallel analysis was used to estimate the number of factors to extract (Horn, 1965). Next, iterative principal axis factoring was used to extract factors due to its relative tolerance of non-normality and sensitivity to weak factors (Tabachnik & Fidell, 2012), and oblique (oblimin) rotation was used as factors were hypothesized to be interrelated. Items were interpreted based on the size of the factor loadings on the pattern matrix and theoretical fit. Items exhibiting complex loadings, defined as loadings greater than or equal to .40 on multiple factors, were excluded from further analysis. Third, scale scores derived from the CRRS based on the factor analysis were investigated by calculating internal consistency reliabilities (Cronbach’s alpha and McDonald’s omega), test–retest reliability, inter-scale correlations, and descriptive statistics.

Fourth, associations between the CRRS scores and demographic measures were investigated by calculating Pearson’s and point-biserial correlations. Fifth, associations between CRRS scores and measures of ADHD, ODD, CD, CU, and impairment were examined by computing Pearson correlations and linear regressions. The linear regressions determined whether CRRS scores were uniquely associated (i.e., controlling other CRRS scores) with psychopathology and impairment outcomes. Sixth, diagnostic groups were compared on CRRS scores by computing one-way analyses of variance (ANOVAs) with CRRS scores as the dependent variables and psychopathology groups (control vs. ADHD-only vs. CP/ADHD vs. CPCU/ADHD) as the independent variable. Significant group differences were followed up with Ryan–Einot–Gabriel–Walsch post hoc tests that control family-wise error while minimizing loss of power (Kim, 2015).

Data analyses were conducted using SPSS version 28, except for the parallel analysis conducted in MPLUS version 8.8. For all analyses, the significance level was set at .05, and correlations of .10, .30, and .50 and standardized mean differences of .20, .50, and .80 were interpreted as small, medium, and large effects, respectively (Cohen, 1988).

Results

Item Analyses

Descriptive statistics of CRRS items, presented in Table 1, indicated that the full range of the scale was used for all items, implying that items had meaningful variability. Bartlett’s test of sphericity indicated that the correlation matrix was significantly different from random error, χ²(300) = 2,298.47, p < .001, and the KMO statistic was 0.86, well exceeding the threshold of 0.50 for conducting factor analysis. These findings suggested that the correlation matrix was suitable for factor analysis.

Table 1

Descriptive Statistics for Initial Pool of Items Generated for the Contingency Response Rating Scale.

Hypothesized component/item				Percent endorsed
Hypothesized component/item	N	M	SD	Strongly disagree	Disagree	Neutral	Agree	Strongly agree
Punishment dysregulation
Punishment makes him or her mad	196	2.99	1.05	4.1	5.1	15.3	38.3	37.2
Gets annoyed or mad when punished	196	2.92	1.05	3.6	8.7	11.7	44.4	31.6
Becomes angry when corrected or punished	194	2.49	1.17	8.2	12.9	19.1	41.2	18.6
Punishment insensitivity
Does not care if she or he is punished	195	1.87	1.09	27.2	42.96	9.7	14.4	6.2
Seems indifferent to punishment	195	1.27	1.11	26.4	40.4	11.4	15.5	6.2
Shows little or no emotion when punished	196	2.07	1.11	31.6	44.9	9.7	10.2	3.6
Acts like he or she doesn’t care about punishment	194	1.33	1.21	26.89	41.2	11.9	12.4	7.7
Punishment ineffectiveness
Punishment is effective with my child	193	1.35	1.20	7.7	19.5	30.3	36.4	6.2
Punishment does not work with my child	194	1.59	1.20	19.6	34.0	21.6	17.0	7.7
Punishments are ineffective with my child	194	1.46	1.16	20.6	39.2	20.1	13.4	6.7
Punishments work well with my child	185	2.06	1.11	11.4	16.2	36.2	27.6	8.6
Reward receipt dysregulation
Gets overexcited when rewarded	196	1.09	1.07	7.7	13.3	31.1	34.7	13.3
Gets overly silly when rewarded	196	2.33	1.10	12.8	24.6	29.7	28.7	4.1
Acts “out of control” when rewarded	193	1.23	1.12	33.7	26.4	26.4	10.4	3.1
Reward insensitivity
Acts like he or she doesn’t care about rewards	193	1.17	1.05	29.0	41.5	16.6	9.8	3.1
Highly motivated by rewards	194	2.63	0.97	2.1	11.3	25.3	43.8	17.5
Seems indifferent to rewards	194	1.12	0.96	27.8	43.3	19.1	8.2	1.5
Does not care if he or she is rewarded	196	1.09	0.97	29.6	43.9	16.3	8.7	1.5
Reward ineffectiveness
Rewards are ineffective with my child	195	1.30	1.19	26.7	40.5	16.9	11.3	4.6
Rewards are effective with my child	196	2.73	0.98	2.6	9.2	21.9	45.4	20.9
Rewards do not work with my child	193	1.16	1.03	30.6	37.3	20.2	9.8	2.1
Rewards work well with my child	185	2.67	1.00	3.8	7.6	26.5	42.2	20.0
Reward anticipation dysregulation
Once my child learns he or she can earn a reward, he or she can’t think of anything else	195	2.14	1.05	8.7	24.0	26.5	33.2	7.7
Is always asking about getting a reward	194	2.22	1.21	10.8	18.0	23.7	33.5	13.9
Is highly focused on earning rewards	194	2.01	1.13	10.8	22.2	30.9	27.3	8.8

Note. Items were scored using a 0 to 4 metric (see Appendix).

Exploratory Factor Analysis

Parallel analysis suggested that five factors should be extracted. The five-factor solution, shown in Table 2, demonstrated that each factor had at least three items with statistically significant factor loadings. After extraction, the five factors collectively explained 59.7% of the total variance. After rotation, each factor individually explained 11.0% to 20.6% of the total variance. One item exhibited significant loadings on two factors and was eliminated from further analyses.

Table 2

Factor Loadings From the Pattern Matrix of Iterative Principal Axis Factor With Oblique Rotation.

Item	Factor
Item	Pine	Rine	Pdys	Rdys	Cins
Punishment ineffectiveness (Pine)
Punishment does not work with my child	.72	.16	−.01	−.01	.22
Punishment is effective with my child	−.70	−.10	.19	.07	−.08
Punishments work well with my child	−.65	−.23	.25	.11	.05
Punishments are ineffective with my child	.64	.20	−.10	.01	.20
Reward ineffective (Rine)
Rewards work well with my child	−.26	−.82	−.12	−.18	−.01
Highly motivated by rewards	−.19	−.69	.06	.14	−.01
Rewards are effective with my child	−.10	−.68	.02	−.09	−.04
Is highly focused on earning rewards	.09	−.64	−.06	.13	.01
Rewards do not work with my child	.28	.60	−.01	.19	.14
Rewards are ineffective with my child	.27	.38	−.07	.17	.23
Punishment dysregulation (Pdys)
Punishment makes him or her mad	−.03	.02	−.82	.12	−.06
Becomes angry when corrected or punished	.12	−.09	−.81	−.08	.04
Gets annoyed or mad when punished	.04	−.07	−.80	.11	−.02
Reward dysregulation (Rdys)
Gets overly silly when rewarded	.00	.06	.00	.74	.04
Gets overexcited when rewarded	−.02	.00	−.11	.67	−.11
Acts “out of control” when rewarded	−.19	.25	−.05	.64	.19
Once my child learns he or she can earn a reward, he or she can’t think of anything else	.00	−.36	−.02	.47	.00
Is always asking about getting a reward	.14	−.31	−.29	.45	.00
Contingency insensitivity (Cins)
Does not care if he or she is rewarded	−.26	.22	−.05	−.09	.74
Seems indifferent to punishment	.22	−.11	−.08	.05	.72
Shows little or no emotion when punished	.16	−.19	.13	.11	.67
Acts like he or she doesn’t care about punishment	.24	−.11	−.05	−.04	.67
Does not care if she or he is punished	.19	.02	.12	.26	.60
Acts like he or she doesn’t care about rewards	−.09	.28	−.08	−.04	.59
No factor (complex loading item)
Seems indifferent to rewards	−.25	.40	−.21	−.09	.52

Note. Items included on a factor are in bold font.

CRRS Scale Scores

Descriptive Statistics, Reliabilities, and Scale Variances

CRRS scores were calculated by averaging items (with reverse scoring applied when necessary) that loaded on factors identified in the EFA. Descriptive statistics, including means, standard deviations, ranges, and reliability estimates, for these scores are presented in Table 3. Each scale demonstrated acceptable reliability in terms of internal consistency and test–retest reliability. The means of the scales were close to the scale’s midpoint (two), and they exhibited variability within the sample.

Table 3

Descriptive Statistics, Reliability Estimates, and Pearson Correlations for CRRS Scores.

Scale	Descriptive statistics			Reliability			Correlations
Scale	M	SD	Range	α	ω	r _1,2	Pine	Rine	Pdys	Rdys
Punishment ineffectiveness (Pine)	1.71	0.97	0–4	.89	.88	.77	–
Reward ineffectiveness (Rine)	1.40	0.79	0–3.6	.84	.84	.68	.55*	–
Punishment dysregulation (Pdys)	2.80	0.97	0–4	.87	.87	.61	.33*	.03	–
Reward dysregulation (Rdys)	1.94	0.80	0–4	.76	.75	.65	.13	−.02	.33*	–
Contingency insensitivity (Cins)	1.22	0.84	0–3.8	.84	.84	.88	.52*	.49*	.14*	.20*

Note. α = Cronbach’s alpha internal consistency reliability; ω = McDonald’s omega internal consistency reliability; r_1,2 = test–retest reliability (n = 30).

Correlation significantly different from zero at p < .05.

Inter-Scale Correlations

Pearson correlations between the CRRS scores, presented in Table 3, ranged from −0.02 to 0.52. Significant correlations revealed that the contingency insensitivity score was positively correlated with other scores, with correlations ranging from small to large. A significant positive correlation was also observed between punishment ineffectiveness and reward ineffectiveness. Punishment dysregulation exhibited moderate positive correlations with both reward dysregulation and punishment ineffectiveness.

Correlations With Demographic Measures

Pearson and point-biserial correlations were additionally computed to investigate whether CRRS scores were associated with child or parent demographic characteristics. Most of these correlations, presented in Table 4, were nonsignificant and small, suggesting that CRRS scores were largely independent of demographic characteristics.

Table 4

Correlations Between Demographic Measures and CRRS Scores.

		CRRS scores
Measure	N	Pine	Rine	Pdys	Rdys	Cins
Child variables
Age	196	.09	.06	.06	−.03	.08
Sex (girl = 0, boy = 1)	196	−.03	−.12	.01	.09	−.06
Race (White = 0, minority = 1)	186	.12	.15*	.06	.16*	.08
Hispanic (no = 0, yes = 1)	189	−.19*	.01	−.06	.05	−.08
IQ	182	.05	.02	−.03	.01	−.05
Read	175	−.03	.02	−.08	.00	−.05
Writing	179	.02	.05	−.11	−.08	−.02
Math	178	.09	.08	−.06	−.09	.12
Parent variables
Informant (mom = 0, dad = 1)	196	−.04	.07	−.09	−.02	.07
Biological parent (no = 0, yes = 1)	196	−.17*	−.06	−.15*	.01	−.14
Age	173	−.14	−.06	.00	.00	.00
Race (White = 0, minority = 1)	181	.03	−.01	.06	.12	−.04
Hispanic (no = 0, yes = 1)	184	−.23*	.03	−.11	.05	−.07
See child daily (no = 0, 1 = yes)	179	−.06	.01	−.03	.02	−.14

Note. Pine = Punishment Ineffectiveness; Rine = Reward Ineffectiveness; Pdys = Punishment Dysregulation; Rdys = Reward Dysregulation; Cind = Contingency Insensitivity.

Correlation differs from zero at p < .05.

Associations With ADHD, ODD, CD, CU, and Impairment

First, Pearson correlations were computed to examine associations between CRRS scores and psychopathology (ADHD, ODD, CD, CU) or impairment scores (see Table 5). CRRS scores showed significant correlations with all parent-rated outcomes, ranging from small to large effect sizes. Similarly, significant correlations were found for several teacher-rated outcomes, with effect sizes ranging from small to moderate. Next, linear regression was performed to investigate whether CRRS scores predicted these same outcomes (see Table 6).³ All regression models predicting parent-rated outcomes were statistically significant, with explained variance ranging from 20% to 38%. These findings revealed differential associations between CRRS and psychopathology scores. Specifically, contingency insensitivity was associated with ODD, CD, and CU but not ADHD or impairment. Conversely, reward dysregulation was associated with ADHD and impairment but not ODD, CD, or CU. Regressions predicting teacher-rated CD, CU, and impairment were also significant, explaining 5% to 7% of the variance. Parent-rated reward dysregulation was associated with teacher-rated CD and impairment, while parent-rated contingency insensitivity was associated with teacher-rated CU.

Table 5

Pearson Correlations and Correlation Difference Tests Between CRRS and Other Measures.

	ADHD		ODD		CD		CU		Impair
	Parent	Teacher	Parent	Teacher	Parent	Teacher	Parent	Teacher	Parent	Teacher
1. Pine	.36*	.14	.45*	.16*	.31*	.21*	.48*	.21*	.39*	.19*
2. Rine	.19*	.13	.24*	.09	.26*	.18*	.29*	.18*	.20*	.06
3. Pdys	.35*	.03	.52*	.13	.28*	.11	.28*	.08	.29*	.17*
4. Rdys	.32*	.15*	.26*	.15	.25*	.19*	.22*	.02	.27*	.24*
5. Cins	.23*	.13	.37*	.16*	.43*	.20*	.42*	.27*	.25*	.17*
Diff Tests	1 > 2	ns	1, 3 > 2, 4	ns	5 > 2, 4	ns	1 > 2, 45 > 4	5 > 4	1 > 2, 5	ns

Note. Sample sizes ranged from 186 to 192 for parent ratings and 158 to 168 for teacher ratings. Diff Tests = William’s t-tests for comparing single sample correlations (Steiger, 1980); Pine = Punishment Ineffectiveness; Rine = Reward Ineffectiveness; Pdys = Punishment Dysregulation; Rdys = Reward Dysregulation; Cind = Contingency Insensitivity; a = average symptom score from the Disruptive Behavior Disorders Rating Scale (Pelham, Gnagy, et al., 1992). Callous-Unemotional sum score from the Inventory of Callous-Unemotional Traits (Frick, 2004). Average scores from the Impairment Rating Scale (Fabiano et al., 2006). ns = all correlation difference tests were nonsignificant.

p < .05.

Table 6

Regressions Predicting ADHD, ODD, CD, CU, and Impairment From CRRS Scales.

	ADHD		ODD		CD		CU		Impair
	Parent	Teacher	Parent	Teacher	Parent	Teacher	Parent	Teacher	Parent	Teacher
Pine	.13*[.06, .40]	.08[−.12, .28]	.19*[.05, .32]	.06[−.17, .14]	.01[−.16, .15]	.09[−.07, .26]	.31*[.11, .54]	.08[−.14, .27]	.33*[.15, .51]	.15[−.03, .34]
Rine	.07[−.09, .21]	.09[−.10, .29]	.05[−.07, .18]	.02[−.13, .26]	.11[−.06, .29]	.11[−.07, .32]	.01[−.17, .20]	.03[−.16, .22]	−.02[−19, .16]	−.07[−.34, .15]
Pdys	.20*[.06, .37]	−.04[−.19, .14]	.40*[.25, .55]	.07[−.11, .24]	.19*[.05, .31]	.03[−.13, .20]	.12[−.03, .26]	.04[−.15, .21]	.12[−.03, .27]	.07[−.09, .26]
Rdys	.22*[.09, .35]	.17[−.01, .32]	.07[−.04, .18]	.11[−.05, .26]	.12[−.01, .27]	.18*[.01, .34]	.09[−.03, .21]	−.04[−.20, .12]	.19*[.04, .33]	.21*[.03, .38]
Cins	.00[−.12, .14]	.02[−.15, .16]	.18*[.03, .32]	.09[−.07, .24]	.32*[.12, .51]	.06[−.11, .19]	.22*[.05, .38]	.21*[.05, .36]	.04[−.10, .17]	.09[−.08, .25]
F	11.14*	1.64	23.18*	1.76	12.01*	3.06*	15.69*	2.69*	10.12*	3.08*
Adj R²	.22	.02	.38	.02	.23	.06	.28	.05	.20	.07

Note. Values in the table are standardized regression coefficients with bootstrapped 95% confidence intervals in brackets. Bootstrapped confidence intervals were bias-corrected and accelerated (BCa) and estimated with 1,000 samples. Pine = Punishment Ineffectiveness; Rine = Reward Ineffectiveness; Pdys = Punishment Dysregulation; Rdys = Reward Dysregulation; Cins = Contingency Insensitivity.

= p < .05.

Comparisons of Psychopathology Groups

One-way ANOVAs comparing psychopathology groups (Control vs. ADHD-only vs. CP/ADHD vs. CPCU/ADHD) revealed significant differences in punishment ineffectiveness, punishment dysregulation, reward dysregulation, and contingency insensitivity but not reward ineffectiveness (see Table 7). Post hoc tests revealed that the CP/ADHD and CPCU/ADHD groups had higher punishment ineffectiveness and punishment dysregulation scores than the control and ADHD-only groups. In addition, the CPCU/ADHD group had higher contingency indifference scores than all other groups and higher reward dysregulation than the control and ADHD-only groups. The CP/ADHD group also had higher reward dysregulation than the control group.

Table 7

Means, Standard Deviations, ANOVA Results, and Effect Sizes for Psychopathology Group Comparisons on CRRS Scores.

	TD	ADHD-only	CP/ADHD	CPCU/ADHD		Standardized mean difference effect sizes
	M (SD)	M (SD)	M (SD)	M (SD)	F-value	TDvs. ADHD	TDvs.CP/AD	TDvs. CPCU	ADHDvs.CP/AD	ADHD vs. CPCU	CP/ADvs.CPCU
Pine	1.06_a (0.60)	1.25_a (0.81)	1.81_b(0.91)	2.12_b(1.13)	8.99*	−0.19	−0.77*	−1.08*	−0.57*	−0.89*	−0.32
Rine	1.10(0.45	1.28(0.74)	1.40(0.76)	1.59(0.93)	1.97	−0.23	−0.38	−0.63	−0.15	−0.40	−0.24
Pdys	2.31_a(0.76)	2.05_a(1.06)	3.09_b(0.81)	3.02_b(0.91)	15.25*	0.27	−0.80*	−0.73*	−1.07*	−1.00*	0.07
Rdys	1.48_a(0.74)	1.75_ab(0.85)	2.04_bc(0.73)	2.17_c(0.77)	4.58*	−0.35	−0.72*	−0.88*	−0.37	−0.54*	−0.17
Cins	0.82_a(0.55)	0.98_a(0.74)	1.19_a(0.74)	1.62_b(1.08)	5.74*	−0.19	−0.44	−0.95*	−0.25	−0.76*	−0.51*

Note. TD = typically developing group (n = 17); ADHD = ADHD-only group (n = 37); CP/AD = conduct problems and ADHD group (CP/ADHD, n = 98); CPCU = conduct problems, ADHD, and CU group (CPCU/ADHD, n = 40); Pine = Punishment Ineffectiveness; Rine = Reward Ineffectiveness; Pdys = Punishment Dysregulation; Rdys = Reward Dysregulation; Cins = Contingency Insensitivity. Within each row, means that do not share a subscript differed in Ryan–Einot–Gabriel–Welsch F post hoc comparisons. Degrees of freedom for all ANOVAs were 3, 188. Child age and sex were included as covariates in preliminary analyses but were dropped from final models because they were never significant. Standardized mean difference effect sizes were computed as: (M_{Group 1}−M_{Group 2})/SD_{Full sample}.

p < .05 for overall ANOVA or for post hoc tests.

Discussion

This study sought to develop and evaluate the psychometric properties of the CRRS, a novel assessment tool to evaluate children’s responses to rewards and punishments. CRSs are crucial for understanding the development, maintenance, and treatment of children’s externalizing psychopathology and callousness. However, evaluating CRSs has been underemphasized in clinical practice, potentially attributable to the lack of a brief, easy-to-use, and psychometrically sound measure of children’s response to reward and punishment that can be easily integrated into a clinical evaluation. This study makes an initial contribution to addressing this gap. The study’s findings revealed that the instrument’s items were categorized into five subscales: (a) punishment ineffectiveness, (b) reward ineffectiveness, (c) punishment dysregulation, (d) reward dysregulation, and (e) contingency insensitivity. The five subscales exhibited low to moderate correlations with each other, and data supported their test–retest and internal consistency reliabilities (see Table 3). The subscales captured meaningful variations among children that were not explained by child or parent demographics (see Table 4). Furthermore, some subscales of the CRRS showed significant and unique associations with psychopathology and impairment scores.

Our initial hypothesis was that the EFA would support a seven-factor model of the CRRS, encompassing factors of effectiveness, dysregulation, and insensitivity, separately for reward and punishment, and with reward dysregulation differentiated into anticipation and receipt of reward (see Table 1). The EFA yielded a five-factor model (see Table 2), partially supporting our initial hypothesis. In support of our initial hypothesis, punishment dysregulation, reward effectiveness, and punishment effectiveness emerged as distinct factors. However, our hypothesis regarding the differentiation of reward dysregulation into anticipation and receipt of reward was not supported. Also, all insensitivity items loaded on a single factor. It is unclear why our initial hypothesis was not fully supported. One possibility is that these distinctions are challenging for parents to observe or report on, potentially reflecting a limitation of the new assessment tool. Alternatively, it may reflect an actual finding that children who exhibit insensitivity to punishment also exhibit insensitivity to reward.

Our second hypothesis was that scores based on factors would exhibit adequate test–retest and internal consistency reliability and significant yet moderate inter-scale correlations. This hypothesis was also partially supported (see Table 3). In support of this hypothesis are findings that the scales exhibited satisfactory internal consistency, test–retest reliability, and some statistically significant inter-scale correlations. Other findings did not fully support this hypothesis due to some inter-scale correlations being low. These findings and the factor analysis results indicate that parents can reliably differentiate among these nuanced aspects of children’s reward and punishment response styles and that reward and punishment response styles are distinct constructs that are, at most, moderately correlated. These findings also suggest that parents know how their child responds to specific consequences and whether they are effective, emphasizing the importance of a comprehensive assessment of children’s CRSs. For instance, a parent might observe that their child with ADHD experiences distress when punished (i.e., punishment dysregulation) but also report that punishment is generally effective in addressing their child’s misbehavior (i.e., punishment effectiveness). As such, this study provides additional insight to inform assessment and treatment practices.

Our third hypothesis posited that CRRS scores would exhibit significant associations with measures of ADHD, CP, CU, and impairment, as assessed in correlational and regression analyses. Findings corroborated this hypothesis, as evidenced by Pearson correlations (see Table 5). These analyses showed that higher scores across CRRS scales, including higher punishment and reward ineffectiveness, higher punishment and reward dysregulation, and higher contingency insensitivity, were all associated with higher parent-rated child psychopathology (ADHD, ODD, CD, CU) and impairment scores. CRRS scores also demonstrated expected associations with some teacher-rated outcomes, although not all correlations were statistically significant (see Table 5). Interestingly, in comparison to reward ineffectiveness, punishment ineffectiveness exhibited stronger associations with most parent-rated ADHD, ODD, CD, CU, and impairment scores. This finding aligns with studies that demonstrate the importance of including appropriate punishment techniques, such as response-cost strategies, alongside reward-based approaches to manage attention and behavior difficulties (Carlson & Tamm, 2000; Pfiffner & Oleary, 1987; Pfiffner, Oleary, et al., 1985; Pfiffner, Rosen, & O’Leary, 1985). These findings highlight the potential utility of the CRRS, with further development, in identifying children who may benefit from intervention targeting their specific CRSs.

To further explore the specific relationships between CRRS scores and outcomes, multiple linear regressions were computed with CRRS scores as predictors and ADHD, ODD, CD, CU, and impairment as the dependent variables. These analyses aimed to determine whether CRRS scores exhibited unique associations with outcomes even after accounting for potential interdependencies among the subscales. Results (see Table 6) revealed that higher punishment ineffectiveness scores uniquely predicted each parent-rated outcome except CD. Conversely, reward ineffectiveness did not uniquely predict any parent-rated outcome. This finding suggests that, from parents’ perspectives, punishment effectiveness plays a more critical role in controlling children’s misbehavior than reward effectiveness. In other words, parents may perceive children’s misbehavior primarily stemming from a lack of punishment effectiveness rather than a lack of reward effectiveness. This observation aligns with previous studies suggesting that many parents view punishment as the primary tool for effective child behavior management (Holden et al., 1999).

Regressions further revealed that reward dysregulation emerged as a significant predictor of both parent-rated and teacher-rated impairment. This suggests that children who exhibit excessive excitement or silliness when rewarded or even anticipating a reward are more likely to experience functional limitations than their peers. These findings align with existing research highlighting significant associations between various forms of dysregulation and impairment (e.g., Caro-Canizares et al., 2020; Graziano & Garcia, 2016; Waschbusch et al., 2004, 2022; Waschbusch & Willoughby, 2008). Furthermore, contingency insensitivity emerged as a significant predictor of both parent and teacher ratings of CU, indicating a robust and unique link between these constructs. Consistent with this observation, group comparisons revealed that the CPCU/ADHD group exhibited significantly higher contingency insensitivity compared with all other groups, including the CP/ADHD group. These results echo prior theoretical and empirical work demonstrating a significant and unique association between a diminished punishment response and CU traits in youth (Budhani & Blair, 2005; Dadds & Salmon, 2003; Frick & Marsee, 2018; Nichols et al., 2015; Zhang et al., 2023). These findings further highlight the CRRS’s potential to inform treatment planning by elucidating the specific patterns of how children respond to different types of consequences. For instance, children exhibiting high reward dysregulation alongside impairment might benefit from interventions targeting emotional regulation in response to rewards, alongside traditional interventions for ADHD, CP, CU, and impairment. Similarly, interventions to increase sensitivity to punishment consequences could benefit children with high levels of contingency insensitivity and CU traits. By identifying these nuanced response patterns, the CRRS holds the promise of contributing to the development of more targeted and effective interventions for children with externalizing psychopathology and functional difficulties.

Taken together, the results of this study paint a compelling picture of how parents view responses to reward and punishment strategies in children with high levels of CP, CU, and ADHD. As parent ratings suggest, some youths become enraged when punished, while others demonstrate excessive excitement or silliness when rewarded. Still other youth appear insensitive to all forms of consequences. In other words, a fascinating tapestry of individual differences emerges in children’s CRSs, even within the broader category of attention and behavior problems. Digging deeper into these individual variations and their potential role in explaining treatment response, as initial research has begun to explore (Staff et al., 2021; van Langen et al., 2021), holds immense promise. Arguably, these findings suggest that many parents are grappling with how to effectively discipline their children with CPCU/ADHD, lending support to the notion that parenting programs based on social learning theory should be an integral component of treatment for youth with CPCU (Hawes et al., 2014). Moreover, the results highlight a striking inconsistency in how children with CPCU/ADHD respond to punishment. Parents report these youth as prone to anger when punished and indifferent to punishment. Designing research to understand this pattern better and developing strategies for implementing punishment more consistently with these children is a crucial goal for future research endeavors (Waschbusch et al., 2020).

This study’s findings hold promise, but it is crucial to acknowledge some limitations. First, the modest sample size, mainly composed of treatment-seeking children, limits the generalizability to other groups, such as high-risk youth from disadvantaged backgrounds. Replicating these results in larger, diverse samples is vital to solidify our understanding of how response styles influence externalizing psychopathology, as well as other forms of psychopathology, across contexts. Second, the lack of additional measures of reward and punishment responses prevented the evaluation of the CRRS’s criterion validity. Future studies incorporating established measures can strengthen its validity claims by demonstrating convergence with existing instruments. Third, the paucity of father ratings hindered the separate analyses of maternal and paternal perspectives. Examining the role of different informants is important, aligning with established practices in assessing other aspects of child functioning (De Los Reyes & Epkins, 2023). Capturing multiple informant viewpoints will enrich our understanding of contingency response patterns. Fourth, while the limited item count for scales minimizes respondent burden and reduces the chance of overlap with other measures, it also compromises the breadth of content captured. Future iterations of the CRRS could explore strategies to balance item count with content comprehensiveness, potentially through innovative administration forms incorporating branching logic. Finally, the lack of a specific time frame for informants to consider when responding raises questions about potential impacts on their reports. Specifying a time frame (e.g., “at present” or “in the past month”) in future revisions of the CRRS could enhance consistency and clarity in responses.

Even considering these limitations, this study is a significant first step toward a psychometrically sound, user-friendly measure of children’s responses to reward and punishment responses, valuable for applied settings. We emphasize our view of this as an initial foray. Measure development is an iterative process: evaluating, revising, and re-evaluating (American Educational Research Association et al., 2014). This study marks the CRRS’s first step, and much remains to be done. First, replicating the factor structure and other findings in more diverse clinical samples (e.g., children with internalizing, externalizing, or no disorders), larger samples, and multiple informants will solidify the measure’s robustness and guide revisions. Second, further criterion validity studies are needed to assess whether CRRS scales significantly correlate with other reward and punishment response measures, like performance tasks, home observations, and clinical data. Third, longitudinal studies will provide crucial information on the CRRS’s predictive validity. Finally, examining whether CRRS scores can predict or personalize treatment responses remains a vital long-term goal. For example, it would be fascinating to see if the CRRS scores can guide tailoring standard behavior therapy to match children’s CRSs and improve treatment outcomes.

This initial exploration paves the way for future studies to refine and validate the CRRS. Ultimately, we aim to empower practitioners with a tool that can illuminate the intricate tapestry of children’s responses to reward and punishment, paving the path toward more targeted and effective interventions for those struggling with externalizing psychopathology and functional difficulties.

Footnotes

Appendix

Authors’ Note

Daniel A. Waschbusch approved the final version of the manuscript. He was responsible for the overall conceptualization of the paper, conducted analyses, and wrote the first draft of the paper.

Author Contributions

All other authors contributed to interpreting the data and preparing the manuscript.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest concerning the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by a grant from the National Institute of Mental Health (7R34MH085796) to the first author.

Ethical Approval

The study was approved by the Institutional Review Board. Informed consent was collected from participants.

ORCID iDs

Daniel A. Waschbusch

Vanessa T. Cao

Data Availability Statement

Data and study materials are available upon request to the first author.

Notes

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing.

American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th text revision ed.). https://doi.org/10.1176/appi.books.9780890420249.dsm-iv-tr

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). https://doi.org/10.1176/appi.books.9780890425596

Babinksi

D. E.

Waschbusch

D. A.

(2022). Disruptive behavior disorders in children and adolescents. In Storch

E. A.

(Ed.), Comprehensive clinical psychology: Vol. 5. Case conceptualization and treatment: Children and adolescents (2nd ed., Vol. 5, pp. 412–423). Elsevier.

Bansal

P. S.

Babinski

D. E.

Waxmonsky

J. G.

Waschbusch

D. A.

(2022). Psychometric properties of caregiver ratings on the Inventory of Callous Unemotional Traits in a nationally representative sample of 5 to 12 year-olds. Assessment, 29(2), 242–256. https://doi.org/10.1177/107319112096456

Becker

S. P.

Fite

P. J.

Garner

A. A.

Greening

Stoppelbein

Luebbe

A. M.

(2013). Reward and punishment sensitivity are differentially associated with ADHD and sluggish cognitive tempo symptoms in children. Journal of Research in Personality, 47(6), 719–727. https://doi.org/10.1016/j.jrp.2013.07.001

Blair

R. J. R.

Leibenluft

Pine

D. S.

(2014). Conduct disorder and callous-unemotional traits in youth. The New England Journal of Medicine, 371(23), 2207–2216. https://doi.org/10.1056/NEJMra1315612

Blair

R. J. R.

White

S. F.

Meffert

Hwang

(2014). Disruptive behavior disorders: Taking an RDoC(ish) approach. Current Topics in Behavioral Neurosciences, 16, 319–336. https://doi.org/10.1007/7854_2013_247

Bollen

K. A.

Barb

K. H.

(1981). Pearson’s R and coarsely categorized measures. American Sociological Review, 46, 232–239.

10.

Budhani

Blair

R. J.

(2005). Response reversal and children with psychopathic tendencies: Success is a function of salience of contingency change. Journal of Child Psychology and Psychiatry, 46(9), 972–981. https://doi.org/10.1111/j.1469-7610.2004.00398.x

11.

Byrd

A. L.

Hawes

S. W.

Waller

Delgado

M. R.

Sutherland

M. T.

Dick

A. S.

Trucco

E. M.

Riedel

M. C.

Pacheco-Colón

Laird

A. R.

Gonzalez

(2021). Neural response to monetary loss among youth with disruptive behavior disorders and callous-unemotional traits in the ABCD Study. NeuroImage: Clinical, 32, 102810. https://doi.org/10.1016/j.nicl.2021.102810

12.

Byrd

A. L.

Loeber

Pardini

D. A.

(2014). Antisocial behavior, psychopathic features and abnormalities in reward and punishment processing in youth. Clinical Child and Family Psychology Review, 17(2), 125–156. https://doi.org/10.1007/s10567-013-0159-6

13.

Carlson

C. L.

Tamm

(2000). Responsiveness of children with attention deficit-hyperactivity disorder to reward and response cost: Differential impact on performance and motivation. Journal of Consulting and Clinical Psychology, 68(1), 73–83. https://doi.org/10.1037/0022-006X.68.1.73

14.

Caro-Canizares

Serrano-Drozdowskyj

Pfang

Baca-Garcia

Carballo

J. J.

(2020). SDQ dysregulation profile and its relation to the severity of psychopathology and psychosocial functioning in a sample of children and adolescents with ADHD. Journal of Attention Disorders, 24(11), 1557–1564. https://doi.org/10.1177/1087054717691829

15.

Carver

C. S.

White

T. L.

(1994). Behavioral—Inhibition, Behavioral Activation, and affective responses to impending reward and punishment—The BIS/BAS Scales. Journal of Personality and Social Psychology, 67(2), 319–333. https://doi.org/10.1037/0022-3514.67.2.319

16.

Chen

(2015). What is the optimal number of response alternatives for rating scales? From an information processing perspective. Journal of Marketing Analytics, 3(2), 69–78. https://doi.org/10.1057/jma.2015.4

17.

Chyung

S. Y. Y.

Roberts

Swanson

Hankinson

(2017). Evidence-based survey design: The Use of a midpoint on the Likert scale. Performance Improvement, 56(10), 15–23. https://doi.org/10.1002/pfi.21727

18.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.

19.

Colder

C. R.

O’Connor

R. M.

(2004). Gray’s reinforcement sensitivity model and child psychopathology: Laboratory and questionnaire assessment of the BAS and BIS. Journal of Abnormal Child Psychology, 32(4), 435–451. https://doi.org/10.1023/b:jacp.0000030296.54122.b6

20.

Colder

C. R.

Trucco

E. M.

Lopez

H. I.

Hawk

L. W., Jr.

Read

J. P.

Lengua

L. J.

Weiczorek

W. F.

Eiden

R. D.

(2011). Revised reinforcement sensitivity theory and laboratory assessment of BIS and BAS in children. Journal of Research in Personality, 45(2), 198–207.

21.

Dadds

M. R.

Salmon

(2003). Punishment insensitivity and parenting: Temperament and learning as interacting risks for antisocial behavior. Clinical Child and Family Psychology Review, 6(2), 69–86. https://doi.org/10.1023/a:1023762009877

22.

De Los Reyes

Epkins

C. C

. (2023). Introduction to the special issue. A dozen years of demonstrating that informant discrepancies are more than measurement error: Toward guidelines for integrating data from multi-informant assessments of youth mental health. Journal of Clinical Child & Adolescent Psychology, 52(1), 1–18. https://doi.org/10.1080/15374416.2022.2158843

23.

Eyberg

S. M.

Nelson

M. M.

Boggs

S. R.

(2008). Evidence-based psychosocial treatments for children and adolescents with disruptive behavior. Journal of Clinical Child & Adolescent Psychology, 37(1), 215–237. https://doi.org/10.1080/15374410701820117

24.

Fabiano

G. A.

Pelham

W. E., Jr.

Waschbusch

D. A.

Gnagy

E. M.

Lahey

B. B.

Chronis

A. M.

Onyango

A. N.

Kipp

Lopez-Williams

Burrows-MacLean

(2006). A practical impairment measure: Psychometric properties of the Impairment Rating Scale in samples of children with attention-deficit/hyperactivity disorder and two school-based samples. Journal of Clinical Child & Adolescent Psychology, 35(3), 369–385. https://doi.org/10.1207/s15374424jccp3503_3

25.

Fosco

W. D.

Babinski

D. E.

Waschbusch

D. A.

(2023). The Disruptive Behavior Disorders Rating Scale: Updated factor structure, measurement invariance, and national caregiver norms. Journal of Pediatric Psychology, 48(5), 468–478. https://doi.org/10.1093/jpepsy/jsad006

26.

Fosco

W. D.

Hawk

L. W.

Rosch

K. S.

Bubnik

M. G.

(2015). Evaluating cognitive and motivational accounts of greater reinforcement effects among children with attention-deficit/hyperactivity disorder. Behavioral and Brain Functions, 11(1), 1–9. https://doi.org/10.1186/s12993-015-0065-9

27.

Frick

P. J.

(2004). Inventory of Callous-Unemotional Traits. The University of New Orleans. http://labs.uno.edu/developmental-psychopathology/ICU.html

28.

Frick

P. J.

Marsee

M. A.

(2018). Psychopathy and developmental pathways to antisocial behavior in youth. In Patrick

C. J.

(Ed.), Handbook of psychopathy (2nd ed., pp. 456-475). Guilford Press.

29.

Furukawa

Alsop

Shimabukuro

Tripp

(2019). Is increased sensitivity to punishment a common characteristic of attention deficit/hyperactivity disorder? An experimental study of response allocation in Japanese children. ADHD Attention Deficit and Hyperactivity Disorders, 11(4), 433–443. https://doi.org/10.1007/s12402-019-00307-6

30.

Gray

J. A.

(1987). Perspectives on anxiety and impulsivity: A commentary. Journal of Research in Personality, 21(4), 493–509. https://doi.org/10.1016/0092-6566(87)90036-5

31.

Gray

J. A.

McNaughton

(2003). The neuropsychology of anxiety: An enquiry into the functions of the septo-hippocampal system (2nd ed.). Oxford University Press.

32.

Graziano

P. A.

Garcia

(2016). Attention-deficit hyperactivity disorder and children’s emotion dysregulation: A meta-analysis. Clinical Psychology Review, 46, 106–123. https://doi.org/10.1016/j.cpr.2016.04.011

33.

Hawes

D. J.

Price

M. J.

Dadds

M. R.

(2014). Callous-unemotional traits and the treatment of conduct problems in childhood and adolescence: A comprehensive review. Clinical Child and Family Psychology Review, 17(3), 248–267. https://doi.org/10.1007/s10567-014-0167-1

34.

Holden

G. W.

Miller

P. C.

Harris

S. D.

(1999). The instrumental side of corporal punishment: Parents’ reported practices and outcome expectancies. Journal of Marriage and the Family, 61(4), 908–919. https://doi.org/10.2307/354012

35.

Horn

J. L.

(1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2), 179–185.

36.

Kim

H. Y.

(2015). Statistical notes for clinical researchers: Post-hoc multiple comparisons. Restorative Dentistry and Endodontics, 40(2), 172–176. https://doi.org/10.5395/rde.2015.40.2.172

37.

Kimonis

E. R.

Frick

P. J.

Skeem

J. L.

Marsee

M. A.

Cruise

Munoz

L. C.

Aucoin

K. J.

Morris

A. S.

(2008). Assessing callous-unemotional traits in adolescent offenders: Validation of the Inventory of Callous-Unemotional Traits. International Journal of Law and Psychiatry, 31(3), 241–252. https://doi.org/10.1016/j.ijlp.2008.04.002

38.

Luman

Tripp

Scheres

(2010). Identifying the neurobiology of altered reinforcement sensitivity in ADHD: A review and research agenda. Neuroscience & Biobehavioral Reviews, 34(5), 744–754. https://doi.org/10.1016/j.neubiorev.2009.11.021

39.

Luman

van Meel

C. S.

Oosterlaan

Geurts

H. M.

(2012). Reward and punishment sensitivity in children with ADHD: Validating the sensitivity to punishment and sensitivity to reward questionnaire for children (SPSRQ-C). Journal of Abnormal Child Psychology, 40(1), 145–157.

40.

Matthys

Vanderschuren

L. J.

Schutter

D. J.

(2013). The neurobiology of oppositional defiant disorder and conduct disorder: Altered functioning in three mental domains. Development and Psychopathology, 25(1), 193–207. https://doi.org/10.1017/S0954579412000272

41.

Matthys

Vanderschuren

L. J.

Schutter

D. J.

Lochman

J. E.

(2012). Impaired neurocognitive functions affect social learning processes in oppositional defiant disorder and conduct disorder: Implications for interventions. Clinical Child and Family Psychology Review, 15(3), 234–246. https://doi.org/10.1007/s10567-012-0118-7

42.

McMahon

R. J.

Frick

P. J.

(2005). Evidence-based assessment of conduct problems in children and adolescents. Journal of Clinical Child & Adolescent Psychology, 34(3), 477–505. https://doi.org/10.1207/s15374424jccp3403_6

43.

Nichols

S. R.

Briggs-Gowan

M. J.

Estabrook

Burns

J. L.

Kestler

Berman

Henry

D. B.

Wakschlag

L. S.

(2015). Punishment insensitivity in early childhood: A developmental, dimensional approach. Journal of Abnormal Child Psychology, 43(6), 1011–1023. https://doi.org/10.1007/s10802-014-9950-1

44.

Pelham

W. E., Jr.

(1998). Disruptive behavior disorders interview. Comprehensive Treatment of ADHD.

45.

Pelham

W. E., Jr.

Evans

S. W.

Gnagy

E. M.

Greenslade

K. E.

(1992). Teacher ratings of DSM-III-R symptoms for the disruptive behavior disorders—Prevalence, factor-analyses, and conditional probabilities in a special-education sample. School Psychology Review, 21(2), 285–299.

46.

Pelham

W. E., Jr.

Fabiano

G. A.

(2008). Evidence-based psychosocial treatment for attention-deficit/hyperactivity disorder: An update. Journal of Clinical Child & Adolescent Psychology, 37(1), 184–214.

47.

Pelham

W. E., Jr.

Fabiano

G. A.

Massetti

G. M.

(2005). Evidence-based assessment for attention deficit hyperactivity disorder in children and adolescents. Journal of Clinical Child & Adolescent Psychology, 34(3), 449–476. https://doi.org/10.1207/s15374424jccp3403_5

48.

Pelham

W. E., Jr.

Gnagy

E. M.

Greenslade

K. E.

Milich

(1992). Teacher ratings of DSM-III-R symptoms for the disruptive behavior disorders. Journal of the American Academy of Child and Adolescent Psychiatry, 31(2), 210–218. https://doi.org/10.1097/00004583-199203000-00006

49.

Pfiffner

L. J.

Oleary

S. G.

(1987). The efficacy of all-positive management as a function of the prior use of negative consequences. Journal of Applied Behavior Analysis, 20(3), 265–271. https://doi.org/10.1901/jaba.1987.20-265

50.

Pfiffner

L. J.

Oleary

S. G.

Rosen

L. A.

Sanderson

W. C.

(1985). A comparison of the effects of continuous and intermittent response cost and reprimands in the classroom. Journal of Clinical Child Psychology, 14(4), 348–352. https://doi.org/10.1207/s15374424jccp1404_14

51.

Pfiffner

L. J.

Rosen

L. A.

O’Leary

S. G.

(1985). The efficacy of an all-positive approach to classroom management. Journal of Applied Behavior Analysis, 18(3), 257–261. https://doi.org/10.1901/jaba.1985.18-257

52.

Radley

K. C.

Dart

E. H.

(2016). Antecedent strategies to promote children’s and adolescents’ compliance with adult requests: A review of the literature. Clinical Child and Family Psychology Review, 19(1), 39–54. https://doi.org/10.1007/s10567-015-0197-3

53.

Ray

J. V.

Frick

P. J.

(2020). Assessing callous-unemotional traits using the total score from the Inventory of Callous-Unemotional Traits: A meta-analysis. Journal of Clinical Child & Adolescent Psychology, 49(2), 190–199. https://doi.org/10.1080/15374416.2018.1504297

54.

Ray

J. V.

Frick

P. J.

Thornton

L. C.

Steinberg

Cauffman

(2016). Positive and negative item wording and its influence on the assessment of callous-unemotional traits. Psychological Assessment, 28(4), 394–404. https://doi.org/10.1037/pas0000183

55.

Sonuga-Barke

E. J.

Cortese

Fairchild

Stringaris

(2016). Annual research review: Transdiagnostic neuroscience of child and adolescent mental disorders–differentiating decision making in attention-deficit/hyperactivity disorder, conduct disorder, depression, and anxiety. Journal of Child Psychology and Psychiatry, 57(3), 321–349. https://doi.org/10.1111/jcpp.12496

56.

Srinivasan

Basu

A. K.

(1989). The metric quality of ordered categorical-data. Marketing Science, 8(3), 205–230. https://doi.org/10.1287/mksc.8.3.205

57.

Staff

A. I.

Oosterlaan

van der Oord

Königs

van den Hoofdakker

B. J.

Luman

(2021). Child neurocognitive functioning influences the effectiveness of specific techniques in behavioral teacher training for ADHD: Moderator analyses from a randomized controlled microtrial. JCPP Advances, 1(3), Article e12032. https://doi.org/10.1002/jcv2.12032

58.

Steiger

J. H.

(1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87(2), 245–251. https://doi.org/10.1037//0033-2909.87.2.245

59.

Tabachnik

B. G.

Fidell

L. S.

(2012). Using multivariate statistics (6th ed.). Harper-Collins.

60.

van Langen

M. J. M.

van Hulst

B. M.

Douma

Steffers

van de Wiel

N. M. H.

van den Ban

Durston

de Zeeuw

. (2021). Which child will benefit from a behavioral intervention for ADHD? A pilot study to predict intervention efficacy from individual reward sensitivity. Journal of Attention Disorders, 25(12), 1754–1764. https://doi.org/10.1177/1087054720928136

61.

Waschbusch

D. A.

Babinski

D. E.

Fosco

W. D.

Haas

S. M.

Waxmonsky

J. G.

Garon

Nichols

S. D.

King

Santor

D. A.

Andrade

B. F.

(2022). Inhibitory control, conduct problems, and callous-unemotional traits in children with ADHD and typically developing children. Developmental Neuropsychology, 47(1), 42–59. https://doi.org/10.1080/87565641.2022.2032713

62.

Waschbusch

D. A.

Porter

Carrey

Kazmi

S. O.

Roach

K. A.

D’Amico

D. A.

(2004). Investigation of the heterogeneity of disruptive behaviour in elementary-age children. Canadian Journal of Behavioural Science/Revue Canadienne des Sciences du Comportement, 36(2), 97–112. https://doi.org/10.1037/h0087221

63.

Waschbusch

D. A.

Willoughby

M. T.

(2008). Attention-deficit/hyperactivity disorder and callous-unemotional traits as moderators of conduct problems when examining impairment and aggression in elementary school children. Aggressive Behavior, 34(2), 139–153. https://doi.org/10.1002/ab.20224

64.

Waschbusch

D. A.

Willoughby

M. T.

Haas

S. M.

Ridenour

Crum

K. I.

Helseth

S. A.

Altszuler

A. R.

Ross

J. M.

Coles

E. K.

Pelham

W. E., Jr.

(2020). Effects of behavioral treatment modified to fit children with conduct problems and callous-unemotional traits. Journal of Clinical Child & Adolescent Psychology, 49(5), 639–650. https://doi.org/10.1080/15374416.2019.1614000

65.

Watkins

M. W.

(2018). Exploratory factor analysis: A guide to best practice. Journal of Black Psychology, 44(3), 219–246. https://doi.org/10.1177/0095798418771807

66.

Waxmonsky

J. G.

Fosco

W. D.

Waschbusch

D. A.

Babinski

D. E.

Baweja

Pegg

Cao

V. T.

Shroff

D. M.

Kujawa

(2022). The impact of irritability and callous unemotional traits on reward positivity in youth with ADHD and conduct problems. Research on Child and Adolescent Psychopathology, 50(8), 1027–1040. https://doi.org/10.1007/s10802-022-00901-9

67.

Wechsler

(2011). Wechsler Abbreviated Scale of Intelligence—Second Edition. Harcourt.

68.

Weijters

Cabooter

Schillewaert

(2010). The effect of rating scale format on response styles: The number of response categories and response category labels. International Journal of Research in Marketing, 27(3), 236–247. https://doi.org/10.1016/j.ijresmar.2010.02.004

69.

Willoughby

M. T.

Mills-Koonce

W. R.

Waschbusch

D. A.

Gottfredson

N. C.

, & Family Life Project Investigators, . (2015). An examination of the parent report version of the Inventory of Callous-Unemotional Traits in a community sample of first-grade children. Assessment, 22(1), 76–85. https://doi.org/10.1177/1073191114534886

70.

Wright

K. D.

Waschbusch

D. A.

Frankland

B. W.

(2007). Combining data from parent ratings and parent interview when assessing ADHD. Journal of Psychopathology and Behavioral Assessment, 29(3), 141–148. https://doi.org/10.1007/s10862-006-9039-4

71.

Zhang

Aloi

Bajaj

Bashford-Largo

Lukoff

Schwartz

Elowsky

Dobbertin

Blair

K. S.

Blair

R. J. R.

(2023). Dysfunction in differential reward-punishment responsiveness in conduct disorder relates to severity of callous-unemotional traits but not irritability. Psychological Medicine, 53(5), 1870–1880. https://doi.org/10.1017/S0033291721003500

Measuring Children’s Reward and Punishment Sensitivity: An Initial Psychometric Evaluation of the Contingency Response Rating Scale

Abstract

Keywords

Method

Participants

Measures

Contingency Response Rating Scale

Disruptive Behavior Disorders Rating Scale

Inventory of Callous-Unemotional Traits

Impairment Rating Scale

Procedure

Analytic Plan

Results

Item Analyses

Exploratory Factor Analysis

CRRS Scale Scores

Descriptive Statistics, Reliabilities, and Scale Variances

Inter-Scale Correlations

Correlations With Demographic Measures

Associations With ADHD, ODD, CD, CU, and Impairment

Comparisons of Psychopathology Groups

Discussion

Footnotes

Appendix

Authors’ Note

Author Contributions

Declaration of Conflicting Interests

Funding

Ethical Approval

ORCID iDs

Data Availability Statement

Notes

References