Sage Journals: Discover world-class research

Abstract

This study examines the psychometric properties of the ORAC-PCQ, a fourth-generation actuarial tool designed to assess recidivism risk and criminogenic needs among individuals serving short sentences in Quebec. The instrument consists of three sections: Sociocriminological Profile (eight items), Criminogenic Needs (16 items), and Clinical Information (26 unscored items). Based on a sample of 14,320 individuals, we report normative data, optimal cutoff scores, and predictive validity across four subgroups defined by gender and Indigenous status. The ORAC-PCQ demonstrated satisfactory overall predictive accuracy (AUC = .75), with slightly higher values for non-Indigenous individuals and women. Exploratory and confirmatory factor analyses revealed a stable four-factor structure—Criminal Background, Substance Abuse, Interpersonal Conflicts, and Employment Problems—across all groups, supporting structural validity and measurement invariance. These findings provide initial evidence of both construct and predictive validity for a new risk assessment tool specifically tailored to short-sentence populations.

Keywords

risk assessment gender predictive validity recidivism risk-need responsivity risk factors criminogenic needs construct validity ORAC-PCQ

Introduction

Risk assessment inventories are essential tools for predicting criminal recidivism and identifying criminogenic needs. They also play a crucial role in decision-making, particularly in determining whether a convicted person may benefit from programs and interventions aimed at reducing the likelihood of reoffending. As Olver et al. (2014) highlight, risk assessment and management tools have evolved significantly over the past three decades. The first generation of risk assessment relied on clinical judgment, making evaluations highly subjective and dependent on the professional’s training and experience. As a result, these assessments were considered less reliable than structured alternatives (Ægisdóttir et al., 2006; Grove et al., 2000). By the 1970s, a second generation of tools emerged, grounded in actuarial science (Gottfredson & Moriarty, 2006). These actuarial models systematically incorporated factors—such as criminal history and substance use—that were statistically linked to recidivism risk. However, as Bonta and Andrews (2023) note, this approach remained purely empirical, focusing on static factors that do not change over time and lacking a theoretical foundation. The third generation of risk assessment tools introduced dynamic risk factors—those that can change over time—allowing practitioners to better identify and respond to criminogenic needs. Unlike static risk factors such as prior convictions, dynamic risk factors like antisocial attitudes, substance use, and employment instability are amenable to change and align more closely with criminological and criminal justice psychology theories (Bonta & Andrews, 2023). The fourth generation of risk assessment tools goes a step further by integrating risk assessment with intervention and case management planning. Developed in the 1980s, the rehabilitation model—based on the principles of risk, need, and responsivity (RNR)—has been increasingly used to assess and rehabilitate convicted individuals in Canada and other regions of the world (Bonta & Andrews, 2023). Despite the success of widely used tools like the Level of Service/Case Management Inventory (LS/CMI; Andrews et al., 2004), few risk assessment instruments have been designed specifically for managing carceral populations under this model, particularly individuals serving short sentences.

This study examines the validity of a newly developed fourth-generation risk assessment tool designed for individuals serving short sentences. To establish a benchmark for its use, normative data will be presented, along with empirically derived cutoff points. Next, the tool’s predictive validity will be evaluated across the entire population of persons convicted in Quebec between January 2019 and December 2022, as well as specifically by gender and Indigenous status. Finally, the factorial structure of the scale will be analyzed and compared across these subgroups.

Convicted Person Assessment in Quebec and the Development of a New Actuarial Tool

The Correctional Services of Quebec oversees individuals serving sentences of less than 2 years. In 2002, following the highly publicized murder of a young boy by a convicted person under its supervision, the Act Respecting the Quebec Correctional Service (Government of Quebec, 2002) came into effect. This legislation mandated that the risk of recidivism be assessed for all convicted persons upon their admission into custody. During the initial years of implementation, no standardized assessment tool was officially required, and evaluations were conducted primarily through unstructured clinical judgment by correctional officers. In 2007, however, the LS/CMI was formally adopted as the mandatory risk assessment tool. While numerous studies have demonstrated that the LS/CMI has good predictive validity (Giguère & Lussier, 2016; Olver et al., 2014), its application posed significant challenges within the Quebec correctional context. Given that sentences under provincial jurisdiction are relatively short (less than 2 years, with a median supervision period of just 61 days; Giroux, 2011), the time required to complete the LS/CMI conflicted with the legal obligations of the Correctional Services. On average, officers had only 7 hours to complete the assessment, making its implementation impractical. To address this recurring challenge, authorities in Quebec’s Correctional Services initiated the development of a risk assessment and management tool better suited to their mandate. This led to the creation of the ORAC-PCQ (Outil d’évaluation du risque et de l’analyze clinique des personnes contrevenantes du Québec [Risk Assessment and Clinical Analysis Tool for Convicted Persons in Quebec]). To ensure the tool’s effectiveness, a psychometric assessment was undertaken to evaluate its validity.

Contemporary Issues in Actuarial Tool Validation

Psychometrics

Predicting adjudicated persons’ future behavior with certainty remains an unrealistic goal. However, despite these uncertainties, recidivism risk can be estimated with relative accuracy based on multiple risk factors known to influence criminal behavior. In terms of criterion validity, this predictive capacity represents a key psychometric strength. That said, an instrument’s quality should not be judged solely on its predictive validity (Brouillette-Alarie et al., 2023). If actuarial tools are to be considered legitimate psychometric instruments, they must meet additional psychometric criteria beyond predictive accuracy to ensure their overall reliability and validity.

In line with this perspective, Messick (1989, 1995) advocates for a unified approach to validity, which should be operationalized through a nomological network framework that maps the relationships between various constructs in the measurement process of an instrument. According to Messick’s model, construct validity comprises six distinct yet interrelated aspects: content, substantive, structural, generalizability, external, and consequential validity. To ensure a comprehensive evaluation, all these dimensions must be assessed to establish an instrument’s unified validity. Following this rationale, recidivism assessment and management tools, as psychometric instruments, should not rely solely on a demonstration of predictive validity. Instead, they must undergo a broader validation process that accounts for multiple facets of construct validity.

Most studies evaluating the validity of recidivism risk assessment instruments primarily focus on predictive validity, that is, whether these tools can accurately predict criminal recidivism. A meta-analysis of 28 studies found that most risk assessment instruments demonstrate moderate predictive accuracy (Yang et al., 2010). Furthermore, a separate meta-analysis of 31 studies (Viljoen et al., 2024) concluded that these tools outperform unstructured clinical judgment in predictive validity. While these findings are both encouraging and significant, they remain insufficient for fully assessing the validity of recidivism risk assessment tools. A more comprehensive validation process is needed to establish their overall reliability and effectiveness.

Another critical aspect of construct validity is an instrument’s structural properties, that is, what dimensions the tool assesses and how these dimensions are interrelated. Although relatively uncommon, some studies have examined the factor structure of recidivism risk assessment instruments. For instance, Hsu et al. (2011) analyzed data from the Level of Service Inventory-Revised (LSI-R; Andrews & Bonta, 1995) on convicted individuals in Australia, identifying a four-factor solution for women (Static Risk, Employment, Procriminal Attitudes, and Mental Health) a five-factor solution for men, which included the same four dimensions plus an additional factor labeled Protective Companions. Similarly, Zhang and Liu (2015) examined LSI-R data from Chinese probationers and found a four-factor structure, although the specific factors were not labeled in their analysis. Meanwhile, Gordon et al. (2015) found that the theoretical eight-factor model of the LS/CMI was not supported empirically; instead, their analysis yielded a two-factor structure comprising Criminal Conduct and Lifestyle Factors. Given the limited body of research on this topic, psychometric evidence regarding the internal or conceptual structure of actuarial instruments remains overshadowed by studies focusing on their ability to predict criminal recidivism. Expanding research in this area is essential to ensuring these tools are both theoretically sound and empirically valid.

Gender Considerations

The generalizability of a risk assessment instrument is a key component of its validity—specifically, whether the tool produces reliable results across different groups. In the context of the implementation and validation of the ORAC-PCQ, two subgroups warrant particular attention: women and Indigenous populations. Regarding gender, two main perspectives exist on the generalizability of risk tools. One group of researchers argues that these tools demonstrate comparable predictive validity for both men and women, effectively making them gender-neutral by default (e.g., Olver et al., 2014). In contrast, another group contends that risk assessment instruments have been developed primarily based on convicted male populations, potentially making them inappropriate for assessing convicted women (de Vogel & Nicholls, 2016; van der Knaap et al., 2012; Van Voorhis et al., 2010). Supporting the psychometric perspective, Andrews et al. (2011) found that the weighted mean predictive validity of risk and need domains was equal to or even higher for women than for men—a finding later reinforced by Olver et al.’s (2014) meta-analysis. Despite these findings, the question of gender neutrality in actuarial risk assessment tools remains a subject of ongoing debate. Ensuring that such instruments are applicable to convicted female is an important consideration in the development and validation of new risk management tools.

Applicability to Indigenous Persons

Indigenous populations are significantly overrepresented in both federal and provincial correctional facilities. Although they constitute only 4% of the adult population, Indigenous peoples account for 28% of admissions to provincial and territorial correctional services and 27% of admissions to federal correctional services (Statistics Canada, 2018). In their 2017–2018 annual report, the Correctional Investigator of Canada noted that, despite efforts spanning three decades, no government—regardless of political affiliation—has succeeded in reversing the overrepresentation of Indigenous peoples in Canadian jails and prisons (McIntosh & McKeen, 2018). This issue, first formally recognized by the federal government in 1984, remains unresolved, with incarceration rates continuing to rise year after year (Clark, 2019).

For Canada’s federal, provincial, and territorial correctional services, assessing the applicability of risk assessment tools for Indigenous peoples remains a significant challenge, particularly given their overrepresentation in the correctional system. This issue gained prominence following the Ewert v. Canada (2018) case. Jeffrey Ewert, an individual who identifies as Métis, challenged Correctional Service Canada on its use of risk assessment tools developed and validated primarily on non-Indigenous populations. He argued that no research supported their validity for Indigenous peoples, thereby violating the correctional services’ duty to ensure that the information used for assessments is current, accurate, and complete.

This case underscored the critical need to evaluate the generalizability of risk assessment tools in Indigenous populations. In a meta-analysis, Wilson and Gutierrez (2014) found that the total score of the Level of Supervision Inventory (LSI; Andrews, 1982) was significantly more predictive for non-Indigenous individuals than for Indigenous individuals. Similarly, Gutierrez et al. (2013) concluded that while the main risk/need factors were predictive of criminal recidivism among Indigenous peoples, certain factors—such as criminal history, substance use, and antisocial patterns—were significantly more predictive for non-Indigenous persons. More recently, Olver et al. (2024) reported that commonly used risk assessment tools demonstrated broadly moderate predictive validity for various recidivism outcomes in Indigenous samples; however, predictive accuracy was frequently higher for non-Indigenous samples, depending on the tool and outcome examined. These findings highlight the need for further research to ensure that risk assessment tools are both valid and equitable across diverse populations.

Study Objectives

Given the need for an efficient risk assessment tool for individuals serving short sentences and recognizing that validity must be evaluated across multiple dimensions, this article aims to present the first psychometric evidence for a newly implemented tool in Quebec, Canada: the ORAC-PCQ. The study has three main objectives. First, it aims to establish normative data for the tool as a benchmark for its use, while also proposing cutoff points based on empirical findings. Second, it seeks to evaluate predictive validity by examining recidivism outcomes and comparing results across subgroups to assess generalizability. Third, it assesses construct validity by analyzing the tool’s latent structure and comparing it across subgroups to further evaluate its generalizability.

Method

Sample

The current analysis is based on the entire population of individuals who were registered and assessed using the ORAC-PCQ between January 2019 and December 2022 in Quebec (N = 14,320). The ORAC-PCQ is administered exclusively to individuals sentenced to more than 6 months (but less than 2 years, as they fall under the jurisdiction of Quebec’s Correctional Services). No distinction is made between individuals serving custodial sentences and those sentenced to community supervision, as the ORAC-PCQ is completed in the same manner for both groups. Furthermore, an individual’s total score is not influenced by the judge’s decision to impose incarceration or community detention. To evaluate the generalizability of the findings, analyses were conducted and compared across four groups: non-Indigenous men (n = 11,421), non-Indigenous women (n = 2,093), Indigenous men (n = 675), and Indigenous women (n = 131). Table 1 presents the descriptive statistics for the full sample and each subgroup.

Table 1:

Descriptive Statistics

	Whole sample(N = 14,320)	Non-Indigenous men(n = 11,421)	Non-Indigenous women(n = 2,093)	Indigenous men(n = 675)	Indigenous women(n = 131)	Group differences
ORAC-PCQ: Items and scores	%/M (SD)	%/M (SD)	%/M (SD)	%/M (SD)	%/M (SD)	Cramer’s V	Eta-squared
ORAC-PCQ: Sociocriminological portrait
(2.1) Disciplinary breaches in detention facilities	17.2%	18.5%	8.7%	23.3%	13.0%	.098***	—
(2.2) Failure to meet legal obligations	56.1%	57.9%	39.4%	77.7%	56.5%	.163***	—
(4.1) Closed custody	10.1%	11.2%	3.6%	13.4%	4.7%	.094***	—
(4.2) Provincial incarceration	68.1%	71.1%	47.1%	83.2%	69.5%	.195***	—
(4.3) Federal incarceration	12.3%	13.9%	2.3%	16.6%	4.6%	.130***	—
(5.1) Consumed alcohol before age 13	15.8%	15.4%	14.5%	26.8%	22.0%	.069***	—
(5.2) Used drugs before the age of 13	20.2%	20.2%	16.3%	32.9%	25.4%	.078***	—
(5.3) Problematic family environment during childhood	46.9%	43.7%	55.7%	69.4%	63.3%	.135***	—
ORAC-PCQ: Criminogenic needs
(1.1) Absence of a person with a positive influence	31.6%	31.5%	32.4%	31.7%	24.6%	.016	—
(1.2) Negative attitude toward figures of authority	17.8%	18.6%	13.9%	17.3%	20.2%	.043***	—
(1.3) Dysfunctional family ties (last 12 months)	35.9%	33.5%	46.4%	39.5%	53.9%	.103***	—
(1.4) Little consideration toward others	18.2%	19.7%	9.9%	19.4%	12.5%	.091***	—
(1.5) Recurrent difficulties solving interpersonal conflicts	52.1%	50.4%	59.7%	54.3%	60.6%	.068***	—
(2.1) Has not held a job for at least 1 year in the last 5 years	38.7%	37.6%	39.3%	51.1%	61.9%	.074***	—
(2.2) Unoccupied when the offense was committed	52.8%	52.4%	51.0%	61.7%	71.1%	.054***	—
(2.3) Lack of interest in work (throughout life)	21.1%	20.5%	21.3%	28.5%	35.9%	.054***	—
(3.1) Problem drug use (past 12 months)	49.0%	49.4%	43.3%	59.6%	57.0%	.065***	—
(3.3) Problem drinking (past 12 months)	34.5%	33.1%	29.1%	68.9%	65.4%	.175***	—
(3.5) Participation in or referral to at least one treatment program for alcohol and/or drug dependence	58.8%	59.2%	53.1%	68.0%	68.0%	.063***	—
(4.2) Use of violence	35.3%	36.5%	22.6%	53.1%	44.6%	.133***	—
(4.3) Impulsivity	60.8%	61.5%	57.6%	60.7%	55.5%	.030**	—
(5.1) Adheres to criminal values	41.8%	44.7%	28.2%	39.0%	24.2%	.123***	—
(5.2) Mostly associates with people involved in criminal activity (past 12 months)	29.1%	29.1%	26.2%	38.7%	34.4%	.052***	—
(5.3) Dysfunctional family environment during childhood (last 12 months)	17.5%	15.3%	22.2%	36.1%	39.7%	.136***	—
ORAC-PCQ: Total scores
Sociocriminological Portrait (/8)	2.44 (1.78)	2.49 (1.79)	1.86 (1.55)	3.36 (1.78)	2.56 (1.54)	—	.029***
Criminogenic Needs (/16)	5.82 (3.46)	5.80 (3.47)	5.48 (3.33)	7.01 (3.38)	7.10 (3.47)	—	.008***
ORAC-PCQ total (/24)	8.20 (4.74)	8.23 (4.76)	7.30 (4.43)	10.12 (4.71)	9.66 (4.54)	—	.014***
Other measures
Age (in years)	37.13 (12.28)	37.17 (12.37)	37.70 (12.39)	35.20 (10.44)	34.35 (9.83)	—	.002***
General recidivism (1-year follow-up)	23.7%	24.0%	17.7%	35.9%	30.5%	.084***	—

Note. Statistical significance indicators refer to group differences based on chi-square or ANOVA tests. Effect size values shown in bold represent small but meaningful effects, while non-bold values are considered negligible.

p < .05. **p < .01. ***p < .001.

Measures

The ORAC-PCQ

In 2012, correctional authorities sought to develop a new assessment tool for individuals sentenced to more than 6 months, whether in the community or in custody. The tool needed to be both aligned with Quebec’s correctional context and supported by a psychometric structure that met the standards of an actuarial measurement instrument. To initiate this process, a committee of experts in the assessment and intervention of justice-involved individuals was assembled. Over several months, the committee iteratively developed the tool’s first version, continuously adding, modifying, and removing items until reaching a satisfactory form. This process ensured that the items met the predefined requirements, establishing an initial foundation of content validity (Colquitt et al., 2019). Between 2015 and 2016, data collection was conducted within correctional services. To maintain standardization and objectivity participating correctional officers attended preparatory training sessions, with ongoing technical support provided throughout the process. Officers were recruited from all correctional regions of Quebec to ensure representative coverage of convicted individuals across the province. Following this phase, data analyses were performed, leading to the development of the first official version of the ORAC-PCQ, which was formally implemented in early 2019. Simultaneously, a specialized training program was designed to equip correctional officers with the necessary knowledge and skills to effectively administer the tool.

The latest version of the ORAC-PCQ consists of three sections. Section 1, Sociocriminological Portrait (eight items), includes static factors related to an individual’s background and criminal history, such as disciplinary infractions in correctional facilities, legal violations, number of current and prior offenses, closed custody, provincial and federal incarceration, alcohol use before age 13, drug use before age 13, and a problematic family environment during childhood. Section 2, Criminogenic Needs (16 items), assesses dynamic risk factors linked to recidivism, including substance abuse, interpersonal conflicts, and employment-related problems. Section 3, Clinical Information (26 items), provides a comprehensive responsivity profile by evaluating five key dimensions: convicted person characteristics (e.g., membership in a criminal organization, individual serving a long sentence), victimization history (e.g., history of physical, sexual, or psychological assault or neglect), readiness for intervention (e.g., failure to recognize the offense, minimizing responsibility), and health factors (e.g., psychopathy, mental disorders, personality disorders, depression, suicide attempts). Like for the LS/CMI, the Clinical Information section of the ORAC-PCQ is not included in the total risk score, as it comprises responsivity factors: personal characteristics that affect how effectively an individual can engage with correctional interventions, but that are not necessarily established as risk-relevant (Bonta & Andrews, 2023). Instead, this section is intended to help correctional staff design more comprehensive and individualized interventions that align with the person’s learning style, motivation, mental health status, and so on. Responsivity, in this context, reflects the principle of tailoring interventions beyond risk level to enhance engagement in correctional programming and promote successful reintegration. An individual’s risk score is thus determined by the total score from Sections 1 and 2, ranging from 0 to 24.

Since 2017, the ORAC-PCQ has been used to identify convicted persons’ criminogenic needs, assist correctional officers in developing intervention plans, and predict the risk of criminal recidivism after release. Over the years, the tool has undergone extensive analyses and revisions. It has been evaluated using both classical test theory and item response theory, with its items carefully examined to determine their difficulty level, predictive power, and potential bias. Based on these analyses, underperforming items were removed or moved to Section 3, resulting in the finalized version of the tool in 2022. Because no new items were added during this refinement process, earlier data remain valid for analysis, as the final version represents a streamlined subset of previous versions. Now that the ORAC-PCQ has reached maturity, the Correctional Services of Quebec has officially unveiled the tool and introduced it to the wider community.

The actuarial tool is integrated into a computerized system for assessment and follow-up (Système d’outils informatisés d’évaluation et de suivis [Computerized Tools System for Evaluation and Follow-up]), which automatically calculates scores for the three sections of the ORAC-PCQ. The database used for this study was generated through this system, with all data extracted in a fully anonymous manner.

Criminal Recidivism

Official criminal recidivism data were used to assess the predictive validity of the ORAC-PCQ. In this study, criminal recidivism is defined as the occurrence of at least one new conviction for a criminal offense during the follow-up period, with breaches of conditions excluded from this definition. The follow-up period for all participants was 1 year, beginning at the point when they were at risk of reoffending, meaning upon their release in the community.

Analytical Strategy

The objectives of this study were to establish the foundations of the ORAC-PCQ in terms of normative data, predictive validity, and factor structure.

Normative Data

The development of normative data for the ORAC-PCQ followed a two-step procedure. First, for the entire sample and each subgroup, recidivism rate tables were generated, stratified by every possible score on the instrument, using a fixed 1-year follow-up period. Second, cutoff scores were determined by testing multiple threshold combinations, using area under the curve (AUC) analyses to identify both the optimal number of risk categories and the specific cutoff points. Various risk category configurations (ranging from 2 to 5) were tested, and the results ultimately supported a three-category system: low, medium, and high risk. The methodology for defining these three categories—specifically, the process of determining two cutoff scores—is detailed below, along with the pseudo-algorithm used to establish these thresholds.

Initialization

The maximum possible score is denoted by K. A starting score k is initialized, typically k = 2.

Optimization

For a given k, the next optimal score k* is determined such that k* > k + 1. The variable k* is determined such that the AUC is maximized over the training sample. The area under the curve is computed for the pair (k, k*) in the validation dataset. We call this quantity AUC (k, k*).

Iteration

The score k is incremented k → k + 1. Steps 2 and 3 are repeated until k = K-1.

Optimal cutoffs

The optimal cutoffs c₁ and c₂ are the pairs of scores (k, k*) that minimize the area under the curve: c₁, c₂ = arg min AUC (k, k*).

The algorithm described above can be generalized to accommodate three or more risk categories, optimizing additional cutoff points by identifying thresholds to the left and right of k. This approach can also be applied independently to different score components, such as the Sociocriminological Profile or Criminogenic Needs, as well as to the total score of the ORAC-PCQ.

Predictive Validity

The predictive validity of the ORAC-PCQ was assessed using the AUC from receiver operating characteristic (ROC) curve analysis (Ruscio, 2008; Swets et al., 2000). To evaluate the effect size of predictors, the study followed the AUC guidelines established by Rice and Harris (2005), which classify AUC values as follows: .56 (or .44) corresponds to a small effect, .64 (or .36) to a moderate effect, and .71 (or .29) to a strong effect. AUC values are considered statistically significant when their 95% confidence interval does not include .50. Predictive validity was examined for the entire sample, as well as across the four subgroups.

Factorial Structure

Exploratory factor analysis

Exploratory factor analysis (EFA) was conducted to identify the latent constructs of the ORAC-PCQ, following the guidelines of Brouillette-Alarie et al. (2016). The analysis was performed on the entire sample, using tetrachoric correlations, which are less sensitive to the range restriction artifacts commonly observed when Pearson correlations are applied to dichotomous or ordinal data (Kubinger, 2003). Factors were extracted using the weighted least square means- and variance-adjusted (WLSMV) method, as recommended for EFA of dichotomous and ordinal data (Muthén & Muthén, 2017). Given that the extracted factors were expected to be correlated, an oblique (geomin) rotation was applied.

The number of factors retained was determined based on four complementary criteria: (a) the scree plot (Cattell, 1966), (b) parallel analysis (Horn, 1965), (c) Velicer’s minimum average partial (MAP) test (Velicer, 1976), and (d) fit indices (outlined below). Each factor retention method has its strengths and limitations, and using multiple approaches is generally preferred (Henson & Roberts, 2006). The final selection aimed to achieve consensus among these criteria while preserving face validity and theoretical coherence among the extracted factors.

To evaluate the overall model fit, four fit indices were used. The root mean square error of approximation (RMSEA) assesses the lack of fit relative to a perfect factor structure and should ideally not exceed .06 (Hu & Bentler, 1999). The comparative fit index (CFI) evaluates model fit relative to a baseline model with no item relationships, where a CFI of .90+ indicates adequate fit and .95+ indicates good fit (Hu & Bentler, 1999). The Tucker–Lewis Index (TLI) considers both model fit and parsimony, following the same interpretation as the CFI (Hu & Bentler, 1999; Tucker & Lewis, 1973). Finally, the standardized root mean square residual (SRMR) measures the average standardized residuals between observed and hypothesized covariance matrices, with a value of .08 or lower indicating good fit (Hu & Bentler, 1999).

Factor inclusion was determined by a loading threshold of .40 (Stevens, 1992). The magnitude of factor loadings was prioritized over statistical significance, as large samples can yield statistically significant yet practically meaningless cross-loadings. Items with factor loadings between .30 and .39 were retained if (a) their inclusion was theoretically justified and (b) they did not exhibit stronger loadings on another factor. All EFA analyses were conducted using Mplus 8.11 (Muthén & Muthén, 2017).

Confirmatory factor analysis and group invariance testing

Confirmatory factor analysis (CFA) was conducted to evaluate the fit of the factor solution across the four categories of convicted individuals in Quebec: non-Indigenous men, non-Indigenous women, Indigenous men, and Indigenous women. Analyses were performed using Mplus 8.11 (Muthén & Muthén, 2017) with the software’s default parameters for binary and polytomous variables, employing the WLSMV estimator. The factor structure was modeled based on the exploratory factor analysis (EFA) results, incorporating post hoc decisions to retain certain items with factor loadings between .30 and .40. Items that did not load on any factors were excluded from the CFA. The same fit index thresholds used in the EFA were applied to assess model fit in the CFA.

Measurement invariance across groups was assessed using multigroup confirmatory factor analysis. First, configural invariance was tested to determine whether the general factor structure—including the number of factors and the pattern of factor loadings—was consistent across groups. Next, scalar invariance was evaluated by constraining both factor loadings and item thresholds to equality, allowing for an assessment of group differences on latent factors. Metric invariance was not tested, as it is not identified under WLSMV estimation with binary variables (Muthén & Muthén, 2017).

To compare models, the study followed the guidelines of Cheung and Rensvold (2002) and Chen (2007), which propose alternatives to chi-square testing—a method that often produces statistically significant differences in large samples, leading to potential misinterpretations. Instead, measurement invariance was assessed by first verifying the acceptability of the fit indices for both the configural and scalar models. Then, invariance was confirmed if the transition from the configural to the scalar model resulted in only minimal changes in fit indices, specifically: Δ RMSEA ≤ 0.015, Δ CFI ≤ 0.01, and Δ SRMR ≤ 0.03.

Results

As illustrated on Table 1, on average, ORAC-PCQ scores were significantly higher for men than for women, and for Indigenous individuals compared to non-Indigenous individuals. Consistent with these risk scores, observed recidivism rates were also higher among men than women, and among Indigenous individuals than non-Indigenous individuals. The effect sizes were, however, small in magnitude.

Risk Distribution Table

Table 2 presents the risk distribution of the ORAC-PCQ, showing the proportion of recidivists for each possible score on the tool, both for the entire sample and the four subgroups. Across all groups, a clear pattern emerged: higher scores were associated with an increased risk of recidivism. However, this trend was less stable among Indigenous populations and at higher ORAC-PCQ scores, as the small number of individuals per case resulted in less reliable estimates.

Table 2:

Risk Distribution Table of ORAC-PCQ Scores

ORAC-PCQ scores	Whole sample(N = 14,320)	Non-Indigenous men(n = 11,421)	Non-Indigenous women(n = 2,093)	Indigenous men(n = 675)	Indigenous women(n = 131)
	%, n recidivists/n total
0	2.1%7/338	2.2%6/269	1.6%1/64	.0%0/4	.0%0/1
1	3.5%21/603	4.3%20/470	.8%1/123	.0%0/7	.0%0/3
2	7.1%57/806	6.7%43/644	6.7%9/135	22.7%5/22	.0%0/5
3	6.1%53/874	6.4%43/676	3.6%6/167	11.5%3/26	20.0%1/5
4	8.4%84/999	8.7%70/806	6.3%10/158	10.3%3/29	16.7%1/6
5	11.1%113/1019	11.8%97/823	6.9%11/159	9.7%3/31	33.3%2/6
6	16.7%190/1136	17.2%157/911	10.8%19/176	30.2%13/43	16.7%1/6
7	21.7%236/1089	22.8%195/854	11.7%19/162	31.7%20/63	20.0%2/10
8	19.8%202/1021	19.3%158/817	17.3%26/150	34.1%15/44	30.0%3/10
9	24.5%252/1028	25.1%202/804	18.8%31/165	33.3%14/42	29.4%5/17
10	31.1%290/932	31.3%235/752	24.4%33/135	51.2%21/41	25.0%1/4
11	34.5%299/866	35.0%240/686	30.2%32/106	34.9%22/63	45.5%5/11
12	37.5%285/696	38.6%241/624	31.8%27/85	36.6%15/41	22.2%2/9
13	40.7%283/696	40.7%229/562	44.0%37/84	34.1%14/41	33.3%3/9
14	43.1%248/575	41.8%184/440	45.3%39/86	52.4%22/42	42.9%3/7
15	46.3%218/471	46.4%173/373	48.0%24/50	46.2%18/39	33.3%3/9
16	47.8%166/347	47.5%131/276	45.5%15/33	54.3%19/35	33.3%1/3
17	54.2%162/299	53.3%129/242	50.0%15/30	71.4%15/21	50.0%3/6
18	46.3%87/188	44.9%70/156	53.8%7/13	47.1%8/17	100%2/2
19	48.5%65/134	47.8%55/115	66.7%6/9	33.3%3/9	100%1/1
20	51.8%44/85	49.3%35/71	66.7%2/3	60.0%6/10	100%1/1
21	70.6%24/34	68.8%22/32	—0/0	100%2/2	—0/0
22	46.2%6/13	50.0%5/10	—0/0	33.3%1/3	—0/0
23	50.0%3/6	50.0%3/6	—0/0	—0/0	—0/0
24	100%2/2	100%2/2	—0/0	—0/0	—0/0

Cutoff Scores for Risk Categories

Table 3 presents the suggested cutoff scores for the ORAC-PCQ risk categories. In the Sociocriminological Portrait component, scores were classified into low (0–1), medium (2–3), and high (4–8). In the Criminogenic Needs component, the categories were defined as low (0–4), medium (5–8), and high (9–16). This classification into low-, medium-, and high-risk groups for both components was found to strongly correlate with recidivism likelihood in the validation sample. Table 3 also presents the recidivism rates, along with their 95% confidence intervals, for each combination of these two risk components.

Table 3:

Cutoff Scores for ORAC-PCQ Risk Categories

Criminogenic needs	Sociocriminological portrait	Recidivism rates (%)	95% CI
Low (0–4)	Low (0–1)	4.80	[3.63, 5.96]
Low (0–4)	Medium (2–3)	15.81	[13.05, 18.56]
Low (0–4)	High (4–8)	27.86	[20.34, 35.38]
Medium (5–8)	Low (0–1)	7.06	[4.99, 9.12]
Medium (5–8)	Medium (2–3)	33.86	[30.74, 36.98]
Medium (5–8)	High (4–8)	35.85	[31.94, 39.75]
High (9–16)	Low (0–1)	11.82	[5.69, 17.95]
High (9–16)	Medium (2–3)	41.65	[36.72, 46.57]
High (9–16)	High (4–8)	44.28	[40.72, 47.84]

Predictive Validity

Table 4 presents the results of the predictive validity of the ORAC-PCQ, based on ROC curve analysis predicting recidivism as a function of scale scores for each subgroup. The predictive values were high for non-Indigenous individuals, while they were moderate for Indigenous populations.

Table 4:

Predictive Validity of the ORAC-PCQ

	Whole sample	Non-Indigenous men	Non-Indigenous women	Indigenous men	Indigenous women
	(3,397 rec./14,320)	(2,745 rec./11,421)	(370 rec./2,093)	(242 rec./675)	(40 rec./131)
ORAC-PCQ: Items and scores	AUC [95% CI]
Sociocriminological portrait (/8)	.722[.713, .731]	.712[.702, .722]	.770[.745, .795]	.655[.612, .698]	.706[.606, .807]
Criminogenic needs (/16)	.724[.714, .734]	.721[.710, .732]	.756[.730, .782]	.656[.612, .700]	.659[.556, .762]
ORAC-PCQ total (/24)	.745[.736, .755]	.740[.730, .750]	.785[.760, .810]	.675[.633, .718]	.695[.556, .793]

Note. The area under the ROC curve is considered significant when its confidence interval does not include .500. “Rec.” = recidivists.

Predictive validity was higher for women than for men, regardless of Indigenous status. In addition, combining the Sociocriminological Portrait and Criminogenic Needs components (i.e., the total ORAC-PCQ score) improved prediction accuracy compared to using either component alone—except for Indigenous women, where the Sociocriminological Portrait score slightly outperformed the total ORAC-PCQ score.

Exploratory Factor Analysis

Figure 1 presents the scree plot of the eigenvalues from the exploratory factor analysis (EFA) as a function of the number of extracted factors. No significant collinearity issues were observed (r > .90). The scree plot suggested a two- or four-factor solution; however, the two-factor model was theoretically inadequate and exhibited poor fit indices (see Table 5). Velicer’s revised MAP test (2000) suggested a three-factor solution, while both parallel analysis and Kaiser’s criterion (eigenvalues > 1) indicated a six-factor solution. Although consistency across factor retention criteria is preferable, the results pointed in different directions. Ultimately, a four-factor solution was selected, as the three-factor model did not exhibit sufficient fit indices, while the six-factor model contained “empty factors” (i.e., factors with only one or two items). The four-factor solution minimized cross-loadings and negative loadings, making it the cleanest and most theoretically sound factor structure.

Figure 1:

Scree Plot of the Eigenvalues From the Exploratory Factor Analysis

Table 5:

Eigenvalues, Parallel Analysis Results, and Fit Indices for Each Number of Extracted Factors

Number of factors extracted	Eigenvalues	Parallel analysis		Fit indices of the factor solution
Number of factors extracted	Eigenvalues	Raw	95th Percentile	RMSEA	CFI	TLI	SRMR
1	8.093	4.956	1.090	.090 [.089, .091]	.766	.743	.115
2	2.252	1.762	1.077	.077 [.077, .078]	.843	.811	.096
3	1.848	1.503	1.067	.065 [.064, .066]	.899	.865	.075
4	1.747	1.485	1.058	.049 [.048, .050]	.949	.924	.050
5	1.302	1.251	1.050	.043 [.042, .044]	.966	.943	.040
6	1.036	1.083	1.044	.038 [.036, .039]	.976	.955	.032
7	.880	.950	1.037	.029 [.028, .030]	.988	.974	.024
8	.848	.929	1.031	.024 [.023, .026]	.992	.982	.019

Note. In parallel analysis, the number of factors to retain corresponds to the last point at which the observed (raw) eigenvalue exceeds the 95th percentile of the eigenvalues generated from random data. The MAP test suggested a three-factor solution.

Table 6 presents the factor loadings for the four-factor solution. The first factor, Criminal Background, reflects an individual’s history of disciplinary breaches in detention facilities and other legal obligations, previous incarceration sentences, and adherence to criminal values. The second factor, Substance Abuse, captures alcohol and drug consumption, whether during youth or in the past 12 months, as well as participation in a dependency treatment program. The third factor, Interpersonal Conflicts, includes experiences of a problematic family environment during childhood, strained family relationships in the past 12 months, and negative attitudes toward authority figures. It also encompasses low consideration for others, difficulty resolving interpersonal problems, impulsivity, and the use of violence. The fourth factor, Employment Problems, pertains to an individual’s lack of employment history, being unoccupied at the time of the offense, low interest in work, and frequent association with individuals involved in criminal activity. Notably, two items did not load onto any factor: lack of ties to a person exerting a positive influence and presence of criminal activity in at least one significant family member within the past 12 months.

Table 6:

Factor Loadings of the Exploratory Factor Analysis of the ORAC-PCQ (N = 14,320)

	Factor 1:Criminal background	Factor 2:Substance abuse	Factor 3:Interpersonal conflicts	Factor 4:Employment problems
Items	Factor loadings
Sociocriminological portrait
(2.1) Disciplinary breaches in detention facilities	.823	−.005	.117	.015
(2.2) Failure to meet legal obligations	.744	.161	.076	.031
(4.1) Closed custody^a	. 385	.160	.164	.054
(4.2) Provincial incarceration	.851	.068	−.112	.015
(4.3) Federal incarceration	.746	−.039	−.063	−.044
(5.1) Consumed alcohol before age 13	.037	.875	−.008	−.056
(5.2) Used drugs before the age of 13	.035	.811	.022	.026
(5.3) Problematic family environment during childhood	−.119	.314	.412	.061
Criminogenic needs
(1.1) Absence of a person with a positive influence	.044	.003	.261	.274
(1.2) Negative attitude toward figures of authority	.179	−.144	.650	.009
(1.3) Dysfunctional family ties (last 12 months)	−.359	.187	.615	.087
(1.4) Little consideration toward others	.159	−.257	.717	.056
(1.5) Recurrent difficulties solving interpersonal conflicts	−.162	.026	.790	−.051
(2.1) Has not held a job for at least 1 year in the last 5 years	−.022	−.001	−.034	.940
(2.2) Unoccupied when the offense was committed	−.006	.035	−.150	.901
(2.3) Lack of interest in work (throughout life)	.077	−.090	.137	.675
(3.1) Problem drug use (past 12 months)^a	.171	. 364	.095	.291
(3.3) Problem drinking (past 12 months)^a	.065	. 306	.140	−.026
(3.5) Participation in or referral to at least one treatment program for alcohol and/or drug dependence	.350	.476	−.053	.024
(4.2) Use of violence	.049	.010	.753	−.152
(4.3) Impulsivity	.036	.072	.719	−.134
(5.1) Adheres to criminal values	.613	.000	.088	.257
(5.2) Mostly associates with people involved in criminal activity (past 12 months)	.338	.154	.062	.403
(5.3) Dysfunctional family environment during childhood last 12 months)	−.004	.195	.139	.183

Note. Estimator = WLSMV, rotation = geomin, tetrachoric correlations used. RMSEA = .049 [90% CI = .048, .050], CFI = .949, TLI = .924, SRMR = .050. Factor loadings of .40+ are in bold.

Because this item had a loading close to .40 in a theoretically appropriate factor, it was included in that factor (bold italic).

The Criminal Background factor showed a correlation of .292 with Substance Abuse, .475 with Interpersonal Conflicts, and .418 with Employment Problems. In turn, Substance Abuse correlated .364 with Interpersonal Conflicts and .306 with Employment Problems. Finally, Interpersonal Conflicts had a correlation of .420 with Employment Problems.

Although the extracted factors were not optimized to maximize predictive validity, we report their individual predictive performance for the full sample as supplementary information. The predictive validity of the four factors was as follows: Criminal Background had an AUC of .734 [95% CI = .725–.742], Substance Abuse had an AUC of .661 [95% CI = .650–.671], Interpersonal Conflicts had an AUC of .650 [95% CI = .639–.660], and Employment Problems had an AUC of .659 [95% CI = .648–.670].

Confirmatory Factor Analysis and Group Invariance Testing

Table 7 presents the model fit indices for the Confirmatory Factor Analysis (CFA) of the ORAC-PCQ across each subgroup. This analysis included the 22 ORAC-PCQ items that loaded onto a factor in the Exploratory Factor Analysis (EFA), each assigned to its corresponding factor as previously described. No modification indices were applied, as the objective was to compare the fit of the factor structure across subgroups in its original form. Results indicate that the ORAC-PCQ demonstrated relatively consistent model fit across all subgroups, except for Indigenous women, for whom the fit indices were notably lower compared to the other groups.

Table 7:

Confirmatory Factor Analysis of the ORAC-PCQ Across Subgroups

Subgroups	RMSEA	CFI	TLI
Non-Indigenous men (n = 11,421)	.063 [.062, .064]	.903	.889
Non-Indigenous women (n = 2,093)	.062 [.059, .065]	.895	.880
Indigenous men (n = 675)	.060 [.055, .065]	.887	.871
Indigenous women^a (n = 131)	.076 [.063, .089]	.863	.844

The residual covariance matrix for Indigenous women is not positive definite, which could invalidate results for that group.

Table 8 presents the results of the measurement invariance testing. Fit indices for both the configural and scalar models were acceptable, and the differences in fit indices between the two models were negligible. These findings indicate that the factor structure of the ORAC-PCQ is applicable across all subgroups, demonstrating both configural and scalar invariance.

Table 8:

Measurement Invariance Testing Across Groups

Tested models	RMSEA	CFI	TLI	SRMR	Chi-Square
Configural model^a	.055	.914	.903	.084	9,709.050 (p = .000)
Scalar model^a	.055	.910	.903	.086	10,227.842 (p = .000)
Difference in fit indices^b	.000	.004	.000	.002	804.620 (p = .000)

The residual covariance matrix for indigenous women is not positive definite, which could invalidate results for that group. ^b Changes of less than .015 in RMSEA indicate invariance; decreases of less than .01 in CFI indicate invariance; and changes of less than .01 in RMSEA indicate scalar invariance.

Discussion

This study examined the psychometric properties and validity of the ORAC-PCQ, a fourth-generation risk assessment tool implemented in Quebec, Canada, designed to evaluate recidivism risk and criminogenic needs among adjudicated individuals serving short sentences. In line with Messick’s (1989) framework, which emphasizes that validity should encompass multiple constructs within the measurement process, this study assessed various aspects of construct validity beyond predictive validity.

Normative Data for the ORAC-PCQ

Risk distribution data for the ORAC-PCQ were presented as a benchmark for its use, along with empirically derived cutoff points. By stratifying the entire sample and each subgroup’s recidivism outcomes according to every possible risk score, we observed a clear trend whereby higher ORAC-PCQ scores consistently corresponded to elevated rates of new convictions. It is also worth noting that some risk-score categories, particularly in the higher ranges or among smaller subgroups, contained very few participants, which can lead to instability in the observed recidivism rates. When only a handful of individuals fall into a given cell, a single recidivist (or non-recidivist) can appreciably alter the calculated risk percentage. Such fluctuations are especially salient for groups like Indigenous women, whose limited sample size yielded noticeable variability in the highest risk categories. As a result, although the overall pattern remains consistent across most scores, readers should interpret the exact recidivism percentages in these sparsely populated cells with caution. Considering these constraints, the most reliable estimates are those derived from the total (unified) sample, which minimizes sampling variability and provides a more stable basis for interpreting the relationship between ORAC-PCQ scores and recidivism. Moreover, the process by which risk classification cutoffs were determined—employing AUC analyses to compare different threshold configurations—ensured that each boundary within the low-, medium-, and high-risk bands were empirically anchored in actual recidivism patterns. By systematically testing numerous cutoff combinations, we were able to identify thresholds that maximize predictive accuracy while remaining user-friendly for frontline staff. Such an approach helps reduce the subjectivity that can arise from more arbitrary or purely theoretical categorization methods.

Predictive Validity

A central aim of this study was to determine how well the ORAC-PCQ predicts criminal recidivism, both in the overall sample and within specific subgroups (non-Indigenous men, non-Indigenous women, Indigenous men, and Indigenous women). The results were encouraging: the area under the ROC curve (AUC) analyses indicated satisfactory predictive accuracy across all groups. In particular, the ORAC-PCQ exhibited an AUC of .75 for the total sample, which is higher than the average predictive value of commonly used Level of Service (LS) instruments. As a point of reference, a meta-analysis by Olver et al. (2014) reported an average AUC of .67 for the LS across various populations and types of recidivism outcomes. These findings suggest that the ORAC-PCQ, at least within the Quebec context and for individuals serving short sentences, operates at the benchmark typically established by the LS family of tools.

Notably, the ORAC-PCQ’s predictive accuracy also held up across subgroups, showing AUC values of .74 for non-Indigenous men, .79 for non-Indigenous women, .68 for Indigenous men, and .70 for Indigenous women. Although the AUC for Indigenous men and women was somewhat lower, these values still exceeded the conventional .64 threshold for moderate predictive accuracy (Rice & Harris, 2005). This pattern of results mirrors those observed by Olver et al. (2024), who found that general risk/need tools tended to demonstrate acceptable predictive validity among Indigenous persons but typically performed better among non-Indigenous populations. This convergence underscores the importance of continued empirical evaluation of risk tools to ensure their consistent accuracy and fairness across population groups.

Predictive validity figures of the ORAC-PCQ compared favorably to previous findings on Level of Service instruments, which often hover in the low-to-mid .60s among Indigenous populations (Gutierrez et al., 2013; Wilson & Gutierrez, 2014). These results suggest that, at least within the Quebec context, the ORAC-PCQ can maintain a predictive accuracy for Indigenous individuals. However, it is important to interpret these results within the broader social and historical context of correctional involvement. The strong coherence between risk scores and observed recidivism may partly reflect structural inequities that influence both offending patterns and system responses (Giguère et al., 2025). For example, the legacy of colonialism—including systemic disadvantage, intergenerational trauma, and heightened surveillance—can increase both actual risk exposure and the visibility of reoffending among Indigenous persons (Ewert v. Canada, 2018; Hart, 2016). In such contexts, predictive validity may reflect real associations but also risks reinforcing existing disparities if scores are interpreted without attention to context (Gutierrez et al., 2013). These considerations do not diminish the value of actuarial tools like the ORAC-PCQ, which support consistency and transparency in correctional decision-making. Rather, they highlight the importance of embedding such tools within culturally informed and equity-oriented practices that ensure assessments serve not only predictive purposes, but also the broader goal of fair and effective intervention.

Latent Structure and Group Measurement Invariance

The latent structure of the ORAC-PCQ was assessed and compared across subgroups to evaluate the generalizability of its structural properties. While many risk assessment tools include subscales, these are often theoretically driven. Relatively few studies have attempted to empirically confirm these factorial structures using appropriate statistical analyses (for exceptions, see: Brouillette-Alarie et al., 2016; Gordon et al., 2015; Hsu et al., 2011; Zhang & Liu, 2015). Using exploratory factor analysis (EFA), we identified four underlying factors within the 24 items of the ORAC-PCQ: Criminal Background, Substance Abuse, Interpersonal Conflicts, and Employment Problems. While the ORAC-PCQ’s factorial structure does not perfectly align with the Central Eight risk factors of recidivism (Bonta & Andrews, 2023)—nor does it claim to—it remains conceptually consistent with these domains. The Criminal Background dimension aligned with the Central Eight’s domain of History of Antisocial Behavior, and the Substance Abuse factor mapped directly onto the Substance-Use domain. Interpersonal Conflicts merged features often discussed separately in the Central Eight—such as Antisocial Personality Patterns, Antisocial Attitudes, and Family/Marital Challenges—suggesting that, in this short-sentence population, these constructs coalesced into a broader “relational risk” dimension. Finally, Employment Problems paralleled the School/Work Instability domain of the Central Eight, as well as the Antisocial Associates. Confirmatory factor analyses (CFA) upheld the viability of this four-factor solution across key subgroups. Although the fit indices were somewhat attenuated for Indigenous women—possibly due to a smaller sample—overall results indicate consistent underlying constructs across varied demographic groups. From a practical standpoint, this stability of factor structure enhances the ORAC-PCQ’s claim to generalizability: officers and program staff can interpret these four risk dimensions in a similar way, regardless of an individual’s gender or Indigenous identity.

Limitations

This study presents certain limitations that should be recognized. First, all participants were under the supervision of Quebec’s Correctional Services, which only manage sentences of 2 years or less. Consequently, our findings may not be directly generalizable to individuals serving longer sentences in Canada’s federal penitentiaries, whose offense patterns and recidivism trajectories could differ. Second, even though our sample was relatively large, it was drawn primarily from non-Indigenous populations in Quebec. While we did include Indigenous subgroups, the size of these subgroups—particularly Indigenous women—was comparatively small, resulting in less stable estimates and potentially limiting the robustness of subgroup-specific conclusions. Third, the follow-up period for recidivism was restricted to 1 year, which aligns with the shorter sentences typically administered at the provincial level. However, this short follow-up window may overlook longer-term recidivism patterns that develop after the initial post-release period. Fourth, all available ORAC-PCQ data, including those used during the tool’s development and refinement, were included in the present analysis. This reuse of data introduces a potential risk of overfitting, as the tool may perform optimally on the dataset used to inform its design. Finally, because official conviction data were used to measure recidivism, our results do not account for unreported crimes or charges that did not lead to convictions, which may have led to an underestimation of actual reoffending.

Conclusion

The ORAC-PCQ, a new fourth-generation risk tool designed for adjudicated individuals serving short sentences, was developed in accordance with Messick’s (1989, 1995) framework and his concept of a nomological network. Consequently, this study aimed to evaluate the unified validity of the instrument. As discussed—though not extensively detailed in this paper—the items were created, validated, and refined based on the recommendations of a committee of correctional experts to ensure their relevance in measuring recidivism risk and to strengthen their content validity. The factor analysis presented in this study provided evidence of the substantive validity of the instrument’s underlying constructs, demonstrating coherence with criminological theories, existing literature, and comparable risk assessment tools. Further reinforcing the tool’s structural validity, measurement invariance analyses demonstrated that these latent factors function similarly across men, women, and Indigenous subgroups, although the small sample of Indigenous women calls for caution in interpreting factorial stability. Criterion validity was supported by robust predictive accuracy over a 1-year follow-up period, achieving area-under-the-curve values comparable to or exceeding modern standards (e.g., LS scales). Although some doubts remain regarding applicability for Indigenous women, these appear largely tied to the limited sample size, which constrains the stability of subgroup-specific estimates. Overall, the results suggest that the ORAC-PCQ is a promising, empirically grounded instrument for short-sentence populations in Quebec, with potential to inform risk assessment and intervention strategies more broadly.

Building on these findings, several avenues warrant further investigation. First, longer follow-up periods would help clarify how well the ORAC-PCQ predicts recidivism over multiple years and whether its predictive accuracy changes over time, thereby examining its dynamic properties. Second, although this study demonstrated promising results across subgroups, future research should include more diverse ethno-racial samples to ensure the tool’s validity beyond the Indigenous vs. non-Indigenous distinction. Third, collecting additional data on Indigenous women may stabilize factor analyses and risk estimates, thereby confirming (or refuting) the tool’s validity for this subgroup. Fourth, a more granular exploration of item functioning using advanced psychometric approaches (e.g., multidimensional item response theory) could reveal whether particular items disproportionately influence total scores or penalize certain groups. Finally, extending validation efforts to other jurisdictions or countries is essential for confirming the ORAC-PCQ’s broader applicability and determining whether cutoff scores require recalibration in new correctional contexts. Pursuing these directions will help refine the ORAC-PCQ, ensuring it remains both psychometrically robust and practically effective for the populations it serves.

Footnotes

Authors’ Note:

The views expressed are those of the authors and not necessarily those of the Ministère de la Sécurité publique du Québec. No AI tools were used to analyze data, interpret results, or generate theoretical or methodological content. All scientific reasoning, analyses, and discussion points reflect the authors’ original work. We have no conflict of interest to disclose.

ORCID iDs

Guy Giguère

Sébastien Brouillette-Alarie

Guy Giguère is a specialist in psychometrics and a lecturer at Université Laval and the Université du Québec à Trois-Rivières. He conducted various research projects on the evaluation of actuarial tools and correctional programs for the Quebec Ministry of Public Security.

Sébastien Brouillette-Alarie is an Associate Professor at the School of Criminology of the Université de Montréal and serves as the Research Lead for the Canadian Practitioners Network for the Prevention of Violent Extremism.

Yanick Charette is an associate professor at the School of Social Work and Criminology at Université Laval. His research focuses on the contribution of quantitative methods to our understanding of life trajectories of justice-involved individuals with mental health issues.

William Arbour is an Assistant Professor of Economics at the University of Montreal. His research focuses on the economics of crime and education, and he is affiliated with the International Center for Comparative Criminology and CIRANO.

References

Ægisdóttir

White

M. J.

Spengler

P. M.

Maugherman

A. S.

Anderson

L. A.

Cook

R. S.

Nichols

C. N.

Lampropoulos

G. K.

Walker

B. S.

Cohen

Rush

J. D.

(2006). The Meta-Analysis of Clinical Judgment Project: Fifty-six years of accumulated research on clinical versus statistical prediction. The Counseling Psychologist, 34(3), 341–382. https://doi.org/10.1177/0011000005285875

Andrews

D. A.

(1982). The Level of Supervisory Inventory (LSI): The first follow-up. National Institute of Justice.

Andrews

D. A.

Bonta

(1995). The Level of Service Inventory—Revised. Multi-Health Systems.

Andrews

D. A.

Bonta

Wormith

J. S.

(2004). Manual for the Level of Service/Case management Inventory (LS/CMI). Multi-Health Systems.

Andrews

D. A.

Guzzo

Raynor

Rowe

R. C.

Rettinger

L. J.

Brews

Wormith

J. S.

(2011). Are the major risk/need factors predictive of both female and male reoffending? A test with the eight domains of the Level of Service/Case Management Inventory. International Journal of Offender Therapy and Comparative Criminology, 56(1), 113–133. https://doi.org/10.1177/0306624X10395716

Bonta

Andrews

D. A.

(2023). The psychology of criminal conduct (7th ed.). Routledge. https://doi.org/10.4324/9781003292128

Brouillette-Alarie

Babchishin

K. M.

Hanson

R. K.

Helmus

L.-M.

(2016). Latent constructs of the Static-99R and Static-2002R: A three-factor solution. Assessment, 23(1), 96–111. https://doi.org/10.1177/1073191114568114

Brouillette-Alarie

Lee

S. C.

Longpré

Babchishin

K. M.

(2023). An examination of the latent constructs in risk tools for individuals who sexually offend: Applying multidimensional item response theory to the Static-2002R. Assessment, 30(4), 1249–1264. https://doi.org/10.1177/10731911221076373

Cattell

R. B.

(1966). The scree test for the number of factors. Multivariate Behavioral Research, 1(2), 245–276. https://doi.org/10.1207/s15327906mbr0102_10

10.

Chen

F. F.

(2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14(3), 464–504. https://doi.org/10.1080/10705510701301834

11.

Cheung

G. W.

Rensvold

R. B.

(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2), 233–255. https://doi.org/10.1207/S15328007SEM0902_5

12.

Clark

(2019). Overrepresentation of Indigenous people in the Canadian criminal justice system: Causes and responses. Department of Justice Canada. https://www.justice.gc.ca/eng/rp-pr/jr/oip-cjs/oip-cjs-en.pdf

13.

Colquitt

J. A.

Sabey

T. B.

Rodell

J. B.

Hill

E. T.

(2019). Content validation guidelines: Evaluation criteria for definitional correspondence and definitional distinctiveness. Journal of Applied Psychology, 104(10), 1243–1265. https://doi.org/10.1037/apl0000406

14.

de Vogel

Nicholls

T. L

. (2016). Gender matters: An introduction to the special issues on women and girls. The International Journal of Forensic Mental Health, 15(1), 1–25. https://doi.org/10.1080/14999013.2016.1141439

15.

Ewert v. Canada, 2018 SCC 30, [2018] 2 S.C.R. 165. https://decisions.scc-csc.ca/scc-csc/scc-csc/en/item/17133/index.do

16.

Giguère

Hamidi

Bourassa

Brouillette-Alarie

(2025). Assessing cultural bias in recidivism risk tools: Differential item functioning and predictive validity of the ORAC-PCQ and LS/CMI among Indigenous and non-Indigenous men in Quebec’s correctional system [Manuscript submitted for publication].

17.

Giguère

Lussier

(2016). Debunking the psychometric properties of the LS\CMI: An application of item response theory with a risk assessment instrument. Journal of Criminal Justice, 46, 207–218. https://doi.org/10.1016/j.jcrimjus.2016.05.005

18.

Giroux

(2011). Profil correctionnel 2007-2008: La population correctionnelle du Québec [Correctional profile 2007-2008: Quebec’s correctional population]. Ministère de la Sécurité publique du Québec. https://www.bibliotheque.assnat.qc.ca/DepotNumerique_v2/AffichageFichier.aspx?idf=36966

19.

Gordon

Kelty

S. F.

Julian

(2015). Psychometric evaluation of the Level of Service/Case Management Inventory among Australian offenders completing community-based sentences. Criminal Justice and Behavior, 42(11), 1089–1109. https://doi.org/10.1177/0093854815596419

20.

Gottfredson

S. D.

Moriarty

L. J.

(2006). Statistical risk assessment: Old problems and new applications. Crime & Delinquency, 52(1), 178–200. https://doi.org/10.1177/0011128705281748

21.

Government of Quebec. (2002). Act respecting the Québec correctional system (R.S.Q., c.S-40.1). https://www.legisquebec.gouv.qc.ca/en/document/cs/S-40.1

22.

Grove

W. M.

Zald

D. H.

Lebow

B. S.

Snitz

B. E.

Nelson

(2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12(1), 19–30. https://doi.org/10.1037/1040-3590.12.1.19

23.

Gutierrez

Wilson

H. A.

Rugge

Bonta

(2013). The prediction of recidivism with Aboriginal offenders: A theoretically informed meta-analysis. Canadian Journal of Criminology and Criminal Justice, 55(1), 55–99. https://doi.org/10.3138/cjccj.2011.E.51

24.

Hart

S. D.

(2016). Culture and violence risk assessment: The case of Ewert v. Canada. Journal of Threat Assessment and Management, 3(2), 76–96. https://doi.org/10.1037/tam0000068

25.

Henson

R. K.

Roberts

J. K.

(2006). Use of exploratory factor analysis in published research: Common errors and some comment on improved practice. Educational and Psychological Measurement, 66(3), 393–416. https://doi.org/10.1177/0013164405282485

26.

Horn

J. L.

(1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2), 179–185. https://doi.org/10.1007/BF02289447

27.

Hsu

C.-I.

Caputi

Byrne

M. K.

(2011). The Level of Service Inventory–Revised (LSI-R) and Australian offenders: Factor structure, sensitivity, and specificity. Criminal Justice and Behavior, 38(6), 600–618. https://doi.org/10.1177/0093854811402583

28.

L.-t.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. https://doi.org/10.1080/10705519909540118

29.

Kubinger

K. D.

(2003). On artificial results due to using factor analysis for dichotomous variables. Psychology Science, 45(1), 106–110.

30.

McIntosh

McKeen

(2018, June 19). Overrepresentation of Indigenous people in Canada’s prisons persists amid drop in overall incarceration. Toronto Star. https://www.thestar.com/news/canada/overrepresentation-of-indigenous-people-in-canada-s-prisons-persists-amid-drop-in-overall-incarceration/article_8ff7df14-620d-5584-a205-b6638a9a11d3.html

31.

Messick

(1989). Validity. In Linn

R. L.

(Ed.), Educational measurement (3rd ed., pp. 13–103). Macmillan; American Council on Education.

32.

Messick

(1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741

33.

Muthén

L. K.

Muthén

B. O.

(2017). Mplus user’s guide (8th ed.).

34.

Olver

M. E.

Stockdale

K. C.

Helmus

L. M.

Woods

Termeer

Prince

(2024). Too risky to use, or too risky not to? Lessons learned from over 30 years of research on forensic risk assessment with Indigenous persons. Psychological Bulletin, 150(5), 487–553. https://doi.org/10.1037/bul0000414

35.

Olver

M. E.

Stockdale

K. C.

Wormith

J. S.

(2014). Thirty years of research on the Level of Service Scales: A meta-analytic examination of predictive accuracy and sources of variability. Psychological Assessment, 26(1), 156–176. https://doi.org/10.1037/a0035080

36.

Rice

M. E.

Harris

G. T.

(2005). Comparing effect sizes in follow-up studies: ROC Area, Cohen’s d, and r. Law and Human Behavior, 29(5), 615–620. https://doi.org/10.1007/s10979-005-6832-7

37.

Ruscio

(2008). A probability-based measure of effect size: Robustness to base rates and other factors. Psychological Methods, 13(1), 19–30. https://doi.org/10.1037/1082-989X.13.1.19

38.

Statistics Canada. (2018). Adult and youth correctional statistics in Canada, 2016/2017. https://www150.statcan.gc.ca/n1/pub/85-002-x/2018001/article/54972-eng.htm

39.

Stevens

J. P.

(1992). Applied multivariate statistics for the social sciences (2nd ed.). Lawrence Erlbaum.

40.

Swets

J. A.

Dawes

R. M.

Monahan

(2000). Psychological science can improve diagnostic decisions. Psychological Science in the Public Interest, 1(1), 1–26. https://doi.org/10.1111/1529-1006.001

41.

Tucker

L. R.

Lewis

(1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38(1), 1–10. https://doi.org/10.1007/BF02291170

42.

van der Knaap

L. M.

Alberda

D. L.

Oosterveld

Born

M. P

. (2012). The predictive validity of criminogenic needs for male and female offenders: Comparing the relative impact of needs in predicting recidivism. Law and Human Behavior, 36(5), 413–422. https://doi.org/10.1037/h0093932

43.

Van Voorhis

Wright

E. M.

Salisbury

Bauman

. (2010). Women’s risk factors and their contributions to existing risk/needs assessment: The current status of a gender-responsive supplement. Criminal Justice and Behavior, 37(3), 261–288. https://doi.org/10.1177/0093854809357442

44.

Velicer

W. F.

(1976). Determining the number of components from the matrix of partial correlations. Psychometrika, 41(3), 321–327. https://doi.org/10.1007/BF02293557

45.

Viljoen

J. L.

Goossens

Monjazeb

Cochrane

D. M.

Vargen

L. M.

Jonnson

M. R.

Blanchard

A. J. E.

S. M. Y.

Jackson

J. R.

(2024). Are risk assessment tools more accurate than unstructured judgments in predicting violent, any, and sexual offending? A meta-analysis of direct comparison studies. Behavioral Sciences & the Law, 43(1), 75–113. https://doi.org/10.1002/bsl.2698

46.

Wilson

H. A.

Gutierrez

(2014). Does one size fit all? A meta-analysis examining the predictive ability of the Level of Service Inventory (LSI) with aboriginal offenders. Criminal Justice and Behavior, 41(2), 196–219. https://doi.org/10.1177/0093854813500958

47.

Yang

Wong

S. C. P.

Coid

(2010). The efficacy of violence prediction: A meta-analytic comparison of nine risk assessment tools. Psychological Bulletin, 136(5), 740–767. https://doi.org/10.1037/a0020473

48.

Zhang

Liu

(2015). Reliability and validity of the Chinese version of the LSI-R with probationers. International Journal of Offender Therapy and Comparative Criminology, 59(13), 1474–1486. https://doi.org/10.1177/0306624X14538396

Normative Data,Predictive Validity,and Factor Structure of a Fourth-Generation Risk Assessment Tool (ORAC-PCQ) in a Population of Adjudicated Individuals Serving Short Sentences in the Province of Québec (Canada)

Abstract

Keywords

Introduction

Convicted Person Assessment in Quebec and the Development of a New Actuarial Tool

Contemporary Issues in Actuarial Tool Validation

Psychometrics

Gender Considerations

Applicability to Indigenous Persons

Study Objectives

Method

Sample

Measures

The ORAC-PCQ

Criminal Recidivism

Analytical Strategy

Normative Data

Initialization

Optimization

Iteration

Optimal cutoffs

Predictive Validity

Factorial Structure

Exploratory factor analysis

Confirmatory factor analysis and group invariance testing

Results

Risk Distribution Table

Cutoff Scores for Risk Categories

Predictive Validity

Exploratory Factor Analysis

Confirmatory Factor Analysis and Group Invariance Testing

Discussion

Normative Data for the ORAC-PCQ

Predictive Validity

Latent Structure and Group Measurement Invariance

Limitations

Conclusion

Footnotes

Authors’ Note:

ORCID iDs

References