Abstract
Talent identification and selection are crucial for the success of elite sport organizations. Scouts and managers generally select the most promising young athletes based on their current performances, physiological characteristics, and gut feelings. However, psychological characteristics (including perceptual-cognitive and self-regulation abilities) might still be overlooked by selectors. This study aimed at verifying the relationship between psychological characteristics and performance in elite ice-hockey. Eighty-eight youth elite ice-hockey players (forwards and defensemen) eligible for a Major Junior selection draft participated in the study. They were measured at 15 years old on perceptual-cognitive skills (decision-making and anticipation with eye-tracking at a temporal occlusion task) and self-regulated learning abilities (self-reported questionnaire). In addition, their current (draft rank and scouts’ subjective appreciation) and future (points, games played, differential for the following four years) performances were recorded. Multiple linear regression models showed that the scouts’ subjective appreciation was the best predictor of current and future performance. However, when scouts’ appreciation is removed from the models or when positions are analyzed separately, self-regulated learning abilities (effort, planning and reflection subscales) and decision-making could add to the prediction. Overall, this study shows that psychological characteristics could help scouts in the talent identification and selection process, but measuring these characteristics cannot replace their judgment.
Introduction
Achieving excellence is a common goal in high performance sports. To do so, athletes’ individual effort is essential, but rarely enough. Trained coaches that have the development of athletes at heart, high level competitors (teammates and opponents), and access to sports-related equipment and facilities are just a few of the other contributing factors. However, those resources are not readily available to all. Indeed, only the teams playing at the highest levels for an age group typically have access to such a rich talent development environment, and spots on those elite teams are rare. 1 To decide which athletes will occupy those spots, a process of talent identification and selection is set in place. The use of the term ‘talent’ is currently the object of debates among sport scientists. Some argue that talent refers to the current abilities that one possesses in a given field that place them in the top 10% of their age group, 2 whereas others stipulate that it refers to having the necessary characteristics to excel in the future (i.e., potential). 3 One thing that seem agreed upon is that talent is rare. 3 With respect to talent identification, it is defined as the process where players are recognized to have the potential to excel in a more advanced level of competition for a specific sport, which aligns with the second definition provided of talent.4–6 Measures are quantitatively or qualitatively established by researchers, scouts, and coaches to identify the athletes with hopefully the most potential to excel later in life of an area or a country, which eventually leads to talent selection. Talent selection corresponds to the complex process of deciding which contenders will remain in the system from a sample of potential athletes. 1
Ice-hockey in [name of Province and country] provides an example of a talent identification and selection process that spans over multiple years. There, the youth ice-hockey structure is divided into seven categories based on age. Those categories are further separated by performance levels according to players’ abilities (e.g., AAA, AA, BB, A, B). From the age of eight years old, young ice-hockey players are graded and selected for their abilities each year. One of the last and most important events for talent identification and selection in minor hockey is the [name of league] annual draft. Most of the best male ice-hockey prospects aged 15 and 16 from [name of Provinces] attend the event. During the draft, each team has the opportunity to select one player per round over 14 rounds 1 . Players expected to perform better now or in the future are usually selected in the earlier rounds (i.e., those seen with more ‘potential’). 7 These decisions are based on extensive evaluation of players by scouts and coaches throughout the preceding year and at the combine 2 . Each team from the [name of league] has its own staff of five to 10 part-time professional scouts.
Scouting and its limits
To evaluate and rank players, scouts typically base their impression on subjective (e.g., experiential knowledge 8 ) and objective factors including anthropometric measures (e.g., weight, height), physiological assets (e.g., maximal aerobic power, skating speed 9 ), and game performance metrics (statistics such as points and games played). They also discuss with the players and their coaches to assess intangible characteristics including psychological assets (e.g., passion, character, leadership) and ‘red flags’, which represent concerns about traits or behaviors (e.g., selfish on-ice behaviors 10 ). However extensive the scouts’ evaluation may be, research shows that some identified athletes end up underperforming, and vice-versa.3,11,12 There are numerous reasons why.
First, an important issue in talent identification is that sports performance is multidisciplinary. Indeed, athletes hold different sets of skills (e.g., physical, tactical) and this variability needs to be considered in talent selection. 3 As an example, Baker and colleagues 3 explained that some athletes have a relatively homogenous profile in their aptitudes, being good in most aspects of the game. However, others hold a more heterogeneous profile because they are excellent in a specific aspect. In any case, there does not seem to be a clear set of skills that predict future performance and success. 13 For example, Martin St-Louis was never drafted by a National Hockey League team (the most competitive ice hockey league in the world), mainly because of his small size. However, this eventual member of the Hockey Hall of Fame has counted mainly on his grit and his exceptional reading of the game and decision-making abilities throughout his career. The comparison of players is therefore complex. Second, in addition to being multidisciplinary, performance is also complex and does not develop linearly (i.e., early talent indicators are not necessarily linked with future performance 1 ). For example, anthropometric features such as height and weight and current performances are linked in early development. At that stage, physical maturation disparity creates differences in performance and players born early in the selection year or those that are early bloomers hold a physical advantage over their later-developing counterparts (i.e., relative-age effect and maturation bias).14–16 Unfortunately, these biases are still present in talent identification, even though scouts and coaches are now aware of the existence of the biases. 17 Fourth, decisions in talent identification and selection are often sources of disagreements between scouts and coaches, and this variability reduces the reliability of the judgments as to whom should be selected. 18 Finally, but not exhaustively, athletes’ talent assessment methods are likely incomplete. Although efforts are made to measure athletes’ physical and physiological characteristics objectively, the measures that substantiate psychological assessment are typically subjective (i.e., interviews to assess psychological assets).10,11,19 These measures are also underused relative to physical assessments,20–22 despite that they might be promising predictors of performance.19,23,24 Therefore, operationalized measures of psychological concepts could help scouts refine and standardize their judgment.
Psychological characteristics
The relative underuse of psychological characteristics compared to physical assessments in the talent identification process may come from the difficulties in targeting and operationalizing psychological concepts.21,25,26 Valid and time-efficient tools that might not have been intended for talent identification could be used in this sense. Some psychological characteristics have been operationalized and found to be related to current and sometimes future performance and could be useful in the identification of talent. Among them are self-regulated learning and perceptual-cognitive skills.
As defined by Zimmerman, 27 self-regulated learning can be seen as the implication one has in their own learning process with the purpose to improve at a specific task. It includes metacognitive (awareness of its thinking), motivational (being invested in attending its goals), and behavioral (taking actions) components. 28 Self-regulation plays an important role in the learning process of multiple disciplines including sports, as it may contribute to increasing the quantity and quality of deliberate practice an individual is willing to do to improve their performance.29,30 Deliberate practice is by definition effortful and not necessarily enjoyable, and self-regulated learners may be more prone to pursuing this to achieve a goal than other learners. 30 Kitsantas and Zimmerman 31 have brought to light the importance of self-regulation in sports and the role it plays in distinguishing expert from non-expert athletes. Jonker and colleagues 28 illustrated this finding by examining the self-regulatory skills of 222 athletes (110 males, 112 females) aged from 12 to 16 years old with a questionnaire comprised of six subscales (planning, self-monitoring, evaluation, reflection, effort and self-efficacy). Athletes were either at a junior international or junior national level and practiced either an individual or team sport. Results indicated that athletes competing at an international level had significantly higher scores only on the reflection subscale than those at a national level, regardless of the type of sports, and not on the other subscales. Reflection abilities help thinking back on learning experiences to extract as much useful information as possible from them and apply them to future performance. Athletes with good reflection would therefore be better able to develop, making them suitable candidates for talent identification programs. Reflection during adolescence was also linked with future international level attainment. 32 Also, evidence for a relationship between sport performance and the other self-regulation subscales (i.e., self-monitoring, planning, effort, evaluation and self-efficacy) were found in adults.33,34 Other personality traits like agreeableness (i.e., concern for social harmony 35 ) and grit (i.e., facing competition without abandoning 36 ) were also shown to discriminate between selected and non-selected elite female ice-hockey players. 19
Research also tends to indicate that perceptual-cognitive abilities could be used to differentiate elite athletes and novices, and even experts among themselves. 37 Perceptual-cognitive skills in sport refer to the ability of an athlete to identify relevant information in a sport-specific context, and to use that information to accurately anticipate future actions and make appropriate decisions 38 ; importantly, this definition is not uniformly agreed upon among sport scientists 39 and resemble the concept of scanning. 40 Decision-making is the cognitive process of deciding which course of actions to take among several options.41,42 While anticipation is a sub-product of decision-making, it is still assessed on its own as it refers to the capability to foresee what is about to happen next, based on different types of cues (e.g., visual, kinetic, timing).43–47 For talent identification, measuring perceptual-cognitive skills may be useful because athletes that may be able to ‘read’ the game, but not yet have the physical attributes to execute the action associated with their decisions, may eventually develop into good performer. Examples of research showing expertise advantage in terms of perceptual-cognitive skills are numerous. Roca and colleagues 48 examined anticipation and decision-making (temporal occlusion with eye-tracking) among soccer players, comparing professionals and semi-professionals with amateurs. Skilled participants (professional/semi-professional) were more accurate than amateurs in anticipating the actions of the opponents and in deciding how to respond to game situations, and performed more ocular fixations of shorter durations. Note that this pattern is not systematic among elite athletes. 49 Woods and colleagues 50 found similar results in a study about Australian football where they could classify talent-identified and less-talented athletes using a video decision-making task.
Fewer attempts have been successful in finding perceptual-cognitive differences among an exclusively elite sample. Yet, a discrimination tool between same-level athletes would be much more useful for talent identification. Breed and colleagues 51 compared 46 trained youth Australian football players (skilled vs. highly skilled decision-makers based on coaches’ ratings) on a video-based test containing three subtests (pattern recall, ruck cue utilization, and decision-making). Highly skilled players performed better on the decision-making subtest, but not on the other two subtests. O’Connor et al. 52 found similar results in youth football between selected and deselected players for a full-time sport scholarship. Fortin-Guichard et al. 11 also found a relationship between perceptual-cognitive abilities and future performance among young highly trained, but late-selected, ice-hockey players.
Objectives
The present study aims to verify the relationship between psychological characteristics and performance in elite youth ice-hockey beyond what scouts can already evaluate subjectively. To do so, we measured psychological characteristics (learning self-regulation and perceptual-cognitive skills) of ice-hockey players when they were 15 years old. We then linked their psychological characteristics with their current (draft rank and Major Junior scouts’ appreciation) and future (games played, points, points per game, differential over the four following years) performance. We did so for the overall sample, but also by separating the participants according to their playing position because scouts do not necessarily look for the same characteristics when it comes to forwards, defensemen, and goaltenders as the physical, psychological, and perceptual-cognitive demands are often position-specific. 37 It was hypothesized that in addition to the scouts’ appreciation, self-regulation and perceptual-cognitive skills would be significant predictors of both draft selection status and performance statistics four years later. More precisely, we hypothesized that the decision-making score and the reflection subscale score from the self-regulated learning questionnaire would contribute the most to the prediction models in addition to the scouts’ appreciation. This hypothesis is based on studies that showed that decision-making is the perceptual-cognitive skill that weighs the most in talent prediction and that the reflection subscale distinguishes between levels of athletic performance.28,52 Importantly, the studies cited herein were largely not specific to ice-hockey, and little is known about the information scouts use when scoring their appreciation and eventually making selection decisions. Verifying the usefulness of measuring psychological characteristics objectively in talent identification situations could help scouts and coaches to go beyond their subjective appreciation, which should lead to better selection; at the very least, this verification should help scouts and coaches decide about the usefulness of these measures.
Methods
Participants
A total of 88 male youth highly trained ice-hockey players at the [name of level] level participated in this study. They were aged between 15 and 16 years old (Mage = 15.60, S.D. = 0.49) at the time of the assessment, and considered highly trained because they played nationally at the highest Canadian competitive level for their age group, where ice-hockey is the national sport.53,54 The sample was comprised of 28 defensemen (Mage = 15.61, S.D. = 0.50), and 60 forwards (Mage = 15.59, S.D. = 0.50). Since the sample of goaltenders was too small and the perceptual-cognitive demands of this position are too different from that of the other positions (i.e., more in reaction to the main action than active decision-makers), the eight goaltenders (Mage = 15.5, S.D. = 0.53) were removed from the initial sample (that included 96 players). They were all eligible for the 2019 [name of league] draft in [name of country]. Participants were contacted for an interview a few months prior to the draft, by the scouts of a [name of league] team that also collaborated with our research group. The team had six part-time and one full-time professional male scouts aged between 26 and 61, with on average 7.67 years of experience (SD = 3.01). The collected data belong to that ice-hockey team, and participants gave their written informed consent before the interview so that it can be used for research purposes. Those participants were selected because the scouts had a potential interest in those specific players, among a list containing 667 eligible players (252 end up being drafted each year).
Materials
The experimental session took place in an available room in either an arena, a hotel, or a school according to the city of each tested player (scouts and researchers traveled together to home cities of participants to reduce the experimental load and travel expenses for the participants). The video sequences were run on an Intel Core i-7 computer with the Windows 10 system and displayed on a 22-inch computer screen (with a refresh rate of 60 Hz). Verbalisation responses were recorded using a LG digital recorder and manual responses were recorded using a basic keyboard. Eye movements were tracked at 120 Hz using a Tobii X3-120 device (0.4-degree accuracy and 0.24-degree precision as used in other sport psychology research). 37 The experiment was configured using the Tobii Pro Lab software version 1.31 (Tobii Pro, Stockholm, SWE).
Video sequences
Video sequences were used to evaluate anticipation, decision-making, and eye movements. These sequences were filmed with a GoPro HERO7 Black camera (30 Hz). Some of the sequences were filmed from the stands in the left corner of the rink, while others were filmed from behind the goaltender (see Figure 1). All sequences were recorded during a Major Junior team's regular training. Ninety-seven attacking sequences were obtained. There were 2v1, 2v2 and 3v2 plays. The first two authors selected 50 of those 97 sequences. The selection was done based on the video quality of the sequences and the absence of distractors such as a supplementary puck in the background. Three coaches of the collaborative Major Junior team screened the remaining 50 sequences together and collectively agreed on a rank on a 7-point Likert scale for representativeness (from 1 [not representative] to 7 [perfectly representative]). They did so for each hockey position, and the nine most representative sequences for each position were kept. The sequences were edited to freeze at a critical moment (at 120 ms before the last contact of the forward with the puck at the top of the faceoff circle in the attacking zone; Figure 2). Participants viewed the sequences in the same order (position-specific). However, the order was randomized prior to the experimentation.
Perspective of the video sequences.
Steps of the perceptive-cognitive ability task specific to ice hockey.
Measures
Measures obtained from participants were 1) sociodemographic characteristics, 2) learning self-regulatory skills, 3) perceptual-cognitive skills (anticipation, decision-making, and eye-movements), and 4) performance measures (scouts’ appreciation, draft ranks, in-game statistics for the four following years after initial data collection).
Sociodemographic questionnaire
A pen-and-paper sociodemographic questionnaire was administered to collect birth date, injury history, and an estimation of weekly training hours in ice-hockey (in the past year, including strength conditioning and mental preparation).
Self-regulated learning questionnaire
The pen-and-paper Self-regulated learning questionnaire was used to evaluate self-regulation.28,55 It measured the implication one has in their own learning process with the purpose to improve at a specific task. The English version showed acceptable internal consistency (Cronbach's α of each subscale between .73 and .85) and sufficient temporal stability (ICCs of each subscale between .69 and .84). 56 Note that for 73 participants, a French version of this questionnaire was used. For the present study, Table 1 presents the Cronbach's α for both the French and English version, as well as combined for the whole sample. A double translation was done by the research team. The questionnaire had a total of 50 questions divided in six subscales (planning, self-monitoring, evaluation, reflection, effort, and self-efficacy). The average score for each subscale was used in the analyses. The six subscales of the self-regulated learning questionnaire 28 are: (1) Planning, which refers to someone's ability to approach tasks in advance of their actions (9 items on a 4-point Likert scale). (2) Self-monitoring, which is the ability to observe one's own improvement (8 items on a 4-point Likert scale). (3) Evaluation, which refers to the ability to retrospectively think on the process of a task and its outcomes (8 items on a 5-point Likert scale). (4) Reflection, which assesses the ability to ponder about its own ability on a task (5 item on a 5-point Likert scale). (5) Effort, which evaluates someone's ability to invest her/himself toward a goal to reach a certain level of performance (10 items on a 4-point Likert scale). (6) Self-efficacy, which refers to someone beliefs about their own way of facing task requirements (10 items on a 4-point Likert scale).
Cronbach's α of the subscales of the Self-Regulated Learning Questionnaire according to language of completion, as well as combined for the whole sample.
Ice-hockey specific perceptual-cognitive skills
Anticipation
Anticipation was measured with a keyboard response following the temporal occlusion of each video sequence. Participants had to indicate as rapidly as possible what they thought would happen next once the sequence was occluded. One response per sequence was required and a point was awarded for every correct response out of eight (i.e., correctly guessing what would happen next). Depending on each player's playing position in ice-hockey, the video sequences differed, and the response mode as well. Forwards had to predict the action of the forward with the puck from the video. They had to indicate if they thought he would either “pass” or “shoot” by pressing either the “p” or the “s” key of the keyboard. Defensemen had to predict the action of the defenseman closest to the forward with the puck from the video. They had to indicate if they thought he would either “follow the forward” or “block the pass option” by respectively pressing either the down arrow key or the left/right arrow key. To help the participants, a researcher asked them verbally “what will happen next?”.
Decision-making
The decision-making task was based on previous sport science studies.48,57 To assess decision-making, participants had to say aloud what they would do if they were one of the players on screen, according to their position (e.g., forwards had to verbalize as if they were the forward with the puck on screen and so on), and what information on the screen led them to this decision. This had to be done after pressing the key for the anticipation task and could differ from their previous response. To help them remember what they had to do, a researcher asked them “what would you do in this situation?”. Verbalizations were recorded and each mention of a specific and relevant cue allowed one point. Relevance of cues was assessed by experienced scouts and managers prior to the study. Appropriate cues could be information such as the handedness of a specific player or the position of a specific player relative to a teammate. Each sequence had a different maximum number of points. This maximum score was normalized for each position (forward and defenseman). Irrelevant cues were not considered in the scoring system.
Eye movements
At the beginning of each video sequences, prior to temporal occlusion, eye movements were tracked and recorded by the Tobii X3-120 device (0.4° accuracy and 0.24° precision 12 ), which measures the number, duration, and location of ocular fixations. A fixation was defined as a moment when the angular velocity of the eyes relative to the stimuli was 30°/s or less for at least 100 ms. 58 The Areas of interest (AOI) were all the players involved in the clip. Data from the two sequences that were common to both positions were used for analyses.
Performance measures
Draft rankings
Draft rankings were obtained by three methods. First, the actual outcome of the 2019 [name of league] draft was compiled from the official website of the [name of league]. Second, a ranking established prior to the draft by the [name of league] scouting Central [name of Central] was also obtained on the same website. This ranking represents the subjective impression of scouts with no personal needs (i.e., they do not work for a specific team) of the actual skills of the participants at the moment of data collection. Finally, a ranking established by the [name of team] scouts and coaches prior to the draft was obtained. This ranking represents the preferences of the [name of team] staff on the value of each player. Those rankings represented current performance, since they were obtained shortly after the psychological measures.
Scouts’ appreciation score
Scouts from the Major Junior team collaborating with the research team also provided their evaluations of each player made prior to the study. Five scouts gave a subjective score out of nine for each of the following skills: 1) skating speed, 2) hockey sense (i.e., on-ice decision-making abilities) and 3) compete level (i.e., how combative each player was). These skills represented the subjective preferences of the team collaborating to this research. The average of the three scores was used as global appreciation of the player by each scout. The appreciation scores were averaged across the scouts for analysis.
In-game statistics
In-game statistics were collected at the end of each hockey season 3 (2019–2020, 2020–2021, 2021–2022, and 2022–2023) on the [name of league] website or an equivalent league, such as the [name of other league]. The collected statistics were points (goals and assists), differential (i.e., number of times a player's team scored when they were on the ice minus the number of times the opponent scored while this player is on the ice), number of games played, and points per game (number of points divided by number of games). Only statistics from the regular season were extracted (i.e., no statistics from the playoffs). Those performance measures represent future performance, since they were obtained each of the four years after the psychological measures.
Procedure
Prior to the beginning of the study, ethical approval was granted by the [name of ethics committees and approbation number]. Participants were tested individually. Since participants believed that they were coming to a standard scouting interview with a Major Junior team, a research assistant explained to them that the standard interview was maintained, but a second part that would not impact their selection was added. The research assistant explained the self-regulated learning questionnaire and the video-based task assessing perceptual-cognitive skills. Each participant completed the sociodemographic and the self-regulated learning questionnaires and then proceeded with the computerized task. Two researchers and two scouts were present for the computerized task. Then, calibration of the Tobii X3-120 device was completed, and the task began. Each participant first had to do a practice trial, for which they received feedback about the task. The eight following sequences were then presented without feedback. All participants completed the tasks in the same order. The whole experiment took about 40 to 45 min to complete, including the scouting interview.
Analyses
All analyses were conducted using IBM SPSS version 23 software (IBM Corp., Armonk, NY). Means, standard deviations and independent samples t-tests (to compare positions) were used to describe the sample. Forward stepwise multiple linear regressions were run to verify if at least one of the measures from the battery had predictive capacity for each performance measure (i.e., the three rankings and the in-game statistics from each season, but not the scouts’ appreciation which was considered a predictor). Models were also run by positions because perceptual-cognitive demands are different according to playing position. Predictors were the quartile of birth, the six self-regulation scores, the anticipation score, the decision-making score, the number of fixations, the number of AOIs fixated, the percentage time spent fixating, and the scouts’ appreciation score. All models were also run without the scouts’ appreciation scores to explore the value of the psychological characteristics alone to predict performance. Assessment for potential collinearity problem for each model was done by verifying if the tolerance index of each predictor was below .01 (this was not the case on any model). The alpha threshold was set at .05.
Results (main analyses)
Descriptive statistics of the psychological measures
Descriptive statistics are presented in Table 2. Independent samples t-tests revealed that positions differed only on the anticipation score. Table 3 indicates how all the psychological and performance variables correlated with each other. The Pearson's correlations revealed that all the self-regulation subscales correlated moderately to strongly with each other (rs ≥ .34, ps < .01). The three rankings correlated (Spearman) strongly with each other (rs ≥ .84, ps < .01), with the in-game statistics (rs ≥ −.31, ps < .05), and the scouts’ appreciation (rs ≥ −.46, ps < .01) 4 . The scouts’ appreciation also correlated with decision-making (r = .34, p < .01), and all the in-game statistics (rs ≥ .31, ps < .01). When looking at the correlations between psychological and performance variables, the three rankings correlated (Spearman) with decision-making (rs ≥ −.29, ps < .05), and the number of games played was linked with the reflection subscale (r ≥ .27, p < .05).
Descriptive statistics by position.
*p < 0.05
Pearson's correlations between all the measured variables (N = 88).
Notes. a = Spearman correlations, * = p < .05, ** = p < .01.
Predicting the ranking lists
All stepwise regression models are presented in Table 4. Regressions that included all the athletes are presented first, followed by those separated by position (forwards and defensemen).
Linear regression models for the current performance (rankings).
[Name of league] actual draft outcome
The stepwise regression for the actual draft ranking included only the scouts’ appreciation score as a predictive variable and explained 29% of the variance (adjusted R2 = .29). Subsequent analyses revealed a higher percentage of variance explained by the scouts’ appreciation in the draft ranking of the forwards players (adjusted R2 = .31) than the defensemen (adjusted R2 = .22).
[Name of league] central ranking
The best predictive model for the Central ranking included only the scouts’ appreciation score as a predictive variable and explained 23% of the variance (adjusted R2 = .23). Subsequent analyses revealed that the scouts’ appreciation remained a significant and main predictor when positions are considered separately. The predictive model exclusive to the forwards also included the planning subscale and explained 32% of the variance (adjusted R2 = .32). The predictive model exclusive to the defensemen included the planning and the effort subscales as predictive variables and explained 49% of the variance (adjusted R2 = .49).
[Name of league and team] ranking
The best regression model for the [team name] ranking included only the scouts’ appreciation score. It explained 37% of the variance from the team's ranking (adjusted R2 = .37). The same predictor was included when considering only forwards (adjusted R2 = .38). The predictive model exclusive to defensemen included the scouts’ appreciation score and the planning subscale as a predictive variable (adjusted R2 = .43).
Predicting in-game statistics
Games played
Regressions specific to the number of games played in [name of league] (or an equivalent league) are presented in Table 5. The best model included the appreciation score and the reflection subscale as predictive variables. This model explained 16% of the variance in the games played (adjusted R2 = .16). When considering the following four seasons, there were variations between the adjusted R2 of each season (4 to 18%), but the same predictors (in combination or alone) remained significant.
Linear regression models for the future performance (games played).
Once forwards and defensemen were analyzed separately regarding games played, regressions were not constant. The best model for the forwards when considering all seasons together included only the scouts’ appreciation score and explained 16% of the variance (adjusted R2 = .16). The predictive capacity of the scouts’ appreciation score varied across seasons (adjusted R2 between .17 and .28), and was either alone (2019–2020), accompanied by the evaluation subscale (2020–2021) or by the effort subscale (2021–2022). No significant predictors were found for the 2022–2023 season for forwards, and no significant predictors were found for defensemen.
Number of points
Regressions are presented in Table 6 regarding the number of points accumulated. When considering the points total through the four seasons, the best model included only the scouts’ appreciation score as a predictive variable (adjusted R2 = .18). When regressions were carried out for each season, the appreciation score remained the only predictive variable (adjusted R² varied between .14 and .24). Also, no predictive model was found for the 2022–2023 season. Once forwards and defensemen were analysed separately, the scouts’ appreciation score remained a predictive variable, but only for forwards (adjusted R2 between .15 and .27). There was no consistent model for defensemen across seasons (see Table 6).
Linear regression models for the future performance (points).
Points per game
Table 7 presents the regression concerning the points per game statistics. The scouts’ appreciation score was the only predictor included to predict the number of points per game for forwards players in the last three seasons (adjusted R² varied between .15 and .29). For defensemen, no consistent model across the four seasons was found (Table 7).
Linear regression models for the future performance (points per game).
Linear regression models for the future performance (differential).
Differential
When considering the differential as the outcome (total and for each season; Table 8), various combinations of variables explained significant portions of variance depending on whether the models were run season-specific or position-specific (adjusted R² varied between .09 and .47). However, these combinations of variables were not consistent across the models.
Results (exploratory analyses)
Tables 9–13 present forward stepwise regressions predicting current and future performance measures based on psychological measures, but this time without the scouts’ appreciation scores. Results mainly indicate that the decision-making score predicts current performance (rankings; Table 9), and the reflection subscale of the self-regulated learning questionnaire predicts future performance (number of games played; Table 10). Other psychological predictors were inconsistent in predicting current and future performance when the scouts’ appreciation scores were not included in the models.
Linear regression models for the current performance (rankings), without the scouts’ appreciation scores.
Linear regression models for the future performance (games played), without the scouts’ appreciation scores.
Linear regression models for the future performance (points), without the scouts’ appreciation scores.
Linear regression models for the future performance (points per game), without the scouts’ appreciation scores.
Linear regression models for the future performance (differential), without the scouts’ appreciation scores.
Discussion
The aim of the present study was to determine if a brief battery of objective psychological and perceptual-cognitive tests could be useful in talent identification beyond what scouts can already evaluate subjectively. We tested the efficacy of this battery by linking the results at the tests to current performance in mid-adolescence (draft rankings) and future performance in the beginning of adulthood (in-game statistics a few years after the draft). We postulated that scouts’ appreciation would be the best predictor of performance, but that decision-making and the reflection subscale from the self-regulated learning questionnaire would also contribute to the predictions.28,52 This hypothesis was partly confirmed.
Relationship between psychological characteristics and current performance
First, the results of the battery were linked to the outcomes of the 2019 [name of league] draft, the ranking of a list provided by the scouting Central of the [name of league] (which ranked players based on their actual abilities and judgments of scouts of the Central), and of a list from a specific Major Junior team (based on the judgments of their own scouts and coaches). This was done to establish the relationship between psychological characteristics in mid-adolescence and current performance. In line with the hypothesis, results show mainly that the scouts’ appreciation score predicted all three rankings.
The scouts’ appreciation scores represent their subjective assessment of every player's skating ability, hockey sense (i.e., decision-making) and competitiveness level. It is therefore not surprising that these scores predicted the ranking for the [Team name], as the scores were provided by the scouts from that team. However, the disparity between the percentages of variance explained by the models from the three rankings might lie in the different objectives and implications associated with each ranking. Indeed, the three rankings represent a classification of the players according to different scouts’ opinion and their team's needs (e.g., filling a specific position). For example, the ranking handed out by the Central of the [Name of league] is based on players’ abilities (current performances) and an independent group of scouts’ judgments, while the actual outcome of the [Name of league] reflects players’ abilities and scout judgments from different teams that have specific needs. 8 Indeed, teams tend to draft players expected to perform better now or in the future during the first rounds (e.g., those that accumulate more points), as they represent the most obvious choices. After these first rounds, there is more ambiguity since players are more homogenous.7,59 Scouts’ judgment has more impact, and certain characteristics can be considered, such as positions to fill on each team (defensemen, forwards, or goaltenders) or the potential of a player to be a good fit with the team.8,10 These specific needs might modulate the outcome of the draft and explain why the draft outcome deviates slightly from the initial talent predictions.
Decision-making was also hypothesized to predict current performance, even adding to the scouts’ appreciation. However, it appears that the shared variance between the appreciation score and the decision-making score (r = .34 in Table 3) explains perhaps an overlapping portion of the variance of performance. In other words, even though the collinearity of these two predictors was not problematic in itself, their shared variance may have captured the same portion of the variance of performance. This implies that when both predictors are included in a regression, they probably cannot both be included as significant predictors, suggesting that decision-making, at least the way it was measured in the present study, cannot replace nor add precision to the judgments of scouts when evaluating players. Interestingly, exploratory analyses excluding the scouts’ appreciation revealed that the decision-making score contributed to predict two out of three rankings. This suggest that if scouts lacked time to assess every player they are interested in during actual games, they may carefully opt to administer the temporal occlusion task of the present study and use the decision-making score as an indicator, keeping its limited prediction value in mind.
Contrary to what was expected, the subscale scores from the self-regulated learning questionnaire did not predict current performance (i.e., rankings) beyond the scouts’ appreciation. However, when positions are analyzed separately or when scouts’ appreciation scores were removed, relationships were observed between self-regulation and performance. Therefore, again, scouts’ judgments do not seem to be replaceable with self-regulation measures, but could be used in addition. One difficulty when comparing the contribution from the self-regulated learning questionnaire score and that of scouts’ judgments is to determine the role played by the scouts’ knowledge about players’ self-regulation of learning within the scouts’ final judgments about players. We could not replicate the results reported by Jonker et al. 28 entirely about the differences between national and international athletes on the reflection subscale; nevertheless, our exploratory analyses without the scouts’ judgments did provide partial support. In Jonker and colleagues’ study, this questionnaire was administered in a research context, whereas in our study, the questionnaire was handed out prior to a scouting interview where players probably thought they had to show themselves in the most positive light possible. This situation might have led to a social desirability bias and could have impacted the way they answered the questionnaire, thus creating less disparity between participants than there might really be. 60
Relationship between psychological characteristics and future performance
The scouts’ appreciation score was predictive of future performance, especially among forwards players, which again strengthen the original hypothesis. Notably, the scouts’ appreciation scores were less linked with future performance than they were with current performance, strengthening the idea that talent development, even in already-elite athletes, may not be linear. 3 Indeed, scouts appear better to identify the players with good current performance than they are at identifying the future ones. Interestingly, Fortin-Guichard et al. 11 recently showed that psychological characteristics (eye-movements and self-regulation) could be useful to identify future performance in ice-hockey, but only when considering the late-drafted players alone. Therefore, it seems that psychological characteristics may contribute to the identification of future performance beyond what scouts may be able to do, but this contribution may be limited to late-drafted players. This is interesting because it is in the later rounds of a draft that there is more ambiguity between players because they tend to be more homogenous physically and technically.7,59
In addition, psychological characteristics (with an emphasis on reflection abilities) could contribute to the prediction, but this contribution appeared tenuous. It was only specific to some years, position, statistics considered or when scouts’ appreciation was removed. Since reflection is a form of thought regulation 56 which allows people to evaluate and adapt what they have learned to improve themselves, it can contribute to the development of specific skills.28,55 It helps athletes take in what they have learned, what still must be learned, and when to use it in different situations. 61 Therefore, it is possible that ice-hockey players that deeply reflect on what they have learned and what still must be worked on are also players with the most games played in the long run. This ability to put their knowledge into action may please their coaches and afford them more ice time.
Perceptual-cognitive abilities did not seem to predict the future performance as reflected by in-game statistics in the few years following the draft. In fact, some perceptual-cognitive abilities (e.g., decision-making, number of fixations) appeared to be related to the number of points and the differential when defensemen were analyzed separately. However, these relationships appeared tenuous because of the absence of relation with the numbers of games played and the low number of defensemen in the study, hinting towards overfitting. In any case, game-related statistics are to be taken with caution as a measure of talent, even more so because of playing position differences. For example, since ice-hockey is a sport that overall depends on the performance of a group, individual contribution is hard to quantify, and player efforts do not always lead to an appropriate performance rating or points production. This is especially the case with the points that strongly favor forwards. Interestingly, we did not find that perceptual-cognitive abilities could predict future performance beyond scouts’ judgment among forwards, which could be due to another limit inherent to this statistic. Indeed, the fact that a player plays in a specific team implies that the level of cohesion and strength of his team impacts his performance. This means the best scorer in a weak team could obtain less points than an average player in a good team. Differential index can also be problematic since neither the quality of teammates nor of opponents are considered in this score. It is therefore possible that across multiple teams, good and bad players obtain similar plus/minus differentials depending on their respective teams.62–64
In sum, the prediction of current and long-term performance using psychological characteristics do not seem to add much beyond what scouts can already judge. There may be some hints towards a contribution of planning, effort and reflection self-regulatory skills and decision-making, especially when scouts’ appreciation is not considered or when positions are analyzed separately. Also, not surprisingly, scouts appear less precise to identify future performance than current performance.
Limits
The present study is not without limitations. The test battery assessed a limited number of psychological characteristics because of time constraints, leaving potentially important predictors out (e.g., grit, personality). 19 Also, the chosen in-game statistics may be biased by other factors than the athletes’ skills. For example, the number of games played, especially early in a career, may be influenced by the draft rank of a player where players drafted earlier may play more games due to a sunk cost fallacy from their coaches and general managers.59,65 Another limit is the possibility of a selection bias (convenience sample). Participants invited to participate in the study had been pre-screened by the scouting staff of a specific [name of league] team, as they were interested in meeting them. This means that the participants were already serious contenders (and likely homogeneous in terms of talent range 66 ), potentially reducing the discriminant capacity of the battery. However, this approach allowed us to study a group of highly talented athletes that are usually difficult to reach in such large numbers. Also, the way the scouts’ appreciation scores was calculated may present an overly optimistic view of the accuracy of their judgment, because the average of the scores may have artificially reduced the noise in the individual judgments. In addition, the appreciation scores and the rankings may be influenced by cognitive biases (e.g., the halo effect 67 ), potentially overshadowing the explained variance of the objective psychological measures. Finally, the results are hardly generalizable to every selection situation in ice-hockey (e.g., women's hockey or players that are not yet ‘elite’) and further studies in various contexts are warranted.
Conclusion
In conclusion, the results of the present study suggest that psychological characteristics do not add much to what scouts are already able to identify in terms of current and future performance. When scouts’ appreciation is not considered or when positions are analyzed separately, psychological variables that may help scouts to an extent were the decision-making score and the reflection, planning and effort subscales of the self-regulated learning questionnaire. Scouts seem better at identifying current than future performance, which is not surprising given the non-linear nature of talent development. 3 Future studies on the psychological characteristics that may be used in identifying future performance should focus more on late-drafted players than when all the players are considered considering the recent findings about talent identification in ice-hockey. 11 Future studies could also find a way to measure perceptual-cognitive skills among goaltenders (with similar administration time). Finally, the measures presented herein could be studied in combination with multidisciplinary objective measures (e.g., anthropometrics, strength, speed) to better understand their relative contribution to talent identification.
Footnotes
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Social Sciences and Humanities Research Council of Canada, (grant number 435-2020-0394).
