Sage Journals: Discover world-class research

Abstract

The objective of this study was to compose an objective and detailed notational analysis system for 3 vs. 2 + GK small-sided soccer games, in which three roles are examined: attacker with ball, attacker without ball and defender. The actions and the outcome of the actions were registered for each player and in each role. Players earn points for each action and outcome according to an a priori determined scheme. Performance scores for each role are calculated as the average number of points a participant earns per trial. This notation system was tested on 19 highly talented female soccer players and validity and reliability of the system were determined. In addition, practical applications were discussed and the most important items of the notation system were determined and using only these items, a simplified notation system was proposed. The notation system has high ecological validity and can discriminate the high and low categorized players, but further development is necessary to increase the reliability of the system.

Keywords

Association football performance analysis sport analytics

Introduction

Assessing tactical skills of team sport players is challenging but interesting for both sport practice and science. In sport practice, trainers, coaches and scouts want an easy tool to determine the quality of performance, identify strengths and weaknesses and follow the developments of players. Scientifically, an objective method to assess tactical skills of team sport players on the field would be valuable for research on expertise and decision making.

Bard and Fleury¹ were the first to attempt to objectively examine decision making skills by presenting slides of offensive basketball game situations to experienced basketball players and novices, after which they had to verbalize their response. However, the validity and reliability of this test was not reported. Better ecological validity would be acquired using film clips as argued by Helsen and Pauwels.² They were among the first who developed a film-based decision-making test that has been used frequently ever since.^3–6

However, the most ecologically valid way of measuring decision making or tactical skills is by using game play.^7–9 By coding, behaviours exhibited during game play actual performance can be assessed. This is more authentic and represents one’s ability more accurately.¹⁰ In sports and physical education, there is an increasing interest in developing performance assessment instruments that can be used on game play performances. In a review, Arias and Castejón¹¹ showed that the two most often cited assessment instruments are the Team Sport Assessment Procedure (TSAP) and the Game Performance Assessment Instrument (GPAI).

The TSAP of Gréhaigne et al.¹² was designed for invasion games and examines how players gain ball possession and how they play the ball. Ball possession can be gained by conquering or receiving the ball, and then, the player can play a neutral ball, lose the ball, play an offensive ball or execute a successful shot. Based on the frequencies of occurrence, the volume of play and efficiency index can be calculated and those two combined yield a performance score. Although this is an easy-to-use assessment instrument, its major limitation is that it only examines the player in possession of the ball. Since a player carries the ball for less than 2% of the game,^13–15 it is essential that a performance assessment instrument for team sports also includes the performances of players off-the-ball.

The GPAI designed by Oslin et al.¹⁶ is the most frequently used assessment instrument¹¹ and includes both on-the-ball and off-the-ball movements. Oslin et al.¹⁶ aimed for a performance assessment instrument that can be used for any kind of game and identified general game components for which the observer has to assess appropriateness of the player’s behaviour. For example, for each time a player is in ball possession, the observer assesses the decisions made and these are coded as appropriate if a player choses to shoot or pass to an open teammate when the opportunity is available, and coded as inappropriate if a player does not pass at an appropriate time or to a marked teammate. Thus, the observer has to decide whether players are open or marked, whether a pass is given at the appropriate time or not, etc. and this leads to a high level of subjectivity in the assessment process.

Other, more recent, performance assessment instruments used general tactical principles of the game (e.g. ‘penetration’ or ‘offensive coverage’ as in FUT-SAT^17,18) or did not assess the performances of all the players (i.e. attackers and defenders) involved in the game (e.g. Game Performance Evaluation Tool¹⁹; for an overview of performance assessments instruments, see ¹¹ or ²⁰). This inspired us to develop a detailed and, in our view, more objective notation system in which the performances of all players are assessed, that is attacker with ball, attacker without ball and defender. For each role, the actions of the participants are registered as well as the outcome of the actions. Depending on the outcome of the action, the participant earns points for each action corresponding to the a priori determined point distribution, so that the user of the system is not required to judge the quality or appropriateness of the actions performed by the players. Performance scores for each role are calculated as the average number of points a participant earns per trial in that role.

The aim of the current study was to examine the validity and reliability of the notation system among highly talented soccer players. Validity was determined with regard to ecological, content, concurrent and construct validity. To determine the reliability of the notation system, inter- and intra-observer reliability were assessed. Consequently, the most important items of the notation system were determined and using only these items, a simplified notation system was proposed. Finally, practical applications were discussed.

Method

Participants

A total of 19 highly talented female soccer players participated in this study, with a mean age of 16.3 years (SD = 1.1) and a mean soccer experience of 9.9 years (SD = 2.3). They all played in the national soccer talent team, in which they train about 15 to 20 h a week and play in a high level competition for males under 14 years of age. The experiment was approved by the local ethics committee of the research institute and all participants gave their written informed consent prior to the experiment; parental consent was provided for players younger than 18 years.

Procedure

To assess the performances of the players (i.e. attackers and defenders), we chose to use 3 vs. 2 + GK small-sided games (i.e. 3 attackers vs. 2 defenders and a goalkeeper) since these are less complex than 11 vs. 11 matches, facilitate more ball touches per player and are the basics of soccer according to the Royal Netherlands Football Association.²¹ The small-sided game was played on a 40 -m long and 25 -m wide field (dimensions were advised by the head coach of the national soccer talent team) with official sized goals, and official soccer rules, including offside, were applied.

The six players were instructed to start at specific locations (Figure 1). The attackers’ task was to try to score as quickly as possible, whereas the defenders had to prevent that. If the defenders obtained ball possession, they had to try to score at the opposite goal. However, the turnover was only for motivational reasons, the notational analysis was only carried out on the performance prior to the change of ball possession (the participants were unaware of this). The trial ended if a goal was scored, a foul was made or the ball went out of play. The variables that were measured are explained in the section ‘Notation system’. After five trials, the participants switched roles (except for the goalkeeper), so that all participants played on each position. Thus, in one test, a participant played 15 attacking trials and 10 defending trials. In total eight tests were conducted, spread out over 4.5 months. Participants who attended less than five tests were excluded from analysis. A total of 733 trials were analysed; on average, a participant played 34 trials (SD = 5) per position.

Figure 1.

Overview of the small-sided game. Players are located at their specific starting positions.

The tests took place on the regular training pitch of the national soccer talent team and were video recorded with a Go Pro Hero 3 camera (Black Edition, resolution 1920 × 1080, 30 Hz; Go-Pro, USA) that was fixed on a 6.5 -m high platform (Showtec LTB-200/6 Lifting Tower, The Netherlands), and analysed afterwards using the notation system.

Notation system

Our notation system distinguishes three roles for a player: attacker with ball, attacker without ball and defender. For each role, possible actions and outcomes have been identified and defined (Table 1). The first step of the notation system was to analyse the video footage frame by frame by registering all the actions a participant makes, and its outcome, for each role. For positioning not the frequency but the duration of being open or marked was registered. This could easily be done using video coding software like Dartfish (TeamPro 7), which we used.

Table 1.

Actions, outcomes, definitions and allocation of points of notation system.

Role	Action	outcome	Definition	Points
Attacker with ball
	Shooting		The attacker shoots at goal and …
		Goal	… scores	12
		Blocked by defender	… the shot is blocked by a defender	6
		Saved by goalkeeper	… the shot is saved by the goalkeeper	9
		Post/crossbar	… the ball hits the post or crossbar	9
		Wide/over	… the ball goes wide or over the goal (within 1 m)	6
		Far wide/far over	… the ball goes far wide or over the goal (more than 1 m)	0
	Passing		The attacker passes the ball …
		Successful, towards teammate in promising position	… and a teammate in a promising position receives the ball	5
		Successful, forward	… forward to a teammate who receives the ball	2
		Successful, backward	… sideways or backward to a teammate who receives the ball	1
		Intercepted	… and a defender or goalkeeper intercepts the ball	0
		Offside	… towards a teammate in offside position	0
		Out of play	… out of play	0
	Dribbling		The attacker moves the ball, after receiving and prior to passing/shooting (without a near defender) and …
		Maintain ball possession, towards promising position	… the attacker maintains ball possession and moves towards a promising position	5
		Maintain ball possession, forward	… the attacker maintains ball possession and moves forwards	2
		Maintain ball possession, to the side or backward	… the attacker maintains ball possession and moves to the side or backwards	1
		Ball possession lost	… the attackers loses ball possession	0
	Offensive 1:1 duel		The attacker with ball and defender approach within 1 m, the defender is next to or in front of the attacker, and …
		Attacker wins and overtakes	… the attacker wins the duel and overtakes the defender	5
		Attacker retains ball possession but goes back	… the attacker maintains ball possession but does not overtake the defender	3
		Defender plays ball out of play	… the defender plays the ball out of play	2
		Defender wins ball possession and can continue directly	… the defender conquers ball possession and is able to continue to play immediately	0
		Defender wins ball possession but cannot continue directly	… the defender conquers ball possession and is not able to continue to play immediately	0
	Receiving		The attacker receives the ball and …
		Under control	… controls it	1
		Out of control	… does not control it	0
	Foul		The attackers makes a foul	0
Attacker without ball
	Running action		The attacker off the ball accelerates or moves in another direction than the flow of the game and …
		Defender follows, creating more space for ball carrier	… a defender follows the attacker, hereby creating more space for the ball carrier	2
		Got open on own half	… the attacker gets open on his own half of the playing field	2
		Got open on opponent’s half	… the attacker gets open on the opponents’ half of the playing field	4
		Wrong direction/timing	… the attacker does not get open and the defender does not follow him	0
	Offside		The attacker is in offside position	0
	In promising position		The attacker is in a promising position, that is, inside the penalty box and a 2-m wide line from the attacker towards the goal is open	7
	Foul		The attacker off the ball makes a foul	0
	Positioning		A 1-m wide line from the ball carrier to the attacker off the ball is …
		Open, own half, centre	… open, and the attacker off the ball is on his own half of the field and in the centre	2
		Open, own half, side	… open, and the attacker off the ball is on his own half of the field and at the side	1
		Open, opponents’ half, centre	… open, and the attacker off the ball is on the opponents’ half of the field and in the centre	5
		Open, opponents’ half, side	… open, and the attacker off the ball is on the opponents’ half of the field and at the side	3
		Marked	… marked by a defender	0
Defender
	Defensive 1:1 duel		The defender and attacker with ball approach within 1 m, the defender is next to or in front of the attacker, and …
		Attacker wins and overtakes	… the attacker wins the duel and overtakes the defender	0
		Attacker retains ball possession but goes back	… the attacker maintains ball possession but does not overtake the defender	2
		Defender plays ball out of play	… the defender plays the ball out of play	2
		Defender wins ball possession and can continue directly	… the defender conquers ball possession and is able to continue to play immediately	6
		Defender wins ball possession but cannot continue directly	… the defender conquers ball possession and is not able to continue to play immediately	4
	Defensive pressure		The defender accelerates towards the attacker with ball and approaches within 2 m (and more than 1 m) and …
		Attacker goes forward	… the attacker with ball moves forward	0
		Attacker goes backward	… the attacker with ball moves to the side or backwards	3
		Towards 1:1 duel	… the defender approaches to within 1 m and a 1:1 duel follows	2
	Intercepting		The defender intercepts a pass and …
		Under control	… controls the ball	6
		No control	… does not control the ball	2
	Blocking shot		The defender blocks a shot at goal and …
		Defender got ball possession	… the defender gains ball possession	5
		Defender got no ball possession	… the attackers maintain ball possession	2
	Offside trap		The last defender steps forward to put an attacker offside and …
		Well executed	… the defender wins ball possession due to offside	3
		Not well executed	… the timing is not correct and thus the attackers maintain ball possession	−3
	Foul		The defender makes a foul …
		Inside penalty area	… inside the penalty area	−9
		Own half	… on his own half	−6
		Opponent’s half	… on the opponents’ half	−3

Depending on the outcome, the participants earned points for the actions they performed. The allocation of points was a priori determined by soccer experts, and is shown in Table 1. For example, when a player passes the ball towards a teammate, this teammate receives the ball and the pass was directed forward, then the passing player earns two points. Only for positioning a slightly different approach was used, the registered duration in each of the categories of positioning were used to calculate the percentage of time a player spend in each of the categories, and consequently, these percentages were multiplied with the points allocated to each category, as can be found in Table 1. For example, when a player was open, on his own half, in the centre of the field, for 25% of the total time, then this player got 0.25 × 2 = 0.5 points for this category. By adding up the points per trial for each role, and calculating the average number of points a player received per trial, a performance score for each role was computed. There were no minimum or maximum scores, as the performance scores depend on the actions that a player made and on the outcome of these actions.

Data analysis

Validity

In addition to descriptions of the ecological and content validity of the notation system, the concurrent validity and construct validity were calculated.

Ecological validity

Ecological validity reflects the congruency between the constraints during assessment and real-life situations. Using a representative design, in which the task constraints are similar to the natural performance setting, a high ecological validity is achieved.²² Our notation system was applied to 3 vs. 2 + GK small-sided games, this enabled the participants to behave naturally, and thus, with regard to the task constraints of the assessment method the ecological validity of our notation system is high. With regard to the actual soccer game, however, the ecological validity can be improved by assessing the performances of the players while playing 11 vs. 11 on a regular-sized pitch instead of 3 vs. 2 + GK small-sided games. Nevertheless, in comparison with previous research, the assessment method used in the current study is a proper representation of the actual performance environment.

Content validity

Content validity was determined by two experts with over 25 years of experience in coaching soccer at national and international level. They provided feedback on the terms and definitions of the notation system and discussed the allocation of points until consensus was reached.

Concurrent validity

Concurrent validity can be determined by correlating the results of a new measurement technique with a reference criterion that is administered at about the same time.²³ In this study, the head coach^a judged the performances of the players and categorized them as high, medium or low. Categorizations were made for their general performance in the 3 vs. 2 + GK tests and on their specific performances as attacker with ball, attacker without ball and defender. As indication of concurrent validity, Kendall’s tau correlations²⁴ were determined between the categorizations of the coach and the performance scores attained with the notation system.

Construct validity

Construct validity of the notation system was determined by its success in differentiating between the high and low categorized players. Performance scores for the three roles of the high and low categorized players were compared separately using independent t-tests to determine whether the notation system could differentiate between skill level.

Reliability

The reliability of the notation system was determined using intra-observer and inter-observer reliability.

Intra-observer reliability

A total of 75 trials (10% of the complete dataset) were coded twice by the main researcher to determine intra-observer reliability. Hughes et al.²⁵ recommend to use percentage error as indicator of reliability for categorical data and values less than 5% are seen as acceptable. With the exception of positioning, percentage error was calculated for each action and outcome separately, to give insight into the reliability of the separate items. For positioning the duration of being open or marked was registered, and thus the Pearson correlation between the two data sets was determined as reliability score.

Inter-observer reliability

Although the main researcher coded all data, an assistant was also trained for 5 h to use the notation system. After training, a total of 118 trials (16% of the complete dataset) were coded by the assistant to assess inter-observer reliability. The percentage error²⁵ was calculated for all actions and outcomes separately, except for positioning, for which the Pearson correlation between the two coders was assessed.

Simplification of the notation system

As it is labour-intensive to register all actions and outcomes for each role, we also examined whether it is possible to simplify the notation system. For each role, we calculated the average occurrence of each action per player per trial and the percentage of points the players earned with each action in relation to the total number of points they earned for that particular role. We also examined the ability to discriminate the high and low categorized players of each action separately by using independent t-tests. Based on these results, we stepwise excluded actions from the notation system to find a simplified notation system that included as few as possible actions but was still able to differentiate between the high and low categorized players.

Practical applications

For coaches, it is valuable to have an easy method to compare the players to each other and to get an overview of the strengths and weaknesses of each individual player. To fulfil this request, we created two easy-to-read graphs based on the results of the notation system. To compare the performances of the players within a team or group, a graphical representation was created of the performance scores for offence (i.e. the sum of the performance scores for the role of attacker with ball and without ball) and defence of each player. Also, the average group scores were displayed. The individual strengths and weaknesses were explored by calculating the points each participant earned for each action separately. We expressed them as z-scores to facilitate the comparisons between actions and displayed them in a radar graph.

Results

Validity

Concurrent validity

Significant correlations between the categorizations by the coach and the performance scores have been found for general performances, τ = .486, p < .05, attacker with ball, τ = .397, p < .05, attacker without ball, τ = .523, p < .05 and defender, τ = .461, p < .05, indicating that the performance scores as obtained with the notation system were significantly related to the categorizations by the head coach.

Construct validity

In Table 2, the mean and standard deviations of the performance scores for the three roles can be found for the high and low categorized players. The high categorized players obtained significantly higher performance scores with the notation system than the low categorized players in all three roles, all ps < .05, meaning that the notation system can differentiate between the high and low categorized players.

Table 2.

Construct validity test; comparison of high- and low-skilled players, based on categorizations of head coach.

	Performance score
	M (SD)	df	t	p	r
Attacker with ball		7	2.505	0.041	.69
High (n = 6)	4.90 (1.00)
Low (n = 3)	3.39 (0.22)
Attacker without ball		4	3.606	0.023	.87
High (n = 3)	1.56 (0.10)
Low (n = 3)	1.29 (0.08)
Defender		7	3.370	0.012	.79
High (n = 4)	2.08 (0.21)
Low (n = 5)	1.54 (0.26)

Reliability

Intra-observer reliability

Table 3 shows the intra-observer reliability for each action and outcome that was coded frequently in this sample (i.e. more than 5 times). All actions and outcomes were coded with a percentage error within the acceptable 5%, except for running actions, being offside, defensive pressure and intercepting the ball, of which the last two were only slightly above the 5% norm. For positioning, the Pearson correlation between the two data sets was found to be significant, all ps < .001, and ranging from 0.865 to 0.995. Thus, overall the intra-observer reliability was sufficient to good.

Table 3.

Intra- and inter-observer reliability, expressed as percentage error, except for positioning, for which Pearson correlation was calculated.

	Intra-observer		Inter-observer
	Action	Outcome	Action	Outcome
Attacker with ball
Shooting	4.3%	2.2%	5.0%	6.6%
Passing	0.7%	4.7%	1.7%	8.5%
Dribbling	3.3%	4.2%	14.0%	0.8%
Offensive 1:1 duel	3.6%	3.6%	28.6%	7.1%
Receiving	3.2%	2.2%	5.3%	3.9%
Foul	^a		^a
Attacker without ball
Running action	17.9%	6.0%	39.3%	1.4%
Offside	28.6%		30.5%
Promising position	^a		^a
Foul	^a		^a
Positioning
Open, own half, centre	0.969**		0.892**
Open, own half, side	0.995**		0.989**
Open, opponent’s half, centre	0.865**		0.391*
Open, opponent’s half, side	0.932**		0.912**
Marked	0.988**		0.951**
Defender
Defensive 1:1 duel	3.6%	3.6%	19.0%	0.0%
Defensive pressure	6.8%	2.0%	45.9%	3.7%
Intercepting	7.4%	0.0%	9.5%	28.6%
Blocking shot	^a		^a
Offside trap	^a		^a
Foul	^a		^a

was coded too infrequently in this sample to compute reliability score.

p < .05, **p < .001.

Inter-observer reliability

The inter-observer reliability for each action and outcome that was coded more than 5 times in this sample is displayed in Table 3. The percentage error varied from 0.0% to 45.9%, indicating that some items had high inter-observer reliability and others low. For positioning, a significant correlation was found between the two coders.

Simplification of the notation system

Table 4 shows for each role the average occurrence of each action per player per trial, the percentage of points the players earned with that action, and the independent t-test statistics on the performance scores acquired with each action separately. When used separately, only the action defensive pressure yielded a significant difference between the high and low categorized players, the other separate actions could not differentiate the high from the low categorized players.

Table 4.

For each action in each role, the mean occurrence per player per trial, the mean percentage of points earned with that action in relation to the total number of points for that role, the mean and standard deviation of the high and low categorized players and the test of the difference between them.

			High		Low
	M occurrence	M % of points	M	SD	M	SD	df	t	p	r
Attacker with ball
Shooting	0.18	31.0%	1.71	0.65	0.75	0.39	7	2.313	0.054	0.66
Passing	0.72	25.4%	1.15	0.17	1.07	0.13	7	0.628	0.550	0.23
Dribbling	0.56	26.0%	1.23	0.17	1.01	0.04	7	2.183	0.065	0.64
Offensive 1:1 duel	0.12	7.6%	0.35	0.11	0.19	0.16	7	1.821	0.111	0.57
Receiving	0.47	10.1%	0.47	0.09	0.38	0.03	7	1.631	0.147	0.52
Foul	0.00	0.0%	0.00		0.00
Attacker without ball
Running action	0.30	34.7%	0.53	0.07	0.46	0.06	4	1.356	0.247	0.56
Offside	0.14	0.0%	0.00		0.00
In promising position	0.01	2.9%	0.10	0.11	0.00	0.00	4	1.555	0.195	0.61
Foul	0.00	0.0%	0.00		0.00
Positioning		62.4%	0.93	0.05	0.83	0.12	4	1.392	0.236	0.57
Defender
Defensive 1:1 duel	0.18	23.0%	0.51	0.14	0.43	0.11	7	0.871	0.412	0.31
Defensive pressure	1.07	54.6%	1.09	0.19	0.77	0.14	7	2.946	0.022	0.74
Intercepting	0.10	21.5%	0.43	0.23	0.31	0.09	7	0.990	0.355	0.35
Blocking shot	0.02	2.5%	0.08	0.08	0.05	0.05	7	0.709	0.501	0.26
Offside trap	0.00	0.1%	0.00	0.00	−0.01	0.01	7	0.882	0.407	0.32
Foul	0.01	−1.6%	−0.01	0.02	−0.01	0.03	7	0.220	0.832	0.08

As shown in Table 2 (and in Table 5, the first line for each role), the construct validity of the complete notation system is good, meaning that it differentiates the high and low categorized players. To reduce the workload of the system, we examined whether it is possible to simplify the system without losing its discriminating ability. Stepwise elimination of actions from the notation system revealed that by including the three actions shooting, dribbling and offensive 1:1 duel the notation system can discriminate the high- and low-skilled players in the role of attacker with ball (Table 5). For the role of attacker without ball, running action, being in promising position and positioning are necessary and sufficient to significantly differentiate the high from the low categorized players. For defenders, the high and low categorized players can be discriminated by including only the single action defensive pressure.

Table 5.

Possibilities to simplify the notation system and effects on construct validity test for each role. Suggested simplification options are given in bold.

		High		Low
Out	In	M	SD	M	SD	df	t	p	r
Attacker with ball
–	Shooting, passing, dribbling, offensive 1:1 duel, receiving, foul	4.90	1.00	3.39	0.22	7	2.505	0.041	0.69
Foul	Shooting, passing, dribbling, offensive 1:1 duel, receiving	4.90	1.00	3.39	0.22	7	2.505	0.041	0.69
Foul, passing	Shooting, dribbling, offensive 1:1 duel, receiving	3.75	0.91	2.32	0.35	7	2.561	0.037	0.70
Foul, passing, offensive 1:1 duel	Shooting, passing, dribbling, receiving	3.40	0.85	2.13	0.43	7	2.389	0.048	0.67
Foul, passing, receiving	Shooting, dribbling, offensive 1:1 duel	3.29	0.86	1.94	0.35	7	2.544	0.038	0.69
Foul, passing, offensive 1:1 duel, receiving	Shooting, dribbling	2.93	0.80	1.75	0.42	7	2.333	0.052	0.66
Foul, passing, offensive 1:1 duel, receiving, dribbling	Shooting	1.71	0.65	0.75	0.39	7	2.313	0.054	0.66
Attacker without ball
–	Running action, offside, in promising position, foul, positioning	1.56	0.10	1.29	0.08	4	3.606	0.023	0.87
Foul, offside	Running action, in promising position, positioning	1.56	0.10	1.29	0.08	4	3.606	0.023	0.87
Foul, offside, in promising position	Running action, positioning	1.46	0.11	1.29	0.08	4	2.207	0.092	0.74
Defender
–	Defensive 1:1 duel, defensive pressure, intercepting, blocking shot, offside trap, foul	2.08	0.21	1.54	0.26	7	3.370	0.012	0.79
Offside trap	Defensive 1:1 duel, defensive pressure, intercepting, blocking shot, foul	2.08	0.21	1.55	0.26	7	3.332	0.013	0.78
Offside trap, foul	Defensive 1:1 duel, defensive pressure, intercepting, blocking shot	2.09	0.23	1.56	0.26	7	3.027	0.019	0.75
Offside trap, foul, blocking shot	Defensive 1:1 duel, defensive pressure, intercepting	2.02	0.23	1.51	0.26	7	3.027	0.019	0.75
Offside trap, foul, blocking shot, defensive 1:1 duel	Defensive pressure, intercepting	1.51	0.18	1.08	0.17	7	3.690	0.008	0.81
Offside trap, foul, blocking shot, intercepting	Defensive 1:1 duel, defensive pressure	1.59	0.07	1.20	0.19	7	3.822	0.007	0.82
Offside trap, foul, blocking shot, defensive 1:1 duel, intercepting	Defensive pressure	1.09	0.19	0.77	0.14	7	2.946	0.022	0.74

Practical applications

The performance scores on offence and defence are displayed in Figure 2 for each participant. Using this graphical representation, it is easy for coaches to see how the players score in comparison to each other. The best players appear in the top right corner and the weakest in the bottom left corner. The defence specialists (i.e. good in defence, weak in offence) are located in the bottom right corner and the offence specialists (i.e. good in offence, weak in defence) in the top left corner. Several soccer coaches have approved the practical relevance of this graph.

Figure 2.

Individual scores of each participant for defence and offence. The black lines indicate the average scores of the group.

Examples of the individual strengths and weaknesses of two participants are shown in Figure 3. Participant 12 had high performance scores for all three roles, whereas Participant 15 scored low on the roles attacker with ball and defender and above average for the role of attacker without ball. The strengths and weaknesses graphs (Figure 3) reveal that Participant 12 especially excels in passing but may benefit from improving her intercepting skills and although Participant 15 scored on average low on defending, her intercepting skills were above average.

Figure 3.

Individual strengths and weaknesses of Participants 12 and 15, expressed as z-scores. The dashed line indicates the average score of the group.

Discussion

The aim of this study was to take a first step in developing an objective notation system for small-sided soccer games that examines player performances both on and off the ball. The notation system was tested on highly talented female soccer players from the national talent program. Validity and reliability of the notation system were determined, practical applications were shown and a simplified system was proposed to reduce the workload of the complete notation system.

The notation system has high ecological validity as a representative design is used in which the task constraints are similar to the natural performance setting and consequently enables natural behaviour. Assessing the performances of the players while playing 11 vs. 11 regular matches, will even further improve the ecological validity and is interesting for future research. Nevertheless, in comparison with previous research, the method we used to assess performance is a proper representation of the actual performance setting. Furthermore, as two experts with over 25 years of experience in coaching soccer at national and international level contributed to the development of the notation system, the content validity of the notation system was warranted.

The concurrent validity of the notation system was found to be significant for each role and for the overall performance score. However, the correlations between the performance scores and the categorizations by the head coach showed medium to large effects. This could possibly be due to correlating the performance scores with the opinion of one expert instead of a panel of experts. Also the fact that we analysed the small number of 19 players could have affected the results, and furthermore, these players were all enrolled in the national talent program, meaning that they were all highly skilled players and consequently large differences were not to be expected. Applying the notation system on a larger and more heterogeneous skilled group of players will probably yield higher concurrent validity.

Construct validity was determined by comparing the performance scores of the high and low categorized players. In each role, the highly skilled players scored significantly higher than the low categorized players, demonstrating the good ability of the notation system to discriminate the high- and low-skilled players.

The intra-observer reliability was good except for running actions and offside. The inter-observer reliability, however, was good for some actions but low for dribbling, 1:1 duel both offensively and defensively, running action, offside, defensive pressure and intercepting. For most of these, the recognition of the action was found to be more difficult than the determination of the outcome of that action, as the reliability scores of the outcome were more often at an acceptable level than the reliability scores of the actions. The actions that scored low on reliability were all actions that are less objectively identifiable than actions like passes or shots on goal, indicating that improvement in reliability can be expected after clarifying the definitions of those actions. The low reliability of offside is probably due to the fact that it is an item that can be easily forgotten to register and, in addition, the camera’s viewpoint (behind the goal) made it difficult to identify offside. The notation system showed reasonably good intra-observer reliability, but the inter-observer reliability requires more attention. The reliability can be improved by defining the actions and outcomes more clearly and by administering more guided training with the notation system than the current 5 h of practice before starting to assess performances.

Another reason for the low reliability scores may be the complexity of the system, as any actions and outcomes need to be registered. Reducing the workload by eliminating actions from the system may also improve the inter-observer reliability. We found that when for the attacker with ball only the actions shooting, dribbling and offensive 1:1 duel were included, for the attacker without ball running actions, being in promising position and positioning and for the defender only defensive pressure, then the complexity and workload of the notation system were reduced considerably, but its ability to differentiate the high- from the low-skilled players remained.

On the other hand, using specialised camera’s and software that can track the positions of the players and ball²⁶ in combination with specially designed algorithms, the registration of all actions of all players on the field can be automated. An advantage of registering all actions is that it reveals a great deal of specific information about the players, which can be used to create player profiles indicating strengths and weaknesses of each player, as we showed in the practical applications, and these player profiles can be used to evaluate training, to follow the development of the individual players and to set goals for an individualised training program.²⁷

Also, the comparison of the performances of the players within a team is of practical relevance to coaches and scouts. For example, coaches can easily compare players and choose a more offensively or defensively playing midfielder according to their preferred game strategy. For both practical applications that we showed, a benchmark would be of great value. Then players can be compared to age- and gender-matched top-level players. To achieve this, the performances of many players of different age and gender should be assessed with the notation system.

Until now, the notation system has only been used to assess the performances of just 19 players. As these players were all enrolled in the national talent program, and thus preselected on their high skills, large differences in performance among the players were not to be expected. The fact that the notation system was able to discriminate the high from the low categorized players shows the potential of the notation system to assist in talent identification.

Conclusion

The notation system we composed for assessing performances of soccer players in 3 vs. 2 + GK small-sided games seems a good first step towards an objective assessment tool that examines both performances on and off the ball. The notation system differentiates the high- and low-skilled players and had high ecological validity, which may be improved by examining 11 vs. 11 matches. Further development is necessary to increase the reliability of the system and a longitudinal study on the use of the system to assist in player evaluation and selection would be valuable.

Footnotes

Acknowledgements

The authors would like to thank head coach Maria van Kortenhof, the other staff members and the players of the CTO Amsterdam Talent Team and Peter van Dort of the KNVB for their cooperation.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was partly funded by the Royal Netherlands Football Association (KNVB).

Note

References

Bard

Fleury

. Analysis of visual search activity during sport problem situations. J Hum Movem Stud 1976; 3: 22–33.

Helsen

Pauwels

The use of a simulator in evaluation and training of tactical skills in soccer. In: Reilly

Lees

Davids

(eds). Science and football, London: Spon, 1988, pp. 493–497.

Belling

Suss

Ward

. Advancing theory and application of cognitive research in sport: Using representative tasks to explain and predict skilled anticipation, decision-making, and option-generation behaviour. Psychol Sport Exerc 2015; 16: 45–59.

Ripoll

Kerlirzin

Stein

. Analysis of information processing, decision making and visual strategies in complex problem solving sport situations. Hum Movem Sci 1995; 14: 325–349.

Savelsbergh

GJP

Haans

SHA

Kooijman

. A method to identify talent: Visual search and locomotion behavior in young football players. Hum Movem Sci 2010; 29: 764–776.

Vaeyens

Lenoir

Williams

. The mechanisms underpinning successful decision-making in skilled youth soccer players: An analysis of visual search behaviours. J Motor Behav 2007; 39: 395–408.

Dicks

Button

Davids

. Examination of gaze behaviours under in situ and video simulation task constraints reveals differences in information pickup for perception and action. Attent Percept Psychophys 2010; 72: 706–720.

Mann

DTY

Williams

Ward

. Perceptual cognitive expertise in sport: A meta-analysis. J Sport Exerc Psychol 2007; 29: 457–478.

Travassos

Araújo

Davids

. Expertise effects on decision-making in sport are constrained by requisite response behaviours – A meta-analysis. Psychol Sport Exerc 2013; 14: 211–219.

10.

Thomas

. Developing expertise in sport: the relation of knowledge and performance. Int J Sport Psychol 1994; 25: 295–315.

11.

Arias

Castejón

. Review of the instruments most frequently employed to assess tactics in physical education and youth sports. J Teach Phys Educ 2012; 31: 381–391.

12.

Gréhaigne

J-F

Godbout

Bouthier

. Performance assessment in team sports. J Teach Phys Educ 1997; 16: 500–516.

13.

Blomqvist

Vänttinen

Luhtanen

. Assessment of secondary school students’ decision-making and game-play ability in soccer. Phys Educ Sport Pedag 2005; 10: 107–119.

14.

Reilly

Thomas

. A motion analysis of work-rate in different positional roles in professional football match-play. J Hum Movem Stud 1976; 2: 87–97.

15.

Withers

Maricic

Wasilewski

. Match analyses of Australian professional soccer players. J Hum Movem Stud 1982; 8: 158–176.

16.

Oslin

Mitchell

Griffin

. The game performance assessment instrument (GPAI): Development and preliminary validation. J Teach Phys Educ 1998; 17: 231–243.

17.

Teoldo

Garganta

Greco

. Tactical principles of soccer: concepts and application. Revista Motriz 2010; 15: 657–668.

18.

Teoldo

Garganta

Greco

. System of tactical assessment in soccer (FUT-SAT): Development and preliminary validation. Motricidade 2011; 7: 69–83.

19.

García-López

González-Víllora

Gutiérrez

. Development and validation of the game performance evaluation tool (GPET) in soccer. Revista Euroamericana de Ciencias del Deporte 2013; 2: 89–99.

20.

González-Víllora

Serra-Olivares

Pastor-Vicedo

. Review of the tactical evaluation tools for youth players, assessing the tactics in team sports: football. SpringerPlus 2015; 4: 1–17.

21.

Dokter

. The Dutch vision on youth soccer, Zeist, Netherlands: Koninklijke Nederlandse Voetbal Bond, 1993.

22.

Brewer

Crano

Research design and issues of validity. In: Reis

Judd

(eds). Handbook of research methods in social an personality psychology, New York: Cambridge University Press, 2014, pp. 11–26.

23.

Thomas

Nelson

Silverman

. Research methods in physical activity, Champaign, IL: Human Kinetics, 2005.

24.

Field

. Discovering statistics using SPSS, London: Sage Publications, 2009.

25.

Hughes

Cooper

Nevill

Analysis of notation data: reliability. In: Hughes

Franks

(eds). Notational analysis of sport: Systems for better coaching and performance in sport, London: Routledge, 2004, pp. 189–204.

26.

Beetz

Hoyningen-Huene

Kirchlechner

. ASpoGAMo: Automated sports game analysis models. Int J Comput Sci Sport 2009; 8: 1–21.

27.

O’Donoghue

Sports performance profiling. In: McGarry

O’Donoghue

Sampaio

(eds). Routledge handbook of sports performance analysis, London: Routledge, 2013, pp. 127–139.

System for notational analysis in small-sided soccer games

Abstract

Keywords

Introduction

Method

Participants

Procedure

Notation system

Data analysis

Validity

Ecological validity

Content validity

Concurrent validity

Construct validity

Reliability

Intra-observer reliability

Inter-observer reliability

Simplification of the notation system

Practical applications

Results

Validity

Concurrent validity

Construct validity

Reliability

Intra-observer reliability

Inter-observer reliability

Simplification of the notation system

Practical applications

Discussion

Conclusion

Footnotes

Acknowledgements

Declaration of Conflicting Interests

Funding

Note

References