Assessing criterion validity and test-retest reliability of a computer vision-based system for tracking player location in cricket

Abstract

Player tracking is widely utilised to quantify physical and tactical performance of elite athletes, with wearable technologies being the most commonly used systems. However, wearable devices are limited in their ability to capture location data of an opposing team, thereby limiting the collection of complete match information. Alternatively, computer vision-based systems can track all players on the field without the need for wearable devices. This study aimed to evaluate the criterion validity and test-retest reliability of a computer vision-based system for tracking player locations in cricket. Validity was assessed as error in 1) measured distance from a player to a reference location and 2) distance between two players. The reliability of these measurements was evaluated along with system's ability to measure location of a player returning to a specific location multiple times. Validity and reliability were evaluated separately for near-end and far-end of the field relative to camera position. The root mean square error (RMSE) for distance measurements ranged 0.28m–0.49 m (near-end) and 0.82m–1.06 m (far-end). The system marginally underestimated (mean bias: −0.13 m to −0.32 m; homogenous distribution of errors) and overestimated (mean bias: 0.72 m to 0.97 m; heterogenous distribution of errors) measured distances at near-end and far-end, respectively. The system demonstrated excellent reliability, with interclass correlation coefficients (ICC) of 1.00 at near-end and far-end. The technical error of measurement (TEM) ranged 0.05m–0.31 m (near-end) and 0.26m–0.34 m (far-end). These findings support the use of specific computer vision-based system as a valid and reliable tool for capturing player location data to analyse tactical performance in cricket.

Introduction

Cricket performance analysis has traditionally relied on event-based variables, such as runs scored, wickets taken, and fielding actions (e.g., catch taken).^1–4 However, these measures provide limited insight into the spatial aspects of the sport. In sports such as soccer, rugby, and netball, the recent emergence of player tracking technology has offered significant opportunities to capture player positional data and analyse tactics to ultimately enhance team performance. ⁵ Similarly, player location data in cricket has the potential to provide an opportunity for analysis and gain competitive tactical advantage by facilitating more informed strategic decisions by key decision-makers (e.g., coaches and captains). This is particularly relevant for cricket fielding performance, where employing optimal field-setting strategies (e.g., exploiting opposition batters’ weaknesses to minimise their run-scoring and/or dismiss them) can influence match outcome. ⁶ Despite these potential advantages, the adoption of player tracking systems to evaluate tactical performance in cricket remains limited.

In recent years, player tracking technology has advanced rapidly, making its global market highly competitive and valued at approximately USD 7 billion.^7,8 The ability to collect and analyse spatiotemporal data has transformed athlete development, enhanced the sports broadcasting experience for viewers, and revolutionised performance analysis research and practices.^9,10 These systems allow sophisticated measurement of physical performance (e.g., distance covered, speed, sprints) and have also been employed in officiating and injury prevention programmes.^5,11–13 However, another significant application of player tracking lies in the assessment of tactical performance, for example, evaluating scoring efficiency in basketball by analysing spatial distribution of shot attempts and associated point outcomes. Such insights provide a more comprehensive understanding of strategic decision-making during training and competitions.^7,11

The most commonly employed player tracking technologies include global positioning systems (GPS), local positioning systems (LPS), computer vision-based systems (or “optical tracking”), inertial measurement units (IMUs), and radio frequency-based technology.^5,11 These systems provide player location data relative to fixed nodes in a stadium, satellite signals, or calibrated field areas. Computer vision-based tracking systems utilise cameras to capture the entire playing area and report player's x-y coordinates using specialised software. ¹⁴ A notable advantage of computer vision-based system over GPS and LPS is that athletes are not required to wear any technology, which can be a limiting factor as wearable devices can dislodge or misorient during play, resulting in inaccurate data. ¹¹ Furthermore, wearable tracking systems provide data only for the team using the technology, limiting their ability to analyse opposition players’ movement and positional patterns. ¹⁵ Computer vision-based systems overcome these limitations by simultaneously capturing all athletes’ locations without requiring wearable devices, ¹⁶ making them a more comprehensive tool for collecting player location data. Nevertheless, the validity and reliability of such systems can vary based on field dimensions, player density, camera calibration, frame rate, pixel resolution, light and weather conditions, and algorithmic processing, hence their applicability must be critically assessed before implementation.^5,13,15,17

The assessment of player tracking systems is also crucial in sport science research to ensure confidence in the results they produce. Prior research has demonstrated computer vision-based systems are valid and reliable to measure player distance and position in soccer and tennis.^12,14,18 However, the larger playing area in cricket results in increased camera-to-player distances, which may contribute to tracking inaccuracies. ⁵ Despite its widespread use in other professional sports, computer vision-based tracking in cricket has primarily been leveraged for broadcasting enhancements rather than performance analysis. Nevertheless, studies in other disciplines of sport science have indicated that these systems may be used interchangeably with wearable technology such as GPS, IMUs, and marker-based motion capture.^11,19–21 Given the growing need for objective assessment of performance and tactical decision-making in cricket, the current technology presents an opportunity for analysis, particularly in evaluating fielding performance. However, it is crucial to assess the data quality generated by these systems before they can be used by coaches and performance analysts, yet no study has established its validity and reliability. To fill this gap in the literature, the current study aims to evaluate the criterion validity and test-retest reliability of a computer vision-based system that was developed specifically for cricket.

Methods

Study design & setting

This research employed an experimental study design to evaluate a computer vision system. The study was conducted on the playing surface at the Melbourne Cricket Ground (MCG) using the Quidich Tracker (QT) system (Quidich Innovation Labs, Mumbai, India). This proprietary system is used in live TV broadcasts of professional cricket games to enhance viewer's experience through real-time player tracking. The system utilises computer vision technology to identify and track fielder location and generate advanced visualisations. ²² The Quidich Tracker uses one high-resolution video camera operating at 25 Hz, positioned on top of the stadium structure to capture the entire playing area. The camera was calibrated using the standardised pitch length (20.12 m) as a reference. The calibration allows for the conversion of pixel coordinates to real-world measurements, ensuring that each pixel correspond to a defined unit of distance, i.e., 1 pixel = 0.1279 m. Notably, this conversion factor may exhibit marginal variation depending on the specific height at which the camera is mounted at different playing venues. The study received an ethics exemption approval from the Deakin University Human Ethics Advisory Group (2025/HE000204).

Experimental procedures

The experiment was conducted within a 20 × 20 m area (Figure 1.a), using 25 markers placed on the field to create a square grid closer to (near-end) and further from (far-end) the camera (Figure 1.b). The camera was positioned approximately 80 m from the test area at the near-end and 165 m at the far-end. Markers were positioned 5 m apart along both horizontal directions, and the test area was remeasured diagonally to ensure accuracy. All distances were measured using a non-stretch tape measure, a method commonly regarded as a gold standard reference in validation studies. ²³ The lead author and an assistant acted as players in the field. A systematic sampling approach was adopted, where at the start of the trial, both players were positioned at markers located at the maximum diagonal distance from each other within the grid. Their initial positions were recorded using the Quidich Tracker system. Players then moved sequentially to designated markers within the grid, with their locations recorded at each position. The system, operated by staff from Quidich Innovation Labs, captured the two-dimensional coordinates (x and y) corresponding to each instance when a player visited a marker. Both players visited all markers at least once, however, to evaluate system's test-retest reliability, one player revisited a specific marker five times.

Figure 1.

a) Station grid layout, b) Location of the test area.

Data processing and statistical analysis

Staff from Quidich Innovation Labs extracted raw tracking data, consisting of x and y coordinates for both players at each marker, to comma-separated values (CSV) format. The initial version of the raw data extracted was not the final dataset used for the study as it included some incomplete player location information for the far-end. This was subsequently addressed as the Quidich Tracker system was reapplied to the original video, to create a complete set of data that is the subject of this analysis. Data processing and statistical analyses (see details below) were performed using Microsoft Excel ²⁴ and RStudio, ²⁵ and this was conducted independently of all staff from Quidich Innovation Labs.

Criterion validation

The system's criterion validity was assessed for two types of distance measurements, 1) distance from a player to a reference location and 2) distance between two players. Validity was determined by root mean square error (RMSE), mean absolute error percentage and mean bias of the distance measurements. The distribution of these errors was interpreted using Bland–Altman plots. ²⁶

The Euclidean distance from a reference point in one corner of the grid, to each player standing on each grid marker was computed using the recorded x and y coordinates. Given the actual dimensions of the grid and the distances between markers were known, the measured (recorded) distances were compared against the actual distances to determine the error of each measurement (near-end, n = 54; far-end, n = 53).

The Euclidean distance between the two players was computed based on their x and y coordinates. These measured distances were compared with the actual distances to determine the error of each measurement (near-end, n = 26; far-end, n = 25).

Reliability

Test-retest reliability of the system was assessed for distance from a player to a reference location (near-end, n = 25; far-end, n = 25) and distance between two players (near-end, n = 33; far-end, n = 32), as described above. Additionally, it was evaluated using a third type, repeated location measurement (described below). Reliability was determined by intraclass correlation coefficient (ICC) with a single-score rating, and two-way consistency model. ²⁷ ICC values were interpreted as poor (< 0.50), moderate (0.5–0.74), good (0.75–0.90), and excellent (> 0.90). ²⁸ ICC values were determined separately for x and y coordinates for the repeated location measurement method. The statistical measures used in this study are widely employed in research assessing the validity and reliability of computer vision-based systems. ²⁹ A p-value < 0.001 was considered statistically highly significant. ³⁰ Additionally, the technical error of measurement (TEM) was calculated to quantify the degree of repeat measurement error. ³¹

Some grid markers were visited multiple times by the players. For all repeated location measurement to the same marker, the replicate x and y coordinates were compared to evaluate the system's ability to consistently detect location over time (near-end, n = 43; far-end, n = 41).

Results

Criterion validity

The validity of the system's measurements, assessed by comparing measured and actual distances from a player to a reference location, yielded a RMSE of 0.28 m (near-end) and 1.06 m (far-end). The mean absolute error percentage was 1.44% (near-end) and 5.82% (far-end). Bland–Altman plots revealed a slight underestimation with homogeneous error distribution at the near-end (mean bias: −0.13 m; Figure 2), and a consistent overestimation with heterogeneous error distribution at the far-end (mean bias: 0.97 m; Figure 3).

Figure 2.

Bland–Altman plot showing agreement between measured and actual distances from a player to a reference location (near-end), with lower (−0.63) and upper (0.38) limits of agreement.

Figure 3.

Bland–Altman plot showing agreement between measured and actual distances from a player to a reference location (far-end), with lower (0.10) and upper (1.83) limits of agreement.

Relatively similar results were observed for criterion validity using distances between two players approach, with a RMSE of 0.49 m (near-end) and 0.82 m (far-end). The mean absolute error percentage was 2.28% and 4.49%, at the near-end and far-end, respectively. At the near-end, the Bland–Altman plot revealed a consistent underestimation with homogenous error distribution (mean bias: −0.32 m; Figure 4). In contrast, at the far-end, a constant overestimation with heterogenous error distribution (mean bias: 0.72 m; Figure 5) was observed, with error magnitude increasing at greater distances.

Figure 4.

Bland–Altman plot showing agreement between measured and actual distances for difference between two players (near-end), with lower (−1.07) and upper (0.42) limits of agreement.

Figure 5.

Bland–Altman plot showing agreement between measured and actual distances for difference between two players (far-end), with lower (−0.08) and upper (1.52) limits of agreement.

Test-Retest reliability

The system demonstrated excellent test-retest reliability in measuring distances to a fixed reference point. ICCs were 1.00 (95% CI: 1.00–1.00) at both the near-end and the far-end. The TEM was 0.05 m and 0.34 m for the near-end and the far-end, respectively.

The measured distances between players showed excellent repeatability, with ICC of 1.00 (95% CI: 0.99–1.00) at the near-end and 1.00 (95% CI: 1.00–1.00) at the far-end. Corresponding TEMs were 0.31 m and 0.26 m, respectively.

Excellent reliability was observed in both x and y coordinate measurements across repeated visits to the same marker. At the near-end and the far-end, ICCs for both coordinates were 1.00 (95% CI: 1.00–1.00). The TEM was 0.06 m at the near-end and 0.33 m at the far-end.

Discussion

Computer vision-based tracking technology is primarily used in elite-level cricket for TV broadcasting, yet its potential to act as a tool for performance analysis remains largely unexplored. Prior to employing such systems in practice, it is critical to establish confidence in the results they produce. This study is the first to assess the criterion validity and test-retest reliability of a computer vision-based tracking system designed to measure player location in cricket. Validity and reliability of distance measure and static location were assessed at the near-end and far-end of the playing field.

The system was found to have a relatively small distance measurement error, however as measurements were made further away from the camera, the magnitude of the error increased. Relatively higher errors (RMSE and mean absolute error percentage) at the far-end compared to near-end, indicate greater deviations between measured and actual distances. This may be due to optical distortion and parallax-related inaccuracies.^5,32,33 A study in squash reported a similar trend, with errors in static player position being three to four times greater further from the camera, compared to closer to the camera. ³⁴ The difference in error between the near-end and far-end in the current system is approximately half of that observed in squash. The increased error at the far-end may have been amplified, as data collection for this occurred in poorer lighting and weather conditions than for the near-end (see Appendix A). These factors have been shown to contribute to higher measurement error in other studies.^13,15

At the near-end, the system tends to slightly underestimate both types of distance measures, with the mean bias remaining relatively consistent (homogeneous error distribution). In contrast, at the far-end, the system overestimates both types of distance measurements, and the magnitude of the mean bias appears to increase with the magnitude of the measured distances (heterogeneous error distribution). If the constructed limits of agreement fall within acceptable ranges, the system may be considered interchangeable to measure distances with the gold standard, ³⁵ although, minor discrepancies are inevitable. For instance, in Australian Football, GPS and computer vision systems were found to overestimate actual distance covered by 4.8% and 5.8%, respectively. ¹¹ However, in the absence of comparable validation studies in cricket, it is not possible to contrast the current findings with another study. Future studies in similar contexts would enable such comparisons and help establish the generalisability of these findings. Nevertheless, the current study's findings suggest that the present system can accurately track player location and be used as a tool to analyse tactical performance in cricket.

In addition to relatively high accuracy for measurement of distances, the system also exhibited excellent test-retest reliability at both ends of the field, indicated by high ICC values that represent minimal variability across repeated measures. When compared with studies from other sports, the results from this study indicate the current system demonstrates notably higher reliability. Related research in soccer and tennis, which estimated distance covered using computer vision technology has reported ICC values in the range of 0.88–0.93.^12,18 In comparison, the present study's estimation of static player positioning showed nearly perfect reliability at the near-end and the far-end (ICC: 1.00).

While this research advances our understanding of the validity and reliability of computer vision-based system to track players on a cricket field, future studies should address its limitations. The data collection was conducted at one cricket stadium, and depending on stadium structural design the mounting height of the camera (e.g., camera installed on top of the structure vs light towers) can vary between venues. Such variations across venues require different zoom settings and camera calibration, which can potentially introduce minor discrepancies in tracking data. ³⁴ Therefore, conducting similar studies across multiple venues could help enhance the generalisability of these findings. Moreover, to further assess the performance of computer vision-based systems, future research should consider the influence of varying environmental conditions (e.g., light and weather). Future studies should also consider the effect of higher player density, which could be relevant for Test cricket, where fielders are generally positioned closer together. Additionally, previous research has suggested that a multi-camera setup can assist in addressing issues such as player occlusion and improve data quality compared to single-camera setup.^36,37 However, multi-camera setup may be cost and time expensive and adds complexity to data preprocessing and camera synchronisation during setup. ³⁸ Nevertheless, it would be valuable for future studies to compare the validity and reliability of single and multi-camera computer vision-based tracking systems.

Conclusion

This study is the first to evaluate the criterion validity and test-retest reliability of a computer vision-based player tracking system in cricket. The findings demonstrate that the system provides valid and reliable measurements, with minimal error. While the current computer-vision based system is primarily used for live TV broadcasting at the elite-level, the results suggest that it can allow to capture player location information in cricket for performance analysis. The findings allow coaches, captains and performance analysts to be confident in the potential application of the system. Furthermore, the computer vision-based tracking system present new opportunities to explore previously unexamined aspects of tactical performance in cricket, particularly in fielding analysis. Future research should expand on these findings by assessing the effectiveness of such systems across multiple cricket venues and under varying environmental conditions to further establish their robustness and generalisability.