Using consensus methods to standardise judgement-based guidelines required for player management decision-making processes: A case study in professional rugby union

Abstract

Standards are pivotal for generating the evidence required to manage players in professional sport environments like rugby union. Resultantly, using a three-step qualitative approach, this study aimed to formulate a consensus as a subjective standard for evidence generation pertaining to player management. The consensus statement intended to identify evidence on peaks/troughs in player external training loads using Global Positioning System (GPS)-based information in the High-Performance Unit (HPU) of a Gallagher Premiership rugby union club. Initially, a systematic review adhering to the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) framework was conducted to unravel the factors considered (literature-based cues) when identifying peaks/troughs in player external training loads using GPS information. Next, thematic analysis conducted on the data obtained from 7 semi-structured interviews with HPU staff highlighted that they consider 6 factors with 38 elements (practitioner-based cues) during player external training load management. Thereafter, guided by the Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument and by utilising selected elements representing 4/6 factors (healthy player, GPS information, longitudinal durations and practitioner judgements on information), a consensus among practitioners for identifying peaks/troughs in player external training loads was developed with the participation of five HPU members using the nominal group technique (NGT). Practitioners reached an agreement with regard to 12 indicators to subjectively identify peaks/troughs in player external training loads within the considered environment.

Keywords

External training load game strategy global positioning system health informatics periodisation

Introduction

In recent times, there appears to be a significant growth in the use of data for decision-making pertaining to player management across different sport organisations. However, since data has no meaning in isolation,¹ it must transform to its higher-order dimensions like information and evidence to meaningfully support decision-making. As a theoretical construct, Dammann² proposed a health informatics framework that illustrated the unidirectional transitioning between data, information, evidence and knowledge. Importantly, this model suggests that information is created when data is contextualised, information compared to a standard formulates evidence and repeated evidence or the success, and agreement arising from the use of evidence to test hypotheses creates knowledge.² In professional sport environments, practitioners often rely on the evidence generated from different information sources (e.g. medical information, training information) when making decisions relating to player management.³ Research illustrates that such decision-making processes involve the interactions between multiple stakeholders.^4–6 In such contexts, under the absence of adequate standards or research-based guidelines, practitioners engaged in a collective decision-making process may create evidence from information sources based primarily on their own personal biases rather than aligning with holistic organisational objectives.⁷ This in turn has the potential to create noise in decision-making relating to player management.

From an applied viewpoint, in an environment like rugby union, we have illustrated that collective decision-making processes occur when practitioners make judgements regarding player external training load management.⁶ One key outcome of this process is to identify and act upon instances that a player experiences higher than normal (peak) or lower than normal (trough) external training loads.⁶ Such judgements are important for player management as it is believed that poorly managed changes in training load impacts training-induced adaptations and performance.^8–12 Practically, prior research illustrates that the information contextualised from microtechnology data (e.g. Global Positioning System (GPS)) play a key role in creating the evidence needed to manage the external training loads of rugby union players in professional environments.³ In such contexts, from a physiological external training load monitoring perspective,¹³ under the presence of adequate standards, GPS-based information can notify a practitioner when a player experiences an external training load peak or trough. While the final practitioner judgement on whether such evidence truly transitions to knowledge (e.g. likely training-induced adaptations due to repeated peaks) may depend on additional contexts to the decision (e.g. practitioner experiences, internal training load information), such a notification could still help alert potential risks and warrants investigation during player training load management.

Moreover, in a team sport environment like rugby union, the evidence necessary to manage the preparation and performance of an athlete is unique to that specific organisation. That is because each sporting club has its own operational model (e.g. financial strategy, playing style and training philosophy) leading to a distinct set of organisational goals.¹⁴ Accordingly, on a micro-level, the standards required for generating evidence from information sources (e.g. identifying peaks/troughs in external training loads from GPS information) are subjective to the considered sporting environment. Therefore, unless guided objectively by research-based evidence, the absence of a relevant subjective standard (i.e. indicators signifying the collective agreement among practitioners) that aligns with the organisational goals may result in individuals formulating evidence for the existence of player external training load peaks/troughs from GPS-based information based fundamentally on their individual biases and beliefs.

Clinical literature discusses how consensus development methods can organise subjective judgments among practitioners to create judgement-based guidelines for clinical decision-making and to overcome challenges arising due to the lack of research-based evidence.¹⁵ Therefore, consensus development techniques appear to be suitable for developing subjective standards that illustrate the agreement among a group of individuals associated with a collective decision-making process. Moreover, their use has already emerged in sport where consensus statements are being defined as guidelines for managing athletes.^16,17 Furthermore, within clinical and sport literature, Delphi,¹⁸ consensus development conference¹⁰ and nominal group techniques (NGTs)¹⁹ are the primary methods utilised to develop a consensus.²⁰ An aspect common to all these methods is the use of cues (i.e. the factors or dimensions considered by the group when making their decisions) during their execution.¹⁵ Clinical literature discusses three main types of cues: literature-based (derived from existing literature on the topic), practitioner-based (from views of the consensus members) and contextual (factors specific to the environment, e.g. restrictions due to resources).²⁰ Since the cues selected in the consensus question could bias the final outcome, it is hence important that they are selected in a systematic manner prior to a consensus development session.²⁰

Although consensus statements derived as judgement-based guidelines are emerging in sport literature, it is still unclear if practitioners within a professional sport environment can systematically reach a consensus and articulate subjective standards that signify their collective agreement during decision-making. Therefore, this article aims to bridge that gap in the sport literature by exploring the formulation of a consensus as a subjective standard among practitioners within the performance department of a professional rugby union club (signifying the collective agreement aligning with the holistic organisational goals) to create evidence on the existence of player external training load peaks/troughs using GPS-based information.

Methods

Case study design

This study focuses on the external training load management of players in the High-Performance Unit (HPU) of a professional rugby union club competing in the Gallagher Premiership in England. The specific details of the relevant HPU have been presented previously⁶ with outcomes from optimising an information flow necessary for player management.²¹ Furthermore, ethical approval for the current study was obtained from the affiliated university.

Figure 1 illustrates a high-level overview of the steps undertaken to develop the consensus for identifying external training load peaks/troughs from GPS-based information within this case study. Moreover, the Appraisal of Guidelines for Research and Evaluation (AGREE) II statement was used as a reference standard to guide the consensus development process.²²

Figure 1.

High-level overview of the steps conducted in developing the consensus.

Identifying the cues

As specified previously, recommendations on the methods for consensus development when formulating clinical guidelines clearly illustrate that the cues used in the consensus questions could influence the individual and group judgements.^15,20 Therefore, our first task was to articulate the three types of cues (literature-based, practitioner-based and contextual) that could be utilised by the practitioners when developing a consensus to identify peaks/troughs in the external training loads experienced by rugby union players.

Literature-based cues

We first conducted a systematic review adhering to the 2009 Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) statement to extract any literature-based cues relating to the considered consensus development.²³ For conducting the systematic review, in relation to the aims of the study, we framed the question as: ‘In sport literature, for field-based invasion team sport athletes, how are peaks and troughs in the training loads relating to injury risks, performance adaptations, wellness, fatigue, and illness identified using the information derived from Global Positioning System (GPS) based metrices?’.

Using Boolean operators, articles published from the year 2000 to September 2021 were extracted from EBSCO (Academic Search Complete, SPORTDiscus, MEDLINE and CINAHL Complete), PubMed and Scopus databases. Titles and abstracts of the retrieved papers were independently reviewed by two authors (JR and MZ) against the specified inclusion and exclusion criteria (refer to Supplementary material 1) and the articles not meeting these requirements were excluded prior to full-text review. Selected full-text studies were further analysed against the inclusion criteria and a search of references within the selected studies were conducted to ensure that all relevant articles for the review were included. All articles were assessed for quality independently by authors JR and MZ based on the quality index proposed by Downs and Black²⁴ for randomised and non-randomised studies. A consensus was reached regarding any disagreements between the two authors relating to the study selections or quality assessments after discussions. Data synthesis was conducted by author JR and corresponding results were discussed among the authors prior to making inferences. All information on the search terms used, inclusion criteria and detailed results from the review have been provided separately in Supplementary material 1. From the findings, any relevant literature-based cues for the consensus question were identified. Additionally, as per recent expert opinions for consensus development in sports,²⁵ the systematic review results were shared with the consensus group 1 week prior to the first consensus development session.

Practitioner-based and contextual cues

From a practical viewpoint, practitioners consider different factors (cues) when making decisions relating to player training load management. Such cues or factors form the different contextual layers that bind a final decision relating to the external training load management of rugby union players within a considered environment. Hence, author JR conducted semi-structured interviews (each less than 30 minutes) with all HPU staff members (three physiotherapists, three strength and conditioning coaches and one sports scientist) responsible for the players’ external training load management (during the timeframe of the study) within the considered environment to identify factors they considered during decision-making. Details of the interviewees have been outlined in Table 1. The relevant open-ended questions used in the semi-structured interviews have been provided in Table 2. They were specifically designed to extract details on the practitioner interaction with GPS-based information during player external training load management.

Table 1.

Characteristics of the High-Performance Unit (HPU) members interviewed to extract practitioner-based cues and participated in the consensus development sessions.

User ID	Age	Years of experience in professional sport	Interviews	Consensus development
U₁	35	8	Y	Y
U₂	39	14	Y	Y
U₃	27	5	Y	Y
U₄	31	6	Y	Y
U₅	46	12	Y	N
U₆	29	8	Y	Y
U₇	31	4	Y	N

Table 2.

Questions used for the interviews with High-Performance Unit (HPU) staff.

S. No.	Key question
1	Would you consider managing the external training load of a player from an injury risk mitigation perspective, performance adaptation view or both?
2	As a practitioner, what would concern you more, a player exposed to an external training load peak or a trough?
3	Which is easier, managing the external training load of an injured player or that of a healthy player?
4	Would you make inferences on player external training load management from individual GPS metrics (e.g. number of VHSRs) or from the interaction between multiple metrics? And could you please explain which of those measures you would use frequently?
5	Are there specific longitudinal durations you may consider (e.g. previous 2 weeks) when making inferences relating to player external training load management from GPS-based information?
6	Would there be instances that you wouldn’t rely on GPS-based information when making decisions relating to player external training load management?
7	Do you think there can be noise in decision-making due to the different ways in which a practitioner can generate evidence from GPS-based information?

GPS: Global Positioning System.

Interview sessions were audio recorded and transcribed verbatim by the lead researcher prior to data analysis. For data analysis, the six-phase thematic analysis technique introduced by Braun and Clarke²⁶ was implemented to unravel the specific factors (as themes) that practitioners consider during player external training load management. For theme development, the lead author first developed semantic and latent codes from each interview transcript. The developed coding structure was evaluated independently by the authors DW and GR and agreement was reached between them and the lead author prior to the theme development stage. Next, based on the inherent features of the codes and guided by an inductive approach, lower-order themes depicting elements of the different factors were unravelled from the developed codes. Thereafter, those lower-order themes were grouped to form the final higher-order themes depicting the high-level factors that practitioners consider during player external training load management. Moreover, the developed themes were validated independently by authors DW and GR.

In the current study, no separate contextual cues were defined as there were no separate case study-specific factors (e.g. resource limitations) that needed consideration for the consensus development. Furthermore, guided by existing literature, a list of definitions (Supplementary material 2) relating to the study was developed based on the agreement between the authors, which were shared with the consensus group prior to the consensus development sessions.

Developing the subjective standard (consensus)

Methods for developing consensus

The selected cues were used to define the consensus questions needed to develop the subjective standard (consensus) to identify peaks/troughs in player external training loads using GPS information. As justified previously, the aim of this study was to develop a consensus statement that was subjective to the case study organisation. Consequently, it was impractical to invite external experts for consensus development as they would not be aware of the operational dynamics and wider organisational objectives of the considered environment. Therefore, the aim was to develop the subjective standard from the expertise within the considered club such that it represented the collective agreement of the HPU. Moreover, among the three consensus methods introduced previously, Nominal Group Technique (NGT), which focuses on obtaining consensus among a group of individuals using face-to-face meetings^19,27 was well suited for the current study as it enabled a rigorous dialogue and equal contribution between practitioners during consensus formulation. Therefore, NGT was selected as the method to develop the consensus. Furthermore, based on the guidelines of senior management team at the club, five HPU members (Head of Medical, Head of Strength and Conditioning, Head of Sports Science, First Team physiotherapist and Senior Academy Strength and Conditioning Coach) from the seven individuals previously considered for the semi-structured interviews (representing the expertise within the club relating to the consensus question) were selected for the consensus development. The relevant participant characteristics have been provided in Table 1.

Determining consensus

In the current study, a dichotomous scale (agree/disagree) was used to rate the agreement or disagreement for an indicator by each member of the consensus group. We utilised guidelines available in content validity (CV) research when defining the exact criteria for reaching consensus.²⁸ Similar findings from CV research have also been used previously by sport researchers when developing consensus statements related to sports.¹⁷ Hence, aligning with the guidelines by Lynn,²⁸ for the current study, consensus for an indicator relating to each question was reached if all five members voted to either agree or disagree with it.

Conducting the nominal group sessions

Author JR led the nominal group meetings (relevant steps are outlined in Figure 2), and each session lasted a maximum duration of 2 hours and four such sessions were used to develop the subjective standards. The process was repeated separately to derive two independent consensus statements (i.e. for peaks and troughs). Initially, adhering to a time limit of 30-minutes, the five participants silently recorded the indicators (short phrases) that would illustrate a peak/trough in the player’s external loads from GPS-based information. Subsequently, based on a round-robin format, the participants recorded the generated indicators on an online Microsoft Excel sheet. By considering one indicator at a time, the members then discussed them in detail and any common indicators among the recorded ones were grouped. The host (JR) intervened to move to the next item once the conversation regarding an indicator reached saturation (i.e. no more comments from the consensus team). Following the discussions, author JR recorded the indicators on a Google Form and the members used it to perform the first round of ratings. The indicators reaching consensus (all members agree or disagree) after round one rating were directly added to the final list and the ones not reaching consensus were taken back into the discussion. A further round of discussions and voting was conducted on the indicators not reaching consensus from the initial round. Resultantly, the indicators reaching consensus after round two were added to the final list If there were any items still not meeting consensus, based on a collective discussion with the nominal group, they were either modified, rejected or accepted to generate the finalised consensus statements.

Figure 2.

Nominal group session steps.

Results

Literature-based cues (systematic review)

Study selection

In the next sections of the results, we will only discuss the findings from the systematic review that directly relate to our research question. However, we have provided full details of the accumulated results from the systematic review in the Supplementary material 1. The PRISMA flow diagram shown in Figure 3 illustrates how the final list of 22 studies included for results synthesis were streamlined from the 3265 articles identified through database and reference list searching. Initially, there was 96.9% agreement between authors JR and MZ following initial screening of titles and abstracts. Additionally, any conflicts arising during the study selection were resolved after discussions between JR, MZ and GR.

Figure 3.

Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) flow diagram of systematic review.

Quality of selected studies

The finalised mean quality rating of the 22 studies included in the review from the 2 authors was 15.09 ± 1.66 out of the 32 possible points (range: 13–18). According to the quality ranges used by Hooper et al.,²⁹ 12/22 studies can be rated as ‘Fair’ (15–19 score) and the rest (10/22) would receive a ‘Poor’ (≤ 14 score) rating. The complete details of the quality ratings are available on Table C in the Supplementary material 1.

Study aims and outcome measures

From the selected articles, 18/22 evaluated the association between external training loads quantified using GPS-based metrics and injuries (overuse, non-contact, soft tissue, time loss and preventable). Similarly, a further 3/22 articles assessed the relationship between external training loads and performance adaptations like changes in aerobic and anaerobic fitness. Only one study aimed at using prediction as an outcome. None of the selected studies had evidence on identifying external training load peaks/troughs associated with player wellness, fatigue and illness. Furthermore, only one study was focused on rugby union and the others based their experimentation on football (8/22), Australian football (8/22), rugby league (4/22) and Gaelic football (1/22).

The use of GPS metrics and statistical measures

Most instances (13/22) of possible peaks/troughs in player external training loads were illustrated from the total distance metric with absolute (e.g. high-speed running distance, sprint distance) and relative GPS-based measures (e.g. number of exposures > % maximum velocity, time > maximum aerobic speed) also used. In such instances, it was common across most studies (18/22) to transform the relevant GPS metrics to illustrate cumulative loads of players/squads across 1-2-3-4 weeks and 3/22 articles compared loads across pre-season versus in-season time points. Within the studies, the shortest duration considered for a cumulative external training load was 3 days and no study evaluated the impact of weekly loads beyond the previous 4 weeks. However, within the articles, no justifications were provided for considering specific time windows in the data for the assessments.

The relevant indicators signifying potential peaks/troughs in player external training loads relating to injury risks and performance adaptations were articulated based on statistical methods like hypothesis tests, analysis of variance, linear, multiple and logistic regression, generalised additive models, generalised linear models and generalised estimating equations. Moreover, when association was the statistical aim (i.e. to determine the relationship between external training load measure/s and an outcome such as injury), it was possible to assess the strength of evidence on peaks/troughs generated from the latter statistical models based on outcome statistics like p-values, correlation coefficients, effect size, odds ratios (ORs), relative risks (RR), hazard ratios (HRs) and incidence rate ratios (IRRs) utilised within the studies. Similarly, prediction results were assessed using statistical processes such as area under the receiver operating characteristic curve (AUC-ROC), sensitivity, specificity and likelihood ratios.

For instance, to signify a peak, the results from a generalised estimating equation illustrated that there is a very likely harmful effect on time-loss overuse injuries if the 2-week cumulative total distance was greater than 59.185 m (OR = 2.25, 90% CI = 1.17–4.34) for a sample of 35 professional football players in the Netherlands.³⁰

Extracting literature-based cues from the systematic review results

The study designs of articles examined in the review and their corresponding statistical outcomes were not consistent across the different studies. This made it challenging to relate the findings to the case study environment considered in the current article. As an example, an outcome specifying that 3-weekly sprint distances >1.453 m is associated with a greater injury risk when compared with <864 m (OR = 3.667, 95% CI = 0.88–15.21, p = 0.074) for Australian footballers, becomes challenging to interpret as the strength of evidence is defined in relative terms.³¹ Therefore, it wasn’t possible to define specific indicators for identifying peaks/troughs in player external training loads using GPS-based metrics from the literature. However, it was evident that longitudinal durations considered in cumulative external training load information (e.g. 1-2-3-4 weekly loads) was a key factor (i.e. utilised in majority of the studies) considered in the articles. In further two studies, loads were compared across training versus matches and adjustments in load due to previous injury history were also factored. However, because time was the most frequently used factor in the studies, longitudinal durations considered for cumulative GPS information was identified as a literature-based cue for the consensus question.

Practitioner-based cues

As shown in Figure 4, the results from the thematic analysis illustrated the existence of 6 different cues or higher-order themes (player categorisation, periodisation and game strategy, time factors, information factors, player and practitioner judgements and organisational factors), constituting 38 elements (lower-order themes) within them that HPU practitioners consider when making decisions relating to player external training load management. For illustration, Table 3 presents how the player categorisation theme (cue) was identified from the raw data. The description of each factor is specified below with examples from selected elements.

Figure 4.

Factors considered by the practitioners during player external training load management in the considered environment.

Table 3.

Thematic analysis illustration.

Raw data	Lower-order theme (cue element)	Higher-order theme (cue)
Some players do not survive the full program	Player's tolerance to load	Player categorisation
Players completing full session as a reference
Players have different tolerances to load
Greater risk mitigation for less tolerable athletes
Injured players load much different to their baseline	Injured versus legacy (i.e. players under observation at the initial stages of returning to full squad training after rehabilitation due to an injury) versus healthy players
Return-to-play (RTP) goal is to achieve load close to a healthy player
RTP players don't have load baselines
RTP players have more chaotic load exposure than fit players
Healthy players have a training load cover
Healthy players training load goal is performance
Managing load of injured players is more organised
RTP player load easier to manage
Legacy players need exposure close to normal performance demands
Player sensitivity to load depends on positional group	Based on positional groups
Playing positions have different load requirements
Load analysed by playing position
Academy players training load goal is performance	Academy versus first team
Academy players in travelling reserve is not frequent
Performance benchmarks from first team
Young players exposed to repetitive load creates risk of injury
Main squad player load harder to manage
Need to be aware about managing training loads of travelling reserve players	Matchday versus travelling reserve versus non-23 player
Travelling reserves greater possibility to have training load troughs
Important to manage travelling reserves load troughs
Academy players in travelling reserve is not frequent

Factor 1: Time

Different time factors were considered by practitioners during player external training load management. Elements like the turnaround time after games, longitudinal durations considered for analysis (e.g. acute:chronic) and time allocated for decision-making were some of the time-based criteria factored by the practitioners.

“Troughs in the sense of like, have we managed to achieve a sprint exposure or multiple sprint exposures this week, yes or no, and if we're seeing consistently the player is going week to week without getting into that sprint zone, erm, and therefore potentially undertraining.”

Factor 2: Information

Practitioners considered different information sources like GPS information, player age, injury history, resistance training information, etc., when making inferences relating to player training load management. For instance, a practitioner highlighted the factoring of a player's age, training load and injury history information as stated below.

“If they report something subjectively, their age to training history, their injury history is massive on that. Yeah, particularly like, take for example a young guy and he's been put into a position. Say for example Player B, he was forced six games on the bounce and played all the games and he's got quite a high volume. He is more susceptible than someone who is three or four years a senior and has played a lot more. Do you know what I mean, erm, and then conversely if a guy has high injury history, tends to get (cranky) after a certain amount of time from a physical perspective.”

Factor 3: Player and practitioner judgements

This theme highlighted how the judgements (cognition) of practitioners and players were considered when making inferences relating to external training load management. For instance, the following highlights how the expertise of practitioners to derive evidence from an information source was factored during decision-making.

“It's also based off the experts that are surrounding me. So, there are experts who are looking at it more so than what I do on a daily basis and understand so much more and they feed it to me.”

Factor 4: Organisational or environmental

This theme signified the different organisational or environment-based factors that were considered during decision-making relating to player external training load management.

“Not everybody within a decision-making process at times has an equal weighting within the decision and equal understanding of the context around the decision, which might change or an equal understanding of actually the intricacies of the decision.”

Factor 5: Player categorisation

Practitioners categorised players based on different criteria and factored the resulting categories when making inferences relating to player external training load management. For instance, staff considered the effects of load in relation to the playing position of an individual or had greater caution about the training load exposures of travelling reserve players when making decisions.

“Some of our back threes are inherently low sensitive. So, to the athlete, we may know that their time at VHSR (very high-speed running) or their time in sprint is more indicative.”

Factor 6: Periodisation and game strategy

The following factor highlighted the importance of considering specific conditions around the periodisation strategy at the club as well as tactical criteria corresponding to games. For example, staff considered periodisation objectives defined for the program and team selection criteria when making decisions regarding player external training load management.

“We are starting to figure out what our program gives us and what the coaches want from the program. Therefore, we need to prepare the players to survive in that and not just survive but thrive.”

Finally, as presented in Figure 5, four cue elements (healthy player, GPS information, longitudinal duration and practitioner judgements on information) corresponding to their respective high-level cues (player categorisation, information, time and player and practitioner judgments), that can be used to directly extract evidence from GPS information, were considered to define the below question for the consensus development.

Figure 5.

Cues considered for defining the consensus question.

‘By considering only current available Global Positioning System (GPS) based metrics, what specific information conditions in them (using preferred comparative views of longitudinal periods) would indicate a peak or trough in the external training load experienced by a healthy rugby union player (do not consider injured or legacy players) in the regular training program?’

Consensus statements from the nominal group sessions

Subjective indicators to identify external training load peaks from GPS information

The 5 nominal group members initially generated 15 indicators to identify peaks in the external training loads from GPS-based information. Following the initial round of discussions and voting, the practitioners reached a consensus on seven indicators. Group consent was reached on the remaining items after the second round of discussions and voting. After grouping similar indicators, the members reached a consensus on nine items. Specifically, six indicators from the nine (Table 4) illustrated conditions on the GPS information to subjectively identify a peak in the external training loads experienced by a healthy player in the regular training program.

Table 4.

Indicators reaching consensus by nominal group members to identify a peak in player external training loads using Global Positioning System (GPS) information. Relative thresholds for the metrics: HSR (distance covered between 60% of V_max [highest velocity recorded by the player]) and 75 of % V_max), VHSR (distance covered between 75% of V_max and 90% of V_max) and sprint (distance above 90% of V_max).

S. No.	Indicator	Consensus
1	Acute (1 week) total volume load of (a) total distance, (b) running distance (> 2 ms⁻¹), (c) high-speed running (HSR) distance, (d) very high-speed running (VHSR) distance and (e) sprint distance is greater than 30% of the average weekly totals of previous 4-week volume loads.	Agree
2	For any volume metric (i.e. total distance, running distance, HSR distance, VHSR distance and sprint distance) on a given day, obtaining a value greater than 30% of the average of similar, comparable days (e.g. comparing training days designed for speed outcomes) during the previous 4-week period.	Agree
3	Recording sprint events on 3 or more days or on 2 consecutive days during a rolling 7-day period.	Agree
4	Very high-speed running or sprint events produced when no VHSR or Sprints events were recorded during previous more than 1-week period.	Agree
5	For a consecutive 3-week period, a continual 10% increase in the weekly total of (a) total distance, (b) running distance (> 2 ms⁻¹), (c) high-speed running (HSR) distance, (d) very high-speed running (VHSR) distance and (e) sprint distance.	Agree
6	When the weekly change of (a) total distance, (b) running distance (> 2 ms⁻¹), (c) high-speed running (HSR) distance, (d) very high-speed running (VHSR) and (e) sprint distance is greater than 30% of their previous week total.	Agree
7	For a player, identifying a peak by considering all weekly data of a chosen metric using a pattern recognition algorithm.	Disagree
8	Relative to the team data, a volume or intensity greater than the mean or 1SD completed by similar positions within same session or week.	Disagree
9	For any metric on a given day, obtaining a value equal to or greater than the average weekly total of that metric (e.g. average weekly total VHSR of 200 m being recorded in a single training day).	Disagree

Subjective indicators to identify external training load troughs from GPS information

Similar to the above, nominal group members initially listed 18 indicators to identify a trough in a player's external training load using GPS information. After grouping similar items, eight indicators were considered for first-round discussions and voting. The group reached a consensus on five items following round 1 and after another round of discussions and voting, group consent was reached on the remaining three indicators. Hence, six of the eight indicators (Table 5) signified a condition to identify a trough in player external training loads using GPS information.

Table 5.

Indicators reaching consensus by nominal group members to identify a trough in player external training loads using Global Positioning System (GPS) information. Relative thresholds for the metrics: HSR (distance covered between 60% of V_max and 75% of V_max), VHSR (distance covered between 75% of V_max and 90% of V_max) and sprint (distance above 90% of V_max).

S. No.	Indicator	Consensus
10	Acute (1 week) total volume load of (a) total distance, (b) running distance (> 2 ms⁻¹), (c) high-speed running (HSR) distance, (d) very high-speed running (VHSR) distance and (e) sprint distance is less than 30% of the average weekly totals of previous 4-week volume loads.	Agree
11	For any volume metric (i.e. total distance, running distance, HSR distance, VHSR distance and sprint distance) on a given day, obtaining a value less than 30% of the average of similar, comparable days (e.g. comparing training days designed for speed outcomes) during the previous 4-week period.	Agree
12	Less than 3 on-feet days within a 7-day rolling period.	Agree
13	No very high-speed running (VHSR) or sprint events produced within 1 week.	Agree
14	For a consecutive 3-week period, a continual 10% decrease in the weekly total of (a) total distance, (b) running distance (> 2 ms⁻¹), (c) high-speed running (HSR) distance, (d) very high-speed running (VHSR) distance and (e) sprint distance.	Agree
15	When the weekly change of (a) total distance, (b) running distance (>2 ms⁻¹), (c) high-speed running distance, (d) very high-speed running distance and (e) sprint distance is less than 30% of their previous week total.	Agree
16	Below 50% very high-speed running (VHSR) distance in any one week in comparison to previous 4-week average.	Disagree
17	50% or greater reduction in weekly (a) total distance, (b) running distance (> 2 ms⁻¹), (c) high-speed running (HSR) distance, (d) very high-speed running (VHSR) distance and (e) sprint distance over two consecutive weeks in comparison to previous 4-week average.	Disagree

At first glance, it may be tempting to scrutinise the use of percentages in the indicators corresponding to the consensus. Hence, we would like to restate that our objective was not on unravelling a set of indicators to identify peaks and troughs in player external training loads using GPS information which may be generalisable across sporting organisations. Instead, the study goal was to present the outcomes of our attempt to establish a consensus relative to the considered environment such that it illustrated the collective agreement among practitioners in the HPU when identifying peaks and troughs from GPS-based information. Consequently, the percentages used in the study illustrated the current subjective opinions of the HPU practitioners. We thus encourage readers not to blindly consider such percentages and use them without validating in their respective sporting environments. Moreover, the choice of percentages in the consensus items with comparisons between time windows (indicators 1, 2, 5, 6, 10, 11, 14 and 15) were articulated by the practitioners based on their interactions with a web-based R Shiny application developed by author JR. That interface was designed to contextualise (graphically) the different GPS information percentages defined in the indicators. The 5 nominal group members interacted with the designed data visualisation during an additional 1-hour meeting and reached an agreement on the relevant percentages.

Discussion

This study explored the possibility of articulating a consensus as a subjective standard among HPU staff for identifying peaks and troughs in player external training loads using GPS-based information at a Gallagher Premiership rugby union club. First, adhering to the PRISMA framework, a systematic review was conducted to extract details on the existing evidence on how peaks and troughs in player external training loads have been identified using GPS-based information. Resultantly, longitudinal durations considered for cumulative GPS information was identified as a literature-based cue from the systematic review. Second, practitioner-based cues or the factors considered by staff when managing external training loads of players in the considered environment were identified using thematic analysis conducted on the data extracted from seven semi-structured interviews with HPU staff. There were six core factors or cues (player categorisation, periodisation and game strategy, time, information, player and practitioner judgements and organisational factors) comprising 38 elements that were considered by HPU staff when managing the external training loads of the players (Figure 4). Third, guided by the AGREE II statement and utilising selected cue elements, a consensus for identifying peaks and troughs in the player external training loads were developed using a nominal group with five HPU members. The finalised consensus resulted in generating 12 indicators (peaks = 6 and troughs = 6) to subjectively identify peaks and troughs in the player external training loads using GPS-based information in the considered environment (Tables 4 and 5).

Evidence from the literature

There is growing scepticism by researchers on how GPS-based training load information is utilised in current sport literature for evaluating relationships between training load, injury and performance.^32–34 Those research assessments were a reason that we did not consider evidence from the identification of external training load peaks/troughs using measures like the acute:chronic workload ratio (ACWR).³⁵ Instead, for the systematic review, we focused specifically on reverting to the first principles by examining studies that identified peaks/troughs directly from GPS-based external training load metrics. Interestingly, it wasn’t possible to extract specific research-based evidence to identify external training load peaks and troughs using GPS information. The reasoning for not articulating direct indicators from the review aligned with the ones observed previously.³⁴

However, it appears that the real problem in using GPS information for player external training load management, decision-making is not with the metrics themselves, but rather on what question that information aims to answer and consequently, the nature of how that information is used in practice. From an applied perspective, practitioners would likely still look to use GPS information as a feedback pathway during player external training load management since it's the best technology at their disposal in current contexts and also because technological developments occur as a continuously improving process. There is thus large scope for the development of judgement-based guidelines within organisations in this specific area of practice due to the lack of research-based evidence to guide practitioners in decision-making.

Decision layers and data-informed decision-making

Our focus in the current study was to concentrate on a single layer of the overall external training load management decision corresponding to identifying external training load peaks/troughs using GPS information. However, as presented by the proceeding practitioner statement obtained from the interviews and illustrated further using Figure 6, the elements of the unravelled factors may integrate through various sequences due to different contexts, to create the other layers of the final training load management decision.

“You evaluate the sport first, and then you use every other tool at your disposal to evaluate what's happened within that. So, when we have a rugby session, we evaluate, did we see what we wanted to see live and you chat to coaches. (Yeah), my perception of that was (and at that moment and this is my thoughts on it). When you look back at the video, does the video confirm or tell us something different to what we actually see. Then we start to (work) that out and then we highlight the positives and negatives within the (backroom). We start to use and utilise different data and GPS is one of those.”

Figure 6.

An example of how the different factors associated with a player training load management decision are layered during execution.

Interestingly, the above statement specifies that within the considered environment, evidence generated from GPS information was only acting as a single contextual layer of the overall external training load management decision. This highlighted that information articulated from GPS data was used to inform or verify a practitioner's judgement within the club rather than drive their decision-making. We believe that this emphasises an important philosophical construct for using data in decision-making relating to player management and should warrant further scientific investigation by researchers. That is because there is a fundamental difference in having data-driven versus data-informed decision-making processes. Specifically, in an industry like manufacturing, data can indeed drive decisions (data-driven) since most processes tends be automated. However, player management workflows in sport are more human-oriented than automation driven. In such latter contexts, a practitioner is the key decision-making entity. Therefore, data should be used to inform a practitioner (data-informed)³⁶ or in more scientific terms, evidence generated from data (e.g. existence of an external training load peak identified from GPS information) can be used to test the hypothesis for falsifying the practitioner's belief .³⁷ Moreover, if the evidence generated from data does indeed manage to disprove the practitioner opinions, new knowledge may be generated relating to player management in the considered environment.

Consensus indicators and their implementation

It was evident from the results that the NGT was well suited to articulate a consensus as a subjective standard by practitioners to organise judgment-based guidelines for decision-making within the considered environment. Examining the identified consensus indicators illustrated that the practitioners in the case study environment considered 1 week and 4 weeks as adequate acute and chronic time windows respectively during player external training load management. It thus appears that the choice of such time periods may have been influenced by the practitioner judgements on the systematic review shared with them prior to the nominal group sessions. This is because the chosen time windows align with those used previously in the literature. However, since the study doesn’t provide concrete evidence to assess such a statement, we will leave it as a hypothesis at this stage to provide a basis for a potential future qualitative research exploration. Specifically, to evaluate how practitioners chose such time windows relating to player training load management during decision-making. Moreover, the indicators utilising percentages demonstrated that practitioners still perceived player training load management from ratio-based perspectives, especially relating to training volume.

From an applied viewpoint and in reverting to Dammann's² hierarchy, the 12 consensus indicators with agreement can be implemented in future as a subjective standard to generate evidence during collective decision-making processes in the HPU. This can be achieved by integrating the indicators into a data visualisation interface³⁸ by utilising a colour coding scheme³⁹ to signify a peak/trough in player external training loads. Furthermore, the guidelines by Cole and Altman,⁴⁰ which specify the use of natural logarithms for dealing with percentage differences could be used in the comparative percentage-based indicators to ensure that the comparisons are symmetric and additive. We wish to explore such an integration in the future. However, while the derived consensus could be implemented as a standard within the considered environment, further research explorations are required prior to a full-scale rollout. This is because the indicators must be improved by adding objectivity (e.g. time series analysis), mainly around the usage of time windows, percentages and ratio-based measures. Furthermore, although consensus methods may capture collective knowledge, they are equally capable of capturing collective ignorance.¹⁵ Therefore, practitioners need to be mindful of such limitations that may exist when developing consensus statements constrained to the expertise within a considered environment. Resultantly, at present, we are quantitatively evaluating the impact that evidence generated from subjective standards like the ones articulated in the current study have on the agreement among practitioners engaged in collective decision-making processes. Thus, examining their potential to reduce decision noise while also validating if such evidence truly transitions to knowledge based on repeatability and agreement arising due to the outcomes of hypothesis tests pertaining to practitioner beliefs.

Study limitations

The subjective standards in the current study were developed in relation to the considered case study environment by utilising expertise within the HPU. Therefore, the consensus statements were articulated to meet the objectives of the given environment. Accordingly, although the methods can be generalisable across other case study organisations, the indicators themselves may not be directly applied to similar settings without validation. The objective of this study was to determine if there was consensus for specific indicators defined by the nominal group members, therefore, to reduce complexity, a dichotomous scale was used. However, if there are requirements to assess the level of consensus, ordinal measurements like Likert scales can be adopted for the voting.

Conclusion

When managing the external training loads of rugby union players in a case study environment, due to the absence of standards, practitioners can generate evidence from information sources based on their personal biases and beliefs. In this study, as the first step to combat noise in decision-making due to such subjectivity, we developed a consensus (signifying the collective agreement among staff) for identifying peaks and troughs in player external training loads using GPS-based information in the HPU of a professional rugby union club in England. The results indicated that practitioners were capable of utilising existing expertise within a case study environment to generate a consensus using the Nominal Group Technique. Moreover, a fundamental organisational objective of change management initiatives like the one discussed in this article is to create a culture for continual improvement. Therefore, as the next step, after incorporating necessary objectivity (e.g. mathematical and statistical analysis to replace percentages) into the indicators, the consensus statements can be integrated as standards into a data visualisation interface to create subjective evidence when managing player external training loads. Since our focus in this study was on a specific case study environment having unique organisational objectives, we do not wish to generalise all the results. Instead, we invite other researchers to utilise the methodological procedures in this article to implement similar systems in their respective environments and unravel the aspects which may be generalisable across organisations. Specifically, the methodological sequence we have adopted could be tested to introduce subjective standards to any other collective decision-making process associated with player management.

Supplemental Material

sj-pdf-1-spo-10.1177_17479541221140192 - Supplemental material for Using consensus methods to standardise judgement-based guidelines required for player management decision-making processes: A case study in professional rugby union

Supplemental material, sj-pdf-1-spo-10.1177_17479541221140192 for Using consensus methods to standardise judgement-based guidelines required for player management decision-making processes: A case study in professional rugby union by Jayamini Ranaweera, Marco Zanin, Dan Weaving and Gregory Roe in International Journal of Sports Science & Coaching

Supplemental Material

sj-pdf-2-spo-10.1177_17479541221140192 - Supplemental material for Using consensus methods to standardise judgement-based guidelines required for player management decision-making processes: A case study in professional rugby union

Supplemental material, sj-pdf-2-spo-10.1177_17479541221140192 for Using consensus methods to standardise judgement-based guidelines required for player management decision-making processes: A case study in professional rugby union by Jayamini Ranaweera, Marco Zanin, Dan Weaving and Gregory Roe in International Journal of Sports Science & Coaching

Footnotes

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Jayamini Ranaweera

Supplemental material

Supplemental material for this article is available online.

References

Hey

. The data, information, knowledge, wisdom chain: the metaphorical link. Retrieved from: https://www.jonohey.com/files/DIKW-chain-Hey-2004.pdf (2004, accessed on 30 November 2021).

Dammann

. Data, information, evidence, and knowledge: a proposal for health informatics and data science. Online J Public Health Inform 2018; 10: e224. 2019/04/02.

West

Williams

Kemp

SPT

, et al.

Athlete Monitoring in Rugby Union: is Heterogeneity in Data Capture Holding Us Back?

Sports (Basel) 2019; 7, 2019/05/01.

Bartlett

Drust

. A framework for effective knowledge translation and performance delivery of sport scientists in professional sport. Eur J Sport Sci 2021; 21: 1579–1587.

Dijkstra

Pollock

Chakraverty

, et al. Managing the health of the elite athlete: a new integrated performance health management and coaching model. Br J Sports Med 2014; 48: 23.

Ranaweera

Weaving

Zanin

, et al. Identifying the Current State and Improvement Opportunities in the Information Flows Necessary to Manage Professional Athletes: a Case Study in Rugby Union. Front Sports Act Living 2022; 4, Original Research. https://doi.org/10.3389/fspor.2022.882516.

Bang

Frith

. Making better decisions in groups. R Soc Open Sci 2017; 4: 170193. 2017/09/08.

Halson

. Monitoring training load to understand fatigue in athletes. Sports Med 2014; 44: S139–S147.

Hulin

Gabbett

Blanch

, et al. Spikes in acute workload are associated with increased injury risk in elite cricket fast bowlers. Br J Sports Med 2014; 48: 8.

10.

Bourdon

Cardinale

Murray

, et al. Monitoring athlete training loads: consensus statement. Int J Sports Physiol Perform 2017; 12: S2161–S2170. 2017/05/04.

11.

McLaren

Macpherson

Coutts

, et al. The relationships between internal and external measures of training load and intensity in team sports: a meta-analysis. Sports Med 2018; 48: 641–658. 2017/12/31.

12.

Fox

Stanton

Sargent

, et al. The association between training load and performance in team sports: a systematic review. Sports Med 2018; 48: 2743–2774. 2018/09/19.

13.

Vanrenterghem

Nedergaard

Robinson

, et al. Training load monitoring in team sports: a novel framework separating physiological and biomechanical load-adaptation pathways. Sports Med 2017; 47: 2135–2142. 2017/03/12.

14.

Ranaweera

Zanin

Weaving

, et al. Optimizing player management processes in sports: translating lessons from healthcare process improvements to sports. Int J Comput Sci Sport 2021; 20: 119–146. https://doi.org/10.2478/ijcss-2021-0008

15.

Black

Murphy

Lamping

, et al. Consensus development methods: a review of best practice in creating clinical guidelines. J Health Serv Res Policy 1999; 4: 236–248. 2000/01/06.

16.

Bahr

Clarsen

Derman

, et al. International Olympic committee consensus statement: methods for recording and reporting of epidemiological data on injury and illness in sport 2020 (including STROBE extension for sport injury and illness surveillance (STROBE-SIIS)). Br J Sports Med 2020; 54: 72.

17.

Barwood

Gibson

Gillis

, et al. Menthol as an ergogenic aid for the Tokyo 2021 Olympic games: an expert-led consensus statement using the modified Delphi method. Sports Med 2020; 50: 1709–1727. 2020/07/06.

18.

Robertson

Kremer

Aisbett

, et al. Consensus on measurement properties and feasibility of performance tests for the exercise and sport sciences: a Delphi study. Sports Med Open 2017; 3: 2–2. 2017/01/05.

19.

Mallett

McLean

Holden

, et al. Use of the nominal group technique to identify UK stakeholder views of the measures and domains used in the assessment of therapeutic exercise adherence for patients with musculoskeletal disorders. BMJ Open 2020; 10: e031591.

20.

Murphy

Black

Lamping

, et al. Consensus development methods, and their use in clinical guideline development. Health Technol Assess 1998; 2: i–iv. 1-88. 1998/04/30.

21.

Ranaweera

Weaving

Zanin

, et al. Digitally optimising the information flows necessary to manage professional athletes: a case study in rugby union. Front Sports Act Living 2022. https://doi.org/10.3389/fspor.2022.850885.

22.

Brouwers

Kho

Browman

, et al. AGREE II: advancing guideline development, reporting and evaluation in health care. Cmaj 2010; 182: E839–E842. 2010/07/07.

23.

Moher

Liberati

Tetzlaff

, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 2009; 151: 264–269.

24.

Downs

Black

. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health 1998; 52: 377–384. 1998/10/09.

25.

Shrier

. Consensus statements that fail to recognise dissent are flawed by design: a narrative review with 10 suggested improvements. Br J Sports Med 2020. 2020/10/02.

26.

Braun

Clarke

. Using thematic analysis in psychology. Qual Res Psychol 2006; 3: 77–101.

27.

Harvey

Holmes

. Nominal group technique: an effective method for obtaining group consensus. Int J Nurs Pract 2012; 18: 188–194. 2012/03/23.

28.

Lynn

Determination and quantification of content validity. Nurs Res 1986; 35: 382–385. 1986/11/01.

29.

Hooper

Jutai

Strong

, et al. Age-related macular degeneration and low-vision rehabilitation: a systematic review. Can J Ophthalmol 2008; 43: 180–187. 2008/03/19.

30.

Jaspers

Kuyvenhoven

Staes

, et al. Examination of the external and internal load indicators’ association with overuse injuries in professional soccer players. J Sci Med Sport 2018; 21: 579–585. 2017/10/29.

31.

Colby

Dawson

Heasman

, et al. Accelerometer and GPS-derived running loads and injury risk in elite Australian footballers. J Strength Cond Res 2014; 28: 2244–2252. 2014/07/24.

32.

Kalkhoven

Watsford

Coutts

, et al. Training load and injury: causal pathways and future directions. Sports Med 2021; 51: 1137–1150.

33.

Impellizzeri

Menaspà

Coutts

, et al. Training load and its role in injury prevention, part I: back to the future. J Athl Train 2020; 55: 885–892. 2020/09/30.

34.

Impellizzeri

Ward

Coutts

, et al. Training load and injury part 2: questionable research practices hijack the truth and mislead well-intentioned clinicians. J Orthop Sports Phys Ther 2020; 50: 577–584. 2020/08/04.

35.

Hulin

Gabbett

Lawson

, et al. The acute:chronic workload ratio predicts injury: high chronic workload may decrease injury risk in elite rugby league players. Br J Sports Med 2016; 50: 31.

36.

Gamble

Chia

Allen

. The illogic of being data-driven: reasserting control and restoring balance in our relationship with data and technology in football. Sci Med Footb 2020; 4: 338–341.

37.

Popper

. Conjectures and Refutations: The Growth of Scientific Knowledge. Routledge, 1962.

38.

Perin

Vuillemot

Stolper

, et al. State of the art of sports data visualization. Comput Graph Forum 2018; 37: 663–686.

39.

Robertson

Bartlett

Gastin

. Red, amber or green? Athlete monitoring in team sport: the need for decision support systems. Int J Sport Physiol Perform 2016; 12: S273–S279.

40.

Cole

Altman

. Statistics notes: percentage differences, symmetry, and natural logarithms. Br Med J 2017; 358: j3683.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.39 MB

0.11 MB