Abstract
In recent years, a growing number of reviews have been published in the field of sports pedagogy. The standards for conducting meta-analyses and systematic reviews, which were originally developed for health sciences and medicine, are frequently used as methodological guidelines. However, these guidelines do not always align with the subject-specific objectives of other disciplines. Consequently, a wide variety of review typologies and methodological approaches have emerged. This has led to uncertainty and confusion regarding the selection of appropriate reviews. The aim of this paper is to improve the understanding of the different types of review. For this purpose, the common characteristics of reviews have been identified. It then proposes a structuring framework that classifies different review typologies according to three distinct aims. This should guide researchers in selecting the most appropriate type of review and methodological approach and should lead to greater consistency and coherence in future research efforts. To evaluate the suitability of the framework, a narrative review of reviews was conducted. The objective was to analyse the extent to which different types of review could be classified within the framework and to identify any inconsistencies. The initial overview confirmed its suitability. Different types of review with varying frequencies were identified. However, the aims and methodological approaches were not always clearly stated. In future, these should be presented more transparently. Furthermore, inconsistencies in the selection and application of specific review typologies were identified. Overall, the findings provide important insights intended to stimulate the future development of specific guidelines.
Introduction
In many scientific disciplines, the rapid increase in publications has led to a lack of transparency and differentiation of knowledge (Gessler and Siemer, 2020; Grant and Booth, 2009; Paré et al., 2015; Snyder, 2019). This challenges scientists to accurately capture the current state of research to consolidate empirical evidence and theoretical perspectives, critically question or further develop them, or offer well-founded knowledge for societal challenges (Newman and Gough, 2020). Reviews are increasingly used to systematically summarise the state of research, especially in medicine and health sciences, where evidence-based research informs policy and practice regarding the effectiveness of health interventions (Snyder, 2019). Given the high practical relevance, it is imperative to develop guidelines and standards to ensure a rigorous methodological approach. Recently, guidelines have been established in some sub-disciplines of sports science (Ardern et al., 2022) but are lacking in other fields like educational research (Gessler and Siemer, 2020; Newman and Gough, 2020). The same applies to sports pedagogy, where various research activities address a wide range of aims spanning different research paradigms (Brandl-Bredenbeck et al., 2021). These aims cannot always be adequately addressed through systematic reviews but instead require other types of review and alternative methodological approaches (Snyder, 2019). A special issue on systematic literature research highlights the importance of reviews, particularly for early-career researchers (Bergmann and Höner, 2021). At the same time, a survey of scientists in qualification phases within the field of sports science, published in the special issue, reveals that knowledge about writing reviews is often gained through trial and error (Gabriel et al., 2021). Experience-based expertise is generally considered significant when writing reviews, although limited expertise and a lack of awareness or need for further development in this regard are also criticised (Paré et al., 2015). Thus, the initial claim of reviews with regard to addressing the complexity of rapidly differentiating scientific knowledge (Grant and Booth, 2009; Paré et al., 2015; Snyder, 2019) stands in contrast to a more experience-based and less methodologically grounded systematic approach.
Our contribution addresses this issue by first providing an overview of the general characteristics of reviews and then presenting different typologies with their respective aims and approaches. We focus not only on well-established types of review but also on those that have received less consideration in methodological contributions so far. Furthermore, we propose a structuring framework that classifies various review typologies according to three distinct aims, offering guidance on how to select suitable types of review in a justified and aim-driven manner. Finally, we systematically analyse selected reviews in the field of sports pedagogy with reference to this framework to identify inconsistencies in the selection and application of specific review typologies that should be addressed when developing specific guidelines in the future.
Objectives and characteristics of a review
Generally, the searching and processing of literature can be understood as an overarching, central academic aim. Regardless of the discipline, this has always been a starting point for research activities (Snyder, 2019). The intention is to gain an initial insight into a research field to be able to specify and justify one's own objectives. This is usually followed by a more in-depth reading of the literature, which forms the basis for further scientific procedures (Döring et al., 2016). From this perspective, the question arises whether any literature search and processing already correspond to a specific review typology or to what extent reviews differ from this as a methodologically justified scientific method.
Paré et al. (2015) highlight a significant difference, noting that reviews are characterised by the fact that the screening and processing of literature are not just prerequisites for further scientific procedures but become a scientific – and in this case, methodological – issue with its own significance. They can be understood as an independent contribution that is not only relevant to one's own research questions, but also of overarching importance (Newman and Gough, 2020). Reviews can thus not only be seen as a ‘way of collecting and synthesizing previous research’ (Snyder, 2019: 333), but also aim ‘to offer something beyond a recitation of previous research’ (Snyder, 2019: 338). Accordingly, Hart (2018: 197) points out: ‘It is not simply a matter of reassembling the parts back into the original order, but looking for a new order’.
Additionally, when conducting reviews, regardless of their specific aims, quality criteria must be considered to ensure that the screening and processing of the literature is accurate, precise and trustworthy (Snyder, 2019). For this purpose, a systematic approach is considered essential (Booth et al., 2016; Snyder, 2019). The following paragraph outlines these quality criteria, which are understood as fundamental, regardless of the respective types of reviews.
Transparency is considered an important quality criterion (Gessler and Siemer, 2020), particularly concerning literature searches. The following should be comprehensible to outsiders: (1) the search terminology and databases (and years) consulted (i.e. what was searched); (2) the decision rules associated with selective or comprehensive inclusion of articles (i.e. what was included/excluded); and (3) a clear statement on whether the quality of the included articles was assessed (and if so, which criteria were used for the assessment). Associated with this, Gessler and Siemer (2020) identify reproducibility as another quality criterion, which relates not only to the screening but also to the processing of literature. Depending on the specific aims of the review, differences in the extent of rigour with which this criterion is pursued can be observed due to the more interpretative nature of data analyses. Regarding literature searches, a transparent presentation of the approach and all important information is required for the completion of the process. The analysis is concerned with ensuring the intersubjective comprehensibility of the results. Relevance is also named as another universally valid criterion for reviews (Paré et al., 2015; Snyder, 2019). As it is pointed out that the aim and question should be specified in advance, a suitable review for answering the question should be selected and justified too. The fundamental requirement under this criterion is to conduct and document reviews in such a way that other researchers can critically question the appropriateness and suitability of the chosen approach regarding the aim at any time. This is also linked to the final criterion of internal consistency, which indicates that with the chosen review, further specific procedures are determined. Thus, different guidelines should be used for the respective types of review to provide orientation for this purpose.
Overview of types of review
As previously mentioned, reviews pursue a variety of different aims beyond summarising literature. Accordingly, the methodological approaches range from strictly systematic methods and the capturing of statistically validated and evidence-based effects (meta-analyses) to thematic overviews determined by specific inquiries (narrative reviews). Within and across this range, there are additional types of review with varying degrees of differences, often overlapping, which makes categorisation challenging. While some common features exist, ‘there is a lack of unique distinguishing features for the most common review types’ (Grant and Booth, 2009: 106). This issue becomes even more critical when considering the differences between and within scientific disciplines. From this perspective, some authors argue that a standardised methodological approach to writing reviews fails to account for the specific characteristics of the individual fields and their interdisciplinary approaches. Instead, the diverse epistemic interests necessitate a broad range of review typologies, each with its own distinct methodological approach (Snyder, 2019). As noted previously, the same applies to the field of sports pedagogy, where various research activities address a wide range of aims related to different research paradigms (Brandl-Bredenbeck et al., 2021).
Differences arise concerning their objectives (e.g. aims and research questions), the rigour of their data selection procedures (e.g. search strategies, quality assessment), the types of contributions included (e.g. studies, conceptual articles) and their analysis procedures (Gough et al., 2012; Paré et al., 2015; see also the SALSA framework: Search, Appraisal, Synthesis and Analysis to characterise different types of reviews, Booth et al., 2016). We build on these categories but first take a step back to consolidate the diverse issues. To this end, we propose a structuring framework that classifies various review typologies according to three overarching objectives (Gough et al., 2012; Paré et al., 2015; Snyder, 2019) to provide orientation in selecting an appropriate type of review. In this context, the aim of one's research interest serves as the starting point for a reasoned selection of a suitable type of review (see also Torraco, 2016), with the corresponding methodological approach being determined subsequently. In line with this idea, we follow the recommendation of Paré et al. (2015: 193), who emphasise that: it is important to recognize that […] there is no set of specific rules that applies to all types of reviews. Hence, it would be misleading to evaluate the quality of different types of research syntheses using the same criteria. It is rather important to rely on guidelines that take into consideration the uniqueness and specificity of each review type […].
Or, as Grant and Booth (2009: 105) highlight, there is no ‘“one size fits all” approach’.
Reviews with the aim of summarising empirical evidence from studies on advanced methodological evaluation procedures and study variables
Reviews associated with this objective exclusively include empirical studies and aim to present empirical evidence across studies on specific research questions. Systematic reviews, historically regarded as the ‘gold standard’ (Snyder, 2019: 334) among reviews, have since differentiated themselves in terms of this objective. Both aggregative methods, which quantitatively combine randomised controlled studies using similar methods and integrative methods – including studies employing different methods such as case studies, interview studies, etc. – are utilised (Paré et al., 2015). Reviews categorised under this objective are characterised by rigorous methodological approaches, typically featuring a formally structured presentation of data collection and methodologically formalised data analyses. Studies included in reviews under this objective are usually assessed for their methodological quality, with only studies meeting formalised quality standards being considered (e.g. peer-reviewed journals; journals indexed in Science Citation Index [SCI], Science Citation Index Expanded [SCIE], Social Sciences Citation Index [SSCI]). Such reviews are highly relevant for providing evidence of intervention effects while considering contextual factors through the synthesis of various studies (Snyder, 2019). As the reviews become more differentiated, other contributions are also included. This is frequently guided by the Population, Intervention, Comparison, Outcome, Study type (PICOS) strategy, which outlines key aspects to consider when formulating the research question and determining the methodological approach (Methley et al., 2014).
The most well-known type of review falling under this category is the meta-analysis. Meta-analysis is a statistical method for combining results from various studies to weigh and compare them. Specific data collection and statistical analysis procedures are used to aggregate quantitative data, converting results into effect sizes to demonstrate the overall effect size and to compare the effects with each other. This allows the consistency or variability of findings to be questioned and demonstrated. A particular strength of meta-analysis lies in the ability to clarify or resolve existing controversies arising from conflicting empirical studies (Paré et al., 2015; Snyder, 2019).
Aggregative methods exclusively incorporating randomised controlled studies with similar methodological designs reach their limits when including studies using other methods (Snyder, 2019). Qualitative systematic reviews address this issue by integrating evidence from both randomised and non-randomised quantitative studies in a narrative manner. Snyder et al. (2019: 335) refer to methods used in these reviews to summarise findings as ‘meta-narratives’ rather than ‘measuring effect size’. Quantitative study findings are synthesised and related to each other to question aspects such as direction, consistency or strength of effects. While statistical analysis methods may complement this approach, other techniques of synthesising evidence remain central, aiming for a more in-depth synthesis of qualitative and quantitative evidence (for various methods, see Dixon-Woods et al., 2005). Grant and Booth (2009) argue that, unlike meta-analysis, such reviews are associated with the aim of expanding the understanding of a particular phenomenon. Other researchers emphasise that qualitative systematic reviews not only include evidence from quantitative but also qualitative studies (Grant and Booth, 2009; Snyder, 2019). The term mixed study review is also used to denote reviews that include additional studies (Gessler and Siemer, 2020; Grant and Booth, 2009). Paré et al. (2015) also point to these developments, noting the increasing diversity in study designs and verifications (for further differentiations, see Munn et al., 2018b; Zawacki-Richter et al., 2020). Regardless of the specific analysis method used, all reviews assigned here, often also referred to simply as systematic reviews, aim to provide reliable empirical results and answer questions regarding aspects of interventions. This means that ensuring the quality assessment of included studies in order to minimise bias is of great importance (Munn et al., 2018b). Specific study variables are typically summarised, often in tabular form.
Reviews with the aim of structuring and analysing the current state of scientific knowledge on a topic under a specific question
The reviews assigned to this aim are intended to survey literature on a topic and provide a comprehensive yet specifically focussed overview. These objectives go beyond summarising findings and aim to develop new perspectives (Snyder, 2019). For example, this could involve restructuring findings (Hart, 2018). Arksey and O'Malley (2005: 20) describe it as ‘a technique to “map” relevant literature in the field of interest’. Such reviews are often understood as an advancement of systematic reviews, as they aim to analyse and structure empirical evidence more comprehensively (Munn et al., 2018a). Arksey and O'Malley (2005) also emphasise that these reviews have their own methodological integrity. Highlighting thematic focuses regarding temporal developments or depending on research paradigms, establishing connections to adjacent fields, or initiating new research projects or theoretical advancements are further emphasised objectives (Snyder, 2019). In addition to empirical contributions, theoretical and conceptual contributions and – depending on the research question – ‘all relevant literature’ (Arksey and O'Malley, 2005: 22) can also be included in these reviews. Thus, the objectives of such reviews go beyond summarising empirical evidence (Paré et al., 2015) and are particularly utilised in disciplines beyond medicine and health sciences (Gessler and Siemer, 2020; Paré et al., 2015). Depending on the objective, a quality assessment of the included contributions is not necessarily required (Gessler and Siemer, 2020). The methodological requirements allow for some flexibility. However, as already argued, literature searches must be presented in a transparent manner. To address the research questions, both analytical and interpretative methods are employed, adhering to qualitative research standards (Arksey and O'Malley, 2005; Levac et al., 2010). Analyses of data are particularly orientated towards the quality characteristic of intersubjective comprehensibility, and objectives and procedures must be specified to assess the suitability of the approach (Snyder, 2019).
For example, the scoping review can be categorised here, which aims to provide a comprehensive overview of a specific research question. Munn et al. (2018a) emphasise that systematic reviews are the appropriate choice when it comes to ‘addressing the feasibility, appropriateness, meaningfulness or effectiveness of a certain treatment or practice’ (Munn et al., 2018a: 3). Conversely, if the focus is on the ‘identification of certain characteristics/concepts in papers or studies, and in the mapping, reporting or discussion of these characteristics/concepts’, the scoping review is the appropriate choice (Munn et al., 2018a: 3). Here, knowledge is captured and presented in a new light through systematic search, selection and analysis on a specific topic (Gessler and Siemer, 2020). Although study variables are also summarised here, the primary focus is on the structuring or analytical objectives under specific questions. Mapping reviews are considered a subset (Gessler and Siemer, 2020; Paré et al., 2015), specifically aimed at mapping or categorising the existing literature on a particular topic (Grant and Booth, 2009).
The narrative review 1 can also be categorised under this objective. Here, the aim is to summarise the content of articles in a condensed yet easily readable and structured form (Green et al., 2006; Paré et al., 2015). If the content of the individual contributions is critically analysed, these reviews can also serve to stimulate reflection. In contrast to scoping reviews, the methodological approach of narrative reviews allows for more flexibility in both the selection and identification of articles and in their analysis. Instead of a complete collection of published articles, the selection based on subjectively defined criteria or the perspective of theoretical saturation is more decisive here, and findings are usually summarised based on research questions. Despite critical objections regarding the methodological approach, narrative reviews are recognised as a relevant source of inspiration for controversial discourse or educational articles (Green et al., 2006).
Reviews aiming to further develop or critically examine the underlying theoretical assumptions of scientific knowledge
The reviews associated with this aim are intended to advance theoretical knowledge bases or assumptions and develop new perspectives. Central to these reviews is the reflection on the underlying theoretical backgrounds of scientific knowledge. Developments are pursued, for example, within integrative literature reviews (Torraco, 2005; 2016). 2 These reviews may include both empirical and conceptual contributions with the aim of summarising theoretical connections from specific perspectives, combining knowledge to develop new theoretical or conceptual frameworks, or expanding existing ones (Paré et al., 2015; Snyder, 2019; Torraco, 2005). Such reviews ‘are conducted on dynamic topics that experience rapid growth [… or] new emerging topics’ and ‘to resolve inconsistencies in the literature and provide fresh, new perspectives on the topic’ (Torraco, 2016: 404–405). According to Torraco's (2005) guidelines, it is crucial to identify a suitable topic or issue and justify why this type is the appropriate review for addressing it. This initially requires a theoretical or conceptual structuring of the topic to present a reference theory or perspective for conducting and analysing the review. From this specific viewpoint, the contributions that were researched are critically analysed for both strengths and weaknesses (Torraco, 2005). Integrative literature reviews also employ systematic search strategies. However, instead of a strictly standardised methodological approach, the reasoned selection of databases, publication sources and types of contributions is crucial here (Torraco, 2016). With a critical note that most ‘publications on literature reviews do not even mention synthesis, whereas others refer to synthesis as a product of literature reviews without discussing its importance and how it is done’ (Torraco, 2016: 409), the justification is equally essential for the analysis process, where it is necessary to disclose ‘how the synthesis and findings of the study were developed from the literature’ (Torraco, 2016: 419). Snyder et al. (2019) similarly describe integrative reviews as characterised by a more subjective approach and an interpretative compilation of results. Torraco (2005; 2016) emphasises a logical and clear conceptual argumentation as the most important feature in developing new insights and future implications.
A similar objective is pursued by the critical review, which summarises the scientific knowledge base to uncover weaknesses, contradictions and controversies. A critical review presents, analyses and synthesises material from various sources. It focuses not on a comparative summary of findings but on evaluating each contribution based on a criterion for more or less acceptability (Grant and Booth, 2009). With this objective as well, the search strategy does not necessarily follow a strictly methodological approach. Furthermore, the qualitative analysis is more interpretative in nature (Paré et al., 2015).
Reviews of reviews
A special category that cannot be classified within the existing taxonomy is reviews of reviews. This descriptive term is used because the terminology varies greatly depending on the review's specific typology, the contributions included and the objectives pursued (Gessler and Siemer, 2020; Paré et al., 2015). For instance, a systematic review of systematic reviews (Paré et al., 2015) refers to a systematic review that only includes other systematic reviews. Reviews of reviews are particularly used to summarise empirical evidence in medicine and the health sciences, where the number of studies and reviews has increased significantly (Paré et al., 2015). In other fields, reviews of reviews often pursue broader objectives. For instance, Gessler and Siemer (2020) conducted an umbrella review to critically examine the methodological approaches of different review typologies. According to our framework, this review would fall under the third objective outlined above and would thus be characterised as a critical review of reviews.
The proposed structuring framework classifies various review typologies according to three distinct objectives, providing guidance on selecting the most suitable type of review for the specific aims. This process is summarised in the decision tree presented in Figure 1, which helps researchers in selecting the most appropriate type of review.
Decision tree to help researchers select the most appropriate type of review.
Exploratory analysis of selected review typologies and approaches using the proposed structuring framework
The following analysis examines the extent to which different types of review can be classified within the proposed structuring framework. Furthermore, it identifies any inconsistencies in the typologies and their respective approaches that should be considered in future developments. As our analysis focuses on a specific area of sports pedagogy, we consider it to be an initial exploratory step. Our main aim is to encourage further discussion and stimulate reflection on the development of review-specific guidelines for the broad field of sports pedagogy in the future. To obtain an initial overview and identify inconsistencies in the selection and application of specific review typologies, we conducted a narrative review. This is particularly suitable for our objective, as narrative reviews aim to summarise content and promote critical reflection. Furthermore, it adheres to quality criteria while providing greater flexibility in the selection, identification and analysis of articles. As only reviews were included, we specify that we conducted a narrative review of reviews (see Figure 1).
Methodological approach to selecting reviews
In line with our objective, we first had to decide on a search strategy that would produce a large and diverse pool of review articles. This would enable us to conduct an exploratory analysis of the suitability of our framework, as well as identifying any inconsistencies in review typologies and their associated methodological approaches, in a transparent and systematic manner. As the field of school sports encompasses a wide range of research activities and aims within sports pedagogical research (Ennis, 2016; Naul and Scheuer, 2020), we expected to find a sufficient variety of types of review to support such an analysis. We therefore limited our research to this domain. As there is currently no common search term for reviews, we used broadly defined search terms (review, overview article, survey, literature overview) combined with school sports or physical education in the BISp and Fachportal Pädagogik databases. These databases are very well-established in Germany and include both German- and English-language articles in the fields of sports and educational science. The overview of the number of results found (see Table 1) confirms that such publications have increased steadily in recent years.
Number of search matches (as of 14 February 2024).
For the initial exploratory analysis, it was deemed suitable and sufficient to consider a pool of 100 reviews provided by the sports science database BISp since 2020. The research was conducted between 2020 and 2024, resulting in a total of 106 contributions being included for further analysis. No quality check was conducted. After reading the titles and abstracts, 31 articles were excluded because they did not qualify as reviews, which was largely due to the broad search terms of survey and overview article. Contributions where the aim and approach were not clearly identifiable were initially included.
The remaining 75 reviews were analysed further using the framework outlined above. The three decision steps (aims, data selection and data analysis) form the three main categories (see Figure 1). To identify differences in the rigour of the data selection procedure, we analysed the included contributions, guidelines, theoretical references, and information about quality checks. The data analysis was evaluated in terms of the methods employed and their theoretical basis. Subsequently, 24 additional articles were excluded due to a lack of detail regarding their methodological approach. Articles in which the selection and analysis processes were only partially presented were also excluded, as they were barely comprehensible. Ultimately, 51 reviews were retained for further analysis.
Descriptive results and analysis
Table 2 provides an initial overview of the number of types of review mentioned by the authors. Moreover, it identifies any inconsistencies in their methodological approaches, according to our framework. The detailed findings of all reviews in terms of the three categories can be accessed in a supplementary file. Overall, the outlined trend of different review typologies and diverse methodological approaches appears to be confirmed. However, clear focal points can be identified, and some review types were either not found or were not referred to as such by the authors.
Number of reviews, structured according to the three aims and the respective methodological approaches.
Referring to the review typologies mentioned by the authors, it can be seen that systematic reviews dominate in number. The majority of these aim to summarise empirical evidence and other variables. However, it is also worth noting that the objectives of the contributions are not always explicitly stated. Reviews that summarise variables from empirical studies were particularly difficult to categorise, as they also provide a focused qualitative overview (Marques et al., 2021; Pfledderer and Brusseau, 2021). This is also an objective of reviews targeting the second aim (see Figure 1). The difference lies in whether the objective is to summarise the literature or to analyse and structure it. The former is achieved through a qualitative summary, whereas the latter requires in-depth qualitative data analysis.
Many of these systematic reviews adopt well-established guidelines from other disciplines for their methodological approaches (e.g. Preferred Reporting Items for Systematic reviews and Meta-Analyses [PRISMA] framework, Moher et al., 2009) and document these in great detail. Sometimes, they even register the review process on PROSPERO (https://www.crd.york.ac.uk/prospero/). It is striking that authors generally refer to the broad inclusion of different types of studies (quantitative, qualitative, mixed-method) when speaking of systematic reviews. However, there are differences in the extent to which these methodological guidelines are followed rigorously. For instance, the quality of the included studies may be evaluated using explicit tools; only articles from peer-reviewed journals may be considered; other publication venues may be included; or no quality assessment may be performed (11 rigorous versus 13 standardised approaches; see Table 2). Some contributions also mention deviations from standardised procedures, explaining why only selected aspects of the guidelines are considered or why the guidelines are used in a modified form (e.g. Nabaskues-Lasheras et al., 2020; Opstoel et al., 2020 3 ). The data analysis processes employed in systematic reviews are also highly diverse. They range from descriptive synthesis to thematic or content analysis, but they generally aim to summarise study variables. However, the classification was sometimes difficult because the procedure was not always described in detail. Nevertheless, some data analysis procedures give the impression of a more in-depth qualitative analysis (Müller and Böhlke, 2023; Nesbitt et al., 2021). These methodological approaches refer to aims that go beyond summarising empirical evidence, though this is not explicitly mentioned by the authors. Some authors even mention further aims (Herrmann and Gerlach, 2020; Nabaskues-Lasheras et al., 2020; Wagner et al., 2022), yet they still conduct systematic reviews despite another type of review being more appropriate according to our framework. It should be noted that some authors specify their reviews as systematic literature review, indicating their intention to do more than summarise empirical evidence (e.g. Herrmann and Gerlach, 2020; Müller and Böhlke, 2023; Nabaskues-Lasheras et al., 2020; Wagner et al., 2022).
However, we also identified several systematic reviews that align with our framework. These include reviews that aim to summarise empirical findings, implement rigorous data selection procedures and synthesise the effectiveness and suitability of measures (e.g. Anico et al., 2022; Leisterer and Jekauc, 2020). The same applies to reviews that their authors describe as meta-analyses: they all complement systematic reviews, providing additional insights. In some cases, the authors justify this by stating that it enables further studies to be included that are not considered in meta-analyses due to missing effect sizes (Cho et al., 2022; Mello et al., 2022). Some formulations in these reviews suggest that qualitative evaluation is considered less important. For instance, Bryant et al. (2021: 260) state that: although the researchers were hoping to conduct meta-analytic reviews, based on the initial screening, it was discovered that several of the actionable criteria listed within each essential component of physical education had little empirical evidence to support or to refute its inclusion as a measure of QPE.
We also identified five contributions in which the authors conducted scoping reviews that correspond to our framework. Overall, however, the picture that emerges regarding this review typology is rather confusing in terms of its objectives and the data analysis procedure. Some contributions appear to refer to scoping reviews only to justify a less stringent methodological approach to data selection, such as including a variety of contributions and omitting a quality check (Kurth et al., 2022; Killian and Mays Wood, 2021; Schnitzius et al., 2021). However, according to our framework, the aim would have required a systematic review with the respective methodological approach.
Other scoping reviews indicate that such reviews set different requirements and are not a quick alternative to systematic reviews. These reviews explain the analysis procedures in detail under reference to iterative processes (Clish et al., 2023; Dyson et al., 2021; Rief et al., 2022). Two of these scoping reviews address further issues and conduct respective qualitative analyses (Clish et al., 2023; Dyson et al., 2021). However, in neither case do the authors refer to integrative literature or critical reviews. The same applies to two contributions which the authors refer to as systematic reviews (Nabaskues-Lasheras et al., 2020; Wagner et al., 2022). In terms of data selection, scoping reviews typically follow a standardised procedure based on adjusted or different guidelines (framework for scoping reviews: Arksey and O'Malley, 2005; extended PRISMA guidelines for scoping reviews: Tricco et al., 2016; von Elm et al., 2019).
However, some authors point out that existing methodological guidelines do not always seamlessly translate to these types of review, particularly when different academic cultures are taken into consideration. For example, Herrmann and Gerlach (2020) emphasise that literature searches conducted in accordance with the PRISMA statement can be problematic due to insufficient indexing. They note that systematic searches often fail to identify all relevant sources and require supplementation through additional strategies. Others highlight the need for guideline adaptation (Meier and Reuker, 2022) or at least acknowledge the limitations arising from this (Faber et al., 2022; Thurm et al., 2024). Consequently, some authors adopt adjusted approaches (Herrmann and Gerlach, 2020; Jastrow et al., 2022; Marques et al., 2021; Rief et al., 2022) or refer to adaptations of the guidelines (Kerner et al., 2022; Nabaskues-Lasheras et al., 2020).
Furthermore, we identified other review typologies that aim to structure or analyse literature. One such review was a narrative review (Barker et al., 2023), which involved an in-depth qualitative data analysis. According to our framework, we would therefore classify it as a scoping review. Another example is a mixed study review (Murtagh et al., 2023), which follows the methodological approach of systematic reviews. Finally, some authors did not specify the typology, referring to reviews or literature reviews. While the latter can clearly be classified as a review conducted under the second aim, the former have different objectives and employ different data analysis procedures. However, neither employed a rigorous, standardised data selection procedure (see Table 2). It is unclear whether the respective review typologies were not mentioned due to a lack of knowledge or acceptance of alternative types of review or whether there is another reason. However, it is notable that the authors of scoping or narrative reviews extensively discuss and justify their different aims and methodological approaches (Barker et al., 2023; Wiklander et al., 2023), which indicates a lack of acceptance.
In general, it can be noted that review typologies besides systematic reviews are underrepresented in (sports pedagogical) research. This is problematic because it forces researchers to adhere to methodological guidelines developed for the first aim, which may not necessarily be applicable to other aims. Deviations from standardised procedures may sometimes be more appropriate. For example, focusing solely on articles in peer-reviewed journals and excluding literature from other sources (e.g. handbook articles or dissertations) may restrict the structuring analysis and further development of scientific knowledge. This issue is particularly evident in the quality assessment of the included studies, which is typically mandated by the guidelines for systematic reviews, but is handled very differently in practice. We found that some reviews pursued a quality check despite this not necessarily being required for their aims (Wagner et al., 2022), while others did not conduct a quality check even though it was recommended for their aims (Jastrow et al., 2022; Pfledderer and Brusseau, 2021). Furthermore, the common practice of conducting a broad search to generate a large number of contributions and then narrowing them down to a few studies for analysis (Cho, 2020; Wagner et al., 2022; Woods et al., 2021) should be critically questioned with regard to other aims.
Discussion – implications for developing specific guidelines for conducting reviews in the field of sports pedagogy
Taking a closer look behind the scenes, we see that there is a confusing diversity of types of review which are understood and implemented in very different ways (Gessler and Siemer, 2020; Grant and Booth, 2009; Paré et al., 2015; Snyder, 2019). Addressing this issue, we introduce a new framework for discussion that clarifies the general characteristics of reviews and categorises different types according to their specific aims.
Our exploratory analysis of selected reviews confirms the fundamental suitability of our framework. We identified reviews that align with our categories, as well as those that reveal inconsistencies in their aims and methodological approaches according to our framework. Regardless of this differentiation, the trend towards different review typologies and methodological approaches seems confirmed. However, as the research was limited to a German-language database between 2020 and 2024, the generalisability of the findings must be questioned. To obtain a more comprehensive overview of different review typologies within the broad field of sports pedagogy, this research should be expanded on in future. In particular, including more internationally representative databases and extending the timeframe would enhance the breadth and applicability of the findings. Nevertheless, our initial overview provides important findings that encourage further discussion and stimulate reflection on selecting an appropriate review typology and developing review-specific guidelines, which will be discussed below.
Firstly, our initial overview of the selected reviews reveals dominant review typologies, as well as review typologies that were either not identified or were not referred to as such by the authors. While meta-analyses and systematic reviews appear to be well-established in sports pedagogy and tend to follow external guidelines uncritically, scoping and narrative reviews are underrepresented in terms of numbers. Integrative and critical reviews were not found at all. These reviews appear to be less well-established or recognised, despite some reviews indicating a clear need for them through the stated aims.
This dominance may explain why most reviews follow a standardised approach that has predominantly followed guidelines for meta-analyses and systematic reviews. However, as mentioned before, these guidelines may not be equally effective for all aims. The same applies to advancements in guidelines for meta-analyses and systematic reviews, which often adhere to non-discipline-specific logic (see also the debate on expanding scoping reviews: Daudt et al., 2013). According to our framework, guidelines that consider the specific objectives are both necessary and promising for reviews following the second or third aim. For example, the PRISMA guidelines for systematic reviews focus on narrative synthesis, which does not fulfil the requirements of more advanced aims that go beyond the summarisation of empirical evidence and study variables. Such aims require the presentation of a theoretical framework to underpin the analyses for structuring, conceptual development, or critical analysis. Overall, we would recommend paying closer attention to the description of the data analysis procedure, especially for reviews under the second or third aim. While the data selection procedure is often described in detail, the specific analysis is usually only mentioned briefly. However, the analysis should also be justified and explained to make the results more comprehensible to others.
This is even more problematic because some reviews imply that they only refer to reviews besides systematic reviews to justify a less rigorous methodological approach. This disregards the fact that these review typologies require a more in-depth qualitative analysis of the data, in line with their objectives. Munn et al. (2018a: 6) highlight this issue, stating that: it is important to mention some potential abuses of scoping reviews. Reviewers may conduct a scoping review as an alternative to a systematic review in order to avoid the critical appraisal stage of the review and expedite the process, thinking that a scoping review may be easier than a systematic review to conduct.
Although the data selection in scoping reviews generally follows slightly less strict criteria, a stronger theoretical embedding is often required to provide the theoretical framework for a well-founded content analysis of the data (see Figure 1).
Secondly, these inconsistencies highlight the need for more thoughtful and sophisticated selection of review typologies and appropriate methodological approaches. This includes explicitly stating the aims and justifying the appropriate review typology and methodological approach according to these aims, as this information was not always clearly identifiable. Our framework serves as a guide, enabling the selection of an appropriate type of review that aligns with a specific research aim. It is based on the fundamental assumption that the searching for and processing of literature in reviews is an independent objective that must be clarified in advance and not merely a preliminary methodological task (Newman and Gough, 2020; Paré et al., 2015). Given the rapidly evolving state of scientific research (Grant and Booth, 2009; Paré et al., 2015; Snyder, 2019), it is crucial to consider various research interests and paradigms when revising review guidelines. We view our framework as a first step. Besides proving the framework's suitability based on an extended review, further development is necessary. Although there are multiple recommendations and adaptations for the data selection procedure, there are hardly any guidelines for the data analysis procedure itself. We would recommend paying more attention to this aspect.
This is particularly relevant in fields beyond medicine and health science due to the specific research interests of each discipline. Paré et al. (2015: 191) note that ‘theory plays a central role in [the information systems literature] both in explaining why things happen the way they do and in predicting what things might happen given certain circumstances’ and that ‘theoretical and narrative reviews […] by far represent the most prevalent types of reviews’ in information systems. This issue underscores the need for discipline-specific guidelines (Levy and Ellis, 2006) to ensure review quality. Gessler and Siemer (2020) come to similar conclusions in the context of vocational education, identifying predominantly conceptual and scoping reviews, as well as a smaller number of systematic reviews and no meta-analyses. They also highlight the lack of discipline-specific guidelines, criticising the fact that no standards have yet been developed in educational sciences. Using their data search, they illustrate the need for discipline-specific guidelines by showing that only journals indexed in databases were included, following standardised guidelines. This left out many relevant contributions (Gessler and Siemer, 2020). The same applies to sports pedagogy, where various research activities address a wide range of objectives spanning different research paradigms (Brandl-Bredenbeck et al., 2021).
In conclusion, rather than selecting the first type of review that comes to mind, we recommend choosing the one that best aligns with the specific aims of the research. This can then be used as a starting point for critically engaging with the corresponding methodological approach, rather than adopting a standardised procedure. To this end, we propose a structuring framework which provides clear and practical guidance that is relevant to both research and professional practice. By enhancing the methodological rigour of the field and paving the way for greater consistency and coherence in future research efforts, our framework is highly relevant to researchers. It enables them to accurately capture the state of the science in relation to their respective research objective. This is also relevant for educators, as a comprehensive overview enables them to integrate empirical findings and theoretical insights into their practice and to further develop it. We hope our contribution will be the first step in this process.
Supplemental Material
sj-docx-1-epe-10.1177_1356336X251369230 - Supplemental material for Reviews in the field of sports pedagogy: A look behind the scenes
Supplemental material, sj-docx-1-epe-10.1177_1356336X251369230 for Reviews in the field of sports pedagogy: A look behind the scenes by Sabine Reuker and Stefan Meier in European Physical Education Review
Footnotes
Acknowledgements
The authors would like to thank Marla Sembach and Bernhard Lang for the support they provided in searching for the reviews and Katharina Sarwari for her proofreading assistance.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Supplemental material
Supplemental material for this article is available online.
Notes
Author biographies
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
