Systematic Review Research in Marketing Scholarship: Optimizing Rigor

Abstract

Literature reviews are an essential feature of academic research because, fundamentally, the advancement of knowledge must be built on prior existing work, and to push the frontiers of knowledge, one must be clear as to where these frontiers presently are. By systematically analyzing, synthesizing, and summarizing bodies of related literature, hypotheses can be tested and/or new theories and insights developed. However, despite the recent increase in systematic review research in business and management, and particularly marketing literature, arguably, many reviews continue to be poorly undertaken and reported due to a lack of a rigorous modus operandi for their journal editors, reviewers, and readers. The purpose of this paper is twofold, first to offer marketing researchers and practitioners a modus operandi to better demonstrate the optimization of rigor when undertaking quantitative systematic review research, and second to represent a call-to-action for marketing scholarship to engage further with optimizing rigorous systematic review research in the future. The paper thereby contributes to marketing literature by offering researchers and practitioners a three-stage protocol to demonstrate the optimization of rigor when undertaking systematic review research.

Introduction

Literature reviews are an essential feature of academic research because, fundamentally, the advancement of knowledge must be built on prior existing work, and to push the frontiers of knowledge, we must be clear as to where these frontiers are presently situated. Literature reviews can be undertaken through two broad approaches. First, through a subjective approach based on a qualitative narrative analysis of the literature and, second, through an objective approach based on a quantitative systematic analysis. Both approaches have advantages and disadvantages and therefore, should be seen as being complementary in discovering emerging trends in article and journal performance, authorship, and research patterns, to help gain an understanding of the evolution and intellectual structure of fields of study (Acedo & Casillas, 2005; Verma & Gustafsson, 2020). Systematic reviews, according to Elsbach and van Knippenberg (2020, p. 1227), are “among the most useful vehicles for advancing knowledge and furthering research.” However, despite the recent increase in the deployment of systematic review research in business and management, and particularly marketing literature, there appears to be a lack of consensus regarding an appropriate methodology for undertaking systematic reviews and hence many continue to be poorly undertaken and reported due to a lack of a rigorous modus operandi (Paul et al., 2021; Tranfield et al., 2003). Based on our own experience reading and writing systematic review papers in various business and management journals, we would concur that many of these studies appear to demonstrate a lack of a rigorous approach to analysis. The purpose of this paper therefore is first to offer marketing researchers and practitioners a modus operandi to optimize rigor when undertaking quantitative systematic review research and thereby contributes to marketing literature by offering a practical three-stage protocol to demonstrate the optimization of rigor when undertaking systematic review research. The purpose of this paper is second to represent a call-to-action for marketing scholarship to engage further with optimizing rigorous systematic review research in the future. The structure of the paper is as follows. First, an overview of systematic review methodologies is presented. Second, a discussion of the need to optimize rigor is presented. Third, a three-stage protocol for researchers to engage with rigorous systematic review methodologies is offered. Finally, the conclusions to the paper are then presented.

What do We Mean by a Systematic Review?

Systematic review research is a complement to the traditional qualitative and interpretive approach to literature reviews utilizing quantitative tools and techniques and, through the identification of citation patterns, arguably provides greater objectivity concerning the classification of published documents, i.e. journal articles, books, conference papers, related to a particular research field (Valenzuela-Fernandez et al., 2020). The popularity of systematic reviews is attributed to both the accessibility of free bibliometric software, such as Gephi and VOS viewer, for constructing and visualizing bibliometric networks, as well as to academic databases, such as Clarivate Analytics’ Web of Science, Elsevier’s Scopus, and Google’s Scholar, which have made obtaining and downloading large amounts of bibliometric data for further analysis relatively easy (Donthu et al., 2021). According to Snyder (2019, p. 333), systematic reviews can “address research questions with a power that no single study has” because such reviews “integrate findings and perspectives from many empirical findings.” Systematic reviews are manifested of several types with the nomenclature including bibliometric reviews (see Martinez-Lopez et al., 2018), meta-analyses (see Crosno et al., 2021), and systematic literature reviews (see Gernsheimer et al., 2021) being published in marketing journals. The two main objectives in systematic review research relate to (1) the number of documents that measure productivity, and (2) the number of citations that measure influence and popularity (Martinez-Lopez et al., 2018). By reviewing academic literature, researchers aim to understand the breadth and depth of existing bodies of work and hence identify gaps in knowledge that can be explored further, and by systematically analyzing, synthesizing, and summarizing bodies of related literature, hypotheses can be tested and/or new theories and insights developed (Paul et al., 2021).

Bibliometric reviews consist of a set of methods that can be deployed to evaluate the social and structural relationships between different research constituents (i.e. authors, countries, institutions, topics) through statistical analysis of large amounts of bibliographic data, commonly focusing on citation/co-citation analyses of published documents (Donthu et al., 2021). Research fields are characterized by patterns of communication between researchers. These patterns of communication manifest themselves in various ways, but foremost among these are citations from one author’s work to another. A citation is the acknowledgement that one article receives from another and generally implies a relationship between parts or the whole of the cited article and parts or the whole of the citing article (Smith, 1981). Citation analysis may focus on either or both published documents and their authors. The raw data that citation counts provide are appealing for analysis as they are “unobtrusive measures that do not require the co-operation of a respondent and do not themselves contaminate the response” (Smith, 1981, p. 84). The basic assumption underlying citation analysis is that researchers cite their influences, so that citations act as surrogates for the influence of the cited work (Acedo & Casillas, 2005). Therefore, the total citations to a certain document, author, or journal offers an acceptable surrogate of that article’s, author’s, or journal’s impact on a corresponding research field. Recognized approaches for ranking the impact of authors and journals include eigenfactors (www.eigenfactor.com) and the h-index (Hirsch, 2005). According to Cronin and Meho (2005, p. 1275), the h-index “helps us to distinguish between a ‘one hit wonder’ and an enduring performer”. Despite the accessibility of free bibliometric software and the number of academic databases available to researchers, according to Franceschet (2010), citation analyses of both authors and journals do not change significantly when compiled on Web of Science and Google Scholar, while rankings based on the h-index show only a moderate degree of variation. The h-index aims to combine an assessment of both an individual researcher’s papers and the citations to these papers. Academic databases such as Web of Science will calculate h-indices and the use of spreadsheets can be used to construct and then visualize the results of the various citation analyses, and bibliometric software, such as Gephi and VOS Viewer, can then be used to construct and then visualize the results of further various co-citation analyses such as co-authorship analysis, bibliographic coupling analysis, and co-occurrence of author keywords analysis (see Van Eck & Waltman, 2010). Co-citation analysis assumes published documents that are cited together are often similar thematically (Hjorland, 2013). For instance, in a co-citation network, two documents are co-cited by a third document when the latter simultaneously cites them. Like with bibliometric reviews, meta-analyses can handle large amounts of literature and provide nuanced summaries of given fields, though the literature considered tends to be less diverse (Donthu et al., 2021). Furthermore, meta-analyses are often used as theory extension tools (Combs et al., 2011). Finally, systematic literature reviews require a narrower scope of study and thus tend to include a lesser number of papers for review (Snyder, 2019).

Systematic reviews also have their limitations, and it is important to recognize that bibliometric analyses are retrospective in nature so developments in a particular field of research appear in the citation data only after some time has elapsed. A published document must be exposed to the academic community for a certain period before it is cited by other researchers and appears in the academic databases. Therefore, the findings from bibliometric analyses tend to be limited due to the methodological constraints that result from their research designs and from their dataset(s) (Coombes & Singh, 2022). One such limitation relates to the nature of the academic database searches of published literature. The academic databases are constantly being updated with new literature as they become published, therefore the data collected for a particular study represents only a ‘snapshot’ of data in the database during the period of data collection. Another limitation relates to the nature of the literature search. For instance, dataset(s) that contain only journal articles could potentially restrict knowledge transfer and it is important to consider the inclusion of further additional documents such as books and conference papers. A further limitation relates to the nature of the search strings that extract the data from the academic databases. Different keyword searches could alter the dataset(s) and hence the results of a particular study.

The Need to Optimize Rigor

According to Paul et al. (2021, p. 5), systematic reviews are a complex form of research which should only be undertaken by “disciplinary and methodological experts” and further posit that the use of a protocol is fundamental to such reviews to optimize and demonstrate rigor. For instance, with its origins in biomedical literature, Tranfield et al.’s (2003) highly cited paper presents three stages of a systematic review which incorporates nine phases: Stage I - Planning the Review, Stage II - Conducting a review, and Stage III - Reporting and Dissemination. It is also increasingly common to read articles that use the Preferred Reporting Items for Systematic Reviews and Meta Analysis (PRISMA) and (PRISMA-P) Protocols (see Moher et al., 2015) to provide the necessary rigor to the article. Using the PRISMA or PRISMA-P statements is likely to let editors, reviewers, and readers know, not only what researchers did and found, but also to optimize the rigor of reporting and make the peer review process more efficient. Due to the increasing popularity of systematic reviews in business and management, and particularly marketing literature, more recently, alternative protocols have been published in business and management, and marketing journals. For instance, see the Bibliometric Analysis Procedure and Best Practice Guidelines (Donthu et al., 2021) and the Scientific Procedures and Rationales for Systematic Literature Reviews (SPAR-4-SLR) protocol (Paul et al., 2021). However, based on our own experience reviewing many systematic review papers in various marketing journals, we argue there still does not appear to be a commonly accepted approach for the search and acquisition of the bibliometric data, the evaluation of this bibliometric data, and the presentation of the results of the various bibliometric analyses. As an alternative, we introduce here what we term the 3-Rs protocol which offers marketing researchers and practitioners a practical three-stage linear process to demonstrate the optimization of rigor to editors, reviewers, and readers when undertaking future systematic review research. An illustration of the 3-Rs protocol is presented in Figure 1. However, it is important to recognize that there is no ‘one size fits all’ approach to undertaking systematic review research because various fields of study will be at different stages of their evolution and intellectual structure at different points in time and certain bibliometric analyses will be more appropriate than others. It is down to the skill of the researcher to determine the most appropriate analyses for their chosen topic. Instead, the 3-Rs protocol offers an approach to undertaking future systematic review research as well as a ‘menu’ of various types of citation and co-citation analyses to demonstrate the optimization of rigor.

Figure 1.

3-Rs protocol.

(I) The first stage of the protocol we have termed Retrieve is to define the aims and scope of the analysis, which must be completed before the search and acquisition of the bibliometric dataset(s). It is also important to become familiar with the nuances of the academic database(s) being used. Consideration of the keyword(s) to be used in the search string criteria is extremely important, will take time to refine, and hence possibly some experimentation with using various search string criteria and corresponding search results from the academic database(s) could well be required. Next, the dataset(s) can then be exported for further analysis using spreadsheets such as Microsoft Excel.

(II) The second stage of the protocol we have termed Review involves evaluating the bibliometric dataset(s) for suitability by manually screening and removing any duplicated or erroneous data. If using datasets from more than one academic database, i.e. from both Web of Science and Scopus - to capture as many published documents as possible - these will need to be merged following manual screening (see Kumpulainen & Seppanen, 2022). This stage could typically present table(s) and the associated narrative showing (a) the details of the final dataset, (b) the top journals, including their individual percentage weightings, publishing research on the chosen topic. This stage could also present histograms and/or pie charts and the associated narrative showing (c) the distribution of published documents between certain periods of time and/or (d) the distribution of citations from published documents between certain periods of time (see Coombes & Nicholson, 2013; Coombes & Singh, 2022).

(III) The third stage of the protocol we have termed Report involves the presentation of the results of the various bibliometric analyses. This stage could typically first present table(s) and the associated narrative of the citation analyses showing (a) the characteristics of the key references, including the identification of the prominent authors and key subjects driving the chosen topic at different points in time, (b) the analysis of citing behaviour by presenting the average number of citations to serve as an indicator of the dynamics and status of the chosen topic’s development, and (c) the citation impact and productivity of the top cited journals by presenting the top journals rank according to the impact factor (Garfield, 1979) of the chosen topic’s published documents compared with their h-indices between certain periods of time (see Coombes, 2023; Coombes & Jones, 2020; Coombes & Nicholson, 2013; Coombes & Singh, 2022). The use of bibliometric software, such as Gephi and VOS Viewer, can then be used to second construct and then visualize the results of further various co-citation analyses. This final part of the third stage of the protocol could typically present visualization(s) and the associated narrative of (d) co-authorship analyses, (e) bibliographic coupling analyses, and/or (f) co-occurrence of author keywords analyses (see Coombes, 2023; Coombes & Singh, 2022). At this final stage, it is also important to acknowledge the limitations of the study.

Conclusion

In sum, despite the increase in systematic review research in business and management, and particularly marketing literature in recent years, arguably, many reviews continue to be poorly undertaken and reported due to a lack of a rigorous modus operandi for their editors, reviewers, and readers. This paper contributes to marketing literature by offering marketing researchers and practitioners a practical three-stage protocol to demonstrate the optimization of rigor to these journal editors, reviewers, and readers when undertaking systematic review research and represents a call-to-action for marketing researchers and practitioners to engage further with optimizing rigorous systematic review research in the future.