Abstract
Our previous work identified that nine leading guidance documents for seven different types of systematic review advocated the same process of literature searching. We defined and illustrated this process and we named it ‘the Conventional Approach’. The Conventional Approach appears to meet the needs of researchers undertaking literature searches for systematic reviews of clinical interventions. In this article, we report a new and alternate process model of literature searching called ‘A Tailored Approach’. A Tailored Approach is indicated as a search process for complex reviews which do not focus on the evaluation of clinical interventions. The aims of this article are to (1) explain the rationale for, and the theories behind, the design of A Tailored Approach; (2) report the current conceptual illustration of A Tailored Approach and to describe a user’s interaction with the process model; and (3) situate the elements novel to A Tailored Approach (when compared with the Conventional Approach) in the relevant literature. A Tailored Approach suggests investing time at the start of a review, to develop the information needs from the research objectives, and to tailor the search approach to studies or data. Tailored Approaches should be led by the information specialist (librarian) but developed by the research team. The aim is not necessarily to focus on comprehensive retrieval. Further research is indicated to evaluate the use of supplementary search methods, methods of team-working to define search approaches, and to evaluate the use of conceptual models of information retrieval for testing and evaluation.
1. Introduction
Our previous work identified that nine leading guidance documents for seven different types of systematic review advocated the same process of literature searching for study identification [1]. This finding suggests that researchers are guided to search for systematic reviews in the same way, irrespective of the type of systematic review they are undertaking or the data they need to identify to address their research question.
Our work represented the first time that guidance for study identification had been reviewed and the process of literature searching summarised across different types of systematic reviews [2]. We illustrated the search process using a conceptual model made up of nine stages, and we named the process ‘the Conventional Approach’, as the methods reported were common convention to leading guidance documents (see Supplemental Figure 5, web-only material). The Conventional Approach appears to suit the needs of researchers undertaking systematic reviews of clinical interventions since it is focused on a transparent report of a comprehensive approach to study identification which aims to minimise bias [2,3].
Our work raised questions about how researchers search for studies for systematic reviews, here we focus on two; one relates to theory and the other to practice.
1.1. A question of theory
The origin of the Conventional Approach is unclear [2]. Was it developed a priori for use in systematic reviews, or has the process emerged organically, and over time, to meet the needs of researchers undertaking literature searching for systematic reviews, and responsively to meet the demands of decision-makers using reviews? The information science literature reports numerous information-processing models which seek to illustrate and examine information-seeking behaviour. The practice of using theoretical models to illustrate and then to evaluate the process of literature searching does not appear to have been adopted as tool to explain and test information retrieval strategies for systematic reviews in the published literature [2]. We wondered, why?
1.2. A question of practice
It is also necessary to ask whether it is appropriate to use the same process of literature searching to identify studies reporting randomised or controlled trials for reviews of clinical interventions as for studies reporting qualitative data for qualitative evidence synthesis? It is true that both types of review require a transparent account of the literature search process; however, the study types needed for one review are materially different from the other. Similar challenges in applying a model of literature searching optimised for systematic reviews of clinical interventions extends to other types of systematic review or evidence synthesis, namely: reviews of economic evaluations, reviews of complex interventions, reviews of diagnostic or prognostic test accuracy, and reviews of theory. Should the process of literature searching used for these reviews account for different study types and should researchers search differently depending on the type of review that they undertake?
This study seeks to bridge the gap between theoretical and conceptual information retrieval process models [4–13] and the practical experience and tacit knowledge gained from undertaking systematic reviews in practice [14,15]. In response to the questions raised above, this article reports the conceptual design of and the theory behind a new information retrieval model for use in complex systematic reviews. We call this model ‘A Tailored Approach’ since it aims to tailor the literature search approach/process to the type of review undertaken and the studies from which an appropriate synthesis could be undertaken. We submit the Tailored Approach as one potential alternative to the Conventional Approach [2].
2. Article aims
The aims of this article are
To explain the rationale for and the theories behind the design of A Tailored Approach;
To report the current conceptual illustration of A Tailored Approach and to describe a user’s interaction with the process model; and
To situate the elements novel to A Tailored Approach (when compared with a Conventional Approach) in the relevant literature to explore the implications of this model for researchers.
2.1. Article structure
The article is structured by the three aims above. This reporting structure conforms to other articles which have set out and reported information retrieval models [4,8,9,11]. Ellis [8], for instance, established the case for his proposed model, then reported the structure and design of the model, before situating it in context. Bates’ [4] Berry Picking model has a similar structure (albeit presenting the same themes in a slightly different order). We aim to follow this method of presentation with similar aims: to identify what we see as the need for the proposed model, to describe the model, and to locate where in the information retrieval pathway or evidence synthesis ecosystem our proposition might sit. One area we have experimented with, which we believe might be novel, is the description of the user’s interaction with the model (covered in Aim 2). This is an area of particular interest to us, as our model proposes different applications of search methods, such that the use of the model in one case will probably not be the same in another.
2.2. Who is this article for?
This article is intended for research teams looking to develop a search approach for complex systematic reviews, as well as experienced research teams which may already undertake much of the work that we describe. While the article reflects (and it seeks to explore) the information needs of complex evidence synthesis, it is the product of information science. We attempt to link information science to evidence synthesis through the information retrieval models which have historically been used to examine (and attempt to explain) a user’s interaction with an information retrieval problem. In this article, we take the problem of searching for complex reviews, and we examine (and attempt to explain) an information retrieval solution. This work is illustrated with practicable examples of evidence synthesis, which we hope will support information professionals, academic librarians, and researchers, who may be unaware of the use of models to explain information retrieval but are responsible for the development of searches to support complicated reviews or, conversely, are aware of the rich heritage of information science and models but seek guidance in their work.
2.3. Definitions used in this article
Definitions used in this article are set out in Table 1.
Definitions used in this article.
2.4. Evidence cited in this article
This article draws from three discrete sources of evidence and one case study to address the aims above:
Eleven 1 leading systematic review guidance handbooks [18–28];
Nine models of information retrieval practice (see Table 2) [4–12,29,31];
Evidence and supporting studies identified as part of a broader body of work, namely the literature review which identified and defined the Conventional Approach [1] and a research project undertaken by the first author which is reported in full elsewhere [2]; and
A case study comparing the Conventional Approach with A Tailored Approach [14].
Nine conceptual models used to explore the Conventional and Tailored Approach to literature searching.
ASK: anomalous state of knowledge; ISP: information search process; INSU: information needs, seeking and use.
We describe the last two points in greater detail in Table 3. The aim of Table 3 is to provide a brief but fuller description of the broader body of work and the case study for a reader who may be unfamiliar with this work and seeks to situate this article in context of the supporting work.
Brief summary of previous work to support A Tailored Approach.
3. Aim 1: to explain the rationale for and the design of A Tailored Approach
3.1. The rationale for a new model
The origin of what we term the Conventional Approach is unclear but it appears to have emerged from the early to mid-1990s and to have evolved as a process to meet the needs of users of, and researchers undertaking, systematic reviews in order to determine the effectiveness of clinical interventions [2]. The Conventional Approach is premised on the belief that it is possible to identify comprehensively all relevant studies which it is thought are necessary to generate a reliable estimate of intervention effect [2,18,19,27,32–36]. Coupled with the development of guidance on how to report transparently the search approach in protocols and final reviews [18,22,28,37,38], the Conventional Approach has superficially met the needs of decision-makers, since an explicit link can be made between the research undertaken and the findings of the systematic review [39–41].
Our article describing the Conventional Approach reports on this in greater detail [1]. In particular, we highlight how the architecture of the literature searching process in the Conventional Approach is aligned to the comprehensive identification of studies reporting randomised trials [2]. Considerable investment has been made to improve the effectiveness and efficiency of literature searching for clinical interventions through indexing (and re-indexing) of studies reporting trials. This investment appears to have been effective [2,32,42–55] but has not been matched by a corresponding investment in the identification of other study designs, methods, or data. Nor has there been any attempt to design an approach to literature searching that offers an alternative to comprehensiveness as an indicator of search quality [2,56–63]. This realisation is nested in calls to consider how we identify studies and approach systematic reviews and if this can be, or should be, done differently [25,33,64–74].
3.2. Why do we need a new model of searching?
The challenges set out above reflect the origin of A Tailored Approach. On the evidence of our case study, and shared experience undertaking complex systematic reviews and evidence synthesis, we contend that the current model of searching used in systematic reviews and evidence syntheses (The Conventional Approach) does not handle the complex needs of complex systematic reviews and syntheses. Compare the following types of research question:
3.2.1. Example 1: a clinical effectiveness review
What is the effectiveness of Dasatinib compared with standard-dose Imatinib for first-line treatment of chronic myeloid leukaemia? Here:
The research question has clearly defined interventions and comparisons (the drug Dasatinib compared with the drug Imatinib; the drugs are proper nouns: they mean only the name of the drug and not anything else);
The population of the review is unified and clearly defined by unambiguous clinical definition (e.g. first-line treatment of chronic myeloid leukaemia); and
To reliably estimate the effect of the interventions, so-called ‘gold standard’ evidence from studies reporting randomised trials is favoured to determine the effect of the drugs to support decision-making.
3.2.2. Example 2: a complex mixed methods review
To assess the health and well-being impacts on adults following participation in environmental enhancement and conservation activities.
The types of intervention used in environmental enhancement and conservation activities are likely to be broad and are difficult to define;
The population of the review is broad – any adults;
The outcomes of health and well-being are broad and can be measured in diverse ways; and
It is likely that multiple study designs might be used to evaluate interventions in this setting, meaning that a range of study designs might need to be identified to address the research question.
The difference between the two examples as it relates to searching is complexity. Several challenges remain to be resolved before initiating searching in example 2, which are clear from the beginning in example 1. Principally, these relate to matters of definition and scope, but they also include the challenge inherent in the topic and the types of evidence required. The absence of complexity in example 1 makes The Conventional Approach a good conceptual fit for searching: the question aligns to a search process led by searches of bibliographic databases which has a routine and linear process to identity studies. Our case study (example 2 above) illustrates why this idea of complexity is important and why The Conventional Approach is a poor fit for searching in complex systematic reviews.
In our case study, we compared The Conventional Approach to A Tailored Approach when searching for a complex systematic review. In the case study, we concluded that the time spent resolving the complexity in the research question and linking this to a novel process of searching, which reflected the available evidence, was more effective when compared with The Conventional Approach. We observed a stark difference between the number of studies identified (21,409 compared with 453) and The Conventional Approach would have led us to miss eligible studies.
Mahtani and colleagues help to identify and understand this idea of complexity with their definition of what makes a systematic review a complex systematic review (see Table 1). The components listed by Mahtani and colleagues are all evident in (or would be required to resolve) example 2 but they are superfluous for example 1. All the components identified by Mahtani and colleagues were evident in our case study and – while their definition came after our case study – they nest with the rationale for A Tailored Approach, in particular the need to resolve the challenges and identify the information needs early.
We compared Mahtani’s definition to the typologies of reviews from Grant and Booth and Sutton and colleagues together with their associated methodologies to see whether certain types of review could be classified as complex, and thus might suit A Tailored Approach. Beyond acknowledging that some review types require comprehensive searches – something which can introduce complexity where the number of references identified outweigh the resources available – we concluded that it is not the type of review which introduces complexity but the topic or purpose of the review.
The models of information retrieval listed in Table 2 all carry the limitation that they do not help us understand when a review is complex and may benefit from an alternative approach to searching, partly because none reflect Mahtani’s definition of a complex review and partly because the majority of the models deal with singular and discrete information problems. In information retrieval terms, this is the comparison between the classic information retrieval model (see Figure 1) and the alternate models are represented in Table 2.
The classic information retrieval model.
While the definition of what makes a review a complex review came after our early work in defining A Tailored Approach, we adopt the definition believing that it is the complexity that determines the need for and use of A Tailored Approach rather than the type of review.
3.3. The design of A Tailored Approach
In conceptualising the design of a new model of information retrieval, we searched for other examples [2].
First, we searched for models of information retrieval which may have been developed for – or which might explain the information retrieval process of – systematic reviews or evidence syntheses. An iterative approach to searching was used, based on citation searching from models we knew, and searching bibliographic databases such as Library, Information Science & Technology Abstracts (LISTA, EBSCOHost) to identify models we did not know [2]. None of the models identified by this search (even those in Table 2) reported an end-to-end process model with the same systematic focus, and which would achieve the same outcome in study identification, as the Conventional Approach [2].
Second, we undertook scoping searches to identify conceptual models which examined information retrieval theory or behaviour. Our search approach was the same as above. This work sought to examine how information retrieval models have been conceptualised and the theory behind and design of prominent models. We sought examples of how authors had conceptualised the scope of information retrieval problems (particularly in directing a choice between comprehensive and non-comprehensive search approaches) and how users interact with a process to make information retrieval decisions.
Table 2 presents these examples. They are compared, for similarity or dissonance, to the Conventional Approach (summarised in Figure 2 and individually in Supplemental Figures 6–14 in web-only material). This work allowed us to engage with the theory behind information retrieval, using a process model as a framework to generate new theories on how to develop a new conceptual process, in particular one which is not necessarily focused on comprehensive literature searches.
A critique of the Conventional Approach.
4. Challenging the existing information retrieval model
The models set out in Table 2 were selected for several reasons which we shall explore below. One thing that all the models have in common is that they represent a challenge to the classic information retrieval model in some way (see Figure 1). We see harmonies between the classic information retrieval model and The Conventional Approach.
5. Representing complexity
Like The Conventional Approach, the classic information retrieval model is linear in design, representing what Kuhlthau terms ‘certainty and order’ [10]. It assumes that a document exists which the user can find and that by finding the document the search query is resolved. Certainty in The Conventional Approach exists due to the primacy of bibliographic database searching (where it is assumed that the majority of studies are represented and will be identified to satisfy enquires) and order is represented by a systematic, sequential process of searching (the model is also linear, represented by consecutive steps, which are broadly speaking consistently followed to resolve searching enquires) [1,2]. Both models illustrate a search process which is linear and illustrated in simple terms [1,4].
It is unlikely that the searching process is actually this linear in either the classic information retrieval model or in The Conventional Approach. Bates’ [4] model offers an alternative visualisation, based on searching in the social sciences (See online material, Supplemental Figure 11). While it is acknowledged that searching in social sciences is complicated [35,75,76], Bates [4] found, and she represents, a form of searching which uses different search methods in endless variation to resolve search enquiries. Similarly, Byström’s model, and separately Du’s model, also represents processes of searching which are markedly different to either the classic information retrieval model or The Conventional Approach [4,7,31]. Both models involve multiple searches, queries are discussed with colleagues, and different search methods are used.
What we perceive in these models, and specifically Bates’ model, is complexity; not only the complexity of the search process but also the complexity implicit in the user’s behaviour as they search. These models, though their alternate visualisations, help us see a clear challenge to linear systems which we have found in our work do not fit complex systematic reviews [2,14]. Importantly, Bates [4], Byström and Hansen [7] and Du [31] have all based their models on user experience, so their models reflect a user’s needs and their behaviour as they navigate information retrieval challenges.
6. Considering user behaviour: behavioural interpretations of information retrieval
Many of the early models take a behavioural approach in analysing and explaining information retrieval problems (models 1–7 in Table 1). Taylor [11], Belkin and colleagues [5,6], and Kuhlthau [10] argue that the classic information retrieval model reflects the system and not the users’ needs of the system. That is, that the systems/models do not account for human behaviour and needs in processing, addressing, and resolving search enquires. We perceive two components here:
Taylor, Belkin, Dervin and Bates challenge us to acknowledge and resolve complexity. They do this by highlighting the importance of (and influence in) query formulation, defining information-seeking behaviour/needs, and identifying the gap in knowledge prior to searching [4–6,11,29].
Bates [4] (specifically – but also Kuhlthau [10], Byström and Hansen [7], and Du [31]) argues that the classical model fails to capture the ‘real life’ behaviour of searchers who adapt their strategy to searching to the particular need of the moment. As above, it is unlikely that classic information retrieval or The Conventional Approach represent real-life searching practice, because current models represent a system which does not acknowledge human behaviour or ‘real life’.
Incorporating an understanding of human behaviour on information retrieval systems and practice has been an influential contribution of models 1–7 when designing A Tailored Approach. These models have highlighted the importance of acknowledging, addressing, and processing the complexity in the research question(s) and aims of complex reviews, specifically in using a team-based approach to resolve uncertainty. Our case study also illustrates Bates’ argument that searching in complex reviews is (almost) happing in endless variation and certainly using different search methods [14]. In short, these models demonstrate the importance of human behaviour on information retrieval processes. We consider a weakness of The Conventional Approach to be that it does not actually represent ‘real life’ information retrieval (as Bates might argue) and that it cannot process or deal with complexity or human behaviour.
7. Primacy of methods
We argue in The Conventional Approach that bibliographic databases have primacy given the emphasis on systematic reviews with reasonable support for this in the literature on evidence synthesis and information retrieval [1,34,77–81]. The models help us understand the challenges here, specifically as it relates to complex reviews. Taylor [11], Belkin and colleagues [5,6], and Kuhlthau’s [10] work questions why we present search enquires which reflect the system not the users’ needs.
The Conventional Approach prioritises searching of bibliographic databases because, in most cases, this is where most studies are identified for systematic reviews. Detailed search strategies are developed to identify studies based on how they are indexed in bibliographic databases, and we assume that the majority of time spent in searching for systematic reviews is spent on developing the search approach in and searching bibliographic databases.
As above, the primacy would not reflect behavioural interpretations nor – as we demonstrate in our case study – does the focus on one search method help us resolve complexity in information retrieval. The models help us to understand that matching search enquires to search methods might be a more suitable approach to resolving information retrieval challenges. They also encourage to look beyond the linear models as represented in the classical model and The Conventional Approach.
8. Singular and multiple pieces of information
The classic information retrieval model, and many of the early models, consider information gathering as a process which is satisfied by information in a singular form. This might reflect the origin of the models, being based on library systems where a single book might be what the user seeks, and mostly the models in Table 2 were developed prior to the online information gathering world we now know. In systematic reviews, and for evidence synthesis more broadly, the search is for studies and often – in complex reviews – different types of studies from different sources. The definition of complex reviews in Table 1 (explored above) helps us understand what is meant by complexity and the models set out in Table 2 help us understand this complexity, encouraging us to engage with it as set out above [16]. The models in Table 2 do not, however, reflect a process to resolve complex enquires in a way which reflects multiple data sources and the transparent reporting requirements often expected and associated with high-quality evidence synthesis. They represent complexity but do not help us understand how to engage with systematically.
The work summarised in Figure 2, and set out above, led to the first illustration of A Tailored Approach (see Supplemental Figure 15: web-only material). It also established seven guiding principles which, drawn from theoretical models, aligned with our tacit experience of undertaking alternative types of review and evidence synthesis. The development of the guiding principles was instrumental in the initial design and later evaluation of A Tailored Approach [14], especially given an absence of any existing models.
8.1. Seven guiding principles which informed the design of A Tailored Approach
The guiding principles are tabulated in Table 4. Each guiding principle is first expressed as a declarative statement (column one). This statement is situated alongside support for the statement (column two) in the form of ‘verbatim extracts’ or references taken from research studies which informed the initial development of A Tailored Approach. The source for these ideas is reported in the methods section above. This is not exhaustive but representative of how A Tailored Approach evolved. Column three of Table 4 summarises narratively where the ideas came from, why they are considered important, and how they developed. This presentation establishes an audit trail between the theories and the design of A Tailored Approach reported below. The current illustration of A Tailored Approach (Figure 3) was developed through multiple drawings and discussion based on the authors experience of undertaking complex reviews and how it related to the various drawings. It is worth noting that the authors have a background of working on various types of evidence synthesis and information retrieval problems together. This should be acknowledged, since it may influence the direction of the research [109]
Seven guiding principles behind A Tailored Approach.
ASK: anomalous state of knowledge.
The current draft of the Tailored Approach [2].
9. Aim 2: to report the current conceptual illustration of A Tailored Approach and to describe a user’s interaction with the process model
In this section of the article, we describe a user’s interaction with A Tailored Approach. The current version of A Tailored Approach is shown in Figure 3. The process is sequential in design. The completion of one stage leads to the start of the next stage and the interaction with the 10 stages of A Tailored Approach should result in a completed and systematic literature search. Table 5 is structured to describe a user’s interaction with the process model in Figure 3. It reads from left to right to describe:
The action undertaken by a user at each stage of the process;
The purpose of each stage;
Anticipated outputs by stage; and
Supporting notes for each stage of A Tailored Approach.
Description of how users interact with A Tailored Approach.
A Tailored Approach assumes an experienced research team made up of: an information specialist, researchers capable of undertaking a systematic review or evidence synthesis, and researchers skilled in the type of analysis planned, be that quantitative or qualitative. Ideally the research team is supported by experts in the topic of review and, in some cases, the end-user such as commissioners and policy makers [87].
10. Aim 3: to situate the elements novel to A Tailored Approach (when compared with a Conventional Approach) in the relevant literature to explore the implications of this model for researchers
We have set out the rationale for, and theory behind, the design of A Tailored Approach (aim 1), and we have reported the current illustration of the conceptual model and described a user’s interaction with it (aim 2). Below, we situate the stages of A Tailored Approach which are either novel or germane to our experience of following the approach, when compared with the Conventional approach (aim 3). This aim is supported by Figure 4 which reports a visual comparison of the two approaches. The aim of this section of the article is to situate A Tailored Approach in context and to explore the implications of A Tailored Approach as well as stimulating discussion on areas of searching practice which might be developed in reporting or future studies.
Comparison between The Conventional Approach and A Tailored Approach.
10.1. Key stage 2. Scoping/review immersion: a good time for a measured pace?
Little explicit methodological guidance or evidence exists on how to effectively undertake scoping searches for systematic reviews using the Conventional Approach [1,113,114]. The lack of guidance is potentially unhelpful for those undertaking scoping for the first time but the lack of the empirical evidence evaluating methods and approaches may be apposite [113,115].
Scoping is necessarily an uncertain stage which is best developed iteratively [27,112]. Armstrong et al. [85] encourage researchers to consider scoping as an opportunity to collect and organise important background information, which implies a measured pace of work that is critical to A Tailored Approach. Further guidance could usefully specify approximately how much time to invest in scoping and how to allocate this time between scoping tasks [2,17]. A previous edition of NICE [134] guidance for the development of public health guidelines (2013) indicated that 1 month was a suitable amount of time to prepare for and undertake literature searches for public health topics. There was no detail on how this time should be allocated or used, or from where this time estimate originated [2]. The implication was that the tacit experience of the research team/stakeholders, and the topic, would guide the work and this was our experience in the supporting case study [14,28]. We suggest two key messages of stage 2, and for alternative/complex reviews generally: (1) follow a measured pace rather than to rush into finalising searching as quickly as possible, and (2) document the scoping searches, perhaps by using a search narrative, to keep track of the development of the search [28,130,131,135].
10.2. Key stage 3. Discussion: where is the evidence? Developing the role of the team
Theoretical models (see Table 2) offer good support for the role of a team as opposed to an individual to reduce the anomaly of knowledge and develop understandings of information needs. However, limited empirical work demonstrates the role of team-based decision making in systematic or complex reviews and in the Conventional Approach [92,136–138].
In the absence of this empirical work, researchers need to intuitively evolve their own working patterns. This was our experience in the supporting case study [14,139]. The research team worked closely with Cochrane and an expert advisory group convened to help support the project [116,140]. The advisory group were consulted to identify studies (in particular unpublished studies or reports), identify organisations or contacts to help identify studies or reports, and comment on potential search terms [113,116,141]. The latter also involved commenting on the context or how the terminology used to describe interventions had changed over time. The advisory group was an important part of our case study and involving experts is recognised in the literature [87,88,116,142]; having formed the basis of guideline development for many years.
One challenge of team-based approaches is to ensure that the collaboration is genuinely inclusive [136]. This relates to the composition of the group and how they might contribute to reviews [88]. This may involve discussion at the start of the project on roles and responsibilities and how people who contribute to, but do not work full-time on, the review can maintain involvement while working on other projects (e.g. information specialists, statisticians, topic experts) [136,143]. This way of working undoubtedly requires further discussion to understand the benefits [136,137] and possible risks [138]. How to fund such resource and ensure that it is available when needed is also an important consideration.
10.3. Key stage 6: selecting search methods
The selection and prioritisation of search methods in A Tailored Approach challenges the prevailing orthodoxy of literature searching as defined in the Conventional Approach and described in best practice guidance. Using search methods in a different order than is presently conventional practice, and the possibility of not using some search methods, initially represents an alarming proposition, raising associated concerns about potential bias.
Where the Conventional Approach seeks to guard against the introduction of bias though a comprehensive search for studies, A Tailored Approach is more objective being defined by the experience of the research team. It is perceptibly a comparison between ‘doing everything’ by using all search methods in a defined, predominantly linear, order (the Conventional Approach) and fitting the order of search methods and the extent to which they are used (or not used) to the research objectives and studies (A Tailored Approach). Accordingly, A Tailored Approach must defend against unintended bias, steps for which are set out below. It should be acknowledged that neither the Conventional Approach nor A Tailored Approach entirely removes the possibility of bias in study identification. The article reporting The Conventional Approach explores some of the gaps in guidance and the issues that this may create [1].
10.4. Greater diligence in reporting the pre-work
Due to the increased choice-making by the research team, and the possibility of producing a biased sample of studies, A Tailored Approach calls for greater diligence in reporting the conceptual and contextual development of the search approach. Kuhlthau’s [10] model suggests that significant exploratory searching occurs before the search enquiry is formed. The same is true of literature searching for systematic reviews: readers only encounter the final and completed search strategy in the protocol or review appendices and often without any guidance on how or why this approach was chosen over any alternative approach or any associated limitations with the approach [130,131]. This not only betrays the amount of work involved in developing searches but also prohibits a clear understanding of the search approach and studies identified.
A Tailored Approach seeks to acknowledge this ‘pre-work’ (illustrated as stages 1–5 in Figure 3) in the belief that, even if only a brief account is recorded in appendices, this detail can guide the reader to understand the strengths and limitations of the literature search. This does not prevent bias (any more than using every search method removes bias from study identification in the Conventional Approach) but it seeks to acknowledge decisions which might introduce bias and their associated limitations. Our experience using A Tailored Approach is to rationalise and explain the approach to literature searching while situating it within the quality and experience of the research team. It is possible, and indeed desirable, to illustrate a priori the approach to literature searching in a study protocol. Any deviation from this pre-specified approach (for example follow-up searches at stage 10) can be addressed by a post hoc annotation of the differences between protocol and review.
10.5. Re-orientating the idea of the primary and supplementary search methods
In the Conventional Approach, bibliographic database searching is consistently identified as the primary search method [1]. A Tailored Approach disputes the notion of a primary search method, instead searching is tailored to information needs, so that any of the methods set out in stage 6 – including bibliographic searching – could be primary and where the greatest amount of resource is invested to identify studies. Supplementary search methods make an important contribution to searching in systematic reviews. This finding has been consistently reported since approximately 1993 [36]. Occasional studies updating this finding [35,144,145] either focus on individual supplementary search methods (e.g. handsearching [146]), citation chasing [147,148], or contacting study authors [149]. The selection of search methods, and the potential for not using some search methods, is recognised but has not yet been examined as part of a composite model to searching in systematic reviews [1,2,66,77,80,81,150–156]. This may be because the evidence for supplementary search methods varies and their validity as effective and efficient search methods is unclear when compared with bibliographic database searching (see implications for further research) [4].
Further work may revisit the suggestion to omit certain search methods where this is considered too radical. An approach that determines the available time for searching and then allocates specific time limits to specific search methods may be more suitable [2,17,77,157,158]. This approach may give more time to web-searching and author contact and less time to bibliographic searching and citation searching in circumstances where the former methods are considered more likely to identify studies than the latter [14,79]. This example was the finding of our case study and it arose through an a priori concern that a focus on bibliographic database searching might have squandered resources where we did not anticipate identifying studies leaving no resource for non-database search methods where we did anticipate finding studies. It seeks to challenge the presumptive belief in the Conventional Approach that bibliographic database searching is where the majority of search resource should be invested since published studies are the foci of the search [2,77]. While this may be true for review of clinical interventions, it does not hold for other types of review or synthesis and it requires further examination [77]. Guidance on searching for Realist Reviews (or evaluation) contributes here too [100,101]. In A Tailored Approach, searches should be driven by the information needs, not necessarily by the hierarchy of evidence [88]. Again, it is worth spending time thinking what data are needed, where these might be reported, and how will these best be identified, rather than focusing on a particular study design [88,100].
10.6. Retaining systematic approaches and transparent reporting
The selection and order of search methods varies by review in A Tailored Approach. This makes it more challenging to provide clear generic guidance on how to undertake A Tailored Approach. It, however, offers the flexibility for the review team to decide on the order of sources and to clearly report the rationale for this decision including any limitations arising from the approach in the final review. A Tailored Approach is equally suited to be defined in a protocol and then reported according to the guidance of choice. Clearly, A Tailored Approach should be reported to the same standard and should follow the same reporting guidance as conventional systematic reviews or other forms of complex searching such as for Realist Review [37,159–161].
10.7. Limitations
Neither A Tailored Approach nor Conventional Approach can anticipate the complexity inherent in complex reviews. The resulting success of any Tailored Approach is relative to the users’ knowledge of the problem that they explore and the review teams experience [8,9]. A review team must be sufficiently comfortable with and confident in the standard methodological approaches recommend in guidance to fit A Tailored Approach to the review and data.
The work reported here is mostly situated in theory and has only been examined empirically in one published case study [14]. Further evaluations of the same study may find different results and we do not suggest that A Tailored Approach could generalise as a perfect solution for use in complex reviews.
The nature of the comparison between A Tailored Approach and the Conventional Approach reported here and elsewhere is illustrative. The purpose of the comparison in this article was to examine one potential alternative model; to describe it, to illustrate it, and to report a users’ interaction with it. In practice, the comparison we make here is unlikely to be this simple. First, in theory, A Tailored Approach as described above could be made up of multiple different iterations, meaning many possible comparisons to the Conventional Approach. Second, the extent to which the Conventional Approach is followed as a linear process is unclear, making comparisons between approaches hard to systematically analyse. While the article is principally a descriptive report, we situate these issues here as a potential limitation of this article.
10.8. Implications for further research
Bates notes (in 1993) that some methods proposed in her model are not yet accepted as valid or effective approaches to information retrieval. This view remains largely valid now (almost 30 years later) for systematic reviews. There is no clear and empirical evidence for privileging, for example, web-searching in systematic reviews over bibliographic database searching, but the unrivalled coverage of the Internet offers a sufficient rationale for its inclusion as a search approach [154,162–164]. Further comparative evidence is needed to explore non-database search methods if Tailored or other approaches are to be developed, setting out clearly the effectiveness, the efficiencies and value of search methods are a clear target for further research.
Certain nuances in approaches to literature searching may be held in the tacit experience of information specialists or research teams and not reported in practice [2]. Elements of what we describe may be common-practice in some researcher teams. Reporting guidelines, word limits, perhaps a misunderstanding as to the importance of reporting the process of literature searching in full, appear to restrict a full report of the search process which inhibits an understanding of day-to-day practice [130]. We call for greater guidance on this and for the availability of fuller reports which would also help guide further research priorities in literature searching for complex reviews.
For complex reviews, it is helpful to indicate the time taken for review-specific tasks, perhaps by sharing a review Gantt chart which sets out proposed and actual timing in an annex of the review [17]. This detail could help future researchers in planning and budgeting research time [143].
While we have not identified any other conceptual models of searching process which could be used in systematic reviews, we acknowledge the emergence of other ‘styles’ of searching which are developing frameworks and questioning the methodological process of study or data identification [15,66,67,99,102,106,165]. Based on the work of Pawson [93], Booth et al. [102,165] have proposed a six-stage framework to guide a Realist search. Similarly, searching for qualitative evidence synthesis is more confidently moving to a searching approach based on sampling as opposed to exhaustiveness (as is used in reviews of effectiveness) [67,99]. This progress is welcomed as researchers look for guidance and as they respond to the challenge of searching an evidence base which grows daily [165].
In acknowledging these approaches, it highlights a subtle difference in the approach to resolving information retrieval enquires, which we explore in this article and the article reporting The Conventional Approach [1]. That is the difference between researchers who pursue information retrieval research using theoretical models and test data sets (compare the models reported in Table 2) and the researchers who examine practical, work-based retrieval problems, which are in the minority [2]. We hope that our work here makes a case for further harmonies between the theoretical visualisation and practical exploration of retrieval problems [2]. An area for further such research would seem to be on developing approaches which incorporate ‘stopping rules’. That is the development, testing, and evaluation on when to stop searching, for instance, where a researcher has sufficient information to address their research query. This is a challenge facing systematic reviews which has not yet been resolved [15,66,164].
11. Conclusions
This article sets out the rationale for and design of a new information retrieval model for use in complex systematic reviews. Supported by description of its development, and explanation of a user’s interaction with the model, we explain the 10-stage Tailored Approach in detail.
A Tailored Approach is intended for use in complex reviews and as one potential alternative to the Conventional Approach. While the Conventional Approach is defined by its use of comprehensive bibliographic and non-bibliographic searches to identify all available studies, A Tailored Approach aims to allocate search methods and to distribute searching time to the process of study identification. The focus of A Tailored Approach may not be on a comprehensive search for studies. Both models aim to retain a systematic approach to study identification.
A Tailored Approach suggests investing time to develop the information needs and search approach and developing these as a team rather than the work of an individual researcher. Further research is indicated to continue to evaluate the use of supplementary search methods and to evaluate the use of theory-driven models of information retrieval for testing and evaluation.
Supplemental Material
sj-docx-1-jis-10.1177_01655515221114452 – Supplemental material for A Tailored Approach: A model for literature searching in complex systematic reviews
Supplemental material, sj-docx-1-jis-10.1177_01655515221114452 for A Tailored Approach: A model for literature searching in complex systematic reviews by Chris Cooper, Andrew Booth, Kerryn Husk, Rebecca Lovell, Julia Frost, Ute Schauberger, Nicky Britten and Ruth Garside in Journal of Information Science
Footnotes
Acknowledgements
We acknowledge with thanks Prof. Chris Hyde’s contribution to the research project. We acknowledge, directly, the work of the authors of the models reported in Table 2, specifically Taylor, Belvin, Dervin, Bates and Kuhlthau: ‘at last, I should find that low door in the wall, which others, I knew, had found before me, which opened on an enclosed and enchanted garden’ [
]. The lead author acknowledges the following people who have influenced and encouraged (but should not be held responsible for) the work reported here: Theo Lorenc, Alison O’Mara-Eves, Isaac Maerro, Alan Gomersall, FJ, Paul Levay, Carol Lefebvre and Julie Glanville.
Author’s Note
Chris Cooper is also affiliated to Department of Clinical, Educational and Health Psychology, University College London, London, United Kingdom.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The ideas explored here were developed as part of a research project funded through the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme (Project Number 16/54/11). The PI was Prof. Chris Hyde. The writing of this article, and visualisations of the models, were unfunded and undertaken by Chris Cooper after the research funding mentioned above had finished. Dr Booth contributed under his University of Sheffield contract. Dr Husk: This article is independent research supported by the National Institute for Health Research Applied Research Collaboration South West Peninsula. The views expressed in this publication are those of the authors and not necessarily those of the National Institute for Health Research or the Department of Health and Social Care. Professor Garside and Professor Britten were partially supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care (CLAHRC) South West Peninsula, now recommissioned as NIHR Applied Research Collaboration (ARC) South West Peninsula. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. Dr Frost, Dr Lovell and Miss Schauberger were not funded for their work.
Supplemental material
Supplemental material for this article is available online.
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
