Abstract
Systematic reviews aim to use formalised and explicitly described methods. However, studies show that systematic reviews pose challenges which can only be resolved using expert judgement that is resistant to explicit formulation. The expertise required to make such judgements can be understood as practical knowledge or phronesis, and is based on lived experiences rather than following clearly defined rules. This study used qualitative methods to investigate the phronesis of expert searchers in the development and conduct of searches for studies for systematic reviews. In particular, the study focused on two ‘supplementary’ search methods: forward citation searching and web searching. Data collection used semi-structured interviews with 15 expert searchers and the analysis used a hermeneutic phenomenological approach. The findings describe five habits of phronesis when searching for studies: Outcome-oriented; persistent; adaptive; critically engaged and holistic. The study brings attention to the use of expert judgement when searching for studies for systematic reviews.
1. Background
Adherence to formalised and explicitly described methods which are pre-specified in a protocol is a key strength of systematic reviews. To different degrees, this is true of both aggregative reviews, which aim to use predefined concepts and methods to assess empirical data; and configurative reviews, which aim to test and refine theories to understand complex phenomena [1]. Librarians and information specialists have an integral role in searching for studies for systematic reviews and have developed detailed guidance on many aspects of this process [2–7]. However, the emphasis on formalisation and explicit description can obscure how expert judgement is also required when carrying out searching and reviewing tasks. Several studies have brought attention to this phenomenon. Boell and Cecez-Kecmanovic [8] describe how the identification and analysis of studies for systematic reviews is an interpretative process in which understanding is gradually gained and refined through reading and re-reading study reports. This interpretive process is exhibited in Moreira’s [9] ethnographic study of a systematic review research team, which records the deliberative decision-making of researchers through the processes of ‘disentanglement’ (i.e. identifying and extracting data from studies) and ‘requalification’ (representing data in a synthesised format). Drawing out the importance of expertise for making interpretive decisions, Lorenc et al. [10] found that the need for expert judgement is intensified in complex reviews which synthesise heterogeneous data. Although methodological approaches have been developed to work with heterogeneity, Lorenc et al. [10] describe how reviewers use these approaches pragmatically rather than mechanistically to remain sensitive to complexity. Similarly, Cooper et al. [11] found that experienced researchers have different understandings of what ‘effectiveness’ means in the context of searching for studies, and Shepherd [12] describes how novice reviewers struggle to make expert judgements in the otherwise highly formalised context of systematic reviews. Melendez-Torres et al. [13] show how meta-analyses and narrative syntheses involve judgements about data which can lead to different conclusions. The judgements described in these studies are not necessarily explicitly documented in protocols or method reports, but they nonetheless influence the approach and findings of systematic reviews.
The expertise required to make competent judgements can be understood as practical knowledge or phronesis [14]. Gadamer [15], whose work underpins the methodology of this study, contrasts practical knowledge with technical knowledge, drawing on Aristotle’s distinction between phronesis and techne. Whereas technical knowledge (techne) is learnt by studying abstract rules and formulae, practical knowledge is learnt through exposure to ‘concrete situations’ in their ‘infinite variety’ [15]. Once acquired, phronesis guides how to achieve a desired end through a form of pre-reflexive understanding more akin to perception (e.g. ‘seeing’ what to do) than deliberative analysis [16]. Due to its pre-reflexivity, there is debate about whether actions guided by phronesis can be retroactively analysed and explained in terms of their motivation [17]. In this study, we assume that, to some extent, the ‘reconstruction of reasons for action not necessarily thought out in advance’ is possible and desirable as a way of gaining understanding of expert judgement [16]. In particular, phronesis can be explored through hermeneutic phenomenological analysis (HPA), which seeks to understand how people make sense of their lived experiences, giving close attention to how they interpret situations that confront them [18].
The expertise that librarians and information specialists contribute to systematic reviews centres on the development and conduct of complex searches of bibliographic databases, and the appropriate use of ‘supplementary’ search methods [5,19–21]. The latter include a variety of search methods which can be used to mitigate the shortcomings of bibliographic databases, or, more substantially, as a major source of studies for systematic reviews where the desired literature is mainly outside of the published domain (e.g. ‘grey’ literature) [22] or where the use of keyword searching in bibliographic databases is unlikely to be successful due to diffuse or poorly defined terminology within a topic area [2,23–26]. In the field of health research, librarians historically took on the role of searching for studies for systematic reviews as an extension of curating and facilitating access to scientific journal articles for clinicians – a task for which bibliographic databases and other electronic resources gradually surpassed hand searching as technology improved and the scale of the literature grew exponentially [20]. Specialist roles, such as clinical librarians and information specialists, were developed in recognition of the need for experts to take on this task as their main specialism [27], and their expertise is increasingly acknowledged through co-authorship of systematic reviews [28] and evidence that involving expert searchers in systematic reviews improves the quality of searching [29] and reduces bias [30]. Thus, the role of the expert searcher within systematic reviews is well established, but, despite this, there is limited investigation of how expert judgement shapes the development of searches for studies [11]. Instead, evaluations of how experts carry out searches for studies typically rely on objective measures, such as the degree to which the application of formal guidance is visible in the reporting of search methods [28,29]. The relative lack of investigation of expert judgement in this context is a gap in understanding, which potentially accentuates the perceived importance of formal guidance relative to the role of expert judgement or phronesis. This study aimed to use HPA to investigate the phronesis of expert searchers in the development and conduct of searches for studies for systematic reviews. In particular, the study focused on two commonly used supplementary search methods: forward citation searching and web searching.
Forward citation searching uses a citation index to identify studies that cite a source study [3,6] and web searching uses search engines and topically relevant websites which are not specifically designed for hosting and retrieving studies [3,6]. We focused on these two search methods due to variability in both if and how they are used in systematic reviews, as exhibited in cross-sectional analyses of the conduct of these search methods [31–33]. In contrast, bibliographic databases are routinely searched for systematic reviews, so the initial decision to use bibliographic databases has less relevance from the point of view of phronesis. Exploring expert judgement in ‘weak situations’, that is, in which options are not determined by clearly define rules, affords the opportunity to consider both why search methods are used in addition to how they are used [34]. However, this does not mean that forward citation searching and web searching are necessarily peripheral search methods; indeed, in some reviews, they have as significant a role in study identification as bibliographic databases [35,36]. This has led to some commentators referring to these methods as ‘complementary’ rather than supplementary search methods [2,35].
2. Methods
2.1. Ethics approval and recruitment
Ethics approval was obtained from the University of Exeter College of Medicine and Health Research Ethics Committee (project reference number: Jul20/D/250; date of approval: 1 July 2020). All participants returned signed consent forms via email.
Recruitment used a purposive sampling strategy that aimed to recruit participants from a variety of research settings with experience of both aggregative and configurative reviews. Potential participants were sent an information sheet via email which described the aim of the study and the requirement to agree a time and date to be interviewed for a maximum of 1 h via MS Teams™ or Zoom™. The information sheet also described how all interview data would be anonymised prior to analysis and stored in a secure digital format. No payments or rewards were offered for participation. Participants were required to have at least 2 years experience of searching for studies for systematic reviews on health and social care topics. We focused on health and social care research due to the formative role that researchers in these fields have had in the development of searching conduct for systematic reviews, for example, through Cochrane and the Campbell Collaboration. Participants were also required to have used either forward citation searching or web searching in this context. The majority of people we approached were information specialists, although some had other role titles as there is variation in how the expert searcher role for systematic reviews is described. However, the substantive part of all potential participants’ roles was either searching for studies or a combination of searching for studies and other systematic review tasks.
Overall, 28 people were approached, of which 15 with relevant experience agreed to be interviewed. The participants’ characteristics are summarised in Table 1. All participants had experience of web searching and 14 had experience of forward citation searching.
Characteristics of participants.
n (%) unless otherwise specified.
2.2. Data collection
A semi-structured interview guide was developed which aimed to facilitate participants to explore their experiences of using forward citation searching and web searching in systematic reviews (see Supplemental Material). Follow-up questions were included alongside the main questions to encourage participants to reflect on their answers. Following a hermeneutic phenomenological approach, the interviews aimed to be responsive to the participants’ answers and pursue relevant issues as they arose [37]. In particular, the interviews aimed to facilitate a dialogue between the interviewer and participant which enabled a shared understanding of phenomena or ‘fusion of horizons’ [38]. This meant that the order in which questions were asked was not always the same nor were all the questions always asked. However, all the interviews consisted of two main sections, the first on forward citation searching and the second on web searching. We did not aim for data saturation, which commentators have argued is not relevant to HPA [39,40]. Instead, we relied on the specificity of the sample and in-depth dialogue to generate sufficient ‘information power’ to explore the phenomenon of interest [39]. The interview guide was piloted with an information specialist colleague. All the interviews were carried out by SB via MS Teams or Zoom video call software between September 2020 and June 2021. Interviews were recorded using the video call software and transcribed by a professional transcription service. All interviews were conducted in English, including one participant for whom English was not their first language but was sufficiently proficient to participate without hindrance.
Prior to the interviews, SB undertook a bracketing interview with GJMT and RA to become more aware of their own perspectives on forward citation searching and web searching. This led to the recognition by SB of a tacit assumption that the search methods had an important role to play in study identification, which might not be shared by all participants. Thus, it was recognised as important to allow the interviewees to present their own views on the value of the search methods without SB challenging them if their views differed.
2.3. Data analysis
Transcripts were anonymised using an alphabetical letter (A to O). Data analysis aimed to describe the phronesis which guided the participants’ searching practice in such a way that was expressive of the group as a whole, while remaining alert to differences in the group [18,38]. To remain sensitive to how phronesis is resistant to explicit formalisation while still being able to articulate a descriptive account of its content, we followed McDowell in conceptualising phronesis in terms of ‘habits of thought and action’, that is, tendencies to perform certain actions which were identified in the data we collected [41]. Thus, we use the phrase ‘habits of phronesis’ when describing our findings.
The analysis used an interpretative process which followed a hermeneutic circle, moving iteratively between analysing transcripts as a whole and analysing parts of transcripts (see Figure 1) [38,42]. We started by reading and re-reading the transcripts to become familiar with the content. An initial round of inductive coding was then undertaken which highlighted key phrases or words that described the experiences of using the search methods for each participant. The transcripts were then read again alongside the initial codes with a view to developing the codes with new understandings which were uncovered by considering how they are related to other sections of a transcript or the transcript as a whole. This process was repeated iteratively with the aim of producing a rich account of each participant’s understanding of forward citation searching and web searching. As the analysis progressed, understanding gained from reading transcripts was explored within other transcripts, which enabled the development of a shared understanding of the phronesis of the participants. As described by Smith et al. [18] through each iteration, our understanding of the transcripts moved from a high-level semantic understanding to a more in-depth, interpretative understanding. We did not separate analysis of forward citation searching and web searching, but we remained alert to differences in their usage. As recommended in Smith et al.’s [18] guidance on HPA, we sought to ensure validity in the analysis by implementing Yardley’s [43] criteria of commitment, rigour, transparency and coherence. Specifically, we were attentive to Yardley’s [43] recommendation for phenomenological analysis to make ‘effective use of prolonged contemplative and empathic exploration’ which aims to ‘transcend superficial, “common sense” understandings’. SB read all the transcripts and undertook the coding; GJMT and RA read selected transcripts and met regularly with SB to discuss the emergent habits of phronesis.
Application of HPA using hermeneutic circle.
3. Findings
Data analysis identified five habits of phronesis that guide searching for studies using forward citation searching and web searching: outcome-oriented; persistent; adaptive; critically engaged and holistic (Figure 2). All five were closely related and used alongside each other by the participants; in particular, adaptive was closely related to persistent and thus these two approaches are discussed together. The interrelation of the five identified habits of phronesis is depicted in Figure 1. Outcome-oriented, persistent and adaptive relate to searching for studies; critically engaged relates to the identification of studies from the results of searches, with feedback from this stage going back into searching for studies. Holistic describes an overall approach to the use of different search methods alongside each other. Other approaches to searching were also identified in the analysis; however, these were typically framed by participants as ‘rule-following’ approaches which relied more on techne than phronesis. These included process-oriented searching, uniform or standardised searching, and exhaustive or comprehensive searching. We refer to these rule-following approaches in our findings where they bring out the contrasting character of the identified habits of phronesis, but do not discuss them in detail.
Five habits of phronesis.
In the following sections, we describe the identified habits of phronesis. Unless it is clear which search method a participant is discussing, we refer to web searching as WS and forward citation searching as FCS alongside the ascription of quotations to participants.
3.1. Outcome-oriented searching
Outcome-oriented searching prioritises the identification of relevant studies above how studies are identified. The participants distinguished this from process-oriented searching, which aimed to show that searches fulfilled the expected criteria for the type of systematic review that was undertaken. Process-oriented searching was a rule-following approach which correlated with seeking to minimise the need for expert judgement, whereas outcome-oriented searching was predicated on expert judgement about how to achieve the desired outcome. Key phrases among participants discussing outcome-oriented searching included ‘do[ing] what works’ (N), ‘focusing…where it’s going to add value’ (G) and searching when it ‘seems like it’s going to be fruitful’ (J).
Outcome-oriented considerations were strongly determinative of whether a search method was used. This was also true of process-oriented considerations, but the rationale was different. One participant contrasted process-oriented with outcome-oriented searching, reflecting on the motivation for focusing on the process rather than the outcome: I always have the impression that what people really want to do is demonstrate and show that that they have done [everything possible] … you know it’s like they don’t want someone to say ‘Well why did you not do forward citation searching?’….I think … there is a real kind of value system around that, and like, doing everything that it was possible to do in the time that was available. (J, our emphasis)
Here, the motivation for process-oriented searching is ‘doing everything possible’, irrespective of whether this is useful. They went on to give a hypothetical example, asking the rhetorical question, ‘[W]hat are the chances, really [in an effectiveness systematic review of RCTs] … that there was a huge trial that you never heard about, but then turned up in a forward citation search?’ Although they accepted this approach was sometimes expected for aggregative reviews which aim to search exhaustively, they considered there were lessons to be learned from other types of review (in particular, configurative reviews) in which searching is ‘more about what seems like [is] going to be fruitful’. WS was more frequently used as standard than FCS, but the emphasis was still on usefulness: ‘If I spend a lot of time doing something, then I need to know it’s going to be useful’ (I, WS). Commonly mentioned situations in which FCS and WS were considered useful included where search terms were hard to define and where studies were not published in journal article format. However, in practice, it was not always possible to know a priori whether a search method would be useful: ‘I think that it’s very unlikely that you actually have the data about that review topic to show that it [i.e. FCS or WS] is worthwhile’ (N). Thus, participants relied on ‘instinct’ based on past experiences: ‘in practice there’s […] an instinct for topics, whether that’s right or wrong. But I would tend to do it then’ (M).
Decisions relating to WS had an additional factor to consider regarding the reproducibility, or lack thereof, of the searches. Participants were mindful that searches using a search engine, such as Google Search or Google Scholar, were not reproducible due to variation in search results for different users. This was also the case for websites that used search engines as the basis for their search function. Thus, even if a carefully constructed search strategy was developed, searching was compromised from the point of methodological rigour: I think it is like this fundamental question about science. I feel more comfortable when searches are scientific in the way they’re designed, and the way you can, you know, conduct them, the way you can evaluate them, and reproducibility is key to that. (N, WS)
One participant in a research consultancy setting was strongly disinclined to recommend using a search engine due to lack of reproducibility, which was seen as a key criterion of searches for a systematic review, that is, a process-oriented consideration. More typically, participants sought to balance the ‘science’ of searching with the end product: is there any point in doing web searching, because you can’t really be completely scientific and reproducible about it? But then ultimately is that … I don’t know, in terms of a review, is it more about the science of it? Or is it about the product, you know, the things that you’ve identified for it? So it’s a bit of a balance of those two things, I think. (F)
To this end, the participants cited examples where search engines had identified relevant studies and used this to justify the inclusion of search engines in systematic reviews, that is, an outcome-oriented justification. This even extended to preferring Google Search, which has known problems with the reproducibility of searches, over search engines which purport to have less variation in the search results: ‘there’s a few [search engines] that don’t track … I tried testing different web search engines but unfortunately the results weren’t as good’ (N). Similar concerns about the technology used for FCS were not raised.
Outcome-oriented considerations guided the development and conduct of search methods after the initial decision to carry out a search was made. The main consideration for FCS was how to select source studies, that is, the studies on which FCS is carried out. Studies were usually selected from the known set of studies which met the inclusion criteria, and richness of data and ‘highly relevant’ were common reasons for choosing such studies. One participant gave an example from a scoping review: [S]ome of the [potential source] studies had a brief mention [of the review topic], but it was so small that we felt it wasn’t going to be as useful as some of these ones where it was the major focus of the article. (F, FCS)
By contrast, participants who preferred a process-oriented approach typically used all known studies for FCS, which they considered less biased, and which was perceived to correlate with the emphasis on exhaustiveness in some types of systematic reviews. One participant noted that this approach ‘treats everything consistently. We’re not having to make a judgment call as to which studies might be most important’ (O). Another participant cited Cochrane guidance to support this decision. Thus, participants were divided about whether outcome- or process-oriented searching was superior when FCS, which was partially but not entirely delineated by whether participants mainly worked on aggregative (process-oriented) or configurative (outcome-oriented) reviews. However, the idea that using all known included studies evades the need for judgement was questioned by one participant: ‘all the studies known to me, it’s already a selection of studies’ (E). They considered that although such studies met the criteria for synthesis purposes, this did not a priori entail that they also met criteria for searching purposes. Instead, they proposed separate criteria for selecting source studies, designed to optimise the outcome of searching: I think it would be more systematic to establish criteria that could help us in selecting studies from known studies, via a different method and then say, ‘Okay. I’ve identified a specific set of studies, and I apply, for example, a criterion of currency, or a criterion of number of participants, or, for cohort studies specifically, the time of follow up’. Because of course a cohort study that has follow up for 10 years, that’s a big, big project, and I’m pretty sure this is a well cited study as well. (E)
Other participants made the related point that publications that do not meet the inclusion criteria could be worth FCS, such as topically relevant commentaries, or studies which fail to meet inclusion on relatively arbitrary criteria, such as publication date or language. This challenged the view that using all included studies was exhaustive.
Whereas at least some participants considered a process-oriented approach was feasible with FCS, all of the participants were sceptical of this for web searches. A typical viewpoint was, ‘whatever you’re doing is a compromise, because you can’t do the type of search that you want to do.’.. (O). In addition to the aforementioned lack of reproducibility, participants described basic search interfaces which severely limited the type of search it was possible to conduct: it’s very difficult to search in a systematic way with the tools that are often … you know, with the very poor tools that are actually available where you can’t do even phrase searching in a lot of websites or … or when you do phrase searching, it’s an AND search. (C)
This necessitated a pragmatic approach to searching, using trial and error with each newly encountered website to achieve the desired end. Demonstrating methodological rigour was considered unrealistic. Although participants were aware of searching guidance, they considered it of limited use on a case-by-case basis: The guidance and the literature, I think, is sparse in this area. So I don’t think it’s a matter of doing what you’re, you know, supposed to do or not supposed to do. I think it’s […] a question of not knowing what you’re supposed to do and doing what works for the particular review. And then I think you develop over time. I’m sure lots of information specialists feel this way as well. You develop your own practice based on what’s worked in the past, and so you continue to do it that way in the future. (N)
For both FCS and WS, although outcome-oriented searches could theoretically be pre-specified and explicitly described, their fullest realisation typically seemed to depend on being responsive to developments in a review and to the searchers’ immersion in the searching process as the review got underway. We explore this in more detail below.
3.2. Persistent searching and adaptive searching
Persistent searching uses multiple attempts to search a resource to identify relevant studies. Persistent searching was sometimes contrasted with exhaustive or comprehensive searching. Whereas comprehensive searching aims to identify all relevant studies within a resource, often with a single large-scale search, persistent searching acknowledges that, for some resources, this is not possible due to limitations of the search interface. Thus, persistent searching aims to identify as many studies as is feasible using multiple smaller-scale searches. The related adaptive searching was detected as complementing persistent searching because of the need to adapt searches when searching persistently to retrieve different results. Adaptation was typically informed by feedback from initial attempts at searching. Adaptive searching was contrasted with standardised or uniform searching, which aims to retain the same approach for consistency between different resources.
3.2.1. Persistent searching
Persistent searching emerged as a strategy for dealing with search interfaces, which are not amenable to complex search strategies, described by one participant as ‘difficult [websites] to search’: [I]f it’s a difficult one to search I usually search it as much as I can, but then I sometimes use Google advanced search to search [using] the website URL … [I]t’s another way, sometimes, to see if I can find anything else. (A)
Here, a persistent approach is clearly described, searching a website ‘as much as I can’ before searching it further via the Google Search Advanced Search page. They went on to contrast this approach with using a single search for a bibliographic database which aims to retrieve all relevant studies: [W]ith a database, once you’ve got your search strategy you can feel like, ‘I’m confident that this will pick up, hopefully, everything that there is as far as precision and sensitivity and so on go’. It’ll pick up the right papers. But with a web search, because of the nature of them you can’t have that confidence from one search. Usually you need to try quite a few different things. (A)
The contrast between the persistent web search and the single or comprehensive bibliographic database search was echoed by other participants, for example, ‘You put some terms in, see what happens, put some more terms in and it’s not like that final MEDLINE search where this is definitely what we’re definitely doing’ (G). This iterative and developmental nature of persistent searching was sometimes seen as haphazard (‘web searching is just messy’ [O]); however, there was still an element of discernment and refinement when persistently WS: I would try and identify … from my database search strategy […] the most highly discriminating terms and search on those, and, again, I wouldn’t necessarily map those out in advance before I started. It would be, like, trial and error, and just seeing how it responded, did I think that the returns I’d tried had been effective, was I retrieving what I would expect to retrieve? If I hadn’t, I might then start … start putting in kind of additional terms. (O)
This approach of trying to identify the most discriminating terms as a starting point was mentioned by several participants. However, there was an acceptance that, unlike bibliographic databases, an objectively optimal way of searching was unlikely to be found in view of uncertainties about search algorithms and unstable content on websites: ‘I suppose the databases are much more of a controlled environment […] So there is that feeling of being able to tinker and improve something in a way that’s objective that you can’t do with web searching’ (G).
One participant, who mainly worked on configurative reviews, described persistent searching when using FCS, which entailed using studies identified by FCS for further citation searching (also described as ‘citation snowballing’ in the literature) [44]. This was deemed useful for gaining an understanding of the context in which research is produced: You get that kind of broader picture of where the literature is coming from … I think a lot of reviews are very focused on the content of the studies they include … and then almost miss the bigger picture, like, the wider context in which the research was produced. And I think the more citation searching you do, and, you know that feeling of ‘Oh I’ve stumbled into this nest of studies again’. (J)
This approach was seen as particularly useful for configurative reviews where the perspectives of the producers of studies were perceived as connected to the outcome of the research.
Due to its iterative nature, the participants noted that persistent searching requires the searcher to know when to stop in order to be manageable within the time constraints of a systematic review: ‘the reason I’m strict with my myself about it is … is, like, not because I think it’s wrong, but because three hours will have passed’ (J, FCS). Understanding how to stop searching depended on knowing the purpose of the search. In the context of WS, the participants gave several examples of targeted searches which aimed to identify a known study or studies with a narrow focus. For example, one participant described using web searches to investigate ongoing studies identified via other means: ‘you think, oh, that was three years ago, I wonder if they’ve actually published anything to do with this’ (K, WS). In a broader but still clearly bounded example, another participant described how WS ‘may be for a particular purpose [such as] you’re only looking for more views studies or … you take a look at the geographical focus … or it may be one certain stakeholder perspective’ (B). These searches could stop when the information need was fulfilled. However, if the purpose of the search was open-ended, the stopping point was harder to define. A broadly outcome-oriented approach, which was persistent within the bounds of usefulness, was detected in these scenarios. For example, one approach when using a search engine was a stopping rule, often after 100 results were screened, or when two or three pages were screened with no relevant results. It was also considered helpful to identify specific websites to search, rather than relying on a search engine, which would give a sense of boundedness. Within these bounds, a persistent approach was used; for example, using the ‘site’ command in Google Search to search websites: ‘if you search with the site command in Google, it’s often quite good, and I have the impression that […] I would find things there that aren’t accessible through the website itself any more’ (J).
3.2.2. Adaptive searching
The participants considered that persistent searching should be combined with adaptive searching by incrementally altering a search with each new attempt at searching a resource. To this end, participants adapted search terms using feedback from the results that they retrieved from previous searches, or simply to ensure that a wide selection of search terms from the bibliographic databases were used for the web searches. However, adaptive searching was also used to be sensitive to the context of the resource that was being searched. One participant gave an example of searching websites for user involvement studies: you start to learn the terminology that people are using … Different organisations have their own terminology for what they mean by ‘user involvement’. So some people call it ‘advocacy’, another organisation will call it something else, ‘patient focused’. So you quickly pick up that you’re not using … you can’t use the same language in each resource, because in fact, the culture of the organisation that you’re searching has a role to play in what you’re going to find. (C)
They concluded, ‘if systematic is being the same, it’s quite hard to be systematic. What I think people need to be is adaptable, and … and tailor what they’re doing to be appropriate to the website they’re searching’ (C). Consistency was still sought, however, if at all possible, rather than introducing variation almost at random: ‘I would try to be consistent [when selecting search terms]’ (E, WS, our emphasis). Participants acknowledged that adapting searches for different interfaces was difficult to do with confidence that a resource was being searched appropriately: It’s not that you don’t know what you’re doing. But like every time it’s a new thing that you’re having to get used to and having to figure out the quirks of. So there’s sort of less confidence in like have I done that right? (M)
The adaptive process was thus also closely aligned with a persistent process in the attempt of a searcher to satisfy themselves that they have searched a resource appropriately.
Several participants described how, because FCS does not typically take place until after the bibliographic database searches have been screened and an initial number of included studies identified, it is not clear until relatively late in the review process whether additional searches are needed to identify a suitable selection of studies (particularly for configurative reviews), or whether there was enough time and resources to screen the results of FCS (either for aggregative or configurative reviews). Adaptiveness was necessitated to take account of these variables, which also drew on outcome-oriented considerations to maximise the usefulness of the approach. One participant noted that FCS was sometimes written into a protocol as a potential ‘reserve’ search method, using an example of a configurative review where FCS was reserved for identifying additional studies if required. Discussing a realist review protocol, they noted that: we used lots of phrases […] in the protocol like ‘if necessary’. ‘If necessary, we will …’ [use FCS]. I guess probably what I’ll do is, there’ll be some studies that will seem to be contributing a lot of data to the review, and those are the ones that I’ll go and look for forward citations. (J)
Once FCS was underway, review teams were sometimes surprised at how many results were retrieved and this necessitated re-visiting the approach that was taken: Sometimes people say well, ‘let’s do the citation search’ and I think they’re assuming that we’re not going to find that many hits … And they’re quite surprised sometimes when we find hundreds or thousands … and then we’ve got to kind of go back and re-think the whole thing again. (D)
3.3. Critically engaged searching
The participants described how outcome-oriented and persistent searching required an understanding of how successful a search was at retrieving relevant studies. From this perceived need to be alert to the content of search results emerged ‘critically engaged’ searching. Critical engagement extends the searcher’s involvement beyond the practical knowledge required to develop and carry out search methods to a reviewer role in study selection.
Participants reported that searching and screening would sometimes overlap: There’s an element of screening that’s involved with both forward citation searching and web searching and reference checking as well … And so there’s always a discussion with the author team who’s going to conduct that work because practically if I’m going to check all of the citing references for a particular study, I need to decide if they meet the inclusion criteria. (N)
This overlap was universally acknowledged when carrying out WS: ‘you need to be really familiar with the inclusion and exclusion criteria, [because] you’re almost kind of screening as you go along’ (K). This overlap of roles occurred partly because search engines and websites rarely have a function for exporting content to reference management software: ‘a challenge … is exporting results, because you usually can’t export them. Usually we have to put them manually into our reference management which is time consuming’ (A, WS). Thus, searchers often elected to select relevant results themselves. Similarly, the precision of web searches was typically low due to basic search interfaces and the wide range of content that was indexed; thus, it was considered inefficient for every identified item to be screened by a second reviewer: ‘If in doubt I would send it to [the review team], but there is an element of me looking at titles thinking, “Oh, that’s nothing to do with this”’ (G). A small number of participants were disinclined to carry out WS themselves because of the need to be familiar with the review content and because they felt that WS did not require specialist search skills: ideally, all of it gets moved down to the reviewer or their research assistants, because I don’t think that it takes necessarily librarian skill to read websites and pick out either citations or interesting bits of information […] I mean, ideally that would be a job for someone else on the review team with content expertise. (L)
This was sometimes done with a view to focusing the available expert searcher time on the more advanced search interfaces of bibliographic databases. One participant who expressed this view in a research consultancy setting was doubtful about the scientific credibility of WS due to problems of reproducibility’ and appeared to consider that investing their time in WS risked ‘validating’ it as systematic search method in the eyes of reviewer colleagues. When FCS, there was the option to export the results which meant it was less common for information specialists to be involved in screening. However, some participants were involved to divide the screening labour: ‘Most screening teams are pretty exhausted by this time, so they elect for me to do a little bit of extra screening on their behalf’ (N).
For participants to be involved in identifying studies, they needed to be critically engaged with the content of the search results. However, the approach they used to select studies was not necessarily determined by the strict application of inclusion criteria. Instead, participants often took an inclusive approach to selecting studies for potential inclusion in a review: ‘I will look [at the results] and if I think there’s the slightest chance it might be relevant then I’ll include it’ (A, WS, our emphasis). This approach was guided by the fact that if the searcher excluded studies which were not exported to reference management software, they would not be seen by a second reviewer. Although citation indexes do facilitate exporting of references, some participants used a similar approach when using Google Scholar to carry out FCS, which has more limited export features than subscription-based citation indexes: ‘I used it just using the regular interface and then scanning the … the references and going through them manually so if I did that then I would have to be quite involved in that bit of the review’ (B). One participant described how there are sometimes mistakes in the citations of studies: you have to be fairly imaginative to take account of the ways that people might have cited a paper, there can be variations and mistakes. So … you need to be looking out … you need to be actively searching for mistakes as well as the real reference that’s been accurately cited. (C)
When using Google Scholar, this extended to searching for mistakes in the citations of studies which were introduced by the automated algorithms which the search engine uses to retrieve citing studies.
The participants noted that they learned about the performance of searches from reading the search results in detail. The search performance feedback gained from critical engagement with the search results was then used when searching persistently to adapt subsequent searches.
3.4. Holistic searching
Holistic searching seeks to ensure that search methods complement each other. To this end, participants assembled search methods, including FCS and WS, into a coherent strategy based on what was consistent with an overall plan. This was contrasted with approaches to searching which focus on assembling search methods, including simply long lists of bibliographic databases, without considering how they work together.
Demonstrating holistic searching, one participant noted that ‘as time goes by, I’m not interested in just putting together a list of databases to show that I’ve searched everywhere. I want to have some rationale for my choices [of search methods]’ (L). Similarly, other participants noted how the ‘default’ option of compiling a long list of databases was not necessarily the optimal way of finding relevant studies for a review: From someone who has supported just tons of very […] ‘systematic’ systematic reviews, and was from that school of comprehensiveness […] the big comprehensive database search is … the linchpin of that, and everything else is additional. But, like … now because I do more chaotic reviews [i.e. configurative reviews], I think that I’m not even that confident that it is the most useful thing. I think it’s probably just the … the big thing that you do first, and then it feels like a lot of the studies that are included in reviews come from it, because that’s where I looked first. (J)
This quotation highlights how the bibliographic database search is often viewed as the main search method and other search methods are relegated to additional methods. In contrast, holistic searching was often associated with assigning a more prominent role to other search methods, such as FCS and WS, alongside searching bibliographic databases. In this respect, holistic searching shares similarities to the already existing literature on using traditionally viewed ‘supplementary’ search methods as ‘complementary’ search methods [23,33]; however, we suggest that there is more emphasis in holistic searching on how the search methods fit into an integrated whole. Discussing how FCS could have a more prominent role in searching for studies, one participant noted that, in the context of large-scale public health systematic reviews, I feel like in a lot of cases what I would like to do is do a much simpler PICO search [i.e. bibliographic database search], not try and bottom out every single synonym and all those kind of things … I have done that for topics where … I did forward and back[ward] citation searching and that was the first bit of the strategy. And I was almost using the PICO element after that to fill in for the stuff that we didn’t capture there. (M)
In this example, using FCS as a starting point might be a more effective or efficient way to identify studies initially, and this can be complemented with a smaller database search to identify studies that might have been missed.
Holistic searching often overlapped with outcome-oriented searching. In particular, an outcome-oriented approach to one search method would typically have strengths and weaknesses which could be balanced by combining it with other search methods which used a different mechanism for retrieving studies: none of these individual methods, particularly in a review that isn’t the traditional kind clinical, efficacy, RCT type review, those more kind of public health, social science reviews, […] feels like it’s enough, but when you use the individual methods together and they start to overlap [inaudible] database search and citation search, and then you might check references and then you might contact people working in that field for extra stuff, all … all those individual methods, as you know, kind of allow for the inadequacies of other methods. (O)
Specific examples were mentioned with respect to FCS and WS: I’m never just relying on that [i.e. WS] on its own […] I’ve already got results from databases and results from this that and the other … it feel likes there are enough facets to the places that information’s coming from that if … if that one is in a little bit of a bubble, that’s okay. (J)
(‘Bubble’ in this quotation refers to the problem that web search results are tailored to the searchers’ search history and geographic location).
Decisions about how to develop holistic searching typically relied on the participants’ experience and expert judgement. Participants did sometimes have ‘maxims’ or evidence-based criteria for making a decision; for example, discussing searching for diagnostic test accuracy reviews, one participant noted ‘there’s sort of a base assumption that PICO works, which if you look at the filter studies and things like that, and diagnostic filters, we know for a fact that in some cases it just doesn’t’ (M). However, the many varied situations that a searcher might face made knowing exactly how to approach a search dependent on ‘instinct and experience’: it’s a judgement call. I guess the fundamental problem in our business is if you want to falsify what you’ve done, if you want to say well, that wasn’t the best way to do it, you’ve got to have a tonne of resources to do it again better […] So I go on instinct and experience. (M)
The perception that it was sometimes hard to explicitly formalise why a particular set of search methods was preferable reflected the tension between the acknowledged aim of systematic reviews to use a rules-based approach and the underlying reality that judgements which rely on the discretion of the review team are inevitable. This was perceived to be a phenomenon which was not only limited to the searching component of a review but also related to wider questions about systematic reviews: I think we probably need more clarity in terms of what people want from it [i.e. searches for studies], and what they are aiming at in terms of the final product. I can’t give you that definition of a systematic review. I don’t know whether people prize transparency more or they prize speed more or … yeah. I don’t know what it is. (M)
4. Discussion
This study has identified and described five habits of phronesis which expert searchers use when searching for studies for systematic reviews using FCS and WS: outcome-oriented; persistent; adaptive; critically engaged and holistic. Our findings suggest that it is not always feasible, or desirable, to approach searching for studies using formalised and a priori described methods. Instead, review teams should make allowance within pre-specified descriptions of searching to be responsive to study identification needs as they arise in the review process, and to dimensions of searching (e.g. ‘persistent’ and ‘adaptive’) which only become apparent when searching is underway. This may also mean, as Moreira [9] and Boell and Cecez-Kecmanovic [8] have also shown, appreciating the non-linearity of searching and reviewing tasks as the ‘critically engaged’ searcher develops an understanding of a review topic through screening the results of searches which feeds back into the development of additional searches.
Traces of the identified habits of phronesis are detectable in guidance and expert commentaries on searching, for example, the recommendation of repeated attempts at searching a website using different search terms [6,26] or purposively selecting ‘key’ studies as source studies for citation searching [2]. These approaches are also visible in reporting of search methods in systematic reviews [31,32]. However, the unique contribution of this study is the articulation of practical reasoning that shapes the development of searching, rather than the specific processes involved in searching. Indeed, it is inherent to phronesis that its dependence on the expert judgement of the practitioner means that it cannot be mechanistically applied. As McDowell says of the possibility of drawing up a code of principles which are sensitive to practical knowledge, however subtle and thoughtful one was in drawing up the code, cases would inevitably turn up in which a mechanical application of the rules would strike one as wrong – and not necessarily because one had changed one’s mind; rather, one’s mind on the matter was not susceptible of capture in any universal formula [16].
Variables which led to the requirement of phronesis in this study included factors such as the aims and objectives of a systematic review, the needs of the end-user, and the time and resources available. Relevant factors were also identified relating to the tools which were used, particularly for WS, wherein the mechanical application of rule-following could lead to suboptimal searching. The difficulty of articulating a rules-based account of searching for studies caused some of the participants to reflect on whether the required judgements fitted within the framework of a scientific endeavour which aimed for methodological rigour. To accommodate for this, a balance was sought between the pursuit of science and the development of a product that was useful for the end consumer, who, for example, would prefer that studies were included rather than omitted because the methods used to identify them lacked methodological rigour. This does not, however, mean that techne is irrelevant for systematic reviews, or that technical guidance is not valuable – but rather that it is unrealistic and indeed not desirable to develop, carry out and assess the validity of systematic reviews entirely by the application of pre-specified criteria and rule-following. Lorenc at al. [10] report similar findings on how reviewers work with heterogeneous data for which guidance is limited, seeking a fine line between applying methods in ‘rigid and uninformative way’ and compromising the integrity of a review by taking too relaxed an approach. Furthermore, both this study and Lorenc et al. [10] found that expert judgement in making these decisions may remain unarticulated in even the most detailed description of methods conduct.
All of the identified habits of phronesis were shown to be distinct from formulised or rule-following approaches to searching (i.e. techne), but perhaps the strongest contrast was between outcome- (phronesis) and process-oriented (techne) approaches. By focusing on the outcome of searching rather than the process, outcome-oriented approaches explicitly challenge the central importance of methodological rigour when carrying out a systematic review [3,45,46]. In particular, a key strength of systematic reviews is that studies are identified by a rigorous searching and screening process using predefined inclusion criteria which prevents bias arising from ‘cherry picking’ studies for inclusion [4,5]. Thus, a risk of outcome-oriented approaches is that searches fail to retrieve studies that are less immediately identifiable, for example, which are not cited by a ‘core’ set of studies used for FCS, or which are relatively hidden within websites or ranked lower down a list of search engine results. Indeed, outcome-oriented approaches may be susceptible to confirmation bias if search parameters are narrowed without due consideration, such as focusing on websites of organisations with a similar perspective on a phenomenon of interest or using studies which report positive results for FCS [47]. To mitigate this problem, it may be helpful to extend the ‘critically engaged’ habit of phronesis to include awareness of the potential for confirmation bias to influence outcome-oriented searching. This might mean searching the websites of a suitably varied set of organisations or using studies for FCS which disconfirm the dominant findings of a review [48]. There is also a long-standing awareness of the value of WS to identify studies which are not published in journal article format as a strategy for tackling publication bias [49,50]. Adaptive and persistent approaches can also be used to extend outcome-oriented searching, and, as participants noted, potential shortcomings of one search method can be mitigated by combining it with other search methods.
Regarding ‘adaptive’ and ‘persistent’ searching using a search engine, it was interesting to note that the participants did not emphasise the need – as recommended in systematic review searching guidance – to clear search histories between carrying out searches to minimise the personalisation of search results based on relevancy feedback from earlier attempts [6]. We suggest that, in view of the aims of adaptive and persistent searching, relevancy feedback may be useful for encouraging a search engine to rank potentially similar items higher in a list of search results; however, when commencing searching for different content, clearing search histories remains useful.
In keeping with the overall approach to HPA, we sought to achieve a ‘fusion of horizons’ which was both expressive of the group as a whole while remaining sensitive to different perspectives [38,42]. This was mainly achieved through attentiveness to contrasting views among participants on whether phronesis or techne was more appropriate, with the emphasis on the practical reasoning which underpinned searching decisions rather than specific cases where they were considered more or less helpful. Sometimes, however, it was possible to articulate factors which led to different perspectives; for example, there were different perspectives on process-oriented (techne) versus outcome-oriented (phronesis) approaches to FCS which were in part reflective of whether participants mainly worked on aggregative or configurative reviews. Yet, even where contrasting views could be linked to different contexts, they were rarely held in an absolute sense, and participants showed awareness of the potential value of the habits of phronesis for developing and carrying out searches for studies for all types of systematic review, from the more ‘traditional’ rule-following Cochrane review to the more exploratory realist review. Hypothetically, differences in knowing when and how to use phronesis might reflect how much experience someone has, in view of how expertise is acquired overtime through practice [51]. However, any such differences between the participants in this study were not identifiable, which perhaps reflects that all of the participants had several years of experience of searching for studies – specifically, only 1 of 15 participants had less than 9 years of experience.
4.1. Strengths and limitations
The participants exhibited a depth of searching experience which was based on many years of practice. Although the participants all worked in health and social care research settings, we think that the findings will be applicable to other topics areas, as the methods used for systematic reviews more widely are probably similar. Indeed, it is unlikely that other topic areas have as extensive guidance on systematic review methods as health and social care research, and they may also have more limited bibliographic database resources and indexing standards, thus the need for expertise on using supplementary search methods may be increased. Although we focused on FCS and WS, these are not peripheral methods, as evidence attests [35,36]. However, there is scope to extend the investigation of expert judgement to other search methods, for which reference to the design of this study may be helpful as a guide. The use of HPA facilitated in-depth investigation of expertise which has hitherto not been explored. We did not, however, iteratively contact participants to follow up lines of investigation, as is recommended in HPA guidance [38]. This was mainly due to the time required to carry out iterative interviews with a large cohort of participants during a challenging time for both interviewer and interviewees during the COVID-19 pandemic. Instead, we sought to exhibit validity using Yardley’s [43] principles of commitment, rigour, transparency and coherence.
5. Conclusion
The expertise of information specialists in searching for studies for systematic reviews extends beyond the formal rules-based approaches set out in guidance and evidence-based practice, and challenges the convention of pre-specified and explicit step-by-step reporting of methods in protocols and systematic review publications. This analysis contributes to accounts of how systematic reviews unfold in ways that may not be explicitly avowed, drawing attention to searching as an area of systematic reviews that is central to their credibility but as yet poorly understood in terms of its expert practice.
Supplemental Material
sj-docx-1-jis-10.1177_01655515221130237 – Supplemental material for The phronesis of expert searcherson using forward citation searchingand web searching to search for studies for systematic reviews: A hermeneutic phenomenological analysis
Supplemental material, sj-docx-1-jis-10.1177_01655515221130237 for The phronesis of expert searcherson using forward citation searchingand web searching to search for studies for systematic reviews: A hermeneutic phenomenological analysis by Simon Briscoe, Rebecca Abbott and G J Melendez-Torres in Journal of Information Science
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
