Abstract
Context:
In response to the growth of evidence-based practice in social work, systematic literature reviews offer significant value to social work but are often met with concerns of time scarcity.
Purpose:
Through a case study search strategy addressing the research question “What are practicing frontline social workers’ experiences of bureaucracy?,” this article seeks to promote efficiency by providing a practical guide for conducting systematic literature searches and an appraisal of database performance in qualitative social work research.
Method:
The total citations, unique hits, sensitivity, and precision for each database were calculated before conducting a cross-study comparison with three previously published social work systematic searches to identify emerging performance trends.
Results/Conclusion:
Relying on a single database is subject to bias and will not provide comprehensive or sensitive findings; however, due to consistent high performance across four systematic searches, Applied Social Science Index and Abstracts, Social Services Abstracts, and Social Science Citation Index are recommended for future literature searching in social work.
Why is an intervention more effective with one cohort and not another? To what extent does policy affect outcomes for service users and practitioners? What are the factors that influence these differences? Evidence-based practice and evidence-based policy making continue to grow in importance, seeking to understand differences in outcomes, appropriateness of policies or interventions, and questioning what works, why, and for whom (Saini & Shlonsky, 2012; Shek, 2008; Thomas & Harden, 2008). Decisions that shape interventions and policies need to be trustworthy, reliable, and comprehensive, increasing the demand for systematic literature reviews (Holden et al., 2008). The body of research available to support social work practice continues to grow. With increasing challenges in remaining up to date to ensure best practice, systematic literature reviews are key to accessing and appraising potentially relevant evidence, providing a comprehensive body of empirical research (Saini & Shlonsky, 2012).
While literature reviews can be selective in what research is included, a systematic review seeks to minimize bias by appraising and summarizing all available evidence in a rigorous and transparent way, in accordance with a predetermined set of inclusion/exclusion criteria (Fisher et al., 2006; McKenzie et al., 2019; Saini & Shlonsky, 2012; Soilemezi & Linceviciute, 2018). Initially limited to randomized controlled trials in medicine and health, the importance of systematic literature reviews has spread beyond this origin and is now applied to a wider range of methods in both quantitative and qualitative research to inform practice and policy development (Dixon-Woods et al., 2006; Fisher et al., 2006; Garside, 2014; Saini & Shlonsky, 2012).
Debate continues around the meta-synthesis of qualitative research due to the specificity of context and participant population, wide range of methods employed, varied theoretical underpinnings, and appropriateness of drawing generalizations (Dixon-Woods et al., 2006; Saini & Shlonsky, 2012; Thomas & Harden, 2008). However, synthesizing a collective body of qualitative research enables a rich, comprehensive analysis to identify contradictions, exceptions, similarities, and gaps in knowledge, providing deeper insights on an intervention or phenomena than is possible when assessing an individual study (Erwin et al., 2011; Saini & Shlonsky, 2012; Soilemezi & Linceviciute, 2018; Thomas & Harden, 2008). The use of systematic literature reviews for qualitative research is a developing methodology and not as well established or evaluated as approaches employed for quantitative research (Flemming & Briggs, 2007). Relying exclusively on quantitative data and meta-synthesis for evidence-based practice, however, risks implementing harmful policies or interventions. Not all outcomes can be evaluated through quantitative measures, and an exclusion of qualitative research can overlook important implications (Ferguson, 2008). Despite the continued time scarcity and increased publications in the field of social work, a commitment to rigorous and transparent systematic reviews increases accessibility by presenting evidence in a clear, forward manner (Saini & Shlonsky, 2012).
Additionally, in the context of social work where time is constantly constrained, up-to-date assessments of database performance is vital to inform practitioners, policy makers, and researchers on appropriate sources for attaining relevant literature. As stated by Shek (2008, p. 504), “with the intensification of the evidence-based practice movement in social work, the role of a systematic and accurate database is of grave importance”; however, there is a dearth of reporting on database performance in systematic literature reviews. Therefore, it is essential that studies continue to report and critically analyze the systematic review process and database performance to further strengthen the skills and knowledge needed to conduct rigorous searches on qualitative research while balancing comprehensiveness with time investment.
Method
This study applied a systematic literature search of qualitative research to address the question “What are practicing frontline social workers’ experiences of bureaucracy?” Each step is discussed, accompanied by a rational for decisions made in order to provide guidance for future researchers and practitioners. An overview of databases used with an evaluation of their performance is included to offer insight on the importance of database selection and how rigor and time can be appropriately balanced.
Adapting the steps proposed by McFadden et al. (2012) and McGinn et al. (2016), the systematic review and database comparison methodology for qualitative research followed 15 steps. Developing a research question Determining the inclusion and exclusion criteria Identifying a list of potential databases Testing the precision of potential databases Final selection of databases for systematic review Constructing concept groups and terms Developing search formula, including specific search facilities across databases Trial and adjustment of search strategy Running the searches making use of available facilities (Boolean phrases, index terms, filters) Screening the title and abstracts Appraising full articles Group consultation Create combined list of citations, manually removing duplications Calculating unique hits, precision, and sensitivity of each database Appraisal of database performance
Developing a Research Question
The impact of bureaucracy on social work practice has been a point of contention, widely discussed in literature and by professional Social Work Associations since the 1990s (see British Association of Social Workers, 2018; Carey, 2009; Harlow, 2003; Jones, 2001; Munro, 2004; 2011; Postle, 2001; Tsui & Cheung, 2004). Yet, a synthesis of empirical research to assess the implications of bureaucratic structures remains a gap in this body of knowledge. The systematic literature searching sought to answer, what are practicing frontline social workers’ experiences of bureaucracy?
Determining Inclusion and Exclusion Criteria
Clear inclusion and exclusion criteria are essential for determining what literature is relevant to the systematic review. The following criteria were developed for the purposes of this study: Period of publication restricted to January 1, 1990, and April 30, 2020. Databases differ in their schedules for updating and indexing literature (Shek, 2008). The 30th April was selected as the upper limit, 2 weeks prior to the first database search, to reduce index and update bias while still retrieving the most up-to-date publications. January 1, 1990, was selected as the lower limit, as 1990 is a key period in which new public management reforms began in the United Kingdom including the National Health Service and Community Care Act (1990; Ellis et al., 1999; Parry-Jones et al., 1998). Empirical qualitative research. Qualitative research focuses on unpacking the experiences and perceptions of phenomena in rich detail while considering context (Saini & Shlonsky, 2012), making empirical qualitative research the most appropriate for answering the current research question. Theoretical debates, critical commentaries, book reviews, and editorial notes were excluded. Quantitative methods were excluded due to the inability to be incorporated into a meta-synthesis. English language. Due to the linguistic limitations of the research team and lack of access to professional translation services. Available as full text. Where full-text copies were not available through the databases, interlibrary loans and e-versions were requested from the institutional library services. Where there were delays of 6 weeks or longer from the point of request, the article was excluded due to time limitations of this study. Peer-reviewed journal articles. Gray literature was excluded on the basis of practicality and accessibility, as discussed in depth in Step 8. Studies must report on social worker’s perspectives and/or experiences of bureaucracy or managerialism to directly answer the research question. Those which focused on the perspectives of manager, funders, service users, or their families were excluded.
Because the objective of this systematic literature review is to determine social workers’ experiences of bureaucracy, not the broader social services sector, the inclusion of literature was limited to studies which defined participants as social workers. Although social workers are employed across a range of settings and fields, an individual is not necessarily required to be a social worker or social work qualified to be a care manager, case worker, child welfare worker, health assistant, or social care worker; therefore, it couldn’t be assumed that participants under such job titles were, in fact, social workers. For example, in a study assessing the factors that influence career choices to stay or leave care management in adult services, participants included both social work and nurse qualified practitioners, demonstrating the interdisciplinary identify of care management (Bradley, 2005). In a further study assessing quality and effectiveness in child protection services in New South Wales, Australia, the rate of child protection workers who held a social work qualification ranged from 27.7% to 30.2% across practice sites (Cortis et al., 2019). These rates of qualification are evidence that an individual is not required to be social worker to gain employment as a child protection worker (Cortis et al., 2019). Lastly, in a Swedish study investigating the experiences of case management in the field of psychiatric disabilities, case managers were identified as psychiatric nurses, mental health nurses, or social workers, evidencing the diversity of professional backgrounds represented under the title of case manager (Markstrom et al., 2009). To reduce ambiguity in the applicability of findings to practicing frontline social workers, studies were only included if participants were recognized explicitly as social workers in their employment capacity or by qualification.
The inclusion of research which incorporates several occupational or professional groups can present limitations to a systematic review by increasing difficulty in defining the relevant participant populations and interpreting the relevance of results to a specific profession (Wirth et al., 2019). This was managed wherein studies with participants from multiple professional backgrounds were only included if results were disaggregated by professional identity. Disaggregation was essential to identify the specific contributions of social workers to the research and findings.
Identifying Potential Databases
While there have been calls for a sensitive and comprehensive social work database in response to the growing demand for evidence-based practice and policy making (McFadden et al., 2012; McGinn et al., 2016; Shek, 2008; Taylor et al., 2007), this is yet to be achieved. Although Social Work Abstracts (SWA) has been considered the “flagship database” for social workers (Shek, 2008, p. 500), an assessment of database performance and the coverage of articles over a 10-year period (1995–2005) compared with PsycINFO, Sociological Abstracts, and Medline found SWA lacking. The study indicated inclusion bias with different treatment across various social work journals, favoring publications produced by the National Association of Social Workers (Shek, 2008). Further deficiencies with SWA included inconsistent indexing, quarterly database updates (where comparative databases were daily, weekly, or monthly), the smallest number of journals included, and a reduced number of social work–related citations.
Inconsistency in SWA was also noted in a longitudinal evaluation of the database (Holden et al., 2008). Assessing the inclusion of 23 social work relevant journals across the period of 1997–2005, SWA failed to contain a complete issue-level coverage for 22 of the 23 journals searched. When compared to the performance of PsycINFO, SWA performed worst across 14 of the journals in question. While the inclusion of journal issues improved over time for PsycINFO, the proportion of missing issues for SWA was “consistently inconsistent,” indicating no sustained improvements over the period (Holden et al., 2008, p. 496). The study supported the findings of Shek (2008), noting SWA’s inclusion bias toward publications produced by the National Association of Social Workers. In a replication study addressing the coverage of core journal issues from 1989–1996, a continuation of inadequacy in SWA was found (Holden et al., 2014).
Consequently, both Holden et al. (2008) and Shek (2008) concluded that SWA cannot be relied upon solely for comprehensive literature searching.
The use of multiple databases
Upon comparing the performance of three previously published systematic literature reviews in the field of social work, no consistency on the best performing database was identified based on indicators of unique hits, sensitivity, and precision. Unique hits indicate the number of relevant hits retrieved that were not present on any of the other databases searched. Sensitivity is the capacity to include all existing relevant literature (McFadden et al., 2012). Sensitivity is calculated by dividing the number of relevant hits a database retrieved by the total number of unique relevant hits from all databases included in the study. Precision is defined as the number of relevant hits divided by the total number of citations retrieved by a single database. Precision is a calculation of how accurately the database excludes irrelevant results (McFadden et al., 2012).
As seen in Table 1, the results present the top three performing databases for each study. These reviews are field-specific, covering areas of child protection social workers’ resilience (McFadden et al., 2012), intimate partner violence programs (McGinn et al., 2016), and decision making in institutional care for elder people (Taylor et al., 2007), indicating that the performance of a database can vary depending on the research question and topics engaged with.
Database Performance Based on Prior Systematic Literature Reviews in Social Work.
Note. ASSIA = Applied Social Science Index and Abstracts; SSCI = Social Science Citation Index; SSA = Social Services Abstracts.
Relying on a single database provides an inadequate scope of literature. In a study assessing the coverage, indexing, and search capacity of SWA compared to Social Services Abstracts (SSA), the authors found the two databases to be complementary (Flatley et al., 2007). Although SSA produced more unique hits in the test searches and a greater journal coverage, the overlap between the two databases (exact hit matches) was reported at only 12%, showing that the databases indexed different content (Flatley et al., 2007). This comparative study emphasizes how, despite the increased work and time required, multiple databases are essential to increase comprehensive searching, ensure the inclusion of all relevant literature, and reduce inclusion biases. Such findings are echoed beyond the social work profession, warning how an overreliance on one database produces inadequate results to conduct an accurate systematic literature review (Brettle & Long, 2001; Stevinson & Lawlor, 2004).
Because social work is a broad profession that engages in a wide range of social issues including mental health, rehabilitation, elder care, education, and refugee/migrant populations to name but a few, 22 databases were considered for inclusion for the current study. These were identified through previous social work–focused systematic literature reviews (McFadden et al., 2012; McGinn et al., 2016; Taylor et al., 2007) and assessing databases with a focus on social science research which were accessible through the institution’s licenses.
Testing the Precision of Potential Databases
Following the steps applied by McGinn et al. (2016) to determine the final selection for the systematic literature review, a test search was conducted across each potential database to assess the level of precision, a calculation to determine which databases would return the most relevant results for a systematic review (McFadden et al., 2012). The test search applied the term “Social work” and, where the database permitted, the term was matched to subject headings. The titles and abstracts of the first 40 English results were reviewed to assess the presence of the test search term and relevance of its application. The results are presented in Table 2.
Results of the Test Search Conducted on 22 Potential Databases.
Final Selection of Databases for Systematic Review
Prioritizing precision, the inclusion of databases was limited to a 90% test result. Index to Thesis was excluded on the basis of limiting the search to peer-reviewed articles (see Step 8 for full discussion). ChildLink and Safeguarding Adults at Risk Information Hub were excluded because the databases had insufficient facilities for running a complex search. Both were unable to utilize Boolean operators to combine terms (De Brun & Pearce-Smith, 2009). ChildLink could not filter search results according to publication type and limited a search to a maximum of 128 characters, including spaces, which was inadequate to include all necessary concepts and terms. Based on this process, the final list was reduced to nine databases for inclusion, Applied Social Science Index and Abstracts (ASSIA), Child Development and Adolescent Studies (CDAS), International Bibliography of the Social Sciences (IBSS), PsycINFO, Social Care Online (SCO), Social Science Citation Index (SSCI), Social Services Abstract (SSA), Sociological Abstracts, and Sociology Database.
Although numerous databases could have been searched through the Social Sciences Premium Collection, reducing the duplication of results and effort, the author decided against this amalgamation in order to draw comparisons from previous systematic literature reviews in social work. Furthermore, the author recognized that not all practitioners, organizations, and institutions have access to the Premium Collection. To make the results more applicable to everyday practice, separate searches were completed.
Constructing Concept Groups and Terms
The following three concepts were derived from the research question.
“What are practicing frontline social workers’ experiences of bureaucracy?” Social worker Experience Bureaucracy
Terms
Relying on a single key term for each concept raises problems as it does not allow the search strategy to explore synonyms and therefore could exclude highly relevant results (Beall, 2007). A key step in developing a search strategy is developing a comprehensive list of relevant terms grouped by concept. The search terms developed for each concept are displayed in Table 3, drawing on key words in the abstract and title of relevant background literature and known synonyms from the authors’ practice background.
Search Concepts and Terms Employed for the Current Study.
A filter for qualitative methods was not used due to the inconsistency in indexing methods across databases and unpredictable inclusion of method detail in titles, key words, or abstracts (Saini & Shlonsky, 2012). Limiting the search strategy to qualitative methods risks excluding relevant studies which have not indicated their qualitative approaches in the initial text searched by the database. Determining inclusion based on methods was conducted manually when reading abstracts and full-text articles in Steps 10 and 11.
Developing Search Formula, Including Specific Search Facilities Across Databases
A draft search strategy was developed for databases hosted on the Proquest platform using the Boolean phrase “AND” to combine terms and “OR” to include the various terms (De Brun & Pearce-Smith, 2009): (“Social work” OR “social service” OR “case management” OR “care management” OR “case work” OR “human services”) AND (Experience OR perspective OR belie* OR response OR reaction OR perception OR opinion) AND (bureaucra* OR managerialism OR “new public management” OR “performance management” OR audit OR “paperwork” OR “form-talk”).
Trial and Adjustment of Search Strategy
The draft search strategy was tested on Proquest ASSIA. This was selected as ASSIA is a database reported to have high sensitivity (McFadden et al., 2012, McGinn et al., 2016) and high precision (McGinn et al., 2016) and used Proquest, the operating platform for five of the nine included databases. Having completed a trial run of the search strategy on ASSIA, the term “audit” was found to produce irrelevant results, being applied as a method of research or policy and program evaluation process, not as a concept relating to experiences of bureaucracy or managerialism in frontline practice. Systematic literature reviews require considerable time and skill investment (Soilmezi & Linceviute, 2018), and to prioritize precision and ensure the highest number of positive results with the least amount of negative hits, the term “audit” was removed from Concept 3.
Gray literature
Following the test search, the inclusion criteria were limited to peer-reviewed articles to restrict results to high-quality research. Beyond an ethical approval processes, the peer-review process is the key mechanism to appraise the quality, methodological rigor, and originality of research before determining adherence to publication standards (Taylor et al., 2007). McFadden et al. (2012) intentionally included theses through the use of the “Index to Theses” database, retrieving 30 hits, none of which were relevant to their research question. After testing the search strategy on ASSIA, the strategy was run through an equivalent database for theses “Dissertation and Theses A&I” on Proquest. Over 1,400 citations were retrieved, even once limited to doctoral thesis and filtered by the time period and English language requirement. Furthermore, the Sociological Abstracts database retrieved more than 830 theses when applying the same search strategy. These tests evidenced substantially more hits than those retrieved by McFadden et al. (2012).
As many theses are only available as printed copies in institutional depositories and with a number of thesis results, including gray literature was deemed impractical for the wider project and time limits imposed. Although there can be delays in the publication process (McGinn et al., 2016), as argued by Taylor et al. (2007), high-quality research presented at conferences and in theses are likely to pursue publication in peer-reviewed journals. Due to the significant time period used for the systematic literature review of 30 years, it is argued that the majority of high-quality theses and conference papers would have had the opportunity to pursue publication within the time period set.
Running the Searches Making Use of Available Facilities (Boolean Phrases, Index Terms, Filters)
The following searches were run.
Applied to ASSIA, CDAS, IBSS, SCO, SSCI, SSA, Sociological Abstracts, Sociology Database.
(“Social work” OR “social service” OR “case management” OR “care management” OR “case work” OR “human services”)
(experience OR perspective OR belie* OR response OR reaction OR perception OR opinion)
(bureaucra* OR managerialism OR “new public management” OR “performance management” OR “paperwork” OR “form-talk”)
Filters applied
Date: January 1990 to April 2020 with the exception of SSCI. The date limitation for SSCI was not month-specific, therefore any articles published after 30th April were removed manually.
Publication: Peer-review journals only, with the exception of SCO. The function “Restrict to A+” was used for SCO to limit results to journal articles.
Language: English
Term search limited to: Anywhere except full text (NOFT) for all databases operating on Proquest. SSCI which was limited to TOPIC which includes title, abstract, and key words. For the SCO and CADS database, no equivalent search function was available.
Strategy two: Applied to PsycINFO
social casework/or case management/or social services/
*human services/
*government/
bureaucracy.mp.
social work.mp.
*accountability/
managerialism.mp.
new public management.mp.
paperwork.mp.
*employee efficiency/ or *employee productivity/
performance management.mp.
experience.mp.
*perception/
*attitudes/
*emotional responses/
1 or 2 or 5
3 or 4 or 6 or 7 or 8 or 9 or 10 or 11
12 or 13 or 14 or 15
16 and 17 and 18
Filters applied
Date: 1990–2020. Articles published after 30th April were manually removed.
Publication: Peer-review journals only.
Language: English.
Term search limited to: Key word
The citations from each database are summarized in Table 4. Citations were imported into excel spreadsheets for screening, providing an audit trail. Search results for each database were saved separately in order to calculate the number of unique hits, rate of precision, and rate of sensitivity for each database.
Number of Citations Retrieved for Each Database Searched.
Note. ASSIA = Applied Social Science Index and Abstracts; CDAS = Child Development and Adolescent Studies; IBSS = International Bibliography of the Social Sciences; SCO = Social Care Online; SSCI = Social Science Citation Index; SSA = Social Services Abstracts.
Screening Title and Abstract
The title and abstract of each hit were screened by the first author. Although screening the title first for immediate rejection has been used as a strategy to reduce the number of citations requiring abstract appraisal and to save time (see Mateen et al., 2013), the decision to screen both title and abstract simultaneously was informed by the lessons of Jones (2004) and Soilemezi and Linceviciute (2018). Screening both title and abstract improves effectiveness, accuracy, and rigor, reducing the risk of relevant studies being missed based on insufficient or inaccurate detail in titles. This approach was realistic, given that the number of unique citations across all of the databases was only 509, compared to the 2,965 retrieved by Mateen et al. (2013).
To maintain an audit trail, the following color code was applied to all citations:
Red: Excluded
Yellow: Uncertain, full article review needed
Green: Include
Black: Duplicate
Building on best practice for auditability, all excluded articles were justified in one or two sentences at the time of exclusion. For example, “Perspectives of managers and admin staff, not frontline social workers,” “Healthcare focus but no mention of social work or social workers,” or “Critical commentary, not empirical research.” As this systematic review was being conducted during the peak of the coronavirus 2019 (COVID-19) global pandemic and UK wide response, the author was mindful of the potential for significant disruptions in research and risk of large time gaps in processing citations. In this context, the importance of traceability and recording individual and team decisions was essential to ensure continuity and consistency in the study.
Appraising Full Articles
The variability of abstract content and structure raised challenges in applying the inclusion criteria, reiterating the importance of clear, well-constructed and detailed abstracts for effectiveness and efficiency in determining relevance (Jones, 2004; Taylor et al., 2007). Where abstracts lacked sufficient detail to determine inclusion or exclusion, the full text was appraised, focusing on the methodology and findings sections. Where decision on inclusion/exclusion could not be determined on the basis of title and abstract, and full text articles were not available online, interlibrary loans and e-versions were requested from the institutional library. However, due to quarantine and country lockdown measures in response to COVID-19, accessibility for some articles faced significant delays. Articles which faced a 6-week or longer time period for retrieval from the point of request were excluded due to the time limitations of the study. Only two articles were excluded before appraising the full text due to accessibility.
Create Combined List, Removing Duplications by Hand in Excel Spreadsheet
Steps 9, 10, and 11 were completed for all nine database searches. In total, the nine database searches retrieved 1,137 citations. The citations were merged into a single excel spreadsheet and duplicates removed by hand, reducing results to 509 unique citations. Each citation was tagged with the relevant databases from which it was retrieved, recording database overlap and enabling the calculation of unique hits per database.
Group Consultation
Eight articles were deemed borderline for inclusion by the first author. These were discussed against the study criteria with the research team until a consensus was achieved. The final number of included and excluded articles can be seen in Figure 1.
Search overview.
Results
In total, 39 articles met the selection criteria and were included in the systematic literature review (see Figure 1). The articles were published across 24 different journals, and although the British Journal of Social Work represented the largest proportion of publications (11 out of 39), the diverse journal coverage reflects the interdisciplinary nature of social work and extensive knowledge base social workers draw upon. Journals not typically associated with social work were evident in the review including Time & Society, Public Administration and Organization, emphasizing the importance of taking a broad range of sources into consideration and not relying on social work exclusive sources to overcome bias and increase the sensitivity of systematic literature reviews.
Calculating Unique Hits, Precision, and Sensitivity of Each Database
The total number of relevant articles (39) was used to calculate the sensitivity and precision of each database and identify the number of unique hits (see Table 5) before conducting a cross-study analysis in the discussion. The SSCI retrieved the highest number of unique hits across the systematic search strategy, while CDAS, SCO, Sociological Abstracts, and Sociological Database retrieved none. SSA provided the greatest sensitivity in results at 69.2%, closely followed by SSCI at 64.1%. The lowest performance of sensitivity was CDAS at 5.1%, followed by Sociological Abstracts at 10.3%. Lastly, despite a low sensitivity rating, PsycINFO provided the highest precision in results (19.4%), yet precision rates were low across all included databases, with five performing at a rate lower than 10%.
Key Indicators of Database Performance Calculated for the Current Study.
Note. ASSIA = Applied Social Science Index and Abstracts; CDAS = Child Development and Adolescent Studies; IBSS = International Bibliography of the Social Sciences; SCO = Social Care Online; SSCI = Social Science Citation Index; SSA = Social Services Abstracts.
Discussion and Implications for Practice
Systematic literature reviews add value to social work practice through the synthesis of literature, providing a comprehensive body of empirical research and an analysis of trends, exceptions, and continued gaps in knowledge, all of which are vital to inform evidence-based practice (Erwin et al., 2011; Saini & Shlonsky, 2012; Soilemezi & Linceviciute, 2018; Thomas & Harden, 2008). With research and publication in the field of social work continuing to grow, there are increasing challenges in remaining up to date in practice; however, systematic literature reviews aid in overcoming barriers to accessing and appraising knowledge (Saini & Shlonsky, 2012). Yet, systematic literature reviews require considerable time and skill investment and there remains a wide body of research which has not yet been subject to systematic meta-synthesis. Given the essential nature of evidence-based practice and continued professional development for competent and professional practice, how can we maximize efficiency in literature searching in response to demands on time and the importance of remaining up to date in the field?
To balance rigor with time restrictions, database selection is an essential consideration for any thorough yet realistic and effective literature search. Although access to some databases is restricted by paywalls, there are typically multiple options in social science research, particularly for social work which crosses many bodies of knowledge and engages in cross-disciplinary work (Parton, 1996; Trevithick, 2008). Because time remains a restricted resource in both research and social work practice, the ability to search all potentially relevant sources is unachievable. In the study presented, potential databases were identified through prior systematic literature reviews in the field of social work and platforms available through the host institution, creating a list of 22 options. Precision was used as a determining factor for inclusion to minimize the number of irrelevant articles retrieved in this study. Calculated through an initial basic test search, inclusion was limited to a 90% or higher precision rating and adequate search facilities. The final nine databases were considered a manageable workload for the limitations of the current project while still covering a range of platforms to enable comprehensive and sensitive of findings.
Upon completing the full systematic search across all nine databases, the results show that four of the databases included were unnecessary, with five databases retrieving 100% of the unique hits (ASSIA, IBSS, PsycINFO, SSCI, and SSA). This could have been reduced further by excluding PsycINFO or IBSS, which each provided only 1 unique hit out of the total 39. These results could not be predicted based on the test search, and it should not be assumed that all systematic searches in social work will produce the same results. However, the results do illustrate that an overreliance on one or two databases is subject to bias and will not provide comprehensive coverage of literature. Reporting both the highest performing and lowest performing databases could offer further guidance in future selection.
The amount of time and work required could have been reduced through merging databases which operate on the same platform. For example, ASSIA, IBSS, SSA, Sociological Abstracts, and Sociology Database are all hosted by the Proquest platform and could have been searched simultaneously, reducing both duplicate results and the number of individual searches performed. The combined single search would have retrieved 33 of the 39 unique hits identified in this study. The decision not to use the merger function in the current study was intentional, as separate searches were essential to assess and compare the performance of individual databases to provide guidance for future literature searching. However, when evaluating the implications for social work practice, a simultaneous search strategy is recommended where possible to streamline the process while still ensuring rigorous and sensitive results.
The appraisal of database performance in the current study was compared to three other systematic reviews which reported on the unique hits, sensitivity, and precision of databases in their systematic search findings (McFadden et al., 2012; McGinn et al., 2016; Taylor et al., 2007). While it is noted that databases have different target audiences and purposes (McFadden et al., 2012), as shown in Table 6, the comparison has identified emerging performance trends across the four studies. PyscINFO is repeatedly reported as a precise database; however, the number of unique hits retrieved by this database was variable. PsycINFO produced 11 of the 53 unique hits in a study on intimate partner violence perpetrator programs (McGinn et al., 2016). Having produced almost 21% of the relevant articles, PsycINFO was an essential resource. However, PsycINFO produced only 2 of the 45 hits retrieved in a study on child protection social workers’ resilience (McFadden et al., 2012), 8 of the 363 unique hits in the study on decision making for institutional care in older populations (Taylor et al., 2007), and only 1 out of 39 unique hits in the current study. Although positioned as a consistently precise database, this comparison indicates that the number of unique hits produced is not reliant on precision and suggests a connection with the nature of the research question and specific field of practice. Furthermore, relying on precision alone to select databases risks missing a vast amount of relevant literature.
Comparison of the Top Three Databases Across Four Systematic Literature Searches.
Note. ASSIA = Applied Social Science Index and Abstracts; SSCI = Social Science Citation Index; SSA = Social Services Abstracts.
Both SSCI and SSA maintained high levels of performance in retrieving sensitive and unique hits in three of the four searches. Similarly, ASSIA was ranked in the top three most sensitive databases in all three studies that used it, evidencing consistent sensitivity across the varied social work topics covered (McGinn et al., 2016; McFadden et al., 2012, current study). Although the specific research questions posed for each study differ considerably (social worker resilience, intimate partner violence perpetrator programs, aged care decision making, and social worker experiences of bureaucracy), ASSIA, SSA, and SSCI are highlighted as valuable, high-performing databases in social work. This could be attributed to the more generalist, inclusive scope of the databases, which adds importance to their inclusion for social work research which transposes multiple disciplines including social policy, law, psychology, sociology, political science, and anthropology (Parton, 1996; Trevithick, 2008). The inclusion of ASSIA, SSA, and SSCI is recommended for future literature searching.
Selection, however, must also be accurate for the specific research question posed. Investigating how decisions are made about the entry of people aged 65+ years into institutional care, findings from a systematic literature search found AgeInfo (a database dedicated to the health and welfare of older people) to be a precise database (Taylor et al., 2007). Furthermore, AgeInfo retrieved 17 unique hits, strengthening the importance of its inclusion for rigorous literature searching on the topic of elder care. As shown by Taylor et al. (2007), topic-specific databases should not be overlooked.
The findings remain tentative, as detailed search strategies and evaluations of database performance in systematic literature searching for social work remain underreported. Additionally, up-to-date and detailed systematic literature review strategies in social work are necessary to develop a more nuanced and informed overview of this interdisciplinary field. The current comparison and findings are limited in that the four studies reporting on performance employed slightly different databases for each systematic search to reflect the specific research questions. For example, Taylor et al. (2007) did not include ASSIA and the current study excluded CINAHL due to the low performance in the test search (see Table 2). Furthermore, none of the studies included SWA. Although this database has been thoroughly critiqued (Flatley et al., 2007; Holden et al., 2008; Shek, 2008), with a growing body of research and ongoing updates to digital technology, timing permits a further assessment of this source.
Conclusion
By detailing the steps followed in conducting a systematic literature search, this article highlights the importance of trialing a draft search strategy to assess the accuracy of terms and their relevance to the key concepts. Terms can then be adjusted to improve precision or sensitivity. Furthermore, a trial can identify realistic limitations and inform inclusion decisions such as incorporating gray literature on the basis of accessibility and number of citations. A trial run enables the researcher or practitioner to improve their search strategy, ultimately saving valuable time in the systematic retrieval of relevant literature.
The relevance and interpretation of the database performance results presented in Table 6 will depend on what the individual or organization is prioritizing: sensitivity, precision, unique hits, or a combination of several of these factors. Social workers, however, can learn from the findings of past research which has shown that database selection should be justifiable and informed by careful consideration to balance comprehensive searching with restricted time and resources. Relying on a single database is subject to bias and will not provide sensitive findings. Despite limitations, the systematic search and cross-study comparison of database performance evidences the importance of ASSIA, SSA, and SSCI for future literature searching in social work and recommends merging searches for databases which operate on the same platform to streamline the process and reduce replication.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the University of Ulster and the Doctoral Training Alliance as a recipient of the Marie Sklodowska-Curie PhD Fellowship Programme. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 801604.
