Sage Journals: Discover world-class research

Abstract

Purpose: Title and abstract screening is one of the most time- and resource-intensive steps in systematic and scoping reviews (SRs); however, artificial intelligence (AI) can accelerate this step without sacrificing methodological rigor. We test ChatGPT-4o's ability to accurately screen citations and present an accessible approach, tailored to social work scholars with limited AI experience. Method: Through prompt engineering, we tested how two vectorization techniques, three algorithms, and class weighting impact ChatGPT-4o's classification accuracy for article inclusion in two SRs compared to models run in a native Python environment (Jupyter Notebook). Results: ChatGPT-4o-generated models were comparable to Jupyter Notebook, with the bag of words or term frequency-inverse document frequency and logistic regression models performing well. Additionally, adjusting for class imbalance improved performance across samples and models. Discussion: This approach is effective, promoting accessibility for researchers and practitioners. We encourage future researchers to replicate and extend the use of chatbots in the screening process.

Keywords

systematic reviews scoping reviews abstract screening artificial intelligence supervised machine learning

Get full access to this article

View all access options for this article.

References

Bannach-Brown

Przybyła

Thomas

Rice

A. S. C.

Ananiadou

Liao

Macleod

M. R.

(2019). Machine learning algorithms for systematic review: Reducing workload in a preclinical review of animal studies and reducing human screening error. Systematic Reviews, 8, Article 23. https://doi.org/10.1186/s13643-019-0942-7

Boughorbel

Jarray

El-Anbari

(2017). Optimal classifier for imbalanced data using Matthews correlation coefficient metric. PLoS ONE, 12(6), e0177678. https://doi.org/10.1371/journal.pone.0177678

Breiman

(2001). Random forests. Machine Learning, 45(1), 5–32. https://doi.org/10.1023/A:1010933404324

Cai

Geng

Westerman

Wang

Vallejo

J. J. G.

(2023). Utilizing ChatGPT to select literature for meta-analysis shows workload reduction while maintaining a similar recall level as manual curation. medRxiv. https://doi.org/10.1101/2023.09.06.23295072

Cambridge English Dictionary. (n.d.). Lemmatization. Cambridge English Dictionary. https://dictionary.cambridge.org/us/dictionary/english/lemmatization

Campion

M. A.

Campion

E. D.

(2023). Machine learning applications to personnel selection: Current illustrations, lessons learned, and future research. Personnel Psychology, 76(4), 993–1009. https://doi.org/10.1111/peps.12621

Chai

K. E.

Lines

R. L.

Gucciardi

D. F.

(2021). Research screener: A machine learning tool to semi-automate abstract screening for systematic reviews. Systematic Reviews, 10, Article 93. https://doi.org/10.1186/s13643-021-01635-3

Chen

. (2025, May 10). ChatGPT free tier limits in 2025: Complete usage guide. Cursor IDE. https://www.cursor-ide.com/blog/chatgpt-free-limits-guide

Chicco

Jurman

(2020). The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics, 21, Article 6. https://doi.org/10.1186/s12864-019-6413-7

10.

Cortes

Vapnik

(1995). Support-vector networks. Machine Learning, 20, 273–297. https://doi.org/10.1007/BF00994018

11.

Covidence. (n.d.-a). Covidence systematic review software. Covidence. Retrieved from www.covidence.org

12.

Covidence. (n.d.-b). Pricing: Covidence—Better systematic review management. Covidence. https://www.covidence.org/pricing/

13.

Ganesan

. (2023, March 12). What are stop words? Opinosis Analytics. Retrieved on June 23, 2024, from https://www.opinosis-analytics.com/knowledge-base/stop-words-explained/

14.

Garkisch

Goldkind

(2025). Considering a unified model of artificial intelligence enhanced social work: A systematic review. Journal of Human Rights and Social Work, 10(1), 23–42. https://doi.org/10.1007/s41134-024-00326-y

15.

Haddaway

N. R.

Westgate

M. J.

(2019). Predicting the time needed for environmental systematic reviews and systematic maps. Conservation Biology, 33(2), 434–443. https://doi.org/10.1111/cobi.13231

16.

Japkowicz

Stephen

(2002). The class imbalance problem: A systematic study. Intelligent Data Analysis, 6(5), 429–449. https://doi.org/10.3233/IDA-2002-6504

17.

Jiang

Gradus

J. L.

Rosellini

A. J.

(2020). Supervised machine learning: A brief primer. Behavior Therapy, 51(5), 675–687. https://doi.org/10.1016/j.beth.2020.05.002

18.

Johnson

M. A.

Schminke

(2020). Thinking big: An integrative conceptual review of the workplace consequences of obesity and a theoretical extension of the processes that create them. Journal of Applied Psychology, 105(7), 671–692. https://psycnet.apa.org/doi/10.1037/apl0000459

19.

Kamaldeep. (2025, January 2). How to improve class imbalance using class weights in machine learning? Analytics Vidhya. https://www.analyticsvidhya.com/blog/2020/10/improve-class-imbalance-class-weights/

20.

Kirasich

Smith

Sadler

(2018). Random forest vs logistic regression: Binary classification for heterogeneous datasets. SMU Data Science Review, 1(3), 9. https://scholar.smu.edu/datasciencereview/vol1/iss3/9

21.

Koenig

Tonidandel

Thompson

Albritton

Koohifar

Yankov

Speer

Hardy III

J. H.

Gibson

Frost

Liu

McNeney

Capman

Lowery

Kitching

Nimbkar

Boyce

Sun

Guo

, …, Newton

(2023). Improving measurement and prediction in personnel selection through the application of machine learning. Personnel Psychology, 76(4), 1061–1123. https://doi.org/10.1111/peps.12608

22.

Landsman

M. J.

Addison

S. M.

Veeh

(2025). A scoping review of how parental substance use affects real child outcomes. [Manuscript in preparation].

23.

May

R. J.

Maier

H. R.

Dandy

G. C.

(2010). Data splitting for artificial neural networks using SOM-based stratified sampling. Neural Networks, 23(2), 283–294. https://doi.org/10.1016/j.neunet.2009.11.009

24.

Muraina

(2021). Ideal dataset splitting ratios in machine learning algorithms: General concerns for data scientists and data analysts. In 7th International Mardin Artuklu Scientific Research Conference (pp. 496–504). https://www.researchgate.net/profile/Ismail-Muraina/publication/358284895_IDEAL_DATASET_SPLITTING_RATIOS_IN_MACHINE_LEARNING_ALGORITHMS_GENERAL_CONCERNS_FOR_DATA_SCIENTISTS_AND_DATA_ANALYSTS/links/61fb97e711a1090a79cc1a8b/IDEAL-DATASET-SPLITTING-RATIOS-IN-MACHINE-LEARNING-ALGORITHMS-GENERAL-CONCERNS-FOR-DATA-SCIENTISTS-AND-DATA-ANALYSTS.pdf

25.

Naidu

Zuva

Sibanda

E. M.

(2023). A review of evaluation metrics in machine learning algorithms. In Silhavy

Silhavy

(Eds.), Artificial intelligence application in networks and systems: Proceedings of 12th computer science on-line conference 2023 (Vol. 3), pp. 15–25. Springer. https://doi.org/10.1007/978-3-031-35314-7_2

26.

Najafali

Camacho

J. M.

Reiche

Galbraith

L. G.

Morrison

S. D.

Dorafshar

A. H.

(2023). Truth or lies? The pitfalls and limitations of ChatGPT in systematic review creation. Aesthetic Surgery Journal, 43(8), NP654–NP655. https://doi.org/10.1093/asj/sjad093

27.

Olofsson

Brolund

Hellberg

Silverstein

Stenström

Österberg

Dagerhamn

(2017). Can abstract screening workload be reduced using text mining? User experiences of the tool Rayyan. Research Synthesis Methods, 8(3), 275–280. https://doi.org/10.1002/jrsm.1237

28.

OpenAI. (2024). ChatGPT (July 18 version) [large language model]. https://chat.openai.com/chat

29.

OpenAI. (n.d.). ChatGPT pricing. ChatGPT. https://openai.com/chatgpt/pricing/

30.

Page

M. J.

McKenzie

J. E.

Bossuyt

P. M.

Boutron

Hoffmann

T. C.

Mulrow

C. D.

Shamseer

Tetzlaff

J. M.

Akl

E. A.

Brennan

S. E.

Chou

Glanville

Grimshaw

J. M.

Hróbjartsson

Lalu

M. M.

Loder

E. W.

Mayo-Wilson

McDonald

, …, Moher

(2021a). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, n71. https://doi.org/10.1136/bmj.n71

31.

Page

M. J.

Moher

Bossuyt

P. M.

Boutron

Hoffmann

T. C.

Mulrow

C. D.

Shamseer

Tetzlaff

J. M.

Akl

E. A.

Brennan

S. E.

Chou

Glanville

Grimshaw

J. M.

Hróbjartsson

Lalu

M. M.

Loder

E. W.

Mayo-Wilson

McDonald

, …, McKenzie

J. E.

(2021b). PRISMA 2020 explanation and elaboration: Updated guidance and exemplars for reporting systematic reviews. BMJ, 372, 372:n160. https://doi.org/10.1136/bmj.n160

32.

Polanin

J. R.

Pigott

T. D.

Espelage

D. L.

Grotpeter

J. K.

(2019). Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses. Research Synthesis Methods, 10(3), 330–342. https://doi.org/10.1002/jrsm.1354

33.

Powers

D. M.

(2020). Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation. arXiv. https://doi.org/10.48550/arXiv.2010.16061

34.

Przybyła

Brockmeier

A. J.

Kontonatsios

Le Pogam

M. A.

McNaught

von Elm

Nolan

Ananiadou

(2018). Prioritising references for systematic reviews with RobotAnalyst: A user study. Research Synthesis Methods, 9(3), 470–488. https://doi.org/10.1002/jrsm.1311

35.

Qaiser

Ali

(2018). Text mining: Use of TF-IDF to examine the relevance of words to documents. International Journal of Computer Applications, 181(1), 25–29. https://www.researchgate.net/profile/Shahzad-Qaiser/publication/326425709_Text_Mining_Use_of_TF-IDF_to_Examine_the_Relevance_of_Words_to_Documents/links/5b4cd57fa6fdcc8dae245aa3/Text-Mining-Use-of-TF-IDF-to-Examine-the-Relevance-of-Words-to-Documents.pdf

36.

Research Screener. (n.d.). Research Screener. https://researchscreener.com/

37.

Rodriguez

M. Y.

Storer

(2019). A computational social science perspective on qualitative data exploration: Using topic models for the descriptive analysis of social media data. Journal of Technology in Human Services, 38(1), 54–86. https://doi.org/10.1080/15228835.2019.1616350

38.

Rogalin

C. L.

Addison

S. M.

(2025). Masculinities under the microscope: A systematic review of experimental methods. [Manuscript accepted for publication]. Psychology of Men & Masculinities.

39.

Rubin

(2023). It takes an average of 17 years for evidence to change practice—The burgeoning field of implementation science seeks to speed things up. JAMA, 329(16), 1333–1336. https://doi.org/10.1001/jama.2023.4387

40.

Santiago

A. M.

Smith

R. J.

(2020). What can “big data” methods offer human services research on organizations and communities? In McBeath

Hopkins

(Eds.), The future of human service organizational & management research (pp. 118–130). Routledge.

41.

Šimundić

A. M.

(2009). Measures of diagnostic accuracy: Basic definitions. eJIFCC, 19(4), 203. https://pubmed.ncbi.nlm.nih.gov/27683318/

42.

Stoll

Wehrli

Lätsch

(2025). Case reports unlocked: Harnessing large language models to advance research on child maltreatment. Child Abuse & Neglect, 160, 107202. https://doi.org/10.1016/j.chiabu.2024.107202

43.

Trudeau

K. J.

Yang

Kraus

D. R.

(2023). Predicting successful placements for youth in child welfare with machine learning. Children and Youth Services Review, 153, 107117. https://doi.org/10.1016/j.childyouth.2023.107117

44.

van Dijk

S. H.

Brusse-Keizer

M. G.

Bucsán

C. C.

van der Palen

Doggen

C. J.

Lenferink

(2023). Artificial intelligence in systematic reviews: Promising when appropriately used. BMJ Open, 13(7), e072254. https://doi.org/10.1136/bmjopen-2023-072254

45.

Victor

B. G.

Perron

B. E.

Sokol

R. L.

Fedina

Ryan

J. P.

(2021). Automated identification of domestic violence in written child welfare records: Leveraging text mining and machine learning to enhance social work research and evaluation. Journal of the Society for Social Work and Research, 12(4), 631–655. https://doi.org/10.1086/712734

46.

Wallace

B. C.

Small

Brodley

C. E.

Lau

Trikalinos

T. A.

(2012). Deploying an interactive machine learning system in an evidence-based practice center: Abstrackr. In Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium (pp. 819–824). https://doi.org/10.1145/2110363.2110464

47.

Wang

Nayfeh

Tetzlaff

O’Blenis

Murad

M. H.

(2020). Error rates of human reviewers during abstract screening in systematic reviews. PLoS ONE, 15(1), e0227742. https://doi.org/10.1371/journal.pone.0227742

48.

Wang

Scells

Koopman

Zuccon

(2023). Can ChatGPT write a good Boolean query for systematic review literature search? In SIGIR ’23: Proceedings of the 46th International ATM SIGIR Conference on Research and Development in Information Retrieval (pp. 1426–1436). https://doi.org/10.1145/3539618.3591703

Making AI Accessible to Social Work Researchers: An Exploratory Analysis of Using ChatGPT to Screen Articles for Systematic and Scoping Reviews

Abstract

Keywords

Get full access to this article

References