Sage Journals: Discover world-class research

Abstract

Performing research syntheses can be challenging due to time, labor, and monetary costs. This manuscript presents the findings of two studies that investigate the usability and feasibility of artificial intelligence (AI) systems to assist with performing research syntheses. In Study 1, participant demographics data coded by humans were compared against data extracted using ChatGPT-4 and Google Bard from 67 single case research design (SCRD) studies in which infants, toddlers, and young children participated. Both AIs achieved interobserver agreement values above 96% for data related to participants’ names, ages, genders, race/ethnicities, and disabilities/conditions. Study 2 compared the time it took two human coders to extract participant demographics data from an additional 40 SCRD studies against ChatGPT-4. ChatGPT-4 extracted data from the studies faster than the human coders. Alhough both AIs may not be ready to autonomously perform research syntheses, findings suggest that both can assist researchers with the coding process.

Keywords

artificial intelligence AI ChatGPT Bard Gemini machine learning systematic review meta-analysis research syntheses

Get full access to this article

View all access options for this article.

References

Allen

I. E.

Olkin

(1999). Estimating time to conduct a meta-analysis from number of citations retrieved. JAMA, 282(7), 634–635. https://doi.org/10.1001/jama.282.7.634

Anghelescu

Firan

F. C.

Onose

Munteanu

Trandafir

A. I.

Ciobanu

Gheorghița

Ș.

Ciobanu

(2023). PRISMA systematic literature review, including with meta-analysis vs. chatbot/GPT (AI) regarding current scientific data on the main effects of the calf blood deproteinized hemoderivative medicine (Actovegin) in ischemic stroke. Biomedicines, 11(6), 1623. https://doi.org/10.3390/biomedicines11061623

Benedek-Wood

McNaughton

Light

(2016). Instruction in letter-sound correspondences for children with autism and limited speech. Topics in Early Childhood Special Education, 36(1), 43–54. https://doi.org/10.1177/0271121415593497

Blank

I. A.

(2023). What are large language models supposed to model? Trends in Cognitive Sciences, 27(2), 987–989. https://doi.org/10.1016/j.tics.2023.08.006

Borah

Brown

A. W.

Capers

P. L.

Kaiser

K. A.

(2017). Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open, 7(2), e012545. https://doi.org/10.1136/bmjopen-2016-012545

Brown

J. A.

Woods

J. J.

(2015). Effects of a triadic parent-implemented home-based communication intervention for toddlers. Journal of Early Intervention, 37(1), 44–68. https://doi.org/10.1177/1053815115589350

Buscemi

Hartling

Vandermeer

Tjosvold

Klassen

T. P.

(2006). Single data extraction generated more errors than double data extraction in systematic reviews. Journal of Clinical Epidemiology, 59(7), 697–703. https://doi.org/10.1016/j.jclinepi.2005.11.010

Ciupe

Salisbury

(2020). Examining caregivers’ independence in early intervention home visit sessions. Journal of Early Intervention, 42(4), 338–358. https://doi.org/10.1177/1053815120902727

Cohen

(1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104

10.

Cooper

(2017). Research synthesis and meta-analysis: A step-by-step approach (5th ed.). Sage.

11.

Cumpston

Flemyng

Thomas

Higgins

J. P. T.

Deeks

J. J.

Clarke

M. J.

(2023). Chapter I: Introduction. In Higgins

J. P. T.

Thomas

(Eds.), Cochrane handbook for systematic reviews of interventions (Version 6.4). https://training.cochrane.org/handbook/current/chapter-i

12.

Curiel

E. S. L.

Sainato

D. M.

Goldstein

(2018). Matrix training for toddlers with autism spectrum disorder and other language delays. Journal of Early Intervention, 40(3), 268–284. https://doi.org/10.1177/1053815118788060

13.

Fleury

V. P.

Schwartz

I. S.

(2017). A modified dialogic reading intervention for preschool children with autism spectrum disorder. Topics in Early Childhood Special Education, 37(1), 16–28. https://doi.org/10.1177/0271121416637597

14.

Google. (n.d.). Gemini Apps FAQ. https://gemini.google.com/faq

15.

Google. (2023). Bard (June 19 version) [Large language model]. https://bard.google.com/

16.

Hardy

J. K.

Hemmeter

M. L.

(2019). Systematic instruction of early math skills for preschoolers at risk for math delays. Topics in Early Childhood Special Education, 38(4), 234–247. https://doi.org/10.1177/0271121418792300

17.

Issaiy

Ghanaati

Kolahi

Shakiba

Jalali

A. H.

Zarei

Kazemian

Avanaki

M. A.

Firouznia

(2024). Methodological insights into ChatGPT’s screening performance in systematic reviews. BMC Medical Research Methodology, 24(1), 1–11. https://doi.org/10.1186/s12874-024-02203-8

18.

Jakubanis

(2023). What are tokens and how to count them? https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them

19.

Killmeyer

Kaczmarek

Kostewicz

Yelich

(2019). Contingent imitation and young children at-risk for autism spectrum disorder. Journal of Early Intervention, 41(2), 141–158. https://doi.org/10.1177/1053815118819230

20.

Krick Oborn

K. M.

Johnson

L. D

. (2015). Coaching via electronic performance feedback to support home visitors’ use of caregiver coaching strategies. Topics in Early Childhood Special Education, 35(3), 157–169. https://doi.org/10.1177/0271121415592411

21.

Lam Hoai

X. L.

Simonart

. (2023). Comparing meta-analyses with ChatGPT in the evaluation of the effectiveness and tolerance of systemic therapies in moderate-to-severe plaque psoriasis. Journal of Clinical Medicine, 12(16), 5410. https://doi.org/10.3390/jcm12165410

22.

Lee

Bubeck

Petro

(2023). Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. The New England Journal of Medicine, 388(13), 1233–1239. https://doi.org/10.1056/NEJMsr2214184

23.

Higgins

J. P. T.

Deeks

J. J.

(2023). Chapter 5: Collecting data. In Higgins

J. P. T.

Thomas

(Eds.), Cochrane Handbook for Systematic Reviews of Interventions (Version 6.4). https://training.cochrane.org/handbook/current/chapter-05

24.

Lipsey

M. W.

Wilson

D. B.

(2001). Practical meta-analysis. Sage.

25.

Locchetta

B. M.

Barton

E. E.

Kaiser

(2017). Using family style dining to increase social interactions in young children. Topics in Early Childhood Special Education, 37(1), 54–64. https://doi.org/10.1177/0271121416678078

26.

Martin

C. A.

Drasgow

Halle

J. W.

(2015). Training teachers to enhance the play skills of young children with developmental disabilities during outdoor time by embedding instructional interactions. Journal of Early Intervention, 37(4), 247–269. https://doi.org/10.1177/1053815115620209

27.

McHugh

M. L.

(2012). Interrater reliability: The Kappa statistic. Biochemia Medica, 22(3), 276–282. https://doi.org/10.11613/BM.2012.031

28.

Michelson

Reuter

(2019). The significant cost of systematic reviews and meta-analyses: A call for greater involvement of machine learning to assess the promise of clinical trials. Contemporary Clinical Trials Communications, 16, 100443. https://doi.org/10.1016/j.conctc.2019.100443

29.

Oh-Young

Buchter

Morgan

C. W.

Clark

(2022). A 10-year systematic review of participant diversity of single-case studies published in two early intervention/early childhood special education research journals. The Journal of Special Education, 56(4), 183–192. https://doi.org/10.1177/00224669221075000

30.

OpenAI. (n.d.). Models. https://platform.openai.com/docs/models/continuous-model-upgrades

31.

OpenAI. (2023). GPT-4 is OpenAI’s most advanced system, producing safer and more useful responses. https://openai.com/gpt-4

32.

Qureshi

Shaughnessy

Gill

K. A. R.

Robinson

K. A.

Agai

(2023). Are ChatGPT and large language models “the answer” to bringing us closer to systematic review automation? Systematic Reviews, 12(1), 72. https://doi.org/10.1186/s13643-023-02243-z

33.

Sinclair

Hansen

S. G.

Machalicek

Knowles

Hirano

K. A.

Dolata

J. K.

Blakely

A. W.

Seeley

Murray

(2018). A 16-year review of participant diversity in intervention research across a selection of 12 special education journals. Exceptional Children, 84(3), 312–329. https://doi.org/10.1177%2F0014402918756989

34.

Spencer

E. J.

Goldstein

Sherman

Noe

Tabbah

Ziolkowski

Schneider

(2012). Effects of an automated vocabulary and comprehension intervention: An early efficacy study. Journal of Early Intervention, 34(4), 195–221. https://doi.org/10.1177/1053815112471990

35.

Spencer

T. D.

Kajian

Petersen

D. B.

Bilyk

(2013). Effects of an individualized narrative interventionon children’s storytelling and comprehension skills. Journal of Early Intervention, 35(3), 243–269. https://doi.org/10.1177/1053815114540002

36.

Wang

Scells

Zuccon

Koopman

(2023). Can ChatGPT write a good Boolean query for systematic review literature search? arXiv. https://doi.org/10.48550/arXiv.2302.03495

37.

Wilson

(2023, March 25). 5 ways that ChatGPT is better than Google Bard (and 3 ways it isn’t). https://www.techradar.com/features/5-ways-that-chatgpt-is-better-than-google-bard-and-3-ways-it-isnt

38.

Windsor

K. S.

Woods

Kaiser

A. P.

Snyder

Salisbury

(2019). Caregiver-implemented intervention for communication and motor outcomes for infants and toddlers. Topics in Early Childhood Special Education, 39(2), 73–87. https://doi.org/10.1177/0271121418815250

39.

Wright

C. A.

Kaiser

A. P.

(2017). Teaching parents enhanced milieu teaching with words and signs using the teach-model-coach-review model. Topics in Early Childhood Special Education, 36(4), 192–204. https://doi.org/10.1177/0271121415621027

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

The Use of Generative Artificial Intelligence in Identifying and Extracting Single Case Participant Demographics Data

Abstract

Keywords

Get full access to this article

References

Supplementary Material