Sage Journals: Discover world-class research

Abstract

Background

As generative artificial intelligence (AI) tools such as ChatGPT become increasingly prevalent in higher education, key questions emerge regarding their effectiveness in supporting classroom learning.

Objective

This study examined whether an AI system could generate psychology course-specific essay questions, grade responses, and provide feedback, as well as how students perceived the process.

Method

We developed a custom web app using ChatGPT with retrieval-augmented generation to create and grade essay questions. Students (N = 37) from an upper-division course at a regional public university completed four AI-graded assessments over six weeks, followed by an exit survey. We compared AI and instructor scores using intraclass correlation coefficients and Bland-Altman analysis.

Results

Agreement between AI and instructor grades was moderate, with intraclass correlations ranging from .21 to .81, and AI tended to avoid assigning extreme scores. Students valued the instant, personalized AI feedback, but only 35% perceived the AI grading as fair.

Conclusion

AI tools such as ChatGPT show promise for supporting assessment and feedback, but limitations in perceived fairness may reduce student acceptance.

Keywords

generative artificial intelligence ChatGPT AI grading AI in higher education retrieval-augmented generation

Get full access to this article

View all access options for this article.

References

Alers

Malinowska

Meghoe

Apfel

(2024). Using ChatGPT-4 to grade open question exams. In Arai

(Ed.), Advances in Information and Communication (pp. 1–9). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-53960-2_1

Ali

FA.

Sharif

SY.

Ata

Patel

Rafay

Syed

HR.

Iqbal

SP.

(2024). ChatGPT develops multiple choice questions (MCQs) for postgraduate specialty assessment – A reality or a myth? Pakistan Journal of Neurological Surgery, 28(1), 142–149. https://doi.org/10.36552/pjns.v28i1.963

Bland

JM.

Altman

DG.

(1986). Statistical methods for assessing agreement between two methods of clinical measurement. Lancet, 327(8476), 307–310. https://doi.org/10.1016/S0140-6736(86)90837-8

Chukwuere

JE.

(2024). The use of ChatGPT in higher education: The advantages and disadvantages. arXiv. https://doi.org/10.48550/arXiv.2403.19245

Coskun

TK.

Alper

(2024). Evaluating the evaluators: A comparative study of AI and teacher assessments in higher education. Digital Education Review, 45, 124–140. https://doi.org/10.1344/der.2024.45.124-140

Dai

Lin

Jin

Tsai

YS.

Gašević

Chen

(2023). Can large language models provide feedback to students? A case study on ChatGPT. 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), 323–325. https://doi.org/10.1109/ICALT58122.2023.00100

Flodén

(2024). Grading exams using large language models: A comparison between human and AI grading of exams in higher education using ChatGPT. British Educational Research Journal, 51(1), 201–224. https://doi.org/10.1002/berj.4069

Fütterer

Fischer

Alekseeva

Chen

Tate

Warschauer

Gerjets

(2023). ChatGPT in education: Global reactions to AI innovations. Scientific Reports, 13(1), 1–14. https://doi.org/10.1038/s41598-023-42227-6

Gao

Xiong

Gao

Jia

Pan

Dai

Sun

Wang

(2023). Retrieval-augmented generation for large language models: A survey. arXiv, https://doi.org/10.48550/arXiv.2312.10997

10.

Koo

TK.

MY.

(2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012

11.

Kooli

Yusuf

(2024). Transforming educational assessment: Insights into the use of ChatGPT and large language models in grading. International Journal of Human–Computer Interaction, 41(5), 3388–3399. https://doi.org/10.1080/10447318.2024.2338330

12.

Lewis

Perez

Piktus

Petroni

Karpukhin

Goyal

Küttler

Lewis

Yih

Rocktäschel

Riedel

Kiela

(2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Proceedings of the 34th International Conference on Neural Information Processing Systems, 9459–9474.

13.

Liu

Tang

Chen

Feng

X. M.

(2016). Correlation and agreement: Overview and clarification of competing concepts and measures. Shanghai Archives of Psychiatry, 28(2), 115–120. https://doi.org/10.11919/j.issn.1002-0829.216045

14.

Morjaria

Burns

Bracken

Levinson

AJ.

Ngo

QN.

Lee

Sibbald

(2024). Examining the efficacy of ChatGPT in marking short-answer assessments in an undergraduate medical program. International Medical Education, 3(1), 32–43. https://doi.org/10.3390/ime3010004

15.

OpenAI. (2022, November 30). Introducing ChatGPT. OpenAI. https://openai.com/blog/chatgpt

16.

OpenAI. (2023). GPT-4 Technical Report. arXiv. https://doi.org/10.48550/arXiv.2303.08774

17.

OpenAI. (2024). Enterprise privacy at OpenAI. OpenAI. Retrieved February 22, 2024, from https://openai.com/enterprise-privacy/

18.

Scouller

(1998). The influence of assessment method on students’ learning approaches: Multiple choice question examination versus assignment essay. Higher Education, 35(4), 453–472. https://doi.org/10.1023/A:1003196224280

19.

Singer-Freeman

Janio

Taylor

(2023). Making a grand contribution: Using ChatGPT to improve assessment in higher education. Assessment Update, 35(5), 10–11. https://doi.org/10.1002/au.30365

20.

Stowell

JR.

Zhu

(2025). Evaluating ChatGPT for automated creation and grading of essay questions in higher education. https://doi.org/10.17605/OSF.IO/T7BHE

21.

Strzelecki

(2023). To use or not to use ChatGPT in higher education? A study of students’ acceptance and use of technology. Interactive Learning Environments, 32(9), 5142–5155. https://doi.org/10.1080/10494820.2023.2209881

22.

Tonmoy

SMTI.

Zaman

SMM.

Jain

Rani

Rawte

Chadha

Das

(2024). A comprehensive survey of hallucination mitigation techniques in large language models. arXiv. https://doi.org/10.48550/arXiv.2401.01313

23.

Wetzler

EL.

Cassidy

KS.

Jones

MJ.

Frazier

CR.

Korbut

NA.

Sims

CM.

Bowen

SS.

Wood

(2024). Grading the graders: Comparing generative AI and human assessment in essay evaluation. Teaching of Psychology, 52(3), 298–304. https://doi.org/10.1177/00986283241282696

24.

(2024). Do AI chatbots improve students learning outcomes? Evidence from a meta-analysis. British Journal of Educational Technology, 55(1), 10–33. https://doi.org/10.1111/bjet.13334

25.

Xiao

Zhi

(2023). An exploratory study of EFL learners’ use of ChatGPT for language learning tasks: Experience and perceptions. Languages, 8(3), 212. https://doi.org/10.3390/languages8030212

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.47 MB

Evaluating ChatGPT for Automated Creation and Grading of Essay Questions in Higher Education

Abstract

Background

Objective

Method

Results

Conclusion

Keywords

Get full access to this article

References

Supplementary Material