The potential of artificial intelligence (AI)-based large language models (LLMs) holds considerable promise in revolutionising education, research and practice. However, distinguishing between human-written and AI-generated text has become a significant task. This article presents a comparative study, introducing a novel dataset of human-written and LLM-generated texts in different genres: essays, stories, poetry and Python code. We employ several machine learning models to classify the texts. Results demonstrate the efficacy of these models in discerning between human and AI-generated text, despite the dataset’s limited sample size. However, the task becomes more challenging when classifying GPT-generated text, particularly in story writing. The results indicate that the models exhibit superior performance in binary classification tasks, such as distinguishing human-generated text from a specific LLM, compared with the more complex multiclass tasks that involve discerning among human-generated and multiple LLMs. Our findings provide insightful implications for AI text detection, while our dataset paves the way for future research in this evolving area.
RadfordAWuJChildR, et al. Language models are unsupervised multitask learners. OpenAI Blog2019; 1(8): 9.
2.
ThoppilanRDe FreitasDHallJ, et al. LaMDA: language models for dialog applications. arXiv 2022. DOI: 10.48550/arXiv.2201.08239.
3.
ShahriarSHayawiK. Let’s have a chat! A conversation with ChatGPT: technology, applications, and limitations. Artif Intell Appl2023; 1: 4.
4.
CottonDRECottonPAShipwayJR. Chatting and cheating: ensuring academic integrity in the era of ChatGPT. Innov Educ Teach Int. Epub ahead of print 13 March 2023. DOI: 10.1080/14703297.2023.2190148.
5.
PerkinsM. Academic Integrity considerations of AI large language models in the post-pandemic era: ChatGPT and beyond. J Univ Teach Learn Pract2023; 20(2): 7.
6.
KingMRChatGPT. A conversation on Artificial Intelligence, Chatbots, and plagiarism in higher education. Cell Mol Bioeng2023; 16(1): 1–2.
7.
YuK-HBeamALKohaneIS. Artificial Intelligence in healthcare. Nat Biomed Eng2018; 2(10): 719–731.
8.
ShahriarS. GAN computers generate arts? A survey on visual arts, music, and literary text generation using generative adversarial network. Displays2022; 73: 102237.
9.
GeorgeASGeorgeAH. A review of ChatGPT AI’s impact on several business sectors. Partn Univers Int Innov J2023; 1(1): 9–23.
10.
BangYCahyawijayaSLeeN, et al. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. arXiv 2023. DOI: 10.48550/arXiv.2302.04023.
11.
BeutelGGeeritsEKielsteinJT. Artificial hallucination: GPT on LSD?Crit Care2023; 27(1): 148.
12.
ManakulPLiusieAGalesMJF. SelfCheckGPT: zero-resource black-box hallucination detection for generative large language models. arXiv 2023. DOI: 10.48550/arXiv.2303.08896.
13.
JiJLiuMDaiJ, et al. BeaverTails: towards improved safety alignment of LLM via a human-preference dataset. arXiv 2023. DOI: 10.48550/arXiv.2307.04657.
14.
DwivediYKKshetriNHughesL, et al. Opinion paper: ‘So what if ChatGPT wrote it?’ Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. Int J Inf Manag2023; 71: 102642.
15.
HarrerS. Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicine. EBioMedicine2023; 90: 104512.
16.
JungherrA. Using ChatGPT and other large language model (LLM) applications for academic paper assignments. Bamberg: Otto-Friedrich-Universität, 2023.
17.
KhalilMErE. Will ChatGPT get you caught? Rethinking of plagiarism detection. arXiv 2023. DOI: 10.48550/arXiv.2302.04335.
18.
LiawAWienerM. Classification and regression by Random Forest. R News2002; 2(3): 18–22.
19.
BurgesCJ. A tutorial on support vector machines for pattern recognition. Data Min Knowl Discov1998; 2(2): 121–167.
20.
KleinbaumDGDietzKGailM, et al. Logistic regression. Cham: Springer, 2002.
21.
ShahriarSAl RokenNZualkernanI. Classification of Arabic poetry emotions using deep learning. Computers2023; 12(5): 5.
22.
ShahriarSTariqU. Classifying Maqams of Qur’anic recitations using deep learning. IEEE Access2021; 9: 117271–117281.
23.
HochreiterSSchmidhuberJ. Long short-term memory. Neural Comput1997; 9(8): 1735–1780.
24.
RamosJ. Using TF-IDF to determine word relevance in document queries. In: 1st Instructional Conference on Machine Learning, ACM, Piscataway, NJ, 2003, vol. 242, pp. 133–142.
25.
PenningtonJSocherRManningCD. Glove: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), Doha, Qatar, October 2014, pp. 1532–1543. Kerrville, TX: Association for Computational Linguistics.
26.
HayawiKShahriarSSerhaniMA, et al. ANTi-Vax: a novel Twitter dataset for COVID-19 vaccine misinformation detection. Public Health2022; 203: 23–30.
KazameiniAFatehiSMehtaY, et al. Personality trait detection using bagged SVM over BERT word embedding ensembles. In: Proceedings of the fourth widening natural language processing workshop, Seattle, USA, July 2020. Kerrville, TX: Association for Computational Linguistics.
30.
Learningexpress. 501 writing prompts. New York, USA: Learning Express, 2014.
31.
PriceL. Fandom, folksonomies and creativity: the case of the archive of our own. In: The human position in an artificial world: creativity, ethics and AI in knowledge organization. Germany: Ergon Verlag, 2019, pp. 11–37.
RaychevVBielikPVechevM. Probabilistic model for code with decision trees. In Proceedings of the ACM SIGPLAN international conference on object-oriented programming, systems, languages, and applications, Amsterdam, 2–4 November 2016. New York, NY: Association for Computing Machinery, pp. 731–747.
35.
KingmaDPBaJ. Adam: a method for stochastic optimization. ArXiv Prepr. ArXiv 14126980, 2014.
PenedoGMalarticQHesslowD, et al. The RefinedWeb dataset for falcon LLM: outperforming curated corpora with web data, and web data only. arXiv 2023. DOI: 10.48550/arXiv.2306.01116.
39.
KeskarNSMcCannBVarshneyLR, et al. CTRL: a conditional transformer language model for controllable generation. arXiv 2019. DOI: 10.48550/arXiv.1909.05858.
40.
OnanA. GTR-GA: harnessing the power of graph-based neural networks and genetic algorithms for text augmentation. Expert Syst Appl2023; 232: 120908.
41.
OnanA. SRL-ACO: a text augmentation framework based on semantic role labeling and ant colony optimization. J King Saud Univ Comput Inf Sci2023; 35(7): 101611.
42.
OnanA. Hierarchical graph-based text classification framework with contextual node embedding and BERT-based dynamic fusion. J King Saud Univ Comput Inf Sci2023; 35(7): 101610.
43.
OnanA. Bidirectional convolutional recurrent neural network architecture with group-wise enhancement mechanism for text sentiment classification. J King Saud Univ Comput Inf Sci2022; 34(5): 2098–2117.
44.
OnanA. Biomedical text categorization based on ensemble pruning and optimized topic modelling. Comput Math Methods Med2018; 2018: 2497471.
45.
OnanA. Two-stage topic extraction model for bibliometric data analysis based on word embeddings and clustering. IEEE Access Pract Innov Open Solut2019; 7: 145614–145633.
46.
OnanAToçoğluMA. A term weighted neural language model and stacked bidirectional LSTM based framework for sarcasm identification. IEEE Access2021; 9: 7701–7722.
47.
OnanA. Sentiment analysis on massive open online course evaluations: a text mining and deep learning approach. Comput Appl Eng Educ2021; 29(3): 572–589.
