AI-Enhanced Quality of Multimodal Physics Word Problems in Authentic Contexts Using the P-MAGIC App

Abstract

This study aimed to develop an app called Physics Multimodal Authentic Generated Interactive Content Problem (P-MAGIC) to gather authentic contextual information by using object recognition and real-time sensors to help students learn the principles of centripetal acceleration in Physics. We implemented an instructional-based prompt generator that adjusts the difficulty level of problems in three levels, with the questions themselves being generated by the Generative Pre-trained version 4o (GPT-4o) model. Furthermore, to evaluate our app, we assessed the quality of the generated questions using automatic and human evaluations. The automatic evaluation demonstrated that the questions were relevant, easy to understand, and diverse. Additionally, feedback from teachers showed high ratings for relevance, completeness, and concept accuracy. Interestingly, problems incorporating both text and images, followed by text, images, and graphs, outperformed those that were based on text and graphs or text and tables. This highlights the positive impact of visual elements on conceptual understanding and student engagement. Furthermore, the strong correlation between automatic and human evaluations indicates that automated assessments can reliably reflect human judgement, making them a scalable tool for evaluations. By providing teachers with the tools to create personalized and context-rich problems in a variety of formats could potentially transform physics education.

Keywords

multimodal physics problems generative AI physics education automatic question generation authentic contexts

Get full access to this article

View all access options for this article.

References

Abraham

S. S.

Khemani

(2016). Hybrid of qualitative and quantitative knowledge models for solving physics word problems. In Proceedings of the twenty-ninth Florida artificial intelligence research society conference (FLAIRS-29). https://cdn.aaai.org/ocs/12939/12939-57691-1-PB.pdf

Airey

(2009). Science, language, and literacy: Case studies of learning in Swedish university physics (Doctoral dissertation). Acta Universitatis Upsaliensis. https://www.diva-portal.org/smash/get/diva2:173193/FULLTEXT01.pdf

Allison

(2026). RAISE the standard: A framework for transparent reporting of artificial intelligence studies in education. Journal of Educational Computing Research, 64(1), 3–15. https://doi.org/10.1177/07356331251377430

Anderson

L. W.

Krathwohl

D. R.

(Eds.), (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives (Complete ed.). Addison Wesley Longman.

Azouaou

Chen

Desmoulins

(2004). Semantic annotation tools for learning material. In International workshop on applications of semantic web technologies for E-Learning. https://www.researchgate.net/publication/32231658_Semantic_Annotation_Tools_for_Learning_Material

Baltruaitis

Ahuja

Morency

L.-P.

(2019). Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2), 423–443. https://doi.org/10.1109/TPAMI.2018.2798607

Devlin

Chang

M.-W.

Lee

Toutanova

(2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (Long and short papers) (pp. 4171–4186). Association for Computational Linguistics. https://aclanthology.org/N19-1423/

Ding

Cen

Wei

(2023). Using large language model to solve and explain physics word problems approaching human level. arXiv. https://doi.org/10.48550/arXiv.2309.08182

El-Adawy

Liao

Lad

Abdelhafez

Dourmashkin

(2024). Streamlining physics problem generation to support physics teachers in using generative artificial intelligence. The Physics Teacher, 62(7), 595–598. https://doi.org/10.1119/5.0201458

10.

Fitzpatrick

C. L.

Hallett

Morrissey

K. R.

Yıldız

N. R.

Wynes

Ayesu

(2019). Response sentences, examples, and authenticity do not help children solve real world problems. Learning and Instruction, 61(1), 111–125. https://doi.org/10.1016/j.learninstruc.2018.10.002

11.

Gatt

Krahmer

(2018). Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. Journal of Artificial Intelligence Research, 61(1), 65–170. https://doi.org/10.1613/jair.5477

12.

Ghosh

Das

(2023). Automatic generation of algebraic representation for physics problems. In Proceedings of the 3rd international conference on educational technology and online learning (ICETOL 2023) full paper proceedings. 2023/06/20.

13.

Gilbert

J. K.

(2010). The role of visual representations in the learning and teaching of science: An introduction. Asia-Pacific Forum on Science Learning and Teaching, 11(1). https://www.eduhk.hk/apfslt/v11_issue1/foreword/index.htm

14.

Gualberto

Kress

(2025). Social semiotics. In Hobbs

Mihailidis

(Eds.), The international encyclopedia of media literacy. Wiley. https://doi.org/10.1002/9781118978238.ieml0226

15.

Hochreiter

Schmidhuber

(1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

16.

Horbach

Aldabe

Bexte

de Lacalle

O. L.

Maritxalar

(2020). Linguistic appropriateness and pedagogic usefulness of reading comprehension questions. In Proceedings of the twelfth language resources and evaluation conference (LREC 2020) (pp. 1753–1762). European Language Resources Association. https://aclanthology.org/2020.lrec-1.217/

17.

Hwang

W.-Y.

Utami

I. Q.

(2024). Using GPT and authentic contextual recognition to generate math word problems with difficulty levels. Education and Information Technologies, 29(13), 17635–17663. https://doi.org/10.1007/s10639-024-12537-x

18.

Hwang

W.-Y.

Wang

C.-Y.

Sharples

(2007). A study of multimedia annotation of web-based materials. Computers & Education, 48(4), 680–699. https://doi.org/10.1016/j.compedu.2005.04.020

19.

Jewitt

(2008). Multimodality and literacy in school classrooms. Review of Research in Education, 32(1), 241–267. https://doi.org/10.3102/0091732X07310586

20.

Jewitt

(2012). Technology, literacy, learning: A multimodal approach. Routledge. https://doi.org/10.4324/9780203964101

21.

Jewitt

Kress

Ogborn

Tsatsarelis

(2001). Multimodal teaching and learning: The rhetorics of the science classroom. Continuum.

22.

Kahan

Koivunen

M.-R.

(2001). Annotea: An open RDF infrastructure for shared web annotations. In Proceedings of the 10th international conference on world wide web (pp. 623–632). ACM.

23.

Küchemann

Steinert

Revenga

Schweinberger

Dinc

Avila

K. E.

Kuhn

(2023). Can ChatGPT support prospective teachers in physics task development? Physical Review Physics Education Research, 19(2), 020128. https://doi.org/10.1103/PhysRevPhysEducRes.19.020128

24.

Kurdi

Leo

Parsia

Sattler

Al-Emari

(2020). A systematic review of automatic question generation for educational purposes. International Journal of Artificial Intelligence in Education, 30(1), 121–204. https://doi.org/10.1007/s40593-019-00186-y

25.

Lemke

J. L.

(2004). The literacies of science. In Saul

E. W.

(Ed.), Crossing borders in literacy and science instruction: Perspectives on theory and practice (pp. 33–47). International Reading Association. https://doi.org/10.1598/0872075192.2

26.

Ling

Afzaal

(2024). Automatic question-answer pairs generation using pre-trained large language models in higher education. Computers and Education: Artificial Intelligence, 6, 100252. https://doi.org/10.1016/j.caeai.2024.100252

27.

Luo

Shan

Zhang

Yuan

Zhang

Yan

(2021). Exploiting syntactic and semantic information for textual similarity estimation. Mathematical Problems in Engineering, 2021(1), 4186750. https://doi.org/10.1155/2021/4186750

28.

Marchotridyo and Ruskanda

(2024). Physics assessment generation through pattern matching and large language models. In Proceedings of the 2024 11th international conference on advanced informatics: Concept, theory and application (ICAICTA), 2024/09/28. https://doi.org/10.1109/ICAICTA63815.2024.10763293

29.

Mazidi

Nielsen

R. D.

(2014). Linguistic considerations in automatic question generation. In Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 2: Short papers) (pp. 321–326). Association for Computational Linguistics. https://doi.org/10.3115/v1/P14-2053

30.

McHugh

M. L.

(2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282. https://doi.org/10.11613/BM.2012.031

31.

Moore

Nguyen

H. A.

Bier

Domadia

Stamper

(2022). Assessing the quality of student-generated short answer questions using GPT-3. In Proceedings of the European conference on technology enhanced learning.

32.

Nokelainen

Kurhila

Miettinen

Floréen

Tirri

(2003). Evaluating the role of a shared document-based annotation tool in learner-centered collaborative learning. In Proceedings of the 3rd IEEE international conference on advanced learning technologies (ICALT’03) (pp. 200–203). IEEE.

33.

Otani

Togashi

Sawai

Ishigami

Nakashima

Rahtu

Heikkilä

Satoh

(2023). Toward verifiable and reproducible human evaluation for text-to-image generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE. https://arxiv.org/abs/2304.01816

34.

Patel

Bhattamishra

Goyal

(2021). Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 conference of the north American chapter of the association for computational linguistics: Human language technologies (pp. 2080–2094). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.naacl-main.168

35.

Polozov

O’Rourke

Smith

A. M.

Zettlemoyer

Gulwani

Popović

(2015). Personalized mathematical word problem generation. In Proceedings of the twenty-fourth international joint conference on artificial intelligence (IJCAI-15) (pp. 381–388). AAAI Press. https://www.ijcai.org/Proceedings/15/Papers/060.pdf

36.

Roy

Roth

(2015). Solving general arithmetic word problems. In Proceedings of the 2015 conference on empirical methods in natural language processing (pp. 1743–1752). Association for Computational Linguistics. https://doi.org/10.18653/v1/D15-1202

37.

Scaria

Chenna

S. D.

Subramani

(2024). Automated educational question generation at different Bloom’s skill levels using large language models: Strategies and evaluation. In Proceedings of the international conference on artificial intelligence in education, ArXiv, abs/2408.04394.

38.

Scharpf

Schubotz

Spitz

Greiner-Petter

Gipp

(2022). Collaborative and AI-aided exam question generation using wikidata in education. arXiv. https://arxiv.org/abs/2211.08361

39.

Shi

Zhao

Liu

Zhang

(2024). Research and design of automatic questioning system based on question generation. In Proceedings of the 2024 12th international conference on information and education technology (ICET), pp. 182-186, Yamaguchi, Japan. https://doi.org/10.1109/ICIET60671.2024.10542827

40.

Singhal

Goyal

Henz

(2016). User-defined difficulty levels for automated question generation. In Proceedings of the 2016 IEEE 28th international conference on tools with artificial intelligence (ICTAI) (pp. 828–835). IEEE. https://doi.org/10.1109/ICTAI.2016.0129

41.

Staacks

Hütz

Heinke

Stampfer

(2018). Advanced tools for smartphone-based experiments: Phyphox. Physics Education, 53(4), 045009. https://doi.org/10.1088/1361-6552/aac05e

42.

Stefanel

(2019). Graph in physics education: From representation to conceptual understanding. In Pospiech

Michelini

Eylon

B.-S.

(Eds.), Mathematics in physics education (pp. 195–231). Springer. https://doi.org/10.1007/978-3-030-04627-9_9

43.

Steuer

Bongard

Uhlig

Zimmer

(2021). On the linguistic and pedagogical quality of automatic question generation via neural machine translation. In Proceedings of the European conference on technology enhanced learning (pp. 289–294). Springer. https://doi.org/10.1007/978-3-030-86436-1_22

44.

Uz Bilgin

Tokel

S. T.

(2019). Facilitating contextual vocabulary learning in a mobile-supported situated learning environment. Journal of Educational Computing Research, 57(4), 930–953. https://doi.org/10.1177/0735633118779397

45.

van Leeuwen

(2005). Introducing social semiotics. Routledge.

46.

Venktesh

Akhtar

M. S.

Mohania

Goyal

(2022). Auxiliary task guided interactive attention model for question difficulty prediction. arXiv. https://arxiv.org/abs/2207.01494

47.

Wang

H.-C.

Maslim

Kan

C.-H.

(2023). A question–answer generation system for an asynchronous distance learning platform. Education and Information Technologies, 28(9), 12059–12088. https://doi.org/10.1007/s10639-023-11675-y

48.

Wanselin

Danielsson

Wikman

(2022). Analysing multimodal texts in science—A social semiotic perspective. Research in Science Education, 52, 891–907. https://doi.org/10.1007/s11165-021-10027-5

49.

Yang

S. J.

Chen

I. Y.-L.

Shao

N. W.

(2004). Ontology enabled annotation and knowledge management for collaborative learning in virtual learning community. Journal of Educational Technology & Society, 7(4), 70–81.

50.

Zhang

Wang

Zhang

Dai

B. T.

Shen

H. T.

(2020). The gap of semantic parsing: A survey on automatic math word problem solvers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(9), 2287–2305. https://doi.org/10.1109/TPAMI.2019.2914054

51.

Yeo

Nielsen

(2020). Multimodal science teaching and learning. Learning: Research and Practice, 6(1), 1–4. https://doi.org/10.1080/23735082.2020.1752043

52.

Zhang

Guo

Chen

Fan

Cheng

(2021). A review on question generation from natural language text. ACM Transactions on Information Systems, 40(1), 1–43. https://doi.org/10.1145/3468889

53.

Zhang

Kishore

Weinberger

K. Q.

Artzi

(2019). BERTScore: Evaluating text generation with BERT. arXiv. https://doi.org/10.48550/arXiv.1904.09675

54.

Zhao

W. X.

Zhou

Tang

Wang

Hou

Min

Zhang

Dong

Yang

Chen

Jiang

Ren

Tang

Liu

Wen

J.-R.

(2023). A survey of large language models. arXiv. https://arxiv.org/abs/2303.18223

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.10 MB