Toward Artificial Intelligence-Human Paired Programming: A Review of the Educational Applications and Research on Artificial Intelligence Code-Generation Tools
Restricted accessResearch articleFirst published online September, 2024
Toward Artificial Intelligence-Human Paired Programming: A Review of the Educational Applications and Research on Artificial Intelligence Code-Generation Tools
Pair Programming is considered an effective approach to programming education, but the synchronous collaboration of two programmers involves complex coordination, making this method difficult to be widely adopted in educational settings. Artificial Intelligence (AI) code-generation tools have outstanding capabilities in program generation and natural language understanding, creating conducive conditions for pairing with humans in programming. Now some more mature tools are gradually being implemented. This review summarizes the current status of educational applications and research on AI-assisted programming technology. Through thematic coding of literature, existing research focuses on five aspects: underlying technology and tool introduction, performance evaluation, the potential impacts and coping strategies, exploration of behavioral patterns in technological application, and ethical and safety issues. A systematic analysis of current literature provides the following insights for future academic research related to the practice of “human-machine pairing” in programming: (1) Affirming the value of AI code-generation tools while also clearly defining their technical limitations and ethical risks; (2) Developing adaptive teaching ecosystems and educational models, conducting comprehensive empirical research to explore the efficiency mechanisms of AI-human paired programming; (3) Further enriching the application of research methods by integrating speculative research with empirical research, combining traditional methods with emerging technologies.
AbdelghaniR.OudeyerP.-Y.LawE.de VulpillièresC.SauzéonH. (2022). Conversational agents for fostering curiosity-driven learning in children. International Journal of Human-Computer Studies, 167(11), 102887. https://doi.org/10.1016/j.ijhcs.2022.102887
AlkaissiH.McFarlaneS. I. (2023). Artificial hallucinations in ChatGPT: Implications in scientific writing. Cureus, 15(2), Article e35179. https://doi.org/10.7759/cureus.35179
4.
AlvesP.CiprianoB. P. (2023). The centaur programmer--How Kasparov's Advanced Chess spans over to the software development of the future. arXiv. preprint arXiv:2304.11172.
5.
Azuaje PirelaM. (2023). Three major challenges for the intellectual property agenda in the face of artificial intelligence and other frontier technologies. Available at SSRN 4378436. https://doi.org/10.2139/ssrn.4378436
6.
BarkeS.JamesM. B.PolikarpovaN. (2023). Grounded copilot: How programmers interact with code-generating models. Proceedings of the ACM on Programming Languages, 7(OOPSLA1), 85–111. https://doi.org/10.1145/3586030
7.
BašićŽ.BanovacA.KružićI.JerkovićI. (2023). ChatGPT-3.5 as writing assistance in students’ essays. Humanities and Social Sciences Communications, 10(1), 1–5. https://doi.org/10.1057/s41599-023-02269-7
8.
BeckL.ChizhikA. (2013). Cooperative learning instructional methods for CS1: Design, implementation, and evaluation. ACM Transactions on Computing Education (TOCE), 13(3), 1–21. https://doi.org/10.1145/2499947.2492686
9.
BeckerB. A.DennyP.Finnie-AnsleyJ.Luxton-ReillyA.PratherJ.SantosE. A. (March, 2023 15-18). Programming is hard-or at least it used to be: Educational opportunities and challenges of ai code generation [Conference session]. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education, Toronto, ON, Canada, (Vol. 1). https://doi.org/10.1145/3545945.3569759
BirdC.FordD.ZimmermannT.ForsgrenN.KalliamvakouE.LowdermilkT.GazitI. (2022). Taking Flight with Copilot: Early insights and opportunities of AI-powered pair-programming tools. Queue, 20(6), 35–57. https://doi.org/10.1145/3582083
12.
BullC.KharrufaA. (2023). Generative AI assistants in software development education: A vision for integrating generative AI into educational practice, not instinctively defending against it. IEEE Software, 41(2), 52–59. https://doi.org/10.1109/ms.2023.3300574
13.
ChenM. X.ZhouL.ZhaoJ. Y. (2022). The link between learning design and learning analytics: Current situation, challenges and implementation paths. Open Education Research, 28(06), 27–36(in Chinese). https://doi.org/10.13966/j.cnki.kfjyyj.2022.06.004
14.
ChittumJ. R.JonesB. D.AkalinS.SchramÁ. B. (2017). The effects of an afterschool STEM program on students’ motivation and engagement. International Journal of STEM Education, 4(1), 11–16. https://doi.org/10.1186/s40594-017-0065-4
15.
ChoksiM. Z.GoedickeD. (2023). Whose text is it anyway? Exploring BigCode, intellectual property, and ethics. arXiv. preprint arXiv:2304.02839.
16.
CottonD. R.CottonP. A.ShipwayJ. R. (2024). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International, 61(2), 228–239. https://doi.org/10.1080/14703297.2023.2190148
17.
DakhelA. M.MajdinasabV.NikanjamA.KhomhF.DesmaraisM. C.JiangZ. M. J. (2023). Github copilot ai pair programmer: Asset or liability?Journal of Systems and Software, 203(9), Article 111734. https://doi.org/10.1016/j.jss.2023.111734
18.
DennyP.KumarV.GiacamanN. (2023, March 15-18). Conversing with copilot: Exploring prompt engineering for solving cs1 problems using natural language [Conference session]. The 54th ACM Technical Symposium on Computer Science Education, Toronto, Canada. https://doi.org/10.1145/3545945.3569823
19.
DennyP.SarsaS.HellasA.LeinonenJ. (2022). Robosourcing educational resources--leveraging Large Language Models for learnersourcing. arXiv. preprint arXiv:2211.04715.
20.
EysenbachG. (2023). The role of ChatGPT, generative language models, and artificial intelligence in medical education: A conversation with ChatGPT and a call for papers. JMIR Medical Education, 9(1), Article e46885. https://doi.org/10.2196/46885
21.
Finnie-AnsleyJ.DennyP.BeckerB. A.Luxton-ReillyA.PratherJ. (2022, February 14-18). The robots are coming: Exploring the implications of openai codex on introductory programming[conference session]. 24th Australasian Computing Education Conference, Virtual Event, Australia.https://doi.org/10.1145/3511861.3511863
22.
Finnie-AnsleyJ.DennyP.Luxton-ReillyA.SantosE. A.PratherJ.BeckerB. A. (2023, January 30 - February 2). My AI wants to know if this will Be on the exam: Testing OpenAI’s Codex on CS2 programming Exercises [conference session]. 25th Australasian Computing Education Conference, Melbourne, Australia. https://doi.org/10.1145/3576123.3576134
23.
GabajiwalaE.MehtaP.SinghR.KoshyR. (2022). Quiz Maker: Automatic Quiz Generation from Text Using NLP. In SinghP.K.WierzchońS.T.ChhabraJ.K.TanwarS. (Eds.), Futuristic Trends in Networks and Computing Technologies. Lecture Notes in Electrical Engineering (936, pp. 523–533). Singapore: Springer. https://doi.org/10.1007/978-981-19-5037-7_37
24.
GoK. R.SoundarapandianS.MitraA.VidoniM.FerreyraN. E. D. (2023). Simple stupid insecure practices and GitHub’s code search: A looming threat?Journal of Systems and Software, 202(8), Article 111698. https://doi.org/10.1016/j.jss.2023.111698
GuX. Q.WangC. L.WangF. (2017). Historical mission and roles of ICT in educational reform in the past 20 years. E-Education Research, 38(06), 9–19. (in Chinese). https://doi.org/10.13811/j.cnki.eer.2017.06.002
27.
HäkkinenP.ArvajaM.HämäläinenR.PöysäJ. (2010). Scripting computer-supported collaborative learning: A review of SCORE studies. E-Collaborative knowledge construction: Learning from computer-supported and virtual environments (pp. 180–194). IGI Global. https://doi.org/10.4018/978-1-61520-729-9.ch010
28.
HindleA.BarrE. T.GabelM.SuZ.DevanbuP. (2016). On the naturalness of software. Communications of the ACM, 59(5), 122–131. https://doi.org/10.1145/2902362
29.
JayagopalD. (2022). Study of program synthesizers and novice programmers. University of California. [Unpublished master dissertation].
30.
JiH.HanI.KoY. (2022). A systematic review of conversational AI in language education: Focusing on the collaboration with human teachers. Journal of Research on Technology in Education, 55(1), 48–63. https://doi.org/10.1080/15391523.2022.2142873
31.
JiaoJ. L.ChenL.WuW. W. (2023). Educational issues triggered by ChatGPT: Possible impacts and counter measures. Chinese Journal of ICT in Education, 29(03), 19–32. (in Chinese). https://doi.org/10.3969/j.issn.1673-8454.2023.03.003
32.
KringsK.BohnN. S.HilleN. A. L.LudwigT. (2023, April 23-28). “What if everyone is able to program?”–exploring the role of software development in science fiction [conference session]. CHI Conference on Human Factors in Computing Systems 2023, Hamburg, Germany. https://doi.org/10.1145/3544548.3581436
33.
LauS.GuoP. (2023, August 7-11). From “Ban it till we understand it” to “Resistance is futile”: How university programming instructors plan to adapt as more students use AI code generation and explanation tools such as ChatGPT and GitHub copilot [conference session]. 2023 ACM Conference on International Computing Education Research-Volume 1, Chicago USA. https://doi.org/10.1145/3568813.3600138
34.
LiberatiA.AltmanD. G.TetzlaffJ.MulrowC.GøtzscheP. C.IoannidisJ. P.ClarkeM.DevereauxP. J.KleijnenJ.MoherD.MoherD. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine, 151(4), W65. W94, https://doi.org/10.7326/0003-4819-151-4-200908180-00136
35.
LuX. Z.WangS. L. (2019). Historical thinking and future development of educational informatization in China: Based on the relationship between educational informatization and educational modernization. Jiangsu Higher Education, 34(12), 1–8(in Chinese). https://doi.org/10.13236/j.cnki.jshe.2019.12.001
36.
MacNeilS.TranA.MogilD.BernsteinS.RossE.HuangZ. (2022, August 7-11). Generating diverse code explanations using the gpt-3 large language model [Conference session]. ICER 2022: ACM Conference on International Computing Education Research, Lugano and Virtual Event, Switzerland. https://doi.org/10.1145/3501709.3544280
37.
MarkelJ. M.GuoP. J. (2021, March). Inside the mind of a cs undergraduate ta: A firsthand account of undergraduate peer tutoring in computer labs [conference session]. The 52nd ACM Technical Symposium on Computer Science Education, Virtual Event, USA. https://doi.org/10.1145/3408877.3432533
38.
MastropaoloA.PascarellaL.GuglielmiE.CiniselliM.ScalabrinoS.OlivetoR.BavotaG. (2023). On the robustness of code generation techniques: An empirical study on github copilot. arXiv. preprint arXiv:2302.00438.
MedeirosR. P.RamalhoG. L.FalcaoT. P. (2019). A systematic literature review on teaching and learning introductory programming in higher education. IEEE Transactions on Education, 62(2), 77–90. https://doi.org/10.1109/te.2018.2864133
41.
MegahedF. M.ChenY. J.FerrisJ. A.KnothS.Jones-FarmerL. A. (2023). How generative ai models such as chatgpt can be (mis) used in spc practice, education, and research? An exploratory study. Quality Engineering, 36(2), 287–315. https://doi.org/10.1080/08982112.2023.2206479
42.
MozannarH.BansalG.FourneyA.HorvitzE. (2022). Reading between the lines: Modeling user behavior and costs in AI-assisted programming. arXiv. preprint arXiv:2210.14306.
MumfordL. (1971). Technics and human development: the myth of the machine (vol. I, pp. 381–410). Harvest Books.
45.
PengS.KalliamvakouE.CihonP.DemirerM. (2023). The impact of ai on developer productivity: Evidence from github copilot. arXiv. preprint arXiv:2302.06590.
46.
PratherJ.ReevesB. N.DennyP.BeckerB. A.LeinonenJ.Luxton-ReillyA.PowellG.Finnie-AnsleyJ.SantosE. A. (2023). It's weird that it knows what I want: Usability and interactions with copilot for novice programmers. arXiv. preprint arXiv:2304.02491.
RomeroM.LepageA.LilleB. (2017). Computational thinking development through creative programming in higher education. International Journal of Educational Technology in Higher Education, 14(1), 1–15. https://doi.org/10.1186/s41239-017-0080-z
50.
RudolphJ.TanS.TanS. (2023). ChatGPT: Bullshit spewer or the end of traditional assessments in higher education?Journal of Applied Learning and Teaching, 6(1), 9. https://doi.org/10.37074/jalt.2023.6.1.9
51.
SarsaS.DennyP.HellasA.LeinonenJ. (2022, August 7-11). Automatic generation of programming exercises and code explanations using large language models [Conference session]. ACM conference on International computing education research, Lugano and Virtual Event, Switzerland. https://doi.org/10.1145/3501385.3543957
52.
SchäferM.NadiS.EghbaliA.TipF. (2024). An empirical evaluation of using Large Language Models for automated unit test generation. IEEE Transactions on Software Engineering, 50(1), 85–105. https://doi.org/10.1109/tse.2023.3334955
53.
SfetsosP.StamelosI.AngelisL.DeligiannisI. (2006). Investigating the impact of personality types on communication and collaboration-viability in pair programming–an empirical study. In AbrahamssonP.MarchesiM.SucciG. (Eds.), Extreme Programming and Agile Processes in Software Engineering (4044, pp. 43–52). Springer. https://doi.org/10.1007/11774129_5.
SobaniaD.BrieschM.HannaC.PetkeJ. (2023). An analysis of the automatic bug fixing performance of chatgpt. arXiv. preprint arXiv:2301.08653.
56.
StahlG.HakkarainenK. (2021). Theories of CSCL. International handbook of computer-supported collaborative learning (pp. 23–43). Springer. https://doi.org/10.4324/9781410601544-21
57.
VaithilingamP.ZhangT.GlassmanE. L. (2022, April 29 – 2022, May5). Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models [conference session]. CHI Conference on Human Factors in Computing Systems,Los Angeles, United States. https://doi.org/10.1145/3491101.3519665
58.
WatsonC.LiF. W. B. (2014, June 21-25). Failure rates in introductory programming revisited [conference session]. 2014 Conference on Innovation & Technology in Computer Science Education - ITiCSE ’14, Uppsala, Sweden. https://doi.org/10.1145/2591708.2591749
59.
WermelingerM. (2023). Using GitHub Copilot to Solve Simple Programming Problems. Proceedings of the 54th ACM Technical Symposium on Computer Science EducationV.1. https://doi.org/10.1145/3545945.3569830
60.
WhiteM.VendomeC.Linares-VásquezM.PoshyvanykD. (2015, May 16-24). Toward deep learning software repositories [Conference session]. 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories, Florence, Italy. https://doi.org/10.1109/msr.2015.38
61.
XiaoY.ChatterjeeS.GehringerE. (2022, November 6-9). A new era of plagiarism the danger of cheating using AI [Conference session]. 20th International Conference on Information Technology Based Higher Education and Training 2022, Antalya, Turkey. https://doi.org/10.1109/ithet56107.2022.10031827
62.
XuW.WuY.OuyangF. (2023). Multimodal learning analytics of collaborative patterns during pair programming in higher education. International Journal of Educational Technology in Higher Education, 20(1), 1–20. https://doi.org/10.1186/s41239-022-00377-z
63.
YangZ.ChenS.GaoC.LiZ.LiG.LvR. (2023). Deep learning based code generation methods: A literature review. arXiv. preprint arXiv:2303.01056.
64.
ZhengQ.XiaX.ZouX.DongY.WangS.XueY.WangZ.ShenL.WangA.LiY.SuT.YangZ.TangJ. (2023). Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv. preprint arXiv:2303.17568.
