The potential of large language models (LLMs) to mitigate the time- and cost-related challenges associated with inductive thematic analysis (ITA) is being increasingly explored in the literature. However, the use of LLMs to support ITA has often been opportunistic, relying on ad hoc prompt engineering (PE) approaches, thereby undermining the reliability, transparency, and replicability of the analysis. The goal of this study is to develop a structured approach to PE in LLM-assisted ITA. To this end, a comprehensive review of the existing literature is conducted to examine how researchers applying ITA integrate LLMs into their workflows and, in particular, how PE is utilized to support the analytical process. Built on the insights generated from this review, four key steps for effective PE in LLM-assisted ITA are identified and proposed. Furthermore, the study explores advanced PE techniques that can enhance the execution of these steps, providing researchers with practical strategies to improve their analyses. In conclusion, the main contributions of this paper include: (i) mapping the existing research on LLM-assisted ITA to enable a better understanding of the rapidly developing field, (ii) proposing a structured four-step PE process to enhance methodological rigor, (iii) discussing the application of advanced PE techniques to support the execution of these steps, and (iv) highlighting key directions for future research.
BraunV.ClarkeV. (2024). Thematic analysis. In Encyclopedia of quality of life and well-being research (pp. 7187–7193). Springer.
2.
BrownT.MannB.RyderN.SubbiahM.KaplanJ. D.DhariwalP.NeelakantanA.ShyamP.SastryG.AskellA. (2020). Language models are few-shot learners. In: Advances in neural information processing systems (Vol. 33, pp. 1877–1901). MIT Press.
3.
Carrera-RiveraA.OchoaW.LarrinagaF.LasaG. (2022). How to con-duct a systematic literature review: A quick guide for computer science research. MethodsX, 9, 101895. https://doi.org/10.1016/j.mex.2022.101895
4.
CarvalhoT.NegmH.El-GeneidyA. (2024). A comparison of the results from artificial intelligence-based and human-based transport-related the-matic analysis. Findings. https://doi.org/10.32866/001c.94401
5.
ChenX.AksitovR.AlonU.RenJ.XiaoK.YinP.PrakashS.SuttonC.WangX.ZhouD. (2023). Universal self-consistency for large language model generation. arXiv preprint arXiv:2311.17311.
DeinerM. S.HoncharovV.LiJ.MackeyT. K.PorcoT. C.SarkarU. (2024). Large language models can enable inductive thematic analysis of a social media corpus in a single prompt: Human validation study. JMIR Infodemiology, 4(1), Article e59641. https://doi.org/10.2196/59641
10.
De PaoliS. (2024a). Further explorations on the use of large language models for thematic analysis. Open-ended prompts, better terminologies and thematic maps. Forum: Qualitative Social Research, 25(3).
11.
De PaoliS. (2024b). Performing an inductive thematic analysis of semi-structured interviews with a large language model: An exploration and provocation on the limits of the approach. Social Science Computer Review, 42(4), 997–1019. https://doi.org/10.1177/08944393231220483
12.
DiaoS.WangP.LinY.PanR.LiuX.ZhangT. (2023). Active prompt-ing with chain-of-thought for large language models. arXiv preprint arXiv:2302.12246.
13.
GaoH.LinT.LiH.YangM.WuY.MaW.LiY. (2023). Self-explanation prompting improves dialogue understanding in large language models. arXiv preprint arXiv:2309.12940.
GirayL. (2023). Prompt engineering with chatgpt: A guide for academic writers. Annals of Biomedical Engineering, 51(12), 2629–2633. https://doi.org/10.1007/s10439-023-03272-4
JiangZ.XuF. F.ArakiJ.NeubigG. (2020). How can we know what language models know?Transactions of the Association for Computational Linguistics, 8, 423–438. https://doi.org/10.1162/tacl_a_00324
19.
KadavathS.ConerlyT.AskellA.HenighanT.DrainD.PerezN. E.and Schiefer, Hatfield-DoddsZ.DasSarmaN.Tran-JohnsonE. (2022). Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221.
20.
KaddourJ.HarrisJ.MozesM.BradleyH.RaileanuR.McHardyR. (2023). Challenges and applications of large language models. arXiv preprint arXiv:2307.10169.
21.
KatzA.FlemingG. C.MainJ. (2024). Thematic analysis with open-source generative ai and machine learning: A new method for inductive qualitative codebook development. arXiv preprint arXiv:2410.03721.
22.
KatzA.GerhardtM.SoledadM. (2024). Using generative text models to create qualitative codebooks for student evaluations of teaching. International Journal of Qualitative Methods, 23, 16094069241293283. https://doi.org/10.1177/16094069241293283
KimH. J.ChoH.KimJ.KimT.YooK. M.LeeS. (2022). Self-generated in-context learning: Leveraging auto-regressive language models as a demonstration generator. arXiv preprint arXiv:2206.08082.
28.
LiC.WangJ.ZhangY.ZhuK.HouW.LianJ.LuoF.YangQ.XieX. (2023). Large language models understand and can be enhanced by emotional stimuli. arXiv preprint arXiv:2307.11760.
29.
LuA.ZhangH.ZhangY.WangX.YangD. (2023). Bounding the ca-pabilities of large language models in open text generation with prompt constraints. arXiv preprint arXiv:2302.09185.
30.
MadaanA.TandonN.GuptaP.HallinanS.GaoL.WiegreffeS.AlonU.DziriN.PrabhumoyeS.YangY. (2024). Self-refine: Iterative refinement with self-feedback. In Advances in neural information processing systems (Vol. 36). MIT Press.
31.
MannstadtI.GoodmanS. M.RajanM.YoungS. R.WangF.Navarro- Mill’anI.MehtaB. (2024). A novel approach for mixed-methods research using large language models: A report using patients’ perspectives on barriers to arthroplasty. ACR Open Rheumatology, 6(6), 375–379. https://doi.org/10.1002/acr2.11662
32.
MarvinG.HellenN.JjingoD.Nakatumba-NabendeJ. (2023). Prompt engineering in large language models. In International conference on data intelligence and cognitive informatics (pp. 387–402). IEEE.
33.
MathisW. S.ZhaoS.PrattN.WeleffJ.De PaoliS. (2024). Inductive thematic analysis of healthcare qualitative interviews using open-source large language models: How does it compare to traditional methods?Computer Methods and Programs in Biomedicine, 255, 108356. https://doi.org/10.1016/j.cmpb.2024.108356
34.
MorseE.LiA.AlbertS.HarpelL.RameauA. (2025). Large language model versus human-generated thematic analysis in otolaryngology qualitative research. The Laryngoscope, 135(1), 1–2. https://doi.org/10.1002/lary.31894
ParkS.KulkarniC. (2023). Thinking assistants: Llm-based conversational assistants that help users think by asking rather than answering. arXiv preprint arXiv:2312.06024.
PrescottM. R.YeagerS.HamL.Rivera SaldanaC. D.SerranoV.NarezJ.PaltinD.DelgadoJ.MooreD. J.MontoyaJ. (2024). Comparing the efficacy and efficiency of human and generative AI: qualitative thematic analyses. JMIR AI, 3, Article e54482. https://doi.org/10.2196/54482
39.
SabbaghanS. (2024). Exploring the synergy of human and ai-driven approaches in thematic analysis for qualitative educational research. Journal of Applied Learning and Teaching, 7(2).
40.
SchroederH.Qu’er’eM. A. L.RandazzoC.MimnoD.SchoenebeckS. (2024). Large language models in qualitative research: Can we do the data justice?arXiv preprint arXiv:2410.07362.
41.
SchulhoffS.IlieM.BalepurN.KahadzeK.LiuC.and SiA.LiY.GuptaA.HanH.SchulhoffS. (2024). The prompt report: A systematic survey of prompting techniques. arXiv preprint arXiv:2406.06608.
42.
ShahC. (2024). From prompt engineering to prompt science with human in the loop. arXiv preprint arXiv:2401.04122.
43.
WestonJ.SukhbaatarS. (2023). System 2 attention (is something you might need too). arXiv preprint arXiv:2311.11829.
44.
YasunagaM.ChenX.LiY.PasupatP.LeskovecJ.LiangP.ChiE. H.ZhouD. (2023). Large language models as analogical reasoners. arXiv preprint arXiv:2310.01714.
45.
ZhangH.WuC.XieJ.KimC.CarrollJ. M. (2023). Qualigpt: Gpt as an easy-to-use tool for qualitative coding. arXiv preprint arXiv:2310.07061.
46.
ZhangH.WuC.XieJ.LyuY.CaiJ.CarrollJ. M. (2023). Redefining qualitative analysis in the ai era: Utilizing chatgpt for efficient thematic analysis. arXiv preprint arXiv:2309.10771.
47.
ZhaoF.YuF.ShangY. (2024). A new method supporting qualitative data analysis through prompt generation for inductive coding. In 2024 IEEE international conference on information reuse and integration for data science (IRI) (pp. 164–169). IEEE.
48.
ZhouD.Sch¨arliN.HouL.WeiJ.ScalesN.WangX.SchuurmansD.CuiC.BousquetO.LeQ. (2022). Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625.
49.
ZhouY.MuresanuA. I.HanZ.PasterK.PitisS.ChanH.BaJ. (2022). Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910.
