Comparative Reflections on Human-Driven and Generative Artificial Intelligence-Assisted Thematic Analysis: A Collaborative Autoethnography

Abstract

Thematic analysis is a well known qualitative analytic method, usually driven by a human researcher to analyze qualitative data. However, in the current age of Generative Artificial Intelligence (GAI) technologies revolution, analyzing qualitative data is evolving. Many research studies have explored the potential of GAI to conduct qualitative data analysis. However, limited studies have explored the collaborative autoethnography qualitative approach in understanding the expectations, challenges and future insights based on two researchers’ personal reflections of using manual approach as well as obtaining support from GAI in analysing data using thematic approach. These reflections are not mutually exclusive but interplay to assist both researchers to understand the dynamics of analysing qualitative data. The study revealed that manual thematic analysis provided in-depth, context-rich insights, capturing cultural and contextual nuances, whereas the GAI-assisted approach offered efficiency and scalability but lacked interpretative depth. Additionally, challenges such as time constraints in manual analysis and prompt variability in GAI-assisted methods were identified, highlighting the need for hybrid approaches to enhance research efficacy. These findings contribute to the research methodologies literature in filling an empirical gap to elevate research efficacy and outcomes as well as present practical implications.

Keywords

generative artificial intelligence thematic analysis collaborative autoethnography AI-assisted data analysis qualitative research methods human-computer collaboration

Introduction

Generative Artificial Intelligence (GAI) is significantly transforming research and analysis landscape globally. GAI has been adopted in handling large volumes of research information and data including streamlining literature reviews and conceptual papers to analyzing data (Christou, 2024). While there are multiple methods to analyze qualitative data, thematic analysis proposed by Braun and Clarke (2013) has been a widely used approach by researchers. With the six phases of thematic analysis proposed by Braun and Clarke (2013), it is time consuming, but it provides a rigorous analysis to make sense of the rich data so that it answers or addresses the research questions (Creswell, 2008). In this current age of AI, there are numerous GAI tools such as ChatGPT to conduct thematic analysis, including generating initial codes and themes to expedite the research process (De Paoli, 2024; Perkins & Roe, 2024). However, the GAI analysis output tend to produce descriptive rather than in-depth, contexual and subtle and data-rich details that is captured well through manual analysis (Perkins & Roe, 2024).

Numerous studies have explored in analyzing qualitative data using manual and traditional research approaches as well as GAI tools such as ChatGPT. These studies have particularly investigated the differences between using manual and GAI assisted qualitative data analysis (Hamilton et al., 2023; Perkins & Roe, 2024; Siiman et al., 2023), however there are limited studies of researchers reflecting their experiences in using manual and GAI assisted analysis, simultaneously to perform thematic analysis. It is vital to explore the researchers’ experiences in providing a unique and original contribution to the research methodologies literature unpacking the details of expectations, challenges and future insights of using manual and GAI-assisted approaches for thematic analysis. We seek to provide instrumental insights on analysing qualitative data in higher education settings. The paper addresses the question: What are our experiences and reflections as qualitative researchers in analyzing qualitative data using traditional and GAI-assisted techniques?

Literature Review

Traditional Methods of Data Analysis

Qualitative data analysis remains a fundamental method in social research, providing a lens through which complex phenomena can be thoroughly explored (Fossey et al., 2002). Traditional approaches to analyze qualitative data include thematic analysis, grounded theory, and content analysis. These approaches have long been essential tools for researchers aiming to uncover patterns, themes, and narratives hidden within qualitative data (Creswell & Poth, 2017). These methods, rooted in interpretive methodologies, allow for deep engagement with the subject matter, making them indispensable across various fields, including sociology, psychology, education, and health studies (Wertz, 2011).

Thematic analysis, for example, has gained popularity due to its flexibility and broad applicability. By systematically identifying and analyzing patterns within the data, thematic analysis facilitates an in-depth understanding of human experiences (Braun & Clarke, 2013). However, this method demands a significant investment of time and effort from researchers. Researchers must immerse themselves in the data, moving through multiple stages of familiarization, coding, and theme development to accurately capture the nuances of the subject matter. Grounded theory, another cornerstone of qualitative research, focuses on theory generation directly from data. This method emphasizes the importance of allowing data to guide the development of theoretical insights, rather than imposing preconceived notions (Charmaz, 2014). One criticism of the grounded theory approach is that its flexible and open-ended design may result in conclusions that are too broad or unclear (MacLennan, 2012), making it challenging to apply the findings to other situations Bryant (2002). Meanwhile, content analysis offers a more structured approach by quantifying and analyzing the presence of specific words, themes, or concepts within qualitative data. This method is especially useful when exploring the prevalence of ideas across large datasets, making it a vital tool in understanding complex social phenomena (Mayring, 2014; Wertz, 2011). However, the labor-intensive and time-consuming nature of these traditional methods cannot be overlooked, especially when handling large volumes of qualitative data (Richards, 2014).

GAI in Data Analysis

In recent years, the advent of GAI has opened new avenues for qualitative research. GAI technologies, including large language models (LLMs), natural language processing (NLP), and machine learning, offer the potential to revolutionize how qualitative data is analyzed. Among these, GAI-driven tools like ChatGPT have attracted significant attention for their ability to assist researchers in processing and analyzing data (e.g., de Oliveira Silva & dos Santos Janes, 2021; Hamilton et al., 2023; Huang & Tan, 2023). ChatGPT, developed by OpenAI, can comprehend and generate human-like text based on extensive training data. Its capabilities extend to coding large datasets, identifying themes, and generating summaries, thereby reducing the time and effort required for data analysis (Burger et al., 2023).

However, the integration of GAI into qualitative research is not without its challenges. Critics argue that GAI tools may lack the depth of understanding and contextual sensitivity that human researchers bring to qualitative analysis. While GAI can process large datasets swiftly, it often struggles to capture the subtleties of human experience that are central to qualitative research (Mesec, 2023). Additionally, the reliance on pre-trained models introduces concerns about biases in GAI-generated analyses, as these models may inadvertently reflect the biases present in the data on which they were trained (Ray, 2023).

One of the primary concerns with using GAI in qualitative research, according to Bano et al. (2023), is the potential loss of contextual understanding. Traditional qualitative methods emphasize the importance of situating data within its broader social, cultural, and historical contexts. Human researchers, with their subtle understanding, can interpret data in ways that account for these contextual factors (van Manen, 1990). In contrast, GAI tools, which rely heavily on pattern recognition (Ray, 2023), may overlook these subtleties. Moreover, bias in GAI-generated analyses is a significant concern. GAI models like ChatGPT are trained on vast datasets, which may include biased or unrepresentative data. Consequently, GAI-generated themes and insights may reflect these biases, leading to skewed or incomplete analyses (Christou, 2023). This is particularly concerning when the GAI reinforces existing power imbalances or fails to capture marginalized voices (Mesec, 2023).

Transparency and interpretability are also critical issues. Traditional qualitative methods involve a high degree of transparency, with researchers typically documenting their analytical processes in detail (Richards, 2014). In contrast, the internal workings of GAI models are often unclear, making it difficult to understand how the GAI arrived at certain conclusions. This lack of transparency according to Grimes et al. (2023) can undermine the credibility of GAI-generated analyses and complicate the process of triangulation, which is essential for rigorous qualitative research (Creswell & Poth, 2017). Furthermore, while GAI tools can process data quickly, they may struggle with tasks requiring deep interpretive engagement, such as identifying emergent themes not explicitly stated in the data (Kasperiuniene, 2021). Human researchers, through their iterative engagement with the data, are often able to identify subtle patterns and connections that might elude GAI. This interpretive flexibility remains a challenge for GAI tools to fully replicate (Charmaz, 2014).

As GAI technologies continue to advance, they are poised to play an increasingly significant role in qualitative research. However, it is crucial for researchers to approach these tools with a critical eye, recognizing both their potential benefits and their limitations. A promising direction for future research involves developing hybrid approaches that combine the strengths of GAI with the depth and contextual sensitivity of traditional qualitative methods. For example, GAI tools might assist with the initial coding of large datasets, while human researchers retain the role of interpreting and refining the emerging themes (Hamilton et al., 2023). Ethical considerations related to the use of GAI in qualitative research also warrant further exploration (Bano et al., 2023). As GAI tools become more prevalent, establishing guidelines and best practices will be essential to ensure these tools are used in ways that respect the integrity of the research process and the voices of research participants. Ultimately, the integration of GAI into qualitative research presents both challenges and opportunities. By carefully considering the strengths and limitations of these tools, researchers can harness their potential to enhance the efficiency and depth of qualitative analysis while preserving the richness and complexity that characterize qualitative research (Kasperiuniene, 2021).

The application of GAI in qualitative data analysis is still emerging, but several recent studies highlight its potential. These studies offer valuable insights into how GAI can be integrated into qualitative research to complement or enhance traditional analysis methods. For instance, Hamilton et al. (2023) compared GAI-generated themes with those identified by human researchers in a guaranteed income pilot study. Although there were significant overlaps between the GAI and human-generated themes, GAI often missed detailed themes related to personal motivations. Similarly, Tabone and de Winter (2023) explored the use of ChatGPT for analyzing human-computer interaction (HCI) research data. They found that GAI could effectively identify patterns but warned that it might lead to misinterpretations without careful human oversight. Mesec (2023) examined how ChatGPT could analyze social work texts, finding that GAI could generate abstract understandings, though human oversight was crucial for maintaining accuracy and relevance. In healthcare research, Alexander et al. (2022) utilized GAI-driven sentiment analysis to process patient feedback, finding that GAI could categorize feedback effectively but struggled to identify underlying causes without human interpretation. Dos Anjos et al., 2024 similarly focused on GAI in educational research, comparing qualitative data analysis using two different GAI tools, Claude 2.0 and ChatGPT 4.0. Claude 2.0 was found to be more effective in recognizing cognitive mediations and referencing specific interview excerpts. Although both models agreed on conceptual evolution, the study emphasizes the need for human oversight to address GAI limitations. Collectively, these studies illustrate the growing interest in integrating GAI into qualitative research, highlighting both the potential and limitations of GAI tools. As GAI technology continues to develop, its role in qualitative research is likely to expand, offering new tools and methods for researchers to explore complex social phenomena.

Methodology

Research Design

This study employs a collaborative autoethnographic research design to reflect on analyzing data through manual thematic analysis by a human researcher and GAI-assisted analysis, facilitated by ChatGPT 4.0. This approach allows for a thorough comparison of the two methods, particularly focusing on their strengths, limitations, and the contextual depth they offer. This design enables a reflective examination of the research process itself, providing insights not only into the outcomes but also into the experiential and methodological details (Jones et al., 2016) of using GAI in qualitative research.

Both researchers engaged in collaborative autoethnography, documenting their reflections at three stages of data analysis: pre-analysis, during analysis, and post-analysis. These reflections provided critical insights into the researchers’ initial expectations, real-time reactions, and post-analysis evaluations. This process included documenting any biases or preconceived notions, adaptations made during the analysis, and the emotional and intellectual impact of the research process. The reflective journaling by the researcher conducting the GAI-assisted analysis also added depth to the autoethnographic record, highlighting the differences in expectation, challenges as well as future engagement and understanding between the two methods. There were two participants in this research:

(1) Anas Al-Fattal is an internationally experienced Assistant Professor at an American university, recognized for his interdisciplinary and global approach. He specializes in exploring consumer behavior, educational marketing, and applied psychology, with a focus on innovative pedagogies. His research combines qualitative and mixed-methods methodologies, often emphasizing the human dimensions of learning and decision-making in diverse cultural and organizational contexts. As a Fellow in the Emerging Technologies Faculty Fellowship Program, he is particularly passionate about the role of GAI in reshaping education and research. His enthusiasm for GAI drives his efforts to bridge technological advancements with human-centered approaches, ensuring meaningful and ethical applications in academic and societal domains.

(2) Jasvir Singh, an international award-winning Senior Lecturer at an Australian university. She received international teaching, research and academic citizenship awards recognizing her outstanding performance. Her expertise is in higher education with a particular interest in exploring international students’ lived experiences of success, employability, career aspirations and learning through qualitative approach. She also investigates the lived experiences of skilled migrants and international academics. She strongly believes in analyzing qualitative data manually as she wants to be close to the data and have a hands-on feel without the intrusion of a computer (Creswell, 2008).

Data Collection: Collaborative Autoethnography

This study employed a collaborative autoethnographic (CAE) approach, which integrates personal reflections with collective analysis, to examine researchers’ experiences of manual and GAI-assisted thematic analysis. Data collection was conducted in three stages: pre-analysis, during analysis, and post-analysis. In the pre-analysis phase, researchers documented their initial expectations and assumptions using co-developed reflective prompts, fostering trust and openness within the team (Al-Fattal et al., 2024; Lapadat, 2017). During analysis, detailed reflexive journals captured the researchers’ processes, including challenges, strategies, and emergent insights. Manual analysis logs focused on coding and interpretive decisions, while GAI-assisted analysis reflections highlighted interactions with GAI tools, such as prompt design and modifications (Sakurai et al., 2021). In the post-analysis stage, researchers continued journaling their reflections to consolidate their experiences and insights. These journals documented critical evaluations of the methods, challenges, and emerging themes, ensuring consistency in data collection across all stages (Hornsby et al., 2021; Smetana et al., 2021).

Data Analysis: Collaborative Autoethnography

The analysis process in this study followed the principles of collaborative autoethnography, focusing on iterative reflection and mutual engagement to address the research question: “What are the experiences and reflections of academics in analyzing qualitative data using traditional and GAI-assisted techniques?”. Researchers individually reviewed their reflective journals from all three stages (pre-analysis, during analysis, and post-analysis) to identify recurring ideas and insights. This process involved open coding, where key phrases, experiences, and observations were systematically categorized. Following individual analysis, the team convened to share and compare their findings, fostering a collaborative interpretation of the data. Consistent with best practices in CAE, the analysis aimed to identify both shared and divergent experiences, with a particular focus on themes such as methodological affordances, trust, and power dynamics (Lapadat, 2017). This iterative process of individual and collective reflection ensured that the final themes accurately represented the diverse perspectives of the researchers, providing a rich and detailed understanding of the data (Chang et al., 2013).

Data Context

The qualitative data that we based our analysis and reflection upon is a study previously conducted by Jasvir. The data was chosen because several papers (Singh, 2019, 2020; Singh & Fan, 2021) have been published from the findings showcasing the reliability of the data. The data for this study comprises of 19 interview transcripts. These interviews were conducted with Chinese graduates who obtained a degree from an Australian university and had work experience in China using a semi-structured format. This format was chosen to capture a wide range of responses, which is crucial for both manual and GAI-driven thematic analysis. The interviews were conducted in English and the duration of the interviews was between 21 and 49 minutes. The interviews were digitally recorded and transcribed verbatim using a professional transcription service. In order to preserve the validity and trustworthiness of the data, all participants reviewed their transcripts (McClure, 2003). Open-ended interview questions were used so that participants could “best voice their experiences unconstrained by any perspectives of the researcher or past research findings” (Creswell, 2013, p. 208). The interview questions were focused on how participants managed challenges and adopted strategies in navigating employment barriers.

The Process of Data Analysis for Interviews

The analysis phase is divided into two distinct but parallel processes, whereby one of the researchers conducted the analysis manually and the other researcher used ChatGPT. This section accounts for the detailed data analysis process:

The manual thematic analysis was carried out by the primary researcher, who was responsible for conducting the interviews as well as transcribing them. This researcher employed van Manen’s (1990) hermeneutic phenomenological approach and Braun and Clarke’s (2013) thematic analysis approach, which emphasizes deep engagement with the data to reveal the lived experiences of participants. The process began with the researcher adopting van Manen’s (1990) approach of repeated readings of the transcripts to fully immerse in the data. This immersion was crucial, as it allowed the researcher to become intimately familiar with the details of each participant’s narrative. Using an inductive approach, themes naturally emerged from the data, guided by van Manen’s (1990) selective reading method, which focuses on identifying significant and meaningful statements. The researcher manually coded the data, carefully grouping similar codes and refining them into cohesive themes through an iterative process, using Braun and Clarke’s (2013) thematic analysis method. This hands-on approach, which avoided the use of software tools, ensured that the researcher maintained a close connection with the data, allowing for a deep and detailed understanding of the phenomenon under study. The final themes identified through this process provided a solid foundation for further analysis and discussion.

In contrast, the GAI-assisted analysis was conducted by a separate researcher who did not participate in the interviewing or transcription process. This researcher utilized ChatGPT 4.0 to analyze the same set of interview transcripts, which were previously prepared by the primary researcher. The GAI was tasked with coding the data, where it automatically assigned keywords and phrases to capture the meanings expressed by participants. ChatGPT employed selective reading techniques to highlight essential phrases and grouped similar and redundant codes to form a manageable number of themes. The GAI’s role was to process the data and reduce these codes into final themes, ensuring that they addressed the research questions and accurately reflected the participants’ experiences. Additionally, the GAI counted the frequency of themes to determine their relative importance within the dataset. Finally, ChatGPT explored relationships and patterns across codes and themes, contributing to the conceptualization of the phenomenon under investigation. The findings generated by the GAI were then reviewed in relation to the research questions and existing literature, providing complementary insights to those obtained through the manual analysis.

Findings

The collaborative autoethnographic process provided both the manual and GAI-assisted researchers the opportunity to reflect on their experiences, expectations, and challenges throughout the study. This reflection is structured around three key themes: expectations, challenges, and insights gained from the process.

Expectations

In relation to expectations, the manual researcher expected that deep engagement with the data would allow for a complete understanding and accurate representation of the participants’ voices. Manual analysis was seen as giving control over the entire process without interference from external tools like GAI or even a computer. This close connection with the data was essential for writing detailed and accurate narratives as well as projecting participants’ voices. The manual researcher reflections are:

I feel very close to the data. There is no ‘third person’ intruding my analysis. I want to be immersed in the data as this is a pure qualitative researcher’s trait. I can remember what is being said by the participants easily, as I usually conduct the interview myself. Then I read and re-read the transcripts over and over again and that process brings me close to the data. I am immersing myself in what the participants are experiencing against the issue. As I am immersed in the data, I am able to write the narrative more easily. I can tell and write the story without any hesitation. Participants’ voice must be upfront.

The manual researcher reflected that her method allowed for a close connection with the participants’ voices and provided a thorough analysis. The process gave confidence in the authenticity of the findings, as it allowed for an interpretation of the data that respected the participants’ experiences.

In contrast, the GAI-assisted researcher expected the GAI to provide fast and efficient results, reducing the time spent on analysis. While there was trust in GAI’s potential, there was also acknowledgment that GAI might not capture the full complexity of the data compared to a manual approach. GAI was viewed as a tool that could complement human analysis by quickly identifying key themes in a short period of time:

The speed of GAI is remarkable. I believe that the GAI system will provide acceptable results, though not as detailed as human analysis. In my previous experiences, I found that GAI is not very accurate in identifying focus and emphasis in text.

Once the analysis was conducted, manually and using ChatGPT, the manual researcher was surprised that the analysis was done quickly through ChatGPT as the manual researcher took weeks to conduct the data analysis:

The analysis with ChatGPT was done in no time through computing the questions and instructions or prompt into ChatGPT. It saved time doing the analysis this way. The content analysis was also done when prompting ChatGPT to do so. This was also done in a nick of time. The quotes were also provided against the themes in ChatGPT. No need for researchers to do the coding and themes generation.

The manual researcher did not expect that GAI could generate qualitative research findings swiftly as it takes time to capture the nuances and depth of the findings through manual scrutiny.

On the other hand, the GAI-assisted researcher was confident that the analysis via ChatGPT will be done rapidly from the onset has he acknowledged the time and effort to conduct manual analysis:

From the start, using GAI felt efficient and allowed for quick processing. I understand that qualitative data analysis requires significant time and effort from the researcher to process the data and derive meaningful results and conclusions.

In sum, both researchers had diverse expectations. The manual researcher views the data analysis process to project, narrate and capture the participants voices through a rigorous and non-intrusive way but the GAI-assisted researcher was inclined to generate the findings in quicker way possible. His thoughts are:

The entire analysis process was easy, as I did not have to spend much time diving into the data. I can feel that I have not been able to capture the essence of what was there in the data. I understand that the power of qualitative research lays in the richness of the narrative which was missed in the results provided.

Challenges

In relation to challenges, the manual researcher’s primary challenge was time. Manual thematic analysis required significant uninterrupted periods to read, reread, and code the data. The reflections are recorded as:

Conducting thematic analysis manually takes time. Time is the biggest challenge for me as during teaching semester I have to stop analysis as I have to dedicate at least 1-2 weeks to analyze a data set. If I lose the time frame, I lose track of analysis. And it takes time to resume the analysis. Hence, I usually set 1-2 weeks (without disturbance) to analyze a data set. I have to go back to the transcript again and again to get the selected quotes. Although I used color coded highlighters, it took time.

The manual researcher faced difficulties in sharing the analysis with collaborators, particularly when they did not have access to the same software tools:

If I am collaborating with researchers, it is hard to show what I have done as the analysis is written in my hard copy book. Therefore, I will need to use Nvivo. But this has challenges too as if the collaborator has no similar edition of Nvivo then that’s a problem as the collaborator cannot access the data.

Additionally, there was a recognition of potential bias in interpreting the data, especially when the manual researcher looking for new contributions for example in the field of employability studies:

I might be biased in interpreting the data as I am looking out on what is new and interesting in the data especially in the employability space. I am also looking out what is the novel contribution.

The GAI-assisted researcher faced challenges with ensuring that the GAI produced meaningful results. Crafting the right prompts was critical to guiding the GAI to deliver accurate themes. AI responses varied across different runs and sometimes grouped or missed themes unexpectedly:

In previous experiences, I found that GAI is not very accurate in identifying focus and emphasis in text. This text was too extensive, and I need to first summarize it and convert it into a prompt. I am concerned that if I do not create the right prompt, the results might not be accurate or representative of the data. I believe this will involve several trials before I can develop an effective prompt.

While the GAI process was quick, there was concern that the results lacked depth compared to what was anticipated from manual analysis, according to the GAI assisted researcher:

I also added the research questions, hoping to guide the GAI’s focus, yet the output was unexpectedly limited, with fewer themes and occasionally unusual groupings of subthemes. I observed that it lacked the ability to capture some of the complexities typically identified in manual analysis.

The findings generated by ChatGPT were superficial without an in-depth understanding and interpretation of the data which is captured by manual analysis. This insight was provided by the GAI assisted researcher upon data analysis:

The GAI’s limitations became more apparent when examining specific themes with rich, layered narratives. For instance, in the manual analysis, themes related to students’ adaptation to foreign educational systems included several relevant insights, such as cultural adjustment, academic confidence, and the influence of social networks. The GAI, however, grouped these into a broader category, missing the intricate distinctions that added depth to the human analysis. This illustrates a challenge for GAI, as it currently struggles to capture subtleties that are often central to qualitative research. These differences reinforced the idea that GAI’s processing might currently suit projects where high-level patterns are sufficient enough, rather than studies requiring deep interpretative insights.

The challenges faced by both researchers were varied: the manual researcher’s main issue was time to generate the findings. The manual analysis took significantly more time, requiring deep engagement with the data to uncover themes and contextual information. The human researcher provided detailed, context-rich findings, including the cultural and market-specific challenges faced by the participants in the data. On the other hand, the GAI assisted researcher had a problem in curating the right prompts and generating in-depth findings (i.e. nuances and themes) as compared to the manual analysis. The GAI analysis was faster, automatically identifying themes based on keywords and patterns. It was effective at summarizing broad themes but lacked the depth and contextual sensitivity seen in the manual analysis, particularly in understanding cultural and career challenges.

Further Insights

In relation to future insights, the GAI-assisted researcher acknowledged that using GAI method has its limitations. For example, GAI lacked the ability to fully capture the complexity of the data as compared to manually analyzing the data. The researcher has proposed that interview transcripts should be developed in such a way to include metalanguage information to assist GAI in limiting such issues:

In previous experiences, I found that GAI is not very accurate in identifying focus and emphasis in text. Perhaps interview transcripts should be developed to include some metalanguage information to help GAI systems overcome such challenges.

Further, the researcher also emphasized the importance of prompt design, as inaccurate or incomplete prompts led to less useful results:

Creating the prompt required careful attention. I needed to create a prompt that would help the GAI system follow a similar procedure as the manual thematic analysis. There may be room for developing prompt variations, such as instructing the GAI to weight specific keywords or increase sensitivity to recurring phrases, which could potentially yield more accurate thematic structures.

Even the manual researcher agrees that learning to use the right prompt is vital in obtaining somewhat accurate finding of the data. This is because ChatGPT is unable to provide deeper interpretation or analysis of the findings. Moreover, the researcher also was very concerned about ChatGPT not providing the precise quotes against the codes/themes:

One must learn and be knowledgeable in using the right prompt to instruct ChatGPT to get the right results/findings. Researcher(s) can only report what ChatGPT has provided without going into deeper interpretation/analysis. The quotes provided against the coding/themes are irrelevant or not like my analysis. Very short quotes are provided hence writing the narrative against the quotes is going to be challenging for a qualitative researcher.

Therefore, the GAI assisted researcher acknowledged that while GAI is promising in generating qualitative findings, it could be considered as a supplementary tool to check on human led analysis:

This project experience has shown that while GAI is promising, particularly for commercial or exploratory applications, it is still in an early phase. For academic research, it might be best suited as a supplemental tool for preliminary theme exploration or as a secondary check on human analysis.

These reflections are concurred by the manual researcher as ChatGPT can be used at initial stages of data analysis to provide an overview of the findings, but it should not be highly relied on. The researcher is a firm believer that as a hardcore qualitative researcher, the analysis and interpretation of the data is vital in showcasing the participants’ experiences:

Overall, ChatGPT can be used to gain initial coding/themes but not to be heavily dependent on. As a pure qualitative researcher, one needs to ‘dive deep’ into the analysis and also the interpretation of data is important to project the research participants voices.

As GAI is significantly transforming research analysis landscape (Christou, 2024), the GAI assisted researcher is highly confident that GAI will evolve in processing analysis for complex qualitative data in the future:

As GAI technology advances, I am optimistic that it will evolve to offer results that capture the full complexity of qualitative data, potentially transforming how we approach large-scale qualitative studies in the future.

Both researchers agreed that while GAI could assist in initial coding, it was not sufficient for producing a complete analysis. The GAI-assisted researcher appreciated the efficiency of GAI-assisted analysis but recognized its limitations. There was a recognition of the need for human oversight in reviewing and interpreting GAI-generated themes to ensure the findings were meaningful and relevant.

Discussion

This study examined the experiences of researchers conducting thematic analysis using manual and GAI-assisted methods, focusing on expectations, challenges, and future insights. Through a collaborative autoethnographic lens, this research provides a detailed understanding of how traditional and emerging methodologies shape the analytical process and outcomes.

The expectations of the researchers highlight a key dichotomy in qualitative research: depth versus efficiency. The manual researcher’s commitment to deep engagement aligns with Braun and Clarke’s (2013) emphasis on the importance of immersion for capturing the richness of qualitative data. This reflects the traditional view that qualitative analysis demands meticulous attention to contextual factors, as also emphasized by Creswell and Poth (2017). By contrast, the GAI-assisted researcher’s focus on speed and efficiency mirrors the growing reliance on GAI tools in qualitative research to streamline labor-intensive tasks (Burger et al., 2023; Hamilton et al., 2023). These contrasting expectations underscore a critical tension in qualitative research: while manual methods ensure interpretive rigor, they are resource-intensive and time-consuming. GAI tools address these limitations by providing rapid coding and initial theme identification. However, as Christou (2024) noted, such tools often lack the capacity to produce data-rich and context-sensitive findings. This trade-off reflects broader methodological debates about integrating technology into qualitative research, where the efficiency of GAI must be balanced against the depth and authenticity of traditional methods.

While some findings in the expectations section may appear intuitive, it is important to clarify that the purpose of this study is not to compare the accuracy of human-driven versus GAI-assisted thematic analysis but to reflect on our experiences engaging with both methods. Reflexivity is a core element of qualitative research (Ravitch & Carl, 2019), and by documenting our expectations, challenges, and insights, we contribute to discussions on how researchers negotiate control, trust, and interpretive authority when using GAI tools in qualitative analysis. GAI is often framed as a tool for efficiency, but its integration into qualitative research raises deeper methodological questions about its role in shaping meaning-making processes (Williams, 2024). Rather than focusing on a direct validation of GAI’s effectiveness, this study positions itself within broader debates on GAI’s role in qualitative research by offering a critical reflection on researcher experiences.

Both methods presented distinct challenges, further illustrating their respective strengths and limitations. Manual analysis, while yielding context-rich insights, required significant time and focus. The manual researcher’s reflections revealed that this method allowed for a detailed understanding of cultural dimensions and individual experiences, consistent with van Manen’s (1990) hermeneutic phenomenological approach. However, the reliance on uninterrupted periods for data immersion and coding presents practical challenges, especially in the context of competing academic responsibilities (Richards, 2014). In contrast, the GAI-assisted analysis highlighted the limitations of relying on GAI tools for qualitative research. GAI tools outputs often grouped several themes into broader categories, overlooking the subtleties of human experience that are central to qualitative inquiry (Mesec, 2023). Furthermore, crafting effective prompts emerged as a significant challenge, with the quality of the GAI’s outputs heavily dependent on the specificity and clarity of the researcher’s instructions (De Paoli, 2024). This reliance on prompt design reflects a methodological vulnerability in GAI-assisted analysis, where the lack of interpretative flexibility can lead to superficial or incomplete findings (Ray, 2023). Additionally, the GAI’s inability to address cultural and contextual dimensions raises concerns about its applicability in studies requiring deep interpretative engagement. For instance, while the manual researcher identified complex cultural challenges faced by Chinese graduates, the GAI grouped these into generalized themes, missing the detailed insights critical for understanding such phenomena.

The study’s findings point toward the potential of hybrid approaches that combine the strengths of manual and GAI-assisted methods. GAI tools can expedite initial stages of analysis, such as coding and broad theme identification, allowing researchers to focus their efforts on deeper interpretative tasks. This aligns with recommendations from Christou (2024) and Hamilton et al. (2023), who advocate for the integration of GAI into qualitative research as a means of enhancing efficiency without compromising depth. Moreover, the importance of prompt design emerged as a critical insight for optimizing GAI outputs. Developing standardized frameworks for crafting effective prompts could improve the reliability and contextual sensitivity of GAI-assisted analysis (De Paoli, 2024). Such frameworks would enable researchers to leverage GAI’s capabilities while mitigating its limitations, ensuring that the findings are both methodologically sound and contextually relevant. However, the integration of GAI tools into qualitative research also raises ethical and methodological considerations. As Grimes et al. (2023) noted, the complexity of GAI algorithms can undermine transparency, complicating efforts to ensure accountability and rigor in qualitative analysis. Addressing these concerns requires a critical approach to GAI integration, where human oversight remains central to the analytical process.

The integration of GAI in qualitative research raises important questions about the evolving paradigm of science. Traditional qualitative methods emphasize researcher subjectivity, reflexivity, and deep contextual interpretation, whereas GAI-assisted analysis introduces algorithmic pattern recognition, potentially shifting the balance between human intuition and machine-driven insights. This development aligns with broader discussions on the role of GAI in research, where scholars debate whether GAI functions merely as a tool to enhance human analysis or represents a shift toward a more hybrid, human-machine epistemology (Williams, 2024). While this study does not aim to resolve this debate, it acknowledges that incorporating AI in qualitative analysis challenges existing methodological frameworks and invites further inquiry into the implications for knowledge production in qualitative research.

Conclusion

This study explored the experiences and reflections of researchers conducting qualitative thematic analysis using manual and GAI-assisted methods, highlighting the expectations, challenges, and future insights. The findings reaffirm the unique strengths of each approach: manual thematic analysis provided in-depth, context-rich insights through immersive engagement with data, while GAI-assisted methods showcased remarkable efficiency in processing large datasets and identifying broad patterns. These findings contribute to the growing discourse on integrating traditional and emerging methodologies, offering valuable perspectives on how researchers can balance depth and efficiency in qualitative research.

The study contributes to the broader field of research methodologies by providing empirical insights into the experiential dimensions of using manual and GAI-assisted methods. It affirms the potential of GAI tools to enhance efficiency, particularly in initial stages of analysis, while reinforcing the importance of human oversight to ensure interpretative rigor and authenticity in qualitative research. This work underscores the importance of hybrid approaches that combine the efficiency of GAI with the depth and contextual sensitivity of manual methods, paving the way for innovative practices in qualitative analysis.

Despite its contributions, this study has limitations. The sample size was limited to the reflections of two researchers, which may not capture the diversity of experiences across different disciplines or datasets. Additionally, the analysis relied on a single GAI tool, ChatGPT, which may limit the generalizability of findings to other GAI platforms. The study also focused on thematic analysis, leaving room to explore the applicability of GAI-assisted methods in other qualitative research approaches, such as grounded theory or narrative analysis. Future studies should address these limitations by incorporating a broader range of participants, datasets, and GAI tools. Comparative studies across disciplines could provide deeper insights into the varied applications of GAI in qualitative research. Moreover, research on developing standardized prompt frameworks and integrating metadata in interview transcripts could enhance the reliability and contextual sensitivity of GAI-assisted analyses. Finally, exploring ethical considerations and best practices for hybrid methodologies will be critical to advancing the field while maintaining the integrity of qualitative research.

Footnotes

ORCID iD

Anas Al-Fattal

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data Availability Statement

The data that support the findings of this study are not publicly available as the authors have chosen not to share them.

References

Alexander

Bahja

Butt

(2022). Automating large-scale health care service feedback analysis: Sentiment analysis and topic modeling study. JMIR Medical Informatics, 10(4), Article e29385. https://doi.org/10.2196/29385

Al-Fattal

Sisavath

Singh

J. K. N.

(2024). Virtual mentorship in academia: A collaborative autoethnography of cross-border academics. The Australian Educational Researcher, 1–22. https://doi.org/10.1007/s13384-024-00795-0

Bano

Zowghi

Whittle

(2023). AI and human reasoning: Qualitative research in the age of large language models. AI Ethics Journal, 3(1), 1–15. https://doi.org/10.47289/AIEJ20240122

Braun

Clarke

(2013). Successful qualitative research: A practical guide for beginners: Sage.

Bryant

(2002). Re-Grounding grounded theory. The Journal of Information Technology Theory and Application, 4(1), 25–42.

Burger

Kanbach

D. K.

Kraus

Breier

Corvello

(2023). On the use of AI-based tools like ChatGPT to support management research. European Journal of Innovation Management, 26(7), 233–241. https://doi.org/10.1108/EJIM-02-2023-0156

Chang

Ngunjiri

F. W.

Hernandez

K. C.

(2013). Collaborative autoethnography. Routledge.

Charmaz

(2014). Constructing grounded theory (2nd ed.). Sage Publications.

Christou

(2023). How to use artificial intelligence (AI) as a resource, methodological and analysis tool in qualitative research? The Qualitative Report, 28(7), 1968–1980. https://doi.org/10.46743/2160-3715/2023.6406

10.

Christou

P. A.

(2024). Thematic analysis through artificial intelligence (AI). The Qualitative Report, 29(2), 560–576. https://doi.org/10.46743/2160-3715/2024.7046

11.

Creswell

J. W.

(2008). Educational research: Planning, conducting and evaluating quantitative and qualitative research (3rd ed.): Pearson Prentice Hall.

12.

Creswell

J. W.

(2013). Qualitative inquiry and research design: Choosing among five approaches (3rd ed.): Sage Publications.

13.

Creswell

J. W.

Poth

C. N.

(2017). Qualitative inquiry and research design: Choosing among five approaches (4th ed.). Sage Publications.

14.

de Oliveira Silva

dos Santos Janes

(2021). The emergence of ChatGPT and its implications for education and academic research in the 21st century. Review of Artificial Intelligence in Education, 2(00), e06. https://doi.org/10.37497/rev.artif.intell.educ.v3i00.6

15.

De Paoli

(2024). Performing an inductive thematic analysis of semi-structured interviews with a large language model: An exploration and provocation on the limits of the approach. Social Science Computer Review, 42(4), 997–1019. https://doi.org/10.1177/08944393231220483

16.

Dos Anjos

de Souza

de Andrade Neto

de Souza

, et al. (2024) (Submitted for publication). An analysis of the generative AI use as analyst in qualitative research in science education. Revista Pesquisa Qualitativa, 12(30), 1–29. https://doi.org/10.33361/RPQ.2024.v.12.n.30.724

17.

Fossey

Harvey

Mcdermott

Davidson

(2002). Understanding and evaluating qualitative research. The Australian and New Zealand Journal of Psychiatry, 36(6), 717–732. https://doi.org/10.1046/j.1440-1614.2002.01100.x

18.

Grimes

Von Krogh

Feuerriegel

Rink

Gruber

(2023). From scarcity to abundance: Scholars and scholarship in an age of generative artificial intelligence. Academy of Management Journal, 66(6), 1617–1624. https://doi.org/10.5465/amj.2023.4006

19.

Hamilton

Elliott

Quick

Smith

Choplin

(2023). Exploring the use of AI in qualitative analysis: A comparative study of guaranteed income data. International Journal of Qualitative Methods, 22, 1–13. https://doi.org/10.1177/16094069231201504

20.

Hornsby

E. R.

Davis

Reilly

J. C.

(2021). Collaborative autoethnography: Best practices for developing group projects. InSight: A Journal of Scholarly Teaching, 16(1), 104–107. https://doi.org/10.46504/16202105ro

21.

Huang

Tan

(2023). The role of ChatGPT in scientific communication: Writing better scientific review articles. American Journal of Cancer Research, 13(4), 1148–1154.

22.

Jones

S. H.

Adams

Ellis

(2016). Introduction: Coming to know autoethnography as more than a method. In Handbook of autoethnography (pp. 17–48). Routledge. https://doi.org/10.4324/9781315427812

23.

Kasperiuniene

(2021). The use of artificial intelligence in social research: Multidisciplinary challenges. In Costa

Reis

Moreira

Longo

Bryda

(Eds.), Computer supported qualitative research. WCQR 2021. Advances in Intelligent Systems and Computing (Vol. 1345). Springer. https://doi.org/10.1007/978-3-030-70187-1_22

24.

Lapadat

J. C.

(2017). Ethics in autoethnography and collaborative autoethnography. Qualitative Inquiry, 23(8), 589–603. https://doi.org/10.1177/1077800417704462

25.

MacLennan

(2012). Immersed in cyberspace: An evaluation of a grounded theory approach to a study of user preferences. Library and Information Research, 36(112), 55–71. https://doi.org/10.29173/lirg493

26.

Mayring

(2014). Qualitative content analysis: Theoretical foundation, basic procedures and software solution. Beltz.

27.

McClure

J. W.

(2003). The experiences of Chinese international postgraduates studying in Singapore. [Unpublished Doctorate dissertation]. Griffith University.

28.

Mesec

(2023). The language model of artificial intelligence ChatGPT: A tool for qualitative analysis of texts. Authorea Preprint . https://doi.org/10.22541/au.168182047.70243364/v1

29.

Perkins

Roe

(2024). The use of Generative AI in qualitative analysis: Inductive thematic analysis with ChatGPT. Journal of Applied Learning and Teaching, 7(1), 390–395. https://doi.org/10.37074/jalt.2024.7.1.22

30.

Ravitch

S. M.

Carl

N. M.

(2019). Qualitative research: Bridging the conceptual, theoretical, and methodological (2nd Ed.). Sage Publications.

31.

Ray

P. P.

(2023). ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems, 3, 121–154. https://doi.org/10.1016/j.iotcps.2023.04.003

32.

Richards

(2014). Handling qualitative data: A practical guide (3rd ed.). Sage Publications.

33.

Sakurai

Shimauchi

Shimmi

Amaki

Hanada

Elliot

D. L.

(2021). Competing meanings of international experiences for early-career researchers: A collaborative autoethnographic approach. Higher Education Research & Development, 41(5), 1012–1028. https://doi.org/10.1080/07294360.2021.2014410

34.

Siiman

L. A.

Rannastu-Avalos

Pöysä-Tarhonen

Häkkinen

Pedaste

(2023, August). Opportunities and challenges for AI-assisted qualitative data analysis: An example from collaborative problem-solving discourse data. In International conference on innovative technologies and learning (pp. 87–96). Springer Nature.

35.

Singh

J. K. N.

(2019). Why do Chinese international students studying in Australia repatriate? Australian Chinese graduates tell it all. Journal of Further and Higher Education, 44(9), 1207–1219. https://doi.org/10.1080/0309877X.2019.1669771

36.

Singh

J. K. N.

(2020). Challenges in obtaining employment in China: Lived experiences of Australian Chinese graduates. Australian Journal of Career Development, 29(3), 153–163. https://doi.org/10.1177/1038416220947085

37.

Singh

J. K. N.

Fan

(2021). International education and graduate employability: Australian Chinese graduates’ experiences. Journal of Education and Work, 34(5-6), 663–675. https://doi.org/10.1080/13639080.2021.1965970

38.

Smetana

Arora

Chaudhuri

Lewis

Goris

(2021). Sustaining academic careers amidst COVID-19: A collaborative autoethnography of women academicians. Journal of Academic Leadership, 10(2), 215–222.

39.

Tabone

de Winter

(2023). Using ChatGPT for human-computer interaction research: A primer. Royal Society Open Science, 10(9), Article 231053. https://doi.org/10.1098/rsos.231053

40.

van Manen

(1990). Researching lived experience: Human science for an action sensitive pedagogy. The Althouse Press.

41.

Wertz

F. J.

(2011). Five ways of doing qualitative analysis: Phenomenological psychology, grounded theory, discourse analysis. Narrative research, and Intuitive inquiry. Guilford Press.

42.

Williams

R. T.

(2024). Paradigm shifts: Exploring AI's influence on qualitative inquiry and analysis. Frontiers in Research Metrics and Analytics, 9, Article 1331589. https://doi.org/10.3389/frma.2024.1331589