Me and the Machines: Possibilities and Pitfalls of Using Artificial Intelligence for Qualitative Data Analysis

Abstract

This paper describes how one researcher learned to overcome artificial intelligence (AI) paralysis and embrace ChatPDF. This freely available AI application uses natural language processing (NLP) to respond to user queries about an uploaded PDF. Researcher insights from experimenting with the AI tool ChatPDF for qualitative data analysis are presented, highlighting the advantages, pitfalls, and application-related considerations. As a two-phase curiosity experiment, the researcher engaged in a theory-building exercise to explore key concepts for understanding when using ChatPDF to assist researchers in qualitative data analysis. The experiment generated insights about the purposeful use of AI tools that incorporate NLP for analysis and the risks of inaccuracy when researchers are not familiar with the data or skilled in prompt engineering. Insights raise questions about whether ChatPDF is a viable research assistant for qualitative researchers, ethical issues with specific forms of qualitative data, and the potential of AI tools for community and student researchers.

Keywords

methods in qualitative inquiry ChatGPT data analysis AI vignettes narrative analysis

Introduction

On April 3^rd, 2023, Twitter user @heykahn, affectionately self-dubbed The AI Guy, tweeted: “ChatGPT is just the tip of the iceberg. 1,000+ AI tools were released in March.” This tweet left me equal parts curious and terrified. An internal dialogue ensued. What if AI can do the job better than me? How do I possibly have the time to learn more tech tools? After spinning a similar line of questioning for the better part of 2 hours and going down the metaphorical AI rabbit hole, I noticed trends in its benefits for supporting research work. I began a slight shift to growth opportunity thinking while remaining skeptical. Should I collaborate with the machine? I’d been doing that for the better part of 7 years now, hadn’t I? Using Trello for team projects, NVivo to assist with qualitative data management, Zotero to store references, Screencast-O-Matic for creating and editing videos, and SPSS for statistical analysis. In a way, working directly with AI was essentially a leveling up of that preexisting relationship. After all, being an overburdened university lecturer during “the COVID years” necessitated quickly acquiring new technological knowledge. One of my greatest fears arising during that experience was becoming technically obsolete. This article emerged from a eureka moment of how AI tools, specifically ChatPDF, might allow me (and others in similar predicaments) to dedicate more time to analyzing dormant study findings urgently requiring dissemination.

ChatPDF (n.d.) is a German-created artificial intelligence (AI) tool that uses natural language processing (NLP) to understand and address user queries. NLP is a branch of AI that enables computers to comprehend, decipher, and generate human language (Woo, 2019; Yew et al., 2023). ChatPDF does this through the integration of ChatGPT, a separate AI tool that uses NLP to emit responses to individual enquires that are human-like. ChatPDF can analyze the user’s input (i.e., the uploaded PDF), recognize the aim or objective received from the researcher, and formulate a relevant response. While not classified as a fully open-source tool accessible to the public under an open-source license, ChatPDF (n.d.) has a freely available version.

Unlike its more famous cousin, ChatGPT, which is known for rapid writing and translation of documents (Tate et al., 2023), generating results from qualitative data (Tabone & de Winter, 2023), composing artistic writing (Kirimani, 2022), climate change research (Biswas, 2023), text analysis and assessment (Zhong et al., 2023), potential for scholarly peer review (Hosseini and Horbach, 2023) and express decoding of programming language (Sobania et al., 2023), among other functions, the potential pragmatic uses for ChatPDF to assist qualitative data analysis (QDA) have not yet been explored. I decided to engage in an experiment studying the accuracy of ChatPDF as an “AI assistant” to condense transcripts on arts-based journey maps of sexual learning and exposure experiences from a project completed with Kenyan youth organization members. The goal was to reproduce each transcript as a personal vignette. A starting premise for the experiment was that ChatPDF would expedite the time from data analysis to reporting the insights to inform youth-led community prevention workshops on adolescent-friendly sexual and reproductive services in the Coast province of Kenya.

This article makes a significant contribution to the landscape of qualitative research in three ways. First, by explaining the steps in a framework I propose might be used or adapted when conducting analysis with the assistance of an AI tool using NLP in QDA, a new and burgeoning field of scholarship. I present the stages of an experiment born out of curiosity that evolved into an incessant need to understand the potential of AI assisting QDA. The raw data used in this experiment with ChatPDF, an AI tool that uses NLP, were derived from a project on the sexual learning and exposure experiences of Kenyan youth organization members. These experiences were shared during a session where arts-based journey mapping (Gray et al., 2019) initiated conversations on sex-related issues. While findings from the analysis will be reported elsewhere, this article will secondly clarify how the transcripts from arts-based techniques for data collection can be rapidly translated to narrative vignettes as the unit of analysis when an AI tool, like ChatPDF, is used. For years, data analysis has been likened to fragmentation and reduction (Miles et al., 2014). AI presents opportunities for rapid reduction—condensing large sets of textual data to parts significant to the research question so that deeper interpretative analysis can ensue. Third, this article presents insights into the potential and pitfalls of qualitative researchers employing the assistance of AI tools in analysis and the presentation of findings of arts-based research. The results from the experiment exhibit a number of benefits when the appropriate cautions are applied—ethical complexities and limitations of this process are also explicated. Finally, a secondary aim of this article is to advance conversations around how freely available AI tools can complement QDA in research while expediting bench-to-beside or research to practice knowledge translation.

Overview of AI Tools Using NLP for Qualitative Data Analysis

For over 2 decades, several authors have investigated the potential of using NLP models to perform a variety of nonautomated and automated analytical functions aiding the QDA process (Ancheta et al., 2020; Cheligeer et al., 2022; Fang et al., 2022; Guetterman et al., 2018; Leeson et al., 2019; Perrin, 2001; Skeen et al., 2022). In more recent years, AI-powered tools that use NLP to perform fully automated actions as part of the QDA process are on the rise. Authors implementing these models reported timesaving, reduction in costs and research-related resources, coupled with the possibility of analyzing larger sample sizes, were noted as the most common benefits. These tools are especially valuable in clinical settings (e.g., Nath et al., 2016) where the institution covers purchasing or membership costs to integrate the digital tools in practice. As an example of investment costs, Table 1 presents collated information on purchasing and estimated training costs for each popular digital tool incorporating NLP as reported on developers’ websites.

Table 1.

Estimated Costs of QDA Tools With NLP Functionality.

AI tools with NLP functionality for QDA	Costs (USD)
BERT (bidirectional encoder representations from transformers)	Purchasing = free
	Training = $300–$400
MAXQDA	Purchasing = $204 for a yearly subscription for one academic
MAXQDA	Training = free to $225 per person
NVivo	Purchasing = one-time purchase of $1,019
NVivo	Training = free to $300–$400
Python	Purchasing = $5–$500 per month
Python	Training = $3,000–$5,000
Leximancer	Purchasing = $1,271 per year
Leximancer	Training = $117 per hour for individual sessions

Larger institutions can more readily afford to invest in training their personnel to use the integrated tools. For smaller community-based and nongovernmental organizations, which are notoriously underfunded, siphoning precious funds for digital tools that have the potential to enhance data analysis efficiency to evidence outcomes may not be the most prudent use of resources. The same access problem might exist for graduate students whose university has not invested in QDA digital tools.

These costs alone are enough to trigger consideration of the potential benefits of freely available AI tools for QDA. ChatGPT is the AI tool du jour. Originally released in 2018, it is a tool owned by OpenAI that has the most research evidence produced thus far. ChatGPT has been explored to increase lecturer productivity (Kraugusteeliana et al., 2023) and automated program repair (Sobania et al., 2023). Others have investigated its potential for research, education, personal skill development and project support (Islam & Islam, 2023). Researchers from China and Australia have examined ChatGPT against other NLP models to compare understanding ability (Zhong et al., 2023). Of the four studies retrieved that mentioned ChatGPT in the context of higher education or academic research (Islam & Islam, 2023; Kraugusteeliana et al., 2023; Tabone & de Winter, 2023; Wen & Wang, 2023), only one (Tabone & de Winter, 2023) demonstrated the potential of the AI tool for QDA through an example.

Tabone and de Winter’s (2023) preprint on using ChatGPT in human–computer interaction (HCI) research used previously analyzed data to understand the viability of ChatGPT to aide in QDA. The primary conclusion from using ChatGPT for the analysis of 300 transcripts was that it enabled fast processing of large amounts of data, which is not possible within traditional boundaries of qualitative research. This has implications for subjective biases of the researchers through applying a transparent and replicable design. Tabone and de Winter (2023, p. 3) defined HCI research as:

the field that focuses on the design and use of computer technology and examines interfaces between people and computers. HCI research may involve various types of methods, including questionnaires, interviews, psychophysics methods, virtual reality setups, and field tests. In this process, the HCI researcher often collects text data.

Findings from this study raise the possibility of exploring further HCI methodologies as a subdomain of qualitative research. While authors such as Longo (2020) evidence how AI has extended QDA methods, it is worth noting that, at the time of this writing, no studies discussing the use of freely available AI tools similar to ChatGPT for QDA were located.

Application of Vignettes in Qualitative Research

A vignette comprises brief descriptive writing focused on a specific moment, scene, character, or idea. Qualitative research has typically utilized vignettes as a methodological tool to present fictious narratives to participants as a means of prompting their reflection on, or discussion of, a particular topic during a data collection session—although real-life vignettes are sometimes used as well (Sampson & Johannessen, 2020). The use of vignettes as a means of presenting data is less popular by comparison; however, there are several successful examples documented in scholarly literature.

Jacobsen (2014) demonstrated that vignettes are a useful method of combining research observations and insights from participant interviews to enhance the reporting of ethnographic accounts and show the value of interview data in ethnographic studies. Reay et al. (2019) also highlighted the potential of vignettes for presenting findings from qualitative research. They argued that a one-size-fits-all approach to presenting data is not helpful and that vignettes are useful when the nature of the data is textual, narrative, or both, with the goal of demonstrating insights around a overarching theme. It is important to note that the term vignettes is sometimes used interchangeably with story stem in Allen et al. (2018), scenario or case study (see Stacey et al., 2014). Reay et al. (2019, p. 8) stated that presenting data through vignettes “foregrounds the interconnections among categories and gives space to show the findings in ways that capture more richness in an easily-readable and credible way” satisfying the criterion of credibility in qualitative research. In Canada, Blodgett et al. (2011) explored using vignettes to centralize Indigenous voices of co-researchers shared in conversational interviews. The authors noted that vignettes are a powerful method of presenting data through story when working in partnership with Indigenous communities. Storytelling aligns with epistemological values of Indigenous peoples and serves to preserve cultural knowledge.

Presenting vignettes from interview and other textual data is becoming increasingly popular in several disciplines where qualitative research is used. Vignettes for presenting findings have been applied to experiences of migration and education (Vartak, 2020), institutionalized young people (Soares, 2019), incarcerated adolescents (McCuish et al., 2019), movement in physical education (Lambert, 2020), coaching (Paquette et al., 2019), families who home school (Bower, 2021), ethics of care when working with refugee families during COVID-19 (Keary et al., 2022), refugees’ sense of community (Fader et al., 2020), chronic illness (Maslen & Lupton, 2020) and Cloud computing (Wulf et al., 2021) among others. Spalding and Phillips (2007) also described the different forms a vignette could take, but their description of portrait vignettes as representing “participants’ character and experience” (p. 957) aligns closely with the narrative vignettes developed through these experiments.

The Experiment: Using an AI Tool to Translate Arts-Informed Data to Vignettes

This experiment involved the translation of arts-based journey map transcripts (i.e., arts-informed data) to narrative vignettes, which was achieved across two experimental phases. The first phase presents a framework for others, supplemented by examples and reflections on the value of these steps from my investigations. Phase 2 of the experiment captures conversations I had with the machine (i.e., ChatGPT in the ChatPDF tool) to begin defining operational definitions for steps applied in the framework.

Phase 1—Steps for AI Conversion of Raw Data to Unit of Analysis

Phase 1 of this experiment occurred through four overarching steps, as illustrated in Figure 1:

1. Researcher is familiar with raw data (e.g., transcript)

2. Upload PDF of transcript

3. Insert prompt question to generate a vignette about the topic of study

3.1. Apply prompt engineering that is, refine prompt questions to ensure the vignette accurately reflects participant’s experience.

3.2. Researcher modification of vignette where necessary

4. Participant member-checks vignette

Figure 1.

Framework for AI Conversion of Raw Data to a Unit of Analysis.

This experiment used 16 personal journey map discussions on sexual learning and exposure, from members of a Kenyan youth organization, that were previously manually transcribed by Kenyan research assistants and translated from Kiswahili to English. All names used in transcripts and reported in Table 2 were pseudonyms chosen by the participants. I had spent a few weeks familiarizing myself with the transcripts when I initially planned to manually generate vignettes. The transcripts were converted from Word to PDF in preparation for uploading to ChatPDF—it is the only accepted file format. All vignettes were generated in the first instance by ChatPDF after individual upload.

Table 2.

Time to Translate Transcript to a Vignette From Journey Map Using ChatPDF.

Participant	# of words in transcript	Translation to vignette speed (secs)
Amore	410	29.28
Digadiga	840	21.02
Ghost	903	24.33
Janette	674	27.10
Keila	732	22.69
Kimonyoski	485	25.73
Kinyonyi	618	28.43
Kirambo	1,095	26.75
Mamuska	343	25.33
Munthaz	442	26.62
Myeng	1,309	24.83
Peps	846	19.20
Rhianna	1,919	26.22
Teff	966	26.19
Teja	337	18.83
Thunush	384	28.69
Total number of transcripts → n = 16	Average transcription word length → n = 769	Average translation speed →n = 25.08

Despite the developer’s description in the tools frequently asked question (FAQ) tab stating a limit of three PDF uploads a day at 120 pages each, I could upload and process all 16 transcript PDFs for the experiment discussed in this article within the same hour. The time it took to translate each transcript to a vignette is recorded in Table 2.

The speed of transcript to vignette translation depended on how prepared I was with the vignette-creation prompt question: Can you produce a vignette from [Name’s] journey map? After prompting seven of the transcript PDFs with the original question, ChatPDF changed its output to a more fragmented version of the participant journey map, as in Figure 2. It is worth noting that, on average, those transcripts with fewer words took longer to generate a vignette—presumably as a result of having less detail from which to build a narrative, but further study is needed to draw any firm conclusions.

Figure 2.

ChatPDF Output Alteration Despite Using the Original Prompt Question.

As the researcher wanted to compare narrative vignettes, the prompt question had to be adapted to have an exceedingly clear directive: Can you produce a narrative vignette from [Name’s] journey map? This increased the time from transcript to vignette translation by 24 seconds. Figure 3 represents the adapted output, which aligned with the previous formats of ChatPDF-generated vignettes using the first question. These actions are known officially as prompt engineering—a crucial skill set for communicating effectively with language models like ChatGPT (White et al., 2023). Given that qualitative data from different participants will tell diverse stories, the workflow input into ChatPDF may not always produce a vignette that reflects the participant’s story with 100% accuracy. ChatPDF depends on the use of informed prompt engineering and, more importantly, the researcher’s familiarity with the data. At that point, it was clear that I was still honing this skill set.

Figure 3.

ChatPDF Output Alteration Despite Using Altered Prompt Question.

As is denoted in Figure 3, after reading the output, I flagged inconsistencies between the raw data and AI-generated vignette. I relied on my familiarization with the data to generate a more accurate representation of the experiences shared and refined the prompt question again: Can you provide an example of how Amore’s journey map could be presented in a vignette that brings in her discussion of menses and different ways she experienced sexual learning while growing up? This example of prompt engineering resulted in a vignette that was the essence of the participant’s experience (Figure 4):

Figure 4.

Refining Prompt Question for Inclusion of Accurate Details.

Similarly, some vignettes had to be regenerated in cases where original transcripts less obviously included sex-related terminologies or had additional information from their lives that was not as relevant to the research topic (see Figure 5). In such cases, a new prompting question was developed to ensure the vignette generated through ChatPDF highlighted the participant’s narrative around sexual learning and exposure. The new prompt was used in Figure 6 to refine the prompt question (when necessary): Can you generate an example of how [Name’s] journey of sexual learning and exposure can be presented as a narrative vignette?

Figure 5.

Refining Prompt Question Required to Focus the Vignette on the Research Topic.

Figure 6.

Adjusted Prompt Question Applied.

Final verbatim vignettes that captured sexual learning and exposure stories of the participants as produced by ChatPDF were cross-checked with original journey map transcripts as an assurance that key aspects of participants’ sexual learning and exposure narratives were not negated. The researcher then modified the vignette to ensure sufficient contextual details were included to reflect the context accurately. For example, in participant Amore’s vignette, the initial and final paragraphs were added to capture the learning she experienced in childhood and since joining the organization (see Figure 7).

Figure 7.

Researcher-Modified AI Vignette.

Given this was a curiosity experiment and not yet part of an agreed process for analysis, two small samples of the AI-generated vignettes were sent to the research assistant in Kenya, who invited the participants to engage in a member-checking process (Candela, 2019) of their vignettes. All the participants confirmed the vignettes were accurately storied to reflect their sexual learning and exposure journeys.

The insights from the experiment are only sample snapshots of the possibilities that translating journey maps and associated transcripts (or other textual data) to vignettes with AI-assistance can yield. It was not simply the act of summarizing or reducing the data that was helpful; ChatPDF accelerated the storying process, which is a laborious task for the researcher.

Phase 2—Me and the Machines: Concept-Building Conversations

Given that the use of ChatPDF for QDA had not yet been explored in the literature, I wanted to understand how it could transform a two-page transcript in 26.75 seconds. At this stage, I did not have the same understanding of what it meant for a tool to be using NLP and was learning the terminology. As academics, there is sometimes an incessant need to define, name or categorize. I needed to know more about the specific type of AI algorithms utilized by ChatPDF and how to clearly distinguish its usage from well-known software that supports QDA, such as NVivo. To find the answer, I initiated a conversation with one machine (i.e., ChatGPT) about the other machine (ChatPDF) and proceeded to move between both to determine key operating concepts for advancing AI-assisted QDA. What follows is an excerpt from our conversations. As the dialogue unfolds, bear in mind that the NLP in both AI tools has a “learn as you go” capacity.

Me: NVivo is also an artificial intelligence tool like you. How would you say it differs from ChatPDF in terms of classifying the types of AI used in each?

ChatGPT: NVivo is a software tool used for qualitative data analysis, and it is designed to help researchers manage, analyze, and visualize qualitative data such as text, images, audio, and video. NVivo is not an artificial intelligence tool in the sense that it does not use machine learning or other forms of AI to analyze data. Rather, it is a tool that helps researchers organize and analyze qualitative data in a systematic and efficient way.

. . . ChatPDF is not designed to analyze data on its own but rather to provide a standardized format for researchers to store and analyze chat data using other tools or methods.

In terms of how we classify the types of AI used in each tool, NVivo would not typically be classified as an AI tool since it does not use machine learning or other forms of AI. However, it could be considered an example of “AI-assisted” qualitative data analysis since it uses software to help researchers manage and analyze large amounts of data.

ChatPDF, on the other hand, would not be classified as an AI tool at all since it does not use any form of AI. It is simply a file format for storing and analyzing chat data.

At this point, I knew that the answer generated was not 100% accurate. However, I tolerated the conversation with the tool a little longer to see what other falsities it could conjure:

Me: Hmm, I am not sure I follow. Because I can give ChatPDF my research transcripts and ask it to turn a transcript into a vignette, for example. So, it is assisting me with qualitative data analysis by organizing my raw data.

ChatGPT: I see what you mean. In that case, if you are using an AI-based tool or algorithm to assist you with qualitative data analysis, then yes, you could classify ChatPDF as an AI tool that assists with qualitative data analysis.

However, it’s important to note that ChatPDF itself is not an AI tool but rather a format for representing chat data. The AI-based algorithm that you are using to generate vignettes from your research transcripts is a separate tool or program that is using the data stored in ChatPDF format.

In general, we can classify AI tools based on the type of AI they use. Some AI tools use rule-based systems, where the tool follows a set of predefined rules to make decisions or perform tasks. Other AI tools use machine learning, where the tool is trained on data to recognize patterns or make predictions.

Me: You said, “The AI-based algorithm that you are using to generate vignettes from your research transcripts is a separate tool or program that is using the data stored in ChatPDF format,” but I uploaded the transcript to ChatPDF then asked it to turn the transcript into a vignette. So how is it a separate tool? Sorry, I am not following.

ChatGPT: I apologize for the confusion. You are correct that in the specific example you gave, ChatPDF itself could be considered an AI tool that assists with qualitative data analysis since it provides a way to store and organize chat data, which can be used by an AI-based algorithm to generate vignettes.

Now we were getting somewhere. Without dialoguing with me and learning from my responses, ChatGPT was not able to accurately define what type of AI ChatPDF was using or even if it could be classified as an AI tool. This was quickly cleared up by a quick search in the FAQs section on ChatPDF.com that revealed “ChatPDF uses GPT 3.5 for now, which is the same as ChatGPT. We are looking at how to add GPT-4. But GPT-4 won’t be available for all messages on the free plan because it costs too much.” This was also a flag that with all the new developments on the AI scene of late, perhaps each tool is not familiar with the other until a conversation is lodged about the topic. This thought crossed my mind given ChatGPT often declares it only has access to information up until 2021. This was also a great reminder that overreliance on AI outputs can lead to misinformation, and the researcher must possess enough content expertise to remain sensitive to identifying such instances.

ChatGPT: I think the source of confusion came from my earlier statement that ChatPDF is not an AI tool, which was meant to convey that ChatPDF itself does not contain any AI algorithms or functionality. However, it can be used with other AI-based tools or algorithms to assist with qualitative data analysis, such as generating vignettes from chat transcripts. For example, ChatPDF can be used as part of an AI-assisted qualitative data analysis workflow where other software tools or algorithms that incorporate machine learning or other forms of AI can be used to assist with the analysis of chat data. For example, NVivo is a software tool that can be used to analyze qualitative data, and it has some AI-based functionality, such as NLP, that can be used to identify patterns or themes in chat data.

In such a workflow, ChatPDF can be used to store and organize the chat data [i.e., uploaded text documents], and then other AI-based tools or algorithms can be used to analyze the data. This can help researchers manage and analyze large amounts of chat data more effectively, potentially identifying patterns or insights that might not be apparent through manual analysis alone.

Me: Thanks, I get it now :)

I had come across the term workflow during my earlier deep-dive on NLP models for qualitative research. In their discussion of mixed-methods text analysis, Parks and Peters (2022) described the importance of a workflow approach. The workflow approach emphasizes the need to critically reflect on the design choices for analysis and the research questions themselves—insights that led to the development of a framework for the experiment (see Figure 1).

ChatGPT: You’re welcome! I’m glad I could help clarify things for you. Don’t hesitate to ask if you have any further questions.

Me: Can you provide a definition of chat data in ChatPDF?

ChatGPT: Sure! In the context of ChatPDF and qualitative data analysis, chat data refers to any text-based communication that occurs between two or more people in real time. This can include conversations that take place over instant messaging platforms, social media messaging apps, or other chat applications.

Examples of chat data that could be stored in ChatPDF format include chat logs from online support groups or forums, chat transcripts from customer service interactions, or transcripts of chat conversations from research studies.

This was an interesting insight. Chat data (i.e., text data from oral presentations of journey maps as used in this experiment) is a core concept for understanding the types of data that can be queried for QDA using ChatPDF. Chat data can be any form of data that the user uploads in a PDF version to the software and is critical concept-building definition for AI-assisted QDA. For example, individual interview or focus group transcripts could be a common form of chat data. However, as generative AI evolves, ChatPDF may also expand what constitutes chat data to include visual and audio data, which is the case for other AI tools.

ChatPDF provides a standardized format for storing and organizing chat data, which can make it easier for researchers to manage and analyze this type of data. By using ChatPDF, researchers can organize chat data into different tiers or levels, such as Speaker, Utterance, or Turn, which can help with the coding and analysis of the data.

A few moments later, after switching over to ChatPDF and completing the last transcript-to-vignette translation, I had to confirm the accuracy of ChatGPT statements that tiers of organization were applied:

Me: Did you use the Speaker, the Utterance or Turn to produce the vignette?

ChatPDF: I used the turn to produce the vignette. The turn is a unit of conversation that consists of one person speaking and another person responding. In this case, I presented Teff’s journey map as a series of turns where she encountered different people who helped her learn about sex and sexual health.

Me: What would be the difference if you used the speaker?

ChatPDF: If I used the speaker to produce the vignette, it would mean that I am presenting Teff’s journey map as if she were speaking directly to the reader. The speaker would be Teff herself, and she would be sharing her experiences and thoughts in her own words.

Using the speaker would give a more personal touch to the vignette, as it would be presented from Teff’s perspective. It could also provide more insight into her emotions and feelings throughout her journey. However, it may not be as concise or organized as using the turn, as Teff may jump around different topics or events without a clear structure.

Me: Can you produce Teff’s vignette using the speaker?

ChatPDF: Sure, here’s an example of Teff’s journey map presented using the speaker:

Hi, my name is Teff and I wanted to share my journey in understanding sex and sexual health. It all started when I was in class seven and my mother would call me and my big sisters to speak about body changes and adulthood. She emphasized the importance of taking care of ourselves so that we wouldn’t get pregnant.

During high school, I heard stories from my friends about what they did with their boyfriends during half term. I was curious about sex but didn’t know much about it. One day, I heard screaming from the neighbors and asked the maid what was happening. The maid acted as an educator because my parents were not willing to speak about these issues with me.

In college, I had friends who would skip classes during the day to have sex with their boyfriends and then explain what happened to me. At my attachment in a clinic, there were peer education sessions where we engaged youths because there were issues of teenage pregnancies on the rise. We also conducted outreaches on such issues and spoke about sex education with community members.

After that, I went to Dream Achievers where I learned terms such as “mjulubeng” [male who has sex with other males] and was exposed to many issues related to sexual reproductive health. Looking back at my journey map, I realize that I learned about sex and sexual health through various sources such as family, friends, educators, and community outreach programs. It wasn’t always easy or comfortable for me to talk about these topics, but each experience helped me gain a better understanding of myself and others around me.

Me: What would be the difference in the vignette if you used “utterance” to produce it?

ChatPDF: If I used utterance to produce the vignette, it would mean that I am presenting Teff’s journey map as a series of individual statements or phrases that she made throughout her journey. Each utterance would be presented as a separate piece of information without any additional context or explanation.

Using utterances could provide a more fragmented and disjointed view of Teff’s journey map, as each statement would be presented in isolation without any clear connection to the others. It may also be more difficult for the reader to follow along with the narrative, as they would need to piece together the different utterances to form a coherent story.

Here’s an example of what Teff’s journey map might look like using utterances:

- My mother used to call me and my big sisters to speak about body changes and adulthood.

- She emphasized taking care of ourselves so we wouldn’t get pregnant.

- I heard stories from my friends about what they did with their boyfriends during half term.

- I was curious about sex but didn’t know much about it.

- The maid acted as an educator because my parents were not willing to speak about these issues with me.

- At my attachment in a clinic, there were peer education sessions where we engaged youths because there were issues of teenage pregnancies on the rise.

- We also conducted outreaches on such issues and spoke about sex education with community members.

- I went to Dream Achievers where I learned terms such as “mjulubeng” and was exposed to many issues related to sexual reproductive health.

As you can see, each statement provides some information about Teff’s journey map, but it lacks the context and structure that would be provided if the tier of organization applied was either Turn or Speaker. This part of the conversation gleaned important insights around operational definitions when using ChatPDF for the purposes of QDA. ChatPDF can organize chat data for vignettes or similar storying formats through three mechanisms. The first is Utterance, which uses individual statements made by the speaker in the chat data. An utterance can be represented by a word, a phrase, a sentence, or a lengthier passage of speech. ChatPDF assigns every word a distinctive number or code that differentiates it from other utterances. The second is Speaker, which refers to the person speaking in the chat data. Each speaker in uploaded raw data is assigned a unique identifier. The speaker is a vital element in the creation of a vignette or other story formats as a unit of analysis because they provide the events being described context and perspective. The third is Turn. A turn is a series of one or more utterances made by one speaker, which are followed by a similar series made by another speaker. Turns are crucial because they provide the discussion structure through conversational interactions. My conversation with the machines highlighted pertinent concepts for researchers using the tool in a way that aligns to their analysis and the importance of prompt engineering skills to recognize inconsistencies in the tool’s outputs.

Implications for Future Use of ChatPDF in QDA

Several insights can be reported based on my learning from the experiment and ChatGPT conversations. First impressions of ChatPDF were efficiency related. There is definite potential for researchers to get insights more rapidly and efficiently from their chat data (i.e., text data) by using ChatPDF as part of an AI-assisted QDA workflow.

Developing a framework also proved useful for understanding the nature of ChatPDF outputs. For example, application of a step-by-step framework ensured that the researcher-modified AI-generated vignettes honored stories shared by participants, demonstrating the rigor needed for AI-assisted QDA. The framework might be useful for future qualitative researchers to apply or adapt. Having previous experience constructing vignettes from similar raw data, the process of ChatPDF generating vignettes proved far less cumbersome following the steps in the framework, and in many cases, the prose was just as inspired and decorated with contextual detail as those produced solely by a researcher. Even in the case of Amore’s vignette, where ChatPDF distorted elements of the participant’s story, being familiar with the data made it easy to correct the error, but this raised the first ethical concern to be considered in using this tool.

Similarly, in Digadiga’s AI-generated vignette, some of the contextual factors around sexual learning and exposure were overlooked. These examples emphasize that the framework’s first step, being familiar with the data, is critical to guaranteeing representativeness and conveying contextual complexities. The second potential ethical concern is the possibility of prejudice in the AI-generated vignettes. The resulting vignettes could reveal hidden biases in the AI system or the training data. In fact, studies exist querying the potential for AI tools like ChatGPT to use of offensive language, be racist, sexist, colonizing, discriminatory, perpetuate stereotypes and push misinformation (Singh, 2023; Zhuo et al., 2023; Zuccon & Koopman, 2023). Scholars such as Bozkurt et al. (2023) presented a counterargument explaining that large language models such as those of ChatGPT (and inherently ChatPDF) utilize an international repository of knowledge that can be deployed to rectify historical and geographic injustices, such as by normalizing the use of Indigenous knowledges on scale with settler and colonial knowledges. Borji (2023) noted that while ChatGPT has precautions in place to safeguard against offensive and harmful language, these may not always be successful. This might maintain current power dynamics and inequities and result in a biased interpretation of the findings. In all instances where AI is used to assist data analysis, it is necessary to be forthcoming about the technique employed as well as any potential flaws and biases in the method.

The operational definition of concepts for using AI in QDA is imperative for the researcher’s active engagement with the tool to produce units of analysis aligned with the study’s purpose. Passive use of AI tools for qualitative research can lead to erroneous representation of participants’ stories as was demonstrated in Figure 3. Thus, the conceptual definitions of chat data and tiers of hierarchies of organization: Utterance, Speaker, and Turn, defined previously, are pertinent to the use of AI tools for assisting, and maybe even advancing, QDA. To ground the validity of these hierarchies in existing literature, readers of this article should consider the qualitative data management software NVivo, which can also be used as a tool by researchers to organize qualitative data. NVivo applies the concept of coding trees or coding hierarchies (Mortelmans, 2019) that include parent, child and sibling codes as levels of organization. The point of difference between NVivo’s hierarchies and those in ChatPDF is the researcher load or amount of effort required by the researcher. The researcher must still know what query to input, but, once done, ChatPDF organizes data based on your request and its inbuilt tiers of organization rather than the manual application of this task as in NVivo.

Possibilities of ChatPDF for Qualitative Research Without Vignettes

Given the capability to query uploaded transcripts in ChatPDF, the tool may help qualitative researchers requiring more structure in their analysis, for example, by performing a deductive thematic analysis where the qualitative researcher has a predetermined set of codes they are searching for in the data. A simple prompt in ChatPDF could locate all instances of a predetermined code immediately and pull out the example, similar to the text search query in NVivo, without the requirement of program knowledge to do the task.

The current experiment in ChatPDF might also help qualitative researchers using textual data in their analysis to produce thematic narratives. ChatPDF might assist researchers who employ other narrative analysis techniques. For example, Riessman’s (2008) tenets of narrative thematic analysis suggests the researcher should remain alert to specific patterns or read for specific topics and semantic subtexts as they analyze. In accordance with these tenets, but adapted for the assistance of ChatPDF, the researcher could give a prompt to extract all quotes related to a specific thematic thread (e.g., struggle for justice, environmental empathy, demonstrating care) by providing the synonyms of the core concept to search that were identified during familiarization with data. Then the researcher could merge related quotes, or even prompt ChatPDF to do so, to form a complete thematic narrative. This process might expedite the analysis and reduce the researcher’s load in terms of the time and labor-intensive processes associated with qualitative research.

Through the terminologies established during this experiment, using the freely available ChatPDF for the purposes of QDA makes rich analysis increasingly possible for researchers in community organizations who do not have investment capability in more formalized research software. Additionally, cost of contracting out for researchers to assist with evidence building or evaluation in community research collaborations can be reduced significantly and create more space for the collaborative interpreting processes in qualitative research. Building on previous insights about the value of AI in higher education institutions (Kuleto et al., 2021; Ma & Siau, 2018) the potential for AI-assisted QDA demonstrated in this experiment should initiate “how might we” conversations between community–university partnerships for upscaling research impacts.

Reflections on ChatPDF for Assisting QDA

In his 1980 book, Electronic Tutors, Arthur C. Clarke wrote “Any teacher that can be replaced by a computer should be” (p.77). If you substitute teacher with researcher, those words ring true for the preliminary conclusions drawn from this experiment. The conversation with both AI tools demonstrates key learnings for future qualitative researchers wishing to engage ChatPDF to assist in QDA with similar raw data formats. The researcher must make methodological decisions about the appropriate tier of organization or organizational hierarchy best suited to the analysis: Utterance, Speaker, or Turn. For example, if preserving the context in which data were gathered is important to the analysis, the researcher would prompt ChatPDF to generate a vignette using Speaker. This obliges the researcher to be well versed in the aims of the study in alignment with the chosen methodological approach. Bullet-pointed utterances from a journey map, for example, would have poor methodological fit with a study employing a narrative inquiry approach.

Similarly, the researcher must also be able to discern the relevance of the AI-generated vignette to the research topic. Prompt engineering skills are inextricably linked to the researcher’s level of familiarization with the raw data. Sufficient familiarity will arm the researcher with a suite of prompts to draw out meaningful data extracts relevant to the overarching research question and will enable the researcher to identify when a further prompt must be inserted. ChatPDF can also detect researcher annotations and codes inserted into chat data (e.g., transcripts) making it easier to identify patterns or trends in the data when prompted to do so using a search function—the chat— (e.g., researcher asks the tool to produce a vignette from Teff’s journey map transcript). However, the researcher must hold prompt engineering expertise, and this can be developed in the first step of the process laid out in Figure 1. Familiarization With Transcripts or Raw Data. Additionally, AI tools, at least in the context of this experiment, had limited capacity to draw on current knowledge unless a dialogue from which it could learn was engaged, making the researcher’s content expertise imperative.

Longo (2020 citing Crowston et al., 2012) noted that when machine learning is given prompts and access to further information to understand the topic being queried by an individual, it showed improved accuracy through larger sample sizes, whereas manual coding approaches worked better with lower sample sizes. The sample in this experiment was not overly large in comparison to quantitative studies. However, the goal of translating data from 16 transcripts to vignettes, to be used as a unit of analysis, would be a time-consuming task comparable to the effort required for coding larger sample sizes with less data or where the transcript text-as-is forms the unit of analysis. The reported experiment demonstrates a reflective account of performing Longo’s (2020) human-in-the-loop approach. Longo defined this approach as involving “human to supervise and to correct predicted codes and use the outcome of this process to create more robust and accurate models (p. 10).

During the experiment, I was able to iteratively improve my understanding of the data being generated as a vignette via the human-in-the-loop approach. ChatPDF quickly regenerated new versions of the vignette when more contextual clues were included in the prompt question. Likewise, other researchers can rapidly come up with ideas about what the data mean when using automated AI tools like ChatPDF to help them find patterns or themes in the data. Moreover, ChatPDF for the nonspecialist to use with QDA is easy compared to tools (e.g., PDFGPT https://pdfgpt.io) that require a deeper understanding of the technical jargon, such as API (i.e., application programming interface) and how to use the associated systems. However, if qualitative research is pursued as a subdomain of HCI research, then the various tools, their applicability for QDA, and associated operational definitions might be the first terrain to navigate.

Authors Tunca et al. (2023) explored the potential of NLP-aided qualitative content analysis using a relatively new QDA and management software innovation that is more formalized, called Leximancer. While the benefits to efficiency and organization when using such software are undeniable, it can come at a high cost to the researcher. It may not be realistic for those researchers working in an underfunded community organization. In terms of ChatPDF’s usability and functionality for the purposes of generating vignettes from the data, the tool generally performed well. At the very least, it provided a foundation for the researcher to get creative, helping to overcome initial writer’s block encountered when the task is completed manually.

Conclusion

Given that ChatPDF features an inbuilt AI language-learning model, qualitative researchers using various units of analysis (e.g., vignettes, thematic summaries etc.) can ask analytical queries based on the aims of their study and the context in which the data were collected. Understanding the different tiers in the organizational hierarchy utilized by ChatPDF accelerated the organization and analysis of data, reducing the researcher load typically associated with the process of QDA. Specifically, using ChatPDF fast-tracked the move from presenting data as a coherent unit of analysis to developing a critically situated analysis. The steps developed in my framework for AI conversion of raw data to unit of analysis are open to emendations and critique. Given the rapid pace at which students (our future researchers) are interacting with AI tools for higher education, engaging in how-might-we conversations on how to advance our AI-related skills for QDA and the prospect of HCI as a new subdomain of qualitative research seems pragmatic.

While not as good as a human research assistant, Chat PDF was a supportive friend who helped move work ahead as long as I fed it the proper meal. This included detailed instructions for the vignettes’ central framing topic, key focusing words previously identified in raw data to ensure each vignette conveyed the nuances participants shared in their journey map, and additional prompts for examples where the software resisted producing the vignette. To continue the food analogy, “you are what you eat,” and in this case, the function of ChatPDF to assist QDA is only as good as the information you feed it. The prompts in ChatPDF to summarize, generate, and re-story qualitative data must accurately reflect design decisions in a study. Cautionary usage is key. This is a reminder that replacing the “I” in intelligence generated by the new tool is not prudent. The researcher holds the knowledge concerning specific contexts and conditions in which data were collected. They have a responsibility to respect stories shared by participants, which involves being diligent about accurate representation. Moreover, AI tools are limited in capacity to draw on current knowledge about specific topics unless a dialogue from which it can learn to more accurately predict is initiated.

Playing with the ChatPDF, I realized that AI tools for researchers are not something to replace me or others like me. Instead, it is a tool to make space to think more critically about the situatedness of research findings, freeing up more time to prepare results for impact. My concluding comment about this curiosity project is that since AI-assisted research tools are here to stay and rapidly evolving, we should not fear them, but use existing research expertise and hone prompt engineering skills to make them work as a capable research assistant!

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Laura Ann Chubb

References

Allen

Bendixsen

Babcock Fenerci

Green

(2018). Assessing disorganized attachment representations: A systematic psychometric review and meta-analysis of the manchester child attachment story task. Attachment and Human Development, 20(6), 553–577. https://doi.org/10.1080/14616734.2018.1429477

Ancheta

J. R.

Gorro

K. D.

M. A. D.

(2020). #Walangpasok on Twitter: Natural language processing as a method for analyzing tweets on class suspensions in the Philippines. 2020 12th International Conference on Knowledge and Smart Technology (KST), Amari Pattaya, Thailand, January 29th - February 01, 2020 (pp. 103–108). IEEE. https://doi.org/10.1109/KST48564.2020.9059411

Biswas

S. S.

(2023). Potential use of ChatGPT in global warming. Annals of Biomedical Engineering, 51(6), 1126–1127. https://doi.org/10.1007/s10439-023-03171-8

Blodgett

A. T.

Schinke

R. J.

Smith

Peltier

Pheasant

(2011). In Indigenous words: Exploring vignettes as a narrative strategy for presenting the research voices of Aboriginal community members. Qualitative Inquiry, 17(6), 522–533. https://doi.org/10.1177/1077800411409885

Borji

(2023). A categorical archive of ChatGPT failures. arXiv preprint. https://doi.org/10.48550/arXiv.2302.03494

Bower

(2021). Um, I was getting bullied at school because I didn’t believe in God”: One family’s experience of autism, school and home education. International Journal of Inclusive Education, 25(8), 914–927. http://dx.doi.org/doi:10.1080/13603116.2019.1588926

Bozkurt

Xiao

Lambert

Pazurek

Crompton

Koseoglu

Farrow

Bond

Nerantzi

Honeychurch

Bali

Dron

Mir

Stewart

Costello

Mason

Stracke

C. M.

Romero-Hall

Koutropoulos

Jandrić

(2023). Speculative futures on ChatGPT and generative artificial intelligence (AI): A collective reflection from the educational landscape. Asian Journal of Distance Education, 18(1). https://www.asianjde.com/ojs/index.php/AsianJDE/article/view/709

Candela

A. G.

(2019). Exploring the function of member checking. The Qualitative Report, 24(3), 619–628. https://doi.org/10.46743/2160-3715/2019.3726, www.proquest.com/scholarly-journals/exploring-function-member-checking/docview/2213787326/se-2

ChatPDF . (n.d.). FAQs. Retrieved from. https://www.chatpdf.com/

10.

Cheligeer

Yang

Nandi

Doktorchik

Quan

Zeng

Singh

(2022). Natural language processing (NLP) aided qualitative method in health research. Journal of Integrated Design and Process Science, 1–18. Ahead of Print. https://doi.org/10.3233/JID-220013

11.

Clarke

A. B.

(1980) (In press). Electronic tutors. Omni Magazine, 2(9), 76–96. https://archive.org/embed/sim_omni_1980-06_2_9

12.

Crowston

Allen

E. E.

Heckman

(2012). Using natural language processing technology for qualitative data analysis. International Journal of Social Research Methodology, 15(6), 523–543. https://doi.org/10.1080/13645579.2011.625764

13.

Fader

Legg

Ross

(2020). Finding a sense of community in youth soccer: A composite vignette of the refugee experience. Journal of Park and Recreation Administration, 38(2). https://dx.doi.org/10.18666/JPRA-2020-9981

14.

Fang

Markuzon

Patel

Rueda

J. D.

(2022). Natural language processing for automated classification of qualitative data from interviews of patients with cancer. Value in Health: The Journal of the International Society for Pharmacoeconomics and Outcomes Research, 25(12), 1995–2002. https://doi.org/10.1016/j.jval.2022.06.004

15.

Gray

Bates

Graham

Han

(2019). Exploring ways to elevate women’s leadership voices to achieve career longevity and gender parity. Centre for Educational Research, Western City University. https://doi.org/10.26183/5c0f54c953ae6

16.

Guetterman

T. C.

Chang

DeJonckheere

Basu

Scruggs

Vydiswaran

V. G. V.

(2018). Augmenting qualitative text analysis with natural language processing: Methodological study. Journal of Medical Internet Research, 20(6), Article e231. https://doi.org/10.2196/jmir.9702

17.

Hosseini

Horbach

S. P.

(2023). Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review. Research Integrity and Peer Review, 8(4), 1–9. https://doi.org/10.21203/rs.3.rs-2587766/v1

18.

Islam

M. N.

(2023). Opportunities and challenges of ChatGPT in academia: A conceptual analysis. Authorea Pre-prints. https://doi.org/10.22541/au.167712329.97543109/v1

19.

Jacobsen

A. J.

(2014). Vignettes of interviews to enhance an ethnographic account. Ethnography and Education, 9(1), 35–50. https://doi.org/10.1080/17457823.2013.828475

20.

Keary

Reupert

Kaukko

Wilkinson

(2022). Education and an ethics of care when working with refugee families during COVID-19. Early Years, 1–14. https://doi.org/10.1080/09575146.2022.2031120

21.

Kirmani

A. R.

(2022). Artificial intelligence-enabled science poetry. ACS Energy Letters, 8(1), 574–576. https://doi.org/10.1021/acsenergylett.2c02758

22.

Kraugusteeliana

Indriana

I. H.

Krisnanik

Muliawati

Irmanda

H. N.

(2023). Utilisation of ChatGPT’s artificial intelligence in improving the quality and productivity of lecturers’ work. Journal Pendidikan dan Konseling, 5(2), 3245–3249. https://doi.org/10.31004/jpdk.v5i2.13650

23.

Kuleto

Ilić

Dumangiu

Ranković

Martins

O. M.

Păun

Mihoreanu

(2021). Exploring opportunities and challenges of artificial intelligence and machine learning in higher education institutions. Sustainability, 13(18), 10424. https://doi.org/10.3390/su131810424

24.

Lambert

(2020). Re-conceptualizing embodied pedagogies in physical education by creating pre-text vignettes to trigger pleasure ‘in’ movement. Physical Education and Sport Pedagogy, 25(2), 154–173. https://doi.org/10.1080/17408989.2019.1700496

25.

Leeson

Resnick

Alexander

Rovers

(2019). Natural language processing (Nlp) in qualitative public health research: A proof of concept study. International Journal of Qualitative Methods, 18(1). https://doi.org/10.1177/1609406919887021

26.

Longo

(2020). Empowering qualitative research methods in education with artificial intelligence. In Costa

Reis

Moreira

(Eds.), Computer supported qualitative research. WCQR 2019. Advances in Intelligent Systems and Computing (vol. 1068, pp. 1–21). Springer International Publishing. https://doi.org/10.1007/978-3-030-31787-4_1

27.

Siau

K. L.

(2018). Artificial intelligence impacts on higher education. MWAS 2019 Proceedings, May 17 – 18, 2018,

College St. Louis

of Business Administration, University of Missouri –, 42. https://aisel.aisnet.org/mwais2018/42

28.

Maslen

Lupton

(2020). Enacting chronic illness with and through digital media: A feminist new materialist approach. Information, Communication and Society, 23(11), 1640–1654. https://doi.org/10.1080/1369118X.2019.1602665

29.

McCuish

E. C.

Hanniball

K. B.

Corrado

(2019). The assessment of psychopathic personality disturbance among adolescent male offenders: Interview strategies and recommendations. International Journal of Forensic Mental Health, 18(1), 35–49. https://doi.org/10.1080/14999013.2018.1531095

30.

Miles

M. B.

Huberman

A. M.

Saldana

(2014). Qualitative data analysis: A methods sourcebook (3rd ed.). Sage Publications.

31.

Mortelmans

(2019). Analyzing qualitative data using NVivo. In Van den Bulck

Puppis

Donders

Van Audenhove

(Eds.), The Palgrave handbook of methods for media policy research (pp. 435–450). Switzerland: Springer International Publishing AG. https://doi.org/10.1007/978-3-030-16065-4_25

32.

Nath

Albaghdadi

M. S.

Jonnalagadda

S. R.

(2016). A natural language processing tool for large-scale data extraction from echocardiography reports. PLoS One, 11(4), Article e0153749. https://doi.org/10.1371/journal.pone.0153749

33.

Paquette

Trudel

Duarte

Cundari

(2019). Participating in a learner-centered coach education program: Composite vignettes of coaches’ and coach educators’ experiences. International Sport Coaching Journal, 6(3), 274–284. https://doi.org/10.1123/iscj.2018-0085

34.

Parks

Peters

(2022). Natural language processing in mixed-methods text analysis: A workflow approach. International Journal of Social Research Methodology, 26(4), 377–389. https://doi.org/10.1080/13645579.2021.2018905

35.

Perrin

A. J.

(2001). The CodeRead system: Using natural language processing to automate coding of qualitative data. Social Science Computer Review, 19(2), 213–220. https://doi.org/10.1177/089443930101900207

36.

Reay

Zafar

Monteiro

Glaser

(2019). Presenting findings from qualitative research: One size does not fit all. In Zilber

T. B.

Amis

J. M.

Mair

(Eds.), The production of managerial knowledge and organizational theory: New approaches to writing, producing and consuming theory (vol. 59, pp. 201–216). Emerald Publishing. https://doi.org/10.1108/S0733-558X20190000059011

37.

Riessman

C. K.

(2008). Narrative methods for the human sciences. Sage.

38.

Sampson

Johannessen

I. A.

(2020). Turning on the tap: The benefits of using ‘real-life’ vignettes in qualitative research interviews. Qualitative Research, 20(1), 56–72. https://doi.org/10.1177/1468794118816618

39.

Singh

(2023). Is ChatGPT biased? A review. https://osf.io/9xkbu/download

40.

Skeen

S. J.

Jones

S. S.

Cruse

C. M.

Horvath

K. J.

(2022). Integrating natural language processing and interpretive thematic analyses to gain human-centered design insights on HIV mobile health: Proof-of-concept analysis. JMIR Human Factors, 9(3), Article e37350. https://doi.org/10.2196/37350

41.

Soares

N. M.

(2019). An inside and an outside: Young people in care who run away. A psychoanalytic approach [Doctoral thesis, Université Paris Cité] https://u-paris.fr/theses/detail-dune-these/?id_these=4117

42.

Sobania

Briesch

Hanna

Petke

(2023). An analysis of the automatic bug fixing performance of ChatGPT. arXiv. https://doi.org/10.48550/arXiv.2301.08653

43.

Spalding

N. J.

Phillips

(2007). Exploring the use of vignettes: From validity to trustworthiness. Qualitative Health Research, 17(7), 954–962. https://doi.org/10.1177/1049732307306187

44.

Stacey

Brière

Robitaille

Fraser

Desroches

Légaré

(2014). A systematic process for creating and appraising clinical vignettes to illustrate interprofessional shared decision making. Journal of Interprofessional Care, 28(5), 453–459. https://doi.org/10.3109/13561820.2014.911157

45.

Tabone

de Winter

(2023). Using ChatGPT for human–computer interaction research: A primer. Delft University of Technology. [Manuscript submitted for publication]. https://www.researchgate.net/publication/367284084_Using_ChatGPT_for_Human-Computer_Interaction_Research_A_Primer

46.

Tate

Doroudi

Ritchie

(2023). Educational research and AI‐generated writing: Confronting the coming tsunami . https://edarxiv.org/4mec3/download?format=pdf

47.

Tunca

Sezen

Balcioglu

Y. S.

(2023). Content and sentiment analysis of the New York Times Coronavirus (2019-nCOV) articles with natural language processing (NLP) and Leximancer. Electronics, 12(9), 1964. https://doi.org/10.3390/electronics12091964

48.

Vartak

(2020). Migration and education in regions with a culture of migration: Observations from Kunkeri village, Konkan, Maharashtra. Journal of Migration Affairs, 2(2), 83–99. https://doi.org/10.36931/jma.2020.2.2.83-99

49.

Wen

Wang

(2023). The future of ChatGPT in academic research and publishing: A commentary for clinical and translational medicine. Clinical and Translational Medicine, 13(3), Article e1207. https://doi.org/10.1002/ctm2.1207

50.

White

Hays

Sandborn

Olea

Gilbert

Elnashar

Spencer-Smith

Schmidt

D. C.

(2023). A prompt pattern catalog to enhance prompt engineering with ChatGPT. arXiv preprint. https://doi.org/10.48550/arXiv.2302.11382

51.

Woo

(2019). Trial by artificial intelligence. Nature, 573(7775), S100–S102. https://doi.org/10.1038/d41586-019-02871-3

52.

Wulf

Lindner

Strahringer

Westner

(2021). IaaS, PaaS, or SaaS? The why of cloud computing delivery model selection: Vignettes on the post-adoption of cloud computing. In Proceedings of the 54th Hawaii International Conference on System Sciences, 2021, Grand Wailea, Maui, Hawaii, Tuesday, January 5, 2021 to Friday, January 8, 2021 (pp. 6285–6294). https://hdl.handle.net/10125/71378

53.

Yew

A. N.

Schraagen

Otte

W. M.

van Diessen

(2023). Transforming epilepsy research: A systematic review on natural language processing applications. Epilepsia, 64(2), 292–305. https://doi.org/10.1111/epi.17474

54.

Zhong

Ding

Liu

Tao

(2023). Can ChatGPT understand too? A comparative study on ChatGPT and fine-tuned BERT. arXiv. https://doi.org/10.48550/arXiv.2302.10198

55.

Zhuo

T. Y.

Huang

Chen

Xing

(2023). Exploring ai ethics of ChatGPT: A diagnostic analysis. arXiv preprint. https://doi.org/10.48550/arXiv.2301.12867

56.

Zuccon

Koopman

(2023). Dr ChatGPT, tell me what I want to hear: How prompt knowledge impacts health answer correctness. arXiv preprint. https://doi.org/10.48550/arXiv.2302.13793