Abstract
This study explores the potential of the AI chatbot ChatGPT to supplement human-centered tasks such as qualitative research analysis. The study compares the emergent themes in human and AI-generated qualitative analyses of interviews with guaranteed income pilot recipients. The results reveal that there are similarities and differences between human and AI-generated analyses, with the human coders recognizing some themes that ChatGPT did not and vice versa. The study concludes that AI like ChatGPT provides a powerful tool to supplement complex human-centered tasks, and predicts that such tools will become an additional tool to facilitate research tasks. Future research could explore feeding raw interview transcripts into ChatGPT and incorporating AI-generated themes into triangulation discussions to help identify oversights, alternative frames, and personal biases.
Keywords
Released in November of 2022, ChatGPT is an artificial intelligence (AI) chatbot that can understand and generate human-like language. The responses are generated using data collected using Reinforcement Learning with Human Feedback (RLHF), which utilizes human feedback to distinguish which response would be most preferable according to the conversation. Research is ongoing to improve the chatbot (Open AI, n.d). There is a wide range of possible uses for such technology, from personal to professional. The system is still in its preliminary stages but constantly improves with user feedback. Within just a few months, ChatGPT reached over 100 million users (Chow, 2023).
Much has been written about the potential for AI technology to replace large sectors of the workforce. It has been widely assumed that this disruption would occur in low-skilled jobs such as manufacturing, retail, and service. Some have cited the growth of technology in these sectors as an argument for a universal basic income or guaranteed income (Bruun & Duka, 2018; Chelliah, 2017; Smith & Anderson, 2014). However, ChatGPT has cast these discussions in a new light. Will it replace even highly skilled jobs that require complex tasks? In this project, we seek to explore the potential of ChatGPT in a traditionally human-centered task, qualitative research analysis. We do this by comparing ChatGPT’s qualitative data analysis outputs with human-created coding. We first coded qualitative interviews with 71 guaranteed income pilot recipients using Colaizzi’s descriptive phenomenological method (Morrow et al., 2015). We then ask ChatGPT to code the qualitative data and compare emergent themes in the human and AI processes. In the first few months of ChatGPT’s existence, a great deal has been written in the popular press about its potential to either support or disrupt a wide array of human-centered tasks, including qualitative research. To our knowledge, however, this study is one of the first academic attempts to directly explore its potential.
Literature Review
We begin by asking ChatGPT, “In 500 words, what is Chat GPT?” In a few seconds, the tool provided the following response. Chat GPT is a large language model developed by Open AI that is capable of generating human-like responses to text-based prompts. GPT stands for “Generative Pre-trained Transformer,” which refers to the model's architecture and training method. The “Transformer” architecture is a type of neural network that was first introduced in a research paper by Google in 2017. Transformers have since become popular in natural language processing tasks due to their ability to handle long-range dependencies and capture contextual information. To train the model, Open AI used a massive amount of text data from the internet, which includes news articles, books, and websites. The data was pre-processed to remove any personally identifiable information or sensitive content before being used to train the model. During training, the model learns to predict the next word in a sentence based on the previous words. By doing so repeatedly, the model can generate entire sentences that are coherent and contextually appropriate. The training process also involves adjusting the model's parameters to minimize the difference between its predicted outputs and the actual data. Chat GPT is particularly well-suited for tasks such as language translation, summarization, and question answering. It can also be used for text generation tasks, such as story writing or chatbot conversations. One of the most impressive features of Chat GPT is its ability to generate responses that are similar to those of a human. This is because the model was trained on a vast corpus of text written by humans, allowing it to learn the nuances of natural language. When given a prompt, the model uses this knowledge to generate a response that is both grammatically correct and contextually appropriate. Despite its impressive capabilities, Chat GPT is not without limitations. The model is not capable of understanding the world in the way that humans do and has limited knowledge of specific domains. It is also prone to generating responses that are biased or contain errors. In conclusion, Chat GPT is a powerful tool for natural language processing tasks that can generate human-like responses to text-based prompts. While it has some limitations, the model's ability to understand context and generate coherent responses has many practical applications in fields such as customer service, content creation, and language translation.
ChatGPT is the fastest-growing app in history. In just three months, the app reached 100 million users. By comparison, the popular app TikTok took nine months to reach the same number of users (Gordon, 2023). Open AI, a company co-founded by popular tech celebrities such as Elon Musk and Sam Altman, shows promise with hefty investments from companies like Microsoft (Chowdhury, 2023). With an expected $1 Billion in revenue in 2024, the company and its app are only expected to expand. The company has begun monetizing parts of the app for revenue (Dastin et al., 2022). For $20 a month, users can access the premium version of the app titled ChatGPT Plus. ChatGPT Plus offers users faster response times and early access to new features (Gordon, 2023).
The creation of ChatGPT has challenged tech giants like Google, as Microsoft is likely to synthesize ChatGPT features into its search app, Bing. As companies race to be the most advanced in AI technology, more advancements are expected. Within three months, ChatGPT has caused disruptions in areas such as academia, media and journalism, mental health, and research as the world learns how this technology can be best utilized. The rapid recent growth of AI technology poses important questions about future implications and problems (Gordon, 2023).
However, activists have been sounding the alarm about technology’s disruptive role in the workforce for several years (Bruun & Duka, 2018; Chelliah, 2017; Smith & Anderson, 2014). We asked ChatGPT, “How does artificial intelligence like ChatGPT threaten the workforce?” Artificial intelligence (AI) technologies like ChatGPT can threaten the workforce in a number of ways. One of the primary concerns is that AI can automate tasks that were previously done by humans, leading to job displacement and changes in the labor market. ChatGPT is an example of a technology that could be used to automate certain customer service or support tasks. For example, it could be used to respond to frequently asked questions, provide basic troubleshooting support, or handle other routine tasks that do not require a human touch. This could potentially lead to job loss for workers who previously performed these tasks. In addition to direct job displacement, AI can also lead to changes in the labor market as a whole. As AI becomes more widespread, workers may need to develop new skills in order to remain competitive in the job market. For example, workers who are displaced from routine jobs may need to transition into higher-skilled positions that require greater creativity, problem-solving, or interpersonal skills. Another potential threat of AI is that it could exacerbate existing inequalities in the workforce. For example, if certain jobs are more easily automated than others, workers in those jobs may be at greater risk of job displacement. Similarly, workers who lack access to education or training programs that teach the skills needed to work with AI may be at a disadvantage in the job market. However, it is important to note that while AI may threaten some jobs, it may also create new opportunities and make certain tasks more efficient. For example, ChatGPT could free up customer service workers to focus on more complex issues or provide a more personalized experience for customers. Overall, the impact of AI on the workforce is complex and multifaceted, and will likely depend on a variety of factors such as the specific technology being used, the industry in question, and the availability of education and training programs to help workers transition into new roles. It will be important for policymakers, businesses, and workers themselves to work together to address these challenges and ensure that the benefits of AI are shared equitably.
Finally, we asked Chat GPT, “What social policies could prevent this disruption?” The potential disruption to the workforce caused by artificial intelligence (AI) technologies like ChatGPT highlights the need for policies that can help ensure that workers are not left behind in the transition to a more automated economy. Here are some social policies that could help prevent this disruption: 1. Education and Training Programs: One of the most important policies for addressing the potential impact of AI on the workforce is to invest in education and training programs that can help workers acquire the skills needed to work with these technologies. Such programs could include technical training for workers in industries that are being automated, as well as general education programs to help workers build more transferable skills. 2. Job Transition Support: Another important policy is to provide support for workers who are displaced by automation. This could include financial assistance, such as unemployment benefits or job training subsidies, as well as counseling and job placement services to help workers transition into new positions. 3. Labor Protections: To ensure that workers are not exploited by AI-driven companies, it is important to establish labor protections that address issues such as fair wages, working conditions, and benefits. This could include laws that require companies to provide a living wage and benefits, as well as protections against discrimination and harassment. 4. Universal Basic Income: Some policymakers have proposed the idea of a universal basic income (UBI), which would provide a guaranteed income to all citizens regardless of their employment status. UBI could help ensure that workers who are displaced by automation are not left without a safety net, and could provide a cushion for workers who are transitioning into new careers. 5. Inclusive Economic Growth: Finally, policymakers should work to promote inclusive economic growth that benefits all members of society. This could include investments in infrastructure and community development, as well as policies that promote economic equality and reduce disparities in income and wealth. In conclusion, the potential impact of AI on the workforce highlights the need for social policies that can help workers adapt to the changing economy. Policies that focus on education and training, job transition support, labor protections, UBI, and inclusive economic growth can all play a role in mitigating the potential disruption caused by AI technologies like ChatGPT.
Echoing ChatGPT’s fourth recommendation, several writers have argued that the growth of AI necessitates a universal basic income or guaranteed income (Bruun & Duka, 2018; Chelliah, 2017; Smith & Anderson, 2014). They argue that as advancements in AI and other technologies affect how human labor is utilized in the workforce, policies must adjust to account for possible unemployment or loss of wages. For example, 2020 presidential candidate and technology entrepreneur, Andrew Yang proposed a “Freedom Dividend” to combat job loss as technology advances (Miller, 2021).
It is fitting, then, that we employ qualitative data from a guaranteed income pilot to test ChatGPT’s capability to provide qualitative data analysis, especially given recent studies highlighting the potential use of AI in research. Tabone and de Winter (2023) demonstrated that ChatGPT can effectively analyze text data from questionnaires, interviews, and think-aloud studies, producing results comparable to traditional manual analyses. Similarly, Mesec (2023) highlighted how AI could be used in the analysis of social work text, finding it capable of abstract understanding and creative paraphrasing. With these recent advancements in mind, the primary objective of our research is to investigate the capacity of ChatGPT, an emerging AI technology, to generate qualitative themes from interview transcripts. We aim to compare these AI-generated themes with human-generated analysis to explore the level of agreement and identify potential differences. Through this investigation, we seek to gain insights into the utility and limitations of AI technology in traditionally human-centered tasks like qualitative research analysis.
Methods
Study Design
The aim of this study is not primarily to analyze participant experiences with GI but rather to compare human-generated qualitative analyses with those produced by the AI language model, ChatGPT. Given this focus, the original interview data is secondary to the methodology of comparison, which is the true phenomenon of interest here. In order to do this, we first employed Colaizzi’s descriptive phenomenological method (Morrow et al., 2015), to generate human-created themes from interviews with GI pilot participants. We then utilized ChatGPT to generate themes from the same data. Finally, we compared the themes identified by both methods for similarities and discrepancies.
The study described below is nested within a larger guaranteed income (GI) program evaluation. The program, called In Her Hands, was born from community listening sessions in the Old Fourth Ward of Atlanta, Dr. Martin Luther King Jr’s home neighborhood. The project serves three Georgia neighborhoods with high concentrations of Black residents, including urban, suburban, and rural areas. Launched in the summer of 2022, the program enrolled 654 low-income women through a lottery system. The program evaluation engages a Community-Based Participatory Research (CBPR) model to prioritize community expertise and decision-making. An Institutional Review Board approved the study in May, 2022. The program staff transferred participant contact information to the research team via a secure server. Contact information was stored in the University’s secure Google Drive and kept separately from survey and interview data. The interviews, which were conducted in late 2022, covered a wide range of topics, including participants' financial situation, their experiences with the GI program, and their personal life. However, it’s important to note that while we discuss these topics, they are not the primary focus of this study. Rather, our interest lies in comparing the human and AI-generated themes from this data set.
Our qualitative analysis for the larger study of guaranteed income includes a longitudinal phenomenological approach used to explore the subjective and lived experiences of individuals. This qualitative research method aims to understand the essence and meaning of those experiences as perceived and described by the participants (Moustakas, 1994). Phenomenological research involves in-depth interviews or other qualitative data collection methods to gather detailed accounts of individuals' experiences and perspectives (van Manen, 1990). The goal is to uncover common themes and patterns from individual experiences and provide a holistic understanding of the phenomena studied (Creswell & Poth, 2017).
Participants and Sampling
Interview participants included recipients of the guaranteed income project described above. The program utilized a lottery to enroll 654 low-income Black women in three sites across Georgia who will receive an average of $850 monthly for 24 months. To qualify for the project, applicants were required to meet three criteria: (1) identify as a woman (inclusive of non-binary and transgender residents) aged 18 years or older; (2) have income at or below 200% of the federal poverty level; and (3) reside in one of the three historically Black communities (urban, rural, and suburban). Demographically, baseline data indicate that the program successfully reaches its target population. Participants across the three sites are 97.2% Black and have a mean age of 37. Nearly three-quarters (71.8%) have children in the home, and 94.8% do not report having a partner in the home. Across all three sites, approximately 28% worked full-time, 17.3% worked part-time, and 29.7% were looking for work at the program baseline. Participants reported a monthly mean income of $1213.72 before receiving the monthly GI payments.
Data Collection
Data collection involves conducting biannual quantitative surveys with all recipients who consent to participate and conducting 60 to 90 qualitative interviews per biannual wave as this number is within the range of best practices for qualitative research, balancing the manageability of data analysis with data validity (Creswell & Poth, 2017). From the pool of participants who consented, we randomly selected a qualitative subset of 30 participants per site. The qualitative interviews for Wave One took place in the fall of 2022, during the first two to three months of program receipt. Three Research Assistants, who are social work doctoral students at a Historically Black College and University (HBCU) in Georgia, conducted the interviews.
The Research Assistants received qualitative interview training through the Research Talks Qualitative Summer Intensive program associated with the University of North Carolina Odum Institute for Research in Social Sciences. Each Research Assistant was assigned a caseload of 10 randomized participants from each of the three sites, resulting in a total caseload of 30 participants per Research Assistant. To accommodate participant preferences and ensure their comfort, we provided the option of face-to-face interviews at a location of their choice or conducting interviews over the phone. Given the circumstances of the pandemic and their own personal lives, the majority of participants chose to have phone interviews. To maximize participation, we developed a contact protocol consisting of three text messages, two emails, and two phone calls. In Wave One, the research team conducted 71 qualitative interviews. The interviews were conducted between August and November of 2022. On average, the interviews lasted approximately 17 minutes, with durations ranging from 8 to 44 minutes.
Data Analysis
Our qualitative analysis involved two phases: (1) a traditional human center protocol and (2) a ChatGPT-generated analysis. In the first phase, qualitative analysis was conducted via a phenomenological research procedure with a research team of six members (the Principal Investigator, the three doctoral students conducting interviews, and two masters level graduate assistants). We specifically followed Colaizzi’s descriptive phenomenological method, in which we independently read and re-read interview transcripts. Through this meticulous review, we identified 'significant statements' or quotes that provide an understanding of how the participants experienced the phenomenon (Morrow et al., 2015; Moustakas, 1994). These significant statements are pivotal in grasping the essence of the participants' lived experiences. We then formulated meanings from these significant statements to construct themes, enabling a comprehensive understanding of the phenomenon under study.
Interviews were first transcribed by Rev.com, then anonymized (removing names and other identifying information) by graduate students. All transcripts were then uploaded to Taguette.com, a free online qualitative coding software where multiple researchers can view and edit a shared data set. Then, the primary interviewer and a randomized secondary reviewer were assigned to highlight significant statements in each transcript following Colaizzi’s method described above, resulting in 1125 unique statements. Each member of the research team then reviewed all significant statements and made individual notes of emergent themes. The team then met to triangulate these themes and commit to a common codebook. We used a free online mind mapping tool (Mural.com) to share individually identified themes and cluster similar codes. Finally, the primary and secondary reviewers returned to the transcripts and recoded significant statements using the common codebook.
In the second phase of the research process, we fed all 1125 identified “significant statements” into ChatGPT. There are character limits for inputting to ChatGPT, so we “fed” the engine 50 significant statements at a time in one “chat” and then asked, “Act as a phenomenological qualitative researcher. All 1125 significant statements above come from interviews with 71 guaranteed income pilot participants. Please identify common themes from the statements.” ChatGPT identified ten themes (described below) in less than 30 seconds.
Finally, to assess the similarities and differences between human-centered coding and AI analysis, a comparison was conducted by the Principal Investigator (PI). The PI clustered similar themes together based on her own perspective and expertise in qualitative analysis. Following the initial analysis, the entire research team engaged in a process of triangulation. This involved reviewing and discussing the PI’s comparisons to ensure a comprehensive and rigorous analysis. Through this iterative process of review and discussion, the research team critically examined the similarities and differences between the two analyses. Consensus was reached through collaborative deliberation.
Verification Procedures
Triangulating the qualitative analysis across six reviewers limited individual bias and strengthened the trustworthiness of the findings. To further ensure data quality, the Principal Investigator engaged in “member checking.” Member checking, which is described by Lincoln and Guba (1985, p. 314) as “the most critical technique for establishing credibility,” is the process of allowing participants the opportunity to give feedback on summative themes. A draft of preliminary findings was emailed to all participants, inviting their feedback. We received responses from 17 participants. Our team assessed the feedback provided by the participants and found it to be primarily positive. We then used ChatGPT to generate a summary of the feedback, which was later verified for accuracy by the Principal Investigator to ensure that the interpretation was based on human-driven analysis. The feedback is mostly positive, with participants expressing gratitude for being part of the program and for the financial help they have received. One participant wishes the payment could be deposited at the beginning of the month. The summary provided by the program is seen as accurate and representative of their experiences. Many participants mention that the program has improved their financial stability and helped them pay bills and rent. They also express appreciation for the program and the team behind it.
This iterative process of member checking, human researcher analysis, and verification through AI-generated summaries allowed us to incorporate participant perspectives and further validate findings. It also provided an opportunity to compare and contrast the interpretations of the research team with those generated by ChatGPT, contributing to our aim of comparing the two approaches.
Bracketing
Phenomenological research often employs the technique of bracketing, also known as epoché, to temporarily set aside preconceptions and assumptions about a phenomenon. This allows the researcher to examine the phenomenon in a less biased manner, focusing on its raw, subjective experience and identifying essential features or structures (van Manen, 1990). Bracketing is an essential step in phenomenological research (Creswell, 2017), as it helps to prevent researcher biases and assumptions from influencing the study results (Moustakas, 1994). However, a constructivist might argue that it is impossible to completely set aside our own biases, as they are the lenses through which we see the world. Because the research described here includes six human researchers and ChatGPT, we asked each researcher to briefly describe their positionality and how this might influence their data evaluation.
Researcher One
As a white female academic who grew up in poverty but now experiences financial privilege, I often feel the significant responsibility of presenting my research, especially for the current study of Black women, in a way that does not reinforce existing stereotypes about persons in poverty. Further, having spent many years evaluating welfare policy, my current bias is that existing safety net policies, while offering short-term relief for families experiencing financial hardship, are frequently designed in a way that dehumanizes recipients and makes it difficult to gain long-term economic stability.
Researcher Two
I identify as an African American Female who has experienced the challenges of poverty and social welfare assistance. Through entrepreneurship and academic education, I achieved homeownership. While my knowledge in qualitative research is newer, in a previous secular role, I interviewed and highlighted the inspirational stories of others who overcame adversity. I have also spent many years advising Black women on empowering themselves through their businesses. My bias in evaluating guaranteed income is that I recognize the value of providing the necessary support and educational resources to help individuals reach their full potential. Lastly, I feel it is important to approach research on Black women in poverty with sensitivity and a commitment to challenging existing stereotypes.
Research Three
I identify as an African American male who grew up in a suburban middle-class family; therefore, I’m well aware of my male, able-bodied and economic privileges. Although I’m aware of my privilege, I am also aware of how my identity has made me a victim of discrimination and racism. My social work background has equipped me with the tools to provide services and empathize with underserved, marginalized populations. Furthermore, because my area of focus centers around macro-level social work, I have quite a bit of education on social policy analysis and qualitative research. Despite my social work background, I am aware that I still hold some unethical bias that shows up in my work as a savior complex when it comes to underserved people of color, which can impact the validity of the research. Therefore, I always keep my biases in check before interacting with participants.
Researcher Four
As an African American woman, wife, and mother who experienced poverty and homelessness, I was no stranger to relying on government assistance programs to meet my needs. However, through self-perseverance, education, and drive to better myself and my family, we managed to grow out of our poverty-stricken situation and no longer needed to rely on government assistance to meet our basic needs. My journey elevating from homelessness to homeownership may create a bias for me in that I understand the value and need for income assistance-driven programs but only as a means to help individuals to build economic independence, not for permanency. Though my background involves working with underserved populations, it is imperative that I approach this research holistically, from all perspectives so as not to influence research validity.
Researcher Five
As a white female graduate student that grew up in poverty, the use of qualitative research is a relatively new concept for me. Due to my limited experience that has solely been on GI pilots in the US, my perceptions may have been affected or potentially biased by my limited experience. The institutions of both my undergraduate and graduate education are predominantly made up of white students and faculty. Although I have a strong interest in social policy research and advocacy, my foundation for this knowledge has been in educational institutions that lack diversity. There may be bias in my evaluation of research methods and policy that affects other races.
Researcher Six
As a white female graduate student that grew up in lower socioeconomic, rural North Carolina, I can understand where many of our UBI/GI participants are coming from. With this being said, I have to check the biases I may have toward them and my desire for them to one day receive a UBI/GI when conducting data evaluation. Furthermore, my hometown, college, and graduate school are predominantly white and, therefore, lack diversity. My previous institutions taught me research methods from a monocultural perspective, which could potentially lead to bias when I conduct research that affects other races.
We then asked ChatGPT to examine its biases with the prompt, “Act as a phenomenological researcher and engage in the process of ‘bracketing.’ What are your biases that might influence your analysis?” As an AI language model, I don't have personal biases in the way that humans do. However, the nature of machine learning and the data used to train me can introduce certain biases, such as underrepresentation of certain demographics in the data, which may impact my analysis. Additionally, my algorithms and programming may have inherent limitations that could affect my ability to accurately analyze and interpret human experiences and perspectives. It's important for any researcher, human or machine, to acknowledge and address potential biases in their work to ensure the validity and reliability of their findings.
Findings
Human Created Themes
Through the researcher triangulation process described above, the team identified 11 emergent themes: (1) family as a motivator, (2) faith as a motivator, (3) navigating financial adversity, (4) inflation, (5) health issues, (6) insufficient community resources, (7) public assistance barriers, (8) gratitude for the GI program, (9) a desire for privacy regarding the GI program, (10) asset development goals, and (11) personal development goals. Each of these themes will be described in further detail below.
Family as a Motivator
Many participants shared the profound influence of their family members, including their children, parents, and siblings, on their financial decisions and personal determination. The participants expressed that their family members served as fundamental motivators, providing them with the drive to navigate their financial challenges and strive for self-determination. One participant highlighted the significant role their children played in motivating them, stating, “My kids, because I probably wouldn’t know which way to go if it wasn’t for them. They keep me going because I know I can’t quit.” Another participant emphasized the importance of their grandkids as a source of strength, saying, “My grandkids…They stay with me. That’s my life, my grandkids. My daughters are grown, so what I do, I do mostly for them.” These quotes reveal the profound emotional connection participants have with their families, who provide them with a sense of purpose and drive to overcome challenges.
Faith as a Motivator
Participants also shared how their faith plays a significant role in motivating them and providing them with hope amidst life’s daily struggles and hardships. Their faith is a source of strength and optimism, enabling them to navigate personal and financial challenges with resilience. One participant expressed gratitude for the support they have received through their faith, stating, “I thank God, in between time, regardless if I have money, he has sent people to bring them clothes, bring us food, bring us anything that we need or whatnot. So it's like an up and down battle or whatnot. You have to be an optimist.”
Another participant emphasized the role of their relationship with God, saying, “My relationship with God gives me my strength and hope to continue on, to be hopeful, and also to want to see the love of God coming back at me. Have to give it to receive it.” These example quotes depict the transformative effect of faith on their lives. Their faith serves as a guiding force that empowers them to face challenges with optimism, gratitude, and a sense of purpose. Previous research has also described the role of the Black church as a vital cultural institution that provides hope, support, and spiritual guidance to its members (Brewer & Williams, 2019).
Navigating Financial Adversity
Participants in this study openly acknowledged the impact of limited financial resources and the struggle of living paycheck to paycheck. They highlighted the systemic challenges they face in a society that lacks critical financial support, such as living wages and affordable childcare. One participant attributed their financial difficulties to insufficient income and lack of child support, stating, “So, I guess my financial problem is just, I just don't get paid enough money. I worked a lot and I work a lot, but having kids with no support... It's really child support, that's really it, because I make enough to support me and my kids, but the extra stuff is what I don't have enough for, or living from check-to-check, check-to-check.”
Participants further shared their strategies for managing these financial challenges, which are born out of necessity in a society with limited support systems. Firstly, they prioritize their needs over wants and make careful financial decisions based on their available resources. One participant described their mindset of assessing needs and affordability, stating, “I look at what I need, I look at what I can afford, and that’s how my mindset is. I don’t want for something that I don’t have. If I haven’t, it’s like okay, now how can I afford it? Then I do without it.”
Pooling resources with family and friends emerged as another strategy mentioned by several participants. However, it is important to note that this collaborative effort is often driven by both necessity and choice. One participant shared their experience of somewhat reluctantly taking care of other people’s children while they went to work, reflecting frustration with the lack of accessible and affordable childcare options. The participant stated, “And it's like they [friends and family] have to miss work because of daycare or nobody's there to watch their kids. Now they're throwing kids on me because they were like, 'Well, you can't work anyway because you really don't have nobody to watch your four youngest, so watch you might as well watch mines too while I go to work.'”
These strategies reflect the resourcefulness and resilience of participants in the face of financial adversity. However, it is crucial to recognize that a lack of critical financial support systems in society necessitates these adaptations.
Inflation
The participants in our study also frequently expressed their concerns about the impact of inflation on their financial well-being. They highlighted the rising prices of essential goods and services, which added additional strain to their already challenging financial situations. As reported by the Bureau of Labor Statistics (2022), national inflation rates rose by 9.1% in 2022, exacerbating the financial difficulties experienced by many individuals and families. Participants emphasized how the increased costs of basic necessities, such as food, affected their ability to make ends meet. They expressed frustration with the escalating prices, which made it increasingly difficult to afford essential items. One participant shared their struggle, stating, “And then everything has gone up. The price of food, the price of everything. So even if a person is trying to come out of [poverty], it's like you can't not rely on the system because it is like, you can never make enough.”
The rising cost of food was an especially significant concern for participants who relied on food assistance programs like SNAP (Supplemental Nutrition Assistance Program). They described how the increased prices impacted their ability to provide nutritious meals for themselves and their families. One participant highlighted the soaring price of eggs, stating, “I've been getting food stamps for the last six years. I feel like it helped better six years ago. But the way food is so expensive, a carton of eggs is $8, and I got three of them to feed, plus me, plus my boyfriend. So it's like, shoot, the eggs are gone in a week.”
Other participants also expressed their belief that food assistance programs should be adjusted to account for the rising costs of food. They felt that the current benefit levels were insufficient to meet their increased needs due to inflation. One participant suggested, “I would say with the SNAP, because food is getting high. Milk is getting high. The formula and stuff like that, they should give them more than what they're giving us because since the food went up, I think SNAP should have gone up.”
Health Barriers
In addition to facing financial hardships, participants in our study expressed how these barriers affected their physical and mental well-being. Insufficient access to medical care further compounded their health concerns, making it challenging for them to achieve stability while dealing with mental and physical health issues. Many mentioned the absence of medical insurance or coverage, particularly among Black citizens. As a result, some individuals avoided necessary medical care. One participant expressed this dilemma, stating, "It’s hard out here for Black women and men that don’t have Medicare or anything. Some of them won’t even go to the doctor because they don’t have the money to go."
They also recognized that their mental well-being played a crucial role in their ability to overcome challenges. One participant candidly shared, “My greatest barrier is my mental stability. It is not the outside sources, because there are sources everywhere. It's not my family, it's not anything, it's me. I stop myself from doing a lot of things out of fear or uncertainty or just out of nervousness or anxiousness. So I'm working on that.”
However, the financial burden of seeking professional help posed an added challenge for some participants dealing with mental illness. They described the impact of their mental health conditions and the struggle to prioritize their own well-being due to limited financial resources. One participant shared their experience with mental illness and the difficulty of accessing affordable healthcare, saying, “I literally have a mental illness problem where I can easily tick, tick, boom, and voices start coming because that's what psychosis is, when you lose reality of society or reality of the world, anything. And then, you know, it's…way too late. I tell my doctor or tell my OB without paying them, which I can't pay because I have kids that need stuff. So I have to put my kids over me.”
Kuskoff et al. (2022) similarly find that low-income mothers often prioritize the welfare of their children over their own. These experiences illustrate the profound impact of financial barriers on participants' physical and mental health. They also shed light on the experiences of Black women who face intersectional challenges, including institutional and daily oppression and racism. Arline Geronimus' (2023) Weathering Hypothesis suggests that navigating systemic racism and the cumulative effects of chronic stress contribute to the deterioration of Black individuals' health over time.
Insufficient Community Resources
In interviews, participants also highlighted the significant challenges they face due to the lack of appropriate resources and infrastructure in their communities. They expressed concerns about the safety of their neighborhoods, the scarcity of well-paying job opportunities, and the need for more activities and community organizing efforts to improve their overall well-being. Participants emphasized the need for comprehensive improvements, including enhanced transportation systems, accessible healthcare facilities, increased job opportunities, affordable housing options, and targeted programs to uplift and support the community.
Some participants expressed frustration with the state of their community, noting the lack of collective action and limited initiatives to address the community’s needs. One participant shared their disappointment, stating, “I don’t really deal with the community because they don’t do nothing. They don’t want to do nothing in this community. I can’t speak up for the community because the community doesn’t speak up for itself.” Others emphasized the necessity of strong leadership and comprehensive support systems within the community. They called for individuals who are committed to education, both for children and adults, and who can provide practical resources and guidance. One participant stressed this need, stating, “They need some real leaders. You need some grown women and some grown men that are really out here ready to educate the children and the adults and have real resources for them. These people need hands-on people with them to run them step by step. They need somebody to teach them ethics.”
Safety concerns were also prevalent among participants, as they highlighted incidents of violence and the impact on the community’s well-being. One participant shared a distressing incident, stating, “Not too long ago, the kids were playing at the playground and some guys walk up and started shooting and killed a man on the playground while the kids were playing, a group of kids out there.” In these quotes, participants clearly articulated the desire for safer neighborhoods, increased opportunities, and targeted programs to uplift their communities but a lack of community resources and infrastructure to achieve them.
Public Assistance Barriers
As we alluded to in earlier themes, participants in our study consistently emphasized the challenges they face with public assistance programs and the impact these barriers have on their journey toward financial stability. They described the rules and procedures governing public assistance as obstacles that prevent them from adequately caring for their families and attaining long-term stability. Many participants expressed frustration with the insufficiency of public assistance in meeting their family’s needs. They emphasized the high cost of living and the increasing expenses they encounter daily. One participant advocated for changes in the EBT food stamp program, stating, “I would absolutely change the EBT Food Stamp program where they're not giving it to enough people that are actually working and trying to make it because the cost of living is high. Everything is increasing. I would make it where everybody's eligible to receive something.”
Additionally, participants highlighted the difficulties they face in accessing childcare assistance, which often has stringent qualification criteria. They voiced the need for more accessible and supportive childcare options that would allow them to pursue employment opportunities. One participant questioned the existing system, stating, “Their [child care assistance] qualification is very high... You need too much. How can we get a job when we have kids at home? Why not give us daycare first, and then we can find a job? Because we cannot find a job, start working and get check stubs and stuff like that, and we can't work in with kids.”
Gratitude
Counter to their experiences with public assistance programs, participants in our study expressed deep gratitude for the In Her Hands program and shared how it has positively transformed their lives. Initially, some participants were skeptical, considering the program too good to be true. However, as they began receiving payments, they experienced a significant improvement in their financial well-being and overall stability. They emphasized the newfound independence and the ability to rely less on others for support. One participant described their experience, stating, “I’m excited about the program. I used to have to borrow from Peter to pay Paul… But it’s not like that now. I’m able to be independent and not have to rely on other people to help me."
Participants especially appreciated the program’s support during challenging personal times. One participant expressed gratitude, saying, “Honestly, the program has been one of the best things that’s happened to us because it’s just a little bit of support in these very hard times.” For participants facing homelessness and housing insecurity, the program played a crucial role in providing immediate relief and stability. One participant shared their experience, highlighting the program’s impact on their housing situation, stating, “Basically, we were homeless… and we’re still trying to find a place. The program came in handy last month… Although it was the first payment, that’s what got us in a hotel and kept us afloat.” The overwhelming sense of gratitude extended beyond individual participants, with many expressing disbelief that such assistance was being provided. One participant captured this sentiment, stating, “I couldn’t believe that somebody was actually helping people like this."
We acknowledge the potential influence of our role as researchers associated with the program and the unspoken pressure participants may feel to speak positively about the program. Further, our team grappled with whether to present this theme as it could inadvertently portray the program as a savior when in reality, it seeks to supplement institutional systems of oppression that should rightfully be dismantled. However, we ultimately decided that it is crucial to acknowledge the participants' expressions of gratitude and their genuine experiences of improvement and hope through the program’s support.
Privacy
In our conversations with program recipients, it became evident that maintaining privacy about their participation in the program is of utmost importance to them. Participants expressed a strong desire to keep their involvement confidential and refrain from disclosing it to others. They emphasized the need to keep their personal matters private and cited concerns about potential requests for money from acquaintances or the fear of judgment. For these reasons, many participants chose to share their participation only with close family or friends whom they trust. One participant shared their perspective, stating, “I don’t talk about it because I feel like that’s my personal business. And I don’t want everybody to know my business of what I get and what I don’t have.” Another participant expressed their preference for privacy, explaining, “I'm very private, so they don't know. I feel like if they would know, they would have their hands out, and it wouldn't really help me... So I didn't tell anyone about this. Nobody knows that I'm receiving this, but me, myself, and I.”
Asset Development Goals
In our qualitative interviews, participants also shared their aspirations for asset development and financial stability. Many participants discussed their previous lack of savings to handle unexpected job losses or health crises, highlighting the vulnerability they faced. However, after enrolling in the program, they expressed newfound goals and ambitions to build assets that would protect their families and create generational wealth. One common goal mentioned by participants was to improve their education and advance their careers. They recognized that furthering their education would open up opportunities for higher-paying jobs and greater financial stability.
Improving housing situations and homeownership were also prominent goals expressed by participants. They envisioned providing better living conditions for themselves and their families, including making necessary repairs and securing stable housing. For some, the program offered the opportunity to save and accumulate resources for future homeownership. One participant shared their aspiration, stating, “That’s my plan - to save up to buy a mobile home for me and my kids to have something to call our own.” Another shared, “[My goal] was to actually pay off some debt, become financially stable, get my credit right in order to buy a house.”
Participants also discussed their entrepreneurial aspirations and the program’s impact in solidifying their dreams of owning their own businesses. The additional income provided by the program allowed them to invest in their entrepreneurial endeavors and establish the necessary credit lines. One participant shared their vision, stating, “[The program is] helping me establish the credit line because I don't have good credit, so I have to make deposits. So it is helping me with the deposits. I can make those necessary deposits to create those credit lines to establish the credit that I need to establish. Business credit to establishing a business.”
Other participants expressed goals such as owning their own store, pursuing trade certifications, and furthering their education through obtaining a GED. These aspirations highlight the multifaceted goals participants have for improving their financial well-being and creating opportunities for themselves and their families.
Personal Development Goals
In addition to their goals of homeownership, education, and small business development, participants in our qualitative interviews also expressed a strong desire for personal growth and development. Engaging in the program presented an opportunity for self-improvement and a chance to make a positive impact on their communities. One participant articulated their aspiration to give back and uplift their community, not just their immediate surroundings but also similar communities facing similar struggles. They emphasized the importance of networking and building connections to support collective growth and progress. As the participant shared, “I want to build the community up. I mean not just this community, but similar communities to us that may have similar interests, similar struggles, and...really network and build. That’s my biggest goal for the next two years.”
Participants also expressed a desire to improve their own circumstances and create a better future for themselves. Another participant expressed, “I want to be in a better position than I am at this moment.” Participants further highlighted a shift in their priorities. They expressed a desire to gain new experiences and broaden their horizons. One participant shared their aspiration, stating, “I want more for myself now… Material things, those stuff don’t matter no more. I want to get out and see the world.”
AI Created Themes
As described above, we fed all 1125 identified “significant statements” into ChatGPT and then prompted, “Act as a phenomenological qualitative researcher. All 1125 significant statements above come from interviews with 71 guaranteed income pilot participants. Please identify common themes from the statements.” ChatGPT provided the following response. Based on the 1125 significant statements from the interviews with 71 guaranteed income pilot participants, the following common themes emerge: • Financial struggles: Many participants expressed financial struggles and difficulties in making ends meet. They mentioned that even with a job, they were paying high amounts for housing and struggling to support their families. • Government assistance: Several participants mentioned receiving government assistance such as food stamps, TANF, Medicaid, and Section 8. They expressed gratitude for the assistance but also acknowledged that the assistance is limited and doesn’t always cover all their needs. • Impact of guaranteed income: Many participants expressed that the guaranteed income has provided them with some breathing room and allowed them to manage their finances better. They also mentioned that the income has motivated them to do better, especially for the sake of their children. • Struggles with parenting: Some participants mentioned that being a parent of multiple children can be stressful, especially when they don’t have enough support from the fathers of their children. • Desired improvements: Several participants mentioned wanting to use the income for home repairs, such as replacing windows and flooring. Some also mentioned wanting to go on a trip to Niagara Falls. • Location and living conditions: Several participants mentioned living in poor and high-crime areas, which can be difficult and stressful. Some expressed a desire to move to a better location. • Perceptions of government assistance: Some participants expressed that government assistance is only available for those who are very poor and that those who need a little assistance often don’t qualify. • Gratitude for assistance: Many participants expressed gratitude for the government assistance they have received, which has helped them afford things they otherwise wouldn’t be able to. • Impact of poverty: Participants described the various ways in which poverty affects their daily lives, such as not having hot water, no air conditioning, and difficulty affording medical treatments. • Burden of medical issues: Several participants mentioned that medical issues can be a burden, especially if they require frequent treatments, as it can be difficult to take time off work. These themes suggest that guaranteed income has the potential to alleviate some of the financial struggles and stress that participants face, but that additional assistance and improvements are still needed to help them achieve financial stability and a better quality of life.
Comparison of Human and AI-Created Themes
Human and AI Created Qualitative Themes, Organized by Similarity.
Arguably, we see some (but not perfect) overlap in about half of the human-centered themes and 80% of the ChatGPT themes. The AI-generated themes tend to be more specific to participants' economic circumstances and immediate concerns. As mentioned in our bracketing section, the research team feels a strong responsibility to avoid replicating stereotypes of Black women and instead focus on their strengths. Indeed, personal motivations and aspirations were noted by the human research team, but not ChatGPT. The human research team also had more contextual knowledge about social and racial inequity to frame some of the participants' concerns as, for example, “insufficient community resources,” rather than “local living conditions.”
The two themes identified by ChatGPT that the research team did not identify were “struggles with parenting” and “desired improvements.” Upon personal reflection, both themes are legitimate interpretations of the data. However, it is comical that ChaptGPT noted in the “desired improvements” themes that “Some also mentioned wanting to go on a trip to Niagara Falls.” Returning to the raw data, the term “Niagra” only appears once in all interviews.
When we asked ChatGPT to compare and contrast the human and AI-generated themes, it had the following to say, Overall, the human-generated themes provide a comprehensive and holistic understanding of participants' experiences, taking into account the broader social, economic, and cultural contexts that shape their lives. The AI-generated themes, while offering efficiency and scalability in data processing, may lack the nuanced understanding and interpretive flexibility that human researchers bring to the analysis. Combining the strengths of both human and AI analysis can lead to a more comprehensive and nuanced understanding of qualitative data.
Discussion
First released on November 30, 2022, as a “free research preview,” it is reasonable to expect ChatGPT to improve rapidly in the months and years ahead. Further, other companies are racing to release their versions of the highly successful software. To explore its current limitations (as of February 2023), we asked ChatGPT, “What are the limitations of ChatGPT?” While ChatGPT and other AI models have shown impressive capabilities in natural language processing and text generation, there are several limitations to their functionality. Here are some of the key limitations of ChatGPT: 1. Limited Contextual Understanding: ChatGPT operates on the basis of pattern recognition, which means that it has limited ability to understand the context and underlying meaning of the text it is processing. This can lead to errors in generating text that accurately reflects the intended meaning of the input. 2. Biases: ChatGPT is trained on large datasets of text, which can contain biases and prejudices that are reflected in the model’s output. For example, if the training data contains sexist or racist language, the model may generate text that reinforces these biases. 3. Lack of Common Sense Knowledge: ChatGPT lacks common sense knowledge, which can lead to nonsensical or inappropriate responses to certain prompts. For example, if prompted with a question like “Can you fly to the moon without a spacesuit?” ChatGPT may generate a response that is technically correct but lacks common sense. 4. Overreliance on Training Data: The quality of ChatGPT’s output is highly dependent on the quality and relevance of the training data used to develop the model. If the training data is biased, incomplete, or unrepresentative, this can impact the accuracy of the model’s output. 5. Difficulty in Generating Long-Form Content: While ChatGPT can generate coherent and cohesive short-form text, such as a paragraph or a few sentences, it can struggle to maintain coherence and structure over longer-form content such as essays or research reports. 6. Limited Memory: ChatGPT has limited memory and can only retain a certain amount of information at a time. This means that when generating text, it may not be able to maintain consistent references to previous information or maintain a consistent tone throughout the text. 7. Inability to Generate Original Ideas: ChatGPT can generate text that is similar in style and content to the training data it was based on, but it is not capable of generating original ideas or insights that go beyond the scope of the training data. 8. Lack of Emotional Intelligence: ChatGPT lacks the emotional intelligence to understand and respond appropriately to the emotions and feelings of the user. This can lead to insensitive or inappropriate responses in situations where empathy and understanding are required. Overall, while ChatGPT and other AI models have made significant advances in natural language processing, they still have limitations and are not yet able to replicate the full range of human language abilities. It is important to be aware of these limitations when using ChatGPT or other AI models and to use them appropriately in contexts where their strengths can be leveraged and their limitations understood.
In conducting this comparison, we did find many parallels between our human and AI-generated qualitative analyses, with the human coders recognizing some themes that ChatGPT did not and vice versa. Specifically, ChatGPT did not discuss the motivating factors of children and faith for participants, the desire for privacy, or asset and personal development goals. Inflation was not mentioned specifically by ChatGPT but was alluded to in its “financial struggles” theme: “even with a job, they were paying high amounts for housing and struggling to support their families.” Because ChatGPT does not currently have access to the live internet, it has limited knowledge of events before 2021. Human researchers, more aware of current events, could better recognize the theme of inflation given current national economic conditions.
More broadly, it becomes evident that human-centered coding offers several advantages over AI analysis. Human coders brought a depth of analysis, sensitivity to nuances, and interpretive flexibility that AI models like ChatGPT lack. The human analysis benefits from the contextual knowledge and subject matter expertise of the researchers, enabling them to recognize subtle connections and understand the impact of structural and policy inequities on participant experiences. This contextual knowledge enhances the interpretive richness and depth of the analysis, allowing for a more comprehensive understanding of the data. Furthermore, human analysis benefits from the iterative and collaborative nature of the coding process, where researchers engage in discussions, reflection, and validation to refine and revise their thematic analysis. The lack of transparency in the internal workings of AI models makes it challenging to understand the specific patterns or criteria used for theme generation.
However, incorporating AI in qualitative analysis offers several advantages. Firstly, AI enables efficient and speedy data processing, allowing researchers to analyze large volumes of textual data quickly. AI can also explore patterns and themes that may not be immediately apparent to human coders. By leveraging computational algorithms, AI can detect hidden connections and uncover new insights in the data. This scalability and ability to identify overarching themes make AI valuable in analyzing extensive datasets. The comparative analysis in the current study suggests that AI should be seen as a complement to human analysis, rather than a replacement. Human-centered coding remains essential for capturing the depth, context, and interpretive nuances that AI may not fully grasp. By combining human expertise and AI technologies, researchers can enhance the qualitative analysis process and gain a more comprehensive understanding of the data.
Further, there are many ways to improve our initial test of the technology’s capacity in qualitative research. For example, future qualitative research might explore feeding the raw interview transcripts into ChatGPT rather than the human-determined “significant statements.” In the current study, with the large volume of interviews (71) and ChatGPT’s current character limits, this was untenable here. Additionally, there is a myriad of applications and add-ons in development that could improve ChatGPT’s functionality for researchers. For example, scite. ai offers an assistant, powered by ChatGPT, that provides summaries of academic literature with citations in response to user queries.
Overall, we find that AI like ChatGPT provides a powerful tool to supplement complex human-centered tasks such as qualitative research analysis. For example, rather than conducting a ChatGPT analysis after completing our human-generated analysis, we could have incorporated AI-generated themes into our triangulation discussions to help identify oversights, alternative frames, and personal biases. While the future of AI development is unknown, we predict that tools like ChatGPT will become an additional tool to facilitate research tasks, just as Google Scholar, citation management, data analysis, and grammar-checking software have already done.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Give Directly.
