Sage Journals: Discover world-class research

Abstract

Generative artificial intelligence is regularly tasked with navigating the complexities of human-to-human interaction, including peer-to-peer mental health support and improving interpersonal emotion regulation within customer service settings. Qualitative findings question whether artificial intelligence deserves a space in this area; however, many recent studies indicate that artificial intelligence may improve human-to-human interaction. In the current study, and across two samples (N_Givers = 522; N_Receivers = 580; N_Total = 1,102) gathered to detect small effects, we examine a specific use case of generative artificial intelligence in the workplace and whether it can be leveraged to improve prosocial human-to-human messaging above and beyond human-written prompts using a quantitative survey experiment. Specifically, we train and examine the impact of generative artificial intelligence on participants’ perceptions of employee recognition messages in the workplace. We found that participants: (a) believed the trained generative artificial intelligence was in line with Human Resources (HR) industry best practice; and (b) learned from the generative artificial intelligence, believing it would lead them to change their behavior. Generative artificial intelligence messages were more in line with HR industry best practice than human-written messages, and these messages were longer and more positively valenced. Further, after controlling for message length, had more nouns, adjectives, and verbs than human-written messages. Consistencies with the current literature and the place generative artificial intelligence has in human-to-human interaction moving forward are discussed.

Plain Language Summary

AI can Help Write Better Recognition Messages at Work

This study explored whether ChatGPT can enhance how people write recognition messages to coworkers in the workplace. With data from over 1,100 participants, the researchers compared messages written by humans to those generated or improved by AI. Participants believed the AI messages aligned more closely with Human Resources' best practices. People also reported that they learned from the AI and would change how they write future messages based on its suggestions. These findings suggest that AI may play a valuable role in improving how people communicate support and recognition in professional settings.

Keywords

gratitude recognition artificial intelligence human resources ChatGPT

Introduction

Artificial intelligence (AI) is regularly tasked with making complex tasks more manageable for humans, including correcting spelling and grammar, automating mundane tasks, generating computer code, and finding intricate patterns in multidimensional data. Given the recent success of OpenAI and other large language models (Vaswani et al., 2017), artificial intelligence is now tasked with writing open-ended social messages requiring intimate knowledge of moral affect at scale (see McCullough, 2002). Large language models are being tasked to navigate conversations predicated on an intimate knowledge of “fuzzy concepts” (or sets; Zadeh, 1965), which have long frustrated humans due to the lack of clear conceptual definitions (e.g., intelligence, empathy, and see Loki’s Wager). For example, recent evidence suggests that AI can enhance written prosocial messages in emotionally complicated settings, including offering empathy to those struggling with mental health concerns (Sharma et al., 2023). In one study from Sharma et al. (2023), untrained peer supporters were recruited from TalkLife, a peer-to-peer support network for mental health concerns, to be assigned to write responses on their own or receive feedback through prompts from a trained AI. Overall, responses generated by the AI were rated equal to or more empathic than the human-only responses 63.60% of the time. Further, peer supporters reported using the service at least once 77.58% of the time, suggestions from the AI were used 83.08% of the time, and 77.70% of participants wanted an AI feedback system to be deployed on TalkLife (Sharma et al., 2023). Indeed, the supporters were perceived as more empathic, used the service regularly, and believed it to be a useful tool.

Though these results are impressive, they are not unique and are becoming more ubiquitous despite ongoing concerns from some providers that “robots [are] incapable of being empathic in a caring relationship” (Doraiswamy et al., 2020, p. 6). Indeed, findings where large language models promote and aid in prosocial human interaction are increasingly common (Henkel et al., 2020; Huang & Rust, 2018). For instance, AI has been leveraged to assist customer service employees in regulating customer emotions (Henkel et al., 2020). This human-AI collaboration has been found to be beneficial. For example, AI has aided employees in regulating customer emotions leading to more positive customer ratings. As a result, employees’ affective well-being was also improved through a reduction in perceived stress (Henkel et al., 2020). In both instances, the AI-coaching is being used to move negative emotional experiences back to baseline (i.e., emotion dysfunction to euthymia; Henkel et al., 2020; Sharma et al., 2023)—a pattern within the psychological sciences that has been identified for decades (Averill, 1980; Fredrickson, 1998). However, the question remains whether AI-coaching can move those in a presumed euthymic state to a state of flourishing, as much of positive psychology argues (e.g., Seligman & Csikszentmihalyi, 2000). These studies highlight the remarkable capabilities of AI in enhancing human emotional experiences and interactions, particularly in terms of empathy and emotional regulation. Building on this foundation, it becomes pertinent to explore how AI can contribute to more advanced aspects of positive psychology, such as gratitude, which goes beyond merely returning to a state of emotional equilibrium.

Expressions of Gratitude as a Starting Point—A Well-Supported Positive Psychology Intervention

Gratitude, a concept deeply rooted in various world religions such as Christianity, Judaism, and Islam, represents a more profound and holistic approach to human well-being. Those who regularly practice gratitude often experience enhanced self-esteem, greater resiliency, and improved quality of relationships (Rashid & Seligman, 2018). Conversely, the absence of gratitude, or ingratitude, is frequently seen as a moral failing with negative implications for both the individual and the wider society including workplace relationships (Amato, 1982; Emmons et al., 2003; McCullough, 2002; McCullough et al., 2002). Gratitude provides a unifying framework for this study. As a central construct in positive psychology and a cornerstone for employee recognition practices, gratitude has often been linked with individual well-being, mental health cost savings, and improved organizational functioning (Lovell et al., 2022, 2023, 2024, 2025).

Gratitude has been expanded on theoretically for decades, and gratitude interventions have become common in the empirical literature and in the business industry (e.g., see Algoe, 2012; Algoe et al., 2020; Emmons et al., 2003; Locklear et al., 2022; McCullough, 2002). In one of the original gratitude intervention studies, participants who recorded five things for which they felt grateful each week for ten weeks rated their life more favorably, reported better health, increased their time spent on exercise, and reported greater well-being compared to the other conditions (Emmons et al., 2003). Other studies have found that delivering gratitude letters to others improved happiness and decreased depressive symptoms to a large degree, but the effects returned to baseline over follow-up (Seligman et al., 2005). The improvements in happiness have been replicated in subsequent studies (Mongrain & Anselmo-Matthews, 2012).

Recent studies have resulted in similar outcomes. For example, participants who focused on written gratitude letters to another individual saw improvements in positive affect compared with written non-directed gratitude (Cregg et al., 2022). More ecologically valid studies (i.e., longitudinal non-treatment focused designs) have started to argue for a reinforcing relationship. Indeed, more positive affect predicts more expressions of gratitude, and more expressions of gratitude predict more positive affect (Snippe et al., 2018). Thus, it is unsurprising that a recent meta-analysis investigating the impact of gratitude interventions on well-being has argued small- to medium-sized effects decreases in depressive (g = −0.23) and anxious (g = −0.55) symptoms (Cregg & Cheavens, 2021).

Given the consistency of the findings, it is unsurprising that some companies have begun to facilitate the development and delivery of gratitude interventions at scale (otherwise known as “recognition” or “employee recognition” within industry). Perhaps unsurprisingly, Human Resources and industry researchers have conducted their own studies to assess the impact of so-called recognition programs (gratitude interventions) on employee well-being measures (Lovell et al., 2022, 2023, 2024, 2025). In addition to taking place within an ecologically-valid setting, these findings differ from those presented above. Instead of focusing on the well-being of the giver of the message, these studies often focus on the receiver's well-being. One such example is the O.C. Tanner Company, who in their annual Global Culture Report, has indicated that recognition that is in line with Human Resources (HR) Industry best practices (i.e., inclusive, relevant, specific, purposeful, meaningful, sincere, and empathic) is associated with feelings of support, feeling valued, and feeling heard (Lovell et al., 2022, 2023, 2024, 2025). Previous reports indicate that recognition that aligns with HR best practices is associated with lower anxiety and stress, higher engagement, more feelings of “purpose,” and greater well-being (Gallup, 2023; Lovell et al., 2022, 2023, 2024, 2025). Thus, employee-to-employee gratitude messaging delivered at scale (i.e., to millions) has the potential to have dramatic impacts on the well-being of both the person who sends the message (e.g., Cregg et al., 2022; Emmons et al, 2003; Seligman et al., 2005; Snippe et al., 2018) as well as the person who receives it (Lovell et al., 2022, 2023, 2024, 2025). First understanding whether the AI-coached messaging is in line with HR best practices compared with human-written messaging would be an initial first step in training an AI-coach aimed at improving prosocial messaging among employees.

Limitations and Gaps in the Current Literature

While strides and innovation are normative in artificial intelligence research, exploratory research in this early stage of development is critical. First, many human-AI collaboration studies attempt to improve relationship processes; however, data is typically gathered from the giver or the receiver (e.g. Khadpe et al. 2020; Sowa et al., 2021). Rarely is data collected from both perspectives (i.e., the giver and the receiver). Positive effects are theoretically assumed to reflect onto the other party, but this isn’t explicitly tested except for a few instances (e.g., Bernard et al., 2023). Second, while understanding the general effects of human-AI collaboration is important, the current literature has yet to explore potential causal mechanisms that may contribute to these effects. Multiple studies have indicated that human-AI collaboration improves (and is associated with more satisfying) written messaging (Henkel et al., 2020; Sharma et al., 2023), and because this messaging is constrained to linguistic features (as opposed to voice tone, facial expressions, or other body language), there should be linguistic differences between the human-written and AI-generated messages. However, we need more information to support this claim.

Research Aims and Hypotheses

Given the current literature, this study has four aims. The first two aims are confirmatory and seek to determine (a) whether participants are satisfied with the AI coaching, and (b) whether participants learn from the coaching and would be willing to revise their message based on the coaching. Given the previous literature, we anticipate participants will indicate that they have learned something from the coaching and would be willing to change their behavior. The third aim of the current study is also confirmatory and aims to examine whether participants believe the material is more in line with industry/HR best practice. Again, given the previous literature, we expect participants will find the messages generated by the AI to be more in line with industry best practices than human-written responses. In the fourth and final exploratory aim of this study, we seek to understand some of the causal mechanisms at play within the messages. Specifically, we aim to examine whether length, sentiment, and part-of-speech differences exist between the AI-generated and human-written messages.

Method

Transparency and Openness: Data Availability Statement

Below, we describe our sampling plan. No data exclusions or manipulations were made to the data. Measures are described below. All code, data, and research materials have been uploaded to the associated OSF webpage for review and reproducibility (https://osf.io/tnbsz/). Data were analyzed using R version 4.3.1 (R Core Team, 2013) and Bayesian analyses were conducted using brms version 2.19.0 (Bürkner, 2017) and BayesianFirstAid version 0.1 (Bååth, 2014). Finally, linguistic elements were extracted using NLTK version 3.6.6 (Loper & Bird, 2002) using Python version 3.11.6 (Van Rossum, 2007).

Power Analysis and Procedure

Preliminary power analyses were conducted before data were gathered, indicating a sample of 504 was needed to detect a small-sized effect (i.e., d = 0.25) with 0.80 power and a 0.05 false-positive rate. To strengthen reliability, we oversampled slightly yielding two independent samples (N_Givers = 522; N_Receivers = 580). This two-sample design was chosen intentionally to cover both sides of the recognition exchange (i.e., givers and receivers), consistent with our aims and the limitations identified in the previous literature (e.g. Khadpe et al., 2020; Sowa et al., 2021). By collecting independent and samples designed to detect small effects, we ensured adequate coverage of both perspectives.

All participants were recruited from Lucid and directed to take a survey. Lucid is the largest available sample aggregator and facilitates direct-to-respondent sampling through their proprietary marketplace platform (Coppock & McClellan, 2019). We can ensure unique responses are received for our dataset through the marketplace interface. Panel providers individually handle respondent payments, ensuring complete confidentiality. The value of incentives for survey participants ranged from $1.50 to $3.45 USD, and they are often provided as redeemable points for merchandise, gift cards, and cash. The survey was built with Alchemer software (formerly SurveyGizmo).

Both samples were independently gathered in October 2023, the response rate was 83% for the receiver sample and 74% for the giver sample, and both surveys were approximately 20 min long. In the first section of both the giver and receiver survey, participants were given a series of 20 manufactured vignettes that provided the participant with a scenario and asked which of two responses (i.e., one human-written and the other AI-generated) the participant preferred (see Table 1). The human-written responses were written by real humans in real-life settings and received from [The O.C. Tanner Company]—a software-as-a-service employee recognition company. Participants were then directed to answer demographic questions. After this first task, the procedure differed between the two surveys.

Table 1.

Example Vignette (Vignette 1) from the Receiver Sample.

Imagine you are an employee at a call center providing customer service for healthcare supplements. Often, these days can be long, grueling, and you easily become bored, tired, and irritable. A few of your colleagues have even quit because of the stress. You notice it has started to impact your colleague’s delivery in calls with the customers. Your manager recognizes the frustration, annoyance, and stress and shares this feedback with you and your team.
In response, you take the feedback seriously. You go above and beyond with your clients. You take care to change your tone of voice, become actively engaged in your calls, and encourage others around you to do the same. You also remember a few friends and family members who have been looking for work and refer them to fill the empty positions.
You receive an e-Card from an employee recognition app—it is from your manager. Which e-Card would make you feel most recognized for your efforts?
- Hi [Your Name], just want to recognize you for your excellence in delivering for our clients, and also your teamwork in sending me referrals. I really appreciate our partnership. :) Wishing you a great week
- Hi [Your Name], I wanted to take a moment to recognize your exceptional delivery for our clients. Your attention to detail and commitment to excellence have not gone unnoticed. Additionally, I deeply appreciate your proactive teamwork in consistently sending me referrals. Your efforts have truly strengthened our partnership. Thank you for your dedication and collaboration. Wishing you a great week ahead!

In the second giver survey task, participants read 20 vignettes where they received coaching on a message they “wrote.” Participants then rated whether the coaching aligned with industry best practices, whether they learned something from the coaching, and whether the coaching would lead them to alter the original message.

Participants were also tasked with reading 20 vignettes in the second receiver survey task. However, they were randomly assigned to one of two experimental conditions. In the first condition, participants were randomly assigned to “receive” a message written by a human or a prompt generated by generative AI. Participants then rated whether the message they received aligned with industry best practices.

Prompt Engineering

The generative model was created using OpenAI’s gpt-3.5-turbo model as this model is capable, cost-effective, works well for traditional completions, and generates natural language (OpenAI, 2023). From here, prompt engineering (not fine-tuning) was used to generate the coached responses using recommendations for output customization (see White et al., 2023). Each output request was explicitly mapped to the dimensions of HR best practice (i.e., inclusive, relevant, specific, purposeful, meaningful, sincere, and empathic), with prompts structured in line with definitions provided by the Global Culture Report.

Participants

Participants in both the givers and receivers samples reported being similar ages (Mean_Givers = 45.68, SD_Givers = 16.89; Mean_Receivers = 45.86, SD_Receivers = 16.77) with no significant differences between the two samples (p = .86)—this pattern held for income (Median_Givers = $40,000; Median_Receivers = $40,500, p = .80) and hours worked (Median_Givers = 30; Median_Receivers = 30, p = .29). Most participants identified as female (Givers = 49.8%; Receivers = 51.8%), slightly fewer identified as male (Givers = 47.5%; Receivers = 45.0%), and the remaining individuals identified with transgender female (Givers = 0.8%; Receivers = 1.4%), transgender male (Givers = 1.0%; Receivers = 1.2%), or a category not listed (Givers = 1.0%; Receivers = 1.0%). Further, most participants identified as White Non-Hispanic (Givers = 67.8%; Receivers = 66.0%), followed by Black (Givers = 11.9%; Receivers = 12.8%), White Hispanic (Givers = 8.6%; Receivers = 8.5%), Asian (Givers = 2.3%; Receivers = 2.2%), Black Hispanic (Givers = 1.5%; Receivers = 2.1%), American Indian or Alaskan Native (Givers = 1.9%; Receivers = 0.5%), Middle Eastern or North African (Givers = 0.2%; Receivers = 0.9%), Native Hawaiian or Pacific Islander (Givers = 0.4%; Receivers = 0.5%), and the rest of the sample identified as “Other” (Givers = 5.4%; Receivers = 6.5%). Lastly, most participants came from customer service backgrounds (Givers = 13.2%; Receivers = 17.4%), followed by those with backgrounds in information technology (Givers = 10.0%; Receivers = 11.0%). Other backgrounds including general administration, finance, HR, manufacturing, marketing, patient care, product development, sales, shipping, and training. These occupations were represented by less than 10% of the sample.

Measures

Best Recognition Practice

Eight items were used to measure best recognition practices within the giver’s sample, and six items were used to measure best recognition practices within the receiver’s sample—all were on a 1 to 7 Likert-style scale ranging from Strongly Disagree to Strongly Agree. Within the giver’s sample, participants were asked the degree to which they agreed the coaching was helpful, inclusive, relevant, specific, purposeful, meaningful, sincere, and empathic. Within the receivers’ sample, participants were asked the degree to which the message they were randomly assigned to receive was specific, inclusive, purposeful, meaningful, sincere, and empathic.

These constructs have been consistently linked to improved business outcomes with nearly a decade of evidence supporting their utility. For instance, organizations perceived to be inclusive have employees who are 4× more likely to be promoters, 7× more likely to believe senior leaders can achieve organizational goals, and 10× more likely to report thriving at work. When empathy is high, there is a 50% increase in the odds that employees perceive their organization as supportive. Likewise, when purpose is high, organizational innovation is 5× more likely, and the same is true when individuals report their organization as having meaningful work.

Within the receivers’ sample, across all vignettes, a unidimensional factor solution fit the data well (i.e., CFI > 0.95, TLI > 0.95, RMSEA < 0.08, SRMR < 0.08; see Kenny, 2020) even under the stringent assumptions of tau-equivalence (CFI = 0.98, TLI = 0.98, RMSEA = 0.07, 90% CI [0.07, 0.08], SRMR = 0.04) and internal consistency was excellent (α = .93, 95% CI [0.92, 0.93]). These findings justify using a total sum score to compare the treatment (i.e., AI-generated message) and control (i.e., human-written) messages and allowed us to decrease our false-positive rate by conducting one overarching test per vignette.

Participant Learning and Behavior Change

These separate constructs were measured using two items. To measure learning, participants were asked, “Did you learn anything from the coaching?” and could answer “Yes” (coded as 1) or “No” (coded as 0). Similarly, to measure behavioral change, participants were asked, “Given the feedback, would you revise your original e-Card?” participants could answer “Yes” (coded as 1) or “No” (coded as 0).

Linguistic Measurement

In Python version 3.11.6, the total number of words, positively-valenced words, and negatively-valenced words were counted using a natural language toolkit (NLTK; Loper & Bird, 2002). Sentiment was measured using Valence Aware and sEntiment Reasoner (VADER) in Python which assigns a sentient weight to words and punctuation from high negatively-valenced to highly positively-valenced words. This also included a semi-normal transformation. Finally, parts of speech were tagged and counted using the tag library in NTLKIN Python.

Data Analysis

For the first aim of the current study, measures of central tendency (i.e., mean, median, or mode) and spread (i.e., standard deviation) were used to describe the most frequent response after matching the measure to the skew of the respective distribution. In the second and third aims of the current study, Bayesian binomial tests from the BayesianFirstAid package in R were used to compare differences in proportions (e.g., the proportion of individuals who prefer to give/receive messages generated by an AI or written by humans; Bååth, 2014). The Beta (1, 1) prior distribution was used, given it is uninformative and will allow the likelihood to inform the posterior. In this study’s fourth and fifth aim, Bayesian regression from the brms package in R was used to determine whether AI-generated or human-written messages were more in line with HR best practices and whether there were linguistic differences between messages (Bürkner, 2017). Standard brms priors were used, allowing the likelihood (rather than the prior) to inform the posterior. Percentages with 95% highest posterior density intervals (HPDIs) that did not include 50% were considered statistically reliable differences. Similarly, confidence intervals (CIs) that did not include zero were also interpreted as reliable evidence of a difference.

Results

Aim 1: Do Participants Believe the AI Coaching Aligns with HR Best Practice?

Across all measures, the most frequent (and the median) response for a left-skewed distribution was participants “agreed” (6 on a 1 to 7 scale) the coaching was helpful (Mean = 5.44; SD = 1.46), inclusive (Mean = 5.32; SD = 1.48), relevant (Mean = 5.44; SD = 1.44), specific (Mean = 5.47; SD = 1.40), purposeful (Mean = 5.49; SD = 1.39), meaningful (Mean = 5.46; SD = 1.42), sincere (Mean = 5.46; SD = 1.45), and empathic (Mean = 5.35; SD = 1.45).

Aim 2: Are Participants Learning from the AI Coach, and Would it Change Their Behavior?

Overall, 79.0% (95% HPDI [75.6%, 82.5%]) of participants reported learning something from the coaching, and 63.5% (95% HPDI [59.4%, 67.6%]) of participants reported they would revise their original message after receiving the coaching. Even within the poorest rated vignettes, when at the lowest, most participants reported learning something (75.6%, 95% HPDI [71.8%, 79.2%]) and reported they would make revisions to their original message (59.0%, 95% HPDI [55.0%, 63.2%]). Finally, at the highest, 84.2% (95% HPDI [81.1%, 87.4%]) of participants reported learning something from the coaching, and 68.5% (95% HPDI [64.3%, 72.4%]) would revise the original message in response to the coaching.

Aim 3: Do Receivers Perceive the AI-Generated or Human-Written Messages as Being More in Line with HR Best Practices?

Participants in the AI-generated condition rated messages higher (i.e., more in line with HR best practice) than those assigned to receive human-written messages. Seventeen out of 20 vignettes possessed statistically reliable treatment effects (i.e., credible intervals did not include zero). While the effects were not statistically reliable in 3 out of 20 vignettes, the average treatment effect still favored the AI-generated messages in each instance. Averaging across vignettes, the effect was small and statistically reliable (d = 0.34, 95% CI [0.18, 0.49]). The largest statistically reliable treatment effect fell within the medium-sized range (d_{Vignette 34} = 0.67, 95% CI [0.52, 0.82]), whereas the smallest statistically reliable treatment effect was trivial in size (d_{Vignette 31} = 0.18, 95% CI [0.01, 0.34]; see Table 2).

Table 2.

Bayesian Regression Analyses of Real Experiment Determining Whether Message Receivers Perceive the AI-Generated Messages Are More in Line With HR Industry Best Practice Compared With Human-Written Messages.

Vignette	Cohen’s d	95% CI LB	95% CI UB
21	0.647	0.493	0.803
22	0.219	0.064	0.377
23	0.614	0.460	0.770
24	0.348	0.190	0.506
25	0.371	0.214	0.537
26	0.194	0.035	0.355
27	0.189	0.033	0.347
28	0.115	−0.050	0.282
29	0.627	0.472	0.780
30	0.251	0.090	0.411
31	0.012	−0.155	0.175
32	0.176	0.007	0.341
33	0.414	0.251	0.570
34	0.670	0.516	0.820
35	0.464	0.307	0.616
36	0.154	−0.012	0.316
37	0.215	0.061	0.365
38	0.479	0.320	0.640
39	0.260	0.099	0.418
40	0.290	0.127	0.456

Note. Cohen’s ds whose 95% credible intervals (CI) did not include zero were assumed to be indicative of an effect. Positive values were indicative of favoring the AI-generated messages whereas negative values were indicative of favoring the human-written messages. In 17/20 instances (and with few exceptions; i.e., 28, 31, 36), participants rated AI-generated messages as falling more in line with HR industry best practices compared to human-written messages. CI = Credible Interval; UB = Upper Bound; LB = Lower Bound.

Aim 4: Are There Length, Sentiment, Parts-of-Speech Differences Between the AI-Generated and Human-Written Messages?

Compared with the human-written messages, the AI-generated messages had more words, more syllables, a greater number of positive words, and the number of negative words did not differ by condition. Overall, sentiment within the AI-generated messages was more positive regardless of the metric used. Regarding parts of speech, there were no differences in the number of prepositions or adverbs between conditions; however, in the AI-generated messages, there were more adjectives, pronouns, nouns, and verbs than in the human-written messages. When controlling for the total number of words in a message, the AI-generated messages still had more syllables, positive words, positive sentiment, adjectives, nouns, and verbs than the human-written messages—pronouns no longer had a reliable effect (see Table 3).

Table 3.

Bayesian Poisson Regression Outcomes of Length, Sentiment, and Part of Speech Differences Between AI-Generated and Human-Written Messages. With the Exception of Negative Words, Prepositions, and Adverbs, All Other Outcomes Positively Favored the AI-Generated Over Human-Written Messages. All Betas Were Left in Their Original Units.

Outcome	b	95% CI LB	95% CI LB
Length	0.470*	0.345	0.593
Positive words	0.864*	0.612	1.121
Negative words	0.419	−0.386	1.217
Sentiment	0.092*	0.024	0.155
Semi-normalized sentiment	3.628*	1.967	5.337
VADER	0.103*	0.048	0.159
Average syllable length	0.301*	0.177	0.425
Prepositions	0.236	−0.051	0.533
Adjectives	0.676*	0.319	1.040
Adverbs	0.368	−0.033	0.777
Pronouns	0.328*	0.099	0.560
Nouns	0.375*	0.215	0.541
Verbs	0.471*	0.211	0.732

Note. CI = Credible Interval; UB = Upper Bound; LB = Lower Bound.

95% Credible Interval does not include zero and is thus indicative of a reliable relationship.

Discussion

The current study had four aims. The first two aims were confirmatory—do individuals perceive AI-generated messages to be in line with best practice, do participants report learning from the AI coach, and does the coaching result in changes in behavior? We found most participants “agreed” the coaching was in line with HR industry best practices: 79% of participants reported learning something from the coaching, and 63.5% of participants reported they would revise their message in response to the coaching. The third aim was also confirmatory and sought to understand if receivers perceived the AI-generated or human-written messages to be more in line with HR industry best practice (Aim 3). Participants rated AI-generated messages as falling more in line with HR industry best practices than human-written messages.

In the final aim, we sought to explore possible reasons why participants preferred and rated the AI-generated messages higher than the human-written responses by examining language features. Overall, we found the AI-generated messages had higher word counts, greater numbers of syllables and descriptive words, and more positive sentiment than the human-written responses.

Discussion of Study Aims: Consistencies with the Current Literature

Regarding the first two aims, the findings appear largely in line with the current literature. Participants reported the messages were in line with HR industry best practices, that they learned from the coach, and that they would revise their original message in response to the coaching. Though some researchers have been skeptical that generative AI can reliably navigate the complexities of human interaction and produce prosocial human-to-human messaging (e.g., expressions of empathy; Blease et al., 2019), this paper adds to the already robust literature that generative AI can reliably assist humans in navigating the complexities of human-to-human interaction (Henkel et al., 2020; Sharma et al., 2023). Indeed, in a short period of time, researchers from various industries, including mental health, customer service, and now, human resources, have all shown how generative AI can assist humans in peer-to-peer mental health support (Sharma et al., 2023), interpersonal emotion regulation of customer service interactions (Henkel et al., 2020), and generating prosocial messaging consistent with a moral affect (i.e., the current study).

In the third aim, the findings build off what has already been shown in the first two aims. Receivers perceive the AI-generated messages as being more in line with HR industry best practices than the human-written responses (Lovell et al., 2023, 2024). This finding is consistent with Sharma et al. (2023), who found that the AI-generated responses were perceived as more empathic than the messages written just by humans. These findings may make some uncomfortable. Indeed, the humanist might hope for a world in which the giver of a prosocial message might have the perfect and desired impact on the receiver of the same message untainted by an artificial bolstering. However, ChatGPT has been primarily trained by observing human-generated content publicly available online (Schade, 2023). Thus, engineering ChatGPT to engage in prosocial human messaging may allow users to monopolize, tap into, and interface with some of the most positive parts of publicly available human-generated content. Simply put, this engineering enables current users to leverage past human-generated content to improve their own. Thus, AI operates as an important adjunctive coach in this workplace intervention.

In the current study's final aim, we examined differences between AI-generated and human-written messages by examining parts-of-speech. First, we found the AI-generated messages were longer (i.e., had more words), had a greater number of positive words, and had more positive sentiment. Longer messages with more positively valenced words may allow the positive affect to be maintained while the giver writes the messages and the receiver reads it. Second, after controlling for message length, the AI-generated messages had more adjectives, nouns, and verbs than the human-written messages. Though limited by the number of messages analyzed, preventing us from robust word clouds or most frequent word pairings (40 AI-generated and 40 human-written), these initial findings might start the initial recipe for understanding the language of well-received gratitude. For instance, these results might suggest that lengthy messages with positive sentiment that focus on actions (verbs) and describe (adjectives) people (nouns) might be the initial underpinnings of the language of gratitude. However, more research here is required.

Strengths, Limitations, and Future Research Directions

Some strengths of the current study include the study design, the overall positive reception of the AI coach, and the efforts to understand differences between the AI-generated messages over the human-written responses. The study design comprised two samples sufficiently powered to detect small-sized effects, were demographically representative of the general U.S. population, and represented both sides of the interaction (givers and receivers). Next, the data consistently indicated the AI coach had the desired effect throughout each aim. Participants reported the coaching was in line with HR industry best practices, learning from the coach, and that the coaching would change their behavior. Lastly, we examined differences between the AI-generated over the human-written messages. The generated messages were longer, had more syllables, had a greater number of descriptive words, and carried more positive sentiment than the human-written messages.

One limitation of the current study was the hypothetical and lab-like setting in which the data were gathered. Indeed, participants were given a set of 40 standardized vignettes and were asked to place themselves in a manufactured scenario. While this methodological decision lends well to experimental design, it is removed from the more externally valid situation of actually giving and receiving these messages. While we addressed this problem as much as possible by utilizing stimuli that reflect a “real-life” message, it does provide an “unrealistically powerful manipulation” of a pure treatment that does not reflect a typical day-to-day reality of numerous and competing frames of information (Kinder & Palfrey, 1993, p. 27; Sniderman & Theriault, 2004; see also Chong & Druckman, 2012). Thus, generalization may be somewhat limited due to inflated treatment effects (Gaines et al., 2007). However, some recent work does argue that participants often cannot tell the difference between messages written by humans or AI, preferring the messages written by artificial intelligence over the human-generated content, even in high-intensity situations like couple therapy (Hatch et al., 2025).

Another limitation of the current study was the small number of vignettes that were delivered to participants. Though the final aim of this study aimed to start to explore why participants preferred one message over the other, 40 vignettes is hardly enough to form any strong conclusions. Thus, some recommendations for future research include gathering data in more externally valid scenarios (actual givers and receivers) and expand the number of messages written by humans or and an AI coach to robustly begin to understand why participants prefer one message over another. Further, cross-cultural and cross-sector evaluations could be worthwhile as cultural factors may shape how employees perceive AI-driven recognition. Together, these recommendations would allow for prescriptive recommendations to be given directly to HR practitioners.

A Note on Empathy, Other Fuzzy Constructs, and Metaphysics

Debates on whether AI is capable of genuine empathy (or workplace best practice) often hinge on claims that machines lack subjective experience, embodiment, or intentionality. While these are important philosophical concerns, they remain metaphysical and untestable. Following Turing’s (1950) functionalist perspective, we do not attempt to resolve whether AI feels empathy or other fuzzy constructs. Instead, we focus on the pragmatic question: can AI generate outputs that are perceived as empathic in human interactions? Our findings suggest that the functional expression of empathy—how messages are received and experienced—can be achieved through generative AI, regardless of whether the system has inner states.

Conclusion

Generative AI is regularly being tasked with navigating the complexities of human-to-human interaction in a superior way including providing recommendations on peer support for mental health, interpersonal emotion regulation within customer service settings, and improved prosocial human messaging within the workplace. In the current study, using a large sample, experiments, prompt engineering, and generative AI, we found participants (a) believed the AI-coach was in line with HR industry best practice, (b) participants are learning from the AI-coach and it would lead them to change their behavior, (c) receivers believed the AI-generated messages to be more in line with HR industry best practice than human-written messages, and (d) these messages were longer, positively valanced, and after controlling for message length, had a greater number of nouns, adjectives, and verbs than human-written messages.

Footnotes

ORCID iDs

S. Gabe Hatch

Cristen Dalessandro

Bryar Topham

Alexander Lovell

Ethical Considerations

Our research was produced as part of our work with a private organization in the US. Since our private organization is not federally funded, nor affiliated with any federally funded institution, nor conducting research in conjunction with any federal agency, we are not required to obtain approval from an IRB to complete nor publish this research. According to the updated 2018 Common Rule, “this policy applies to all research involving human subjects conducted, supported, or otherwise subject to regulation by any Federal department or agency that takes appropriate administrative action to make the policy applicable to such research” (2018 Common Rule, 45 C.F.R. § 46.101(a)). Nevertheless, we maintain a robust ethics standard. All researchers within our organization are required to complete PI ethics courses through CITI to meet IRB certification criteria. Additionally, each research protocol is reviewed to ensure compliance with statutory obligations, regardless of our non-requirement to do so. Further, research protocol are continually monitored through completion to ensure compliance with the Belmont Report and principles of sound research design and ethics. Lastly, we maintain an advisory committee that regularly reviews research protocols, as needed, to ensure our research procedures and guidelines meet evolving requirements within the United States, regardless of our non-requirement to comply. With all this said, in our review and in consultation with external IRB reviewers, this study is exempt from IRB review under exemption category 2 (45 C.F.R. § 46.101(b)). During the completion of this research, no qualifying events occurred, or substantive changes made that would change this exemption. Our survey included informed consent. Further, while our survey instrument collected general demographic data in addition to the questions central to our study as presented, no personally identifiable information was collected from respondents. This ensures that this exempted study continues to be minimal risk for research participants. This research was conducted for the O.C. Tanner Institute, and analyzed data was confidential to protect participant identity. Participants also had the option to discontinue the survey at any time.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by the O.C. Tanner Company.

Declaration of Conflicting Interests

The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: This research was supported by the O.C. Tanner Company (Salt Lake City, UT, USA), which is where the manuscript authors are employed. Publication may lead to the development of products licensed to O.C. Tanner, in which the authors—as employees of the O.C. Tanner Company—may have a business and/or financial interest.

Data Availability Statement

All code, data, and research materials have been uploaded to the associated OSF webpage for review and reproducibility ().

References

Algoe

S. B.

(2012) Find, remind, and bind: The functions of gratitude in everyday relationships, Social and Personal Psychology Compass, 6, 455–469. https://doi.org/10.1111/j.1751-9004.2012.00439.x

Algoe

S. B.

Dwyer

P.C.

Younge

Ovesis

(2020). A new perspective on the social functions of emotions: Gratitude and the witnessing effect, Journal of Personality and Social Psychology, 119(1), 40–74. https://doi.org/10.1037/pspi0000202

Averill

J. R.

(1980). On the paucity of positive emotions. In Assessment and modification of emotional behavior (pp. 7–45). Springer US.

Amato

J. A.

(1982). Guilt and gratitude: A study of the origins of contemporary conscience. Greenwood Press.

Bååth

(2014). Bayesian first aid: A package that implements Bayesian alternatives to the classical test functions in R. In Proceedings of useR! 2014 (p. 2). The R Foundation for Statistical Computing.

Bernard

Poncin

Bovet

Tamches

Cantin

Pralong

Borasio

G. D.

(2023). Giving and receiving thanks: a mixed methods pilot study of a gratitude intervention for palliative patients and their carers. BMC Palliative Care, 22(1), 1–13. https://doi.org/10.1186/s12904-023-01172-x

Blease

Kaptchuk

T. J.

Bernstein

M. H.

Mandl

K. D.

Halamka

J. D.

DesRoches

C. M.

(2019). Artificial intelligence and the future of primary care: Exploratory qualitative study of UK general practitioners' views. Journal of Medical Internet Research, 21(3), e12802.

Bürkner

P. C.

(2017). Brms: An R package for Bayesian multilevel models using Stan. Journal of statistical software, 80, 1-28. https://doi.org/10.18637/jss.v080.i01

Chong

Druckman

J. N.

(2012). Dynamics in mass communication effects research. In Semetko

H. L.

Scammell

(Eds.), The SAGE handbook of political communication (pp. 307–326). SAGE Publications.

10.

Cregg

D. R.

Cheavens

J. S.

(2021). Gratitude interventions: Effective self-help? A meta-analysis of the impact on symptoms of depression and anxiety. Journal of Happiness Studies, 22, 413–445. https://doi.org/10.1007/s10902-020-00236-6

11.

Cregg

D. R.

Musarra

Cheavens

J. S.

(2022). Grateful to help you or grateful to help me? An investigation of the effects of expressing gratitude to benefit oneself. Research Square. https://doi.org/10.21203/rs.3.rs-1875007/v1

12.

Coppock

McClellan

O. A.

(2019). Validating the demographic, political, psychological, and experimental results obtained from a new source of online survey respondents. Research & Politics, 6(1), 1–14. https://doi.org/10.1177/2053168018822174

13.

Doraiswamy

P. M.

Blease

Bodner

(2020). Artificial intelligence and the future of psychiatry: Insights from a global physician survey. Artificial Intelligence in Medicine, 102, 101753. https://doi.org/10.1016/j.artmed.2019.101753

14.

Emmons

R. A.

McCullough

M. E.

Tsang

J.-A.

(2003). The assessment of gratitude. In Lopez

S. J.

Snyder

C. R.

(Eds.), Positive psychological assessment: A handbook of models and measures (pp. 327–341). American Psychological Association. https://doi.org/10.1037/10612-021

15.

Fredrickson

B. L.

(1998). What good are positive emotions? Review of General Psychology, 2(3), 300–319. https://doi.org/10.1037/1089-2680.2.3.300

16.

Gaines

B. J.

Kuklinski

J. H.

Quirk

P. J.

(2007). The logic of the survey experiment reexamined. Political Analysis, 15(1), 1–20. https://doi.org/10.1093/pan/mpl008

17.

Gallup. (2023). Empowering workplace culture through recognition. Gallup World Headquarters.

18.

Hatch

S. G.

Goodman

Z. T.

Vowels

Hatch

H. D.

Brown

A. L.

Guttman

Bailey

R. J.

Esplin

C. R.

Harris

S. M.

(2025). Correction: When ELIZA meets therapists: A Turing test for the heart and mind. PLOS Mental Health, 2(8), e0000426.

19.

Henkel

A. P.

Čaić

Blaurock

Okan

(2020). Robotic transformative service research: deploying social robots for consumer well-being during COVID-19 and beyond. Journal of Service Management, 31(6), 1131–1148. https://doi.org/10.1108/JOSM-05-202-0145

20.

Huang

M. H.

Rust

R. T.

(2018). Artificial intelligence in service. Journal of Service Research, 21(2), 155–172. https://doi.org/10.1177/1094670517752459.

21.

Kenny

D. A.

(2020). Measuring Model Fit. https://davidakenny.net/cm/fit.htm

22.

Khadpe

Krishna

Fei-Fei

Hancock

J. T.

Bernstein

M. S.

(2020). Conceptual metaphors impact perceptions of human-AI collaboration, Proceedings of the ACM on Human-Computer Interaction, 4(163), 1–26. https://doi.org/10.1145/3415234

23.

Kinder

D. R.

Palfrey

T. R.

(Eds.). (1993). Experimental foundations of political science. University of Michigan Press.

24.

Locklear

L. F.

Sheridan

Kong

D. T.

(2022). Appreciating social science research on gratitude: an integrative review for organizational scholarship on gratitude in the workplace, Journal of Organizational Behavior, 44(2), 225–260. https://doi.org/10.1002/job/2624

25.

Loper

Bird

(2002). Nltk: The natural language toolkit. arXiv preprint cs/0205028.

26.

Lovell

Beckstrand

Cox

Thorpe

Chau

Noble

LaPine

Jameson

Tanner

Patterson

Dalessandro

Berry

Young

(2022). Community: The 2023 Global Culture Report. O.C. Tanner Institute.

27.

Lovell

Beckstrand

Cox

Thorpe

Chau

Noble

LaPine

Jameson

Tanner

Patterson

Dalessandro

Berry

Hatch

S. G.

Wymer

Topham

Johnson

(2023). Shift: The 2024 Global Culture Report. O.C. Tanner Institute.

28.

Lovell

Beckstrand

Cox

Thorpe

Chau

Noble

LaPine

Jameson

Tanner

Patterson

Dalessandro

Berry

Young

(2024). Generative Care: The 2025 Global Culture Report. O.C. Tanner Institute.

29.

Lovell

Beckstrand

Cox

Thorpe

Chau

Noble

LaPine

Jameson

Tanner

Patterson

Dalessandro

Berry

Young

(2025). Inspire: The 2026 Global Culture Report. O.C. Tanner Institute.

30.

McCullough

M. E.

(2002). Savoring life, past and present: Explaining what hope and gratitude share in common. Psychological Inquiry, 13(4), 302–304.

31.

McCullough

M. E.

Emmons

R. A.

Tsang

J. A.

(2002). The grateful disposition: A conceptual and empirical topography. Journal of Personality and Social Psychology, 82(1), 112–127. https://doi.org/10.1037/0022-3514.82.1.112

32.

Mongrain

Anselmo-Matthews

(2012). Do positive psychology exercises work? A replication of Seligman et al. Journal of Clinical Psychology, 68(4), 382–389. https://doi.org/10.1002/jclp.21839

33.

OpenAI. (2023). Models Documentation. https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo

34.

R Core Team. (2013). R: A language and environment for statistical computing.

35.

Rashid

Seligman

M. P.

(2018). Positive psychotherapy: Clinician manual. Oxford University Press.

36.

Schade

(2023). How ChatGPT and our language models are developed. https://help.openai.com/en/articles/7842364-how-chatgpt-and-our-language-models-are-developed

37.

Seligman

M. E. P.

Csikszentmihalyi

(2000). Positive psychology: An introduction. American Psychologist, 55(1), 5–14. https://doi.org/10.1037/0003-066X.55.1.5

38.

Seligman

M. E. P.

Steen

T. A.

Park

Peterson

(2005). Positive psychology progress: empirical validation of interventions. American Psychologist, 60(5), 410–421. https://doi.org/10.1037/0003-066X.60.5.410

39.

Sharma

Lin

I. W.

Miner

A. S.

Atkins

D. C.

Althoff

(2023). Human–AI collaboration enables more empathic conversations in text-based peer-to-peer mental health support. Nature Machine Intelligence, 5(1), 46–57. https://doi.org/10.1038/s42256-022-00593-2

40.

Sniderman

P. M.

Theriault

S. M.

(2004). The structure of political argument and the logic of issue framing. Studies in Public Opinion: Attitudes, Nonattitudes, Measurement Error, and Change, 3(3), 133–65.

41.

Snippe

Jeronimus

B. F.

aan het Rot

Bos

E. H.

de Jonge

Wichers

(2018). The reciprocity of prosocial behavior and positive affect in daily life. Journal of Personality, 86(2), 139–146. https://doi.org/10.1111/jopy.12299

42.

Sowa

Przegalinska

Ciechanowski

(2021). Cobots in knowledge work: Human-AI collaboration in managerial professions. Journal of Business Research, 125, 135–142. https://doi.org/10.1016/j.jbusres.2020.11.038

43.

Turing

A. M.

(1950). Computing machinery and intelligence (1950). Mind, 59(236), 33–60.

44.

Van Rossum

. (2007, June). Python programming language. In Proceedings of the USENIX annual technical conference (pp. 1–36). USENIX Association.

45.

Vaswani

Shazeer

Parmar

Uszkoreit

Jones

Gomez

A. N.

Kaiser

Ł.

Polosukhin

(2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998–6008.

46.

White

Hays

Spencer-Smith

Schmidt

D. C.

(2023). Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design. https://doi.org/10.48550/arXiv.2303.07839

47.

Zadeh

L. A.

(1965). Fuzzy sets. Information and Control, 8(3), 338–353.

Thanks a Bot: Leveraging Artificial Intelligence for Improved Workplace Appreciation

Abstract

Plain Language Summary

Keywords

Introduction

Expressions of Gratitude as a Starting Point—A Well-Supported Positive Psychology Intervention

Limitations and Gaps in the Current Literature

Research Aims and Hypotheses

Method

Transparency and Openness: Data Availability Statement

Power Analysis and Procedure

Prompt Engineering

Participants

Measures

Best Recognition Practice

Participant Learning and Behavior Change

Linguistic Measurement

Data Analysis

Results

Aim 1: Do Participants Believe the AI Coaching Aligns with HR Best Practice?

Aim 2: Are Participants Learning from the AI Coach, and Would it Change Their Behavior?

Aim 3: Do Receivers Perceive the AI-Generated or Human-Written Messages as Being More in Line with HR Best Practices?

Aim 4: Are There Length, Sentiment, Parts-of-Speech Differences Between the AI-Generated and Human-Written Messages?

Discussion

Discussion of Study Aims: Consistencies with the Current Literature

Strengths, Limitations, and Future Research Directions

A Note on Empathy, Other Fuzzy Constructs, and Metaphysics

Conclusion

Footnotes

ORCID iDs

Ethical Considerations

Funding

Declaration of Conflicting Interests

Data Availability Statement

References