The seven roles of generative AI: Potential & pitfalls in combatting misinformation

Abstract

Misinformation can shape beliefs and undermine democracy, making effective mitigation essential. Generative AI (gen AI) presents risks and opportunities in this space—it can generate dubious content but also detect and counter misinformation at scale. AI’s ability to support and persuade people, facilitate discourse, and enhance media literacy underscores its potential. However, risks such as hallucinations, bias reinforcement, and manipulation highlight the need for responsible implementation.

In this paper, we explore gen AI’s role as an informer, guardian, persuader, integrator, collaborator, teacher, and playmaker, examining strengths, weaknesses, opportunities, and threats via strengths, weaknesses, opportunities, and threats (SWOT) analysis. For policymakers and technology leaders, we highlight the importance of regulations, transparency, human oversight, and AI literacy to ensure that gen AI can serve as a tool for truth rather than deception.

Some people have expressed substantial concerns about AI’s impact on the health of the information environment, whereas others have highlighted the potential benefits of AI-driven solutions to counter misinformation. This paper primarily focuses on generative AI (gen AI) systems such as large language models while recognizing the potential implications for other AI applications. Given that research on AI systems and misinformation is still in its infancy, we adopted a cautious analytical approach to summarizing the available evidence. Using an updated strengths, weaknesses, opportunities, and threats framework, ¹ we categorized well-supported findings as strengths or weaknesses and more speculative or emerging claims as opportunities or threats. Inspired by people’s tendency to anthropomorphize AI, ² we discuss these findings in relation to seven roles that gen AI may play in misinformation management: informer, guardian, persuader, integrator, collaborator, teacher, or playmaker. Each role has weaknesses and risks, necessitating clear recommendations to mitigate potential harm (more details in Table 1 and below).

Table 1.

The Seven Roles of AI in Misinformation Management

AI role	Strengths	Weaknesses	Opportunities	Threats	Recommendations
Informer: Retrieves, summarizes, and tailors information across sources and languages	Rapid access, tailored explanations, multilingual search	Bias, hallucinations, lack of source verification	Improve speed and search	Amplifies misinformation, skews information flows, and homogenizes knowledge; vulnerable to synthetic data loops and source obfuscation	Regulatory oversight, transparency, digital literacy
Guardian: Credibility guard that detects, flags, and helps verify suspicious claims and media	Real-time monitoring, detects misinformation, “neutral” fact-checking	Limited by training data, struggles with nuance, flags false positives	Cross-language fact-checking, human–AI validation	Potential misuse for political/economic influence	Bias checks, trusted oversight, critical engagement
Persuader: Tailored, dialogue-based gen AI to counter misperceptions	Persuasive at scale, counters conspiracy beliefs, enhances consensus	Risk of manipulation of humans, depends on training data biases	Personalized interventions, support (e.g., public health messaging)	Flooding of low-quality content, microtargeted persuasion	Regulate microtargeting, ensure source credibility
Integrator: Mediator that synthesizes views into balanced, common-ground summaries	Facilitates debate, synthesizes diverse perspectives	Susceptible to algorithmic bias, may reinforce misinformation	Support of democratic debate, improve decision-making	Can be used to manipulate for antidemocratic purposes	Regulatory oversight, human–AI collaboration
Collaborator: Copilot in inquiry, organizes sources and scaffolds questions	Enhances efficiency and supports research	Opaque reasoning creates illusion of understanding	Enhances information access, processing, and production	Users overestimate AI reliability; overreliance damages critical thinking	Encourage reflective use and AI literacy, ensure transparency
Teacher: Personalized tutoring with formative feedback on information evaluation and reasoning	Provides scalable support, rapid individual feedback	Bias, hallucinations, lack of contextual understanding	Support educators, improve access, refine comprehension	Passive, uncritical learning, reinforcement of laziness and mainstream views	Maintain human oversight, integrate AI literacy, restricted and reflective use
Playmaker: Maker of adaptive game scenarios to practice verification and prebunking	Creates interactive learning experiences, adaptive storytelling	Lacks creativity, reinforces stereotypes, requires human refinement	Educational games can boost critical thinking; automates game development, enhances accessibility	Biased content, ethical concerns, and copyright risks	Ensure fairness, human oversight in design

The Strengths, Weaknesses, Opportunities, & Threats of AI Roles

The following sections examine gen AI’s roles in greater detail as they relate to misinformation detection and dissemination. It is important to recognize that gen AI systems are shaped by the incentives, values, and preferences of the human actors who develop and engage with these systems. Developers may operate under commercial, reputational, and intellectual incentives and prioritize some AI functions, such as efficiency and novelty, over others, like moderation or safeguarding. Likewise, users bring their own motivations (for example, curiosity and efficiency seeking) and awareness of potential risks, which influence how AI is used in these contexts.

The seven roles described here are neither mutually exclusive nor exhaustive. Rather, they provide a heuristic framework for understanding how AI might function in complex information environments. Many strengths and weaknesses, such as risks of bias, manipulation, and overreliance, cut across roles, underscoring the need for integrated policy responses.

Informer

As an informer, gen AI serves as an on-demand explainer and synthesizer that helps users access, summarize, and compare information across sources and languages. In practice, this includes search-embedded assistants and retrieval-augmented systems that generate overviews, translations, and audience-appropriate explanations.

Strengths & Weaknesses

AI enables efficient multilingual searches across vast multimodal datasets.^{3 –8} AI systems can be accurate for factual queries, insofar as training data are accurate, and tailor explanations to different levels of complexity, enhancing accessibility. Yet, factuality is a challenge, as AI struggles with source evaluation and often generates information without proper verification.^{9 –11} Hallucinations remain an issue, alongside AI’s difficulties with recognizing and correcting its own errors. ¹² Biases embedded in training data and fine-tuning processes can amplify stereotypes and skewed perspectives. ¹³ Many commercial models lack transparency regarding their training data and reasoning. Susceptibility to intentional or unintentional prompt manipulation exacerbates misinformation risks.^14,15

Opportunities

AI integration with complementary tools could improve search accuracy, such as deploying localized AI models on personal devices to incorporate relevant context information. AI systems could help people find credible news. ¹⁶ Human-in-the-loop systems can support expertise, ethical judgment, and nuanced decision-making in contexts where AI alone might fall short.^{17 –20}

Threats

AI lacks true expertise and may rely on dubious sources and false narratives, especially where data voids exist.^{21 –26} AI can create illusory understandings while cementing misconceptions ²⁷ and perpetuating biases.^28,29 Perceiving AI systems as authoritative or unbiased could increase susceptibility to flawed content.^{30 –32}

Control over AI systems confers significant power over information flows, raising concerns about manipulative or biased influence. Ethical and legal risks complicate AI deployment, with proprietary models potentially exposing copyrighted or personally identifiable content.^{7,33 –35} Low-quality AI content threatens search reliability as models increasingly reference synthetic data, ³⁶ and there is a risk of feedback loops leading to homogenization of the knowledge universe. ³⁷

Recommendations

These threats highlight the need for regulatory oversight, transparency, and responsible use. Regulatory frameworks should discourage models that promote unreliable information by mandating responsible AI development and deployment. ³⁸ There is precedent for regulatory frameworks that seek to achieve this. For instance, the European Commission’s General-Purpose AI Code of Practice ³⁹ suggests safety and security model reports, which require statements on misinformation spread and the likelihood of hallucination. More specifically, it advocates that model reports include a detailed description of the architecture, capabilities, propensities, and affordances of the AI model, as well as transparency about how it has been developed, including its training method and data, and how it differs from other models available on the market. ³⁹

Quality control safeguards, including credibility-based ranking algorithms, human-in-the-loop systems, and regular bias checks, are required to mitigate misinformation amplification. ⁴⁰ Transparency should be prioritized, including disclosing training-data sources, labeling AI-generated content, and facilitating independent audits and public scrutiny. Legal protections against the misuse of personal data and copyrighted material by AI models could help ensure ethical AI deployment. Digital literacy initiatives promoting effective AI use, with equitable access, are required.

Guardian

As a guardian, gen AI supports the detection, triage, and verification of dubious content by classifying claims, matching them to evidence, and surfacing credibility cues. Examples include automated fact-checking pipelines, AI-assisted content moderation, manipulated-image/deepfake detection, and browser plug-ins that warn users about likely misinformation.

Strengths & Weaknesses

AI systems can rapidly analyze information, monitor platforms in real time^41,42 and produce convincing countermessaging,^43,44 facilitating the identification and correction of emerging misinformation.^30,40,45,46 Nonetheless, AI’s potential as a guardian is limited by training data due to blind spots in certain areas and trouble detecting new misleading tactics. It has a limited capacity to verify photos ⁴⁷ and also struggles to recognize misinformation that is subtle or contains logical fallacies.^{48 –50} AI may erroneously flag accurate content as it lacks cultural, contextual, and historical awareness and understanding of literary devices such as sarcasm.^51,52

Opportunities

As AI systems are largely language agnostic, they can be used to automatically detect and fact-check misinformation across countries and domains through browser plug-ins or online platforms.^53,54 This may be optimized through human-in-the-loop systems, with fact-checkers validating AI assessments. AI-generated corrections of political misinformation may be particularly useful, as they can be perceived as neutral, thereby reducing motivated reasoning.^{55 –57}

Threats

Organizations could use fact-checking systems to benefit their political or economic interests. Reliance on AI systems to detect and correct misinformation could evoke a false sense of security and reduce critical analysis. Although AI systems are often described as language agnostic, their performance may actually vary across languages and cultural contexts due to uneven availability of high-quality training data. This is important to note since it creates disparities in misinformation detection capabilities, particularly in underresourced linguistic domains.

Recommendations

Regulation could require platforms to conduct regular bias checks and apply system debiasing where necessary. ⁴⁰ Trusted organizations should invest in AI capabilities for combating misinformation and employ initiatives without being deterred by concocted debates about freedom of speech and truth arbiters, as misinformation can often be objectively and reliably identified. ⁵⁸ To avoid blind trust of AI systems, educational interventions that foster critical engagement must be developed.

Persuader

As a persuader, gen AI delivers dialogue-based messaging intended to correct misperceptions. Applications include chatbots against conspiracy theories, targeted protruth messaging that highlights credible sources and scientific consensus, and conversational agents that scaffold reflective reasoning about claims.

Strengths & Weaknesses

AI systems can rapidly synthesize diverse knowledge sources to generate persuasive messages at scale,^26,43,59,60 even in domains perceived as hallmarks of human interaction, such as social dilemma situations requiring expressions of empathy. ⁴³ As a persuader, AI can help counter conspiracy beliefs through tailored, evidence-based conversations. ⁶¹ However, people may show reactance to AI-generated persuasion in some contexts. ⁴³ AI biases and sensitivity to prompt engineering, combined with its limited ability to recognize its own limitations, can lead to highly persuasive yet misleading messages.^{21 –25}

Opportunities

The persuader could amplify accurate information in important domains, including public health, and reinforce perceptions of consensus to serve the public interest. AI can also produce personalized content to enhance persuasion.^{60,62 –64}

Threats

Large-scale content generation may be used to “flood the zone” with low-quality information. Persuasive messages, especially if microtargeted, present a substantive threat if used by malicious actors against vulnerable populations^63,64 and can go so far as to threaten democracy. ⁶⁵ Low AI literacy may also increase susceptibility to persuasive attacks due to uncritical receptivity. ⁶⁶

Recommendations

Given that persuasion depends on multiple factors, including source credibility,^67,68 message repetition,^69,70 perceived consensus,^71,72 communication channel,^73,74 and level of transmission, ⁷⁵ information sources must be made explicit. However, source transparency alone is likely insufficient^24,26,63 and must therefore be accompanied by appropriate regulation of microtargeting in sensitive domains such as elections.

Integrator

As an integrator, gen AI mediates discussion by synthesizing diverse viewpoints, mapping areas of agreement and disagreement, and drafting balanced summaries to support deliberation. For example, AI mediators can assist citizens’ panels and online forums by generating neutral briefs of issues and structured option comparisons.

Strengths & Weaknesses

The integrator may facilitate democratic deliberation by helping individuals find common ground in divisive political debates.^76,77 While humans show bounded rationality and biased reasoning,^{77 –81} AI can rapidly integrate vast amounts of data across languages and modalities, identifying patterns and trends. However, data integration can be skewed by biases in training data, algorithm design, fine-tuning, and feedback loops.^{20 –25}

Opportunities

AI-generated summaries of debates can be perceived as superior and more balanced than human-mediator statements, which, at scale, may be able to support democratic deliberation. ⁷⁶ Better information synthesis may provide a foundation for improved policymaking.

Threats

Integration algorithms could be manipulated to encourage preordained outcomes by malign actors and facilitate antidemocratic discourse. Aggregated opinions may exert normative pressure on people to conform, ⁸² and dissenting voices may be inappropriately suppressed.

Recommendations

Regulatory oversight may be required to ensure careful engineering and deployment of AI integrators. For instance, AI mediators may play a role in advisory and decision-making bodies, such as focus groups and citizens’ assemblies, where it could be required that human mediators oversee this process, rather than being replaced by AI alternatives. ⁸³ For AI tools to serve as effective mediators, they must also be fine-tuned to integrate minority voices and perspectives. Policymakers should therefore balance the benefits of consensus-building with protections for legitimate minority positions.

Collaborator

As a collaborator, gen AI functions as a copilot for inquiry, helping users formulate questions, locate and organize sources, outline analyses, and iteratively reflect on evidence quality. Use cases include literature mapping and metacognitive prompts that scaffold systematic source evaluation.

Strengths & Weaknesses

AI systems can be useful as guides, coaches, and research assistants,^84,85 including in the context of information evaluation. However, AI systems often operate as black boxes, making it difficult to assess how they classify information or detect misinformation. ⁸⁵ The collaborator’s reliability depends on the availability of sufficient high-quality data and precise instructions. Quick and well-formulated AI responses can create a false sense of accuracy, leading users to accept misinformation without scrutiny^{86 –88} and overestimate their knowledge.^27,85

Opportunities

In academic and investigative settings, AI could assist by identifying patterns and expanding access to diverse sources of information.^28,89,90 AI-driven tools could enhance metacognitive skills by supporting interactive learning processes and boosting reflection on information quality.^85,91

Threats

The effectiveness of AI collaborators remains largely untested in the context of misinformation. Risks of misdirection exist, particularly for users with limited knowledge and AI vigilance.^85,92,93 Overreliance on the collaborator may reduce users’ cognitive skills^{94 –96} and their ability to critically assess information, leading to skill erosion over time.^97,98 Furthermore, AI systems can have a tendency to feed users content that aligns with their preexisting beliefs rather than challenging them. ⁹ The cognitive demands of critically assessing AI-generated outputs may also lead to cognitive overload in users unfamiliar with the topic area and misinformation-detection strategies. ⁸⁵

Recommendations

AI–human collaboration must be approached with critical reflection. AI should act as a copilot rather than a truth arbiter replacing critical human judgment, encouraging users to evaluate its outputs rather than accept them uncritically. AI literacy programs should emphasize AI vigilance and metacognitive support.^85,99,100 AI collaboration must be transparent, ensuring users understand its strengths and limitations.^85,101

Teacher

As a teacher, gen AI provides scalable, formative feedback and guidance that can personalize practice on information evaluation and reasoning tasks. Examples include AI tutors that critique drafts, rehearse fact-checking strategies, and give hints aligned with curricular goals and assessment rubrics.

Strengths & Weaknesses

AI may be able to facilitate education against misinformation, but the evidence available to date is scarce and mixed. ¹⁰² Key weaknesses are that AI lacks contextual understanding, soft skills, and ethical reasoning (for example, misinterpreting complex arguments or satire), making it unreliable as a teacher.^2,96

Opportunities

AI offers scalability and rapid and individual feedback.^{103 –105} Virtual agents could support media literacy,^{106 –108} and AI tools can simplify complex concepts and refine comprehension. ¹⁰⁹ With improved reasoning transparency, AI may enhance students’ ability to engage with information critically. ¹¹⁰

Threats

AI in education risks fostering passive learning, namely reliance on AI-generated answers rather than active task engagement,^97,111 which may lead to decline in cognitive skills. AI can reinforce mainstream perspectives over innovative ideas, ¹¹² subtly making students more susceptible to biased narratives.

Recommendations

AI should serve as a teaching aid rather than a substitute for educators, maintaining a human-in-the-loop approach and a human-centered mindset in education.^113,114 Implementation in schools must be accompanied by AI literacy programs that teach critical evaluation, bias recognition, and effective prompting.^{85,103,115,116} AI-driven manipulation, emotional profiling, and social scoring in schools should be prohibited. ¹¹⁷ Educational institutions should also establish guidelines for AI use that emphasize assessment of the learning process, as well as its outcomes, to encourage students to use AI as an effective learning companion rather than a shortcut.

Playmaker

As a playmaker, gen AI enables adaptive, game-based learning by generating scenarios and feedback that allow learners to rehearse counter misinformation skills. Use cases include AI-assisted serious games and mixed-reality simulations in which learners practice the navigation of misleading information.

Strengths & Weaknesses

Serious games have shown potential to foster fact-checking skills and critical thinking.^{118 –121} But AI systems struggle with logical reasoning and consistency, leading to unpredictable content generation in live gameplay. ¹²² Storytelling can be negatively affected by group stereotypes, ²⁹ AI’s tendency to lose track of long-term interactions due to limited memory, ¹²³ and lack of creativity and depth.

Opportunities

AI can streamline game development by automating level design, narrative generation, and character creation, making serious game development more feasible.^{115 –117} Additionally, AI can make games more accessible through real-time translation and adaptive difficulty.^{124 –127} AI-assisted simulations could also help train educators to counter conspiracy theories in classrooms. ¹⁰⁷

Threats

Overreliance on the playmaker risks producing generic, biased, and unoriginal content, reducing the effectiveness of serious games in fostering critical thinking.^128,129 Ethical and legal concerns remain unresolved, including displacement of human designers, concerns related to AI-driven monetization and potential manipulation of players, and copyright issues.

Recommendations

AI-generated game content must prioritize accuracy, fairness, and inclusivity. ¹²⁶ Developers should ensure human oversight in AI-assisted game design to mitigate bias and maintain ethical standards. Players should be made aware of algorithmic, cultural, and economic biases embedded in AI-generated content.

Conclusion

Research at the nexus of AI, misinformation, and human behavior is still emerging. Studies often lack long-term validation and rigorous controls or rely on small samples, self-report measures, and short-term assessments. More rigorous studies, including randomized controlled trials, are needed to better understand AI’s strengths, weaknesses, opportunities, and threats.

Overall, AI systems must be used in ways to maximize benefits while mitigating risks. Human oversight is essential to ensure that AI systems complement rather than replace human judgement. Transparency in AI content generation and dissemination is needed to prevent misuse. AI’s persuasive capabilities should be monitored and designed to encourage reflection rather than passive acceptance. Education initiatives, in general but especially in educational settings such as schools, should promote AI literacy, equip users with skills to assess AI outputs critically, and hinder the reinforcement of biases and stereotypes.