Abstract
This paper introduces the Integrated Analysis Matrix (I-AM), a novel hybrid AI-assisted and human-led methodological approach towards the mindful integration, analysis, and interpretation of complex multimodal, mixed methods datasets – applied here to educational research, particularly where Game-based Learning (GBL) environments generate large volumes of entangled qualitative and quantitative data. Drawing on ludonarrative Design-based Research rooted in Sociocultural, Ecological Systems, and Complexity theories, the iterative development of the I-AM is
Keywords
Introduction
The Challenge of Multimodal Analysis and Analytical Frameworks
Educational research is increasingly grappling with multimodal and large-scale datasets (Cohn et al., 2024; Yan et al., 2024), particularly in Game-Based Learning (GBL) contexts where data arise from video or audio recordings, participant journaling or other forms/formats of writing, field notes, surveys, and more. The integration of multiple assessment domains (e.g., narrative, metacognitive, social interaction) creates substantial complexity in data collection, analysis and interpretation, requiring sophisticated methodological and analytical approaches to maintain scientific rigour and practical feasibility. Traditional mixed methods research tools, while robust, often require significant resources to triangulate data (Lameras & Arnab, 2021) and may not capture the dynamic interplay of quantitative metrics (e.g., lexical diversity, metacognitive scores) and qualitative insights (e.g., thematic, discourse, or narrative analyses), as pointed out by Gao et al. (2023) and later Yan et al. (2024). A persistent challenge is moving beyond merely combining different methods, to achieving true integration that produces insights exceeding the sum of their constituent parts. This often reflects deeper epistemological and theoretical limitations. The “problem of integration” (Bagnall et al., 2024, p. 12) in mixed-methods research, including describing analytical framing and generating meta-inferences, remains significant.
Within the learning sciences, the emergence of AI-driven data analysis has opened pathways for faster transcription, automated coding, and sentiment or content analysis. However, purely computational methods risk losing context and ignoring nuance (Davison et al., 2024). Conversely, purely qualitative approaches, while preserving nuance, can be impractically time-consuming (Zhang et al., 2024) for extensive datasets (Marshall & Naff, 2024). GBL’s effectiveness is often moderated by individual differences between learners, which complicates its application across diverse student populations. This has led to calls for further research (Di Mitri et al., 2024; Saqr & López-Pernas, 2024) to clarify how GBL can be tailored to maximise its benefits across different educational contexts. A critical gap is the absence of widely accepted, standardised instruments for measuring key developmental outcomes in TRPGs (Liapis & Denisova, 2023), such as narrative complexity, collaborative problem-solving, social-emotional learning, or player experience. This makes cross-study comparisons, replication, and evidence building difficult. Accordingly, there is a pressing need (Gao et al., 2023; Vogl, 2023) for a hybrid methodology (Turobov et al., 2024) that fuses the human sense of accuracy and interpretative richness with AI-driven efficiency.
A Hybrid Analytical Framework
The Integrated Analysis Matrix (I-AM) was conceptualised to address these challenges. Developed iteratively, I-AM offers: 1. 2. 3. 4.
This paper introduces the I-AM for discussion by means of illustration, applying it to a dataset (anonymised; depersonalised), focusing on one participant (codenamed GAG05B) from a Tabletop Role-Playing Game (TRPG) research study, as an example case within a case. Doing so allows me to demonstrate the volume and level of complexity that data generated by a single TRPG player can reach before even considering their fellow players.
Studying TRPGs in informal learning environments requires capturing multidimensional data – data entangled 1 in layers of social, emotional, cognitive, and narrative functions and contexts, etc. The impetus for I-AM emerged from the scale and heterogeneity of these data (including over 500 h of TRPG gameplay audio alone), along with the mixed nature of quantitative (e.g., readability scores) and qualitative (e.g., journal entries, gameplay transcripts) information that I needed to work through. The ‘origin study’, as it were, investigated the potential impact of TRPG play on the development of communicative meaning-making skills over time. GAG05B’s data offer a glimpse of that study’s treasure trove of entangled information.
Developing language and academic skills and managing processes for mastering content knowledge through augmented Tabletop Role-playing Game (TRPG 2 ) play introduces a unique set of complexities. Studying TRPGs includes studying the dynamic interweaving of social interactions, cognitive processes, emotional experiences, and multimodal communicative events. These all unfold within the shared imaginary world that players and their Game Master 3 (GM) – collectively referred to as players unless specified otherwise – co-create. This co-created game world is not just a backdrop, but an active environment shaped and developed over time through collaborative storytelling, role play, and improvisation that is deeply embedded in the players’ collective and individual ecologies of knowing and being. 4 Unlike traditional thematic matrices or multimodal coding protocols, the I-AM integrates structured AI-handled computational analysis with consistent human-centred interpretive validation, enabling rapid yet context-sensitive synthesis across transcript, observational, and other artefact datasets.
Background
Theoretical Frameworks and Lenses of Enquiry
TRPGs and other game-based learning activities fall under ludonarrative 5 contexts, in which “ludo” (gameplay mechanics, rules, decision-making) interweaves with “narrative” (storytelling, co-creative world-building). Such ludonarrative ecologies support meaning-making through multimodal semiotics and socially shared imagination mediated through collaborative, structured roleplay. Understanding these gameful interactions (Almås et al., 2023) requires a holistic, integrative lens bridging numeric patterns (e.g., turn-taking frequency, metacognitive survey scores) with nuanced, emergent participant narratives.
Rooted in Vygotsky’s sociocultural theory, learning is viewed as socially mediated (Chen, 2025), underscoring peer scaffolding. Ecological perspectives (Cowley, 2022; Thorne, 2012; Van Lier, 2000) extend this by considering the dynamic interaction between learners and their environment. While ecological perspectives highlight the interdependence of learners, tools and environments, Complexity Theory (Al-Hoorie et al., 2023; Panahi et al., 2023) recognises the dynamic and adaptive nature of learning systems. By situating TRPGs within a rhizomatic net of these frameworks, we see that cognitive, emotional, prosocial and motivational dimensions all interweave (Gascoine et al., 2017; Toraman et al., 2020). A methodology that integrates quantitative metrics (e.g., lexical diversity, readability scores, scores of metacognitive awareness indexes) and thick qualitative data (e.g., interview and gameplay transcripts, player journal entries), able to both differentiate and integrate varieties and sources of data, becomes vital for capturing the emergent dynamics of these playful ecologies.
Understanding Tabletop Role-Playing Games
For this paper, TRPGs are defined as social games where players gather to engage in episodic, collaborative storytelling of and as fantasy characters. An imaginary game world is co-created and developed through rules-based interactions between players, player characters, and a cast of non-player characters, creatures and environments as interpreted by designated players. In-game actions, activities, events and challenges are presented, refereed and resolved by a predetermined play-story authority, supported by the selected ruleset (game title) and agreed-upon amendments.
What distinguishes a Game Master (play-story authority) from other players in a TRPG is their role in ordering in-game events, co-interpreting and relaying the effects of actions in the game, often role-playing “Everyone Else in the World 6 ” and communicating the state of the game world at any particular point in time through some form of semiotic mediation (Arsenault, 2023).
Understanding Communicative Meaning-Making
Communicative meaning-making skills refer to the integrated knowledge, abilities, and strategies that enable effective engagement in communication, interpretation, and creation of meaning across various contexts. These skills are essential for effective participation in communicative contexts, adapting to changing semiotic landscapes, and are crucial for successful communication, learning, and personal and professional growth.
Traditional research tools and methodologies – while robust within their specific paradigms – seem to fall short when tasked with capturing the nuanced layers of multidimensional 7 data that emerge from such complex systems of meaning-making praxis 8 as TRPG play. Quantitative analytical tools, with their emphasis on measurability and generalisability, 9 may overlook the rich, intermingled, context-specific insights provided by multimodal and multilingual narrative data (Kumpulainen et al., 2021). Conversely, purely qualitative approaches, though adept at delving deep into individual experiences, may miss broader patterns in and relationships between data that quantitative analysis could reveal (Easterbrook et al., 2023). When it comes to studying TRPG play as described in this paper, existing mixed methods research struggles to effectively integrate and make sense of the complex, four-dimensional tapestries woven through shared imaginative sensemaking activities and practices (Kapitany et al., 2022; Morris, 2022; Weichold & Rucińska, 2022) that collectively constitute the TRPG experience as interrelated and embedded, fractal ecologies of communal (Giordano, 2022; Lave, 1991; Wenger, 1998) learning affordances engaged in time.
Using Ludonarrative Lenses
Ludonarrativity in TRPGs refers to the interplay between Gameplay Mechanics and Narrative 10 within a TRPG’s framework that facilitates learnful (Reinhardt & Han, 2021) outcomes. It specifically refers to how the rules, player decisions, and procedural elements of TRPGs (the ‘ludo’ aspect) synergise with collaborative storytelling, co-creative world-building, and character development (the ‘narrative’ aspect) to produce informal yet responsive, meaningful, context-sensitive and thereby “immersive” 11 learning experiences (Kronenberg, 2012, pp. 62–63). Integrating ludonarrative scaffolds into the I-AM’s configuration (i.e., mindful selection and matching up of tools and datasets, structured combinations of analysis, dynamically linked dashboards and interfaces) adds systems-based structure (Arnab et al., 2015; Gee & Gee, 2017; Sousa et al., 2023), in turn affording stable anchor points for cross-study comparisons, replication and continued evidence building.
Material and Methods: Seeking Structure
Focusing on the evolving landscape of Game-based Learning, particularly within the learning affordances of commercially available ‘over-the-counter’ games (Hogue, 2021) – here TRPGs – there is an imperative for research tools that navigate the interplay between quantitative/quantitised 12 data (metrics) and qualitative, multidimensional experiences (multimodal narratives). The challenge is not merely technical but conceptual, requiring a rethinking of data integration to holistically capture TRPGs as learning ecologies.
The Integrated Analysis Matrix (I-AM) was designed to ameliorate some of the limitations of traditional analytical approaches in integrated methods research (Plowright, 2011). The genesis of this interleaved analytical solution is rooted in a laboratory dialogue that critically examined existing methodologies, e.g., concurrent triangulation mixed-method design (Arias Valencia, 2022), revealing a gap in the analytical toolkit for researchers exploring the multifaceted phenomena encountered in TRPG studies. Investigating the impact of TRPG play on participant communicative meaning-making skills 13 resulted in a large collection of communicative artefacts. 14 These were analysed using quantitative metrics (readability scores, lexical analyses) and qualitative insights from thematic and conversation analyses alongside metacognitive awareness indices. With over 500 h of gameplay audio alone, however, a less time-consuming way to do more comprehensive analyses needed to be found.
Contextualising the Challenge
A Step-by-Step Map of the Iterative Phases That Constitute the I-AM
The multi-faceted recreational ecologies (Thorne, 2012; Van Lier, 2010) co-created through Tabletop Role-playing gameplay, characterised by their adaptability, complexity and dynamism, challenge the conventional boundaries of educational research methodologies.
The central catalyst for the development of I-AM was the need to both generalise (conceptually, as solution-seeking praxis, not empirically – small DBR study) and particularise: to understand broad patterns while preserving individual stories. This approach highlights the fractal qualities of gameplay and players, illuminating unique learning pathways in TRPG settings. This multiplicity pointed to a gap in existing methodologies – a lack of an integrated analytical framework capable of simultaneously harnessing the breadth of quantitative data and the depth of qualitative insights to navigate the apparent chaos of such complex systems. To illustrate the development of I-AM, I include a series of narrative glimpses into that trial-and-error journey, including some of the training/testing data used to see if I-AM was working (better). This process involved a single-participant Exact transcripts of GAG05B’s journal entries were tested for readability and complexity using standard, freely available online tools, with the resultant metrics (quantified data) being put through standard quantitative analytics alongside the relevant results from two metacognitive awareness monitoring tools. The same texts were then subjected to qualitative examination using the relational abilities of an AI chatbot. The idea was to see if there was a way to combine all these methods of analysis into one, and then test it to see if valid, reliable information can consistently be gained from it. [Related
Sources of GAG05B’s Collected Data, With Descriptions
Quantitative Data
Key Aspects Motivating the Incorporation of Quantitative Analytics in Ludonarrative Contexts
A Sample of GAG05B’s Gameplay Notes Showing a Detailed Journal Entry From 2023/09/20
To recap, the inclusion of quantified data in the proposed methodology is thought to be instrumental in providing a holistic view of the development of communicative meaning-making skills (Arias Valencia, 2022; Di Mitri et al., 2024), bridging the gap between objective measurement and subjective experience (Gierus et al., 2025) to foster a deeper understanding of learning in informal, storied (Gee, 2007, 2013, 2015) learning environments.
Some Shortcomings of Existing Quantitative Approaches in Ludonarrative Contexts
To illustrate, Table 5 contains the numeric (i.e., simplified) data obtained by running GAG05B’s transcribed journal texts through readability calculators and quantitative text-analysis formulae. Note that the scores calculated for the text in Figure 1 (typed up and presented verbatim in Text Box 1) are represented in Row 6 of Table 5. Jasmine walks towards the coridor and Zaphaar shouts out her name and dissapears. Jasmine hears him and she just went through the way. Jasmine sees two doors and and hears people talking together, and she tell everyone about it. And Thayer makes a decision to open the other door. And encounter three humans, two Orcks two Rakas which no one has heard of since their kingdom was distroyed. Thayer takes an action by using his quaterstarf and kills one of the humans by snapping his neck. He tries to attack the other human and breaks his neck to death. Galtar kills the ork that that Thayer and Jasmine had already hurt. Galtar gets hit by the human and gets hurt but not much. The human releases his rock and sends a message to spear that they have trouble down stairs. One of the Rakas tries to hit destine but misses and hits its friend the ork badly. Garax moves into the room and kills the ork with is mighty strong anurmed punch and kills the one ork then he uses his breath weapon and roasts the orc and the human. Thier tries to hit the human and he did not hit because of the humans armor. Aerdith uses his arrow and aims at the human and kills him. Jasmine tries to hit the ork and misses because of the armor. Will uses his spirit weapon moarning star hits the Raka. Destine uses his fire fist and kills the Raka by punching it. Thayer finds 3 gold, I Garax found 6 gold a greataxe a javeeline and a amulet of the Kahar. Will found a note in dwarfish. Thayer finds out that the note has the same stamp with the last. The letter says: The Raka have authority to negotiate for the release of six Tieflings into the Rakas custody. The usual compensation will apply. We go to the other room. Linguistic Metrics Calculated for GAG05B Writing Samples to Illustrate the Pitfalls of OversimplificationText Box 1. Verbatim Type-Up of GAG05B’s Gameplay Journal Notes for Session 7 Taken on 2023/08/30
The content of Table 5 represents eight different texts by one author. The various metrics calculated for each text create an impression of a writer with a fairly consistent lexical style (Lexical Density/Diversity scores) but wildly fluctuating markers of communicative contexts (LIWC-22 and Coh-Metrix results). However, reading these game journal texts (e.g., Text Box 1 above and Text Box 2 below) anchors the communicative context, raising new questions about the quantitative flattening of semiotic artefacts. Garax and Galtar and Zaphaar are up on the bay. Jasmine moves toward one of the coffins and finds thet contains Crakoloo and in her hand it has a starf (The python starf) which can turn into a snake when you command, and its only used by a Cleric, a Druid or a warlock. Thayer goes ahead and tries to open Kokies coffin and finds out there is a trap but he tripped it when he was trying to remove it and gets burnt by fire. In the coffin he finds old bones of a elf dressed in elvish clothes with a necklace that contains magic. Aerdith tried to open the Bards coffin and detects poison arround but fails to open it. and Thayer, will also fails but Jasmine comes to their rescue and they find a lute. The Lute is a magical and cast illusions. Destine opens the dwarfs coffin and finds a rod. Will cast a spell to speak with the dead and he asks them if they could help and they all say yes. Will ask them partially if he could get their weapons and they all grant them. Galtar and Garax start suspecting the water and they ask Zaphar if he knows anything and he detects magic. Thayer leads the others to another room and they see coffins, traps, and from far they spot a huge sword striked into a skull. Azdraka, an ancient dragon who was killed by the people who were burried in there. Jasmine moves toward the sword and inspects magic and tells Everydbody. Light bringer assumes that there is powerful magic occuring or coming to the island. Thayer downstairs opens one of the coffins in the room and finds a paper boat and a bottle containing black smoke. Galtar, while on his way to tell the others about what is happening dowstairs and ends up falling into a trap. Aerdith opens the othe coffin and find gogles that give you night vision and a bottle that releases unstopping watter. Thayer takes the sword out and finds out that kills dragon. Garax and the others upstairs detect that there is a telepotation incoming. Will downstairs gets hit by an unknown thing really bad. Garax goes and hide and so did Zaphar and Lightbringer. Thayer inspects and see a huge creature heading toward Galtar. Outside, while they are hiding, the teleport opens and a lady appears seeming to be light bringers friend and she is the Lore master named Karister. Downstairs one Will makes a spell and kills Two willow the wisps and one is still alive. Destine notice that the thing attacking them is an ait elemental. Then destine pour ink on Galtar. Aerdith opens the everlasting watter bottle to see where the creature steps with its footprint on the water. After a long round of fighting, Jasmine comes to the rescue by killing the element. Galtar tells everyone that the water is still and they all go up out. while going up we encounter Spear in his dragon form flying towards us. “The lore master and the life bringer see it fit to allighn with this group, lunch is served” he said with a surprise. Garax rages and tries to throw his Javeline but mises. Spear breaths in to attack but Kariste uses her protection. Thayer makes a decision to throw a his fireball necklace and he roasts everybody including Spear, and some are in conciousness. Zaphar Goes and bite and also he hits Zaphaar. Galtar uses his breath weapon and electricutes spear. Karister circles her hands twice and pushes the energy forward. And spear is now paralized as long as shee is concentrated. Light bringer heals everybody who is in need of healing. Spear breaks the spell. Jasmine uses her bow and arrow and hits spear twice. Galtar makes a decision to use his fist of unbroken air and hits. Galtar attacks with is axe on Spears leg and cuts his leg. Aerdith used his arrow and some spells and hits spear bad. Kurister makes a sykick laws spell and hits the dragon badly. Garax the great runs up and slashes him on his neck with his great axe. Spear gets angry and in draconic he says imprison while looking at Kurrister. Jasmine tries to shoot spear but misses and hits zaphaar. Will makes a Guiding bolt and spirit to hit and does. Destin punch's with his flaming fists and punctures its voice box. Galtar cut through the dragons scale and removes some skin. Aerdith shoots his arrow. Garax the great slashes the head of and kills the green dragon.Text Box 2. Verbatim Type-Up of GAG05B’s Gameplay Journal Notes for Session 9, Taken on 2023/09/20
More than numeric analysis, an integrative approach that combines quantitative metrics with qualitative insights is called for to do ludonarrative texts justice (Leon & Lipuma, 2024). Such a mixed-methods approach would not only leverage the strengths of quantitative analysis but also enrich the understanding of communicative meaning-making by incorporating the depth and contextuality afforded by qualitative research of ludonarrative ecologies (Glenhaber, 2022).
Qualitative Data
Key Challenges to Qualitative Analytics in Ludonarrative Contexts
A Diagrammatic Conceptualisation of I-AM’s Interleaved Structure and How Various Data Types Could be Mapped Against Each Other
Addressing these challenges necessitates a careful, methodical approach that leverages both the strengths of qualitative research and the support of technological tools for data management and analysis (Gao et al., 2023). It also underscores the importance of interdisciplinary collaboration (Borger et al., 2023), bringing together expertise from educational research, data science, and ludic design studies to navigate the complexities of multimodal data analysis.
Findings: Trial and Error
In light of the identified need for mindful, methodical solutions to the challenges identified (see Material and Methods: Seeking Structure), this section details the iterative development of the I-AM from its initial conceptualisation to its refined structure, emphasising how it is designed to capture, transform, analyse, and synthesise diverse data types for optimal interpretation. Consequently, the constituent subsections elaborate on the evolution and application of each ‘arm’ of integration: the numeric component (see Meeting the Challenge) and its responsibility to ensure that quantitative/quantitised data is analysed appropriately, and the I-AM’s narrative component (see Developing the Numeric Component of I-AM) with its similar responsibility towards qualitative/qualitised data, providing a transparent account of the analytical journey and demonstrating how this matrix serves as a robust instrument for navigating the complexities inherent in integrated methods (Plowright, 2011) research.
Meeting the Challenge
Recognising the challenges set out in Tables 4 and 6, I-AM is proposed as a prototype solution, designed to help TRPG researchers consider: 1. The data they want to collect, 2. How that data may be captured, 3. What data transformations might serve their research purposes, and 4. How their data could be analysed, integrated and synthesised for optimal interpretation.
It marries the computational prowess (Sufi, 2024) and natural language processing abilities of large multimodal language model (LMLM) generative pre-trained transformer (GPT) ‘artificial intelligence’ (AI) chatbots 16 with the nuanced, interpretive capabilities and accuracy requirements of human researchers to design, collect, capture, transform, process, analyse, and interpret complex datasets. This human-AI hybrid model, with their combined abilities to ‘read’ numeric and narrative data, facilitates a deeper exploration of the data available in TRPGs and play.
The I-AM is structured (Figure 2) to process and synthesise comprehensive datasets by categorising and analysing data across thematic dimensions (e.g., deductive codes) drawn from the study’s theoretical framework and corresponding to the granularity (depth and detail) of its datasets. The sequential application (from broad to narrower in scope) of Thematic-, Content-, Discourse-, and Conversation Analyses on narrative texts (i.e., narrative components of the I-AM design) coincides with the calculation and categorisation of said texts’ readability scores, linguistic metrics and recorded metacognitive awareness measurements (i.e., numeric components of the I-AM design). Anonymised data are organised alphabetically (codenames; modality; data type) and chronologically (yyyymmdd-protocol) to maintain consistency in analysis across different sources and data types, and to facilitate future inter-participant or temporal event analyses.
The evolution of this design is fuelled by a commitment to developing a framework that is not only adaptable and responsive to the specificities of one TRPG-based skills development study, but one that might also offer a model for broader applications in integrated methods research, addressing some of the shortcomings and concerns identified earlier in the paper (Di Mitri et al., 2024; Yan et al., 2024). By embracing the complexity of ludonarrative learning environments and the rich data they generate, I-AM represents the beginnings of research innovation aimed at bridging the gap between quantitative precision and qualitative richness beyond mere triangulation.
The journey of the Integrated Analysis Matrix through its iterations – from the initial I-AMi through the more refined I-AMiv – reflects an evolving understanding of the complex analytical needs inherent in mixed methods research. This progression is characterised by a deepening integration of numeric and narrative analytical components, revealing numeric data affinities for specific narrative codes. This interleaved approach to the design and (re)iteration of the matrix captures the multifaceted learning and development possible via TRPG play in informal learning environments.
Developing the Numeric Component of I-AM
The numeric component of I-AM, where/how quantitative and quantitised data analytics are integrated into I-AM (Figure 3), has undergone significant refinement across its iterations; each step driven by the goal to more effectively capture and analyse quantitative and quantitised data within the rich context of TRPG play. A Screenshot of the Metacognitive Awareness Section of the I-AM Numeric Component, Showing the (a) Thematic Breakdown Extracted From the Alignment Between (b) Jnr-MAI and (c) CHILD Items (cf. Figure 2) Alongside Its (d) Legend for Decoding Shading, Codes and Marks
I-AMi – The First Iteration
I-AMi
The inception of the numeric component focused on identifying key quantitative metrics that could provide insight into the cognitive and emotional developments of TRPG participants. 17 Initial metrics included readability scores (Gunning Fog Index, Flesch-Kincaid Grade Level) and linguistic analyses (Lexical Density, Lexical Diversity), aiming to offer a base understanding of the complexity and nuance in participant communication (i.e. Post-play interview transcripts, game journal texts, gameplay audio transcripts) (Supplemental Material).
Analysis Conducted
Baseline metrics for readability, lexical density, and lexical diversity were established based on participant writing (coded WRI).
Key Findings
Examples of How Quantitative Analytics are Complicateda by Idiosyncrasies (Punctuation) in GAG05B’s Writing
aHighlighted readability scores should not exceed 100%.
The WhatsApp Version of GAG05B’s PC Backstory. Analytics: Gunning Fog Index: 21.80 | Flesch-Kincaid Grade Level: 18.84
Challenges
Workarounds to avoid such miscalculations needed to be found. In this instance, the learner had provided two drafts of their Player Character backstory (a WhatsApp version as Figure 4 and the handwritten version included as Figure 5), which averaged out to more reasonable scores. The Coleman-Liau Readability Score and the FORCAST readability formula were included in the battery of tests as additional points for verification. A Photo of GAG05B’s First Draft PC Backstory. Analytics: Gunning Fog Index: 11.10 | Flesch-Kincaid Grade Level: 9.11 | Coleman-Liau Index: 5.83 | FORCAST Readability: 8.4 | Average Reading Level Consensus (Calc) = Grade Level 9
I-AMii – The Second Iteration
I-AMii
Building upon the foundational metrics, the second iteration introduced the incorporation of advanced linguistic tools such as the Coh-Metrix and LIWC-22. This expansion aimed to deepen the analysis by including measures of cognitive processing, emotional tone, and social dynamics. The introduction of metacognitive awareness tools (CHILD 3-5 for observational MAI data and Jr. MAI for self-reporting data) marked a significant step towards capturing a broader spectrum of learning dynamics, emphasising the role of More Knowledgeable Others and self-reflection in the learning process.
Analysis Conducted
The LIWC-22 and Coh-Metrix solutions were incorporated to track changes in qualitative and metacognitive expressions over time. Results of the Jr. MAI and CHILD 3–5 tools were included. Transcripts of one-on-one interviews (semi-structured, conducted in situ post-gameplay) were separated into Researcher and Participant Talk, whereafter Participant Talk texts (coded INT) were run through the same battery of tests as WRI texts.
Key Findings
Including the LIWC-22 and Coh-Metrix results revealed potential compatibilities with the results of the metacognitive awareness monitoring tools. The possibility of using the four MAI categories of the CHILD 3–5 observation items (i.e. Emotional, ProSocial, Cognitive, Motivational) as foundational inductive codes in qualitative analysis was noted.
Challenges
Aligning Jr. MAI (ver. B) items to the proposed codes (Figure 6) proved to be prone to subjective interpretation. Aligning Jr. MAI (ver. B) Items With Codes Lifted From CHILD 3–5 (Hence the Movement of Question Numbers on X-axis). GAG05B’s Scores Show More Stability in Cognitive Items
I-AMiii – The Third Iteration
I-AMiii
The third iteration represents a more sophisticated blend of scientific insight and skills, and AI chatbot capabilities (Gee & Zhang, 2024), enhancing the numeric component’s ability to process and categorise data efficiently. The use of the AI chatbot to pre-process and categorise quantitative metrics into the thematic areas – Emotional, ProSocial, Cognitive, and Motivational – allowed for a more nuanced integration of numeric data with qualitative insights.
Analysis Conducted
AI chatbot-assisted categorisation of lexical analyses and metacognitive awareness metrics. Selection and alignment of readability scores.
Key Findings
AI chatbot-assisted thematic categorisation required multiple iterations to ensure alignment with researcher-defined codes. While AI efficiently processed large data sets (68 numeric data points, a WRI game journal text, and an INT transcript per participant per week, plus 3–4.5 h of gameplay audio each week), iterative human verification, validation, and explanation (theoretical alignment) were necessary to refine outputs and maintain theoretical alignment.
Challenges
Managing the increased data complexity and ensuring alignment between diverse data sources posed significant analytical challenges.
I-AMiv – The Fourth Iteration
Analysis Conducted
Focusing on a holistic integration of all data dimensions, machine learning models were employed – using the AI chatbot’s deep learning capabilities – to identify patterns in the data that might predict learning trajectories.
Key Findings
Earlier inclusion of AI chatbot technology would have sped up the development process of the I-AM significantly. Three datasets (early measurements, mid-study measurements, late measurements) per person are insufficient for drawing reliable conclusions.
Challenges
The complexity and proliferation of machine learning models require comprehensive pre-training and constant revision to ensure the validity and reliability of AI chatbot recommendations and analytical contributions.
Developing the Narrative Component of I-AM
The narrative component of the I-AM design, where qualitative and qualitised data is incorporated into the matrix, reflects a parallel evolution to that of the I-AM’s numeric component, emphasising the depth and contextual understanding that qualitative analysis brings to mixed methods research.
I-AMi – The First Iteration
I-AMi
The initial focus was on establishing a systematic approach to thematic analysis (Figure 7), utilising the narrative data from game journals, interviews, and gameplay sessions. The aim was to identify emergent themes that provide a foundational layer of qualitative insights. Concept for the I-AMi Thematic Framework Development Sheet, Showing its Modular Aspirations to Allow for Swappable Tool “Blocks” and Compatibility Mapping
Analysis Conducted
A deductive coding framework (as identified during development of the Numeric Component of I-AM) was applied to GAG05B’s texts and interview transcripts (Figure 8) for Thematic Analysis. GAG05B’s Interview Section of an I-AM Worksheet After Gameplay on 2023/07/19 (‘Session Zero’ Reflective Voice Note), Including Scores From Both Metacognitive Awareness Tools for That Day
Key Findings
A refined (inductive) coding scheme emerged, i.e. intentional speech, consensus making, emotional well-being, friends, agency, immersion, transformation.
Challenges
Gameplay transcripts from only one recorder (best quality out of four) were impossible to analyse due to crosstalk, inaudibility and environmental noise.
I-AMii – The Second Iteration
I-AMii
Further development of the narrative component moved towards integrating Discourse Analysis, Narrative Analysis, and Conversation Analysis. This iteration aimed to explore meaning-making practices, individual and group narratives, and the dynamics of interaction within the TRPG setting. The application of these analyses provided a richer tapestry of qualitative data, revealing the complex interplay of language, identity, and social interaction in ludonarrative learning environments.
Analysis Conducted
A Demonstration of How I-AM’s Nested Approach to Qualitative Analysis Was Applied as Part of the Second I-AM Design Iteration
Example Passage
“What I liked mostly about today is the teamwork that, the teamwork that I did with GEG20B by roasting all the rats with our thunder, with our lightning breaths. The cooperation, I also liked the whole setting of the game, like how it’s playing, ja. It’s cool; I was very shocked when THG01B told me that people were dying, that they thought it was everlasting. It’s like imitation of real life but using your imagination.” – GAG05B, Reflective Voice Note – 2023/07/19 (see also Figure 8).
Key Findings
The identification of prosocial behaviours and their correlations with specific game events was improved. New observations on the motivational impacts of gameplay were made, e.g., the boost of identifying a successful gameplay strategy.
Challenges
Transcribing gameplay audio from backup recorders that were distributed around the room (including recorders that failed at different times) was done to capture more participant voice data. Matching these transcriptions to each other proved challenging; however, the main challenge of how to
I-AMiii – The Third Iteration
I-AMiii
I-AMii introduced a more refined process for developing and refining analytical themes through the sequential application of Thematic, Discourse, Narrative, and Conversation Analyses. By strictly examining what is necessary to add and develop the overall qualitative analysis, I-AMiii hoped to ensure a comprehensive exploration of the narrative component. The inclusion of graphical representations and diagrams should further enhance our ability to visualise and interpret the interconnections between themes and codes, offering researchers a nuanced understanding of the data.
Analysis Conducted
The LIWC-22 and Coh-Metrix solutions were incorporated to track changes in metacognitive expression over time.
Key Findings
An evolution of GAG05B’s strategic thinking was indicated, seemingly driven by specific game challenges related to teamwork. This highlighted the role of peer interactions in the shaping of their cognitive processes (see also Figure 9). LIWC-22 Data for GAG05B’s Writing Shows the Most Movement (Variation) in ‘Cognitive Processes’
Challenges
Managing the increased data complexity and ensuring alignment between diverse data sources posed significant analytical challenges, particularly in terms of categorisation. AI chatbot limits in terms of token-counts (i.e. how much chatting can be done in the same conversation/thread) currently cap sensible engagement. “Conversations” need to be planned per participant, unit of measurement or purpose to maximise thread efficiency. 18
I-AMiv – The Fourth Iteration
Analysis Conducted
All entries in GAG05B’s journal were transcribed and subjected to preliminary Thematic, Discourse, Narrative and Conversation analyses via the AI chatbot. 19 A manual review and refinement of the analyses were conducted in sequence (i.e. Thematic-, Discourse-, Narrative-, then Conversation Analysis).
Key Findings
Earlier inclusion of LMLM GPT technology would have sped up I-AM development. Chatbot neutrality 19 revealed nuanced researcher-participant bias that would have gone unnoticed otherwise.
Challenges
The complexity of machine learning models requires disciplined pre-training and consistent revision of a custom LMLM GPT. The rate of LMLM iteration, adjustment and proliferation – and the impact these may have on your active GPT model’s behaviour – necessitates consistent revision and testing of custom chatbots.
Integrating the Numeric and Narrative Components: I-AM (Almost There)
The iterative development of the numeric (Figure 3) and narrative (Figure 10) components signifies a commitment to crafting an analytical tool that respects the complexity of mixed methods research in the context of TRPGs and ludonarrative designs for learning. Each iteration has contributed to a more integrated and nuanced structure, capable of facilitating increasingly comprehensive insights into the processes of learning and development afforded by TRPG play. A Screenshot of the Qualitative Analyses Section of the I-AM Narrative Component Showing the (a) Thematic Breakdown (See Figure 3) Mapped Onto the Summary Analyses of (b) Qualitative and (c) Quantitative Metrics, Including a Coh-Metrix (d) Graph Vis-à-vis Captured Data Check Item, as Well as the Researcher’s (e) Notes Section for the Analysis of the Indicated Text
The culmination of the I-AM design process involves synthesising the numeric and narrative components developed through iterative revisions. This integration is the golden ticket, merging quantitative precision with qualitative depth to form a cohesive matrix. The convergence of these methodologies within the fourth iteration enables a robust analysis of TRPG-based learning environments, increasingly facilitating a holistic understanding of both the measurable and the multidimensional experiences (see Figure 11) of participating players such as GAG05B: 1. 2. 3. 4. I-AM Render of GAG05B’s Written Meaning-Making Journey Throughout the Study (Plotting Written Text Coh-Metrix & Lexical Style Metrics Against Metacognitive Awareness Measurements for Some Weeks) Shown From Two 3D Perspectives
By embedding compatible components within a unified framework, I-AM hopes to address the multifaceted nature of the data, ensuring that neither the richness of qualitative narratives nor the objectivity of quantitative metrics overshadows the other. This balanced approach allows for detailed, nuanced insights that can potentially inform educational practices and learner engagement strategies more effectively.
Discussion
Linking Findings to Theory
The development of communicative meaning-making in the TRPG environment is best understood through the synergistic lens of Sociocultural, Ecological-, and Complex Systems theories, which collectively move beyond a static representation to capture the dynamic and adaptive nature of learning. Drawing on sociocultural theory, GAG05B’s evolving communicative meaning-making highlights the role of peer scaffolding and collaborative imagination (i.e., engagement in Zones of Proximal Development/ZPDs) in shaping higher-order thinking (Almås et al., 2023). This process of interaction, as GAG05B’s development as player and Player Character within the game demonstrates, aligns with Vygotsky’s notion of internalisation (Vygotsky & Cole, 1978). Cognitive and linguistic structures, initially developed through social interaction with more capable others (e.g., fellow players, Game Master), are gradually integrated into the learner’s individual understanding.
This sociocultural engagement directly constituted and dynamically shaped the ecological space of the TRPG environment. From an ecological perspective, this environment – encompassing fellow players, the chosen ruleset, the shared physical space, the co-created imaginary world, participant agency, gaming artefacts, and emergent ephemera – formed a rich Learning Ecology (Damşa et al., 2019; Hellermann & Thorne, 2022; Van Lier, 2000). Within this actively growing and responsive ecology, language use and metacognitive development emerged not in isolation, but as a direct function of continuous, mediated social interactions. The I-AM foregrounds how these learning trajectories were not isolated cognitive events but were deeply embedded in and shaped by these nested ecological systems, allowing us to trace coherence between personal regulation and environmental scaffolding, supporting Bronfenbrenner’s systemic view.
Complexity Theory further illuminates the non-linear, emergent patterns within this dynamically evolving ecological system. As each play session introduced new shifts in ‘initial conditions’ – whether through new game challenges, evolving group dynamics, or novel narrative elements – the ecological perspective highlights how these factors could mitigate or amplify changes in language production. For instance, the social engagement, while fostering development, could at times overextend participants, potentially leading to temporary regressions to less accurate or more basic linguistic forms. The observed complexity in GAG05B’s gameplay interactions, writing, and interview feedback on playtesting and the day’s gameplay in general, and self-reflection thus aligns with Larsen-Freeman’s (2006) perspective on iterative, self-organising learning processes and the emergence of dynamic patterns. GAG05B’s meaning-making journey (Figure 11) is not discrete or phased; instead, it unfolds through fluctuating patterns of gradual and sudden changes, characterised by the emergence of new, distinct nodes of semiotic complexity. The I-AM, by mapping linguistic and cognitive shifts as pattern formations across modalities, operationalises complexity theory’s emphasis on interaction-dominant systems, revealing how meaningful, transformative learning emerges from high-agency, ludonarrative encounters within this adaptive, socially constructed ecology.
An Integrated Analysis Matrix: Addressing Core Methodological Challenges
I-AM represents a determined step towards methodological innovation in the study of Game-based Learning, particularly within the context of TRPGs. As highlighted in the introduction (see The challenge of multimodal analysis and analytical frameworks), educational research – especially within GBL – faces significant challenges in managing and integrating large-scale, multimodal datasets that combine quantitative metrics with qualitative insights. Traditional mixed-methods approaches often struggle to move beyond mere combination to achieve integration, falling short in capturing the dynamic and multifaceted interplay of cognitive, emotional, and social processes in complex learning environments. Furthermore, a critical gap exists in the absence of standardised, comprehensive instruments for measuring key developmental outcomes in TRPG research.
The I-AM was specifically conceptualised to address these persistent challenges. It stands as a novel hybrid AI-assisted and human-led approach that not only encapsulates the quantitative and qualitative dimensions of mixed-methods research data but also harmonises them in a way that amplifies their respective strengths by lowering the barrier to doing more. This matrix is uniquely tailored to explore the dynamic learning specific to TRPGs – grounded in a theoretical framework that weaves together Socio-Cultural Theory, Ecological Systems Theory and ecological perspectives on learning that tie in strands of Connectionism and Complexity Theory, allowing for a nuanced reconsideration of such concepts as Lave and Wenger’s (Boon, 2022) Communities of Practice (CoP), Legitimate Peripheral Participation (Giordano, 2022), Situated Cognition (Cunningham & Crandall, 2019), Embodied Learning (Bustamante et al., 2020) in Semiotic Domains (Bacalja et al., 2024), and contemporary ludonarrative perspectives. This deep integration is intended to allow researchers to capture the dynamic, intricate interplay of multidimensional cognitive, emotional, and social processes that single-researcher-centred, traditional mixed-method approaches were not designed for, providing a robust foundation for both theoretical advancement and practical application in educational research.
The manifest advantages of this human-AI hybrid model directly address the methodological shortcomings identified, moving beyond mere data triangulation to achieve greater integration and deeper insight: 1. 2. 3. 4.
Crucially, the development of the I-AM is deeply rooted in a Design-Based Research (DBR) framework (Radović et al., 2022). This situated application within a real-world TRPG study (see Developing the Numeric Component of I-AM and Developing the Narrative Component of I-AM) was the testing ground for the I-AM’s iterative evolution. Each cycle of data collection, AI-assisted preprocessing, human-led analysis, and subsequent interpretation informed the refinement and recalibration of the matrix and its associated AI tools. The value of this iterative process lies in its capacity to enhance both the accuracy of the analytical outputs and the theoretical alignment of the framework with the complex realities of ludonarrative learning.
This iterative integrity (Point 3, above) is further amplified by the DBR approach. Continuously testing and refining the I-AM in the context of actual TRPG gameplay and participant data ensures that the matrix remains responsive to the dynamic nature of the learning environment and theoretically sound in its capacity to capture emergent phenomena. The evolution of the I-AM through this situated, iterative process is a testament to the power of DBR in developing methodologies that are both theoretically principled and practically effective for navigating the complexities of educational research.
Concluding Remarks
The I-AM emerges from scrutinising the applicability of mixed methods research to the study of TRPG play. By drawing on theoretical frameworks that underscore the significance of play in learning and development, and co-opting the constantly developing deep learning capabilities of LMLM GPT technology, the I-AM stands as an example of the spirit of adventure to be found in the collaborative, transdisciplinary nature of contemporary design-based educational research. It embodies a response to the call for tools, methods, and methodologies that not only bridge the divide between quantitative and qualitative realms but also enhance the capacity of researchers to generate holistic, impactful insights into the multidimensional interweaving processes of learning and becoming.
With this, I propose the I-AM as a playtestable methodology for researchers seeking to navigate the complexities of ludonarrative or gameful studies in Education, offering glimpses of analyses that are not only reflective of the multidimensional nature of TRPG studies in Education, but possibly also adaptable to the broader fields of Education, Social Sciences and Humanities research.
Methodological Advancements
The development of I-AM marks a methodological advancement in the field of mixed methods research, especially for studies involving multilingual and/or multimodal ludonarrative elements in informal learning environments. The I-AM expands current methodological approaches by allowing for greater presentation and inclusion of complex datasets, moving mixed methods research beyond triangulation (where important data might need to be reduced to the level of background information) towards integration (where data complexity is foregrounded). This advancement is particularly significant in its ability to seamlessly integrate and interpret both high-volume quantitative data and rich qualitative narratives. The iterative testing and refinement of I-AM, as presented and encouraged, highlight its capacity to provide a more tuneable (in terms of granularity) understanding of player interactions, decision-making processes, and emotional engagements within TRPG sessions. These capabilities should not only push the boundaries of traditional research methodologies but also reveal new possibilities for in-depth studies in educational settings where complex social interactions play a crucial role. In short, by iterating the numeric (quantitative) and narrative (qualitative) dimensions within a single integrated matrix, I-AM offers an innovative approach to: 1. 2. 3.
Limitations
While the I-AM offers a robust integrative model, its application does require the researcher(s) to be familiar with at least narrative methods and the ethical navigation of participant agency and safety in co-constructed ecologies of meaning-making, such as TRPG play environments. The use of the I-AM further requires researcher access to digital infrastructure, explicitly some form of LMLM GPT AI chatbot technology (or better), as well as fluency in basic data-processing tools (e.g., Microsoft Excel).
With regards to the work reported on here, it is important to acknowledge that: 1. This paper serves as an 2. The participant data is drawn (with informed consent) from a 3. This paper 4. GPTs and related models are sure to
Practical Implications
The practical implications of I-AM are manifold – particularly for researchers looking to delve into the complexities of performed and storied Game-based Learning. By working this matrix, researchers may be able to more effectively manage and analyse large sets of diverse data types, e.g., from player performance metrics and quantitised discourse data to reflections on and recordings of gameplay experiences. This capability would facilitate a deeper exploration of how ludic and narrative elements combine to impact learning, offering insights that can guide the design of educational games and the implementation of Game-based Learning strategies. For educators and game designers, the insights garnered through I-AM could inform the creation of more engaging and educationally effective TRPG experiences, tailored to the needs and contexts of diverse learner populations. Examples hereof include: 1. 2. 3.
Future Directions (A Call to Adventure!)
As I-AM continues to evolve, there are ample opportunities for further research and development. Future directions include refining the matrix’s structure and algorithms to enhance data integration, expanding its applicability to other forms of educational technology, and exploring its effectiveness in different cultural and educational contexts. Additionally, ongoing collaboration across disciplines such as educational psychology, data science, and digital humanities could yield innovative adaptations and applications of the matrix. Such interdisciplinary work will not only improve I-AM’s utility but also contribute to its evolution as a dynamic tool for educational research.
Immediate considerations in our research trajectory include: 1. 2. 3.
In sum, the Integrated Analysis Matrix stands as a playtestable methodology for bridging quantitative and qualitative modes in complex, ludonarrative or gameful educational research. As TRPGs continue to flourish as informal learning ecologies, I-AM can help researchers and educators see the bigger picture of communicative, cognitive, and socioemotional development.
Researchers are encouraged to engage with I-AM, play with it, break it, remix it and adapt it to their specific needs – then share their findings and foster a collaborative effort to refine and expand its capabilities. This is as much a call for practitioners and theorists as it is a call for adventurers to find somewhere to be together and play a TRPG that tickles your fancy. I have a hunch that it’ll be worth it (in multiple dimensions). So… How do you want to do this?
20
Supplemental Material
Supplemental Material - Developing the Integrated Analysis Matrix (I-AM): A Data-Minding Approach for Better Ludonarrative Design-Based Research in Education
Supplemental Material for Developing the Integrated Analysis Matrix (I-AM): A Data-Minding Approach for Better Ludonarrative Design-Based Research in Education by Frederik Willem ‘Willie’ Matthys Knoetze in International Journal of Qualitative Methods
Footnotes
Acknowledgments
Great appreciation is expressed for the support of Olivier Potgieter in coding (Python) the 3D data visualisation cube script.
Ethical Considerations
The study was conducted in accordance with the Declaration of Helsinki (2013), and approved by the Institutional Ethics Committees of Stellenbosch University (Project ID: 27157; 19 June 2023) and Coventry University (Project Ref. Nr: P148405; 28 February 2023).
Consent to Participate
Informed consent was obtained from all subjects involved in the study in the form of written consent by Adults/Parents/Guardians and subsequent iterative verbal assent by Participants.
Consent for Publication
Informed consent was obtained from all subjects involved in the study in the form of written consent by Adults/Parents/Guardians and subsequent iterative verbal assent by Participants.
Author Contributions
The author is solely responsible for the work in its entirety; AND has approved the submitted version; AND agrees to be personally accountable therefore and for ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated, resolved, and documented in the literature.
Funding
The author received no financial support for the research, authorship, and/or publication of this article.
Declaration of Conflicting Interests
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Data Availability Statement
The data presented in this study are available on request from the corresponding author due to institutional embargoes on examinable manuscripts and data.
Supplemental Material
Supplemental material for this article is available online.
Notes
Appendix
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
