A Mixed Methods Approach to Analyzing Embodied Interaction: The Potentials of Integrated Mixed Methods Analysis of Video Interaction Data

Abstract

Video has become a widespread tool for capturing naturalistic behavioral data. While mixed methods show great potential in understanding the active nature of children’s interaction, only a few studies have developed mixed methods for video-based interaction research. This paper presents a mixed methods embodied interaction model appropriate for studying complex embodied interaction and draws on methodological insights from a mixed methods project to capture changes in children’s interaction with touchscreen technologies. The paper details how integrated mixed methods analysis was needed to fully explain the results and how mono-method studies would provide misleading results. The theoretical advancements generated during the project illustrate how mixed methods for video-based studies have valuable properties for understanding complex interactional phenomena, such as learning.

Introduction

Interaction is at the very fundament of life and central to being and developing as a human, where learning from others and the world of knowledge, we cumulate by being a cultural animal (Tomasello, 2019). This is not least true for children, where interaction is the key underlying many developmental and learning processes. Interaction between people and things can also be a messy study area, as many factors influence it, ranging from the psychological to the social to organizational frames for how people conduct themselves. Mixed methods show great potential for uncovering children’s mobile and active nature, for example, by integrating positional and observational data from children (Bhuyan & Zhang, 2020; Walker et al., 2009). Mixed methods have, however, been developed less as a research strategy for examining the embodied nature of direct interactional data between children, peers, and adults. Here, potential developments lie as the traditional fields concerned with studying interaction are battling the challenges arising from the complex datasets that naturalistic video recordings generate.

Video data has, with the accessibility, affordability, and mobility of cameras, become a staple tool for data collection. Video data provides a rich set of data points for researchers interested in studying human interaction in detail that is inherently multimodal, that is, including both verbal and non-verbal actions as meaningful parts of communication. The rich nature of video data, as it captures behavior in context, lends itself to mixed methods inquiry, as the data is open to an array of interpretational possibilities. In educational settings, video has been used for both qualitative and quantitative analytical purposes (Jacobs et al., 1999). Wall et al. (2013) argue that understanding the values of qualitative and quantitative approaches when merging these in mixed methods designs can have great potential for understanding learning. However, as DeCuir-Gunby et al. (2011) critically noted, there is a lack of mixed methods approaches to analyzing video data in educational settings, and they moreover point to a need to develop integrated mixed methods approaches to video analysis.

The purpose underlying integrated mixed methods analysis is to integrate qualitative and qualitative analysis that reveals something that goes beyond a mono-method approach to analysis (Bazeley & Kemp, 2012; Fetters & Freshwater, 2015). This is an important principle and poses a potentially fruitful agenda for video-based interaction studies. However, the practice of analytical integration of qualitative and quantitative data has been challenging (Vogl, 2019), and this is additionally more complicated with the added richness from video data, as layers of interactional complexity are added from video recordings that include the entire embodied repertoire of behavior of participants and their interactions with an often technologically enhanced environment. Likewise, with a mixed methods undertaking, the analytical complexity increases, adding additional data sources and/or possible interpretations into the research process.

To illustrate the virtues of mixed methods analysis, this paper draws on a project examining how digital touchscreens are reshaping traditional forms of interactions in preschool, such as shared book reading (Samuelsson et al., 2022a), children’s play (Samuelsson et al., 2022b), and childhood computational literacy (Samuelsson, 2023), drawing on a methodological framework of studying children’s interactions as embodied, integrating qualitative and quantitative aspects.

Today, touchscreen technologies are ubiquitously available across settings, often also for very young children (Rideout & Robb, 2020). Touchscreens and their use in early childhood are still relatively new and highly contested phenomena (Hassinger-Das et al., 2020). There are pediatric concerns about young children’s screen use (American Academy of Pediatrics, 2011), for example, concerning adverse effects on children’s language learning from screens (Kuhl et al., 2003), and later developmental delays associated with early screen time (Takahashi et al., 2023). However, there are also nuances, not least from studies pointing out the relative role of social interaction and screen use showed how situations with touchscreens have different effects depending on the amount of interactivity that the social and digital environment affords (Hirsh-Pasek et al., 2015; Roseberry et al., 2014). It is thus of concern to examine how interaction with touchscreens unfold, as there seems to be a large interactional dispersion between children’s solitary engagement with screens and the screen used in social constellations (Roseberry et al., 2014). From this background, Zosh et al. (2023) provided a recent call for diverse methodological development to understand better the details of interaction concerning children and screens. The methodological tools developed in this paper provide one of many ways for this type of interaction to be understood.

This paper aims to show an integrated mixed methods analytical approach developed and how it revealed empirical and theoretical insights about the changing nature of child–adult interactions in the digital transformation of shared book readings. Furthermore, it is outlined how the project outcomes would not have been possible to reveal from a sole qualitative or quantitative approach. Studies have pointed to how mixed methods provide perspectives from both qualitative and quantitative angles (DeCuir-Gunby et al., 2011; Johnson & Onwuegbuzie, 2004), for example, when adding qualitative voices to a larger survey in an explanatory sequential design (cf. Creswell & Plano Clark, 2018) but more fully integrated approaches are also possible. This paper details how the mixed methods approach became transformational in understanding the phenomena under study, showcasing the synergistic (Hall & Howard, 2008) result of the mixed methods approach, which proved highly informative for understanding interaction reconfigured in contemporary technological settings. The paper also exposes the key role of mixed methods for studying embodied interaction, that is, interaction that includes both verbal and non-verbal forms of interaction with both the physical and social environment.

The following sections show the affordances of visual and video methods, how they are employed in mixed methods studies, and how they transform interactional studies. It discusses how video data’s embodied and ecologically sensitive nature has added difficulty to interaction studies and how a mixed methods approach addresses some of these challenges. It is then exemplified by the details of the project examining children’s touchscreen interaction and the mixed methods methodological view of embodied interaction that was developed following the project’s integrational challenges.

Visual Methods, Video Recordings, and the Prospects of Mixed Methods Analysis

Several fields concerned with human action and interaction have been turning toward human conduct’s visual and embodied character and how it can be studied (Pink, 2015). One such approach has foregrounded the multimodality of visual representation and used this as a key way of analytically approaching visual phenomenon, breaking visual modes into interconnected parts made to make meaning (Jewitt & Kress, 2003; Kress, 2009). Applied to human interaction, which is inherently multimodal (Finnegan, 2002), we communicate using ensembles of action, gestures, talk, and interaction with the environment and effectively draw on such resources when we make sense of the world and interact with each other (Goodwin, 2000). A similar turn has been the case of ethnographical fieldwork, where a set of approaches and technological advances such as affordable and inobtrusive cameras have made fieldwork more moveable (Ingold, 2011), and increasingly concerned with the visual (Pink, 2006, 2015) and embodied interaction inevitably captured by video recordings.

Video recordings offer new ways of looking at interaction. In qualitative micro-analysis, the works of Charles and Marjorie Goodwin have been influential, showing how understanding is orchestrated from cooperative actions in a range of settings from advanced professional practices such as archeology (Goodwin, 2018) to the schoolyard (Goodwin, 1990). In this line of qualitative inquiry, video has enabled an “embodied turn” (Mondada, 2016), where interaction has increasingly been seen as multimodal (Streeck et al., 2011). Video has here offered what Heath et al. (2010, p. 3) call “a realignment akin to the effect of the microscope on biology” by its ability to study how interaction unfolds on the micro level and how people and the in-situ properties of everyday technologies make this possible.

A dividing line in studies of interaction has been the difference in approaches between studies from a quantitative approach, often prioritizing randomized and elicited forms of data, and qualitative studies favoring naturalistic data from real-life settings (Mondada, 2012). Specifically, the foci of these approaches become radically different, whereas qualitative and conversational analytic approaches closely track and examine interaction as turn-by-turn unfolding. Quantitative studies, on the other hand, draw from the overall patterns of interaction and the phenomena under study. Here, a mixed methods approach can be potentially beneficial if we see interaction as both an embodied and cooperative phenomenon and a discontinuous part of ever-changing environments that are part of technologically infused interaction. This is also challenging, as scholars have noted the inherent trickiness of integrating mixed methods in studies concerning visual material (Shannon-Baker & Edwards, 2018). However, a joint methodological consideration has the promise of capturing the multifaceted character potentiated by rich interactional data.

Embodied Interaction and the Challenges of Multimodality

One successful approach to analyzing interaction has been conversation analysis (CA). In CA, intricate transcription techniques have been developed to closely examine the turn-by-turn unfolding of interaction. In this approach, interaction is fundamentally seen as a sequential phenomenon. A proposal is followed by a response, or a verbal question is followed by a verbal answer, for example. Traditionally, these studies of interaction have been less concerned with embodied actions and put less emphasis on environmental scaffolds, such as artifacts (for important exceptions, see early works of Goodwin). This has, however, changed in recent years following a multimodal and embodied turn in CA, following more frequent and intricate use of video in studies (Mondada, 2012). Here, a key challenge has been how to modulate the detailed transcription devices of CA to capture multimodal details (Mondada, 2016) and how increased reliance on technological devices forms interaction (Heath & Luff, 2012). Following the emic perspective, the underlying conversation analysis focus has remained on the turn-by-turn unfolding of interaction at the expense of longitudinal patterns of changing interaction. Some commentators (see Seedhouse, 2022) have pointed to how this misses complexities inherent in the processes that build up educational interactions. Learning is shaped not only in the moment-by-moment interaction but also across time and sometimes at unexpected locations. This poses a considerable methodological challenge to educational studies as learning builds up from iterated embodied interaction over time, and this is a phenomenon of direct interaction also unfolding over time in a learning process. This paper points to how some of these complexities stem from the embodied and environmentally coupled nature of interaction and how mixed methods analysis is ideal for untangling the details by tracking both micro and macro levels of interaction.

Embodied Interaction, Learning, and the Role of a Mixed Methods Undertaking

In studies of interaction and learning, theories putting weight on the embodiment of human psychology have steadily been gaining ground (Clark, 1999). Central to these theories is that human interaction is fundamentally seen as a dynamical coupling of people’s minds and the physical world (McGann et al., 2013), placing the fundamental analysis point on the relational feature of interaction (Overton & Lerner, 2012). This has moved theoretical interest and analytical concern from the notion of individual minds (or brains) to interaction with the social and physical environment and how learning emerges in this relation. Here, the role of artifacts for interaction and learning takes foremost relevance and stresses the point of studying complex psychological phenomena such as learning as a situated phenomenon where people cooperatively use the environments and artifacts in complex interaction (Hutchins, 1995). Here, the relevance of video-based methods becomes a critical tool because they capture individual human actions in their natural context and the unfolding interaction between people and things. This also pushes the boundaries of interaction, including the moment-by-moment unfolding, the interaction with environments, and the tools often critically used in modern technological settings, such as workplaces and educational settings (Heath & Luff, 2012). From this vantage point, learning as an embodied and situated phenomenon can be thoroughly studied, and this opens methodological opportunities for the technological “realignment” (cf. Heath et al., 2010, p. 3) of video-based studies in the educational area studied during the project.

In the following section, the paper describes the methodological process developed during a video-based project on preschool education comparing shared touchscreen interactions with traditional shared book-reading scenarios and the theoretical and methodological insights that the mixed methods generated. This highlights the importance of using integrated qualitative and quantitative data to analyze human interaction. Interaction data, such as video recordings of interactions between children and adults in this report, are open-ended and embodied. Therefore, video datasets of some proportion are open to mixed methods designs as they can be analyzed quantitatively and qualitatively. A mixed methods design can be carried out when one part of a data collection is deemed inadequate for a project, but also as a methodological and analytical technique that aims at wholistic understanding of a complex phenomenon (cf. Creswell & Plano Clark, 2018), such as the types of interaction studied in this case. At the heart of the mixed methods undertaking is the complementarity of mixed data and analysis of this. Here, it is detailed how the project’s initial core mixed methods analytical design ran into problems of integration, which led to theoretical insights that needed to be developed to encompass integrative views of embodied interaction unfolding in situ in a real-life learning setting.

Project Summary

This project centered on children’s embodied communication and learning and was inspired by multimodal and video-based ethnography (Dicks et al., 2006; Pink, 2015) as a main data collection method. The video recordings have been combined with child observational study techniques (Pellegrini et al., 2013), aiming to detail the overall context where children interact with teachers during shared reading sessions or the digital counterpart with interactive books on an iPad touchscreen tablet.

From the outset, the project can be categorized using Creswell and Plano Clark’s (2018) description of convergent mixed methods design during a case study, the case here being the changing shared reading practices at an early childhood center. In this way, comparable episodes of interaction could be sampled for mixed methods analysis based on ecologically valid data, being based on two, in this context, naturally occurring types of activities. From this approach, it was possible to study patterns of children’s embodied behavior quantitatively and the multimodal interaction turn by turn qualitatively, aiming to converge these results.

Participants

The preschool department was located outside Stockholm, Sweden, with nine children, two pedagogues, and one teacher assistant. The child participants were 2-year-olds (M = 28.4 months, SD = 2.7) at the project start. The preschool was visited three times a week for 2 months, where relevant activities were recorded.

Data Collection

The project primarily used video-based collection methods and other techniques of multimodal ethnography (Dicks et al., 2006). Multimodal ethnography heavily relies on visual information in the data collection process, often involving video and/or other visual materials such as photographs or artifact collection. This project uses multimodal ethnography to collect data from multimodal interactions in shared reading and shared interactive touchscreen sessions in their institutional and interactional context. In short, the multimodal data collection methods were used to capture the embodied nature of interaction and also relate this to the particular context. Data collection captured the multimodal interactions of children and teachers through video recordings of the preschool activities and shared sessions, supplemented by field notes and photographs.

Sampling and Unit of Analysis

The projects examined multimodal interaction between children, teachers, and books or the touchscreen tablet iPad. Comparable episodes were selected where teachers and children shared attention over the book or touchscreen, directed toward interaction in this immediate space. The unit of analysis is episodes of shared attention (Tomasello, 2019) between children, teachers, and a touchscreen or book. When selecting episodes, criteria relevant to the embodied framework underlying the research process have been considered. For example, only episodes where children and teachers are bodily oriented toward the artifact under use (the book or iPad) in a manner typical for a traditional shared book-reading scenario were chosen for analysis for comparability. Figure 1 displays examples of shared interaction in the reading and iPad scenarios.

Figure 1.

Examples of the sampled interaction sessions: (a) shows the shared book reading with two children and (b) displays a similar orientation but with the interactive iPad application.

The recordings allowed analysis of how multimodal communication, comprised of verbal and non-verbal actions such as gestures and touch actions, comes together in interaction. Sampled data relevant to the project was primarily 11.5 hours of video recordings of which relevant episodes from the video that corresponded with the above criteria were used, that is, episodes similar to those embodied configurations shown in Figure 1. The recordings were supplemented with visual and text-based ethnographic data, with over 200 photographs and field notes from the shared reading and touchscreen activities.

Analytical Procedures

The analysis utilized a mixed methods analytical approach to capture both the frequencies of child and teacher actions and qualitative moment-by-moment qualities of multimodal interaction in the shared book-reading or interactive touchscreen scenarios.

An annotation system in ELAN created for the project was used (see Samuelsson et al., 2022a for annotation system and supplementary files). Both verbal and non-verbal actions from children and teachers were coded, including touch actions to the screen, such as pressing and swiping motions to the touchscreen surface. Thus, the codes covered embodied actions children and teachers used during interaction. The bracketed coding offered in ELAN allowed coding continuous actions from two or several participants, thus capturing an interweaving set of actions performed in the sessions.

Quantitative Analysis

After coding, the total number of actions for a coded sequence could be calculated and divided by the sequence’s time. This created the category of actions per minute, or APM, a per-minute average of actions calculated comparatively for the touchscreen and book-reading scenarios. Moreover, a subset of coded behaviors was divided into verbal actions, comprising talk per minute or TPM, and non-verbal actions per minute or OPM. This analysis showed the overall quantitative action patterns during the project for the touchscreen and book-reading scenarios.

Qualitative Multimodal Interaction Analysis

To further understand interaction in multimodal detail, we turned to the previously discussed techniques of multimodal interaction (Goodwin, 2018; Streeck, 2009). At this stage, the analysis was focused on the turn-by-turn cooperative interactional achievements common to this analytical tradition. Following the sampled sessions, exemplified in Figure 1, a close analysis of how the participants interacted on a turn-by-turn basis could be performed. In this tradition, means of embodied interaction, such as gesture and touch actions, are seen as means of action and communication. Select episodes were chosen and carefully transcribed for verbal, bodily actions and interaction with the environment following a variation of Streeck (2009).

Convergent Analysis

Comparing the quantitative analysis results and the qualitative unfolding of interaction, the project at this stage ran into the problem of integration, as the quantitative and qualitative results, at first, pointed in different directions. These results are described in the following section, showing how this integrational step forced theoretical and analytical development during the project.

Project Findings and Developing an Integrated Embodied Interaction Analysis

The quantitative analyses of the project showed that teachers’ interaction patterns held a relatively stable pattern between traditional shared reading sessions and the shared digital tablet sessions, unlike the teachers, who had a similar frequency of action in the two scenarios (see Samuelsson et al., 2022a). However, children’s action patterns changed dramatically between the two scenarios. While the total number of actions (APM) was comparable between the traditional (8.1) and the digital scenarios (8.7), children’s verbal language (TPM) decreased in interactions featuring the tablet (TPM 2.1), compared to the traditional book (TPM 5.4), instead making way for an increased number of touchscreen actions (OPM). Figure 2 summarizes the action patterns for children from the project, calculated into per-minute averages in the two scenarios). There was thus a considerable shift toward non-verbal actions (OPM) that were 2.7 in the shared book-reading scenarios and 6.6 during the iPad interactions.

Figure 2.

Summarized children’s embodied action analysis results, adapted from Samuelsson et al. (2022a).

A conclusion that could have been drawn from this quantitative analysis, seen in Figure 2, would be that children communicate less when using the tablet, seeing how the action patterns can be described as almost reversed regarding the children’s TPM actions. However, the qualitative part of the project pointed to a more complex understanding of embodied interaction, and the following section details these discoveries.

Qualitative Embodied Action Analysis

Analyzing the coded sessions from ELAN and going into qualitative detail of the sessions, interesting patterns emerged. Drawing from the coded video data, it became clear that the decrease in verbal action was not a failure of children to respond to the teacher. Comparatively, a text-based, turn-by-turn analysis of the situation would entail, for example, that a question that is not responded to breaks some of the standard interaction conventions. However, following the multimodal analysis of these turns, it became clear that interactions with the touchscreen tablet prompted touch-action responses from children. The presses and screen-directed movements were not done separately from the ongoing verbal action of teachers but were part of how children communicated and responded in the touchscreen scenarios. Needed was an integrated form of analysis that included the features of embodied interaction that the video-based method afforded.

Developing an Embodied Model of Interaction From Mixed Methods Insights

To further explain the data, what Fetters and Molina-Azorin (2017, p. 301) refer to as “merged interpretation” was required to encompass the results thoroughly. In the book-reading scenarios, children would respond to a teacher with words or sometimes only with a gesture (quantitatively shown in the TPM of 5.4 and OPM of 2.7). This pattern fits how a traditional shared book-reading session is expected to progress. However, in the touchscreen scenarios, children responded communicatively with teachers through touchscreen actions, notably here consisting of a large majority of children’s actions with an OPM of 6.6 and TPM of 2.1. However, our conclusion from the integrated analyses would not be that children communicate less in the touchscreen scenarios (as the quantitative analysis would entail) but that children communicate differently through other means of communication.

The insights from the project led us to believe that new theoretical and methodological devices were needed to fully understand the results, encompassing what people say and what they do with the technologies that afford new ways of action and interaction. We took an embodied and action-based perspective that could better encompass the non-verbal actions of children (McGann et al., 2013; Streeck et al., 2011) to develop a view of interaction that more fully incorporated both types of communicative actions observed. In this approach, more concern is given to the dynamic interaction between participants and the artifacts and environments they act in (Di Paolo et al., 2018). The model developed from the merged mixed methods analysis could now show how participants’ embodied actions shaped the overall quantifiable patterns of interaction.

Integrating Theories of Interaction through Mixed Methods Insights

Rather than the standard conceptualization of interaction studies, where interaction builds on turns of interaction, we could see how single interaction moves shaped the conditions for the following turns. A major insight was how touchscreen action to the iPad was communicative and how these could stand in for verbal responses in the iPad scenario where a comparable episode in the book reading would entail a verbal response. The clearest example of this was that children sometimes responded to questions from the teachers with a touchscreen action that, in a similar book-reading scenario, would elicit a verbal answer, thus not breaking the interaction but showing a new type of pattern afforded by the new technology present.

We went on to develop a concept of a cyclical model of action and interaction, drawing from the embodied theoretical lens and the insights from merging and trying to integrate the mixed methods results. From this theoretical device, actions not only communicate but also actively change the conditions for further turns of interaction. The model for interaction developed during the project is illustrated in Figure 3, drawing from the role of the body in communication from the embodiment literature and the role of environmental affordances as a major step in unfolding interactional turns.

Figure 3.

Illustration of the theoretical model for interaction developed and used from the integrated mixed methods approach.

Figure 3 explains how a single action (communicational or embodied) can interact with and alter the environment, which also changes the possibilities for future action. The new model developed during the project better encompassed the overall (quantitative) results covering how communication changed and the detail of each turn (qualitative) where interaction could be altered by actions performed. Instead of seeing interactions as one-action turns, for example, one person does or communicates something, and the other party responds, action and interaction were now conceptualized as cycles. A cyclical model of interaction can be explained through:

1. An actor performs an action. This can be saying or communicating through gesture and/or performing an embodied action, such as pressing a function on the touchscreen.

2. This action affects the environment, the tool under use, and/or interaction with others.

3. The action alters the tool and environment, which can provide new information for social interaction or new affordances of the tool.

4. The action turn has now opened new action potentials for the next turn of action.

This model worked better to provide a merged interpretation of the data. The integrated approach explained the overall pattern of children’s changing verbal actions and why this interaction is being reshaped with new technologies; for example, as a touchscreen action provides a new informational change to the environment, it also can be seen as communicative in a way more consistent with the data.

The new model of interaction also better helped explain discontinuous change and how interaction drastically could change in touchscreen interaction scenarios, making use of the integrated mixed analyses. The model of interaction helped explain both changing quantitative patterns of interaction and the discontinuous interaction that sometimes was in the close qualitative multimodal interaction analyses. For example, a child pressing a new application (in step 1) could change the pedagogical situation drastically (stage 2), altering the pedagogical interaction (stage 3), changing the situation from a literacy event to an arts-based interaction, thus opening a completely new set of potentials (stage 4).

The integrated mixed methods pointed to the overall interaction pattern and the power of people’s actions, particularly equipped with devices where a single press can open an array of options as the touchscreen and its applications did. This challenges some standard interaction conventions, where talk is often seen as the primary driver of conversation. The project reveals how a set of actions considerably changed interaction dynamics, seeing interaction as embodied cycles of action and interaction.

Adding this interactional complexity to the project, the mixed methods steps are outlined below. Following this additional step toward integration, the overall research process is outlined in Figure 4, which now contains an additional step to the standard core design flowchart (cf. Creswell & Plano Clark, 2018), adding the step of theoretical development for generating integrated results.

Figure 4.

A flowchart illustrating the research process, analytical steps, integrational problems, and solutions.

The mixed methods methodological approach to the video data provided insight into children’s actions and the changes in interaction in the digital age. This also points to an important insight for studies of interaction, as while single actions often follow the predicted pattern, some actions do not, and one action can radically alter what happens next. This points to the importance of using mixed methods approaches to study interaction, especially in populations such as this, where the methodological devices can capture children’s full range of expressive means and the discontinuous nature of interactions underlying educational interaction today. Moreover, considerable debate has surrounded the role of screens and what they do with children’s language development (Kuhl et al., 2003) and social interaction (Roseberry et al., 2014). The model here provides methodological tools to capture the changes to interactional phenomena that come with touchscreens.

The theoretical insights gained in the project would not have followed if the project had been divided into two single mono-method studies, and thus crucially speak to mixed methods as a key to triangulating interactional data and corroborating mono-methodological findings. The case speaks to the explanatory power of mixed methods for analyzing video data and the importance of integrated human action and interaction analyses.

Discussion

The paper has argued for a mixed methods approach to understanding human interaction as captured by video data. It has drawn from a project examining the changing nature of children’s actions and interaction with new technologies, showing how the mixed methods approach forwarded new perspectives on a complicated multimodal dataset. The integrated approach provided a view of child–adult interaction that went beyond the findings yielded from a mono-method approach to analysis. Crucially, it has been argued that this integration stage could only be reached following theoretical development fueled by the mixed methods integrative approach that included both the interactional turn-by-turn details from the qualitative studies and the overarching interactional patterns from the quantitative approach. Together, these formed insights into how embodied interaction unfolds that could be used to track children’s action patterns during the technological change at the preschool studied in the project.

Moreover, the perspective of interaction resulting from the mixed methods undertaking and integration revealed how interaction with technological devices such as touchscreens is part of interactional transformations, and in this specific case, how this challenges some of the norms of interaction when viewed as an embodied phenomenon, analyzed multimodally for both its quantitative patterns and its qualitative discontinuous nature. This leads to important results in understanding contemporary interaction and how educational child–teacher interaction can take new forms. Moreover, it adds critical details for understanding contemporary educational interactions and has been used to capture the changing nature of interaction. In the particular case (detailed in Samuelsson et al., 2022a), the changing nature of interaction moved children’s interactional output toward non-verbal behavior. This is of considerable concern for language learning initiatives in early childhood contexts that are increasingly digitalized. The project also points to the need for methodological development regarding children and screen use (Zosh et al., 2023), the role of mixed methods undertaking, and the particular application for understanding the embodied nature of changing interactional settings. While several pediatric studies have pointed to the association between screen time and delayed developmental and communicational outcomes (Takahashi et al., 2023), we can, from this framework, point to some possible interactional reasons why this may be. Moreover, the understanding of interaction involving screens gained from the interactional framework can also be used to act on the communicational affordances of the situation. For example, understanding that children’s touchscreen actions can be communicational actions can provide understanding for parents, educators, and other professionals to continue the conversation and move the interaction toward verbal-language and face-to-face interaction if this is the goal.

A fundamental insight from the project and the mixed methods analytical framework developed was how any single action performed could greatly alter the forthcoming interaction, for example, by changing the artifact under use and altering the type of activity, showing the unfolding dynamics of interaction the importance of integrating both qualitative and quantitative multimodal data. For example, in interactive tablet readings, changing the app with a simple press could work as a means of communication and alter the educational situation entirely (see Samuelsson et al., 2022a). The project showed how interaction follows regular and irregular patterns, as some actions radically reshape the character of ongoing activities. A core of the embodied perspective is how our actions alter the world around us (Clark, 1999; Di Paolo et al., 2018), and this can fruitfully be captured both quantitatively and qualitatively. However, the changing patterns of interaction also showcased new forms of embodied touchscreen interaction, where verbal language plays a decreased part. Here lies power in the interactional model presented, as data of these new forms of interaction can be informative for pedagogues and other settings where adults interact with children and touchscreen devices are involved. Understanding the intricacies of touchscreen interactions can provide empirical evidence of when they might be used or when not to use them, and also, when used, how they transform the nature of interaction.

While interaction in the project mostly followed the expected patterns, humans also act in unpredictable ways. The mixed methods framework lets us capture both these aspects of the nature of human interaction. This core realization is telling because, as humans, we sometimes want to act in unexpected ways. This was shown in the project’s later studies examining children’s play, where a child’s whim could radically change the play theme and interaction dynamics between children. Likewise, using a toy or touchscreen in a playful, unpredicted way shifted children’s play encounters (see Samuelsson et al., 2022b). Mixed methods are ideal for capturing both continuous and discontinuous interaction patterns and tracking the complex change that technologies are bringing to contemporary settings featuring children.

The theory of interaction developed during the project is at the core of what it is to be human and has considerable bearing on how we learn, develop, and alter the social and cultural setting where we live (Tomasello, 2019). Doing things affects not only the moment-to-moment interaction but may also alter our context, which may ripple into effects on the learning environment. This also speaks to strategies merging qualitative research, which can combinedly capture smaller samples in detail and the richness of interactive contexts, with quantitative research’s ability to capture the fundamental patterns of interaction. Mixed methods can unify the interaction levels underpinning learning interactions (Seedhouse, 2022), and this paper has pointed out the key role of a mixed methods undertaking of this.

Contribution to the Field of Mixed Methods

The paper presents an integrated mixed methods embodied interactional analytical technique and provides a crucial step in studies of video-based interaction data. This was exemplified in the project where the integrated mixed methods analysis uncovered crucial aspects of the interaction studied. In hindsight, it is methodologically telling that the project would not have revealed the insights it did following just one line of data analysis (cf. Fetters & Freshwater, 2015). Not only did the mixed methods approach provide a more compelling picture but it is also striking that a single-method study of the same material may have provided a misconstrued conception of what was happening. Voicing the notion of Onwuegbuzie (2012) that there is a fruitful middle-way in methodological approach, the project shows how such a middle-way provides a holistic picture of human actions and how embodied actions are key parts in forming how the interaction unfolds, today fueled by the extension of digital technologies being part of these unfolding interactions. This provides a crucial argument for integrated mixed methods research utilizing video data from real-life interactions as these capture embodied interactions unfolding in contexts that may be shifting as part of these actions. Mixed methods approaches here provide a powerful package that can track these aspects of learning and interaction and work to explain these in an integrated way.

The research process illustrates the virtues of mixed methods in this area. The literature on integrating analytical strategies in mixed methods research, such as Bazeley’s (2018) integration points, can also be complemented by generating theoretical insights. Such theoretical models can iteratively feed into the project itself or spur new research questions for following studies, furthering the claim of Vogl (2019) that 1 + 1 can equal 3 in mixed methods inquiry. Integrating qualitative and quantitative findings provided a conceptual reshaping of the understanding of interaction, giving new possibilities for interpreting the results. This generated the new theoretical perspective shown in Figure 3, explaining the, at first, surprising dispersion between qualitative and quantitative results drawing from the theoretical and methodological development of an embodied model for interaction and analysis. The theoretical perspective then, in turn, led to explanatory power and new insights in the project, showing how the sometimes complicated integration of visual data, noted by Shannon-Baker and Edwards (2018), can yield results and propel further theoretical and methodological developments. In this project, the integrational steps described were vital in creating an understanding of the surprising results in ways that could fathom a more holistic understanding of the technological change in the cases studied.

The overall research process and steps, outlined in Figure 4, show the methodological steps of analysis, integration, and theoretical development during the project and the sometimes laborious efforts needed to approach integrative understanding. This abductive nature of mixed methods research, going in between theoretical and empirical understanding, should be further explored as a research strategy and methodological option, underscoring Bazeley’s (2018) point of how qualitative and quantitative data interact in integrated analysis. This can be seen as informative for interaction research, today often studying complex forms of interaction involving people cooperating around advanced technologies (cf. Goodwin, 2018). Critically, the integrated view shows how this entails a sometimes radical change to children’s action patterns, shifting how the interaction unfolds in situ with new technological affordances (see Samuelsson et al., 2022a, 2022b). Video data is key to helping us capture this, and multimodal mixed methods analysis of embodied behavior tracks the complexities and constantly shifting nature of these interactions.

The project vouches for integrating mixed methods analysis of video-based human action and interaction studies. Many approaches to examining interaction have long been centered on questions and techniques concerned with micro-level analyses and studying interaction as unfolding turn by turn (for example, in CA). The project shows how a mixed methods approach can fruitfully boost a micro-level focus on interaction, taking from interaction’s fundamentally embodied and simultaneously patterned and non-patterned nature. Further studies should assess, for example, how technologies shape the affordances for action and its effects on emerging action patterns. Our analytical approach can inspire new uses and similar methodological developments.

Limitations

While promising to battle the methodological complexities brought on by interaction’s increasingly visual and digital nature, the current approach presented in this paper carries crucial limitations. The proposed method and view of interaction are relatively novel and thus have not yet been largely validated across any critical mass of studies and not in other settings than the early educational settings studied in the project. More studies can fruitfully use the method and steps provided here and by Samuelsson et al. (2022a) on other educational settings and places undergoing technological change.

Moreover, the messy form of data generated through multimodal data collection and the embodied analytical method also introduces complexity to the project, which may be unnecessary in some cases. However, the insights generated in the project can be convincing for other researchers to undertake this challenge. The approach presented answers to the hard problems posed by multimodal and visual analyses of video data (Heath et al., 2010) and shows the strenuous but worthwhile methodological challenge posed and how it can be necessary to understand contemporary interaction fully.

Conclusion

This paper reported on the utility of a mixed methods approach for understanding the technological reshaping of children’s interaction. Reflecting on the overall results of the project, it is clear that an integrated mixed methods approach yielded insights into the project that would otherwise have been missed or even misinterpreted. This has shown the importance of adopting mixed methods approaches to fields concerned with children’s actions and interaction, such as education and interaction design studies. Combining the embodied view of interaction and using mixed methods analysis significantly impacted how we understand interaction amid technological change.

The mixed methods approach provided critical theoretical insights about human interaction by seeing it as a simultaneously patterned and unpredictable phenomenon. Human beings and the interaction between us is a complex area of study. However, integrating modes of video data analysis can track some of this complexity and provide a well-rounded picture of this complex phenomenon. Mixed methods offer a way forward for understanding unfolding embodied interaction, thus giving insights into the building blocks of how we interact and learn with each other in the shifting contemporary context.