Rhythm in literary apps

Abstract

This article addresses how rhythm may function in literary apps. The article has two aims: increasing the knowledge of how literary apps work as texts, by exploring their aspects of rhythm, and developing the understanding of the theoretical term of rhythm. The authors propose a rhythmanalysis in which two different types of rhythm – reading rhythm and narrative rhythm – are taken into account. The two types of rhythm may both occur at different structural levels in the text. This approach is applied to the analysis of rhythm in the popular literary app, Florence (Wong et al., 2018, Florence Tablet application software), drawing on concepts from multimodal social semiotics (Van Leeuwen, Introducing Social Semiotics, 2005), although leaning towards a more reception-oriented approach than the traditional text-oriented analysis in social semiotics. Literary apps are defined in this context as multimodal fictional narratives that can lead to an aesthetic experience for the reader (Iser, 1984, Der Akt des Lesens); however, non-narrative apps, such as poetry, may also be defined as literary apps. These apps may be read on a tablet or a smartphone. This article elucidates some of the many facets of rhythm related to the multimodal design of a literary app, which invites different forms of interactivity than the linear reading and page-turning of print-based picture books. The findings of the analysis show how rhythm not only contributes to the multimodal cohesional aspects of literary apps, but is fundamental to the meaning potential of the literary app.

This article has two aims: first, to increase knowledge about how literary apps work as texts by exploring their aspects of rhythm, and second, to develop a theoretical understanding of reading and narrative rhythm and the relations between these concepts. While acknowledging rhythm as a cohesive mechanism (Van Leeuwen, 2005), this article shows how rhythm also relates to the ideational and interpersonal metafunctions, thereby expanding the perspective on the meaning-making potentials of rhythm. Furthermore, we also show how rhythm contributes to the aesthetic experience invited by the multimodal features of the text, while answering the question: What are some of the important features of rhythm in literary apps?

The article proposes a rhythmanalysis based on the understanding that rhythm forms an important part of the reader’s meaning making in the use of literary apps. The approach is applied to the analysis of rhythm in the popular literary app, Florence (Wong et al., 2018), drawing on concepts from multimodal social semiotics (Van Leeuwen, 2005), although leaning towards a more reception-oriented approach, that is, how the text may be received by readers, than the traditional text-oriented analysis in social semiotics. The article’s starting point is a definition of rhythm as a change between two poles or opposites (Van Leeuwen, 2005: 182). This application of rhythm is not in the conventional sense of metre or rhyming patterns, but across textual features.

In this article, the concept of literary apps and examples of theory-developing research, of the kind applied here, are elaborated. The concept of rhythm and its theoretical background are also discussed. The methods are outlined, and Florence is introduced as a multimodal text in the Findings section, followed by a more detailed rhythmanalysis of two selected chapters.

Literary Apps

Literary apps are defined here as multimodal fictional narratives that can lead to an aesthetic experience for the reader (Iser, 1984); however, non-narrative apps, such as poetry, may also be defined as literary apps. Literary apps may be read on a tablet or a smartphone. One important feature of these apps is their interactivity, ¹ comprising digital elements that may be touched, pressed, and swiped.

Literary apps gained popularity as a textual medium around 2010 – the time that tablets became ubiquitous, presenting a relatively new field of research with theoretical and analytical perspectives yet to be explored and developed, particularly in terms of rhythm. Literary apps, as digitally-mediated, multimodal texts that have different textual features from printed picture books, present new literary and aesthetic experiences (Hagen, 2020); hence, there is a growing body of research in literary genres that are supported by mobile digital applications (Frederico, 2017; Zhao and Unsworth, 2017). Literary apps may be seen as a subcategory of digital literature, defined as literature produced by authors who utilize digital technology in the production, distribution, and reception processes (Rustad, 2012: 11). Thus, this article extends understandings within the broader field of digital literature, but with a focus on literary apps that have emerged throughout the last decade.

Research addressing theory development in the field of literary apps has included studies that draw on picture book theory (Al-Yaqout and Nikolajeva, 2015), narrative theory (Stichnothe, 2014), post-structural literary theory (Turrión, 2014), and intermedial theory (Henkel, 2015). Analytical frameworks from multimodal social semiotics have been employed on literary apps by Frederico (2017) and Zhao and Unsworth (2017). In particular, Zhao and Unsworth (2017) studied the media-specific function of interactivity, postulating a social semiotic theory of this feature and distinguishing between two basic types of interactivity: intra-text, which calls for interpretation within the narrative, and extra-text, such as hotspots, through which one can, for example, choose a language and control the sound.

In Frederico’s (2017: 57) analysis, rhythm was addressed specifically in relation to textual meanings and coherence, contributing to the limited field of research on rhythm pertaining to literary apps. The current research addresses this gap, proposing an interdisciplinary framework for analysing the multimodality of apps, integrating reader-response theory and social semiotics. Saemmer (2012) touches upon the meaning potentials of sound and images in digital computer-based texts, indicating that some temporal semiotic units of sound can equally be applied to the analysis of sound, text, or image. These reflections lead her to call for the ‘exploration of this tropological potential of couplings between text, movement and manipulation’ (Saemmer, 2013: 6), which she sees as one of the most interesting challenges facing digital textuality.

The current article endeavours to understand the coupling of text, movement, and manipulation, which Saemmer mentions, within the context of a narrative touch-screen based text. The current research also extends the concept of reading rhythms in literary apps, described to some extent by Frederico (2017), to explore the relations between narrative rhythm and reading rhythm. Rhythm also includes understandings of interactive features. These include extra-text, but more commonly intra-text features (Zhao and Unsworth, 2017), which along with their interpretive nature, contribute to the aesthetic experience.

Rhythm

In this section, the use of rhythm in this article is defined, and the specific categories of reading and narrative rhythm are discussed. Rhythm, as mentioned in the introduction, is widely defined as a change between two poles or opposites, such as ‘to and from’, ‘up and down’, ‘loud and low’, and ‘night and day’ (Van Leeuwen, 2005: 182). Rhythm occurs in the alternations between these poles, but in order to identify the change, such opposites must be mapped. Marxist sociologist, Lefebvre (2004: 15), whose view on rhythm relates to the social semiotic perspective, theorizes that ‘Everywhere where there is interaction between a place, a time, and an expenditure of energy, there is rhythm.’ This view can be extended to the understanding that the act of reading a literary app is a dynamic interaction between the rhythms of the user and the text across time.

Rhythm is an essential part of multimodal cohesion, as it is an indispensable element of the structure of multimodal texts that unfold over time (Van Leeuwen, 2012: 169). Van Leeuwen describes, ‘Rhythm provides cohesion, segments the speech, or the action, or the music, into communicative moves that propel the semiotic event forward.’ Following this understanding, with interruptions to the rhythm, the cohesion of the text falls apart. In general terms, with the social semiotic aspects set aside, cohesion is essentially the act of keeping something together. In particular, Van Leeuwen (2005: 179) identifies multimodal cohesion as ‘four ways in which different kinds of semiotic resources are integrated to form multimodal texts and communicative events’, namely rhythm (the focus of the current article), composition, information linking and dialogue.

Van Leeuwen’s theory of multimodal cohesion applies to texts and communicative events that contain multiple modes. A literary app is a certain kind of multimodal text, and reading it may be seen as a communicative event. In relation to how a text is bound together and what makes the whole different from the parts, Van Leeuwen (2012: 176) states, ‘Modes can become so utterly intertwined with one another that they no longer make sense on their own.’ In other words, it is insufficient to analyse each mode in turn, but to also examine the relations between these modes. Rhythm can potentially bind the text together across modes. Establishing an approach to analyse the rhythm of literary apps can provide new understandings of multimodal cohesion.

An interesting feature that emerges from an examination of rhythm in apps is the way in which the reader influences the rhythm in fundamental ways. When browsing, changing scenes, or performing various interactive tasks, the user affects the app’s rhythm, and thus, its inner coherence. However, in many instances, the potential for interactivity is inscribed in the app, thus, inviting a particular range of interactive rhythms.

A certain linearity is important in all texts that play out over time, and events follow a particular order. While tempo refers to measures of timing or speed (e.g. fast and slow), rhythm is the placement of elements in time in a regular and repeated pattern. As opposed to tempo, rhythm is a broader concept, which includes sensory experiences that can be difficult to measure. Van Leeuwen (2005) divides categories of rhythm into phrases, measures, and pulses. The application of rhythm in this article differs somewhat from that of Van Leeuwen as it highlights other aspects of rhythm, such as narrative and reading rhythms, and levels of rhythm, such as overarching, sequential and overlapping, as explained further in the Methods section. The division of rhythm into measures of equal duration is challenging in texts, such as literary apps in which the user’s approach to different reading tasks and interactive possibilities is variable. This relates to the view of digital humanist, Ariana Mayer, and digital author and theorist, Serge Bouchardon’s thoughts on the reading of digital literature as an event: ‘an ocean of data that only become meaningful during the brief time of our active presence’ (Mayer and Bouchardon, 2020: 29).

Rhythm seen as a sensory experience is also eminent in Van Leeuwen’s theorizations because rhythm can be seen, heard and felt, and is connected to the human senses and the body. In humans, rhythm is biologically based (Van Leeuwen, 2005: 181). Heartbeats and breathing are basic rhythms in all living bodies, and rhythm appears both within the individual and socially. Van Leeuwen (2012: 169) states, ‘Everything we do has to be rhythmical and in all our interactions we synchronize with others as finely as musical instruments in an orchestra’, thus, emphasizing the importance of rhythm for all human action and interaction. The sensory aspect is vital to the app as a literary text because the sensory experience is essentially an aesthetic one. Literary theorist, Wolfgang Iser, known for his theories on aesthetic response (Iser, 1984) considered ‘aesthetics’ as ‘perception’, and ‘aesthetic effect’ as a form of realization stemming from the human senses. This view of aesthetics reflects the etymological meaning of the Greek word, aísthēsis – sensation, because as Lefevbre (2004: 21) explains, ‘The rhythmanalyst calls on all his senses.’

According to social semiotic theory, the meaning of any utterance is realized through three metafunctions. The ideational metafunction is concerned with how language is used to represent content, the interpersonal metafunction involves how language is used interrelationally among humans, and the textual metafunction refers to the inner coherence of a text (Halliday, 1978). In the context of rhythm, which is a mechanism of cohesion, the textual metafunction predominantly comes into play, creating meaning across semiotic resources.

In this article, rhythm is analysed by introducing two categories of multimodal rhythm – narrative and reading rhythm – showing how these categories materialize and work together in an app. The call for a new approach to the analysis of rhythm has two dimensions: the reading rhythm is connected to the aesthetic aspects of the text, linked to how the readers respond to the text, and how they interact with it, as well as the narrative rhythm, which is tied to narrative structures in the text. Iser (1981: 103) claims that a text comes to life when it is read, which means it occurs in the meeting with its reader. While this view of literature is relevant, there are still organizing structures of texts that are not immediately apparent to the reader. Together, these two types of rhythm shed light on different aspects of the text: the reader’s experience of the text, and the text’s structure. The perceptive use of the app exists within the frame of the narrative context. For this reason, narrative rhythm is considered along with reading rhythm, which is closely related to interactive touch.

Narrative rhythm includes the sequences of states or events in fiction that structure the narrative. The important feature of these states or events is that they represent opposites, aligning with the view of rhythm as a change between two poles (Van Leeuwen, 2005: 182). The narrative rhythm is constituted through the shift between opposites. Narrative rhythm concerns the events in stories, which represent ideas beyond its own system of signs, and relatedly, are connected to the ideational metafunction (Halliday, 1978). Despite the fictive story, the events in it refer to the idea of real-life events, and the narrative rhythm reveals possible processes in humans’ lives, or ‘aspects of the world as it is experienced by humans’ (Kress and Van Leeuwen, 2006: 42).

There is no rhythm without repetition but, according to Lefebvre (2004: 6), there is ‘always something new and unforeseen that introduces itself into the repetitive’. In the context of narrative rhythm, this means that even when there is a repeated pattern, the story still changes and continues to unfold. However, the narrative rhythm is not necessarily about repetition per se, but, rather, the ‘to and fro’, or a change between two states, with a development that slightly modifies the rhythm with each move.

It is acknowledged that concept of narrative rhythm is not the only organizational principle in narratives, and it is possible to combine its analysis with other narratological concepts. Narrative rhythm, however, sets out to describe certain rhythmical parts of the narrative, namely opposites, particularly elucidating the connections between rhythm and ideational meaning.

While narrative rhythm can be found in the structures of the text, the focus of the reading rhythm concerns the reader’s sensory experience of the text, and therefore considers the often-overlooked role of the sensory in social semiotics. Meaning in a literary text may be perceived as neither definable nor objective, but as a human experience (Iser, 1984) and, in the case of reading a literary app, the reader’s use of the digital text’s interactive features influences this reading experience. This includes how the interactive tasks are performed by the app user, how the reader interacts with the verbal text and other multimodal features, and how these interactions with the tablet work together with rhythms in sounds, images, and moving images. Lefebvre (2004: 20) argues that rhythm analysts must use the rhythms of their own bodies as the measure of other systems; hence, reading rhythm is an embodied practice, and this bodily rhythm can occur at a subconscious level.

The interpersonal metafunction (Halliday, 1978) concerns the fact that ‘a semiotic system must be able to create a relation between the producer and the receiver’ (Machin, 2007: 17). In the context of literary apps, interpersonal meanings may concern the nature of the relationships among the fictive characters (Mills et al., 2018), as well as that between the text (created by a producer), and the reader. The interaction between a text and its reader may cause additional interpersonal meanings. In Florence, the app lets the reader speak on behalf of Florence and participate in her actions in the sense that the reader conveys her part of the conversation in the form of a vicarious and bodily experience (Mills et al., 2018). As the rhythm may influence the readers’ feelings toward the fictive characters in a story, and arises from the reader’s communicative interaction with the app, it may also, to some extent, realize the meaning potential related to the interpersonal metafunction.

Methods

This article is part of a larger project concerning aesthetics and multimodality in literary apps. The project first included 21 apps, of which 7 were chosen for detailed analysis. The interest in analysing rhythm emerged as several apps were scrutinized, with rhythm emerging as significant to the meaning and progression of the narrative. Three apps were then selected for a detailed rhythmanalysis, because each highlighted different unfolding rhythms.

The app, Florence, was selected for this article because of the interesting interrelationships between the design of the images, music, and other sounds, and the related possibilities for user interactivity, while making the text of interest aesthetically. In addition, it shows different ways through which rhythm may occur in a literary app. The analysis of this app also revealed some particularly interesting insights on the meaning-making potentials of rhythms due to its lack of verbal text. The method of moving from one scene to the other in Florence does not resemble the turning of pages in a picture book, and is therefore, not easily compared to the reading rhythms invited by picture books (as described in Rhedin, 1992). Thus, the rhythms established by the juxtaposition and sequencing of scenes differs fundamentally from the rhythms established by the linear, beginning to end, turning of the pages of a picture book, illustrating how literary apps constitute reading rhythms that differ to other media.

To map these rhythms, the researcher’s hand interaction with the app on the iPad was filmed. Each opposite in terms of touching the iPad, music and moving pictures was noted in a table. The narrative rhythm was analysed by writing down the opposites in the narrative itself. As the table was completed, a model for the analysis of narrative rhythm and reading rhythm at different levels was developed based on how the rhythm of the texts emerged and was realized at different levels. The model (see Figure 1) was then used on the material again, as a starting point for the final analysis.

Figure 1.

Structure of analysis.

Figure 1 shows the categories of rhythm used for this article’s findings. The analysis showed how reading rhythm and narrative rhythm can occur at different levels. The distinction between levels is made to illustrate that rhythms are not limited to sequences of text, but also appear in the app’s overarching structure and even in the transition between sequences, which we call ‘overlapping’ rhythms.

The next section briefly describes Florence as a multimodal text. The Findings section then focuses on rhythm, first considering its use in the overarching textual structure, and secondly, the analysis of rhythms across other levels, including sequential and overlapping rhythms.

Findings

Florence – a multimodal story founded in the visual

Florence (2018) was developed by Mountains© and published by Annapurna Interactive®, first for iOS and later for several other platforms. Ken Wong initiated the production and served as lead developer and creative director, along with composer, Kevin Penkin, producer, Kamina Vincent, and a team of supporting developers. The narrative focuses on the ambitions and relationships of 25-year-old Florence, making the text potentially of interest to a wide audience that includes teenagers and adults.

Described by its developers as a game, Florence is like a game in that the reader unlocks new sequences and chapters upon completing previous ones. The story is divided into 6 acts and 20 chapters. The verbal language is mainly used for the chapter headings, to outline each chapter’s main theme, and for Florence’s indifferent replies to her mother’s phone calls. In essence, a non-verbal story, Florence can be compared to a wordless picture book, and its narrative events may thus more often be subject to a wider range of readings than those in books that also contain verbal language (see, e.g., Christensen, 2019). To summarize the main storyline, Florence, the protagonist, leads a mundane life as an accountant. Since her childhood, she has been passionate about the visual arts; however, this dream has receded in the face of reality. She falls in love with a cellist, Krish, and a large part of the story is about their unfolding relationship, which eventually ends. This, however, allows Florence to revive her passion, invest in her artistic talent, and change the direction of her life.

The rather mundane life that Florence leads in the beginning of the story is highlighted by the interactive elements, which are limited to routine actions, such as brushing her teeth (sliding a finger back and forth), scrolling through her social media pages, and preparing the balance sheets in her accounting job. The interactive tasks change and become more varied as Florence’s life becomes less mundane. The monotony of life is underlined by pops of colour in otherwise greyscale scenes. For example, in a scene showing Florence on her daily commute, the people around her are in greyscale, while her mobile phone screen is coloured to show interest. On her job, she sits by the screen, and in the evening, she eats take-out sushi in front of the television. Her life seems to unfold more on screens than physically, which serves as a metacommentary on the very medium through which her story plays out. However, this changes when she meets Krish, and she gains an enlivened perspective on life.

Colour serves several other purposes in the story. For instance, yellow is associated directly with Krish, who is first introduced by yellow notes and cello music. At one point, the use of yellow in the background indicates that Florence is thinking about Krish. The colour of the empty speech bubbles that appear in the conversations between Florence and Krish indicate the mood of each conversation. For example, red is used for angry words in the chapter entitled ‘fight’, with corresponding sharp sounds and edges in the illustrations. Pink is used in the story to symbolize romance, such as at the end of Florence and Krish’s first date. Their speech bubbles turn pink, and the scene culminates with a picture of Florence and Krish kissing against a pink background. However, greyscale is used in sad or mundane situations, such as when Florence rides the bus to work. The significance of music is equally conspicuous. For example, Krish and Florence are accompanied by cello music and piano, respectively, and these are used to indicate whether they are ‘in tune’ or fighting with one another. The cello and the piano serve as their voices, as there is no recorded dialogue. The importance of the visual modes and the music may also denote the artistic sides of Krish and Florence: his musicality, and her interest in the visual arts.

Overarching rhythms

One overarching narrative rhythm in Florence is mainly created through a back-and-forth shift between closeness and distance between Florence and Krish in the relationship (see Figure 2).

Figure 2.

Narrative rhythm in Florence.

The text contains another overarching narrative rhythm – the presence of Florence’s creative work. In the beginning, she is allowed to do crafts, but later, her time is occupied by schoolwork. Eventually, she pursues her passion for art full time. This is also connected to the ‘back and forth’, and ‘ups and downs’ of her relationship with Krish and, relatedly, the fulfilment of her wishes versus those of others. In the narrative, others’ wishes that take priority include her mother’s desire for her to find a stable job, and Krish’s ambition to realize his potential as a musician.

Turning to the reading rhythm, the main structure of Florence consists of chapter groupings, organized by sequences. The reader turns to the next sequence by clicking an arrow at the bottom of the screen. When the chapter is finished, the reader clicks a checkmark. This structure is repeated for each chapter, creating an overarching structural reading rhythm.

While the narrative rhythm relates to the ideational metafunction, pointing to important events in Florence’s life, the reading rhythm is constituted through the interactions between the tablet and the reader’s hand, touching hotspots indicating how to get through to the next scenes and chapters. In the overarching structure, unlike the sequential level described below, the touch function can be seen as an ‘extra-text interactivity’ as these functions have little possibilities for interpretation in the narrative context but, rather, clearly indicate how to move on through the story (Zhao and Unsworth, 2017: 94). In the following section, two key chapters in Florence, ‘music’ and ‘fight’, are analysed to show how rhythm can unfold within sequences.

Sequential rhythms

The two chapters in this section were selected because they serve to illustrate some of the significant aspects of rhythm. The chapters are different in terms of how the readers move their hands, meaning that the reading rhythm varies. In addition, one of the chapters is dominated by music, the other one by other sounds, indicating different ways of interacting with the tablet. The narrative consists of a dialogue in one chapter, while the other chapter presents a sequence of action. These are two fundamentally different narrative sequences, showing how narrative rhythm can unfold in different ways.

Description of Chapter 3, ‘music’

In the first scene of Chapter 3 (Figure 3[1]), the reader must tap the play button on a phone to begin the narrative action. The effect of this user action is that Florence is seen walking, facing her phone, and wearing a headset. Half of a phone screen is shown beside her, and the reader must like or swap pictures for Florence to keep walking [2]. The music is an electronic, rhythmical, repeated beat that changes between a percussive piece, and a simple legato motif comprised of two tones that resemble human humming layered on top of the beat, playing in a looping sequence. Suddenly, the music stops, and the screen turns black and displays an empty battery symbol [3]. Thereafter, Florence is depicted without her headphones, facing forward instead of down [4]. A low sound of a cello melody in a minor key, sentimental rather than sad, begins playing as yellow notes appear that the reader pushes to make Florence go forward [4]. The music gradually becomes louder, and more and more yellow notes appear, exceeding the speed at which the reader can push them [5–6]. Toward the end of the scene, Florence begins flying, without the reader needing to push more interactive elements. A piano line is gradually added to the cello music. The scene ends with Krish playing the cello, with his eyes closed, as if lost in the music, facing the reader [7]. The reader then pushes an arrow symbol to arrive at a framed picture of Florence against a yellow background, with a heart to her left [8].

Figure 3.

Representative screenshots from Florence, Chapter 3, ‘music’. Mountains©/Annapurna Interactive®. Reproduced with permission.

Narrative rhythm and reading rhythm

A narrative rhythm is constituted by Florence’s awareness or lack of awareness of the outside world. In the beginning, her smartphone is the salient object on the screen. Florence is ignorant of the outside world at first, pushing the skip and like buttons on her phone. As the battery of her phone dies, the reader’s focus shifts to Florence, as her depiction becomes the salient object, placed in front. Florence is no longer ignorant, but she is still not fully aware of the outside world. She looks up and becomes aware of something outside of herself and her interactions with the phone, hesitantly at first, then floating, seemingly unaware of her surroundings again. In the end, she becomes very aware of the outside world, that is, Krish playing his cello. In the end, Florence is sitting on her windowsill, but unaware of the outside world, thinking of something other than what she physically sees, indicated by the little heart, leading the reader to believe that she is thinking of Krish. Her focus changes throughout the scene (from her social media page to Krish) from inward to outward, but it is not a repetition of the same thoughts.

The reading rhythm, however, concerns the reader’s physical interaction with the app. As the reader pushes the play button on Florence’s iPhone, a dull, repeated beat begins playing. In an equally dull manner, the reader pushes the ‘like’ or ‘change’ button on his or her social media page – an interface resembling Instagram with pictures and a prominent ‘like’ button in the shape of a heart (Figure 3[2]); the lack of real significance of the interface is indicated by the fact that only half of the phone’s screen is shown.

At first, the reader is invited to feel the protagonist’s hesitation, as the yellow notes appear at a tempo different from that of the cello music, but as Florence begins floating along with the music, so does the reader vicariously. As the yellow notes appear faster, the rhythm of the music sets a backdrop for the pulse of the user’s fingers on the visible music notation in a way that interacts with the tempo of the music, for instance, aligned with the pulses (Van Leeuwen, 2005). The music is represented by notes, in this case only the melody line that is played through a cello. Underlying this melody line is a piano arrangement that appears as Florence begins floating along with the music. This arrangement makes it even harder for the reader not to interact with the rhythm because the piano line uses sequences of 16th notes or semiquavers, making the measures even clearer for the reader. The reading rhythm in this sequence is influenced by the interactive relationship between music, images, and the user’s touch.

Turning to the relationship between the narrative rhythm and the reading rhythm in this chapter, Florence’s lack of interest in the physical world and focus on the screen are reinforced by the reader’s focus on the screen, while tapping the buttons on Florence’s phone. The process of shifting from a feeling of hesitation to floating along with the music, which the user may experience vicariously with Florence, is influenced by the reader’s tapping of the notes that appear on the screen, initially out of time with the music, and subsequently, in a way that makes it possible for the reader to tap in time with the music. In this way, the reader understands aspects of the narrative, including the shifts in Florence’s focus that form the narrative rhythm through the reading rhythm. Hesitance, an aspect of the narrative, is perceived because of the reader’s interaction (the tapping of the notes) with the tablet.

Potential realizations of meaning through metafunctions

The ideational meanings in this section emerge through the narrative processes and events, and the representations of the characters and their surroundings in the story. These meanings represent ideas beyond the app’s own system of signs, and are intertwined with the narrative rhythm, which influences the understanding of the narrative processes. The narrative rhythm does so by oscillating between the opposites of the narrative, that is between Florence’s awareness and unawareness of the outside world.

The interpersonal metafunction, which expresses aspects of the communication between the characters and particularly in a literary app, between the app and the reader’s possible feelings towards the characters in this interaction, is realized through both the presence and the absence of the reader’s hand touching the screen. The reader’s identification with the protagonist may be reinforced as the reader participates vicariously in Florence’s story as described in the preceding section. Even though the reading rhythm is particularly connected to the interpersonal metafunction, the narrative rhythm contributes to the user’s immersion in the story and, relatedly, the development of the reader’s empathy toward the characters, contributing to realizing interpersonal meanings as well.

The textual metafunction, which is connected to cohesion in terms of rhythm, is realized through the rhythm’s organizational qualities, such as dividing the sequence into meaningful parts that structure the narrative (the shifts in focus that create the narrative rhythm), and the reading rhythm moving the text forward through the user touching the screen. For example, if the reader does not push the notes or scroll through Florence’s social media page, the narrative stops unfolding.

In sum, it becomes clear through the rhythmanalysis of this chapter that the narrative rhythm, in particular, accentuates the ideational metafunction, while the reading rhythm accentuates the interpersonal metafunction. The textual metafunction, however, is connected to both types of rhythm.

Description of Chapter 14, ‘fight’

Chapter 14, ‘fight’, starts with portraits of Florence and Krish (Figure 4[1]) with various facial expressions, gestures, and head movements indicating tension (e.g. downward head tilt, hand to head, downcast eyes). The reader scrolls down to the scene showing them washing dishes [2]. When the characters turn toward one another in the next scene [3], the reader can then put together Florence’s wordless speech bubbles like a jigsaw puzzle [4], whereas Krish’s equally wordless bubbles are pre-programmed, that is, put together automatically. As Krish speaks, a short cello motif is played, and while the reader finishes Florence’s bubbles, a piano motif is heard. After this conversation, two black-and-white framed pictures of Florence and Krish appear one after the other [8–9], facing different directions. Clicking the arrow, the reader is presented with a jigsaw puzzle whose pieces do not fit together [10–11].

Figure 4.

Representative screenshots from Florence, Chapter 14, ‘fight’. Mountains©/Annapurna Interactive®. Reproduced with permission.

Narrative rhythm and reading rhythm

The narrative rhythm is indicated through the dialogue between the pair, particularly in the speaking and the silence, and in the rhythmic turn-taking or ruptures in the exchange of replies. Toward the end of the conversation, as they start speaking simultaneously, the scene illustrates how the rhythm of the normal turn-taking of a conversation is resolved in a fight.

Similarly, a rhythm exists in the characters’ postural shifts in each scene. For example, in the pictures of the fight, they look toward and then away from one another, subsequently appear facing one another in the puzzle, culminating in their bodies turned clearly away in the aerial view of the two lying in bed. This dynamic reflects the ongoing conflict and ruptures in the rhythms of their relationship, while the relationship between the two characters, together or apart, remains focal.

Concerning the reading rhythm, in a number of sequences in Florence, Florence and Krish engage in wordless conversations, the rhythm of which appears distinct. Bubbles without verbal language cascade down the screen. Krish’s speech bubbles and their content are pre-programmed, while those of Florence have to be put together as jigsaw puzzle pieces by the reader. The conversations indicate a rhythm, not only between the protagonist and her partner, but also between the reader and the tablet, and this rhythm is influenced by sound, music, and animation. For example, in the beginning of one conversation, the rhythms do not flow easily but are uneven. Paradoxically, the reader must push the pictures aside to put the pieces together (Figure 4[5–6]). The edges of the puzzle pieces become sharper as the fight escalates [4–5] and, at the end of this sequence, the main characters begin talking simultaneously as people tend to do in heated discussions. Notably, the regular sounds that appear as the puzzle pieces in Krish’s speech bubbles are joined automatically, and influence how and at what speed the reader assembles Florence’s pieces. The cello sound accompanying Krish’s bubbles and the countering of Florence’s piano sound also influence the time between the replies. As Krish speaks faster, the reader may be influenced to do so on Florence’s behalf by putting the pieces together faster, following the rhythms of the sounds accompanying Krish’s puzzle pieces falling into place.

The way that the speech bubbles are put together represents the ‘pulsing’ of speech (Van Leeuwen, 2005: 183), with the sound of each puzzle piece representing one pulse, the visual puzzle piece representing a measure, and the whole speech bubble representing a phrase. This in turn underlines the conversation’s rhythm and how the rhythm itself plays a crucial role in ‘getting the message across’ (Van Leeuwen, 2005: 181). This is emphasized to the level at which the actual words are lacking; only the rhythm is left, along with visual representations in the form of the increasingly sharp edges of the puzzle pieces. This ‘pulsing’ serves a principal role in meaning-making because it underscores the rhythm that carries ‘the key information of each measure . . . If you heard only these syllables you might still get the meaning’, as described by Van Leeuwen (p. 183). In the case of the conversations in Florence, the interesting point is the emphasis on the rhythms of human conversation without words, while still ‘getting the meaning across’ by inviting readers to understand that this is a conflict or fight.

Concerning the relationship between the two types of rhythm, the narrative rhythm is indicated through the conversation’s content, as well as the rhythms, ruptures, and speed of the turn-taking. The specific content is imagined, as the reader cannot see or hear the actual words of the conversation. The narrative content is indicated through the conversation’s rhythm. The reading rhythm is the realization of taking turns through the reader’s actions and the tempo of putting together the puzzle pieces that complete the speech bubbles, which is influenced by the sounds that are played as Krish’s jigsaw pieces are put together automatically. In this example, the close relationship between the narrative rhythm and the reading rhythm becomes clear. The first is the rhythm of the conversation between the two main characters, and the second rhythm is the way that the reader participates in their conversation when interacting with the literary app’s features. This highlights the vicarious experience of being part of the conversation, for instance, by feeling the frustration in the fight.

Potential realizations of meaning through metafunctions

The narrative rhythm contributes to ideational meanings, representing ideas beyond the app’s own system of signs by showing how the main characters take turns in the conversations. Speech bubbles represent the speech process, and the rhythm represents the substance of the conversation – both clearly connected to the ideational metafunction.

The reading rhythm reveals interpersonal meanings through the feeling of frustration that builds up in the conversation, for example, through the quickening turn-taking. However, interpersonal meanings are also realized through the substance of the conversation, which is connected to the narrative rhythm because, without imagining this substance, that is, the ideational meanings of the narrative, the reader’s feelings toward the characters would not be actualized. As a textual principle, the rhythm structure includes longer and shorter sequences of speech as the reader assembles speech bubbles to continue the narrative.

Overlapping rhythms

As Figure 1 shows, a literary app contains not only overarching rhythms and sequential rhythms but also rhythms that cross narrative sequences, namely overlapping rhythms. Regarding these overlapping rhythms, the music plays a special role in maintaining the coherence across the montage of images in the app. In this case, the music influences the reading rhythm, because the music and other sounds do not stop as the reader moves to a new sequence, creating continuity in the narrative. The rhythms within the changing scenes and junctures of the visual narrative are sometimes distinct and abrupt, such as the fragmented images of dislocated puzzle pieces during the conflict between Florence and Krish. However, the music contributes to the app’s rhythm and coherence by continuing through many of the scenes and longer sequences. The music sometimes influences the reading rhythm, changing as the reader begins a different interactive activity. The way that music functions in the app helps the reader sense the whole, and thus concerns the reading rhythm in particular. However, narrative rhythm may also overlap across scenes. For instance, within a chapter, the reader moves to the next scene, in which pictures of Florence and Krish face toward or away from one another, demonstrating how they feel over several sequences.

Discussion

The rhythms of an app differ from those of a film, in which the reading rhythms are fixed and, although it is possible to rewind, pause and continue, the film sequences do not change per se (e.g. see the film analysis of Van Leeuwen, 2005). Literary app users can linger in the experiences, choosing not to move forward in the story, or may even perform a certain interactive task for a long time because they enjoy the task itself. Different tasks open up different possibilities for individual engagement, and the same reader may perform the task differently when reading the app over again, creating a slightly different reading experience.

However, this does not mean that each reader composes their own exclusive rhythm. The readers are not completely free, as the app invites them to perform tasks in a certain way to move the story forward, and it contains pre-programmed elements, sounds and animations that may influence the readers to perform the tasks in certain ways, following certain rhythms. For instance, as the reader taps the yellow notes when Florence meets Krish for the first time, one may do this in a way that is influenced by the ‘pulses’ in the music (Van Leeuwen, 2005: 183).

In a picture book, the experience of rhythm is construed in the turning of the pages as well as in the experience of time in the narrative, and the told time (Rhedin, 1992). In a novel, the move forward is caused by the readers themselves, not only in turning the pages, but also in decoding and interpreting the verbal language, which in turn result in aesthetic experiences, that is, the readers’ individual, sensuous experiences (Iser, 1984). Compared to watching films or reading novels, the reading of literary apps is neither fixed nor free; it is partly individual and partly influenced by the way in which the interactivity, music or images call for action. The unique rhythms of a literary app consist of the interplay between the reading rhythm and the narrative rhythm.

During the fight between Florence and Krish, the reader may experience the agitation of the fight through the rhythm of the puzzle. Readers may also feel the hesitation of Florence as she walks toward Krish because they are forced to push the yellow notes in a different tempo than the music. These are potential realizations of aesthetic experiences, which occur as interpersonal meanings are realized, as the experiences influence the readers’ feelings toward the fictive characters. Rhythm as a back and forth of narrative events points toward the ideational meanings of the real World, whereas both types of rhythm – the reading rhythm and the narrative rhythm – realize textual meanings through organizing the text by inviting the reader to act in a specific way or through organizing a narrative structure. The sensory experience of the reading rhythm in an app connects to the reading experience as an aesthetic one because ‘rhythms shape human experience in timespace and pervade everyday life and place’ (Edensor, 2010: 1). In this way, rhythm contributes to the entirety of the literary experience.

Lemke (2011) calls for a phenomenological perspective to complement the semiotic one. He claims that such a perspective ‘reminds us of the importance of time, pacing, feeling, affect, and embodiment, all of which can be construed semiotically, but which seem to elude being completely accounted for in formal, categorial terms’ (p. 141). To a certain extent, the rhythm encompasses the embodiment, feeling, affect, time, and pacing. Perhaps the manner of perceiving rhythm, as discussed in this article, should not be categorized as a social semiotic one, although Van Leeuwen’s (2005) concepts are the starting point of the analysis. Nonetheless, this approach is an attempt to consider the aspects of rhythm in the meaning-making process that are not ‘being completely accounted for’ (Lemke, 2011: 141).

Conclusion

This article elucidates some of the many facets of rhythm related to the multimodal design of a literary app, which invites different forms of interactivity than the linear reading and page-turning of print-based picture books. The approach to analysing rhythm employed in this article takes into account that there are two types of rhythms that may both appear on different levels. The analysis has shown how rhythm not only contributes to the multimodal cohesional aspects of literary apps, but is also fundamental to meaning-potential. Rhythm dictates the spatiotemporal patterning of events across narratives, whether intermittent or steady, fluid or ruptured. Attending to these spatio-temporal specificities can generate new insights into the interconnectedness of multimodal design and cohesion within literary apps. However, to fully understand and demonstrate the concept of rhythm in literary apps more broadly, the analysis of multiple texts could potentially illustrate further ways in which rhythm might function in various digital narrative formats and applications.

The findings are significant in changed reading environments, both in education and beyond, in which the social practice of reading literary texts is now carried out across multiple digital and non-digital formats. This article has shown how the multimodal relationships between image, music, sound, and other interactive elements in the design of literary apps differ from those of printed book formats, creating hybrid rhythms characterized by a new and dynamic interactivity.