Squaring Literacy Assessment With Multimodal Design

Abstract

Over the past few decades, there has been a marked shift away from conceptualizing literacy as a functional skill set toward its recognition, particularly for children and youth, as a social achievement that is buttressed, in part, by access to digital tools and new media. Yet, beyond the mere consumption of multimedia and the mundane assemblage of words, images, and other resources, we ask, “What does a successful multimedia literacy performance look like and how might ‘designful’ multimedia thinking and composition be taught, learned, and assessed?” In addressing these issues, we present a fine-grained description and analysis of the work of a 13-year-old Singaporean named “Jeremy,” who produced a personal digital story of considerable theoretical and practical interest to us as researchers and new literacy scholars. Building on prior research in the field of multiliteracies, we argue that educators (and students) must cultivate their own senses of “semiotic awareness” before meaningful assessment of children’s multimodal design work can be conceived or implemented. We also sketch a preliminary approach to assessing multimodal literacies and explicate a range of interconnected representational possibilities that we expect will prompt a timely and urgent reconsideration of multimodal meaning design in school settings.

Introduction

In the lives and education of children and youth, literacies may now most usefully be understood not so much in terms of representational systems that are inherited and engendered, but, perhaps, as an ever expanding and revised set of blueprints: designs for architectures of meaning continually built and rebuilt (cf. Barton, 2007; Harris, 2000; Kress, 2003, 2010). We recognize, too, that digital technologies of various descriptions are increasingly the buttresses that bear these integrally multimodal semiotic architectures. Thus, understanding the roles and potentials of digital mediation in the designing of meaning (Kalantzis & Cope, 2005; Kress, 2003, 2010; New London Group, 1996), rather than digital technologies per se, is of greatest importance. Though we can identify the tools, procedures, and practices of the architect, this offers only a cursory view onto architectural design; so it also is with technologies of literacies.

Based on research in a Singaporean educational context, and building on prior work in the field of “multiliteracies” (Cope & Kalantzis, 2000, 2009; New London Group, 1996; Unsworth, 2001), this paper demonstrates the complex, relational nature of multimodal meaning design—as yet scarcely understood, particularly within classrooms—and argues that until we reach more nuanced understandings of multimodal meaning design, assessments made of students’ digital multimedia products, and multimodal literacy capacities, will be of very limited value. Preliminary to any evaluation of children’s multimodal design work, we maintain, educators must cultivate their own “semiotic awareness” (Nelson, 2006, 2008; Nelson & Kern, 2012), that is, critical attention to relational, multimodal aspects of meaning design, without which meaningful assessment schemes can neither be conceived nor implemented.

David Barton (2007) remarks, “the child is not repeating or recapitulating the adult’s world, the child is creating a world anew” (p. 135), to the point that digital technologies are now an organic and, in the Vygotskian sense, “internalized” (Vygotsky, 1978) feature of children’s literate experience. Admitting this, with a view to literacy assessment, we question whether comfort and familiarity with computer technologies necessarily imply facility and efficacy in digital meaning design. There is little doubt that young people have, as we all do, increasing need and occasion to design, produce, and distribute meaning in digital, multimodal forms (Kress & van Leeuwen, 2001), for example, in creating a slideshow for a class project and sending or interpreting a Multimedia Messaging System (MMS) on a mobile telephone. However, are multimodal composers necessarily also designers? We believe they are not.

Authentic design—of anything, including meaning itself—needs a creative capacity, the ability to fashion the raw material of conventional forms, practices, and meanings into novel ones. “A designer,” as famously expressed by revolutionary architect, engineer, and thinker R. Buckminster Fuller, “is an emerging synthesis of artist, inventor, mechanic, objective economist, and evolutionary strategist” (Buckminster Fuller Institute, 2010).

However, as Buckingham (2007) observes, the digital, multimodal texts that young people generate are too often “characterized not by spectacular forms of innovation and creativity . . . but by relatively mundane forms of information retrieval” (p. 92): the antithesis of design. Notwithstanding a substantial research literature that documents and celebrates some youth designfully expressing a digital “voice” to personally and socially transformative effects (e.g., Alvermann, 2008; Hull, 2003; Hull & Katz, 2006; Pleasants, 2008; Vasudevan, Schulz, & Bateman, 2010), we see that the multimedia texts young people create can often be mere assemblages of words, images and other resources casually appropriated online via some hastily chosen search terms and a few mouse clicks.

If literacy educators accept that designing texts and meanings is more than, and preferable to, the mundane consumption that Buckingham reports, then they are confronted with the question of how to learn and teach designful multimodal thinking and composing; an issue confounded by the fact that literacy practitioners are typically not themselves exemplars of Buckminster Fuller’s “designer.” An even more primary question then becomes, what exactly does a successful multimedia literacy performance look like? Traditionally, language and literacy teachers have approached the evaluation of students’ compositions (namely linguistic texts) as fairly unproblematic. Clear appraisal of quality, accuracy—even more subjective aesthetic aspects—was an assumed pedagogical capacity. However, assessing dynamic, relational qualities of meaning in the interstices between mode, ¹ text, and context can be unfamiliar and uncomfortable territory for teachers, who, adopting familiar, comfortable choices of assessment strategies and procedures, would tend to overlook the multimodally designful qualities in their students’ work (Johnson & Kress, 2003).

This paper examines such a case. Following a brief review of scholarship relevant to our core themes of meaning design, multimodality, and assessment, we detail the circumstances in which a 13-year-old Singaporean youth named “Jeremy” ² produced a personal digital narrative, or “digital story,” which we have come to see as a remarkably designful multimodal text. The specific research questions underpinning our analysis are as follows:

• Research Question 1: To what extents and how can the various elements of Jeremy’s digital story be seen to semiotically cohere?

• Research Question 2: What might this suggest about Jeremy’s developing capacities for multimodal textual design?

• Research Question 3: To what extent do these capacities align with the assessment mechanisms applied to Jeremy’s work in school?

• Research Question 4: How might such expectations and assessments be reformed to more closely align with and take due account of Jeremy’s demonstrated design capacities?

We then analyze the multimodal features and relationships in Jeremy’s story, rendering sensible and comprehensible, we believe, evident qualities of the designer that he clearly is. Finally, we draw these empirically oriented discussions together to preliminarily recommend some core considerations for thinking afresh about multimodal meaning design and literacies.

Literacies, Multimodal Design, and Educational Assessment

There has been a marked shift away from conceptualizing literacy as a purely functional skill set toward its recognition as a “social achievement,” foregrounding its empowering and self-enhancing potentials (Scribner, 1988) and intrinsically “ideological” nature (Street, 1984, 1996). This decidedly sociocultural, critical turn in literacy was signposted by landmark ethnographic studies of diverse, situated literacy practices among, for example, the Vai of West Africa (Scribner & Cole, 1981), middle- and working-class African Americans and Caucasians in the Piedmont Carolinas (Heath, 1983), and Iranian males (Street, 1984, 1996), among others. A common conclusion to be drawn from this work is that reading and writing cannot be understood outside of the social contexts and practices in which they may occur. Furthermore, a necessary implication is that any singular conception of literacy would be a fictitious abstraction. By their nature, “literacies are legion” (Lemke, 1998, p. 283; italics added).

A second, related insight to have emerged as a foundational tenet of the “New Literacy Studies” (Gee, 1990/2008; Street, 1996, 2003) is that textual meaning, being inherently social, must be understood to exist within and beyond the boundaries of any text itself, that is, what a text means has only partly (and not even necessarily) to do with the conventional significances of its printed words (e.g., see Gee, 1990/2008, Chapter 5). Linguist Roy Harris (2009) asserts a stronger version of this argument: that the meaning of a word, as with any sign form, is “radically indeterminate” (p. 81). Harris explains that “contextualization,” not a dictionary definition, is what permits a word to have any meaning at all, whether conventional or idiosyncratic, linguistic or extra-linguistic. Persuaded though we may be, a literacy pedagogy founded on radical indeterminacy is challenging to envisage.

The question of how to accommodate such indeterminacy, or “multiplicity,” within an operational pedagogical approach was first addressed by an international consortium of scholars known as the “New London Group” in a now highly influential Harvard Educational Review piece on “multiliteracies” (New London Group, 1996). A primary organizing and operational principle of multiliteracies is “design” that “sees semiotic activity as a creative application and combination of conventions . . . that the process of Design transforms at the same time it reproduces these conventions” (New London Group, 1996, p. 74). This transformation of “available designs” (objects, ideas, practices, processes) is integrally semiotic and material, but also fundamentally social, in that the resultant “re-designs” become newly available as meaning-making resources that potentially anyone could recruit and similarly transform for her own purposes.

In view of so-called “traditional” literacy, design rejects the conception of reading and writing as exercises in decoding and encoding stable meanings, and it reframes both of these activities as creative processes, wherein diverse multimodal resources (e.g., pixels, printed words, knowledge of genres, personal memories, ethical positions) are considered, selected, configured, composed, synthesized, and revised, conveying semiotic, material and social (e.g., political) transformations and implications (Kress, 2003, 2010). So pedagogies of design have the mission and the potential to appropriately prepare young people to productively link “word” (or text) and “world” (Freire, 1970) and thereby to thrive in the evermore plural and digitally mediated communication environments of today and tomorrow.

Yet, teachers, curriculum planners, and researchers who want to explore and actualize the positive potentials of design pedagogies and multimodal meaning making face a dilemma. Put simply, the assessment practices associated with school literacy curricula and instructional programs that privilege “linguistic” (Jewitt, 2005, p. 330) competencies (again, principally decoding and encoding) are not necessarily suitable for complex texts involving the selective, purposeful combination of a variety of modes of representation. For example, Vincent (2006), a primary school teacher and researcher in Australia, determined that criteria used for measuring and making judgments about verbal texts were of no use to him in assessing the content and quality of his students’ multimedia narratives, although this is not to say that other adapted assessment guidelines (e.g., visual design or film studies) or expertise might have served a useful purpose under other circumstances. We see at least two paths forward where the nexus of multimodality and educational assessment is thinly available in practical terms.

First, educators and materials writers can try to measure what they can from the technical and substantive aspects observable in students’ finished work. For example, the production elements featured in Standley and Via’s (2004, pp. 71-74) multimedia digital storytelling evaluation rubric are as follows: transitions and edits, sound, camera technique, lighting, framing, planning and storyboarding, content and theme, accuracy and information, acting and dialogue, originality and creativity, and documentation. Importantly, each of these factors is measurable on an ascending scale of accomplishment from one, three or five points, adding up to a total possible score of 55 for all the scales ³ .

While Standley and Via’s framework makes clear what is valued by educators adopting (and adapting) a film makers’ approach, it is largely silent about the aptness or appropriate use of modes in meaning making from an audience perspective or the motivations and the emerging sign-making intentions of multimodal authors, overall (cf. Kress, 1997). Indeed, it is difficult to imagine how inherently subjective, intangible dimensions like these could be made explicit and counted in an agreed fashion in and across multiple instructional contexts, given the constraints of typically standardized, sporadic school-based assessment practices.

A second and alternative approach to the assessment of students’ multimodal work side steps the issue of grading individual artifacts by focusing instead on the personal development and growth of children as multimodal authors. For example, Bearne and Wolstencroft (2007) detail steps in planning and teaching a wide range of “visual approaches” to writing with young learners. As children are increasingly expected to “read” images and other modes along with print in literacy work, they need to make decisions about if and when to use combinations of modes depending on their communicative purposes, audience, and the resources (physical and virtual) they have to hand (Kress, 2003).

A strength of Bearne and Wolstencroft’s approach is that it allows students’ voices to be heard concerning their design intentions and agency in how they craft texts in various ways depending on purpose, audience, available resources and content. Nonetheless, while informed decision-making dominates, the mechanisms by which modes of representation variously interact to produce meanings or effects are not explained or illustrated. Furthermore, the manner in which multimodal texts cohere or consistently present a point of view or story (or not) within and/or across modes is unexplained and perhaps taken for granted. Helpfully, other researchers have specifically explicated patterns of multimodal meaning in children’s work (e.g., Kendrick, McKay, & Mutonyi, 2009; Kress, 2010; Pahl, 2005), but these examples tend toward the descriptive and typically examine texts of insufficient complexity (like drawings with captions) to suggest a range of criteria for developing an “apt metric” (Kress, 2010, p. 182) for multimodal literacy assessment.

In sum, educators have to make a choice at the center of multimodality and educational assessment: either attempt to assess students’ texts quantitatively or describe and evaluate learners’ growing compositional capabilities in more qualitative terms. However, we suggest a middle path that accepts, for the present, the predominance and consequences of quantitative assessment while also promoting a more meaning-focused, relational approach to evaluating students’ multimodal literacy performances. Yet, to draw benefits from this best-of-both-worlds viewpoint, we need to demonstrate empirically how and why quantitative assessment approaches can miss the mark in evaluating new media literacy performances and capabilities.

Research Context and Method

In the summer of 2008, teachers in the English ⁴ department at Singapore’s Fox Hill Secondary School (FHS) began a school-wide infrastructure and curriculum development program, where each teacher and student had on-demand access via wireless laptop computers to a raft of digital tools, including an online learning management system for the distribution and collection of instructional material and the Google Applications for Education ⁵ suite. Also underway at that time was a research program, which aimed to describe the English language curriculum at FHS, analyze the impact of its one-to-one laptop initiative in terms of teacher professional judgment, and report on likely conditions for the development and implementation of scalable and sustainable one-to-one English language pedagogical practices (Towndrow & Vaish, 2009).

Two years into the study, we began a descriptive qualitative case study (Hitchcock & Hughes, 1995) attending to the production and teacher processing of students’ digital stories. In particular, we watched Jeremy’s piece several times and followed up with conversations with him and his English teacher to verify certain facts about the task set up and implementation of the digital storytelling project.

Jeremy’s English language subject teacher, Kim Chew, was in her 14th year of full-time professional service at FHS. She was notably amiable and hard working, describing herself as “satisfied” with her established pattern of teaching that was geared toward preparing students for success in high-stakes, hand-written tests (personal communication). Though, she was not averse to involving her students in creative multimedia work, like digital storytelling, which she understood to motivate her students and, perhaps more importantly, which fulfilled a key directive of the 2010 National English Language Syllabus ⁶ to integrate “rich language for all” (Singapore Ministry of Education, 2008, p. 2) specifically in the form of digital multimedia texts, into her lessons. Her digital storytelling program comprised a 10-week unit of work (UOW) in narrative writing culminating in an assessed summative performance task about “significant life events”: On their laptops, the students produced multimodal versions of the narratives they had recalled, researched, and written. Kim planned to assess the students’ work using a modified digital storytelling rubric downloaded from the Internet (see Appendix A). ⁷ She gave this document to all the students in advance so that the assessment criteria would be transparent to them.

In addition, on one notable occasion during the 5th week of the UOW, Kim bolstered the students’ understanding of the task at hand by demonstrating what, to her mind, constituted an effective performance of multimodal communication. The demonstration lesson was organized into three parts. In the first section (approximately 5 min) she delivered a whole-class lecture consisting mainly of curriculum talk on narrative writing and the organization of ideas and resources in storytelling. In the next segment (approximately 25 min), Kim delivered an almost impassioned enactment of Jules Verne’s 20,000 Leagues Under the Sea (Miller, 1969), augmenting the exaggerated prosody of her own speech with emphatic gestures and the pictures and movement in a pop-up book from her personal childhood collection, all of which the class took in with rapt attention. Finally, her storytelling concluded, Kim returned to whole-class lecture mode until the end of the period (approximately 15 min). The content of this talk focused once more on curriculum and organizational content. The teacher expounded the storyteller’s art by drawing a number of task-relevant comparisons between “traditional” oral storytelling (i.e., what she had just done) and digital storytelling. For example, Kim explained how she had used the pop-up book as an aid to engage the class (her audience) visually in the story.

Jeremy’s digital composition titled, “Digital Story Telling: Places My Mum Fondly Remembers,” describes second-hand his mother’s memories of childhood by presenting a series of retrospective vignettes from her experiences growing up in a Housing and Development Board (HDB) ⁸ flat in Singapore’s center-west neighborhood of Queenstown. It is a simple, but engaging and affecting piece, and Kim gave it the highest rating in the class: 72 out of a possible 100 marks (Appendix B). As researchers, we wondered what this assessment might tell us about Jeremy’s understanding of new media communication and digital tools and his capabilities as a multimodal author. Not very much, we surmised.

Though our work had little chance of feeding back directly into FHS, we nevertheless undertook a broadly social semiotic micro-analysis of Jeremy’s work as a means to gaining greater purchase on the ineffable, yet important design dimensions of his piece, and on multimodal literacies at large. Our analysis exemplifies a social semiotic approach in that:

. . . the text is foregrounded as the semiotic work of the designer, and the primary concern . . . is the kinds of choices s/he made within the contextual constraints of the curriculum, the pedagogy as the social relations in the classroom and the technologies of printing/digital design, the discourses realized and the potential effects on learning. (Jewitt, 2009, p. 35; cf. Bezemer & Kress, 2008)

We qualify our interpretation of a social semiotic approach as “broad” because we find it most helpful to recruit analytic tools from “mainstream semiotics” (Hodge & Kress, 1988), and diverse theories of communication and design, as well as the expected Hallidayan frameworks (cf. Halliday, 1978; Hodge & Kress, 1988; Kress, 2003, 2010; Kress & van Leeuwen, 1996/2006, 2001; O’Toole, 1994; Unsworth, 2001; van Leeuwen, 2005). Our particular interest is in discerning patterns of “intersemiotic” meaning potential (cf. Jakobson, 1959/2000; Kress’, 2003, notion of “transduction”)—describing the creative meaning-making possibilities that arise when different modes are put into conversation, so to say—and associated patterns of “multimodal cohesion” (van Leeuwen, 2005).

Multimodal Analysis

Textually, Jeremy’s story comprises a collection of hand-drawn and digitally produced still images, audio recordings of his own spoken words, and a soundtrack song, all purposefully coordinated within a span of 3 minutes and 2 seconds. Visual transitions between images and “camera” effects (pans and zooms) are also deployed as elements in Jeremy’s design for meaning. In our analysis, we first unpacked the piece inductively and “along modal lines,” attending by turns to the represented content in each discrete semiotic mode through a process of open and axial coding, adopting and adapting a constant comparative method (Strauss & Corbin, 1998). With each elaboration and refinement of the coding scheme, the respective semiotic properties of each part of the story became increasingly salient in connection and contradistinction to other parts, highlighting orders of intersemiotic meaning potential. We assured rigor in these processes by cross-checking our interpretations with each other and triangulating our findings with other data sources. We also formerly consulted Jeremy and Kim Chew on separate occasions about the credibility of our analysis. This member checking (Mertens, 1998) allowed us to clarify our perspectives before finally writing them up. ⁹

The deconstruction and reconstruction of multimodal features and intersemiotic relations within the piece were accomplished with the aid of a fine-grained multimodal transcription method (see Figure 1), developed in earlier work (see, for example, Hull & Nelson, 2005; Nelson, Hull, & Roche-Smith, 2008). Our approach to transcription, illustrated in Figure 1, isolates each modal channel and arranges these diverse components as they co-occur in the temporal flow of the story. Visual (pictorial), spoken-language, written-language, and musical forms of representation are shown “side-by-side,” and the various visual transitions and video effects used by Jeremy to segue between images are described.

Figure 1.

Multimodal transcription extract (Frames 10-14).

In addition, we added a “Time Code” channel to indicate the duration of the appearance of each visual image on screen. Visual images are the class of semiotic element within the piece that, due to its static nature, is most clearly bounded as the story unfolds in the flow of time; the appearance and disappearance of each image from the space of the screen is marked and obvious, unlike music, for example, which is characterized by a fluid temporality that makes it harder to segment. Jeremy, as a function of his design process, created a tight correspondence between particular words and sentences and particular images; he wrote the script and then imagined and created images that illustrate story segments. In this way, the images delineate meaningful multimodal sections within the story, and so we see that the static image presents and defines a kind of “de facto minimal unit” (cf. Hull & Nelson, 2005) for the analysis of intersemiotic meaning potential in all co-deployed modes.

Though we find it useful here to segment our reported analysis according to mode to demonstrate our interpretation of the semiotic workings of the piece, we do not mean to suggest that this is the only combination of parts to which the whole of the story is reducible, nor even that the whole is reducible at all. It is the irreducibility of the congeries of semiotic relations in the story that we wish to highlight, in fact. To this end, we point up some of the evident design decisions made in the composing process and aim to lay bare something of the ingenuity that generated this multimedia product.

The Auditory Channel

We use the term audio channel to categorically subsume modes of communication within the story that are apprehended through listening or hearing. These include the aural-linguistic mode, referring to Jeremy’s spoken words and to the lyrics of his chosen theme music, and the aural-musical mode, which concerns the other communicative qualities of the musical sound track (e.g., rhythm, tempo, melody).

Aural-linguistic

In Singaporean-accented Standard English, ¹⁰ Jeremy relates the story of his mother’s childhood spent in Queenstown. The narrative is transcribed below:

1 My mum used to live in an old three-room HDB flat on the fourth storey in Queenstown.

2 Her house had only two bedrooms, one for parents and the other for herself and her three

3 siblings. There were two double-deck beds, and in the old days there was a rule that only

4 boys can sleep on top. So my mum and her sister had to sleep at the bottom. My mum

5 describes her house as presentable, cozy and comfortable. The houses in her

6 neighborhood were exactly the same as her block. She always liked to visit the Indian

7 sundries store to indulge in her favorite tidbits, especially cuttlefish. As her mum does not

8 like them to eat tidbits, there was one time she got really angry. As their mum was not

9 home, my mum and her sister went down to buy cuttlefish. They entered the lift and

10 opened the cuttlefish. And FOOM! The cuttlefish flew all over! They were shocked and

11 afraid that they quickly ran home. When their mum was on the way home, she saw the

12 cuttlefish all over in the lift. She knew it was them because they were like cuttlefish

13 addicts. My mum studies in CHIJ Kellock, where in her classrooms there were only a

14 blackboard, tables, chairs, and lots of mats, because most of the time they were seated on

15 the floor. After school she likes to visit the library with her friends to study and read.

16 Although my mum is not a very avid shopper, she still shops at Queenstown Shopping

17 Centre for her necessities. My mum is a Catholic, and she goes to the Church of the

18 Blessed Sacrament to pray. My mum loved to go to the Botanic Gardens, where she can

19 play with her sister and do what they like to do. The Botanic Gardens is a place she will

20 remember for her lifetime because she was ever stung by a bee there. Another place my

21 mum would remember is the Singapore Netball Association. She would be there three

22 times a week to train for whatever competition there was. My mum had lived there for

23 thirty years and sure has many memories. ¹¹

As with many approaches to digital storytelling (e.g., Lambert, 2002), Jeremy’s story was first scripted, and then read and recorded. The nature of his process is evident, too, in the fact that the presentation of each visual frame ends and begins with a complete sentence, the only exception being in Lines 13 to 15 (Frames 12 and 13), where a main clause, “My mum studied in CHIJ Kellock,” is introduced while the first frame, depicting the outside of the building, is shown and the subordinate clause, “where in her classrooms there were . . . ,” is carried over into the next frame, depicting the interior. Again, Jeremy drew pictures to correspond to the script he had already written, and in his story he speaks to the pictures in sequence. Unsurprisingly, his speech has a somewhat stilted quality that betrays reading aloud. Jeremy did not ad lib or deviate from his script. He did, however, inject emotion into his words at moments of dramatic action, such as when the bag of cuttlefish explodes in the lift; his “FOOM!” is emphatically loud, and we, his audience, are especially to understand that “the cuttlefish flew ALL over,” a point-driven home by an exaggerated lengthening of the vowel sound in the word all (Lines 9-10; Frame 10).

Most of the events of the story take place in past time, except for the ending (Lines 22 and 23; Frame 20), which returns to the present (“. . . she sure has many memories”), and in Lines 16 to 17 (Frames 15 and 16), which jar us back into present day also by discussing Jeremy’s mother’s shopping habits, religion, and church membership. The mention of his mum’s spirituality as an adult may be a logical lead-in to introducing her Catholic education and upbringing, but the mention of shopping represents a curious rupture in the narrative-past structure of the story. Also curious are several grammatical “miscues” (Goodman, 1965, 1969), underlined in the transcript above, that confuse past and present time, such as when Jeremy says that his mum “studies at CHIJ Kellock” (Line 13; Frame 12) and “likes to visit the library with her friends” (Line 15; Frame 14). The effect of such inconsistencies was an understandable impression on the part of his teacher of inelegance and inaccuracy, which eventuated in an assessment of the ostensibly most language-related aspects of the piece as mediocre.

Aural-musical

The musical score of Jeremy’s story is a popular song titled I Won’t Go Home Without You, released in 2007, by the rock band Maroon 5. ¹² The lyrics of the song tell a story, in the first person, of a rueful man who has presumably betrayed his partner’s trust, causing her to leave. The singer/narrator bemoans life’s misfortunes (“Why does every moment have to be so hard?”) and begs the partner for “one more chance to make it right.”

The thematic content of this song would seem to bear little or no obvious relation to Jeremy’s own narrative. Indeed, it seems likely that he simply dropped the song into the story timeline without modification or alignment to other elements of the piece (e.g., pictures, spoken words) in any special way. The song features three repeated changes in phrasing and tempo, and these shifts mainly do not occur in correspondence with any other discernible “pivot point” in the piece, such as a significant juncture in the spoken narrative or on the cusp between still images. One notable exception is at the end where the credits begin. Here there is an added flourish in the way the lyrics are sung, a departure from the normal refrain (“I should not ever let you go”) capped with a dramatic staccato elongation to the word go (or go-oh-oh-oh). This musical convention signals the approaching end of a song, often a final repetition of the refrain; likewise, it marks the wrapping up of Jeremy’s story. Moreover, he amplifies this sense of closure, quite literally, by raising the volume of the music played during ending credits. He engineers the same effect at the beginning, on the opening title slide, which serves, again, to bound the narrative.

We would mention one other feature of the musical portion of the story that potentially makes a semiotic contribution to foregrounding aspects of Jeremy’s expressive intent. While the story is certainly not about a painful break-up, it is about “going home,” in the sense of revisiting the past. And his score—as reflected by its title (“I won’t go home without you”), which is repeated three times throughout the song—is also about going home, or rather the desire to go home, but not without an unnamed female “you.” In each channel, spoken language, and musical lyrics, the narrator invokes the ideas of returning “home” and the presence of a female figure to whom that home also belongs.

The Visual Channel

Within the visual channel, representation in three different modes is evident: the visual-pictorial mode, which refers to imagistic depictions in the piece; the visual-kinetic mode, describing qualities of visual movement; and the visual-linguistic mode, accounting for the incorporation of written language symbols and scripts through various means within the multimodal text.

Visual-pictorial

The main visual components of the story are a series of 17 mixed-media (pencil, marking pen, and crayon) drawings that Jeremy himself produced and then digitized with a scanner. These appear in the story as Frames 2 through 4 and 6 through 20; Frames 2 and 20 feature the same image. In terms of draftsmanship, most of these drawings are spare and non-naturalistic (e.g., line drawings of stick figures and two-dimensional rectangular buildings). Notable exceptions are in Frames 4, 12, 13, and 14, in which two-point perspective is attempted in the representation of “double-deck beds,” a schoolhouse, classroom desks, and a library table, respectively (see Figure 2). Yet, simplistic though these drawings are, they may be seen to directly, visually reinforce the theme of the narrative: a nostalgic reflection (by proxy) on a particular childhood.

Figure 2.

Drawings incorporating perspective (Frames 4, 12, 13 and 14).

Although Jeremy is an early teen, his drawings seem to have been produced by a much younger artist; his renderings of human figures are particularly characteristic of a typically earlier phase of artistic development. According to Viktor Lowenfeld and colleagues, based on their pioneering, enduring work on the developmental analysis of children’s drawing, Jeremy’s drawn representations would seem typical of the “schematic stage,” normally evident in the art of 6- to 8-year-olds, rather than the developmental period of “naturalistic” drawing most commonly evidenced by the work of youth of Jeremy’s age (Lowenfeld & Lambert Brittain, 1987). His main concern seems to have been with aligning the visual style of his piece with the presumed expectations and preferences of his peers. The simple shapes, idiosyncratic perspective, and flat backgrounds filled in with hasty single-color crayon strokes all lend salience ¹³ to the immaturity of the artist and thereby may also evoke the notion of childhood itself, which is a core element of his narrative. Artlessness, under the right circumstances, can be quite artful.

Moreover, the childish quality of the renderings may be further emphasized by the dissonant inclusion of peculiar, sophisticated details in a number of images. For example, in Frames 10 and 11 (Figure 3)—respectively depicting the scenes in the story where the girls scatter cuttlefish on the floor of the elevator and the mother returns to find the mess—among the very few visual details presented are floor numbers above the lift, the first of which, for the ground floor, is “illuminated.” Also illuminated is a triangle signaling that the lift is going up. This level of technical specificity is inconsonant with the relatively very simplistic, “low-modality” (Kress & van Leeuwen, 1996/2006; van Leeuwen, 2005) pictorial context in which it is found and may further accentuate the image’s association with childhood.

Figure 3.

Transitions between Frames 9 and 10, 10 and 11.

We would call attention to one other notable aspect of this collection of images: in all frames that depict people, 16 of the total 22, an expression conveying a particular emotional state clearly registers on each face. Once again, none of these faces entails more than some hurried and notional hair atop a circle enclosing two vertical strokes for eyes and single-line mouth. But much like the essentialized emoticons that they resemble, each face displays an expression of emotion that is so widely conventional as to be practically unambiguous, especially within its respective pictorial context. For instance, in Frame 4 (Figure 4), two stick figures with spiky hair lie on the top beds of two sets of bunk beds, while two stick figures with longer hair stand together in between them. The two spiky-haired figures display broad u-shaped grins, and the figures in the middle frown in seeming disappointment. Other resources are available to the viewer to help disambiguate the scene, but one would hardly need to read the speech bubble exclaiming “Hmph! So unfair,” for example, to glean a sense of disparity and dissatisfaction. (Again, interactions among modes are discussed at the conclusion of this Analysis section.) Likewise, in Frame 16 a longhaired stick figure, identical to the disgruntled figures in Frame 4, kneels before a crucifix in a gesture of prayer. Apart from the image of Christ on the cross, this is the only figure in the piece that is depicted with a straight horizontal line for a mouth. All other figures in the piece are depicted as smiling, frowning, or “shocked,” that is, agape, with a circular “open” mouth. By contrast, the horizontal line in Frame 16 would seem to suggest a neutral position; a serious, respectful solemnity, perhaps.

Figure 4.

Detailed images of Frames 4, 16, and 18.

Visual-kinetic

Jeremy’s story features only two effects that simulate camera movement, one in Frame 1 and one in Frame 20. The first effect is a slow “zoom in” on the title in the opening frame, and the second is a slow pan down on an image of his mother’s apartment building, with his mother as a child smiling in the window; the second image is a repeat of the image in Frame 2, but a close-up. These two effects visually bound the story. The first literally draws the viewer into the visual field of the multimedia text. The second signals the end of piece, focusing more closely now on the image of the mother in the window with which the story began and emphasizing the building itself by scanning the length of it; in this way, two core elements of the story, the mother and her old home, are simultaneously foregrounded, also visually “restating” the theme of the story.

In contrast to the sparing use of camera effects, the piece displays a wide array of transition effects, which visually segue from one image to the next. These include vertical and horizontal wipes, fades, cube spins, shrinking and expanding irises, and page peels. The only shifts between images that do not involve transition effects occur between Frames 1 and 2 (the title and the first hand-drawn image) and between Frames 20 and 21 (the last hand-drawn image and the start of the credits). The transition effects also visually bound the story, then, but with a visual “silence,” figuratively put. The absence of transitions also temporally coincides with the presence of the abovementioned camera effects, which further reinforces the sense of circumscription or boundedness at the beginning and end the narrative.

Beyond marking the boundaries of the narrative, the transitions also serve a semantic function. More than just a means of spanning between one image and the next, the special movement patterns of a number of the transitions deployed speak to the thematic content of the visuals they bridge. For example, consider the transitions made between Frames 9, 10, and 11. (Refer to Figure 3, once again.) Frame 9 depicts two stick figures, representing Jeremy’s mother and her sister, deciding to buy cuttlefish against the wishes of their mother. Again, Frame 10 is an image of the two figures scattering the cuttlefish in the lift, and Frame 11 shows the angry mother discovering it. The visual transition made between 9 and 10 involves a vertical wipe that moves from right to left. The transition from 10 to 11 is also a vertical wipe, but from left to right. Semiotically, these effects may cooperate with the images and language of the story to frame this incident as an episode within the story. The first wipe effect may provide the basis for actualizing a cluster of complementary meanings: its movement closely mimics the movement of an elevator door, and the unidirectional “entrance” of Frame 10 into the visual field parallels the entrance of the girls into the elevator, and into the complicating action. The second wipe also recalls an elevator door, but in reverse, which, when juxtaposed to the preceding wipe effect, may signify the escape made by the girls or even a more metaphorical “reversal of fortune.”

Another visual effect that might convey a semantic import, along with fulfilling its transition function, is the ripple effect that occurs between Frames 19 and 20. This kind of hazy ripple is a prototypical device, commonly used in movies and on television, for transporting the viewer to and from dream states, memories, fantasies, and alternate times and realities. Visually bookending this ripple effect in Jeremy’s story are a description of his mother’s netball experience as a child (Frame 19; corresponding to Lines 24-27 in the transcript above) and the present-time conclusion to the piece: “My mum had lived there for thirty years and sure has many memories” (Frame 20; Lines 27 and 28). The ripple effect aptly bridges these timeframes.

Visual-linguistic

Jeremy incorporates written text within the visual channel in his piece to perform several distinct semiotic functions. The most obvious use of written language is in the title and credits, Frames 1, 21, and 22. Titles, of course, conventionally serve the purpose of “naming” the entire story, all that is to follow, and credits acknowledge contributions of various kinds to the finished piece. Jeremy has chosen to represent these frames using only written text, with no background imagery or other augmentation. This stylistic sameness, a kind of “visual rhyme” (van Leeuwen, 2005), reinforces the sense of the narrative content of the piece being “book-ended” by metacommentary, that is, what the piece is to be called and by whose efforts it came about. The other (nearly) text-only slide is Frame 5, which shows the words presentable, cozy, and comfortable, in yellow curvy letters and encased in aurae of purple, green, and red, respectively. The background is a two-dimensional wood-grain surface, reminiscent of 1970s wall paneling. Here, Jeremy’s design may be seen to support his intended meaning in at least two ways. First the words are represented in the same color, yellow, which constructs a parallel relation among them, all being descriptors for his mother’s childhood home. Positive connotations though they each convey, they each represent a discrete quality, and this discreteness is constructed visually by the different-colored halos. In this way, Jeremy achieves a simultaneous, felicitous encoding of sameness and difference. (See Figure 5 for Frames 1, 21, and 5.)

Figure 5.

Frames 1, 5, and 21.

The hand-drawn images also integrate written language to different effects. These may be categorized under the general headings of labels, signs, and speech. Labels are used to disambiguate features within the visual field that may not be readily obvious, but are important enough to the narrative to merit particular attention. For example, in Frames 2 and 6, which, respectively, depict Jeremy’s mother’s housing block and similar blocks, at the base of each building is a rectangle that would likely be taken for a door were it not marked with the word lift (see Figure 6). Under other circumstances, it probably wouldn’t matter if these rectangles were perceived by an audience to be doors, elevators, or something else. In fact, it wouldn’t at all diminish the comprehensibility of Jeremy’s story to leave these lifts unlabeled. Yet, to do this would detract from the literary artfulness of Jeremy’s piece in that the lift plays a focal role in, or becomes a key context for, the action of the story, which is revealed in Frames 10 and 11, which depict the cuttlefish incident. Jeremy’s labeling device is, in this case, a visual-linguistic form of foreshadowing.

Figure 6.

Lift labels (Frames 2 and 6, with detail images of each at right).

By signs we mean labels placed within depicted contexts as they might actually appear in the real or imagined counterparts to these images. In some cases these signs correspond directly what is being said, such as in Frame 7, where Jeremy introduces the Indian sundries store his mother visited as a child and includes a sign above the counter that says “Indian Sundries Store.” Elsewhere, though, signs serve to remind the audience of details introduced earlier. In Frame 6, for example, a sign in front of a row of housing blocks reads, “Queenstown.” Jeremy does not say anything about Queenstown in this portion of the story; he mentioned it only once at the beginning of piece, during the appearance of Frame 2, in which no visual sign is depicted, interestingly. Written language in the form of speech, then, brings an affective, evaluative dimension to the largely neutral, factual telling of his mother’s story. In Frame 9, the girls exclaim, “Yay, Mummy not home!” and “Let’s go buy cuttlefish!” The “Yay” and exclamation points convey an explicit sense of youthful defiance and mischievousness that Jeremy’s spoken words, although quite similar in content and phrasing, do not. In these various ways, written language adds semiotic complexity to the narrative laminate and often, by design, bring the author’s overall message into sharper focus (see Figure 7, for detail images from Frames 7, 6, and 9).

Figure 7.

Written language as sign (Frames 7 and 6) and speech (Frame 9).

Intersemiotic Relations

As the reader may have already gathered from the above, there are patterns of potential meaning that may be seen to crosscut the discrete modes of communication deployed in Jeremy’s story. We discuss these as follows under the general headings of “structure” and “theme,” relating, respectively, to the multimodal organization of the story and the designs for communicating its intended messages.

Structure

In terms of its structure, Jeremy’s piece evidences a coherence built-upon alignment of the affordances of different copresent semiotic elements, which is to say that different parts of his digital story are used to make meanings that they most easily and effectively do, and in a way that supports other elements in doing the same. The most salient example of such structural coherence is in the abovementioned bounding of the story. Several devices signal the beginning and ending of the presentation of the narrative content of the piece complementarily. Recall that in the auditory mode, the volume of the background track is lowered at the start of the spoken narration and raised again to flag the start of the credit roll. Likewise in the visual mode, the camera movements, the zoom-in and pan-down effects, are used to mark the beginning and end of the story. The oral-linguistic content presented at corresponding points in the narrative comprises an “orientation,” in Labov’s (1972, 1997) phrase, and a “coda,” which respectively situate the action of the story at its onset and recapitulate its main point:

My mum used to live in an old three-room HDB flat on the fourth storey in Queenstown. (Lines 1 and 2; Frame 2)

My mum had lived there for thirty years and sure has many memories. (Lines 27 and 28; Frame 20)

These sentences are, by design, singled out, so to say, within the flow of speech by the ruptures created in the auditory and visual channels. In the manner of what Lanham (2006) has called “attention traps,” these devices alert the viewer to more consciously attend to these features, on which we, as inherently signifying creatures, are apt to consider what they might mean. We would emphasize, too, that these resources for expression only produce the effect of rupture because they are used strategically and sparingly. In this, Jeremy exemplifies one capacity of the expert designer of textual meaning (cf. Bearne & Wolstencroft, 2007): He demonstrates the awareness that in selecting from among available resources, what is unseen, or perceived as absent, can be just as critical to conveying a message as what is included and sensible. Of course, we do not mean to suggest here that Jeremy’s design is an entirely novel one; in point of fact, he draws, in original ways, on stylistic conventions commonly seen in films and on television. This is entirely appropriate and necessary, particularly given his declared intention of creating a story to be viewed and understood by his classmates and teacher as well as his mother. “Semiotic resources,” Kress (2010) explains, “are socially made and therefore carry the discernible regularities of social occasions, events and hence a certain stability.” But these resources are not only made in a fixed sense; they are “constantly remade” (p. 8), as Jeremy has done.

Theme

As we understand it, the main idea of Jeremy’s story is to recount where his mum frequently went in the past. But this summary belies the complexity that he accomplished in his piece via multimodal means. Again, the spoken-language portion of the piece is a fairly straightforward, almost dispassionate recounting of details and events from his mother’s childhood. However, the multimodal laminate, taken as a whole, represents, in a Bakhtinian sense, a richly layered diversity of voices and perspectives (Bakhtin, 1981). As mentioned above, the images bring a younger, more childlike quality to Jeremy’s early-teenaged voice and composed, somewhat stiff prose. Yet, as with the structural ruptures, there are sophisticated details within the drawings that also provide clues that this childhood is only invoked and imagined. As well, the language and facial expressions within the images, like a Greek chorus of sorts, punctuate the evenly told narrative with affect, emotional exclamations, and evaluative commentary. The lyrics of the background track, “I won’t go home without you,” compel the viewer to do some analogical or metaphorical semiotic work of her own and to integrate these meanings within an understanding of the whole. The transition effects, too, extend to the viewer a semiotically varied range of opportunities for deepening the meaning potential of the other resources presented, to actualize, for example, the mimetic or iconic quality of the elevator door-like movements between Frames 9 and 11 and the conventional symbolism of the flash-forward ripple effect between Frames 19 and 20. All of these individual elements, whether they suggest similar or different interpretations, comprise a collection of semiotic resources with the potential to deepen and complicate the content of Jeremy’s story, and convey his communicative intent, in a multiplicity of ways.

Understanding and Assessing Multimodal Meaning Design: Discussion and Recommendations

Our expectation is that the analysis presented above specifically attests to the multimodal intricacy and ingenuity of Jeremy’s composition, which largely went unnoticed and unassessed. All the same, a crucial question to deal with immediately is whether much of the above analysis should be discounted as the result of “reverse engineering,” or the ex post facto invention and imposition of patterns of meaning that the author did not intend. To be clear, authorial intention is a key component of multimodal literacies, as we discuss shortly. But to leap to questions of intention and its likely extent is to miss a necessary point.

We engage multimodal literacies where the imaginative limits of the text producer and the interpreter intersect: again, both are designers and producers of meaning (cf. Kress, 2003, 2010), and so any interpretation made of a multimodal text is a fundamentally collaborative achievement. What we should first notice and appreciate, then, is not whether each of the patterns of multimodal connection discussed above were fully intended, but rather that they are discernible. While we, as researchers of New Literacies, may be specially motivated and attuned to actualize multimodal patterns of meaning in text, richly layered meanings are not constructed of thin air, as it were. We could certainly not have explicated any significance in Jeremy’s piece in the detailed way that we have done if his selection and orchestration of resources did not afford, and even suggest, these interpretations. So altogether deliberate or not (we do not know and cannot measure), we admit a notable aptness and elegance in Jeremy’s meaning design, borne out in patterns of potential meaning that are sensible and salient in the textual product. This admission begs a further explanation for how and why these qualities were not duly reflected in the original assessment of his work.

Revisiting Kim’s classroom practices, the narrative writing UOW had to address a wide set of learning objectives over a relatively short period of time. Kim’s approach quickly demonstrated exemplary performance in oral storytelling in terms of voice, fluency, articulation, pronunciation, expressiveness and the use of images/manipulative elements to enhance or decorate the story-listening experience. But leaving aside what Kim already knew about literacy and its assessment, it is difficult to see how her students’ meaning-design work could be supported by and woven through the technical substrate of her assessment rubric (Appendix A).

To what, then, should we attribute this performance–assessment mismatch? It is certainly not due to lack of interest in multimodality on the part of education policy makers. Multimodal meaning making is an increasingly important priority in language arts education in Singapore, evidenced by the inclusion of “viewing” and “representing” alongside the traditional four language skills (reading, writing, listening, and speaking) as assessable components of the 2010 National English Language Syllabus (Singapore Ministry of Education, 2008). Of course, as with other “accountability”-obsessed educational cultures, there is a “hidden curriculum” (Apple, 1982) too, which tends to sideline the creative and communicative aspects of learning with new media and prioritizes “fundamental” skills-building and pen-and-paper tests (cf. Tan & McWilliam, 2009). But the primary reason, we conclude, for the inventiveness of Jeremy’s design going unnoticed is that understandings of “viewing” and “representing,” conceptual and practical, in Singapore as elsewhere, are vastly underdeveloped on the parts of teachers and other stakeholders. Teachers cannot be expected to engender or assess these critical new-media literacy capacities if they do not know what they comprise and how to recognize and interpret their realizations in instances of practice. But this is much easier said than done. In truth, though a number of academics and policy makers have developed helpful frameworks for understanding the conceptual and practical entailments of so-called 21st century skills and new literacies (e.g., Kimber & Wyatt-Smith, 2009; C. Lemke, Coughlin, Thadani, & Martin, 2003; Partnership for 21st Century Skills, 2011), there is as yet no clear consensus as to what the requisite “new knowledges and core capacities,” to apply Kimber and Wyatt-Smith’s phrase, for success in work and life in the present and future may be. Moreover, with particular respect to multimedia communication and multimodal composition, there are no universally applicable or generic strategies for approaching a meaning-design problem. As Kress (2003) explains, multimodal meaning design is a fundamentally situated response to questions of “what is needed now, in this one situation, with this configuration of purposes, aims, audience, and with these resources, and given my interests in this situation” (p. 49, italics in original).

Looking forward, then, we return to the notion of intention and its role in multimodal literacy learning and practice. To recall, we maintain that intention is neither a necessary nor a sufficient condition for assessing performance on a multimodal literacy task. However, intention, which we discuss in the narrower sense of “semiotic awareness” (Nelson, 2006, 2008; Nelson & Kern, 2012), plays a integral role in promoting the salience in communication of particular meaning designs, as well as in engendering the continued development of multimodal literacy itself. Semiotic awareness may be understood as an alertness to the representational possibilities that any form can afford, in which contexts and for whom and how and why. It generally follows, too, that the extent to which a new media designer, like Jeremy, is semiotically aware will predict the degree to which his design choices will articulate across different modes and media, befit purposes at hand, and speak to the experiential bases and interests of given audiences, who must similarly engage their own semiotic awareness capacities toward actualizing the intended meanings.

So we say that the development of semiotic awareness is at the heart of multimodal literacies—to be multimodally literate, again borrowing Buckminster Fuller’s fitting phrasing, is to be a “comprehensive anticipatory design scientist” (Buckminster Fuller Institute, 2010, emphasis added). While good design is no less good when it comes about unconsciously or accidentally, consistently successful designers approach any design task deliberately, always judging the utility of a range of available resources (e.g., words, pictures, concepts, processes) in view of changing communicational, practical, and aesthetic needs. And just as with any form of iterative, conscious practice, with experience the designer may gain a keener and keener sensitivity to the meaning-making potentials and regularities and combinative possibilities of the concepts, forms, materials, and tools she uses—a process, that is, of expanding literacy capacities. Urgently, this is a process that young people and their teachers must begin and sustain if school-based learning is to adequately address communicational needs and developments in the wider digitally supported world.

Even so, how does one even begin to recognize the dynamic meaning potentials of diverse multimodal resources in communicative use? (To a person with only a hammer, as the saying goes, everything tends to look like a nail.) One approach is to practice deconstructing students’ multimodal compositions, that is, reverse engineering designs of meaning as we have done here. This is a means by which educators, like Kim, might hone their own semiotic sensitivities toward reforming their assessment measures to more fully capture and credit the qualities of meaning design work that students do, with the presumed positive washback effect of involving their students in more meaningful, relevant, transparent design tasks. When students, for their part, begin to see and understand their own work (and that of their peers) in critical semiotic terms, they become better able to recognize the meaning-making affordances of diverse resources with respect to their expressive needs, to anticipate others’ interpretations of their design choices, and to verbalize their design intentions. As these capacities increase, the imaginative limits of student and teacher come into fuller contact, overlap. Playing out the spatial metaphor, the more one’s field of vision (namely semiotic awareness) expands, the more likely it is that one will gain a greater perspective onto what another sees, especially if the purview of that other is also expanding. Literally, as student and teacher each become more semiotically aware, the meaning designs (compositions and interpretations) that they each construct more likely suppose and suggest common interpretive frames. A presumable knock-on effect, then, is that the validity and reliability of multimodal literacy assessments are augmented, inasmuch as the nature of the meaning-design exercise is clarified and agreed on by teacher and student, and each party can more consciously, critically anticipate and appreciate the meaning-making work of the other.

Concluding Remarks

In this paper we have attempted to show what a successful multimedia literacy performance looks like and how one might begin discerning its characteristics. These matters, we contend, operate beyond familiarity and comfort with digital tools and new media, and therefore require the development and application of an assessment approach that draws meaning from context, and views creativity and coherence in terms of the transformation of available designs.

While we fully acknowledge the limitations of our work in terms of only considering one case and in barely penetrating the very complex issue of how best to understand and assess multimodal literacies, we nonetheless hope we have accomplished our primary aims of (a) specifically demonstrating the need to look differently at and think differently about the meaning design work that children and youth do and (b) sketching some core conceptual and practical considerations for such alternative looking and thinking, toward a broader revisioning of school-based literacy assessment. As with all case study work, we leave it to others to determine the trustworthiness of our methods and transferability of our findings to different contexts.

That said, we clearly see, evidenced in Jeremy’s case, that 21st century communication, and accordant pedagogical priorities, demand that meaning making itself be at the center of literacy learning and teaching, as the New London Group (1996), Kress (2003, 2010), Gee (2004) and like-minded colleagues have espoused for now well over a decade. But turning the logical force of this argument into real, practical traction, we feel, needs more concrete delineation of the nature and implications of multimodal meaning-design problems in classrooms, along with more practicable, adaptable approaches for solving these problems and evaluating the solutions.

With all of the assessment and other tasks competing for her attention, we are not so naïve as to recommend that Kim would better have spent weeks painstakingly transcribing and analyzing each one of her students’ pieces to appropriately evaluate them. In real-world educational institutions, there are few maxims, if any, whose violation can quash pedagogical reform faster than feasibility. Likewise, we understand that for Kim, quantitative assessment of students’ work was not merely efficient, but also compulsory. Dictated by the Singapore Ministry of Education (and increasingly common globally), numerical marks were the measure by which most significant decisions as to student placements, rankings, and futures were made. But even given such constraints, through alertness and practice, teachers like Kim can develop a feel for qualities of meaning design, not unlike, for example, the integrative appreciation that guides an experienced writing teacher in assessing the different aspects of a student’s narrative; her global sense of the writer’s craft informs her appraisal of the mechanical, structural, and literary components. And these more particular appraisals feed back into iterative refinements of the initial, overall impression. So too can it be with assessment of multimodal literacies.

When teachers and students become sensitized to meaning relationships, they become better able to take in and then deconstruct the multimodal gestalt, that is, to judge what semantic contributions each part can make to other parts and to the whole. Quantitative assessment is not best disposed to capturing complex, nuanced qualities of relation, as we hope we have shown, but we also believe it can be provisionally accommodated. The current phase of our project, working “backward” from the analysis of pieces like Jeremy’s, involves devising and testing rubrics that describe types and levels of semiotic connection to facilitate more designful, but still efficient assessment of digital multimodal texts. We will report on this work in due course, but, for the present, we underscore the preliminary importance of semiotic awareness.