Sage Journals: Discover world-class research

Abstract

This article presents the design and functionalities of interactive software for multimodal analysis currently being developed in the Multimodal Analysis Lab, Interactive Digital Media Institute (IDMI) at the National University of Singapore. The software is being used for the annotation, analysis, search and retrieval of semantic patterns in unified but complex semiotic acts – for example, the interaction of gesture, gaze, intonation, camera angle, and music in a film. In addition to providing a digital platform for multimodal analysis, the software provides the site for further development of multimodal theory as the analytical techniques and tools produce insights into the nature of the multimodal phenomena. The approach is located within the digital humanities paradigm that promotes the use of computer techniques and technologies for humanities, arts, and social science research.

Keywords

climate change debate interactive software intersemiosis multimodal analysis social semiotics

Get full access to this article

View all access options for this article.

References

Bateman

(2008) Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal Documents. Basingstoke: Palgrave Macmillan.

Bateman

Schmidt

K.-H.

(2011) Multimodal Film Analysis: How Films Mean. London: Routledge.

Berry

D.M.

(2011) ‘The Computational Turn: Thinking about the Digital Humanities’, Culture Machine 12: 1–22.

Boeriis

Holsanova

(2012) ‘Tracking Visual Segmentation: Connecting Semiotic and Recipient Perspectives’, Visual Communication 11(3), this issue.

Boykoff

M.T.

(2011) Who Speaks for the Climate? Making Sense of Media Reporting on Climate Change. New York: Cambridge University Press.

Boykoff

M.T.

Boykoff

J.M.

(2004) ‘Balance as Bias: Global Warming and the US Prestige Press’, Global Environmental Change 14: 125–36.

Budd

Craig

Steinman

(1999) Consuming Environments: Television and Commercial Culture. New Brunswick, NJ: Rutgers University Press.

Carvalho

(2007) ‘Ideological Cultures and Media Discourses on Scientific Knowledge: Re-Reading News on Climate Change’, Public Understanding of Science 16(2): 223–43.

Cheong

Y.Y.

(2004) ‘The Construal of Ideational Meaning in Print Advertisements’, in O’Halloran

K.L.

(ed.) Multimodal Discourse Analysis: Systemic-Functional Perspectives, pp. 163–95. London: Continuum.

10.

Halliday

M.A.K.

Greaves

W.S.

(2008) Intonation in the Grammar of English. London: Equinox.

11.

Halliday

M.A.K.

Matthiessen

C.M.I.M.

(2004) An Introduction to Functional Grammar, 3rd edn. London: Arnold.

12.

Kress

Van Leeuwen

(2006) Reading Images: The Grammar of Visual Design, 2nd edn. London: Routledge.

13.

Montgomery

(2008) ‘The Discourse of the Broadcast News Interview’, Journalism Studies 9(2): 260–77.

14.

O’Halloran

K.L.

. (2011) ‘Multimodal Analysis within an Interactive Software Environment: Critical Discourse Perspectives’, Critical Discourse Studies 8(2): 109–25.

15.

Podlasov

Tan

O’Halloran

K.L.

(in press 2012) ‘Interactive State-Transition Diagrams for Visualization of Multimodal Annotations’, Intelligent Data Analysis: An International Journal 16(4).

16.

Rohlfing

. (2006) ‘Comparison of Multimodal Annotation Tools: Workshop Report’, Gesprächsforschung, Online-Zeitschrift zur verbalen Interaktion (7): 99–123.

17.

Royce

(1998) ‘Synergy on the Page: Exploring Inter-Semiotic Complementarity in Page-Based Multimodal Text’, JASFL Occasional Papers 1(1): 25–49.

18.

Smart

(2011) ‘Argumentation across Web-Based Organizational Discourses: The Case of Climate Change’, in Candlin

C.N.

Sarangi

(eds) Handbook of Communication in Organisations and Professions, pp. 363–86. Berlin: Mouton de Gruyter.

19.

Smith

B.A.

. (2011) ‘Analyzing Multimodality in an Interactive Digital Environment: Software as Metasemiotic Tool’, Social Semiotics 21(3): 359–80.

20.

Unsworth

(2006) ‘Towards a Metalanguage for Multiliteracies Education: Describing the Meaning-Making Resources of Language–Image Interaction’, English Teaching: Practice and Critique 5(1): 55–76.

21.

Van Leeuwen

(1999) Speech, Music, Sound. London: Macmillan.

Interactive software for multimodal analysis

Abstract

Keywords

Get full access to this article

References