This edition first published 2019
© 2019 John Wiley & Sons, Inc.
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by law. Advice on how to obtain permission to reuse material from this title is available at http://www.wiley.com/go/permissions.
The right of Sigrid Norris to be identified as the author of this work has been asserted in accordance with law.
Registered Office
John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, USA
Editorial Office
The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, UK
For details of our global editorial offices, customer services, and more information about Wiley products visit us at www.wiley.com.
Wiley also publishes its books in a variety of electronic formats and by print‐on‐demand. Some content that appears in standard print versions of this book may not be available in other formats.
Limit of Liability/Disclaimer of Warranty
While the publisher and authors have used their best efforts in preparing this work, they make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives, written sales materials or promotional statements for this work. The fact that an organization, website, or product is referred to in this work as a citation and/or potential source of further information does not mean that the publisher and authors endorse the information or services the organization, website, or product may provide or recommendations it may make. This work is sold with the understanding that the publisher is not engaged in rendering professional services. The advice and strategies contained herein may not be suitable for your situation. You should consult with a specialist where appropriate. Further, readers should be aware that websites listed in this work may have changed or disappeared between when this work was written and when it is read. Neither the publisher nor authors shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.
Library of Congress Cataloging‐in‐Publication Data
Names: Norris, Sigrid, 1961– author.
Title: Systematically working with multimodal data : research methods in multimodal discourse analysis / Sigrid Norris.
Description: Hoboken, NJ : John Wiley & Sons, 2019. | Includes bibliographical references and index. |
Identifiers: LCCN 2018027393 (print) | LCCN 2018058184 (ebook) | ISBN 9781119168331 (Adobe PDF) | ISBN 9781119168348 (ePub) | ISBN 9781119168317 (hardcover) | ISBN 9781119168324 (pbk.)
Subjects: LCSH: Discourse analysis–Technology. | Discourse analysis–Study and teaching. | Modality (Linguistics)
Classification: LCC P302.865 (ebook) | LCC P302.865 .N67 2018 (print) | DDC 401/.41–dc23
LC record available at https://lccn.loc.gov/2018027393
Cover Design: Wiley
Cover Image: © Peshkova/iStock.com
To Alan
| Figure 2.1 | Mediated action: feeding a baby dragon. |
| Figure 3.1 | Ethical considerations: people have a basic right not to take part in a study. |
| Figure 3.2 | Recording in public places where we were not interested in particular individuals’ actions: no need for consent. |
| Figure 3.3 | A specific individual recorded in public places: a need for consent. Reproduced with permission of the participant. |
| Figure 3.4 | Position of researchers and camera in relation to the video recorded (inter)action. |
| Figure 3.5 | Snapshot of screen on the day the YouTube clip was selected as data piece. |
| Figure 3.6 | The researcher as viewer and multimodal analyst of the video. |
| Figure 3.7 | Diagram 1 – birdseye view of the table and computer set up. |
| Figure 3.8 | Position of the researcher in room A.2 |
| Figure 3.9 | Position of the researcher in room B.2 |
| Figure 3.10 | A view of all collected data of one dyadic team working via video conferencing technology, collated, synchronized, and time stamped. Reproduced with permission of the participants. |
| Figure 3.11 | Position of the researchers while recording Family 2. Reproduced with permission of the participants. |
| Figure 3.12 | A view of all collected data of Family 2’s video conferencing (inter)action, collated, synchronized, and time stamped. Reproduced with permission of the participants. |
| Figure 4.1 | The site of engagement that embraces the researcher(s) and the data set. |
| Figure 4.2 | Screenshot of a representative data piece. |
| Figure 4.3 | The site of engagement that embraces the researcher and a data piece. |
| Figure 4.4 | The site of engagement of the data set: YouTube clip from Ellentube on 2 July 2016. |
| Figure 4.5 | The site of engagement of a YouTube video data set that includes the researcher as relevant social actor. |
| Figure 4.6 | A data piece: Kai Sings “Cake by the Ocean”. |
| Figure 4.7 | The site of engagement of a researcher/viewer watching/analyzing the clip Kai Sings “Cake by the Ocean”. |
| Figure 4.8 | Sub‐data set: comprising the large scale higher‐level mediated action of one dyad taking part in the experimental study with many smaller scale higher‐level actions. |
| Figure 4.9 | Site of engagement of the entire data set including the researcher(s). |
| Figure 4.10 | Data: a screenshot of a dyad working on a task via video conferencing technology. Reproduced with permission of the participants. |
| Figure 4.11 | Site of engagement: researcher analyzing the recorded (inter)action of a team working on a task via video conferencing technology. Reproduced with permission of the participants. |
| Figure 4.12 | Sub‐data set: comprising the large scale higher‐level mediated action of one family taking part in the video ethnography with many smaller scale higher‐level mediated actions. |
| Figure 4.13 | Site of engagement of the entire data set including the researcher(s). |
| Figure 4.14 | A site of engagement that embraces the researchers and a one‐family data set. |
| Figure 4.15 | Data: a screenshot of a recorded video conferencing (inter)action. Reproduced with permission of the participants. |
| Figure 4.16 | Site of engagement: researcher analyzing the recorded video conferencing (inter)action. Reproduced with permission of the participants. |
| Figure 5.1 | A narrow site of engagement for micro analysis example YouTube. |
| Figure 5.2 | A narrow site of engagement for micro analysis teamwork example. Reproduced with permission of the participants. |
| Figure 5.3 | A narrow site of engagement for micro analysis example family video conferencing. Reproduced with permission of the participants. |
| Figure 6.1 | Transcript of the broader layout of the stages in the YouTube clip Kai Sings “Cake by the Ocean.” |
| Figure 6.2 | Layout of the first 40 seconds in the video clip – camera angles. |
| Figure 6.3 | Layout transcript for the first 6 seconds, YouTube example. |
| Figure 6.4 | Proxemics transcript for the mother of the triplets. Reproduced with permission of the participants. |
| Figure 6.5 | Upper‐body posture transcript for the man in the dyad. Reproduced with permission of the participants. |
| Figure 6.6 | Leg and foot movement transcript for the man in the dyad. Reproduced with permission of the participants. |
| Figure 6.7 | Noting times and actions. |
| Figure 6.8 | Gesture transcript of the man on the right in the images. Reproduced with permission of the participants. |
| Figure 6.9 | Gaze transcript for the woman with the pony tail. Reproduced with permission of the participants. |
| Figure 6.10 | Head‐movement transcript (Images 1–8) for the woman with the pony tail. Reproduced with permission of the participants. |
| Figure 6.11 | Head‐movement transcript (Images 9–18) for the woman with the pony tail. Reproduced with permission of the participants. |
| Figure 6.12 | Facial expression transcript. Reproduced with permission of the participants. |
| Figure 6.13 | Object handling transcript for the man in the dyad. Reproduced with permission of the participants. |
| Figure 6.14 | Spoken language transcript overlaid over the top of the gesture transcript. Reproduced with permission of the participants. |
| Figure 6.15 | Written language/image transcript. Reproduced with permission of the participants. |
| Figure 6.16 | Producing a final transcript: gaze and head‐movement transcript collated. Reproduced with permission of the participants. |
| Figure 7.1 | Natural experienced time cycles and rhythms. |
| Table 3.1 | Data collection table. |
| Table 3.2 | Sample of a data collection table for our YouTube example. |
| Table 3.3 | Sample of a data collection table for our experimental study. Reproduced with permission of the participants. |
| Table 3.4 | Sample of a data collection table for our video ethnographic study. Reproduced with permission of the participants. |
| Table 4.1 | Data set table. |
| Table 4.2 | The data set: comprising three dyads working on two tasks each, plus individual interviews clearly showing the length of the recordings, as well as notes. Reproduced with permission of the participants. |
| Table 4.3 | (Part of) a full data set: comprising participants (adults and children), their relationships, devices, and animals present in the video conferencing project as well as the length of the recordings. Reproduced with permission of the participants. |
| Table 5.1 | Higher‐level mediated action table of an imaginary data set. |
| Table 5.2 | Bundled higher‐level mediated action table of an imaginary data set. |
| Table 5.3 | Higher‐level mediated action table YouTube example. |
| Table 5.4 | Incorrect higher‐level mediated action table YouTube example. |
| Table 5.5 | Color coded bundles of higher‐level mediated action table YouTube example. |
| Table 5.6 | Bundled higher‐level mediated action table YouTube example. |
| Table 5.7 | Higher‐level mediated action table teamwork example. Reproduced with permission of the participants. |
| Table 5.8 | Incorrect higher‐level mediated action table teamwork example. Reproduced with permission of the participants. |
| Table 5.9 | Color coded bundles of higher‐level mediated action table teamwork example. Reproduced with permission of the participants. |
| Table 5.10 | Bundled higher‐level mediated action table teamwork example. Reproduced with permission of the participants. |
| Table 5.11 | Higher‐level mediated action table video ethnography example. Reproduced with permission of the participants. |
| Table 5.12 | Incorrect higher‐level mediated action table video ethnography example. Reproduced with permission of the participants. |
| Table 5.13 | Color coded bundles of higher‐level mediated action table video ethnography example. Reproduced with permission of the participants. |
| Table 5.14 | Bundled higher‐level mediated action table video ethnography example. Reproduced with permission of the participants. |
The book has taken a substantial amount of time to write and I thank the editors at Wiley for their patience. Many people have been helpful along the way, but if I was naming everyone who had had an impact upon the many pages before you, these acknowledgments would turn into the longest chapter of the book. I think most people who have had contact with me over the last few years, who have asked a question with regard to multimodal data collection, data analysis, reliability of qualitative research, or the like, would have had an impact upon my writing of this book. Here, I would like to thank all of you for your interest, curiosity, and trust in me having an answer to these questions. I do hope that you will find at least some of your answers in the pages ahead and I hope that you will find the book useful. I know that I have promised many a reader that I would answer questions, that I would write a book that explained how I work with data, and I hope that you will not be disappointed.
There are also a few people who I will name and first, some members of the Multimodal Research Center, who have had the greatest impact upon my writing and who deserve special thanks. I thank Tui Matelau, Yulia Khan, and Ivana Rajic for their questions as they were working with the phases and steps and their support in writing this book; a particular thank you goes to Jesse Pirini, who read very early drafts of some sections and who found my writing of the how‐to sections for data analysis incredibly useful. Special thanks also go to Jarret Geenen, who visited the Center in early 2017 and was a catalyst in my explaining some theory in great detail. Both Jarret’s input with regard to the need for theory to be covered and Jesse’s input during this early writing stage of phases and steps solidified my thinking. I would also like to thank Elina Tapio and Chloe Grace Fogarty‐Bourget, who were visiting researchers at the Multimodal Research Centre and tried out some of the phases and steps for data analysis. They both had valuable feedback and questions with regard to the systematic approach. Further, I would like to thank Edgar Bernad‐Mechó, another visiting scholar, who also tried out some phases and steps and who emphasized that I needed to include not only data analysis, but also to outline data collection and transcription in the same book.
Of course, all participants, in our projects have been of utmost importance. Without participants, we cannot do our work. But here, I am most grateful to the participants who are featured in this book. Every participant and every family featured has viewed the videos, tables, and images and permitted them to be published as shown. Featuring color images of real participants and being able to use excerpts of our data to give students and researchers hands‐on experience makes this book special. Therefore, I would like to thank the following participants of the studies for their permission to publish their images, names and videos as part of this book: Abbie O’Rourke, Ali, the Budd Family and the Charko Family, Cameron Fraser, Dunja Vajsakovic, Isla Connors, Dina‐Sara Vajsakovic, Edgar Bernad‐Mechó, Hana Mlinac, Ivana Mlinac, Ivana Rajic, Jay Nam, Jake Connors, Jo Connors, Luke Norris, Melanie, Michael Evans, Rachelle Ferguson, Shymala Sidharth, Sophie Connors, and Brownderbag Tagaloa.
I would like to thank the Faculty of Design and Creative Technologies, the School of Communication Studies, and the AUT Multimodal Research Centre at Auckland University of Technology in New Zealand for funding the projects and making the writing of this book possible.
I would also like to thank Freiburg Institute for Advanced Studies (FRIAS), University of Freiburg, Germany and the People Programme (Marie Curie Actions) of the European Union's Seventh Framework Programme (FP7/2007–2013) under REA grant agreement no. [609305] for making the completion of this book possible.
Then, there is family. Family is always most important for my writing. Luke and Kevin, with their humor and light‐heartedness as well as the many beautifully prepared meals, particularly by our nutritionist, Luke, always helped me to switch off. Alan, as always, has been more supportive than I can ever explain. He has accompanied me to many destinations from New Zealand to Germany, a ski resort in Austria, a monastery in Spain, the hills of Tuscany, Sicily, Florence, and Rome in Italy, to Chamonix as well as around Nice in France, to Los Angeles and San Francisco in the USA, to Budapest in Hungary, to Stockholm and Lingköping in Sweden, or to Aarhus and Aalborg in Denmark, all the while bearing with me as I was trying to write this book. He sought out hotels that allowed me to continue my writing and has been the kindest companion, who only once stopped me from finishing a section. This was in San Sebastián, where I was typing away, while he had been out and around town. When he came back to the hotel, I was still typing away, but he looked at me earnestly and requested I stop writing and have a look at the beautiful city. How right he was! We had an unforgettable afternoon and evening in San Sebastián and when we left the next morning, I was still happy that I had not finished the section. Now, whenever I come across the section that I did not finish in San Sebastián, I remember our wonderful time there. But this is only one beautiful memory during my book‐writing travels and for me, Alan is visible in many parts of this book and I dedicate this book to him.
While so many people have had an impact upon my writing of this book, all shortcomings, of course, are my very own. I do hope that the book will be useful to many students, teachers, and researchers alike and I hope you will enjoy the many examples and forgive the shortcomings.
This book is accompanied by a companion website:
www.wiley.com/go/Norris/multimodal‐data
The website includes:
