Words allow us to communicate and interact successfully, but they don’t do it alone! In conversation, words are just one dimension of a highly complex system of communication, which can include the position and movement of our bodies and hands, the expressions we make with our faces, the pitch, melody, and timbre of our voices, and the physical context in which we are surrounded. Speakers make use of all these channels in coordination when making meaning. One of the avenues my research interests have taken explores how these different modalities work together.

Multimodal communication is particular relevant for quotation and enactment, which is when we use our voice and bodies to present talk and action that (we are suggesting) originally belonged to, or was produced by, some other individual. Quotation is a big part of story telling, and is also important for teaching. This line of research has made use of methodologies drawn from Conversation Analysis, learned while pursuing my masters in Linguistics at the University of Colorado, Boulder. Analyzing the conversational context in which these multimodal quotations are embedded, I’ve looked at how they become a meaningful part of interactions from music pedagogy to text messaging. Drawing our whole bodies in the acting out of previously experienced, (or sometimes mock), bits of talk or action allows us to show rather than describe, a powerful tool in communication.

A bit of transcript demonstrating a nonverbal vocalization, along with images of the co-produced gesture.

Related Publications

Tolins, J. (2013). Assessment and direction through non lexical vocalizations in music instruction. Research on Language and Social Interaction, 46, 47-64.

Tolins, J., & Samermit, P. (under review). Quotative enactment in text-mediated conversations.

See also a guest blog on the joys and challenges in working with multimodal communication for the ROLSI blog:

Presenting multi-modal analyses: Capturing and communicating nonverbal behaviors