Mimetic schemas in multimodal contact-establishing communication
Contact-establishing is deservedly characterized as the major function of communication. Until recently, it has been mostly subjected to linguistic analysis aimed at identifying its discursive markers. Meanwhile, contact-establishing frequently appears in gesturing. The current work develops a cognitive view to gesture and speech alignment, and addresses multimodal contact-establishing communication as mediated by mimetic schemas or bodily schemas shared by communicants who engage in face-to-face and body-to-body interaction. Based on a multimodal experiment where participants engage into task-oriented expository dialogue, we identified two most common contact-establishing recurrent gestures, palm-up-open-hand (PUOH) and palm-down-open-hand (PDOH) gestures contingent on two mimetic schemas, SHOW and RESTRAIN. In the study, we explore the distribution of these two schemas in their sub-schemas in gesture and speech. Following the participants’ contact-establishing PUOH and PDOH gestures (manifesting SHOW and RESTRAIN sub-schemas) and verbal cues (cognitive, pragmatic, and functional semantic dimensions), we determined the multimodal alignment patterns mediated by SHOW and RESTRAIN mimetic schemas in contact-establishing communication. Additionally, the clusters of PUOH and PDOH contact-establishing gestures were determined via their linguistic correspondences since they account for common thinking-for-speaking growth points and the language profiles of PUOH and PDOH contact-establishing gestures. The results allowed to scale the mimetic sub-schemas as manifesting event and referent features such as situatedness, embodiment, performativity, referent definiteness, referent foregrounding, reification, dynamicity, addressing, agentivity, referentiality. Overall, we established that in cognitive dimension non-situated events prevail in PUOH gestures while situated events prevail in PDOH gestures. In pragmatic dimension there is the difference in constativity in PUOH gestures and in performativity in PDOH gestures. In functional semantic dimension PDOH gestures commonly occur with acts, while PUOH gestures are more frequented with attributes. Additionally, we identified that within-cluster distance in PDOH gestures is more obvious than in PUOH gestures, which signifies that the typological differences (mediated indirectly by linguistic characteristics) in PDOH gestures are more distinct. The data obtained provide new evidence in multimodal contact-establishing communication.
Figures
Kiose, M. I., Leonteva, A. V., Agafonova, O. V. and Petrov, A. A. (2024) Mimetic schemas in multimodal contact-establishing communication, Research Result. Theoretical and Applied Linguistics, 10 (2), 46-70. DOI: 10.18413/2313-8912-2024-10-2-0-3
While nobody left any comments to this publication.
You can be first.
Grishina, E. A (2012). Pointing by hand as a system (based on the multimedia Russian corpus), Topics in the Study of Language, 3, 3–50. (In Russian)
Iriskhanova, O. K. (2017). On thinking for communicating: the multimodal dimension of aspectuality. Cognitive Studies of Language, 30, 57–60. (In Russian)
Iriskhanova, O. K. and Nikolayeva, A. I. (2023). Cognitive basis for the variation of recurrent gestures in explanatory discourse: palm-up-open-hand and palm-down-open-hand gestures. Voprosy Kognitivnoy Lingvistiki, 4, 26–34. https://doi.org/10.20916/1812-3228-2023-4-26-34 (In Russian)
Iriskhanova, O. K., Kiose, M. I., Leonteva, A. V. and Agafonova, O. V. (2022). Multimodal spatial deixis in speech and gestures: frames of reference in explanatory discourse, Voprosy Kognitivnoy Lingvistiki, 4, 17–31. https://doi.org/10.20916/1812-3228-2022-4-17-31 (In Russian)
Kibrik, A. A. and Podlesskaya, V. I. (2009). Night Dream Stories: A Corpus Study of spoken Russian discourse. Languages of Slavonic Culture, Moscow, Russia. (In Russian)
Kiose, M I. (2021). Multimodality as a research technology, in Multimodal dimensions of discourse, Languages of Slavonic Culture, Moscow, Russia, 33–62. (In Russian)
Lyashevskaya, O. N. and Kashkin, E. V. (2015). Types of information about lexical constructions in Russian FrameBank. Trudy instituta russkogo jazyka im. V.V. Vinogradova, 6, 464–555. (In Russian)
Paducheva, E. V. (2008). Vyskazyvanie i ego sootnesennost' s dejstvitel'nost'ju: Referencial'nye aspekty semantiki mestoimenij [Utterance and it’s alignment with reality: Referential aspects of the semantics of pronouns], LKI, Moscow, Russia. (In Russian)
Alibali, M. W., Spencer, R. C., Knox, L. and Kita, S. (2011). Spontaneous gestures influence strategy choices in problem solving. Psychological science, 22(9), 1138–1144. https://doi.org/10.1177/0956797611417722 (In English)
Bavelas, J. B., Chovil, N., Coates, L. and Roe, L. (1995). Gestures specialized for dialogue. Personality and Social Psychology Bulletin, 21(4), 394–405. DOI: https://doi.org/10.1177/0146167295214010 (In English)
Beilock, S. L. and Goldin-Meadow, S. (2010). Gesture changes thought by grounding it in action. Psychological science, 21(11), 1605–1610. DOI: https://doi.org/10.1177/0956797610385353 (In English)
Bressem, J., Ladewig, S. H. (2011). Rethinking gesture phases: Articulatory features of gestural movement? Semiotica 184–1/4, 53–91. https://doi.org/10.1515/semi.2011.022 (In English)
Chu, M., Meyer, A., Foulkes, L. and Kita, S. (2014). Individual differences in frequency and saliency of speech-accompanying gestures: The role of cognitive abilities and empathy. Journal of Experimental Psychology: General, 143(2), 694–709. https://doi.org/10.1037/a0033861. (In English)
Cienki, A. J. (2013). Image schemas and mimetic schemas in cognitive linguistics and gesture studies. Review of Cognitive Linguistics, 11(2), 417–432. https://doi.org/10.1075/rcl.11.2.13cie (In English)
Cienki, A. (2017). Ten lectures on Spoken language and Gesture from Perspective of Cognitive Linguistics. Issues of Dynamicity and Multimodality. Brill, Leiden, Boston, the Netherlands, USA. (In English)
Cienki, A. (2021). From the finger lift to the palm-up open hand when presenting a point: A methodological exploration of forms and functions. Languages and Modalities, 1, 17–30. https://doi.org/10.3897/lamo.1.68914 (In English)
Cienki, A. and Müller, C. (2008). Metaphor, gesture, and thought, in Gibbs, R. W. Jr. (Ed.), The Cambridge handbook of metaphor and thought, Cambridge University Press, Cambridge, UK. P. 483–501. https://doi.org/10.1017/CBO9780511816802.029 (In English)
Clark, H. H. (2016). Depicting as a method of communication. Psychological Review, 123(3), 324–347. http://dx.doi.org/10.1037/rev0000026 (In English)
Clift, R. (2020). Stability and Visibility in Embodiment: The ʻPalm up’ in interaction. Journal of Pragmatics, 169, 190–205. https://doi.org/10.1016/j.pragma.2020.09.005 (In English)
Cooperrider, K., Abner, N. and Goldin-Meadow, S. (2018). The Palm-Up Puzzle: Meanings and Origins of a Widespread Form in Gesture and Sign. Front. Commun., 3:23. https://doi.org/10.3389/fcomm.2018.00023 (In English)
Ford, C. E., Thompson, S. A. and Drake, V. (2012). Bodily-Visual Practices and Turn Continuation. Discourse Process, 49, 192–212. https://doi.org/10.1080/0163853X.2012.654761 (In English)
Freedman, N. (1972). The analysis of movement behavior during the clinical interview, in Siegman, A. W. and Pope, B. (eds.), Studies in dyadic communication, NY: Pergamon Press, New York, USA, 153–175. https://doi.org/10.1016/B978-0-08-015867-9.50012-5 (In English)
Göksun, T., Goldin-Meadow, S., Newcombe, N. and Shipley, T. (2013). Individual differences in mental rotation: what does gesture tell us?. Cognitive processing, 14(2), 153–162. DOI: https://doi.org/10.1007/s10339-013-0549-1 (In English)
Holler, J., Kendrick, K. H. and Levinson, S. C. (2018). Processing language in face-to-face conversation: Questions with gestures get faster responses. Psychon Bull Rev, 25, 1900–1908 https://doi.org/10.3758/s13423-017-1363-z (In English)
Iriskhanova, O. K., Kiose, M. I., Leonteva, A. V. and Agafonova, O. V. (2023). Multimodal languaging: Reification profiles in language and gesture. Linguistic Frontiers, 6(2), 78–91. https://doi.org/10.2478/lf-2023-0014 (In English)
Kang, S., Tversky, B. and Black, J. B. (2015). Coordinating gesture, word, and diagram: explanations for experts and novices. Spatial Cognition & Computation, 15(1), 1–26. DOI: https://doi.org/10.1080/13875868.2014.958837 (In English)
Kendon, A. (1995). Gestures as illocutionary and discourse structure markers in Southern Italian conversation. Journal of Pragmatics, 23(3), 247–279. DOI: https://doi.org/10.1016/0378-2166(94)00037-F (In English)
Kendon, A. (2004). Gesture: Visible Action as Utterance. Cambridge University Press, Cambridge, UK. https://doi.org/10.1017/CBO9780511807572 (In English)
Kiose, M. I., Rzheshevskaya, A. A., Izmalkova, A. I. and Makeev, S. D. (2023). Foregrounding and accessibility effects in the gaze behavior of the readers with different cognitive style. Computational Linguistics and Intellectual Technologies. Papers from the Annual International Conference “Dialogue”, 22, 225–232. https://doi.org/10.28995/2075-7182-2023-22-225-232 (In English)
Kita, S., van Gijn, I. and van der Hulst, H. (1998). Movement phases in signs and cospeech gestures and their transcription by human encoders, in Ipke Wachsmuth & Martin Fröhlich (eds.), Gesture and sign language in human-computer interaction, 23–35. Berlin: Springer. http://dx.doi.org/10.1007/BFb0052986 (In English)
Levinson, S. C. (2003). Space in Language and Cognition: Explorations in Cognitive Diversity. Cambridge University Press, Cambridge, UK. https://doi.org/10.1017/CBO9780511613609 (In English)
Marrese O. H., Raymond C. W., Fox B. A., Ford C. E. and Pielke M. (2021). The Grammar of Obviousness: The Palm-Up Gesture in Argument Sequences. Front. Commun., 6:663067. https://doi.org/10.3389/fcomm.2021.663067 (In English)
McNeill, D. (2005). Gesture, gaze, and ground, MLMI'05: Proceedings of the Second international conference on Machine Learning for Multimodal Interaction, Edinburgh, UK, 1–14. https://doi.org/10.1007/11677482_1 (In English)
Müller, C. (2004). Forms and uses of the Palm Up Open Hand: A case of a gesture family? The semantics and pragmatics of everyday gestures. Proceeding of the Berlin conference, Berlin, Germany, 233–256. (In English)
Nippold, M. A. and Scott, C. M. (2009). Expository Discourse in Children, Adolescents, and Adults: Development and Disorders. New York: Psychology Press., New York, USA. DOI: https://doi.org/10.4324/9780203848821 (In English)
Rodero, E. (2022). Effectiveness, Attractiveness, and Emotional Response to Voice Pitch and Hand Gestures in Public Speaking. Frontiers in Communication, 7: 869084. DOI: https://doi: 10.3389/fcomm.2022.869084 (In English)
Shaw, E. P. (2013). Gesture in Multiparty Interaction: A Study of Embodied Discourse in Spoken English and American Sign Language. PhD thesis. Georgetown University, Washington DC, USA. (In English)
Shusterman, A. and Li P. (2016). A Framework for Work on Frames of Reference, in Barner, D. and Baron, A. S. (eds.), Core Knowledge and Conceptual Change, Oxford University Press, Oxford, UK. P. 191–206. (In English)
Streeck, J. (2009). Forward-gesturing. Discourse Processes, 46(2-3), 161–179. https://doi.org/10.1080/01638530902728793 (In English)
Van Dijk, T. (1990). Issues in Functional Discourse Analysis, in Pinkster, H. and Genee, I. (eds.), Unity in Diversity: Papers Presented to Simon C. Dik on his 50th Birthday, De Gruyter Mouton, Boston, USA, 27–46. https://doi.org/10.1515/9783110847420.27 (In English)
Yasui, E. (2013). Collaborative idea construction: Repetition of gestures and talk in joint brainstorming. Journal of Pragmatics, 46(1), 157–172. https://doi.org/10.1016/j.pragma.2012.10.002 (In English)
Zlatev, J. (2005). What’s in a schema? Bodily mimesis and the grounding of language, in Beate Hampe (ed.), From perception to meaning: Image schemas in cognitive linguistics, 313–343. Berlin: Mouton de Gruyter. (In English)
Zlatev, J. Image schemas, mimetic schemas, and children’s gestures (2014). CognitiveSemiotics, 7(1), 3–29. https://doi.org/10.1515/cogsem-2014-0002 (In English)
The research presented in Sections 1 and 2 is part of the project “Multimodal research of the speaker’s communicative behavior in different discourse types” (FSFU-2020-0021) carried out at Moscow State Linguistic University. The research presented in Sections 3, 4, and 5 is part of the project “Kinesic and vocal aspects of communication: parameters of variance” (FMNE-2022-0015) carried out at the Institute of Linguistics RAS.