Can gestures communicate the time of described events? The perception of temporal gestures executed by a companion robot
In this paper, we investigate whether temporal gestures indicating different events in time can convey information about the relative timing of these events. We depart from the assertion, that in many cultures time is metaphorically represented as an axis, oriented from left to right, i.e. recent or future events have rightmost position, as compared to the events of the past. We test the hypothesis that temporal gestures, referring to the events on the axis, may communicate to the addressee the temporal reference of the described events. The hypothesis was tested in an experiment (N = 22 people, average age 21.6 years, 17 female) where two companion robots told stories, accompanied by left-oriented, symmetric or right oriented gestures. Each story included events from two different cases – recent and past, and each statement was accompanied by (a) left-oriented or symmetric gesture (first condition), or (b) right-oriented or symmetric gesture (second condition). The task of the subjects was to correlate the described events with time, assigning each statement with recent or past event. As a result of this study, the hypothesis that the orientation of temporal gestures can communicate the time of the designated events was partially confirmed. The results correspond to the concept of time oriented for the narrator from left to right, where the events of the past are in the center, and later events (recent events, or ‘today’) are located on the right. This conceptual orientation in the experiment was interpreted by the human listeners, although from their point of view, “recent events” are located on the left side of the metaphoric axis.
Figures
Kotov, A. A., Zinina, A. A., Arinkin, N. A. and Berkuta, D. V. (2024) Can gestures communicate the time of described events? The perception of temporal gestures executed by a companion robot, Research Result. Theoretical and Applied Linguistics, 10 (2), 100-116. DOI: 10.18413/2313-8912-2024-10-2-0-5
While nobody left any comments to this publication.
You can be first.
Barsalou, L. W. (1999). Perceptual Symbol Systems, Behavioral and Brain Sciences, 22, 577–660. https://doi.org/10.1017/S0140525X99002149(In English)
Barsalou, L. W. (2005). Abstraction as dynamic interpretation in perceptual symbol systems, in Gershkoff-Stowe, L. and Rakison, D. (eds.), Building object categories, Carnegie Symposium Series, Erlbaum, 389–431. (In English)
Boroditsky, L. (2000). Metaphoric structuring: Understanding time through spatial metaphors, Cognition, 75 (1), 1–28. https://doi.org/10.1016/S0010-0277(99)00073-6(In English)
Boroditsky, L., Fuhrman, O. and McCormick, K. (2011). Do English and Mandarin speakers think about time differently?, Cognition, 118 (1), 123–129. https://doi.org/10.1016/j.cognition.2010.09.010(In English)
Casasanto, D. and Bottini, R. (2010). Can mirror-reading reverse the flow of time?, in Hölscher, C., Shipley, T. F., Olivetti Belardinelli, M., Bateman, J. A. and Newcombe, N. S. (eds.), Spatial Cognition VII. Spatial Cognition 2010. Lecture Notes in Computer Science, Springer, Berlin, Heidelberg, 6222, 335–345. https://doi.org/10.1007/978-3-642-14749-4_28(In English)
Clark, H. H. (1973). Space, time, semantics, and the child, Cognitive development and acquisition of language, 27–63. https://doi.org/10.1016/B978-0-12-505850-6.50008-6(In English)
Cooperrider, K. and Núñez, R. (2007). Doing time: speech, gesture, and the conceptualization of time, Center for Research in Language Technical Reports, 19, 3–19. (In English)
Fuhrman, O. and Boroditsky, L. (2007). Mental time-lines follow writing direction: Comparing English and Hebrew speakers, Proceedings of the Annual Meeting of the Cognitive Science Society, 29 (29), 1007-1011. (In English)
Fuhrman, O. and Boroditsky, L. (2010). Cross-cultural differences in mental representations of time: Evidence from an implicit nonlinguistic task, Cognitive Science, 34 (8), 1430–1451. https://doi.org/10.1111/j.1551-6709.2010.01105.x(In English)
Grishina, E. A. (2012). Ukazaniya rukoj kak sistema (po dannym Mul'timedijnogo russkogo korpusa) [Hand indications as a system (on the data of the Multimedia Russian Corpus)], Voprosy Yazykoznaniya, 3, 3-50. (In Russian)
Grishina, E. A. (2018). Russkaya zhestikulyaciya s lingvisticheskoj tochki zreniya. Korpusnye issledovaniya [Russian gesticulation from the linguistic point of view, Corpus studies], Yazyki slavyanskoj kul'tury, Moscow, Russia. (In Russian)
Grishina, E., Savchuk, S. and Sichinava, D. (2012). Multimodal Parallel Russian Corpus (MultiPARC): Main Tasks and General Structure, LREC 2012 Workshop on Best Practices for Speech Corpora in Linguistic Research, Istanbul, Turkey, 13–16. (In English)
Gu, Y., Zheng, Y. and Swerts, M. (2019). Which is in front of Chinese people, past or future? The effect of language and culture on temporal gestures and spatial conceptions of time, Cognitive Science, 43 (12), e12804. https://doi.org/10.1111/cogs.12804(In English)
Haviland, J. B. (1993). Anchoring, iconicity, and orientation in Guugu Yimithirr pointing gestures, Journal of Linguistic Anthropology, 3 (1), 3–45. https://doi.org/10.1525/jlin.1993.3.1.3(In English)
Kendon, A. (1980). Gesticulation and speech: Two aspects of the process of utterance, The Relationship of Verbal and Nonverbal Communication, 25 (1980), 207–227. (In English)
Kibrik, A. A. (2011). Reference in Discourse, Oxford University Press, UK. (In English)
Kopp, S., Krenn, B., Marsella, S., Marshall, A., Pelachaud, C., Pirker, H., Thórisson, K. and Vilhjálmsson, H. (2006). Towards a Common Framework for Multimodal Generation: The Behavior Markup Language, in Gratch, J., Young, M., Aylett, R., Ballin, D. and Olivier, P. (eds.), Intelligent Virtual Agents, 4133, Springer, Berlin, Heidelberg, 205–217. http://dx.doi.org/10.1007/11821830_17(In English)
Kotov, A. A. and Zinina, A. A. (2015). Funkcional'nyj analiz neverbal'nogogo kommunikativnogo povedeniya [Functional Analysis of Non‑Verbal Communicative Behavior], Proceedings of the International ConferenceComputer linguistics and intellectual technologies, Moscow, Russia, 287–295. (In Russian)
Lakoff, G. and Johnson, M. (2003). Metaphors we live by, University of Chicago press Chicago, IL. (In English)
McNeill, D. (1992). Hand and mind: What gestures reveal about thought, University of Chicago press Chicago, IL. (In English)
Miles, L., Nind, L. and Macrae, C. (2010). Moving through time, Psychological Science, 21 (2), 222. (In English)
Nagai, Y., Hosoda, K., Morita, A. and Asada, M. (2003). A constructive model for the development of joint attention, Connection Science, 15 (4), 211–229. https://doi.org/10.1080/09540090310001655101(In English)
Núñez, R. E. and Sweetser, E. (2006). With the future behind them: Convergent evidence from Aymara language and gesture in the crosslinguistic comparison of spatial construals of time, Cognitive Science, 30 (3), 401–450. https://doi.org/10.1207/s15516709cog0000_62(In English)
Ouellet, M., Santiago, J., Israeli, Z. and Gabay, S. (2010). Is the future the right time?, Experimental Psychology, 57 (4). https://doi.org/10.1027/1618-3169/a000036(In English)
Sidner, C. L., Kidd, C. D., Lee, C. and Lesh, N. (2004). Where to look: a study of human-robot engagement, Proceedings of the 9th International Conference on Intelligent User Interfaces, New York, NY, United States, 78–84. https://doi.org/10.1145/964442.964458(In English)
Ulrich, R., Eikmeier, V., de la Vega, I., Ruiz Fernández, S., Alex-Ruf, S. and Maienborn, C. (2012). With the past behind and the future ahead: Back-to-front representation of past and future sentences, Memory & Cognition, 40, 483–495. https://doi.org/10.3758/s13421-011-0162-4(In English)
Vilhjálmsson, H., Cantelmo, N., Cassell, J. E. Chafai, N., Kipp, M., Kopp, S., Mancini, M., Marsella, S., Marshall, A., Pelachaud, C., Ruttkay, Z., Thórisson, K., van Welbergen, H. and van der Werf, R. (2007). The Behavior Markup Language: Recent Developments and Challenges, Intelligent Virtual Agents, 99–111. http://dx.doi.org/10.1007/978-3-540-74997-4_10(In English)
Xiao, C., Zhao, M. and Chen, L. (2018). Both Earlier Times and the Future Are “Front”: The Distinction Between Time-and Ego-Reference-Points in Mandarin Speakers’ Temporal Representation, Cognitive Science, 42 (3), 1026–1040. https://doi.org/10.1111/cogs.12552(In English)
Zinina, A., Arinkin, N., Zaydelman, L. and Kotov, A. (2019). The role of oriented gestures during robot’s communication to a human, Proceedings of the International ConferenceComputer linguistics and intellectual technologies, Moscow, Russia, 800–808. (In English)
Zinina, A., Kotov, A., Arinkin, N. and Zaidelman, L. (2022). Learning a foreign language vocabulary with a companion robot, Cognitive Systems Research, 77, 110-114 https://doi.org/10.1016/j.cogsys.2022.10.007(In English)