2313-8912Research Result. Theoretical and Applied Linguistics2313-891210.18413/2313-8912-2024-10-4-0-73678Human Language Behaviour in Machine-Generated Environments<strong>Technosemantics of gesture: on the possibilities of using Perm sign notation in software-generated environments</strong><strong>Technosemantics of gesture: on the possibilities of using Perm sign notation in software-generated environments</strong>BelousovKonstantin I.BelousovKonstantin I.belousovki@gmail.comTaleskiAleksandarTaleskiAleksandartaleski87@yahoo.comAgaevArtem R.AgaevArtem R.agaev-artem1@yandex.ruPerm State UniversityHuman Semantics LLC202410400

This paper is dedicated to the development of a concept and software solution for generating human movements based on a semantically-oriented language notation created by the authors. The language notation is presented as a formula with a flexible structure of concepts and rules for their implementation, allowing easy adaptation of movement parameter changes to match an ideal or real sample. To model movements represented in the language notation, a cross-platform application was developed using the Blender 4.2 for the visualization and generation of gestures for anthropomorphic models. The movement control system consists of the following stages: translating it into a gesture notation record; parsing this record; constructing an internal representation of the movement, which is a sequence of frames. Frames contain information about which bone (body part) of the performer they refer to, how they affect its position, and at what time from the start of the movement this frame is relevant. In the final stage, the internal representation of the movement is transformed into the performer&rsquo;s movement, which can be either virtual anthropomorphic 3D models or physical software-hardware systems in the form of anthropomorphic robots. To better correspond to anthropomorphic behavior, in addition to &ldquo;ideal samples&rdquo; of human movement, models of human gesture behavior presented in a multimodal corpus specially created for this purpose by the team were used. The material consists of audiovisual recordings of spontaneous oral texts where recipients describe a wide range of their own emotional states. The data obtained during the experimental research confirmed the flexibility, enhanced controllability, and modularity of the language notation, as well as the ability to model the continuous space of human motor activity.

This paper is dedicated to the development of a concept and software solution for generating human movements based on a semantically-oriented language notation created by the authors. The language notation is presented as a formula with a flexible structure of concepts and rules for their implementation, allowing easy adaptation of movement parameter changes to match an ideal or real sample. To model movements represented in the language notation, a cross-platform application was developed using the Blender 4.2 for the visualization and generation of gestures for anthropomorphic models. The movement control system consists of the following stages: translating it into a gesture notation record; parsing this record; constructing an internal representation of the movement, which is a sequence of frames. Frames contain information about which bone (body part) of the performer they refer to, how they affect its position, and at what time from the start of the movement this frame is relevant. In the final stage, the internal representation of the movement is transformed into the performer&rsquo;s movement, which can be either virtual anthropomorphic 3D models or physical software-hardware systems in the form of anthropomorphic robots. To better correspond to anthropomorphic behavior, in addition to &ldquo;ideal samples&rdquo; of human movement, models of human gesture behavior presented in a multimodal corpus specially created for this purpose by the team were used. The material consists of audiovisual recordings of spontaneous oral texts where recipients describe a wide range of their own emotional states. The data obtained during the experimental research confirmed the flexibility, enhanced controllability, and modularity of the language notation, as well as the ability to model the continuous space of human motor activity.

TechnosemanticsMovement generationLanguage notation оf gesturesGestureVisualizationMultimodal corpus3D graphicsInterpretabilityTechnosemanticsMovement generationLanguage notation оf gesturesGestureVisualizationMultimodal corpus3D graphicsInterpretabilityСписок литературыAbbie,&nbsp;M. (1974). Movement notation, The Australian journal of physiotherapy, 20&nbsp;(2), 61&ndash;69. https://doi.org/10.1016/S0004-9514(14)61177-6 (In English)Bashan,&nbsp;M., Einbinder,&nbsp;H., Harries,&nbsp;J., Shosani,&nbsp;M. and Shoval, D. (2024). Movement Notation: Eshkol and Abraham Wachmann, Verlag der Buchhandlung Walther K&ouml;nig, K&ouml;ln, Germany. (In English)Belousov,&nbsp;K.&nbsp;I., Sazina,&nbsp;D.&nbsp;A., Ryabinin,&nbsp;K.&nbsp;V. and. Brokhin,&nbsp;L.&nbsp;Yu. (2024). Sensory Technolinguistics: On Mechanisms of Transmitting Multimodal Messages in Perceptual-Cognitive Interfaces, Automatic Documentation and Mathematical Linguistics, 58 (2), 108&ndash;116. https://doi.org/10.3103/s0005105524700079 (In English)Benesh,&nbsp;R. and Benesh,&nbsp;J. (1956). An Introduction to Benesh Dance Notation, A &amp; C Black, London, UK. (In English)Bernardet,&nbsp;U., Fdili Alaoui,&nbsp;S., Studd,&nbsp;K., Bradley,&nbsp;K., Pasquier,&nbsp;P. and Schiphorst,&nbsp;T. (2019) Assessing the reliability of the Laban Movement Analysis system, PLoS ONE, 14&nbsp;(6): e0218179. https://doi.org/10.1371/journal.pone.0218179 (In English)Birdwhistell,&nbsp;R.&nbsp;L. (1952). Introduction to Kinesics: An Annotation System for Analysis of Body Motion and Gesture, Foreign Service Institute, Washington, DC, USA. (In English)Bull,&nbsp;P. and Doody,&nbsp;J.&nbsp;P. (2013). 8 Gesture and body movement, De Gruyter eBooks, 205&ndash;228. https://doi.org/10.1515/9783110238150.205 (In English)Calvert,&nbsp;T. (2015). Approaches to the Representation of Human Movement: Notation, Animation and Motion Capture, Dance Notations and Robot Motion, Springer Tracts in Advanced Robotics, 111, 49&ndash;68. https://doi.org/10.1007/978-3-319-25739-6_3 (In English)Dael,&nbsp;N., Mortillaro,&nbsp;M. and Scherer,&nbsp;K.&nbsp;R. (2012). The Body Action and Posture Coding System (BAP): Development and Reliability, Journal of Nonverbal Behavior, 36&nbsp;(2), 97&ndash;121. https://doi.org/10.1007/s10919-012-0130-0 (In English)Dell,&nbsp;C. (1977). A Primer for Movement Description: Using Effort-shape and Supplementary Concepts, Dance Notation Bureau Press, New York, USA. (In English)Duprey,&nbsp;S., Naaim,&nbsp;A., Moissenet,&nbsp;F., Begon,&nbsp;M. and Ch&egrave;ze,&nbsp;L. (2017). Kinematic models of the upper limb joints for multibody kinematics optimisation: An overview, Journal of Biomechanics, 62, 87&ndash;94. DOI: 10.1016/j.jbiomech.2016.12.005 (In English)Ekman,&nbsp;P. and Friesen,&nbsp;W.&nbsp;V. (1978). Facial Action Coding System, Consulting Psychologists, Palo Alto, CA, USA. (In English)El Raheb,&nbsp;K. and Ioannidis,&nbsp;Y. (2014). From dance notation to conceptual models: a multilayer approach, Proceedings of the 2014 International Workshop on Movement and Computing, MOCO, ACM, New York, 25&ndash;30. (In English)El Raheb,&nbsp;K., Buccoli,&nbsp;M., Zanoni,&nbsp;M., Katifori,&nbsp;A., Kasomoulis,&nbsp;A., Sarti,&nbsp;A. and Ioannidis,&nbsp;Y. (2023). Towards a general framework for the annotation of dance motion sequences, Multimed Tools Appl, 82, 3363&ndash;3395. https://doi.org/10.1007/s11042-022-12602-y (In English)Eshkol,&nbsp;N. and Wachmann,&nbsp;A. (1958). Movement Notation, Weidenfeld and Nicolson, London, UK. (In English)Farnell,&nbsp;B.&nbsp;M. (1996). Movement Notation Systems, The World&rsquo;s Writing Systems, in Daniels P. T. (ed.), 855&ndash;879. (In English)Frishberg,&nbsp;N. (1983). Writing systems and problems for sign language notation, Journal for the Anthropological Study of Human Movement, 2&nbsp;(4), 169&ndash;195. (In English)Frey,&nbsp;S., Hirsbrunner,&nbsp;H-P. and Jorns,&nbsp;U. (1982). Time-Series Notation: A Coding Principle for the Unified Assessment of Speech and Movement in Communication Research, Gunter NarrVerlag, T&uuml;bingen, Germany. (In English)Grushkin,&nbsp;D.&nbsp;A. (2017). Writing Signed Languages: What For? What Form?, American Annals of the Deaf, 161&nbsp;(5), 509&ndash;527. https://doi.org/10.1353/aad.2017.0001 (In English)Guest,&nbsp;A.&nbsp;H. (1984). Dance Notation: The Process of Recording Movement on Paper, Dance Horizons, New York, USA. (In English)Guest,&nbsp;A.&nbsp;H. (2005). Labanotation: The System of Analyzing and Recording Movement (4th ed.), Routledge, New York, USA. https://doi.org/10.4324/9780203823866 (In English)Harrigan,&nbsp;J.&nbsp;A. (2008). Proxemics, Kinesics, and Gaze, The New Handbook of Methods in Nonverbal Behavior Research, 136&ndash;198. https://doi.org/10.1093/acprof:oso/9780198529620.003.0004 (In English)Izquierdo,&nbsp;C. and Anguera,&nbsp;M.&nbsp;T. (2018). Movement notation revisited: syntax of the common morphokinetic alphabet (CMA) system, Front. Psychol, 9:1416. https://doi.org/10.3389/fpsyg.2018.01416 (In English)Karg,&nbsp;M., Samadani,&nbsp;A.-A., Gorbet,&nbsp;R., Kuhnlenz,&nbsp;K., Hoey,&nbsp;J. and Kulic,&nbsp;D. (2013). Body Movements for Affective Expression: A Survey of Automatic Recognition and Generation, IEEE Transactions on Affective Computing, 4&nbsp;(4), 341&ndash;359. https://doi.org/10.1109/t-affc.2013.29 (In English)Kendon,&nbsp;A. (1997). Gesture, Annual Review of Anthropology, 26&nbsp;(1), 109&ndash;128. https://doi.org/10.1146/annurev.anthro.26.1.109 (In English)Key,&nbsp;M.&nbsp;R. (1977). Nonverbal communication: a research guide and bibliography, The Scarecrow Press, Metuchen, N.J., USA. (In English)Kilpatrick,&nbsp;C.&nbsp;E. (2020). Movement, Gesture, and Singing: A Review of Literature. Update: Applications of Research in Music Education, 38&nbsp;(3), 29-37. &nbsp;DOI: 10.1177/8755123320908612 (In English)Laban,&nbsp;R.&nbsp;von and Lawrence,&nbsp;F.&nbsp;C. (1974). Effort: Economy of Human Movement 2nd ed., Macdonald &amp; Evans, London, UK. (In English)Laumond,&nbsp;J. and Abe,&nbsp;N. (2016). Dance Notations and Robot Motion, Springer International Publishing AG, Cham (ZG), Switzerland. https://doi.org/10.1007/978-3-319-25739-6 (In English)Liu,&nbsp;H., Zhu,&nbsp;Z., Iwamoto,&nbsp;N., Peng,&nbsp;Y., Li,&nbsp;Zh., Zhou,&nbsp;Y., Bozkurt,&nbsp;E. and Zheng,&nbsp;B. (2022). BEAT: A Large-Scale Semantic and Emotional Multi-modal Dataset for Conversational Gestures Synthesis, Computer Vision &ndash; ECCV 2022, 612&ndash;630. https://doi.org/10.48550/arXiv.2203.05297 (In English)Murillo,&nbsp;E., Montero,&nbsp;I. and Casla,&nbsp;M. (2021). On the multimodal path to language: The relationship between rhythmic movements and deictic gestures at the end of the first year, Frontiers in Psychology, 12, 1&ndash;8. https://doi.org/10.3389/fpsyg.2021.616812 (In English)Novack,&nbsp;A.&nbsp;M. and Wakefield,&nbsp;E.&nbsp;M. (2016). Goldin-Meadow S. What makes a movement a gesture?, Cognition, 146, 339-348. https://doi.org/10.1016/j.cognition.2015.10.014 (In English)Qi,&nbsp;X., Liu,&nbsp;C., Li,&nbsp;L., Hou,&nbsp;J., Xin,&nbsp;H. and Yu,&nbsp;X. (2024). Emotion Gesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generation, IEEE Transactions on Multimedia, 1&ndash;11. https://doi.org/10.1109/TMM.2024.3407692 (In English)Streeck,&nbsp;J. (2010). The Significance of Gesture: How it is Established, Papers in Pragmatics; 2 (1-2). https://doi.org/2.10.1075/iprapip.2.1-2.03str (In English)Shafir,&nbsp;T., Tsachor,&nbsp;R. and Welch,&nbsp;K.&nbsp;B. (2016). Emotion Regulation through Movement: Unique Sets of Movement Characteristics are Associated with and Enhance Basic Emotions, Frontiers in Psychology, 6. https://doi.org/10.3389/fpsyg.2015.02030 (In English)Stults-Kolehmainen,&nbsp;M.&nbsp;A. (2023). Humans have a basic physical and psychological need to move the body: Physical activity as a primary drive, Frontiers in Psychology, 14. https://doi.org/10.3389/fpsyg.2023.1134049 (In English)Tonoli,&nbsp;R.&nbsp;L., Costa,&nbsp;P.&nbsp;D.&nbsp;P., Marques, L.&nbsp;B.&nbsp;d.&nbsp;M.&nbsp;M. and Ueda,&nbsp;L.&nbsp;H. (2024). Gesture Area Coverage to Assess Gesture Expressiveness and Human-Likeness&rsquo;, International Conference on Multimodal Interaction (ICMI Companion &lsquo;24), 4&ndash;8 November 2024, San Jose, Costa Rica. ACM, New York, NY, USA. https://doi.org/10.1145/3686215.3688822 (In English)Trujillo,&nbsp;J.&nbsp;P., Vaitonyte,&nbsp;J., Simanova,&nbsp;I. and &Ouml;zy&uuml;rek,&nbsp;A. (2018). Toward the markerless and automatic analysis of kinematic features: A toolkit for gesture and movement research, Behavior Research Methods, 51&nbsp;(2), 769&ndash;777. https://doi.org/10.3758/s13428-018-1086-8 (In EnglishVan&nbsp;Elk,&nbsp;M.,&nbsp;van&nbsp;Schie,&nbsp;H.&nbsp;T. and Bekkering,&nbsp;H. (2009). Short-term action intentions overrule long-term semantic knowledge, Cognition, 111&nbsp;(1), 72&ndash;83. https://doi.org/10.1016/j.cognition.2008.12.002 (In English))Yang,&nbsp;S., Wu,&nbsp;Z., Li,&nbsp;M., Zhang,&nbsp;Z., Hao,&nbsp;L., Bao,&nbsp;W. and Zhuang,&nbsp;H. (2023). QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2321&ndash;2330. https://doi.org/10.48550/arXiv.2305.11094 (In English)Yoon,&nbsp;Y., Cha,&nbsp;B., Lee,&nbsp;J.-H., Jang,&nbsp;M., Lee,&nbsp;J., Kim,&nbsp;J. and Lee,&nbsp;G. (2020). Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity, ACM Transactions on Graphics, 39&nbsp;(6). https://doi.org/10.1145/3414685.3417838 (In English)Zhi,&nbsp;Y., Cun,&nbsp;X., Chen,&nbsp;X., Shen,&nbsp;X., Guo,&nbsp;W., Huang,&nbsp;S. and Gao,&nbsp;S. (2023). LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generation, Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 20807&ndash;20817. https://doi.org/10.1109/ICCV51070.2023.01902 (In English)