1. Introduction

Contact-establishing function of communication is by right extensively explored in language studies. As known, it is mostly addressed via establishing discursive markers or via pragmatic or speech acts. By contrast with linguistic nature of contact-establishing, its multimodal nature is mostly neglected. Meanwhile, this is the bodily behavior in gesturing that presumably provides contact-establishing in a more persuasive way than language. Contact-establishing gestures are used in different contexts to build rapport with the interlocutor(s) due to their physical properties: palm orientation (fingers oriented towards the listener) and/or its movement towards the listener (Bavelas et al., 1995); they manifest interactive nature since they convey communicative intentions and emotions, form request, facilitate cooperation (Yasui, 2013), as well as initiate, maintain, regulate, or terminate interaction in general (Alibali et al., 2011; Gökson et al., 2013; Kang et al., 2015; Rodero, 2022). Their multi-functional nature promotes the need to classify them; still up to now the studies did not offer a methodological solution to the problem.

This paper advances a cognitive solution to exploring multimodal contact-establishing via its bodily nature and identifies mimetic schemas or bodily schemas shared by communicants who engage in face-to-face and body-to-body interaction (Cienki, 2013; Zlatev, 2014) which underlie the selection of gestures and speech. We employ the gestural and speech data obtained in the multimodal experiment (Iriskhanova et al., 2023), to explore the distribution of mimetic schemas in contact-establishing gestures and speech. We expect that contact-establishing gestures can be differentiated on the grounds of the differences they display in mimetic schemas which help construe them. Since two recurrent gesture families, palm-up-open-hand (PUOH) and palm-down-open-hand (PDOH) gestures (Müller, 2004; Cienki, 2021; Iriskhanova & Nikolaeva, 2023), demonstrating the mimetic schemas SHOW and RESTRAIN were mostly observed in our compiled corpus of contact-establishing gestures, we 1) develop the three-dimension framework (cognitive, pragmatic and functional semantic dimensions) proposed in (Van Dijk, 1990) and further developed in multiple studies to explore the linguistic nature of mimetic sub-schemas of SHOW and RESTRAIN, 2) explore the linguistic and gestural nature of SHOW and RESTRAIN mimetic schemas and their sub-schemas differentiated in (Iriskhanova, Nikolaeva, 2023), 3) determine the clusters of PUOH and PDOH contact-establishing gestures via their linguistic correspondences since they account for common thinking-for-speaking growth points (McNeill, 2005); 4) establish the language profiles of PUOH and PDOH contact-establishing gestures.

Therefore, in the current paper, we survey new perspectives of advancing the methodology of mimetic schemas while applying it to identifying the typological characteristics of gesture families, here PUOH and PDOH gestures. This approach which advances the procedure of indirect measures (Kiose, 2021), here exploring gestures via contingent speech, allows to explore and differentiate multi-functional contact-establishing gestures. Following these distinctions, we hypothesize that i) contact-establishing gestures fall into two families of recurrent gestures, PUOH and PDOH which represent the mimetic schemas SHOW and RESTRAIN; ii) SHOW and RESTRAIN mimetic schemas and sub-schemas display alignment patterns with cognitive, pragmatic and functional semantic speech cues; iii) gesture typology of PUOH and PDOH contact establishing gestures can be determined via concurrent linguistic cues.

In Section Two we provide an overview on the Theoretical Framework on the paper, which covers a) contact-establishing (phatic) multimodal communication, b) a cognitive mimetic approach to exploring multimodal behavior expressed in contact-establishing gestures and speech. In Section Three we present the experiment design and data collection. Section four manifests the experiment results with reference to three developed hypotheses and is followed by discussion of the results. In Final Remarks section we draw attention to the research outcomes for multimodal communication theory.

2. Theoretical Framework

This section will subsequently address contact-establishing (phatic) multimodal communication. It will introduce a cognitive mimetic approach to exploring multimodal behavior expressed in contact-establishing gestures and determine the methodology to identify the speech cues to concur with gestures.

2.1. Contact-establishing PUOH and PDOH gestures in mimetic schemas SHOW and RESTRAIN

During the communication we express our ideas, feeling, emotions, stance, etc., but not only through words, but also through nonverbal means, such as proxemics and gestures, which include body, head and hand movements. Contact-establishing function can be observed in gestures that can play a role of discourse markers (Kendon, 1995) due to their interactive nature in conversations. Such gestures are used in different contexts to establish contact due to their physical properties: palm orientation (fingers oriented towards the listener) and/or its movement towards the listener (Bavelas et al., 1995). Some studies argue in favour of the interactive nature of some gestures, as they can convey communicative intentions and emotions, form request, facilitate cooperation (Yasui, 2013), as well as initiate, maintain, regulate, or terminate interaction in general (Alibali et al., 2011; Gökson et al., 2013; Kang et al., 2015; Rodero, 2022). Their interactive potential can be used to manage dialogues through turn-taking (Abner et al., 2015). Such gestures are also vital in problem-solving communication (Beilock & Goldin-Meadow, 2010). Contact establishing gestures can be of various forms and sometimes can convey more than one meaning. E.g. they can be both contact establishing and emphatic/pointing, discourse structuring, etc. They are also known for being pragmatic, illocutionary, or discourse gestures (Kendon, 2004). The latter observation accounts for the need to develop their typology to differentiate their functional and structural features.

In our paper, we focus on two gesture configurations: palm up open hand and palm down open hand, which are regarded parts of two families of recurrent gestures: palm up and palm down (e.g. see Cienki, Müler, 2008; Cooperrider et al., 2018). The first type, PUOH, has been described in multiple studies, as it is often used in English communication (Chu et al., 2014). The physical properties of the gesture are described as follows: “rotation of the palms upwards or outwards towards the recipient, with—if standing—a raise of the arms outwards away from the body. These are then momentarily held static in parallel, iconically displaying a temporary halt to the progressivity of the interactional sequence” (Clift, 2020: 204). The origin of this gesture is supposed to lie in object transfer, e.g. presenting, offering, or hand-over something (Streeck, 2009) or it can include the idea of a container or presentation (Cienki, Müller, 2008). The embodied nature of this gesture is also reflected in the use with verbal discourse, as one can hold it during one or multiple turns (Clift, 2020). Thus, we would argue, that it can be used as a means of uniting discourse parts. This ability of PUOH is regarded to be of argumentative nature, as it can be used to express stance (Kendon, 2004; Ford et al., 2012; Shaw, 2013; Clift, 2020; Marrese et al., 2021). In particular, PUOH is described to have similarities with verbal particles which index “obviousness”. It was found to be used with declarative assertions, interrogatives, imperatives, or without any verbal cues (Marrese et al., 2021). In some other works, PUOH is mentioned as a performative gesture, enacting speech acts, e.g., it can be used to dismiss an offer or idea (e.g., see Cienki, Müller, 2008). The second gesture configuration, palm down open hand, PDOH, is a less studied configuration. It is linked to negation and share semantics of stopping or interrupting some action in progress, e.g., physical or communicative action or mental activity (Kendon, 2004). It can be synchronized with grammatical negation in speech expressed in verbs, verb phrases, clauses or sentences (Harrison, 2009, 2018).

Both of these gesture families, PUOH and PDOH, are explored as manifesting different mimetic schemas in the pioneering works of Cienki (2021) and Iriskhanova, Nikolaeva (2023). Following Zlatev (2005, 2014) and Cienki (2013, 2017), mimetic schemas are viewed as a type of image schemas with the first being bodily schemas shared by communicants who engage in face-to-face and body-to-body interaction (Zlatev, 2014). In (Cienki 2021) the sub-types of PUOH and PDOH gestures are scaled as demonstrated higher and lower Representational, Pragmatic and Interactive transparencies (for the researcher) or functions in discourse. In the current study, following the functions of PUOH gestures outlined in (Kendon, 2004; Ford et al., 2012; Shaw, 2013; Clift, 2020; Marrese et al., 2021; Iriskhanova, Nikolaeva, 2023) we support the idea of PUOH being an embodiment of presenting and offering some ideas or discourse parts. Thus, we distinguish it as representing the memetic schema SHOW/PRESENT. Following the functions of PDOH gestures and its semantic properties outlined in (Kendon, 2004; Cienki, 2021; Iriskhanova, Nikolaeva, 2023), we suggest that PDOH gesture represent the memetic scheme of RESTRAIN as it is used to interrupt, stop or negate the idea or speech flow. Importantly, in (Iriskhanova, Nikolaeva, 2023) 6 variants of SHOW mimetic schema (with PUOH gestures) and 5 variants of RESTRAIN mimetic schema (with PDOH gestures) are identified. These outcomes serve to develop the typology of mimetic sub-schemas which is adopted in the present study and is used to explore PUOH and PDOH gestures contingent with linguistic cues as representing these schemas.

2.2. Language cues in mimetic schemas SHOW and RESTRAIN

To develop the typology of speech cues potentially contingent to mimetic sub-schemas of SHOW and RESTRAIN, we addressed the discourse features in the clauses specifying 1) cognitive characteristics of communication event and object of reference, 2) pragmatic characteristics of communication event and object of reference, 3) functional semantic characteristics of the communication event and object of reference. In advancing a three-dimension typology, we mostly adhere to the model of discourse comprising its cognitive, pragmatic and functional semantic features developed by T. Van Dijk (Van Dijk, 1990). While determining cognitive characteristics of communication, we identified 1a) the event of situated communication or non-situated (objects of reference present or absent from communicative event) (Levinson, 2003; Shusterman & Li, 2016) which was proved to affect the gestural mode (Grishina, 2012), and 1b) the object of reference being of a bodily type or of a more abstract type (here – a word) which was priorly found to mediate the use of gestures (Iriskhanova et al., 2023). Relevant pragmatic features were attested, 2a) pragmatic type of event of communication, question, negation or statement, 2b) pragmatic type of object of reference in its viewpoint formulation and presentation related to expositive discourse (Nippold, Scott, 2010). Functional-semantic features describe the semantic role of nominals relating to the object of reference and semantic roles (Lyashevskaya & Kashkin, 2015) of other context components to construe the communication event.

Furthermore, we expect that PUOH and PDOH gestures can obtain language profiles via contingent speech cues, i.e. gestures can be indirectly typologized via a contingent communicative mode (speech). This possibility lies within common thinking-for-speaking growth points (McNeill, 2005) which stimulate the choice of communicative modes in discourse. To establish the speech profiles of gestures we adopt the indirect measurement procedure described in (Kiose, 2021) which proposes to use crossmodal correspondences to identify the cognitive typological distinctions in one of the modes. Presumably, language profiles will allow to specify the relevant contrasting features of PUOH and PDOH gestures.

3. Experiment design and data collection

3.1. Experiment design

In order to explore contact-establishing in speech and gesture, we used the data obtained in the multimodal experiment (Iriskhanova et al., 2023) in which the pairs of participants aged 18-21 explained the differences in the use of close synonyms stimulated by the discourse task to develop common ground. The total number of the participants was 30 and the total length of the multimodal video was 204 minutes, which was approximately 13.5 minutes for each pair. The interlocutors were sitting at the table opposite each other. The choice of such positioning was conditioned by the intention to simulate a real-life communication which could take place in a common situation (e.g., in a café) and where the participants could look at each other. To analyze multimodal behavioral patterns, we used motion capture costumes Perception Neuron Motion Capture. The recording was performed using one camera mounted on a tripod, that was placed opposite the speakers and a camera built in each pair of eye-tracking glasses that provided the perspective seen by each interlocutor. After the experiment the data on contact-establishing gestures and accompanying speech was analyzed in a computer program ELAN^[1], which enabled annotating verbal and nonverbal patterns. Total multimodal video set provided for the analysis within the framework of the current study accounts approximately 57 minutes.

3.2. Data collection

Overall, the corpus contained 366 contact-establishing gestures with 271 PUOH and PDOH recurrent gestures which were further analyzed to differentiate the mimetic sub-schemas (6 SHOW sub-schemas and 5 RESTRAIN sub-schemas) and speech cues representing cognitive, pragmatic and functional semantic dimensions.

In Figure 1 we present the visual images of 5 frequent PUOH gestures and 4 frequent PDOH gestures with accompanying speech. In identifying the gesture subtypes, we applied the procedure developed in (Iriskhanova, Nikolaeva, 2023). Figures 1a and 1b show MANIFEST and HOLD gestures of the PUOH family. Both these gestures display tension observed both in the flexion of single fingers and in the configuration of the hand (Bressem & Ladewig, 2011) which occur in the stroke phase or in the period of apparent gestural effort (Kita et al. 1998). These gestures are similar in terms of the palm configuration, however there are differences in the axis and character of the movement of the hand. HOLD gestures are manifested through a cupped static hand. MANIFEST gestures are different in terms of the hand configuration: it is flatter than that of the HOLD.

Figures 1c and 1d show WEIGH and MOVE AWAY gestures. These gestures display movement (Bressem & Ladewig, 2011) in the stroke phase which exposes the meaning of the gesture (Seyfeddinipur 2006, cited in Bressem & Ladewig, 2011: 61). WEIGH gestures are represented by a flat or cupped hand with a single or multiple hand up and down movement(s). MOVE AWAY gestures have the same forms as WEIGH gestures with a single or multiple movement(s) away from the center.

Figure 1e shows MOVE TOWARDS gesture. These gestures in contrast with MOVE AWAY gestures, present a smoother movement. They are characterized by a flat or cupped hand moving away from the speaker towards the interlocutor. 

In Figures 1f and g we present COVER and PRESS gestures of PDOH family. These gestures display tension and movement while the hand shape is of the same type, with fingers either stretching (dorsal type) or bent towards the palm (volar flexion) as termed in (Bressem & Ladewig, 2011: 68). COVER gestures are represented by a flat or cupped static hand. PRESS gesture has the same forms but there is a single or multiple down oriented movement(s).

Figure 1h. manifests TRACE gestures. These gestures are of movement type, commonly smooth, with fingers in either dorsal or volar flexion type, while the movement is horizontal and smooth.

To show the annotation procedure in identifying the discourse features via speech cues, let us consider a speech sample in which the participants commented on the difference between ложь and вранье (lie and falsehood): тыговоришьэтоложь / авраньеэтокогданапримерпротебя .. кто-топлохосказалэээкто-то .. / ниииинелицеприятноотебевысказался (what you say is a lie / and falsehood it is when for instance about you .. someone said something bad someone .. / disss dissastisfactorily expressed his opinion. The sample involves 3 minimal discourse units which correspond to the clauses or the speech fragments segmented by a prolonged pause; while two criteria are considered, syntactic and phonological (Kibrik & Podlesskaya, 2009). In the sample, the first discourse unit ты говоришь это ложь (what you say is a lie) relates to the event of situated communication while the speaker clearly construes the object of reference as an abstract notion bearing reference to manifestation of language (word) since we observe the indication to its mode of articulation. Pragmatically, this fragment is a statement and the object of reference is presented explicitly and is highly foregrounded being a part of a rhematic clause component (Iriskhanova, 2017; Kiose et al., 2023). Semantically, we observe a non-referential use of ложь (lie) since it occupies a predicate position (Paducheva, 2008), still it manifests attributive semantics; other clause components ты (you) and говоришь (say) indicate a subject and an act. The second discourse unit а вранье это когда например про тебя .. кто-то плохо сказал эээ кто-то .. (and falsehood it is when for instance about you .. someone said something bad someone ..) cognitively relates to the event of non-situated communication since this someone is clearly absent from the communicative situation shared by the speaking communicants. The speaker similarly to the first discourse unit construes the object of reference as an abstract notion. Pragmatically, this unit is also a statement; as opposed to the first discourse unit we observe the presence of vague reference in the construal of object of reference appearing in the use of approximator (metadiscourse accompanying comments in например (for instance) and placeholder (impersonal pronoun кто-то (someone) used repeatedly). Semantically, вранье (falsehood) is used as an object; other clause components relate to experiencer use of the subject in тебя (you), an act and a parameter. The third discourse unit (кто-то) / нииии нелицеприятно о тебе высказался ((someone ..) / disss dissastisfactorily expressed his opinion) cognitively develops the object of reference of the prior unit; therefore, it also relates to the event of non-situated communication and the object of reference is construed as an abstract notion. Pragmatically, this unit is also a statement; still as opposed to the second unit it does not manifest vague reference. Semantically, implicit вранье (falsehood) is used as an object; other clause components relate to experiencer use of the subject in тебе (you), an act and a parameter.

We propose that the distribution of mimetic schemas SHOW and RESTRAIN in contact-establishing gesture will be synchronized with specific cognitive, pragmatic and semantic cues and the latter in their turn will suffice to differentiate between the sub-schemas and typologize the gestures; although due to polysemous nature of language and gestures we do not expect rigid correlations.

4. Results and Discussion

In this Section we present the results which 1) describe the distribution of linguistic cues as mediated by PUOH contact-establishing gestures in SHOW mimetic schema and determine the clusters of its sub-schemas; 2) describe the distribution of linguistic cues as mediated by PDOH contact-establishing gestures in RESTRAIN mimetic schema and determine the clusters of its sub-schemas. To proceed, we establish the correspondences of PUOH and PDOH contact-establishing gestures and linguistic cues defined above. In the Section we also 3) propose the language typology of PUOH and PDOH contact-establishing gestures via their language profiles, which is facilitated by the idea that speech and gesture account for common thinking-for-speaking growth points. To establish the alignments and typology, annotation of all units was performed in ELAN and the distributions of mimetic sub-schemas, gestures and cognitive, pragmatic and semantic cues were determined. Below, we present the data describing their distribution.

4.1. Language and gesture in SHOW mimetic schema

In Table 1 the co-distribution of PUOH gestures demonstrating 6 mimetic sub-schemas of SHOW and cognitive, pragmatic and functional semantic language cues is shown.

Table 1 allows to establish 1) distinctions in the use of cognitive, pragmatic and functional semantic cues contingent on the sub-schemas of SHOW, 2) scale the SHOW sub-schemas considering their cognitive, pragmatic and functional semantic dimensions in speech, 3) cluster the SHOW sub-schemas considering their cognitive, pragmatic and functional semantic dimensions in speech.

The use of cognitive cues shows that on the whole the distribution of situated and non-situated communication events does not manifest any specificity in the sub-schemas (with the ATTRACT sub-schema being infrequent) with prevailing non-situated events; however, the HOLD sub-schema displays prevalence to higher situated events use and the WEIGHT sub-schema – to non-situated events. Presumably, these results prove that these two sub-schemas display a different construal potential. In our prior studies we found that in the construal of situated events participants commonly employ more pragmatic gestures and in the construal of non-situated events they tend to use more representational gestures (Iriskhanova et al., 2022), we may conclude that the HOLD sub-schema manifests a higher pragmatic potential, while the WEIGHT sub-schema displays a higher representational potential. Notably, the sub-schemas employ similar distribution of word and referent (object) construal. Overall, we observe that SHOW schema is used twice more often to convey non-situated communication with object as referents.

The use of pragmatic cues shows that the HOLD sub-schema is adopted unaccustomedly frequently with questions. Following the observations obtained in the experiment testing the use of gestures in question – response sequences (Holler et al., 2018), 82 per cent of gestures applied in questions were pragmatic gestures, we may conclude that HOLD sub-schemas display higher pragmatic potential contrasted with other sub-schemas. If we contrast the pragmatic construal of object of reference with the sub-schemas, we can observe that the MANIFEST sub-schema is more frequently applied to construe the object as being more definite, which may seem predictable since in manifesting we preconstrue a definite object. This means that the MANIFEST sub-schema displays a higher representational potential, still this is not representing an object of reference but the action with this object which is shown to the communicant. Consequently, apart from representational potential, MANIFEST gestures demonstrate deictic potential. Somewhat surprisingly, this sub-schema is less frequently than other sub-schemas used with the foregrounding speech cues. Importantly, the same situation occurs with the MOVE AWAY sub-schema. Presumably, here the directed action itself substitutes for the use of foregrounding in speech, which suffices to claim that the major function of these sub-schemas is clearly deictic.

The use of semantic cues shows that HOLD and WEIGH sub-schemas are synchronized with speech cues of states more frequently than other sub-schemas; the MOVE TOWARDS sub-schema shows a similar tendency although not so evidently (while MANIFEST and MOVE AWAY do not). To search for a relevant explanation, we address the semantic cues of object of reference. As shown, HOLD and WEIGH sub-schemas commonly manifest non-referential use of object of reference (in major cases in predicate position) while the referent is of a different type (however, mostly inactive or patient). This means that HOLD and WEIGH sub-schemas provide access to the object of reference but not to its features, functions or use. Presumably, holding and weighing the object even in an imagination allows to form bodily or sensory relations with its further construal. Consequently, these schemas are mostly related to representing an object of reference. As for the MOVE TOWARDS sub-schema which also is frequently used with the speech cues of states, presumably apart from relating to the state of the object of reference, it is oriented onto the interlocutor since it is commonly synchronized with speech cues of address.

The obtained results allow to scale the memetic sub-schemas as positioning the event and object of reference in terms of their cognitive, pragmatic and functional semantic potential. To scale the potential, we considered the quotients (the ratio) in each binary or polynomial opposition. Therefore, cognitive potential is estimated in terms of 1) Situatedness (Situated event / Non-situated event), 2) Embodiment (Object-referent / Word-referent). Pragmatic potential is estimated in terms of 1) Performativity (Performative in exclamations and questions / Constative act in statements), 2) Referent definiteness (Definite / Vague referent), 3) Referent foregrounding (Foregrounded / Backgrounded referent). Functional semantic potential is estimated in terms of 1) Reification (Reification displayed in Object and Subject / Predication displayed in Place, Time, Parameter, Attribute, Act, State, Address), 2) Dynamicity (Dynamicity displayed in Act / Stativity displayed in State), 3) Addressing (Address / No address), 4) Agentivity (Agentic / Non-agentic use), 5) Referentiality (Referential / Non-referential use). As seen, these oppositions describe both event and referent construal in mimetic sub-schemas. It is worth mentioning that in case there is scale discontinuity, we marked it with double index. In case the sub-schema is not found in a position, we do not include it into the table. If to identify a ratio, the result could not be obtained, we did not consider it. In Table 2 the dimension scales of SHOW mimetic schema are presented.

While we observe several scales ranging cognitive, pragmatic and functional semantic dimensions of mimetic sub-schemas, we can identify several regularities in their functioning. For instance, ATTRACT and HOLD commonly appear at different point of the scales. To cluster the sub-schemas, we apply Hierarchical clustering analysis (performed in Jamovi[2]). Six variables were clustered using Euclidian distance measure and clustering method Ward.D2. Two major clusters can be identified (ATTRACT vs other sub-schemas), still, to specify the use of sub-schemas within the second cluster we established three clusters (Figure 3). 

To describe each cluster, we addressed the dimension scales of SHOW mimetic schema.

Cluster 1 (ATTRACT) manifests non-situated event with the referents of abstract nature, high performativity, addressing and agentivity and low dynamicity.

Cluster 2 (WEIGH and MOVE TOWARDS) manifests non-situated events with the referents of low bodily nature, with low performativity and high referent foregrounding, high dynamicity, addressing, agentivity and referentiality.

Cluster 3 (MOVE AWAY, MANIFEST, HOLD) manifests situated events with bodily referents, with varied performativity, referent definiteness, low referent foregrounding, low addressing, agentivity and referentiality.

4.2. Language and gesture in RESTRAIN mimetic schema

In Table 2 the co-distribution of PDOH gestures demonstrating 5 mimetic sub-schemas of RESTRAIN and cognitive, pragmatic and functional semantic language cues is shown.

Table 2 allows to establish 1) distinctions in the use of cognitive, pragmatic and functional semantic cues contingent on the sub-schemas of RESTRAIN, 2) scale the RESTRAIN sub-schemas considering their cognitive, pragmatic and functional semantic dimensions in speech, 3) cluster the RESTRAIN sub-schemas considering their cognitive, pragmatic and functional semantic dimensions in speech.

The use of cognitive cues shows that the distribution of situated and non-situated communication events is on the whole similar with prevailing situated events apart from the TRACE sub-schema (and the schema PUSH AWAY which is infrequent). Interestingly, while RESTRAIN schema is commonly used to construe situated events, the schema SHOW typically construes non-situated events. Developing the ideas presented in (Iriskhanova et al., 2022), we thus infer that RESTRAIN schema overwhelmingly manifests high pragmatic potential. However, the TRACE sub-schema displays higher representational potential since it is used in construe non-situated events. Importantly, as opposed to SHOW schema used majorly twice more often to construe referents than words, the schema RESTRAIN manifests more ambiguous construal patterns. While COVER, LOCATE and (probably) PUSH AWAY prevailingly relate to referent rather than words, the sub-schemas PRESS and TRACE while displaying the same tendency are also quite common in the construal of word. Supposedly, these two sub-schemas manifest common for construal of both word and object patterns of keeping them under control and searching for their boundaries.

The use of pragmatic cues shows that PRESS and TRACE again display uncommon pattern of frequent use in statements; however, this observation only supports the idea that these sub-schemas are used to develop the communicative topic while keeping the object of reference under control or tracing them. Besides, PRESS is even more common in exclamations which helps maintain the object of reference under control and emphasizes the pragmaticity of this action. The prevalence of vague reference in the referent construal in PRESS additionally specifies that the idea of maintaining the referent under control originates from the desire to clarify it. PRESS is also more frequently used with backgrounding speech cues, which again conforms to the idea of searching for the ways to clarify the object of reference.  

The use of semantic cues shows additionally attests to the specificity of the PRESS sub-schema since it is used uncommonly frequent with states and parameters. This observation allows to propose that the search for ways of clarifying the object of reference is found in intensifying or alleviating the manifestations of features in referents. It is worth noticing that COVER frequently conveys address. Probably, the speaker wants to attract the attention of the interlocutor to the object of reference which is kept concealed. TRACE is used to display state and result; this seems expected due to the fact that it commonly appears in statements. Importantly, none of these sub-schemas demonstrate non-referential use of object of reference, while PRESS manifests the use of both agent and inactive referent since the speaker construes himself as someone who maintains control over the object of reference. In Figure 4 the dimensions of RESTRAIN mimetic schema are presented.

To cluster the sub-schemas, we apply the Hierarchical clustering analysis (performed in Jamovi). 6 variables were clustered using a Euclidian distance measure and clustering method Ward.D2. 2 major clusters can be identified (TRACE and PUSH AWAY vs other sub-schemas), still, to specify the use of sub-schemas within the second cluster we established 3 clusters (Figure 5). 

To describe each cluster in RESTRAIN schema, we followed the procedure described above.

Cluster 1 (TRACE and PUSH AWAY) manifests non-situated event with the referents of abstract nature, low performativity, high referent foregrounding and low agentivity.

Cluster 2 (PRESS) manifests very high referent vagueness, low foregrounding, very high dynamicity and agentivity.

Cluster 3 (COVER and LOCATE) manifests situated events with bodily referents, with high performativity, high addressing, high agentivity and referentiality.  

Overall, the results demonstrate that SHOW sub-schemas manifest higher diversity than RESTRAIN sub-schemas. Among SHOW sub-schemas, the ATTRACT sub-schema displays major difference from other sub-schemas; among RESTRAIN sub-schemas, this is the PRESS sub-schema which manifests major difference. These differences account for various dimensions; however, ATTRACT demonstrates specificity in all three dimensions and PRESS specificity is mostly attributed to pragmatic and functional semantic dimensions. However, PRESS appears regularly in contact-establishing gesture and ATTRACT is far less common, which may explain its variance.

4.1. Language typology of recurrent PUOH and PDOH gestures

Additionally, we used the linguistic data to specify the family of recurrent PUOH and PDOH gestures. Therefore, we applied indirect measurement or the analysis of distribution of one communicative, perceptual or semiotic mode to typologize the aligned mode (Kiose, 2021). In this case indirect measurement was used to specify the typological characteristics of one communicative mode (gestures) applying the concurrent typological characteristics of the other contingent communicative mode (speech). Methodologically, we presume that since the explored linguistic characteristics relate to cognitive, pragmatic and functional semantic features of speech, they simultaneously determine the choice of mimetic schemas displayed in gestures. This idea stems from the theory of growth points (McNeill, 2005) or dynamic units of online verbal thinking which mediate the process of information construal. To proceed, we performed Hierarchical clustering with all gesture types representing 11 sub-schemas. We can hypothesize that if the distinction onto PUOH and PDOH gestures (and their mimetic schemas) is typologically relevant and significant, their distribution in contact-establishing gestures will manifest specific distribution in linguistic characteristics. Otherwise, two inferences are possible. Either the distribution of linguistic characteristics is not an effective measure to specify the typological distribution of gestures due to highly polysemous nature of language or the distribution of gestures does not manifest any obvious typological features. Presumably, a two-cluster solution might serve to identify whether recurrent PUOH and PDOH gestures do constitute two typologically different groups.

In Figure 6 we present the two-cluster solution.

Figure 3 shows that two clusters clearly manifest different distribution of linguistic characteristics due to significant between-cluster distance. Meanwhile, we observe that within-cluster distance in Cluster 2 is more obvious than in Cluster 1, which signifies that the typological differences (mediated indirectly by linguistic characteristics) in Cluster 2 are more distinct. The distribution of gestures within each cluster although does not fully correlate to PUOH and PDOH gesture distribution which should be attended.

Cluster 1 majorly unites PUOH gestures with an exception of PUSH AWAY and TRACE gestures; still in both cases these are the gestures which manifest high within-cluster specificity as seen in Figure 3. Bearing in mind that our corpus contains very few PUSH AWAY gestures, we can discard them. TRACE gestures although are represented in a larger number. If we address Table 3, we can identify that they fell into the cluster of PUOH gestures due to the fact that mimetic sub-schemas related to TRACE display specificity in all three linguistic dimensions – cognitive, pragmatic and functional semantic. This yields a conclusion that TRACE gestures occupy a somewhat intermediary typological position related to both PUOH and PDOH gesture semantics. A sound explanation for this fact may be that TRACE gestures imply rotation which following Cienki (2021) plays a fundamental role in the production of PUOH and PDOH. A second possible explanation may be that TRACE gestures display a functional difference; apart from the major function of “inviting the participants to take on a shared perspective on this object” (Müller, 2004: 233), they single out its contour thus representing not the “object of presence” manifested to reach the interlocutor, but the “object of representation” or just personally experienced object as Freedman specifies it (Freedman, 1972: 158, in Cienki, 2021: 25). Importantly, other gestures within this cluster do fall within PUOH group, which strengthens the idea of their typological unity. Additionally, we can identify the most prototypical PUOH gestures or the gestures with very low within-cluster variance, these are MANIFEST, HOLD, WEIGH and MOVE gestures. 

Cluster 2 majorly unites PDOH gestures with the exception of ATTRACT gestures, which was quite expected due to their small number in our corpus. Consequently, the fact of their attribution should be considered in a separate study on a larger corpus data. Meanwhile, other three gesture types, PRESS, COVER, LOCATE fell into the same group, which attests to their typological affinity in contact-establishing gestures. It is noticeable although that these gestures display higher within-cluster variance but this may account for the smaller number of cluster members; still, they display lower prototypicality effects. This fact may also be accounted for in the use of rotation (Cienki, 2021).

What also seems relevant is that the application of tripartite linguistic dimensions following (Van Dijk, 1990) appeared efficient in attesting to the distinction of two gesture groups, PUOH and PDOH gestures. Further application of the classifications of events and reference in terms of their cognitive, pragmatic and functional semantic nature, we specified the differences between two major types of recurrent gestures.

Below, in Figs. 7a and 7b the integrated language profiles of PUOH and PDOH gestures are presented.

The Figures do not involve the profiles of ATTRACT PUOH gestures and TRACE PDOH gestures due to their low frequency in the first case and due to their variance in the second case. As it can be seen from Figures 7a and 7b, the differences between language profiles of PUOH and PDOH gestures lie in all 3 dimensions. In cognitive dimension we observe the prevalence of non-situated events in PUOH gestures and the prevalence of situated events in PDOH gestures as well as the prevalence of object-referent construal in PUOH gestures. In pragmatic dimension there is the difference in the prevalence of constativity in PUOH gestures and in performativity in PDOH gestures. In functional semantic dimension in PDOH gestures prevail with acts, while PUOH gestures are more frequented with attributes, which additionally stress the prevalence of constativity in PUOH gestures and point at higher dynamicity of PDOH gestures. Several less significant differences may also be observed; still, they were identified in Tables 2 and 3.

5. Final remarks

The research on multimodal contact-establishing presented in this paper contributes to the ongoing research in speech and gesture studies in several respects.

First, it supplements the existing typologies of recurrent gestures by providing a solid language ground which helps explain their semantics. In the study, we addressed the alignment patterns of recurrent palm-up-open-hand (PUOH) and palm-down-open-hand (PDOH) gestures and their language counterparts displaying two mimetic schemas, SHOW and RESTRAIN. Adopting the 3-level typology of verbal cues, describing the cognitive, pragmatic, and functional semantic dimensions, we determined their distribution among 6 mimetic sub-schemas of SHOW (MANIFEST, HOLD, WEIGH, MOVE AWAY, MOVE TOWARDS, ATTRACT) and 5 mimetic sub-schemas of RESTRAIN (COVER, PRESS, LOCATE, TRACE, PUSH AWAY). The results evidence that SHOW sub-schemas manifest higher diversity than RESTRAIN sub-schemas. We found that among SHOW sub-schemas, the ATTRACT sub-schema displays higher difference from other sub-schemas; among RESTRAIN sub-schemas, this is the PRESS sub-schema which manifests higher difference. These differences account for various verbal dimensions; however, ATTRACT demonstrates specificity in all three dimensions and PRESS’s specificity is mostly attributed to pragmatic and functional semantic dimensions. However, PRESS appears regularly in contact-establishing gesture and ATTRACT is far less common, which may explain its variance. Therefore, by offering a linguistic description to gesture performance, the study offers a clearer differentiation of mimetic schemas and sub-schemas with regard to their use in contact-establishing.

Second, the study contributes to the discussion on questions related to the use of linguistic data to specify the family of recurrent PUOH and PDOH gestures in contact-establishing. To proceed, we applied indirect measures to reveal the typological characteristics of gestures as one communicative mode applying the concurrent typological characteristics of the other contingent mode, which is speech. The results show that the differences between language profiles of PUOH and PDOH gestures lie in all 3 dimensions. In cognitive dimension the prevalence of non-situated events in PUOH gestures and the prevalence of situated events in PDOH gestures was found, additionally we established the prevalence of object-referent construal in PUOH gestures. In pragmatic dimension we found the difference in the prevalence of constativity in PUOH gestures and in performativity in PDOH gestures. In functional semantic dimension PDOH gestures display contingency with acts, while PUOH gestures commonly appear with attributes.

The results prove the efficiency of alignment analysis to explore the multimodal nature of contact-establishing. They also display the perspectives of indirect measures analysis to explore the gesture profiles via verbal cues. Most noticeably, the differences between two mimetic schemas were observed via the differences in the use of verbal cues. Additionally, the present study specifies the cues which contribute to this specificity. The results obtained in the present study might be applied to develop neural networks which constitute the base for various robotic systems, conversational agents, as well as to enhance computer-human communication.

Overall, this paper makes a contribution to both multimodal research of contact-establishing and to gesture studies. The data provide new evidence in the phenomena of recurrent gestures observable in contact-establishing, and in doing so, it expands the fields of investigation in multimodal studies.