Multimodal Dynamics of Hand Gestures and Pauses in Multiparty Interactions

This study examines multimodal patterns linking hand gestures and pauses in multiparty interactions, distinguishing between within-speaker pauses and between-speaker pauses. Using the MULTISIMO corpus, which includes annotated audio-visual recordings of collaborative dialogues, we analysed, for each category of gesture, their distribution, pause duration, and the timing of gesture onset and offset relative to the pause. Results showed distinct gestural timing patterns involving within-speaker and between-speaker pauses. Self-adaptors were associated with longer pauses, possibly reflecting increased cognitive demands or speech planning. While syntactic position influenced pause duration, with shorter pauses at utterance endings, it did not impact gesture onset and offset. Additionally, most pauses containing gestures occurred within utterances, regardless of gesture type.