EXPERIMENTAL METHODS OF EXPLORING MULTIMODAL DISCOURSE: CROSSMODAL ALIGNMENT
The study advances Crossmodal Alignment Framework to explore multimodal discourse in its three formats – semiotic, communicative and perceptive – via multimodal experiment. It considers the alignment patterns obtained from two semiotic modes (text and image), transferred in two communicative modes...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Volgograd State University
2024-11-01
|
| Series: | Vestnik Volgogradskogo Gosudarstvennogo Universiteta. Seriâ 2. Âzykoznanie |
| Subjects: | |
| Online Access: | https://l.jvolsu.com/index.php/en/archive-en/930-science-journal-of-volsu-linguistics-2024-vol-23-no-5/modelling-parametres-of-the-text-for-the-development-of-artificial-intelligence-systems/2826-kiose-m-i-potekhin-v-o-zubkov-o-d-experimental-methods-of-exploring-multimodal-discourse-crossmodal-alignment |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Summary: | The study advances Crossmodal Alignment Framework to explore multimodal discourse in its three formats – semiotic, communicative and perceptive – via multimodal experiment. It considers the alignment
patterns obtained from two semiotic modes (text and image), transferred in two communicative modes (speech and gesture), sensed by two perception modes (visual and audial). The common research framework determines
the patterns as modulated by discourse tasks. The study features the results of multimodal experiments with the participants engaged in three discourse tasks: 1) receptive, which presumes obtaining information from text and
image stimuli; semiotic alignment patterns are identified indirectly via participants’ gaze response; 2) productive, in which the participants communicate the information in monological format; communicative alignment patterns are identified directly via their speech and gesture; 3) receptive-productive, which presupposes the participants perceive information visually and audially; alignment patterns are identified directly via participants’ gaze behavior contingent on the stimuli areas of interest and indirectly via their speech response. Data analysis allows to
determine and scale the degree of crossmodal alignment to discourse tasks, which helps identify the input of each mode to solving these tasks. The research framework and obtained results contribute to further development
of multimodal discourse methods. |
|---|---|
| ISSN: | 1998-9911 2409-1979 |