Dynamic Tuning and Multi-Task Learning-Based Model for Multimodal Sentiment Analysis
Multimodal sentiment analysis aims to uncover human affective states by integrating data from multiple sensory sources. However, previous studies have focused on optimizing model architecture, neglecting the impact of objective function settings on model performance. Given this, this study introduce...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-06-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/15/11/6342 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849330637470695424 |
|---|---|
| author | Yi Liang Turdi Tohti Wenpeng Hu Bo Kong Dongfang Han Tianwei Yan Askar Hamdulla |
| author_facet | Yi Liang Turdi Tohti Wenpeng Hu Bo Kong Dongfang Han Tianwei Yan Askar Hamdulla |
| author_sort | Yi Liang |
| collection | DOAJ |
| description | Multimodal sentiment analysis aims to uncover human affective states by integrating data from multiple sensory sources. However, previous studies have focused on optimizing model architecture, neglecting the impact of objective function settings on model performance. Given this, this study introduces a new framework, DMMSA, which utilizes the intrinsic correlation of sentiment signals and enhances the model’s understanding of complex sentiments. DMMSA incorporates coarse-grained sentiment analysis to reduce task complexity. Meanwhile, it embeds a contrastive learning mechanism within the modality, which decomposes unimodal features into similar and dissimilar ones, thus allowing for the simultaneous consideration of both unimodal and multimodal emotions. We tested DMMSA on the CH-SIMS, MOSI, and MOEI datasets. When only changing the optimization objectives, DMMSA achieved accuracy gains of 3.2%, 1.57%, and 1.95% over the baseline in five-class and seven-class classification tasks. In regression tasks, DMMSA reduced the Mean Absolute Error (MAE) by 1.46%, 1.5%, and 2.8% compared to the baseline. |
| format | Article |
| id | doaj-art-2d0ddb57b7904b539d2ede4e04445c0b |
| institution | Kabale University |
| issn | 2076-3417 |
| language | English |
| publishDate | 2025-06-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj-art-2d0ddb57b7904b539d2ede4e04445c0b2025-08-20T03:46:50ZengMDPI AGApplied Sciences2076-34172025-06-011511634210.3390/app15116342Dynamic Tuning and Multi-Task Learning-Based Model for Multimodal Sentiment AnalysisYi Liang0Turdi Tohti1Wenpeng Hu2Bo Kong3Dongfang Han4Tianwei Yan5Askar Hamdulla6School of Computer Science and Technology, Xinjiang University, Urumqi 830017, ChinaSchool of Computer Science and Technology, Xinjiang University, Urumqi 830017, ChinaInformation Research Center of Military Science, Beijing 100142, ChinaSchool of Computer Science and Technology, Xinjiang University, Urumqi 830017, ChinaSchool of Computer Science and Technology, Xinjiang University, Urumqi 830017, ChinaInformation Research Center of Military Science, Beijing 100142, ChinaSchool of Computer Science and Technology, Xinjiang University, Urumqi 830017, ChinaMultimodal sentiment analysis aims to uncover human affective states by integrating data from multiple sensory sources. However, previous studies have focused on optimizing model architecture, neglecting the impact of objective function settings on model performance. Given this, this study introduces a new framework, DMMSA, which utilizes the intrinsic correlation of sentiment signals and enhances the model’s understanding of complex sentiments. DMMSA incorporates coarse-grained sentiment analysis to reduce task complexity. Meanwhile, it embeds a contrastive learning mechanism within the modality, which decomposes unimodal features into similar and dissimilar ones, thus allowing for the simultaneous consideration of both unimodal and multimodal emotions. We tested DMMSA on the CH-SIMS, MOSI, and MOEI datasets. When only changing the optimization objectives, DMMSA achieved accuracy gains of 3.2%, 1.57%, and 1.95% over the baseline in five-class and seven-class classification tasks. In regression tasks, DMMSA reduced the Mean Absolute Error (MAE) by 1.46%, 1.5%, and 2.8% compared to the baseline.https://www.mdpi.com/2076-3417/15/11/6342multimodal sentiment analysiscontrastive learningmulti-task learningdynamic tuning |
| spellingShingle | Yi Liang Turdi Tohti Wenpeng Hu Bo Kong Dongfang Han Tianwei Yan Askar Hamdulla Dynamic Tuning and Multi-Task Learning-Based Model for Multimodal Sentiment Analysis Applied Sciences multimodal sentiment analysis contrastive learning multi-task learning dynamic tuning |
| title | Dynamic Tuning and Multi-Task Learning-Based Model for Multimodal Sentiment Analysis |
| title_full | Dynamic Tuning and Multi-Task Learning-Based Model for Multimodal Sentiment Analysis |
| title_fullStr | Dynamic Tuning and Multi-Task Learning-Based Model for Multimodal Sentiment Analysis |
| title_full_unstemmed | Dynamic Tuning and Multi-Task Learning-Based Model for Multimodal Sentiment Analysis |
| title_short | Dynamic Tuning and Multi-Task Learning-Based Model for Multimodal Sentiment Analysis |
| title_sort | dynamic tuning and multi task learning based model for multimodal sentiment analysis |
| topic | multimodal sentiment analysis contrastive learning multi-task learning dynamic tuning |
| url | https://www.mdpi.com/2076-3417/15/11/6342 |
| work_keys_str_mv | AT yiliang dynamictuningandmultitasklearningbasedmodelformultimodalsentimentanalysis AT turditohti dynamictuningandmultitasklearningbasedmodelformultimodalsentimentanalysis AT wenpenghu dynamictuningandmultitasklearningbasedmodelformultimodalsentimentanalysis AT bokong dynamictuningandmultitasklearningbasedmodelformultimodalsentimentanalysis AT dongfanghan dynamictuningandmultitasklearningbasedmodelformultimodalsentimentanalysis AT tianweiyan dynamictuningandmultitasklearningbasedmodelformultimodalsentimentanalysis AT askarhamdulla dynamictuningandmultitasklearningbasedmodelformultimodalsentimentanalysis |