Research on multimodal social media information popularity prediction based on large language model
To address the limitations of strong feature dependency, insufficient generalization, and inadequate performance in few-shot/cold-start settings in existing multimodal social media popularity prediction algorithms, a MultiSmpLLM model based on large language model with instruction fine-tuning and hu...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | zho |
| Published: |
Editorial Department of Journal on Communications
2024-11-01
|
| Series: | Tongxin xuebao |
| Subjects: | |
| Online Access: | http://www.joconline.com.cn/zh/article/doi/10.11959/j.issn.1000-436x.2024193/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Summary: | To address the limitations of strong feature dependency, insufficient generalization, and inadequate performance in few-shot/cold-start settings in existing multimodal social media popularity prediction algorithms, a MultiSmpLLM model based on large language model with instruction fine-tuning and human alignment was proposed. Firstly, the task of multimodal social media popularity prediction for cold-start users was defined. Secondly, multimodal fine-tuning instructions were constructed, and the large language model (Llama3) was instructionally fine-tuned using the low-rank adaptation (LoRA) and parameter freeze (Freeze) method. Finally, an improved direct preference optimization (DPO) algorithm IDPOP was developed by constructing preference data and adding a parameter-tuned penalty to the DPO loss function, resolving instability and non-convergence in RLHF and incorrect optimization in standard DPO for social media popularity prediction. Experiments show MultiSmpLLM outperforms conventional multimodal prediction models and multimodal large language models such as GPT-4o. |
|---|---|
| ISSN: | 1000-436X |