Research on multimodal social media information popularity prediction based on large language model
To address the limitations of strong feature dependency, insufficient generalization, and inadequate performance in few-shot/cold-start settings in existing multimodal social media popularity prediction algorithms, a MultiSmpLLM model based on large language model with instruction fine-tuning and hu...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | zho |
| Published: |
Editorial Department of Journal on Communications
2024-11-01
|
| Series: | Tongxin xuebao |
| Subjects: | |
| Online Access: | http://www.joconline.com.cn/zh/article/doi/10.11959/j.issn.1000-436x.2024193/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841537125725503488 |
|---|---|
| author | WANG Jie WANG Zitong PENG Yan HAO Bowen |
| author_facet | WANG Jie WANG Zitong PENG Yan HAO Bowen |
| author_sort | WANG Jie |
| collection | DOAJ |
| description | To address the limitations of strong feature dependency, insufficient generalization, and inadequate performance in few-shot/cold-start settings in existing multimodal social media popularity prediction algorithms, a MultiSmpLLM model based on large language model with instruction fine-tuning and human alignment was proposed. Firstly, the task of multimodal social media popularity prediction for cold-start users was defined. Secondly, multimodal fine-tuning instructions were constructed, and the large language model (Llama3) was instructionally fine-tuned using the low-rank adaptation (LoRA) and parameter freeze (Freeze) method. Finally, an improved direct preference optimization (DPO) algorithm IDPOP was developed by constructing preference data and adding a parameter-tuned penalty to the DPO loss function, resolving instability and non-convergence in RLHF and incorrect optimization in standard DPO for social media popularity prediction. Experiments show MultiSmpLLM outperforms conventional multimodal prediction models and multimodal large language models such as GPT-4o. |
| format | Article |
| id | doaj-art-bd05c145bb0c4603840c6eef2030bf29 |
| institution | Kabale University |
| issn | 1000-436X |
| language | zho |
| publishDate | 2024-11-01 |
| publisher | Editorial Department of Journal on Communications |
| record_format | Article |
| series | Tongxin xuebao |
| spelling | doaj-art-bd05c145bb0c4603840c6eef2030bf292025-01-14T08:46:18ZzhoEditorial Department of Journal on CommunicationsTongxin xuebao1000-436X2024-11-014514115679134326Research on multimodal social media information popularity prediction based on large language modelWANG JieWANG ZitongPENG YanHAO BowenTo address the limitations of strong feature dependency, insufficient generalization, and inadequate performance in few-shot/cold-start settings in existing multimodal social media popularity prediction algorithms, a MultiSmpLLM model based on large language model with instruction fine-tuning and human alignment was proposed. Firstly, the task of multimodal social media popularity prediction for cold-start users was defined. Secondly, multimodal fine-tuning instructions were constructed, and the large language model (Llama3) was instructionally fine-tuned using the low-rank adaptation (LoRA) and parameter freeze (Freeze) method. Finally, an improved direct preference optimization (DPO) algorithm IDPOP was developed by constructing preference data and adding a parameter-tuned penalty to the DPO loss function, resolving instability and non-convergence in RLHF and incorrect optimization in standard DPO for social media popularity prediction. Experiments show MultiSmpLLM outperforms conventional multimodal prediction models and multimodal large language models such as GPT-4o.http://www.joconline.com.cn/zh/article/doi/10.11959/j.issn.1000-436x.2024193/large language modelpopularity predictioninstruction fine-tuninghuman alignment |
| spellingShingle | WANG Jie WANG Zitong PENG Yan HAO Bowen Research on multimodal social media information popularity prediction based on large language model Tongxin xuebao large language model popularity prediction instruction fine-tuning human alignment |
| title | Research on multimodal social media information popularity prediction based on large language model |
| title_full | Research on multimodal social media information popularity prediction based on large language model |
| title_fullStr | Research on multimodal social media information popularity prediction based on large language model |
| title_full_unstemmed | Research on multimodal social media information popularity prediction based on large language model |
| title_short | Research on multimodal social media information popularity prediction based on large language model |
| title_sort | research on multimodal social media information popularity prediction based on large language model |
| topic | large language model popularity prediction instruction fine-tuning human alignment |
| url | http://www.joconline.com.cn/zh/article/doi/10.11959/j.issn.1000-436x.2024193/ |
| work_keys_str_mv | AT wangjie researchonmultimodalsocialmediainformationpopularitypredictionbasedonlargelanguagemodel AT wangzitong researchonmultimodalsocialmediainformationpopularitypredictionbasedonlargelanguagemodel AT pengyan researchonmultimodalsocialmediainformationpopularitypredictionbasedonlargelanguagemodel AT haobowen researchonmultimodalsocialmediainformationpopularitypredictionbasedonlargelanguagemodel |