Research on multimodal social media information popularity prediction based on large language model - WANG Jie; WANG Zitong; PENG Yan; HAO Bowen (Tongxin xuebao)
To address the limitations of strong feature dependency, insufficient generalization, and inadequate performance in few-shot/cold-start settings in existing multimodal social media popularity prediction algorithms, a MultiSmpLLM model based on large language model with instruction fine-tuning and human alignment was proposed. Firstly, the task of multimodal social media popularity prediction for cold-start users was defined. Secondly, multimodal fine-tuning instructions were constructed, and the large language model (Llama3) was instructionally fine-tuned using the low-rank adaptation (LoRA) and parameter freeze (Freeze) method. Finally, an improved direct preference optimization (DPO) algorithm IDPOP was developed by constructing preference data and adding a parameter-tuned penalty to the DPO loss function, resolving instability and non-convergence in RLHF and incorrect optimization in standard DPO for social media popularity prediction. Experiments show MultiSmpLLM outperforms conventional multimodal prediction models and multimodal large language models such as GPT-4o.