Enhancing human-centered dynamic scene understanding via multiple LLMs collaborated reasoning
Abstract Human-centered dynamic scene understanding plays a pivotal role in enhancing the capability of robotic and autonomous systems, where video-based human-object interaction (V-HOI) detection is a crucial task in semantic scene understanding, which aims to comprehensively understand HOI relatio...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Springer
2025-03-01
|
| Series: | Visual Intelligence |
| Subjects: | |
| Online Access: | https://doi.org/10.1007/s44267-025-00074-1 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849391928784715776 |
|---|---|
| author | Hang Zhang Wenxiao Zhang Haoxuan Qu Jun Liu |
| author_facet | Hang Zhang Wenxiao Zhang Haoxuan Qu Jun Liu |
| author_sort | Hang Zhang |
| collection | DOAJ |
| description | Abstract Human-centered dynamic scene understanding plays a pivotal role in enhancing the capability of robotic and autonomous systems, where video-based human-object interaction (V-HOI) detection is a crucial task in semantic scene understanding, which aims to comprehensively understand HOI relationships within a video to benefit the behavioral decisions of mobile robots and autonomous driving systems. Although previous V-HOI detection models have made significant advances in accurate detection on specific datasets, they still lack the general reasoning ability of humans to effectively induce HOI relationships. In this study, we propose V-HOI multi-LLMs collaborated reasoning (V-HOI MLCR), a novel framework consisting of a series of plug-and-play modules that could facilitate the performance of current V-HOI detection models by leveraging the strong reasoning ability of different off-the-shelf pre-trained large language models (LLMs). We design a two-stage collaboration system of different LLMs for the V-HOI task. Specifically, in the first stage, we design a cross-agents reasoning scheme to leverage the LLM to perform reasoning from different aspects. In the second stage, we perform multi-LLMs debate to get the final reasoning answer based on the different knowledge in different LLMs. Additionally, we develop an auxiliary training strategy using CLIP, a large vision-language model to enhance the base V-HOI models’ discriminative ability to better cooperate with LLMs. We validate the superiority of our design by demonstrating its effectiveness in improving the predictive accuracy of the base V-HOI model through reasoning from multiple perspectives. |
| format | Article |
| id | doaj-art-3c791fa0f62f46a4ab84071b067b56c5 |
| institution | Kabale University |
| issn | 2097-3330 2731-9008 |
| language | English |
| publishDate | 2025-03-01 |
| publisher | Springer |
| record_format | Article |
| series | Visual Intelligence |
| spelling | doaj-art-3c791fa0f62f46a4ab84071b067b56c52025-08-20T03:40:53ZengSpringerVisual Intelligence2097-33302731-90082025-03-013111110.1007/s44267-025-00074-1Enhancing human-centered dynamic scene understanding via multiple LLMs collaborated reasoningHang Zhang0Wenxiao Zhang1Haoxuan Qu2Jun Liu3Singapore University of Technology and DesignHohai UniversityLancaster UniversityLancaster UniversityAbstract Human-centered dynamic scene understanding plays a pivotal role in enhancing the capability of robotic and autonomous systems, where video-based human-object interaction (V-HOI) detection is a crucial task in semantic scene understanding, which aims to comprehensively understand HOI relationships within a video to benefit the behavioral decisions of mobile robots and autonomous driving systems. Although previous V-HOI detection models have made significant advances in accurate detection on specific datasets, they still lack the general reasoning ability of humans to effectively induce HOI relationships. In this study, we propose V-HOI multi-LLMs collaborated reasoning (V-HOI MLCR), a novel framework consisting of a series of plug-and-play modules that could facilitate the performance of current V-HOI detection models by leveraging the strong reasoning ability of different off-the-shelf pre-trained large language models (LLMs). We design a two-stage collaboration system of different LLMs for the V-HOI task. Specifically, in the first stage, we design a cross-agents reasoning scheme to leverage the LLM to perform reasoning from different aspects. In the second stage, we perform multi-LLMs debate to get the final reasoning answer based on the different knowledge in different LLMs. Additionally, we develop an auxiliary training strategy using CLIP, a large vision-language model to enhance the base V-HOI models’ discriminative ability to better cooperate with LLMs. We validate the superiority of our design by demonstrating its effectiveness in improving the predictive accuracy of the base V-HOI model through reasoning from multiple perspectives.https://doi.org/10.1007/s44267-025-00074-1Scene understandingLarge language modelsKnowledge-based reasoning |
| spellingShingle | Hang Zhang Wenxiao Zhang Haoxuan Qu Jun Liu Enhancing human-centered dynamic scene understanding via multiple LLMs collaborated reasoning Visual Intelligence Scene understanding Large language models Knowledge-based reasoning |
| title | Enhancing human-centered dynamic scene understanding via multiple LLMs collaborated reasoning |
| title_full | Enhancing human-centered dynamic scene understanding via multiple LLMs collaborated reasoning |
| title_fullStr | Enhancing human-centered dynamic scene understanding via multiple LLMs collaborated reasoning |
| title_full_unstemmed | Enhancing human-centered dynamic scene understanding via multiple LLMs collaborated reasoning |
| title_short | Enhancing human-centered dynamic scene understanding via multiple LLMs collaborated reasoning |
| title_sort | enhancing human centered dynamic scene understanding via multiple llms collaborated reasoning |
| topic | Scene understanding Large language models Knowledge-based reasoning |
| url | https://doi.org/10.1007/s44267-025-00074-1 |
| work_keys_str_mv | AT hangzhang enhancinghumancentereddynamicsceneunderstandingviamultiplellmscollaboratedreasoning AT wenxiaozhang enhancinghumancentereddynamicsceneunderstandingviamultiplellmscollaboratedreasoning AT haoxuanqu enhancinghumancentereddynamicsceneunderstandingviamultiplellmscollaboratedreasoning AT junliu enhancinghumancentereddynamicsceneunderstandingviamultiplellmscollaboratedreasoning |