LLM-driven agent for speech-enabled control of industrial robots: A case study in snow-crab quality inspection
This study investigates the integration of large language models (LLMs) into a voice- and vision-based robotic control system in autonomous industrial applications. The main objective is to demonstrate that an LLM-based agent can interpret natural instructions, dynamically plan movements, and execut...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2025-09-01
|
| Series: | Results in Engineering |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2590123025027276 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Summary: | This study investigates the integration of large language models (LLMs) into a voice- and vision-based robotic control system in autonomous industrial applications. The main objective is to demonstrate that an LLM-based agent can interpret natural instructions, dynamically plan movements, and execute robotic actions without domain-specific supervised learning, thereby enabling autonomous robotic planning. The proposed system relies on a voice interface, an LLM agent, and tools for real-time robot control. A dedicated communication module was developed to ensure the full control of a KUKA industrial robot using WebSocket, without resorting to proprietary solutions. To validate the approach, a case study was conducted using a robotic cell, which was applied to snow-crab sorting, where computer vision provides real-time perception. The experimental evaluation covered a wide range of commands, including movement instructions, complex planning tasks (e.g., trajectory generation), and visual queries based on crab quality, size, and anatomy. The results showed that the model exhibited robust interpretation capabilities with an overall success rate of 98.46%. These performances highlight the potential of LLMs to facilitate human-robot interaction in real industrial environments, while reducing programming complexity and increasing system autonomy. |
|---|---|
| ISSN: | 2590-1230 |