LLM-driven agent for speech-enabled control of industrial robots: A case study in snow-crab quality inspection
This study investigates the integration of large language models (LLMs) into a voice- and vision-based robotic control system in autonomous industrial applications. The main objective is to demonstrate that an LLM-based agent can interpret natural instructions, dynamically plan movements, and execut...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2025-09-01
|
| Series: | Results in Engineering |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2590123025027276 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849330485251014656 |
|---|---|
| author | Ibrahim Kadri Sid Ahmed Selouani Mohsen Ghribi Rayen Ghali Sabrina Mekhoukh |
| author_facet | Ibrahim Kadri Sid Ahmed Selouani Mohsen Ghribi Rayen Ghali Sabrina Mekhoukh |
| author_sort | Ibrahim Kadri |
| collection | DOAJ |
| description | This study investigates the integration of large language models (LLMs) into a voice- and vision-based robotic control system in autonomous industrial applications. The main objective is to demonstrate that an LLM-based agent can interpret natural instructions, dynamically plan movements, and execute robotic actions without domain-specific supervised learning, thereby enabling autonomous robotic planning. The proposed system relies on a voice interface, an LLM agent, and tools for real-time robot control. A dedicated communication module was developed to ensure the full control of a KUKA industrial robot using WebSocket, without resorting to proprietary solutions. To validate the approach, a case study was conducted using a robotic cell, which was applied to snow-crab sorting, where computer vision provides real-time perception. The experimental evaluation covered a wide range of commands, including movement instructions, complex planning tasks (e.g., trajectory generation), and visual queries based on crab quality, size, and anatomy. The results showed that the model exhibited robust interpretation capabilities with an overall success rate of 98.46%. These performances highlight the potential of LLMs to facilitate human-robot interaction in real industrial environments, while reducing programming complexity and increasing system autonomy. |
| format | Article |
| id | doaj-art-a3e9480c1a3a4809859d38deec7d79e1 |
| institution | Kabale University |
| issn | 2590-1230 |
| language | English |
| publishDate | 2025-09-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Results in Engineering |
| spelling | doaj-art-a3e9480c1a3a4809859d38deec7d79e12025-08-20T03:46:54ZengElsevierResults in Engineering2590-12302025-09-012710666010.1016/j.rineng.2025.106660LLM-driven agent for speech-enabled control of industrial robots: A case study in snow-crab quality inspectionIbrahim Kadri0Sid Ahmed Selouani1Mohsen Ghribi2Rayen Ghali3Sabrina Mekhoukh4Université de Moncton, Campus of Shippagan, 218 Bd J. D. Gauthier, Shippagan, E8S 1P6, NB, Canada; Corresponding author.Université de Moncton, Campus of Shippagan, 218 Bd J. D. Gauthier, Shippagan, E8S 1P6, NB, CanadaUniversité de Moncton, Campus of Moncton, 18 Av Antonine-Maillet, Moncton, E1A 3E9, NB, CanadaUniversité de Moncton, Campus of Shippagan, 218 Bd J. D. Gauthier, Shippagan, E8S 1P6, NB, CanadaUniversité de Moncton, Campus of Shippagan, 218 Bd J. D. Gauthier, Shippagan, E8S 1P6, NB, CanadaThis study investigates the integration of large language models (LLMs) into a voice- and vision-based robotic control system in autonomous industrial applications. The main objective is to demonstrate that an LLM-based agent can interpret natural instructions, dynamically plan movements, and execute robotic actions without domain-specific supervised learning, thereby enabling autonomous robotic planning. The proposed system relies on a voice interface, an LLM agent, and tools for real-time robot control. A dedicated communication module was developed to ensure the full control of a KUKA industrial robot using WebSocket, without resorting to proprietary solutions. To validate the approach, a case study was conducted using a robotic cell, which was applied to snow-crab sorting, where computer vision provides real-time perception. The experimental evaluation covered a wide range of commands, including movement instructions, complex planning tasks (e.g., trajectory generation), and visual queries based on crab quality, size, and anatomy. The results showed that the model exhibited robust interpretation capabilities with an overall success rate of 98.46%. These performances highlight the potential of LLMs to facilitate human-robot interaction in real industrial environments, while reducing programming complexity and increasing system autonomy.http://www.sciencedirect.com/science/article/pii/S2590123025027276Large language models (LLMs)Voice interfaceKUKA industrial robotHuman-robot interactionAutonomous robotic planningSnow crab sorting |
| spellingShingle | Ibrahim Kadri Sid Ahmed Selouani Mohsen Ghribi Rayen Ghali Sabrina Mekhoukh LLM-driven agent for speech-enabled control of industrial robots: A case study in snow-crab quality inspection Results in Engineering Large language models (LLMs) Voice interface KUKA industrial robot Human-robot interaction Autonomous robotic planning Snow crab sorting |
| title | LLM-driven agent for speech-enabled control of industrial robots: A case study in snow-crab quality inspection |
| title_full | LLM-driven agent for speech-enabled control of industrial robots: A case study in snow-crab quality inspection |
| title_fullStr | LLM-driven agent for speech-enabled control of industrial robots: A case study in snow-crab quality inspection |
| title_full_unstemmed | LLM-driven agent for speech-enabled control of industrial robots: A case study in snow-crab quality inspection |
| title_short | LLM-driven agent for speech-enabled control of industrial robots: A case study in snow-crab quality inspection |
| title_sort | llm driven agent for speech enabled control of industrial robots a case study in snow crab quality inspection |
| topic | Large language models (LLMs) Voice interface KUKA industrial robot Human-robot interaction Autonomous robotic planning Snow crab sorting |
| url | http://www.sciencedirect.com/science/article/pii/S2590123025027276 |
| work_keys_str_mv | AT ibrahimkadri llmdrivenagentforspeechenabledcontrolofindustrialrobotsacasestudyinsnowcrabqualityinspection AT sidahmedselouani llmdrivenagentforspeechenabledcontrolofindustrialrobotsacasestudyinsnowcrabqualityinspection AT mohsenghribi llmdrivenagentforspeechenabledcontrolofindustrialrobotsacasestudyinsnowcrabqualityinspection AT rayenghali llmdrivenagentforspeechenabledcontrolofindustrialrobotsacasestudyinsnowcrabqualityinspection AT sabrinamekhoukh llmdrivenagentforspeechenabledcontrolofindustrialrobotsacasestudyinsnowcrabqualityinspection |