Exploring the performance impact of soft constraint integration on reinforcement learning-based autonomous vessel navigation: Experimental insights
Reinforcement learning has shown promise in enabling autonomous ship navigation, allowing vessels to adapt and make informed decisions in complex marine environments. However, the integration of soft constraints and their impact on performance in RL-based autonomous vessel navigation research remain...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2024-01-01
|
| Series: | International Journal of Naval Architecture and Ocean Engineering |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2092678224000281 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1846109779349471232 |
|---|---|
| author | Xin Jiang Jiawen Li Zhenkai Huang Ji Huang Ronghui Li |
| author_facet | Xin Jiang Jiawen Li Zhenkai Huang Ji Huang Ronghui Li |
| author_sort | Xin Jiang |
| collection | DOAJ |
| description | Reinforcement learning has shown promise in enabling autonomous ship navigation, allowing vessels to adapt and make informed decisions in complex marine environments. However, the integration of soft constraints and their impact on performance in RL-based autonomous vessel navigation research remain understudied. This research addresses this gap by investigating the implications of soft constraints in the context of the risk-averse ship navigation problem. Four distinct soft constraint functions are proposed, which are integrated with two widely used RL algorithms, resulting in the creation of eight risk-averse autonomous vessel navigation models. To ensure a comprehensive evaluation of their performance, comparative analyses are conducted across seven virtual digital channel environments. Additionally, a novel metric, known as Large Helm Momentum (LHM), is introduced to quantify the smoothness of autonomous vessel navigation. Through thorough experimentation, key considerations for the design of soft constraint functions in the domain of autonomous ship navigation are identified. A comprehensive understanding of how different soft constraint functions influence autonomous driving behavior has been achieved. Key considerations for designing soft constraint functions in the domain of autonomous ship navigation have also been identified. Five principles, namely the constraint association principle, dominance of hard constraints, reward-balance principle, mapping requirement principle, and iterative improvement principle, are proposed to optimize the design of soft constraint functions for autonomous ship navigation, providing valuable guidance and insights. |
| format | Article |
| id | doaj-art-4705607a2abd40ff8c919d9933561fb6 |
| institution | Kabale University |
| issn | 2092-6782 |
| language | English |
| publishDate | 2024-01-01 |
| publisher | Elsevier |
| record_format | Article |
| series | International Journal of Naval Architecture and Ocean Engineering |
| spelling | doaj-art-4705607a2abd40ff8c919d9933561fb62024-12-25T04:21:10ZengElsevierInternational Journal of Naval Architecture and Ocean Engineering2092-67822024-01-0116100609Exploring the performance impact of soft constraint integration on reinforcement learning-based autonomous vessel navigation: Experimental insightsXin Jiang0Jiawen Li1Zhenkai Huang2Ji Huang3Ronghui Li4Naval Architecture and Shipping College, Guangdong Ocean University, Zhanjiang, ChinaNaval Architecture and Shipping College, Guangdong Ocean University, Zhanjiang, China; Key Laboratory of International Shipping Development and Property Digitization of Hainan Free Trade Port, Hainan Vocational University of Science and Technology, Haikou, China; Technical Research Center for Ship Intelligence and Safety Engineering of Guangdong Province, Zhanjiang, China; Guangdong Provincial Key Laboratory of Intelligent Equipment for South China Sea Marine Ranching, Zhanjiang, ChinaNaval Architecture and Shipping College, Guangdong Ocean University, Zhanjiang, China; Technical Research Center for Ship Intelligence and Safety Engineering of Guangdong Province, Zhanjiang, China; Guangdong Provincial Key Laboratory of Intelligent Equipment for South China Sea Marine Ranching, Zhanjiang, ChinaNaval Architecture and Shipping College, Guangdong Ocean University, Zhanjiang, China; Technical Research Center for Ship Intelligence and Safety Engineering of Guangdong Province, Zhanjiang, China; Guangdong Provincial Key Laboratory of Intelligent Equipment for South China Sea Marine Ranching, Zhanjiang, ChinaNaval Architecture and Shipping College, Guangdong Ocean University, Zhanjiang, China; Technical Research Center for Ship Intelligence and Safety Engineering of Guangdong Province, Zhanjiang, China; Guangdong Provincial Key Laboratory of Intelligent Equipment for South China Sea Marine Ranching, Zhanjiang, China; Corresponding author.Reinforcement learning has shown promise in enabling autonomous ship navigation, allowing vessels to adapt and make informed decisions in complex marine environments. However, the integration of soft constraints and their impact on performance in RL-based autonomous vessel navigation research remain understudied. This research addresses this gap by investigating the implications of soft constraints in the context of the risk-averse ship navigation problem. Four distinct soft constraint functions are proposed, which are integrated with two widely used RL algorithms, resulting in the creation of eight risk-averse autonomous vessel navigation models. To ensure a comprehensive evaluation of their performance, comparative analyses are conducted across seven virtual digital channel environments. Additionally, a novel metric, known as Large Helm Momentum (LHM), is introduced to quantify the smoothness of autonomous vessel navigation. Through thorough experimentation, key considerations for the design of soft constraint functions in the domain of autonomous ship navigation are identified. A comprehensive understanding of how different soft constraint functions influence autonomous driving behavior has been achieved. Key considerations for designing soft constraint functions in the domain of autonomous ship navigation have also been identified. Five principles, namely the constraint association principle, dominance of hard constraints, reward-balance principle, mapping requirement principle, and iterative improvement principle, are proposed to optimize the design of soft constraint functions for autonomous ship navigation, providing valuable guidance and insights.http://www.sciencedirect.com/science/article/pii/S2092678224000281Deep reinforcement learningSoft constraintAutonomous shipReward functionArtificial intelligence |
| spellingShingle | Xin Jiang Jiawen Li Zhenkai Huang Ji Huang Ronghui Li Exploring the performance impact of soft constraint integration on reinforcement learning-based autonomous vessel navigation: Experimental insights International Journal of Naval Architecture and Ocean Engineering Deep reinforcement learning Soft constraint Autonomous ship Reward function Artificial intelligence |
| title | Exploring the performance impact of soft constraint integration on reinforcement learning-based autonomous vessel navigation: Experimental insights |
| title_full | Exploring the performance impact of soft constraint integration on reinforcement learning-based autonomous vessel navigation: Experimental insights |
| title_fullStr | Exploring the performance impact of soft constraint integration on reinforcement learning-based autonomous vessel navigation: Experimental insights |
| title_full_unstemmed | Exploring the performance impact of soft constraint integration on reinforcement learning-based autonomous vessel navigation: Experimental insights |
| title_short | Exploring the performance impact of soft constraint integration on reinforcement learning-based autonomous vessel navigation: Experimental insights |
| title_sort | exploring the performance impact of soft constraint integration on reinforcement learning based autonomous vessel navigation experimental insights |
| topic | Deep reinforcement learning Soft constraint Autonomous ship Reward function Artificial intelligence |
| url | http://www.sciencedirect.com/science/article/pii/S2092678224000281 |
| work_keys_str_mv | AT xinjiang exploringtheperformanceimpactofsoftconstraintintegrationonreinforcementlearningbasedautonomousvesselnavigationexperimentalinsights AT jiawenli exploringtheperformanceimpactofsoftconstraintintegrationonreinforcementlearningbasedautonomousvesselnavigationexperimentalinsights AT zhenkaihuang exploringtheperformanceimpactofsoftconstraintintegrationonreinforcementlearningbasedautonomousvesselnavigationexperimentalinsights AT jihuang exploringtheperformanceimpactofsoftconstraintintegrationonreinforcementlearningbasedautonomousvesselnavigationexperimentalinsights AT ronghuili exploringtheperformanceimpactofsoftconstraintintegrationonreinforcementlearningbasedautonomousvesselnavigationexperimentalinsights |