Robust Reinforcement Learning Strategies with Evolving Curriculum for Efficient Bus Operations in Smart Cities
Public transit systems are critical to the quality of urban life, and enhancing their efficiency is essential for building cost-effective and sustainable smart cities. Historically, researchers sought reinforcement learning (RL) applications to mitigate bus bunching issues with holding strategies. N...
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-11-01
|
| Series: | Smart Cities |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2624-6511/7/6/141 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1846102712166383616 |
|---|---|
| author | Yuhan Tang Ao Qu Xuan Jiang Baichuan Mo Shangqing Cao Joseph Rodriguez Haris N Koutsopoulos Cathy Wu Jinhua Zhao |
| author_facet | Yuhan Tang Ao Qu Xuan Jiang Baichuan Mo Shangqing Cao Joseph Rodriguez Haris N Koutsopoulos Cathy Wu Jinhua Zhao |
| author_sort | Yuhan Tang |
| collection | DOAJ |
| description | Public transit systems are critical to the quality of urban life, and enhancing their efficiency is essential for building cost-effective and sustainable smart cities. Historically, researchers sought reinforcement learning (RL) applications to mitigate bus bunching issues with holding strategies. Nonetheless, these attempts often led to oversimplifications and misalignment with the goal of reducing the total time passengers spent in the system, resulting in less robust or non-optimal solutions. In this study, we introduce a novel setting where each bus, supervised by an RL agent, can appropriately form aggregated policies from three strategies (holding, skipping station, and turning around to serve the opposite direction). It’s difficult to learn them all together, due to learning complexity, we employ domain knowledge and develop a gradually expanding action space curriculum, enabling agents to learn these strategies incrementally. We incorporate Long Short-Term Memory (LSTM) in our model considering the temporal interrelation among these actions. To address the inherent uncertainties of real-world traffic systems, we impose Domain Randomization (DR) on variables such as passenger demand and bus schedules. We conduct extensive numerical experiments with the integration of synthetic and real-world data to evaluate our model. Our methodology proves effective, enhancing bus schedule reliability and reducing total passenger waiting time by over 15%, thereby improving bus operation efficiency and smoothering operations of buses that align with sustainable goals. This work highlights the potential of robust RL combined with curriculum learning for optimizing public transport in smart cities, offering a scalable solution for real-world multi-agent systems. |
| format | Article |
| id | doaj-art-61a5560aa2004f56ac2a9b7e9b9f3068 |
| institution | Kabale University |
| issn | 2624-6511 |
| language | English |
| publishDate | 2024-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Smart Cities |
| spelling | doaj-art-61a5560aa2004f56ac2a9b7e9b9f30682024-12-27T14:53:48ZengMDPI AGSmart Cities2624-65112024-11-01763658367710.3390/smartcities7060141Robust Reinforcement Learning Strategies with Evolving Curriculum for Efficient Bus Operations in Smart CitiesYuhan Tang0Ao Qu1Xuan Jiang2Baichuan Mo3Shangqing Cao4Joseph Rodriguez5Haris N Koutsopoulos6Cathy Wu7Jinhua Zhao8Department of Civil and Environmental Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USADepartment of Civil and Environmental Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USADepartment of Civil and Environmental Engineering, University of California at Berkeley, Berkeley, CA 94720, USADepartment of Civil and Environmental Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USADepartment of Civil and Environmental Engineering, University of California at Berkeley, Berkeley, CA 94720, USADepartment of Civil and Environmental Engineering, Northeastern University, Boston, MA 02115, USADepartment of Civil and Environmental Engineering, Northeastern University, Boston, MA 02115, USADepartment of Civil and Environmental Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USADepartment of Urban Studies and Planning, Massachusetts Institute of Technology, Cambridge, MA 02139, USAPublic transit systems are critical to the quality of urban life, and enhancing their efficiency is essential for building cost-effective and sustainable smart cities. Historically, researchers sought reinforcement learning (RL) applications to mitigate bus bunching issues with holding strategies. Nonetheless, these attempts often led to oversimplifications and misalignment with the goal of reducing the total time passengers spent in the system, resulting in less robust or non-optimal solutions. In this study, we introduce a novel setting where each bus, supervised by an RL agent, can appropriately form aggregated policies from three strategies (holding, skipping station, and turning around to serve the opposite direction). It’s difficult to learn them all together, due to learning complexity, we employ domain knowledge and develop a gradually expanding action space curriculum, enabling agents to learn these strategies incrementally. We incorporate Long Short-Term Memory (LSTM) in our model considering the temporal interrelation among these actions. To address the inherent uncertainties of real-world traffic systems, we impose Domain Randomization (DR) on variables such as passenger demand and bus schedules. We conduct extensive numerical experiments with the integration of synthetic and real-world data to evaluate our model. Our methodology proves effective, enhancing bus schedule reliability and reducing total passenger waiting time by over 15%, thereby improving bus operation efficiency and smoothering operations of buses that align with sustainable goals. This work highlights the potential of robust RL combined with curriculum learning for optimizing public transport in smart cities, offering a scalable solution for real-world multi-agent systems.https://www.mdpi.com/2624-6511/7/6/141bus oerationdeep reinforcement learningintelligent transportation systemscurriculum learningsmart cities |
| spellingShingle | Yuhan Tang Ao Qu Xuan Jiang Baichuan Mo Shangqing Cao Joseph Rodriguez Haris N Koutsopoulos Cathy Wu Jinhua Zhao Robust Reinforcement Learning Strategies with Evolving Curriculum for Efficient Bus Operations in Smart Cities Smart Cities bus oeration deep reinforcement learning intelligent transportation systems curriculum learning smart cities |
| title | Robust Reinforcement Learning Strategies with Evolving Curriculum for Efficient Bus Operations in Smart Cities |
| title_full | Robust Reinforcement Learning Strategies with Evolving Curriculum for Efficient Bus Operations in Smart Cities |
| title_fullStr | Robust Reinforcement Learning Strategies with Evolving Curriculum for Efficient Bus Operations in Smart Cities |
| title_full_unstemmed | Robust Reinforcement Learning Strategies with Evolving Curriculum for Efficient Bus Operations in Smart Cities |
| title_short | Robust Reinforcement Learning Strategies with Evolving Curriculum for Efficient Bus Operations in Smart Cities |
| title_sort | robust reinforcement learning strategies with evolving curriculum for efficient bus operations in smart cities |
| topic | bus oeration deep reinforcement learning intelligent transportation systems curriculum learning smart cities |
| url | https://www.mdpi.com/2624-6511/7/6/141 |
| work_keys_str_mv | AT yuhantang robustreinforcementlearningstrategieswithevolvingcurriculumforefficientbusoperationsinsmartcities AT aoqu robustreinforcementlearningstrategieswithevolvingcurriculumforefficientbusoperationsinsmartcities AT xuanjiang robustreinforcementlearningstrategieswithevolvingcurriculumforefficientbusoperationsinsmartcities AT baichuanmo robustreinforcementlearningstrategieswithevolvingcurriculumforefficientbusoperationsinsmartcities AT shangqingcao robustreinforcementlearningstrategieswithevolvingcurriculumforefficientbusoperationsinsmartcities AT josephrodriguez robustreinforcementlearningstrategieswithevolvingcurriculumforefficientbusoperationsinsmartcities AT harisnkoutsopoulos robustreinforcementlearningstrategieswithevolvingcurriculumforefficientbusoperationsinsmartcities AT cathywu robustreinforcementlearningstrategieswithevolvingcurriculumforefficientbusoperationsinsmartcities AT jinhuazhao robustreinforcementlearningstrategieswithevolvingcurriculumforefficientbusoperationsinsmartcities |