EventSegNet: Direct Sparse Semantic Segmentation from Event Data
Semantic segmentation tasks encompass various applications, such as autonomous driving, medical imaging, and robotics. Achieving accurate semantic information retrieval under conditions of high dynamic range and rapid scene changes remains a significant challenge for image-based algorithms. This cha...
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-12-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/17/1/84 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841549001050030080 |
|---|---|
| author | Pengju Li Yuqiang Fang Jiayu Qiu Jun He Jishun Li Qinyu Zhu Xia Wang Yasheng Zhang |
| author_facet | Pengju Li Yuqiang Fang Jiayu Qiu Jun He Jishun Li Qinyu Zhu Xia Wang Yasheng Zhang |
| author_sort | Pengju Li |
| collection | DOAJ |
| description | Semantic segmentation tasks encompass various applications, such as autonomous driving, medical imaging, and robotics. Achieving accurate semantic information retrieval under conditions of high dynamic range and rapid scene changes remains a significant challenge for image-based algorithms. This challenge is primarily attributable to the limitations of conventional image sensors, which can experience motion blur or exposure artifacts. In contrast, event-based vision sensors, which asynchronously report changes in pixel intensity, offer a compelling solution by acquiring visual information at the same rate as the scene dynamics, thereby mitigating these limitations. However, we encounter a significant challenge in event-based semantic segmentation tasks: the need to expend time on converting event data into frame images to align with existing image-based semantic segmentation techniques. This approach squanders the inherently high temporal resolution of event data, compromising the accuracy and real-time performance of semantic segmentation tasks. To address these issues, this work explores a sparse semantic segmentation approach that directly addresses event data. We propose a network named EventSegNet that improves the ability to extract geometric features from event data by combining geometric feature enhancement operations and attention mechanisms. Based on this, we propose a large-scale event-based semantic segmentation dataset that provides labels for each event. Our approach achieved a new F1 score of 84.2% on the dataset. In addition, a lightweight and edge-oriented AI inference deployment technique was implemented for the network model. Compared to the baseline model, the optimized network model reduces the F1 score by 1.1% but is more than twice as fast computationally, enabling real-time inference on the NVIDIA AGX Xavier. |
| format | Article |
| id | doaj-art-7e3d4bf48c10426a858498c639fedeb4 |
| institution | Kabale University |
| issn | 2072-4292 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj-art-7e3d4bf48c10426a858498c639fedeb42025-01-10T13:20:10ZengMDPI AGRemote Sensing2072-42922024-12-011718410.3390/rs17010084EventSegNet: Direct Sparse Semantic Segmentation from Event DataPengju Li0Yuqiang Fang1Jiayu Qiu2Jun He3Jishun Li4Qinyu Zhu5Xia Wang6Yasheng Zhang7Graduate School, Space Engineering University, Beijing 101416, ChinaGraduate School, Space Engineering University, Beijing 101416, ChinaGraduate School, Space Engineering University, Beijing 101416, ChinaGraduate School, Space Engineering University, Beijing 101416, ChinaGraduate School, Space Engineering University, Beijing 101416, ChinaGraduate School, Space Engineering University, Beijing 101416, ChinaInstitute of Tracking and Communication Technology, Beijing 100094, ChinaGraduate School, Space Engineering University, Beijing 101416, ChinaSemantic segmentation tasks encompass various applications, such as autonomous driving, medical imaging, and robotics. Achieving accurate semantic information retrieval under conditions of high dynamic range and rapid scene changes remains a significant challenge for image-based algorithms. This challenge is primarily attributable to the limitations of conventional image sensors, which can experience motion blur or exposure artifacts. In contrast, event-based vision sensors, which asynchronously report changes in pixel intensity, offer a compelling solution by acquiring visual information at the same rate as the scene dynamics, thereby mitigating these limitations. However, we encounter a significant challenge in event-based semantic segmentation tasks: the need to expend time on converting event data into frame images to align with existing image-based semantic segmentation techniques. This approach squanders the inherently high temporal resolution of event data, compromising the accuracy and real-time performance of semantic segmentation tasks. To address these issues, this work explores a sparse semantic segmentation approach that directly addresses event data. We propose a network named EventSegNet that improves the ability to extract geometric features from event data by combining geometric feature enhancement operations and attention mechanisms. Based on this, we propose a large-scale event-based semantic segmentation dataset that provides labels for each event. Our approach achieved a new F1 score of 84.2% on the dataset. In addition, a lightweight and edge-oriented AI inference deployment technique was implemented for the network model. Compared to the baseline model, the optimized network model reduces the F1 score by 1.1% but is more than twice as fast computationally, enabling real-time inference on the NVIDIA AGX Xavier.https://www.mdpi.com/2072-4292/17/1/84event-based vision sensorsevent-based semantic segmentationPointNet++EventSegNet |
| spellingShingle | Pengju Li Yuqiang Fang Jiayu Qiu Jun He Jishun Li Qinyu Zhu Xia Wang Yasheng Zhang EventSegNet: Direct Sparse Semantic Segmentation from Event Data Remote Sensing event-based vision sensors event-based semantic segmentation PointNet++ EventSegNet |
| title | EventSegNet: Direct Sparse Semantic Segmentation from Event Data |
| title_full | EventSegNet: Direct Sparse Semantic Segmentation from Event Data |
| title_fullStr | EventSegNet: Direct Sparse Semantic Segmentation from Event Data |
| title_full_unstemmed | EventSegNet: Direct Sparse Semantic Segmentation from Event Data |
| title_short | EventSegNet: Direct Sparse Semantic Segmentation from Event Data |
| title_sort | eventsegnet direct sparse semantic segmentation from event data |
| topic | event-based vision sensors event-based semantic segmentation PointNet++ EventSegNet |
| url | https://www.mdpi.com/2072-4292/17/1/84 |
| work_keys_str_mv | AT pengjuli eventsegnetdirectsparsesemanticsegmentationfromeventdata AT yuqiangfang eventsegnetdirectsparsesemanticsegmentationfromeventdata AT jiayuqiu eventsegnetdirectsparsesemanticsegmentationfromeventdata AT junhe eventsegnetdirectsparsesemanticsegmentationfromeventdata AT jishunli eventsegnetdirectsparsesemanticsegmentationfromeventdata AT qinyuzhu eventsegnetdirectsparsesemanticsegmentationfromeventdata AT xiawang eventsegnetdirectsparsesemanticsegmentationfromeventdata AT yashengzhang eventsegnetdirectsparsesemanticsegmentationfromeventdata |