An Energy-Efficient Dynamic Feedback Image Signal Processor for Three-Dimensional Time-of-Flight Sensors
With the recent prominence of artificial intelligence (AI) technology, various research outcomes and applications in the field of image recognition and processing utilizing AI have been continuously emerging. In particular, the domain of object recognition using 3D time-of-flight (ToF) sensors has b...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-10-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/24/21/6918 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1846173082887127040 |
|---|---|
| author | Yongsoo Kim Jaehyeon So Chanwook Hwang Wencan Cheng Jong Hwan Ko |
| author_facet | Yongsoo Kim Jaehyeon So Chanwook Hwang Wencan Cheng Jong Hwan Ko |
| author_sort | Yongsoo Kim |
| collection | DOAJ |
| description | With the recent prominence of artificial intelligence (AI) technology, various research outcomes and applications in the field of image recognition and processing utilizing AI have been continuously emerging. In particular, the domain of object recognition using 3D time-of-flight (ToF) sensors has been actively researched, often in conjunction with augmented reality (AR) and virtual reality (VR). However, for more precise analysis, high-quality images are required, necessitating significantly larger parameters and computations. These requirements can pose challenges, especially in developing AR and VR technologies for low-power portable devices. Therefore, we propose a dynamic feedback configuration image signal processor (ISP) for 3D ToF sensors. The ISP achieves both accuracy and energy efficiency through dynamic feedback. The proposed ISP employs dynamic area extraction to perform computations and post-processing only for pixels within the valid area used by the application in each frame. Additionally, it uses dynamic resolution to determine and apply the appropriate resolution for each frame. This approach enhances energy efficiency by avoiding the processing of all sensor data while maintaining or surpassing accuracy levels. Furthermore, These functionalities are designed for hardware-efficient implementation, improving processing speed and minimizing power consumption. The results show a maximum performance of 178 fps and a high energy efficiency of up to 123.15 fps/W. When connected to the hand pose estimation (HPE) accelerator, it demonstrates an average mean squared error (MSE) of 10.03 mm, surpassing the baseline ISP value of 20.25 mm. Therefore, the proposed ISP can be effectively utilized in low-power, small form-factor devices. |
| format | Article |
| id | doaj-art-233bfc7cd72a4640bad8d422dd7501dd |
| institution | Kabale University |
| issn | 1424-8220 |
| language | English |
| publishDate | 2024-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj-art-233bfc7cd72a4640bad8d422dd7501dd2024-11-08T14:41:27ZengMDPI AGSensors1424-82202024-10-012421691810.3390/s24216918An Energy-Efficient Dynamic Feedback Image Signal Processor for Three-Dimensional Time-of-Flight SensorsYongsoo Kim0Jaehyeon So1Chanwook Hwang2Wencan Cheng3Jong Hwan Ko4Department of Semiconductor and Display Engineering, Sungkyunkwan University, Suwon 16419, Republic of KoreaDepartment of Electrical and Computer Engineering, Sungkyunkwan University, Suwon 16419, Republic of KoreaDepartment of Electrical and Computer Engineering, Sungkyunkwan University, Suwon 16419, Republic of KoreaDepartment of Computer Science, National University of Singapore, Singapore 117417, SingaporeDepartment of Electrical and Computer Engineering, Sungkyunkwan University, Suwon 16419, Republic of KoreaWith the recent prominence of artificial intelligence (AI) technology, various research outcomes and applications in the field of image recognition and processing utilizing AI have been continuously emerging. In particular, the domain of object recognition using 3D time-of-flight (ToF) sensors has been actively researched, often in conjunction with augmented reality (AR) and virtual reality (VR). However, for more precise analysis, high-quality images are required, necessitating significantly larger parameters and computations. These requirements can pose challenges, especially in developing AR and VR technologies for low-power portable devices. Therefore, we propose a dynamic feedback configuration image signal processor (ISP) for 3D ToF sensors. The ISP achieves both accuracy and energy efficiency through dynamic feedback. The proposed ISP employs dynamic area extraction to perform computations and post-processing only for pixels within the valid area used by the application in each frame. Additionally, it uses dynamic resolution to determine and apply the appropriate resolution for each frame. This approach enhances energy efficiency by avoiding the processing of all sensor data while maintaining or surpassing accuracy levels. Furthermore, These functionalities are designed for hardware-efficient implementation, improving processing speed and minimizing power consumption. The results show a maximum performance of 178 fps and a high energy efficiency of up to 123.15 fps/W. When connected to the hand pose estimation (HPE) accelerator, it demonstrates an average mean squared error (MSE) of 10.03 mm, surpassing the baseline ISP value of 20.25 mm. Therefore, the proposed ISP can be effectively utilized in low-power, small form-factor devices.https://www.mdpi.com/1424-8220/24/21/6918image signal processorToF sensordynamic feedbackenergy efficiency3D depth imagepoint cloud |
| spellingShingle | Yongsoo Kim Jaehyeon So Chanwook Hwang Wencan Cheng Jong Hwan Ko An Energy-Efficient Dynamic Feedback Image Signal Processor for Three-Dimensional Time-of-Flight Sensors Sensors image signal processor ToF sensor dynamic feedback energy efficiency 3D depth image point cloud |
| title | An Energy-Efficient Dynamic Feedback Image Signal Processor for Three-Dimensional Time-of-Flight Sensors |
| title_full | An Energy-Efficient Dynamic Feedback Image Signal Processor for Three-Dimensional Time-of-Flight Sensors |
| title_fullStr | An Energy-Efficient Dynamic Feedback Image Signal Processor for Three-Dimensional Time-of-Flight Sensors |
| title_full_unstemmed | An Energy-Efficient Dynamic Feedback Image Signal Processor for Three-Dimensional Time-of-Flight Sensors |
| title_short | An Energy-Efficient Dynamic Feedback Image Signal Processor for Three-Dimensional Time-of-Flight Sensors |
| title_sort | energy efficient dynamic feedback image signal processor for three dimensional time of flight sensors |
| topic | image signal processor ToF sensor dynamic feedback energy efficiency 3D depth image point cloud |
| url | https://www.mdpi.com/1424-8220/24/21/6918 |
| work_keys_str_mv | AT yongsookim anenergyefficientdynamicfeedbackimagesignalprocessorforthreedimensionaltimeofflightsensors AT jaehyeonso anenergyefficientdynamicfeedbackimagesignalprocessorforthreedimensionaltimeofflightsensors AT chanwookhwang anenergyefficientdynamicfeedbackimagesignalprocessorforthreedimensionaltimeofflightsensors AT wencancheng anenergyefficientdynamicfeedbackimagesignalprocessorforthreedimensionaltimeofflightsensors AT jonghwanko anenergyefficientdynamicfeedbackimagesignalprocessorforthreedimensionaltimeofflightsensors AT yongsookim energyefficientdynamicfeedbackimagesignalprocessorforthreedimensionaltimeofflightsensors AT jaehyeonso energyefficientdynamicfeedbackimagesignalprocessorforthreedimensionaltimeofflightsensors AT chanwookhwang energyefficientdynamicfeedbackimagesignalprocessorforthreedimensionaltimeofflightsensors AT wencancheng energyefficientdynamicfeedbackimagesignalprocessorforthreedimensionaltimeofflightsensors AT jonghwanko energyefficientdynamicfeedbackimagesignalprocessorforthreedimensionaltimeofflightsensors |