SF-SAM-Adapter: SAM-based segmentation model integrates prior knowledge for gaze image reflection noise removal
Gaze tracking technology in HMDs (Head-Mounted Displays) suffers from decreased accuracy due to highlight reflection noise from users' glasses. To address this, we present a denoising method which directly pinpoints the noisy regions through advanced segmentation models and then fills the flawe...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2025-01-01
|
| Series: | Alexandria Engineering Journal |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S1110016824012572 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841524942992048128 |
|---|---|
| author | Ting Lei Jing Chen Jixiang Chen |
| author_facet | Ting Lei Jing Chen Jixiang Chen |
| author_sort | Ting Lei |
| collection | DOAJ |
| description | Gaze tracking technology in HMDs (Head-Mounted Displays) suffers from decreased accuracy due to highlight reflection noise from users' glasses. To address this, we present a denoising method which directly pinpoints the noisy regions through advanced segmentation models and then fills the flawed regions through advanced image inpainting algorithms. In segmentation stage, we introduce a novel model based on the recently proposed segmentation large model SAM (Segment Anything Model), called SF-SAM-Adapter (Spatial and Frequency aware SAM Adapter). It injects prior knowledge regarding the strip-like shaped in spatial and high-frequency in frequency of reflection noise into SAM by integrating specially designed trainable adapter modules into the original structure, while retaining the expressive power of the large model and better adapting to the downstream task. We achieved segmentation metrics of IoU (Intersection over Union) = 0.749 and Dice = 0.853 at a memory size of 13.9 MB, outperforming recent techniques, including UNet, UNet++, BATFormer, FANet, MSA, and SAM2-Adapter. In inpainting, we employ the advanced inpainting algorithm LAMA (Large Mask inpainting), resulting in significant improvements in gaze tracking accuracy by 0.502°, 0.182°, and 0.319° across three algorithms. The code and datasets used in current study are available in the repository: https://github.com/leiting5297/SF-SAM-Adapter.git. |
| format | Article |
| id | doaj-art-f57a436d5d754df5915cc8dd5b0f608b |
| institution | Kabale University |
| issn | 1110-0168 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Alexandria Engineering Journal |
| spelling | doaj-art-f57a436d5d754df5915cc8dd5b0f608b2025-01-18T05:03:42ZengElsevierAlexandria Engineering Journal1110-01682025-01-01111521529SF-SAM-Adapter: SAM-based segmentation model integrates prior knowledge for gaze image reflection noise removalTing Lei0Jing Chen1Jixiang Chen2School of optics and photonics, Beijing Institute of Technology, Beijing 100089, ChinaCorresponding author.; School of optics and photonics, Beijing Institute of Technology, Beijing 100089, ChinaSchool of optics and photonics, Beijing Institute of Technology, Beijing 100089, ChinaGaze tracking technology in HMDs (Head-Mounted Displays) suffers from decreased accuracy due to highlight reflection noise from users' glasses. To address this, we present a denoising method which directly pinpoints the noisy regions through advanced segmentation models and then fills the flawed regions through advanced image inpainting algorithms. In segmentation stage, we introduce a novel model based on the recently proposed segmentation large model SAM (Segment Anything Model), called SF-SAM-Adapter (Spatial and Frequency aware SAM Adapter). It injects prior knowledge regarding the strip-like shaped in spatial and high-frequency in frequency of reflection noise into SAM by integrating specially designed trainable adapter modules into the original structure, while retaining the expressive power of the large model and better adapting to the downstream task. We achieved segmentation metrics of IoU (Intersection over Union) = 0.749 and Dice = 0.853 at a memory size of 13.9 MB, outperforming recent techniques, including UNet, UNet++, BATFormer, FANet, MSA, and SAM2-Adapter. In inpainting, we employ the advanced inpainting algorithm LAMA (Large Mask inpainting), resulting in significant improvements in gaze tracking accuracy by 0.502°, 0.182°, and 0.319° across three algorithms. The code and datasets used in current study are available in the repository: https://github.com/leiting5297/SF-SAM-Adapter.git.http://www.sciencedirect.com/science/article/pii/S1110016824012572Eye trackingImage segmentationImage denoisingLarge model |
| spellingShingle | Ting Lei Jing Chen Jixiang Chen SF-SAM-Adapter: SAM-based segmentation model integrates prior knowledge for gaze image reflection noise removal Alexandria Engineering Journal Eye tracking Image segmentation Image denoising Large model |
| title | SF-SAM-Adapter: SAM-based segmentation model integrates prior knowledge for gaze image reflection noise removal |
| title_full | SF-SAM-Adapter: SAM-based segmentation model integrates prior knowledge for gaze image reflection noise removal |
| title_fullStr | SF-SAM-Adapter: SAM-based segmentation model integrates prior knowledge for gaze image reflection noise removal |
| title_full_unstemmed | SF-SAM-Adapter: SAM-based segmentation model integrates prior knowledge for gaze image reflection noise removal |
| title_short | SF-SAM-Adapter: SAM-based segmentation model integrates prior knowledge for gaze image reflection noise removal |
| title_sort | sf sam adapter sam based segmentation model integrates prior knowledge for gaze image reflection noise removal |
| topic | Eye tracking Image segmentation Image denoising Large model |
| url | http://www.sciencedirect.com/science/article/pii/S1110016824012572 |
| work_keys_str_mv | AT tinglei sfsamadaptersambasedsegmentationmodelintegratespriorknowledgeforgazeimagereflectionnoiseremoval AT jingchen sfsamadaptersambasedsegmentationmodelintegratespriorknowledgeforgazeimagereflectionnoiseremoval AT jixiangchen sfsamadaptersambasedsegmentationmodelintegratespriorknowledgeforgazeimagereflectionnoiseremoval |