Tuning a SAM-Based Model With Multicognitive Visual Adapter to Remote Sensing Instance Segmentation
The segment anything model (SAM), a foundational model designed for promptable segmentation tasks, demonstrates exceptional generalization capabilities, making it highly promising for natural scene image segmentation. However, SAM's lack of pretraining on massive remote sensing images and...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10776034/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841557084680749056 |
|---|---|
| author | Linghao Zheng Xinyang Pu Su Zhang Feng Xu |
| author_facet | Linghao Zheng Xinyang Pu Su Zhang Feng Xu |
| author_sort | Linghao Zheng |
| collection | DOAJ |
| description | The segment anything model (SAM), a foundational model designed for promptable segmentation tasks, demonstrates exceptional generalization capabilities, making it highly promising for natural scene image segmentation. However, SAM's lack of pretraining on massive remote sensing images and its interactive structure limit its automatic mask prediction capabilities. In this article, a multicognitive SAM-based instance segmentation model (MC-SAM SEG) is introduced to employ SAM on remote sensing domain. The SAM-multicognitive visual adapter (Mona) encoder utilizing the Mona is conducted to facilitate SAM's transfer learning in remote sensing applications. The proposed method named MC-SAM SEG extracts high-quality features by fine-tuning the SAM-Mona encoder along with a feature aggregator. Subsequently, a pixel decoder and transformer decoder are designed for prompt-free mask generation and instance classification. The comprehensive experiments are conducted on the HRSID and WHU datasets for instance segmentation tasks on synthetic aperture radar images and optical remote sensing images, respectively. The evaluation results indicate the proposed method surpasses other deep learning algorithms and verify its effectiveness and generalization. |
| format | Article |
| id | doaj-art-5c24941a9ebf4d02aa1ca37e0b28f369 |
| institution | Kabale University |
| issn | 1939-1404 2151-1535 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| spelling | doaj-art-5c24941a9ebf4d02aa1ca37e0b28f3692025-01-07T00:00:27ZengIEEEIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing1939-14042151-15352025-01-01182737274810.1109/JSTARS.2024.350440910776034Tuning a SAM-Based Model With Multicognitive Visual Adapter to Remote Sensing Instance SegmentationLinghao Zheng0Xinyang Pu1https://orcid.org/0009-0002-0627-4603Su Zhang2Feng Xu3https://orcid.org/0000-0002-7015-1467Key Laboratory for Information Science of Electromagnetic Waves (Ministry of Education), School of Information Science and Technology, Fudan University, Shanghai, ChinaKey Laboratory for Information Science of Electromagnetic Waves (Ministry of Education), School of Information Science and Technology, Fudan University, Shanghai, ChinaKey Laboratory for Information Science of Electromagnetic Waves (Ministry of Education), School of Information Science and Technology, Fudan University, Shanghai, ChinaKey Laboratory for Information Science of Electromagnetic Waves (Ministry of Education), School of Information Science and Technology, Fudan University, Shanghai, ChinaThe segment anything model (SAM), a foundational model designed for promptable segmentation tasks, demonstrates exceptional generalization capabilities, making it highly promising for natural scene image segmentation. However, SAM's lack of pretraining on massive remote sensing images and its interactive structure limit its automatic mask prediction capabilities. In this article, a multicognitive SAM-based instance segmentation model (MC-SAM SEG) is introduced to employ SAM on remote sensing domain. The SAM-multicognitive visual adapter (Mona) encoder utilizing the Mona is conducted to facilitate SAM's transfer learning in remote sensing applications. The proposed method named MC-SAM SEG extracts high-quality features by fine-tuning the SAM-Mona encoder along with a feature aggregator. Subsequently, a pixel decoder and transformer decoder are designed for prompt-free mask generation and instance classification. The comprehensive experiments are conducted on the HRSID and WHU datasets for instance segmentation tasks on synthetic aperture radar images and optical remote sensing images, respectively. The evaluation results indicate the proposed method surpasses other deep learning algorithms and verify its effectiveness and generalization.https://ieeexplore.ieee.org/document/10776034/Instance segmentationremote sensing imagessegment anything model (SAM)transfer learning |
| spellingShingle | Linghao Zheng Xinyang Pu Su Zhang Feng Xu Tuning a SAM-Based Model With Multicognitive Visual Adapter to Remote Sensing Instance Segmentation IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing Instance segmentation remote sensing images segment anything model (SAM) transfer learning |
| title | Tuning a SAM-Based Model With Multicognitive Visual Adapter to Remote Sensing Instance Segmentation |
| title_full | Tuning a SAM-Based Model With Multicognitive Visual Adapter to Remote Sensing Instance Segmentation |
| title_fullStr | Tuning a SAM-Based Model With Multicognitive Visual Adapter to Remote Sensing Instance Segmentation |
| title_full_unstemmed | Tuning a SAM-Based Model With Multicognitive Visual Adapter to Remote Sensing Instance Segmentation |
| title_short | Tuning a SAM-Based Model With Multicognitive Visual Adapter to Remote Sensing Instance Segmentation |
| title_sort | tuning a sam based model with multicognitive visual adapter to remote sensing instance segmentation |
| topic | Instance segmentation remote sensing images segment anything model (SAM) transfer learning |
| url | https://ieeexplore.ieee.org/document/10776034/ |
| work_keys_str_mv | AT linghaozheng tuningasambasedmodelwithmulticognitivevisualadaptertoremotesensinginstancesegmentation AT xinyangpu tuningasambasedmodelwithmulticognitivevisualadaptertoremotesensinginstancesegmentation AT suzhang tuningasambasedmodelwithmulticognitivevisualadaptertoremotesensinginstancesegmentation AT fengxu tuningasambasedmodelwithmulticognitivevisualadaptertoremotesensinginstancesegmentation |