Cross Teaching-Enhanced Multispectral Remote Sensing Object Detection With Transformer
Remote sensing platforms are often equipped with sensors of multiple spectrums to capture the diverse reflective properties of ground areas, typically including the visible spectrum and the near infrared (NIR) spectrum. Moreover, thermal infrared (TIR) sensors capture the radiated heat of targets an...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10770223/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841536161501151232 |
|---|---|
| author | Jiahe Zhu Huan Zhang Simin Li Shengjin Wang Hongbing Ma |
| author_facet | Jiahe Zhu Huan Zhang Simin Li Shengjin Wang Hongbing Ma |
| author_sort | Jiahe Zhu |
| collection | DOAJ |
| description | Remote sensing platforms are often equipped with sensors of multiple spectrums to capture the diverse reflective properties of ground areas, typically including the visible spectrum and the near infrared (NIR) spectrum. Moreover, thermal infrared (TIR) sensors capture the radiated heat of targets and are capable of all-day observation regardless of illumination conditions. By leveraging the complementary features of different spectrums, multispectral fusion techniques enhance the precision and robustness of remote sensing object detection methods. In this article, we present an object detection method for remote sensing imagery named multispectral detection transformer (multispectral DETR). The model fuses multispectral features with deformable attention and utilizes fused features for object detection. The multispectral deformable attention fusion block integrates the flexibility of dynamic weights with the principle of fusion based on local regions. Then, we propose a simple yet effective oriented object detection scheme based on angle prediction. Finally, we introduce a novel cross-teaching method between single-spectral and multispectral models, which alleviates the spectral interference issue caused by inconsistent target visibility. Experimental results demonstrate that multispectral DETR achieves state-of-the-art results on both the RGB-NIR VEDAI and the RGB-TIR DroneVehicle datasets. |
| format | Article |
| id | doaj-art-26b540af7ff548d2a39d8d4411e652cf |
| institution | Kabale University |
| issn | 1939-1404 2151-1535 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| spelling | doaj-art-26b540af7ff548d2a39d8d4411e652cf2025-01-15T00:00:23ZengIEEEIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing1939-14042151-15352025-01-01182401241310.1109/JSTARS.2024.350454910770223Cross Teaching-Enhanced Multispectral Remote Sensing Object Detection With TransformerJiahe Zhu0https://orcid.org/0000-0002-4587-9237Huan Zhang1https://orcid.org/0000-0003-0747-4375Simin Li2https://orcid.org/0000-0001-6355-2978Shengjin Wang3https://orcid.org/0000-0001-7809-1932Hongbing Ma4https://orcid.org/0000-0002-1785-4024Department of Electrical Engineering, Tsinghua University, Beijing, ChinaDepartment of Electrical Engineering, Tsinghua University, Beijing, ChinaInstitute of Electronic Engineering, China Academy of Engineering Physics, Mianyang, ChinaDepartment of Electrical Engineering, Tsinghua University, Beijing, ChinaDepartment of Electrical Engineering, Tsinghua University, Beijing, ChinaRemote sensing platforms are often equipped with sensors of multiple spectrums to capture the diverse reflective properties of ground areas, typically including the visible spectrum and the near infrared (NIR) spectrum. Moreover, thermal infrared (TIR) sensors capture the radiated heat of targets and are capable of all-day observation regardless of illumination conditions. By leveraging the complementary features of different spectrums, multispectral fusion techniques enhance the precision and robustness of remote sensing object detection methods. In this article, we present an object detection method for remote sensing imagery named multispectral detection transformer (multispectral DETR). The model fuses multispectral features with deformable attention and utilizes fused features for object detection. The multispectral deformable attention fusion block integrates the flexibility of dynamic weights with the principle of fusion based on local regions. Then, we propose a simple yet effective oriented object detection scheme based on angle prediction. Finally, we introduce a novel cross-teaching method between single-spectral and multispectral models, which alleviates the spectral interference issue caused by inconsistent target visibility. Experimental results demonstrate that multispectral DETR achieves state-of-the-art results on both the RGB-NIR VEDAI and the RGB-TIR DroneVehicle datasets.https://ieeexplore.ieee.org/document/10770223/Detection transformerfeature fusionknowledge distillationmultispectral remote sensing imageobject detection |
| spellingShingle | Jiahe Zhu Huan Zhang Simin Li Shengjin Wang Hongbing Ma Cross Teaching-Enhanced Multispectral Remote Sensing Object Detection With Transformer IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing Detection transformer feature fusion knowledge distillation multispectral remote sensing image object detection |
| title | Cross Teaching-Enhanced Multispectral Remote Sensing Object Detection With Transformer |
| title_full | Cross Teaching-Enhanced Multispectral Remote Sensing Object Detection With Transformer |
| title_fullStr | Cross Teaching-Enhanced Multispectral Remote Sensing Object Detection With Transformer |
| title_full_unstemmed | Cross Teaching-Enhanced Multispectral Remote Sensing Object Detection With Transformer |
| title_short | Cross Teaching-Enhanced Multispectral Remote Sensing Object Detection With Transformer |
| title_sort | cross teaching enhanced multispectral remote sensing object detection with transformer |
| topic | Detection transformer feature fusion knowledge distillation multispectral remote sensing image object detection |
| url | https://ieeexplore.ieee.org/document/10770223/ |
| work_keys_str_mv | AT jiahezhu crossteachingenhancedmultispectralremotesensingobjectdetectionwithtransformer AT huanzhang crossteachingenhancedmultispectralremotesensingobjectdetectionwithtransformer AT siminli crossteachingenhancedmultispectralremotesensingobjectdetectionwithtransformer AT shengjinwang crossteachingenhancedmultispectralremotesensingobjectdetectionwithtransformer AT hongbingma crossteachingenhancedmultispectralremotesensingobjectdetectionwithtransformer |