A novel pansharpening method based on cross stage partial network and transformer
Abstract In remote sensing image fusion, the conventional Convolutional Neural Networks (CNNs) extract local features of the image through layered convolution, which is limited by the receptive field and struggles to capture global features. Transformer utilizes self-attention to capture long-distan...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2024-06-01
|
| Series: | Scientific Reports |
| Online Access: | https://doi.org/10.1038/s41598-024-63336-w |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841544628865597440 |
|---|---|
| author | Yingxia Chen Huiqi Liu Faming Fang |
| author_facet | Yingxia Chen Huiqi Liu Faming Fang |
| author_sort | Yingxia Chen |
| collection | DOAJ |
| description | Abstract In remote sensing image fusion, the conventional Convolutional Neural Networks (CNNs) extract local features of the image through layered convolution, which is limited by the receptive field and struggles to capture global features. Transformer utilizes self-attention to capture long-distance dependencies in images, which has a global receptive field, but the computational cost for high-resolution images is excessively high. In response to the above issues, this paper draws inspiration from the FusionNet network, harnessing the local detail acquisition capability of CNNs and the global data procuring capacity of Transformer. It presents a novel method for remote sensing image sharpening named Guided Filtering-Cross Stage Partial Network-Transformer, abbreviated as GF-CSTNet. This solution unifies the strengths of Guided Filtering (GF), Cross Stage Partial Network (CSPNet), and Transformer. Firstly, this method utilizes GF to enhance the acquired remote sensing image data. The CSPNet and Transformer structures are then combined to further enhance fusion performance by leveraging their respective advantages. Subsequently, a Rep-Conv2Former method is designed to streamline attention and extract diverse receptive field features through a multi-scale convolution modulator block. Simultaneously, a reparameterization module is constructed to integrate the multiple branches generated during training into a unified branch during inference, thereby optimizing the model’s inference speed. Finally, a residual learning module incorporating attention has been devised to augment the modeling and feature extraction capabilities of images. Experimental results obtained from the GaoFen-2 and WorldView-3 datasets demonstrate the effectiveness of the proposed GF-CSTNet approach. It effectively extracts detailed information from images while avoiding the problem of spectral distortion. |
| format | Article |
| id | doaj-art-e3ee3199d5f64b1b90ee7d9336c9f1b0 |
| institution | Kabale University |
| issn | 2045-2322 |
| language | English |
| publishDate | 2024-06-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj-art-e3ee3199d5f64b1b90ee7d9336c9f1b02025-01-12T12:24:51ZengNature PortfolioScientific Reports2045-23222024-06-0114111810.1038/s41598-024-63336-wA novel pansharpening method based on cross stage partial network and transformerYingxia Chen0Huiqi Liu1Faming Fang2School of Computer Science, Yangtze UniversitySchool of Computer Science, Yangtze UniversitySchool of Computer Science and Technology, East China Normal UniversityAbstract In remote sensing image fusion, the conventional Convolutional Neural Networks (CNNs) extract local features of the image through layered convolution, which is limited by the receptive field and struggles to capture global features. Transformer utilizes self-attention to capture long-distance dependencies in images, which has a global receptive field, but the computational cost for high-resolution images is excessively high. In response to the above issues, this paper draws inspiration from the FusionNet network, harnessing the local detail acquisition capability of CNNs and the global data procuring capacity of Transformer. It presents a novel method for remote sensing image sharpening named Guided Filtering-Cross Stage Partial Network-Transformer, abbreviated as GF-CSTNet. This solution unifies the strengths of Guided Filtering (GF), Cross Stage Partial Network (CSPNet), and Transformer. Firstly, this method utilizes GF to enhance the acquired remote sensing image data. The CSPNet and Transformer structures are then combined to further enhance fusion performance by leveraging their respective advantages. Subsequently, a Rep-Conv2Former method is designed to streamline attention and extract diverse receptive field features through a multi-scale convolution modulator block. Simultaneously, a reparameterization module is constructed to integrate the multiple branches generated during training into a unified branch during inference, thereby optimizing the model’s inference speed. Finally, a residual learning module incorporating attention has been devised to augment the modeling and feature extraction capabilities of images. Experimental results obtained from the GaoFen-2 and WorldView-3 datasets demonstrate the effectiveness of the proposed GF-CSTNet approach. It effectively extracts detailed information from images while avoiding the problem of spectral distortion.https://doi.org/10.1038/s41598-024-63336-w |
| spellingShingle | Yingxia Chen Huiqi Liu Faming Fang A novel pansharpening method based on cross stage partial network and transformer Scientific Reports |
| title | A novel pansharpening method based on cross stage partial network and transformer |
| title_full | A novel pansharpening method based on cross stage partial network and transformer |
| title_fullStr | A novel pansharpening method based on cross stage partial network and transformer |
| title_full_unstemmed | A novel pansharpening method based on cross stage partial network and transformer |
| title_short | A novel pansharpening method based on cross stage partial network and transformer |
| title_sort | novel pansharpening method based on cross stage partial network and transformer |
| url | https://doi.org/10.1038/s41598-024-63336-w |
| work_keys_str_mv | AT yingxiachen anovelpansharpeningmethodbasedoncrossstagepartialnetworkandtransformer AT huiqiliu anovelpansharpeningmethodbasedoncrossstagepartialnetworkandtransformer AT famingfang anovelpansharpeningmethodbasedoncrossstagepartialnetworkandtransformer AT yingxiachen novelpansharpeningmethodbasedoncrossstagepartialnetworkandtransformer AT huiqiliu novelpansharpeningmethodbasedoncrossstagepartialnetworkandtransformer AT famingfang novelpansharpeningmethodbasedoncrossstagepartialnetworkandtransformer |