Deep Learning for Hyperspectral Image Classification: A Critical Evaluation via Mutation Testing
Recently, there has been a surge in the adoption of deep learning (DL) techniques, especially convolutional neural networks (CNNs), to perform hyperspectral image (HSI) classification. Although deep learners have been shown to achieve impressive performance in HSI classification, they are known to b...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-12-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/16/24/4695 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1846102895048523776 |
|---|---|
| author | Zhifei Chen Yang Hao Qichao Liu Yuyong Liu Mingyang Zhu Liang Xiao |
| author_facet | Zhifei Chen Yang Hao Qichao Liu Yuyong Liu Mingyang Zhu Liang Xiao |
| author_sort | Zhifei Chen |
| collection | DOAJ |
| description | Recently, there has been a surge in the adoption of deep learning (DL) techniques, especially convolutional neural networks (CNNs), to perform hyperspectral image (HSI) classification. Although deep learners have been shown to achieve impressive performance in HSI classification, they are known to be extremely sensitive to even slight perturbations to their inputs and models. When applied in safety-critical applications, it is crucial to know how robust they really are against perturbations. However, there is still limited tool support for DL testing in terms of their robustness, nor are the existing RGB testing approaches able to address the HSI-specific challenges. In this paper, we propose a mutation analysis framework specialized for DL models trained to classify HSIs, which facilitates a critical evaluation of the robustness of DL-based HSI classifiers. First, we introduce a set of mutation operators to inject faults into the inputs and models to simulate distortions of remote sensing HSI classifiers. By utilizing the mutation testing technique, we implement a novel framework which supports the multidimensional evaluation of individual DL-based classifiers. Finally, a comparative study of the robustness of seven popular CNN-based HSI classifiers (i.e., 3D-CNN, FDSSC, HybridSN, MCNN, FC3DCNN, DWTDENSE, and Tri-CNN) on six HSI datasets is provided. Results show that FDSSC and Tri-CNN achieve higher robustness in the presence of distortions, and FDSSC maintains a relatively stable level of robustness even with few training samples. These empirical findings can be partly explained by the characteristics of the classifiers’ architectures. The results substantiate the efficacy of our evaluation framework in assessing the robustness of HSI classifiers and thus confirm its contribution to the field of remote sensing image classification. |
| format | Article |
| id | doaj-art-eb1f1c1a09ba4d98bb2abca85b6a0400 |
| institution | Kabale University |
| issn | 2072-4292 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj-art-eb1f1c1a09ba4d98bb2abca85b6a04002024-12-27T14:50:55ZengMDPI AGRemote Sensing2072-42922024-12-011624469510.3390/rs16244695Deep Learning for Hyperspectral Image Classification: A Critical Evaluation via Mutation TestingZhifei Chen0Yang Hao1Qichao Liu2Yuyong Liu3Mingyang Zhu4Liang Xiao5School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, ChinaSchool of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, ChinaSchool of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing 210003, ChinaSchool of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, ChinaSchool of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, ChinaSchool of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, ChinaRecently, there has been a surge in the adoption of deep learning (DL) techniques, especially convolutional neural networks (CNNs), to perform hyperspectral image (HSI) classification. Although deep learners have been shown to achieve impressive performance in HSI classification, they are known to be extremely sensitive to even slight perturbations to their inputs and models. When applied in safety-critical applications, it is crucial to know how robust they really are against perturbations. However, there is still limited tool support for DL testing in terms of their robustness, nor are the existing RGB testing approaches able to address the HSI-specific challenges. In this paper, we propose a mutation analysis framework specialized for DL models trained to classify HSIs, which facilitates a critical evaluation of the robustness of DL-based HSI classifiers. First, we introduce a set of mutation operators to inject faults into the inputs and models to simulate distortions of remote sensing HSI classifiers. By utilizing the mutation testing technique, we implement a novel framework which supports the multidimensional evaluation of individual DL-based classifiers. Finally, a comparative study of the robustness of seven popular CNN-based HSI classifiers (i.e., 3D-CNN, FDSSC, HybridSN, MCNN, FC3DCNN, DWTDENSE, and Tri-CNN) on six HSI datasets is provided. Results show that FDSSC and Tri-CNN achieve higher robustness in the presence of distortions, and FDSSC maintains a relatively stable level of robustness even with few training samples. These empirical findings can be partly explained by the characteristics of the classifiers’ architectures. The results substantiate the efficacy of our evaluation framework in assessing the robustness of HSI classifiers and thus confirm its contribution to the field of remote sensing image classification.https://www.mdpi.com/2072-4292/16/24/4695deep learninghyperspectral image classificationmutation testingmodel robustness |
| spellingShingle | Zhifei Chen Yang Hao Qichao Liu Yuyong Liu Mingyang Zhu Liang Xiao Deep Learning for Hyperspectral Image Classification: A Critical Evaluation via Mutation Testing Remote Sensing deep learning hyperspectral image classification mutation testing model robustness |
| title | Deep Learning for Hyperspectral Image Classification: A Critical Evaluation via Mutation Testing |
| title_full | Deep Learning for Hyperspectral Image Classification: A Critical Evaluation via Mutation Testing |
| title_fullStr | Deep Learning for Hyperspectral Image Classification: A Critical Evaluation via Mutation Testing |
| title_full_unstemmed | Deep Learning for Hyperspectral Image Classification: A Critical Evaluation via Mutation Testing |
| title_short | Deep Learning for Hyperspectral Image Classification: A Critical Evaluation via Mutation Testing |
| title_sort | deep learning for hyperspectral image classification a critical evaluation via mutation testing |
| topic | deep learning hyperspectral image classification mutation testing model robustness |
| url | https://www.mdpi.com/2072-4292/16/24/4695 |
| work_keys_str_mv | AT zhifeichen deeplearningforhyperspectralimageclassificationacriticalevaluationviamutationtesting AT yanghao deeplearningforhyperspectralimageclassificationacriticalevaluationviamutationtesting AT qichaoliu deeplearningforhyperspectralimageclassificationacriticalevaluationviamutationtesting AT yuyongliu deeplearningforhyperspectralimageclassificationacriticalevaluationviamutationtesting AT mingyangzhu deeplearningforhyperspectralimageclassificationacriticalevaluationviamutationtesting AT liangxiao deeplearningforhyperspectralimageclassificationacriticalevaluationviamutationtesting |