Assessing Land Cover Changes Using the LUCAS Database and Sentinel Imagery: A Comparative Analysis of Accuracy Metrics
Classification of remote sensing images using machine learning models requires a large amount of training data. Collecting this data is both labor-intensive and time-consuming. In this study, the effectiveness of using pre-existing reference data on land cover gathered as part of the Land Use–Land C...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-12-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/15/1/240 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841549401382715392 |
|---|---|
| author | Beata Hejmanowska Piotr Kramarczyk |
| author_facet | Beata Hejmanowska Piotr Kramarczyk |
| author_sort | Beata Hejmanowska |
| collection | DOAJ |
| description | Classification of remote sensing images using machine learning models requires a large amount of training data. Collecting this data is both labor-intensive and time-consuming. In this study, the effectiveness of using pre-existing reference data on land cover gathered as part of the Land Use–Land Cover Area Frame Survey (LUCAS) database of the Copernicus program was analyzed. The classification was carried out in Google Earth Engine (GEE) using Sentinel-2 images that were specially prepared to account for the phenological development of plants. Classification was performed using SVM, RF, and CART algorithms in GEE, with an in-depth accuracy analysis conducted using a custom tool. Attention was given to the reliability of different accuracy metrics, with a particular focus on the widely used machine learning (ML) metric of “accuracy”, which should not be compared with the commonly used remote sensing metric of “overall accuracy”, due to the potential for significant artificial inflation of accuracy. The accuracy of LUCAS 2018 at Level-1 detail was estimated at 86%. Using the updated LUCAS dataset, the best classification result was achieved with the RF method, with an accuracy of 83%. An accuracy overestimation of approximately 10% was observed when reporting the average accuracy ACC metric used in ML instead of the overall accuracy OA metric. |
| format | Article |
| id | doaj-art-3bd7f5c2b36742b5bb0fca44533d5d25 |
| institution | Kabale University |
| issn | 2076-3417 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj-art-3bd7f5c2b36742b5bb0fca44533d5d252025-01-10T13:14:54ZengMDPI AGApplied Sciences2076-34172024-12-0115124010.3390/app15010240Assessing Land Cover Changes Using the LUCAS Database and Sentinel Imagery: A Comparative Analysis of Accuracy MetricsBeata Hejmanowska0Piotr Kramarczyk1Department of Photogrammetry, Remote Sensing, and Spatial Engineering, Faculty of Geo-Data Science, Geodesy and Environmental Engeneerinf, AGH University, al. A. Mickiewicza 30, 30-059 Krakow, PolandDepartment of Photogrammetry, Remote Sensing, and Spatial Engineering, Faculty of Geo-Data Science, Geodesy and Environmental Engeneerinf, AGH University, al. A. Mickiewicza 30, 30-059 Krakow, PolandClassification of remote sensing images using machine learning models requires a large amount of training data. Collecting this data is both labor-intensive and time-consuming. In this study, the effectiveness of using pre-existing reference data on land cover gathered as part of the Land Use–Land Cover Area Frame Survey (LUCAS) database of the Copernicus program was analyzed. The classification was carried out in Google Earth Engine (GEE) using Sentinel-2 images that were specially prepared to account for the phenological development of plants. Classification was performed using SVM, RF, and CART algorithms in GEE, with an in-depth accuracy analysis conducted using a custom tool. Attention was given to the reliability of different accuracy metrics, with a particular focus on the widely used machine learning (ML) metric of “accuracy”, which should not be compared with the commonly used remote sensing metric of “overall accuracy”, due to the potential for significant artificial inflation of accuracy. The accuracy of LUCAS 2018 at Level-1 detail was estimated at 86%. Using the updated LUCAS dataset, the best classification result was achieved with the RF method, with an accuracy of 83%. An accuracy overestimation of approximately 10% was observed when reporting the average accuracy ACC metric used in ML instead of the overall accuracy OA metric.https://www.mdpi.com/2076-3417/15/1/240LUCASGoogle Earth EngineSentinel-2 |
| spellingShingle | Beata Hejmanowska Piotr Kramarczyk Assessing Land Cover Changes Using the LUCAS Database and Sentinel Imagery: A Comparative Analysis of Accuracy Metrics Applied Sciences LUCAS Google Earth Engine Sentinel-2 |
| title | Assessing Land Cover Changes Using the LUCAS Database and Sentinel Imagery: A Comparative Analysis of Accuracy Metrics |
| title_full | Assessing Land Cover Changes Using the LUCAS Database and Sentinel Imagery: A Comparative Analysis of Accuracy Metrics |
| title_fullStr | Assessing Land Cover Changes Using the LUCAS Database and Sentinel Imagery: A Comparative Analysis of Accuracy Metrics |
| title_full_unstemmed | Assessing Land Cover Changes Using the LUCAS Database and Sentinel Imagery: A Comparative Analysis of Accuracy Metrics |
| title_short | Assessing Land Cover Changes Using the LUCAS Database and Sentinel Imagery: A Comparative Analysis of Accuracy Metrics |
| title_sort | assessing land cover changes using the lucas database and sentinel imagery a comparative analysis of accuracy metrics |
| topic | LUCAS Google Earth Engine Sentinel-2 |
| url | https://www.mdpi.com/2076-3417/15/1/240 |
| work_keys_str_mv | AT beatahejmanowska assessinglandcoverchangesusingthelucasdatabaseandsentinelimageryacomparativeanalysisofaccuracymetrics AT piotrkramarczyk assessinglandcoverchangesusingthelucasdatabaseandsentinelimageryacomparativeanalysisofaccuracymetrics |