Analysis of the Effectiveness of Selected Machine Learning Algorithms in the Classification of Satellite Image Content Depending on the Size of the Training Sample

• Autorzy: Kupidura Przemysław , Niemyski Stanisław

Warianty tytułu

PL

Analiza skuteczności wybranych algorytmów uczenia maszynowego w klasyfikacji treści obrazów satelitarnych w zależności od rozmiaru próbki treningowej

• Języki publikacji: EN

Abstrakty

EN

The article presents an analysis of the accuracy of 3 popular machine learning (ML) methods: Maximum Likelihood Classifier (MLC), Support Vector Machine (SVM), and Random Forest (RF) depending on the size of the training sample. The analysis involved performing the classification of the content of a Landsat 8 satellite image (divided into 6 basic land cover classes) in 10 different variants of the number of training samples (from 2664 to 34711 pixels), estimating individual results, and a comparative analysis of the obtained results. For each classification variant, an error matrix was developed and on their basis, accuracy metrics were calculated: f1-score, precision and recall (for individual classes) as well as overall accuracy and kappa index of agreement (generally for the entire classification). The analysis showed a stimulating effect of the size of the training sample on the accuracy of the obtained classification results in all analyzed cases, with the most sensitive to this factor being MLC, showing the best effectiveness with the largest training sample and the smallest - with the smallest, and the least SVM, characterized by the highest accuracy with the smallest training sample, comparing to other algorithms.

PL

Artykuł przedstawia analizę dokładności 3 popularnych metod uczenia maszynowego: Maximum Likelihood Classifier (MLC), Support Vector Machine (SVM) oraz Random Forest (RF) w zależności od liczebności próbki treningowej. Analiza polegała na wykonaniu klasyfikacji treści zdjęcia satelitarnego Landsat 8 (w podziale na 6 podstawowych klas pokrycia terenu) w 10 różnych wariantach liczebności próbek uczących (od 2664 do 34711 pikseli), oszacowaniu poszczególnych wyników oraz analizie porównawczej uzyskanych wyników. Dla każdego wariantu klasyfikacji opracowano macierz błędów, a na ich podstawie obliczono metryki dokładności: F1-score, precision and recall (dla pojedynczych klas) oraz ogólną dokładność i wskaźnik zgodności Kappa (ogólnie dla całej klasyfikacji). Analiza wykazała stymulujący wpływ rozmiaru próbki uczącej na dokładność uzyskiwanych wyników klasyfikacji we wszystkich analizowanych przypadkach, przy czym najbardziej wrażliwym na ten czynnik był MLC, wykazujący się najlepszą skutecznością przy największej próbce treningowej i najmniejszą - przy najmniejszej, a najmniej SVM, cechujący się największą dokładnością przy najmniejszej próbce treningowej, w porównaniu do pozostałych algorytmów.

Słowa kluczowe

EN

machine learning; classification; remote sensing; training sample size; SVM; random forest; maximum likelihood classifier; satellite imagery

PL

uczenie maszynowe; klasyfikacja; teledetekcja; rozmiar próbki treningowej; SVM; lasy losowe; klasyfikator największego prawdopodobieństwa; zobrazowania satelitarne

• Wydawca: Polskie Towarzystwo Geograficzne, Oddział Teledetekcji i Geoinformatyki
• Czasopismo: Teledetekcja Środowiska
• Rocznik: 2024
• Tom: T. 64
• Strony: 24--38
• Opis fizyczny: Bibliogr. 39 poz., rys., tab.

Twórcy

autor

Kupidura Przemysław

• Warsaw University of Technology, Faculty of Geodesy and Cartography

autor

Niemyski Stanisław

• Warsaw University of Technology, Faculty of Geodesy and Cartography

Bibliografia

• Belgiu, M., & Drăguţ, L. (2016). Random forest in remote sensing: A review of applications and future directions. ISPRS Journal of Photogrammetry and Remote Sensing, 114, 24-31. DOI: 10.1016/j.isprsjprs.2016.01.011.
• Bigdeli, A., Maghsoudi, A., & Ghezelbash, R. (2024). A comparative study of the XGBoost ensemble learning and multilayer perceptron in mineral prospectivity modeling: a case study of the Torud-Chahshirin belt, NE Iran. Earth Sci Inform, 17, 483-499. DOI: 10.1007/s12145- 023-01184-4.
• Boser, B. E., Guyon, I. M., & Vapnik, V. N. (1992). A training algorithm for optimal margin classifiers. In Proceedings of the fifth annual workshop on Computational learning theory - COLT '92 (p. 144). DOI: 10.1145/130385.130401.
• Breiman, L. (2001). Random Forests. Machine Learning, 45, 5-32. DOI: 10.1023/A:1010933404324.
• Budach, L., Feuerpfeil, M., Ihde, N., Nathansen, A., Noack, N., Patzlaff, H., Naumann, F., & Harmouch, H. (2022). The Effects of Data Quality on Machine Learning Performance. arXiv preprint arXiv:2207.14529.
• Burkholder, A., Warner, T. A., Culp, M., & Landenberger, R. E. (2011). Seasonal trends in separability of leaf reflectance spectra for Ailanthus altissima and four other tree species. Photogramm. Eng. Remote Sens., 77, 793-804.
• Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20, 273-297. DOI: 10.1007/BF00994018.
• Cracknell, M. J., & Reading, A. M. (2014). Geological mapping using remote sensing data: A comparison of five machine learning algorithms, their response to variations in the spatial distribution of training data and the use of explicit spatial information. Comput. Geosci., 63, 22-33.
• Ding, H. (2024). Establishing a soil carbon flux monitoring system based on support vector machine and XGBoost. Soft Comput, 28, 4551-4574. DOI: 10.1007/s00500-024-09641-y.
• Duro, D. C., Franklin, S. E., & Dubé, M. G. (2012). A comparison of pixel-based and object-based image analysis with selected machine learning algorithms for the classification of agricultural landscapes using SPOT-5 HRG imagery. Remote Sens. Environ., 118, 259-272.
• Figueroa, R. L., Zeng-Treitler, Q., Kandula, S., et al. (2012). Predicting sample size required for classification performance. BMC Medical Informatics and Decision Making, 12(8). DOI: 10.1186/1472-6947-12-8.
• Fu, Y., Shen, R., Song, C., Dong, J., Han, W., Ye, T., & Yuan, W. (2023). Exploring the effects of training samples on the accuracy of crop mapping with machine learning algorithm. Science of Remote Sensing, Volume 7, 100081. DOI: 10.1016/j.srs.2023.100081.
• Ghayour, L., Neshat, A., Paryani, S., Shahabi, H., Shirzadi, A., Chen, W., Al-Ansari, N., Geertsema, M., Amiri, M. P., Gholamnia, M., et al. (2021). Performance Evaluation of Sentinel-2 and Landsat 8 OLI Data for Land Cover/Use Classification Using a Comparison between Machine Learning Algorithms. Remote Sensing, 13(7), 1349. DOI: 10.3390/rs13071349.
• Halevy, A., Norvig, P., & Pereira, F. (2009). The Unreasonable Effectiveness of Data. IEEE Intelligent Systems, 24(2). DOI: 10.1109/MIS.2009.36.
• Hand, D. J., Christen, P., & Kirielle, N. (2021). F*: an interpretable transformation of the F-measure. Mach Learn, 110, 451-456. DOI: 10.1007/s10994-021-05964-1.
• Ho, T. K. (1995). Random Decision Forests. Proceedings of the 3rd International Conference on Document Analysis and Recognition, Montreal, QC, 14-16 August 1995. pp. 278-282.
• Kelley, P.R. (2021). Maximum Likelihood Estimation. In Foscher, M.M., Nijakamp, P. (eds) Handbook of Regional Science. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-60723-7_88.
• Koppaka, R., & Moh, T. -S. (2020). Machine Learning in Indian Crop Classification of Temporal Multi-Spectral Satellite Image. In 2020 14th International Conference on Ubiquitous Information Management and Communication (IMCOM) (pp. 1-8). Taichung, Taiwan. DOI: 10.1109/IMCOM48794.2020.9001718.
• Labatut, V., & Cherifi, H. (2012). Accuracy Measures for the Comparison of Classifiers. Proceedings of The 5th International Conference on Information Technology, Amman, Jordanie. 10.48550/arXiv.1207.3790.
• Li, X., Chen, W., Cheng, X., & Wang, L. (2016). A Comparison of Machine Learning Algorithms for Mapping of Complex Surface-Mined and Agricultural Landscapes Using ZiYuan-3 Stereo Satellite Imagery. Remote Sensing, 8(6), 514. DOI: 10.3390/rs8060514.
• Liu, J., Zuo, Y., Wang, N., Yuan, F., Zhu, X., Zhang, L., Zhang, J., Sun, Y., Guo, Z., Guo, Y., et al. (2021). Comparative Analysis of Two Machine Learning Algorithms in Predicting Site-Level Net Ecosystem Exchange in Major Biomes. Remote Sensing, 13(12), 2242. DOI: 10.3390/rs13122242.
• Maxwell, A. E., & Warner, T. A. (2015). Differentiating mine-reclaimed grasslands from spectrally similar land cover using terrain variables and object-based machine learning classification. Int. J. Remote Sens., 36, 4384-4410.
• Maxwell, A. E., Strager, M. P., Warner, T. A., Zegre, N. P., & Yuill, C. B. (2014). Comparison of NAIP orthophotography and RapidEye satellite imagery for mapping of mining and mine reclamation. GIScience Remote Sens., 51, 301-320.
• Maxwell, A. E., Warner, T. A., Strager, M. P., Conley, J. F., & Sharp, A. L. (2015). Assessing machine-learning algorithms and image- and Lidar-derived variables for GEOBIA classification of mining and mine reclamation. Int. J. Remote Sens., 36, 954-978.
• Maxwell, A. E., Warner, T. A., Strager, M. P., & Pal, M. (2014). Combining RapidEye satellite imagery and Lidar for mapping of mining and mine reclamation. Photogramm. Eng. Remote Sens.
• Maxwell, A. E., Warner, T. A., & Fang, F. (2018). Implementation of machine-learning classification in remote sensing: an applied review. International Journal of Remote Sensing, 39(9), 2784-2817. DOI: 10.1080/01431161.2018.1433343.
• Mousavinezhad, M., Feizi, A., & Aalipour, M. (2023). Performance Evaluation of Machine Learning Algorithms in Change Detection and Change Prediction of a Watershed’s Land Use and Land Cover. Int J Environ Res, 17, 29. DOI: 10.1007/s41742-023-00518-w.
• Nalepa, J., & Kawulok, M. (2019). Selecting training sets for support vector machines: a review. Artificial Intelligence Review, 52, 857-900. DOI: 10.1007/s10462-017-9611-1.
• Powers, D. M. W. (2007). Evaluation: From Precision, Recall and F-Factor to ROC, Informedness, Markedness & Correlation. Technical Report SIE-07-001, Flinders University, Adelaide, Australia.
• Ramezan, C.A., Warner, T.A., Maxwell, A.E., & Price, B.S. (2021). Effects of Training Set Size on Supervised Machine-Learning LandCover Classification of Large-Area High-Resolution Remotely Sensed Data. Remote Sensing, 13(3), 368. DOI: 10.3390/rs13030368.
• Raudys, S. J., & Jain, A. K. (1991). Small Sample Size Effects in Statistical Pattern Recognition: Recommendations for Practitioners. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(3).
• Schölkopf, B., & Smola, A. J. (2001). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press. ISBN 9780262256933.
• Seydi, S. T., Kanani-Sadat, Y., Hasanlou, M., Sahraei, R., Chanussot, J., & Amani, M. (2023). Comparison of Machine Learning Algorithms for Flood Susceptibility Mapping. Remote Sensing, 15(1), 192. DOI: 10.3390/rs15010192.
• Shang, M., Wang, S.X., Zhou, Y. et al. (2018). Effects of Training Samples and Classifiers on Classification of Landsat-8 Imagery. Journal of the Indian Society of Remote Sensing, 46, 1333-1340. DOI: 10.1007/s12524-018-0777-z.
• Sim, J., & Wright, C. C. (2005). The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements. Physical Therapy, 85(3), 257–268. DOI: 10.1093/ptj/85.3.257.
• Sobieraj, J., Fernández, M., & Metelski, D. (2022). A Comparison of Different Machine Learning Algorithms in the Classification of Impervious Surfaces: Case Study of the Housing Estate Fort Bema in Warsaw (Poland). Buildings, 12(12), 2115. DOI: 10.3390/buildings12122115.
• Volke, M. I., & Abarca-Del-Rio, R. (n.d.). Comparison of machine learning classification algorithms for land cover change in a coastal area affected by the 2010 Earthquake and Tsunami in Chile. Nat. Hazards Earth Syst. Sci. Discuss. [preprint].
• Zhao, Z., Islam, F., Waseem, L. A., Tariq, A., Nawaz, M., Islam, I. U., Bibi, T., Rehman, N. U., Ahmad, W., Aslam, R. W., Raza, D., & Hatamleh, W. A. (2024). Comparison of Three Machine Learning Algorithms Using Google Earth Engine for Land Use Land Cover Classification. Rangeland Ecology & Management, 92, 129-137. https://doi.org/10.1016/j.rama.2023.10.007.
• Zheng, W., & Jin, M. (2020). The Effects of Class Imbalance and Training Data Size on Classifier Learning: An Empirical Study. SN Computer Science, 1, 71. DOI: 10.1007/s42979-020-0074-0.