Thorough statistical comparison of machine learning regression models and their ensembles for sub-pixel imperviousness and imperviousness change mapping

• Autorzy: Drzewiecki W.

Identyfikatory

DOI

10.1515/geocart-2017-0012

• Języki publikacji: EN

Abstrakty

EN

We evaluated the performance of nine machine learning regression algorithms and their ensembles for sub-pixel estimation of impervious areas coverages from Landsat imagery. The accuracy of imperviousness mapping in individual time points was assessed based on RMSE, MAE and R2. These measures were also used for the assessment of imperviousness change intensity estimations. The applicability for detection of relevant changes in impervious areas coverages at sub-pixel level was evaluated using overall accuracy, F-measure and ROC Area Under Curve. The results proved that Cubist algorithm may be advised for Landsat-based mapping of imperviousness for single dates. Stochastic gradient boosting of regression trees (GBM) may be also considered for this purpose. However, Random Forest algorithm is endorsed for both imperviousness change detection and mapping of its intensity. In all applications the heterogeneous model ensembles performed at least as well as the best individual models or better. They may be recommended for improving the quality of sub-pixel imperviousness and imperviousness change mapping. The study revealed also limitations of the investigated methodology for detection of subtle changes of imperviousness inside the pixel. None of the tested approaches was able to reliably classify changed and non-changed pixels if the relevant change threshold was set as one or three percent. Also for fi ve percent change threshold most of algorithms did not ensure that the accuracy of change map is higher than the accuracy of random classifi er. For the threshold of relevant change set as ten percent all approaches performed satisfactory.

Słowa kluczowe

EN

impervious area; sub-pixel classification; machine learning; model ensembles; landsat

PL

modele regresji; nieprzepuszczalność; subpiksel

• Wydawca: Komitet Geodezji PAN
• Czasopismo: Geodesy and Cartography
• Rocznik: 2017
• Tom: Vol. 66, no. 2
• Strony: 171--209
• Opis fizyczny: Bibliogr. 57 poz., tab.

Twórcy

autor

Drzewiecki W.

• AGH University Faculty of Mining Surveying and Environmental Engineering Department of Geoinformation, Photogrammetry and Remote Sensing of Environment al. Mickiewicza 30, 30–059 Kraków, Poland

Bibliografia

• [1] Aleksandrowicz, S., Wawrzaszek, A., Drzewiecki, W. and Krupiński, M. (2016). Change Detection Using Global and Local Multifractal Description. IEEE Geoscience and Remote Sensing Letters, 13, 8, 1183-1187. DOI: 10.1109/LGRS.2016.2574940.
• [2] Amancio, D.R., Comin, C.H., Casanova, D., Travieso, G., Bruno, O.M., Rodrigues, F.A. and da Fountoura Costa, L. (2014). A Systematic Comparison of Supervised Classifiers. PLoS ONE 9(4): e94137. DOI:10.1371/journal.pone.0094137.
• [3] Bernat, K. and Drzewiecki, W. (2014). Two-stage subpixel impervious surface coverage estimation: comparing classifi cation and regression trees and artificial neural networks. In: Proc. SPIE Vol. 9244, 92441l, Image and Signal Processing for Remote Sensing. DOI: 10.1117/12.2067308.
• [4] Bouckaert, R.R. (2003). Choosing Learning Algorithms Using Sign Tests with High Replicability. In: Gedeon T..D., Fung L.C.C. (eds.) AI 2003: Advances in Artificial Intelligence. AI 2003. Lecture Notes in Computer Science, vol 2903. Springer, Berlin, Heidelberg. DOI: 10.1007/978-3-540-24581-0_61.
• [5] Bouckaert, R.R. (2004). Estimating replicability of classifier learning experiments. In Proceedings of the 21st International Conference on Machine Learning Banff, Canada, 2004. DOI: 10.1145/1015330.1015338.
• [6] Bouckaert, R. R. and Frank, E. (2004). Evaluating the replicability of significance tests for comparing learning algorithms. In: D. Honghua, R. Srikant, and C. Zhang, (eds.), Advances in Knowledge Discovery and Data Mining, 8th Pacifi c-Asia Conference, PAKDD 2004, Sydney, Australia, May 26-28, 2004, Proceedings. Springer, 2004. DOI: 10.1007/b97861.
• [7] Breiman, L. (2001). Random Forests. Machine Learning, 45, 5-32. DOI: 10.1023/A:1010933404324.
• [8] Coelho, G.P. and Von Zuben, F.J. (2006). The infl uence of the pool of candidates on the performance of selection and combination techniques in ensembles. In: Proceedings of the International Joint Conference on Neural Networks, 10588-10595.
• [9] Cohen, J. (1960). A coeffi cient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46. DOI: 10.1177/001316446002000104.
• [10] Daniel, W.W. (1990). Applied nonparametric statistics. Duxbury Thomson Learning, Pacific Grove.
• [11] Dams, J., Dujardin, J., Reggers, R., Bashir, I., Canters, F. and Batelaan, O. (2013). Mapping impervious surface change from remote sensing for hydrological modeling. Journal of Hydrology, 485, 84-95. DOI: 10.1016/j.jhydrol.2012.09.045.
• [12] Demsar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1-30.
• [13] Diedenhofen, B. and Musch, J. (2015). cocor: A Comprehensive Solution for the Statistical Comparison of Correlations. PLoS ONE 10(4): e0121945. DOI: 10.1371/journal.pone.0121945.
• [14] Drzewiecki, W. (2016). Comparison of Selected Machine Learning Algorithms for Sub-Pixel Imperviousness Change Assessment. In: 2016 Baltic Geodetic Congress (Geomatics), 67-72. DOI: 10.1109/BGC.Geomatics.2016.21.
• [15] Drzewiecki, W. (2016). Improving sub-pixel imperviousness change prediction by ensembling heterogeneous non-linear regression models. Geodesy and Cartography, 65, 2, 193-218. DOI: 10.1515/geocart-2016-0016.
• [16] Dunn, O. J. (1961). Multiple comparisons among means. Journal of the American Statistical Association, 56, 293, 52-64.
• [17] Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27, 8, 861-874. DOI: 10.1016/j.patrec.2005.10.010.
• [18] Finner, H. (1993). On a monotonicity problem in step-down multiple test procedures. Journal of the American Statistical Association, 88, 920-923. DOI: 10.1080/01621459.1993.10476358.
• [19] Fisher, R.A. (1937). Statistical methods and scientifi c inference. Hafner publishing Co, New York.
• [20] Foody, G.M. (2009). Classifi cation accuracy comparison: Hypothesis tests and the use of confi dence intervals in evaluations of difference, equivalence and non-inferiority. Remote Sensing of Environment, 113, 8, 1658-1663. DOI:10.1016/j.rse.2009.03.014.
• [21] Friedman, J. (1991). Multivariate Adaptive Regression Splines. The Annals of Statistics, 19(1), 1-141.
• [22] Friedman, J. (2002). Stochastic Gradient Boosting. Computational Statistics and Data Analysis, 38(4), 367-378. DOI: 10.1016/S0167-9473(01)00065-2.
• [23] Friedman, M. (1940). A Comparison of Alternative Tests of Signifi cance for the Problem of m Rankings. The Annals of Mathemathical Statistics, 11, 1, 86-92.
• [24] Garcia, S., Fernandez, A., Luengo, J. and Herrera, F. (2010). Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: experimental analysis of power. Information Sciences, 180 (10), 2044-2064. DOI: 10.1016/j.ins.2009.12.010.
• [25] Hanley, J.A. and McNeil, B.J. (1982). The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve. Radiology, 143, 1, 29-36. DOI: 10.1148/radiology.143.1.7063747.
• [26] Heremans, S. and Van Orshoven, J. (2015). Machine learning methods for sub-pixel landcover classification in the spatially heterogeneous region of Flanders (Belgium): a multi-criteria comparison. International Journal of Remote Sensing, 36, 11, 2934-2962. DOI: 10.1080/01431161.2015.1054047.
• [27] Hussain, M., Chen, D., Cheng, A., Wei, H. and Stanley, D. (2013). Change detection from remotely sensed images: From pixel-based to object-based approaches. ISPRS Journal of Photogrammetry and Remote Sensing, 80, 91-106. DOI: 10.1016/j.isprsjprs.2013.03.006.
• [28] Iman, R.L., Davenport, J.M. (1980) Approximations of the critical region of the friedman statistic. Communications in Statistics, 9, 571-595. DOI: 10.1080/03610928008827904.
• [29] Japkowicz, N. and Shah, M. (2011). Evaluating Learning Algorithms. A Classifi cation Perspective. Cambridge university Press.
• [30] Joshi, M.V. (2002). On evaluating performance of classifi ers for rare classes. In Proceedings of The 2002 IEEE International Conference on Data Mining, pp. 641-644. DOI: 10.1109/ICDM.2002.1184018.
• [31] Kircher, J. (2001). Data Analysis Toolkit #5: Uncertainty Analysis and Error Propagation. University of California Berkeley Seismological Laboratory. Available online at: http://seismo.berkeley.edu/~kirchner/eps_120/Toolkits/Toolkit_05.pdf.
• [32] Klaric, M. (2014). Predicting Relevant Change in High Resolution Satellite Imagery. ISPRS International Journal of Geoinformation, 3, 1491-1511. DOI: 10.3390/ijgi3041491.
• [33] Landis, J.R. and Koch, G.G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics, 30, 1, 159-174. DOI: 10.2307/2529310.
• [34] Li, S., Harner, E.J. and Adjeroh, D.A., (2011): Random KNN feature selection - a fast and stable alternative to Random Forests. BMC Bioinformatics, 12:450. DOI: 10.1186/1471-2105-12-450.
• [35] Lu, D., Li, G., Kuang, W. and Moran, E. (2014). Methods to extract impervious surface areas from satellite images. International Journal of Digital Earth, 7, 2, 93-112. DOI: 10.1080/17538947.2013.866173.
• [36] Lu, D., Li, G., Kuang, W. and Moran, E. (2014). Current situation and needs of change detection techiques. International journal of Image and Data Fusion, 5, 1, 13-38. DOI: 10.1080/19479832.2013.868372.
• [37] Morgan, M.G. and Henrion, M. (1990). Uncertainty: a guide to dealing with uncertainty in quantitative risk and policy analysis. Cambridge University Press.
• [38] Nadeau, C. and Bengio, Y. (2003). Inference for the Generalization Error. Machine Learning, 52, 3, 239-281. DOI:10.1023/A:1024068626366.
• [39] Olofsson, P., Foody, G.M., Herold, M., Stehman, S.V., Woodcock, C.E. and Wulder, M.A. (2014). Good practices for estimating area and assessing accuracy of land change. Remote Sensing of Environment, 148, 42-57, DOI: 10.1016/j.rse.2014.02.015.
• [40] Powers, D.M.W. (2011). Evaluation: from Precision, Recall and F-measure to ROC, Informedness, Markedness and Correlation. Journal of Machine Learning Technologies, 2, 1, 37-63. DOI: 10.9735/2229-3981.
• [41] Quinlan, R. (1993). Combining instance-based and model-based learning. In: Proceedings of the Tenth International Conference on Machine Learning, pp. 236-243.
• [42] Ridd, M.K. (1995). Exploring a V-I-S (vegetation-impervious surface-soil) model for urban ecosystem analysis through remote sensing: Comparative anatomy for cities. Int. J. Remote Sens., 16, 2165-2185. DOI: 10.1080/01431169508954549.
• [43] Ripley, B. (1996). Pattern Recognition and Neural Networks. Cambridge University Press.
• [44] Saito, T. and Rehmsmeier, M. (2015). The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLoS ONE 10(3): e0118432. DOI:10.1371/journal.pone.0118432.
• [45] Santafe, G., Inza, I. and Lozano, J.A. (2015). Dealing with the evaluation of supervised classifi cation algorithms. Artifi cial Intelligence Review, 44, 467-508. DOI: 10.1007/s10462-015-9433-y.
• [46] Shahtahmassebi, A.R., Song, J., Zheng, Q., Blackburn, G.A., Wang, K., Huang, L.Y., Pan, Y., Moore, N., Shahtahmassebi, G., Haghighi, R.S. and Deng, J.S. (2016). Remote sensing of impervious surface growth: A framework for quantifying urban expansion and re-densification mechanisms. International Journal of Applied Earth Observation and Geoinformation, 46, 94-112. DOI: 10.1016/j.jag.2015.11.007.
• [47] Smola, A.J. and Schölkopf, B. (2004). A tutorial on support vector regression. Statistics and Computing, 14, 199-222. DOI: 10.1023/B:STCO.0000035301.49549.88.
• [48] Steiger, J.H. (1980). Tests for Comparing Elements of a Correlation Matrix. Psychological Bulletin, 87: 245-251. DOI: 10.1037/0033-2909.87.2.245.
• [49] Tewkesbury, A.P., Comber, A.J., Tate, N.J., Lamb, A. and Fisher, P.F. (2015). A critical synthesis of remotely sensed optical image change detection techniques. Remote Sensing of Environment, 160, 1-14. DOI: 10.1016/j.rse.2015.01.006.
• [50] Trawiński, B., Smętek, M., Telec, Z. and Lasota, T. (2012). Nonparametric Statistical Analysis for Multiple Comparison of Machine Learning Regression Algorithms. International Journal of Applied Mathematics and Computer Science, 22, 4, 867-881. DOI: 10.2478/v10006-012-0064-z.
• [51] Turner, II B.L. and Meyer, W.B. (1994). Global Land Use and Land Cover Change: An Overview. In: Meyer W.B. and Turner II B.L. (eds.), Changes in Land Use and Land Cover: A Global Perspective. Cambridge University Press, pp. 3-10.
• [52] Wężyk, P., Hawryło P., Szostak, M., Pierzchalski, M. and de Kok, R. (2016). Using Geobia and Data Fusion Approach for Land use and Land Cover Mapping. Quaestiones Geographicae, 35, 1, 93-104. DOI: 10.1515/quageo-2016-0009.
• [53] Wieland, M., Liu, W. and Yamazaki, F. (2016). Learning Change from Synthetic Aperture Radar Images: Performance Evaluation of a Support Vectore Machine to Detect Earthquake and Tsunami-Induced Changes. Remote Sensing, 8 (10), 792, DOI: 10.3390/rs8100792.
• [54] Wilcoxon, F. (1945). Individual comparison by ranking methods. Biometrics, 1(6), 80-83. DOI: 10.2307/3001968.
• [55] Yang, L., Xian, G., Klaver, J. M. and Deal, B. (2003). Urban land-cover change detection through sub-pixel imperviousness mapping using remotely sensed data. Photogrammetric Engineering and Remote Sensing, 69, 9, 1003-1010. DOI: 10.14358/PERS.69.9.1003.
• [56] Yang, Y. and Liu X. (1999). A re-examination of text categorization methods. In: Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pp. 42-49. DOI: 10.1145/312624.312647.
• [57] Zou, G. Y., (2007). Toward using confi dence intervals to compare correlations. Psychological Methods, 12, 4, 399-413. DOI: 10.1037/1082-989X.12.4.399.

Uwagi

PL

Opracowanie rekordu w ramach umowy 509/P-DUN/2018 ze środków MNiSW przeznaczonych na działalność upowszechniającą naukę (2018)