On combining principal components with Fisher's linear discriminants for supervised learning

• Autorzy: Pechenizkiy M. , Tsymbal A. , Puuronen S.
• Konferencja: ADBIS Workshop on Data Mining and Knowledge Discovery (ADMKD'2005) / sympozjum [1st; September 15-16, 2005; Tallinn, Estonia]
• Języki publikacji: EN

Abstrakty

EN

"The curse of dimensionality" is pertinent to many learning algorithms, and it denotes the drastic increase of computational complexity and classification error in high dimensions. In this paper, principal component analysis (PCA). parametric feature extraction (FE) based on Fisher's linear discriminant analysis (LDA), and their combination as means of dimensionality reduction are analysed with respect to the performance of different classifiers. Three commonly used classifiers are taken for analysis: ŁNN, Naive Bayes and C4.5 decision tree. Recently, it has been argued that it is extremely important to use class information in FE for supervised learning (SL). However, LDA-based FE, although using class information, has a serious shortcoming due to its parametric nature. Namely, the number of extracted components cannot be more that the number of classes minus one. Besides, as it can be concluded from its name, LDA works mostly for linearly separable classes only. In this paper we study if it is possible to overcome these shortcomings adding the most significant principal components to the set of features extracted with LDA. In experiments on 21 benchmark datasets from UCI repository these two approaches (PCA and LDA) are compared with each other, and with their combination, for each classifier. Our results demonstrate that such a combination approach has certain potential, especially when applied for C4.5 decision tree learning. However, from the practical point of view the combination approach cannot be recommended for Naive Bayes since its behavior is very unstable on different datasets.

Słowa kluczowe

EN

principal component analysis; linear discriminant analysis; feature extraction; supervised learning

• Wydawca: Wydawnictwo Politechniki Poznańskiej
• Czasopismo: Foundations of Computing and Decision Sciences
• Rocznik: 2006
• Tom: Vol. 31, No. 1
• Strony: 59--73
• Opis fizyczny: Bibliogr. 17 poz.

Twórcy

autor

Pechenizkiy M.

autor

Tsymbal A.

autor

Puuronen S.

• Department of Computer Science and Information Systems, University of Jyvaskyla, P.O. Box 35, 40351 Jyvaskyla, Finland, (phone: [phone]; fax: [phone],

Bibliografia

• [1] Aivazyan S.A., Applied statistics: classification and dimension reduction. Finance and Statistics, Moscow, 1989.
• [2] AladjemM., Multiclass discriminant mappings, Signal Processing, 35, 1994, 1-18.
• [3] Bellman R., Adaptive Control Processes: A Guided Tour, Princeton University Press, 1961. [4] Blake C.L., Merz C.J., UCI Repository of Machine Learning Databases, Dept. of Information and Computer Science, University of California, Irvine CA, 1998.
• [5] Fayyad U.M., Data Mining and Knowledge Discovery: Making Sense Out of Data, IEEE Expert, 11, 5, 1996, 20-25.
• [6] Fukunaga K., Introduction to statistical pattern recognition, Academic Press, London 1990.
• [7] Jolliffe I.T., Principal Component Analysis, Springer, New York, NY, 1986.
• [8] Liu H., Feature Extraction, Construction and Selection: A Data Mining Perspective, ISBN 0-7923-8196-3, Kluwer Academic Publishers, 1998.
• [9] Michalski R.S., Seeking Knowledge in the Deluge of Facts, Fundamenta Informaticae, 30, 1997,283-297.
• [10] Oza N.C., Turner K., Dimensionality Reduction Through Classifier Ensembles, Technical Report NASA-ARC-IC-1999-124, Computational Sciences Division, NASA Ames Research Center, Moffett Field, CA, 1999.
• [ll] Pechenizkiy M., Impact of the Feature Extraction on the Performance of a Classifier: kNN, Naive Bayes and C4.5, in: B.Kegl, G.Lapalme (eds.), Proc. of 18th CSCSI Conference on Artificial Intelligence Al'05, LNAI 3501, Springer Verlag, 2005, 268-279.
• [12] Pechenizkiy M., Tsymbal A., Puuronen S.. Local Dimensionality Reduction within Natural Clusters for Medical Data Analysis, in Proc. 18th IEEE Symp. on Computer-Based Medical Systems CBMS'2005, IEEE CS Press, 2005, 365-37.
• [13] Tsymbal A., Pechenizkiy M., Puuronen S., Patterson D.W., Dynamic integration of classifiers in the space of principal components, in: L.Kalinichenko, R.Manthey, B.Thalheim, U.Wloka (eds.), Proc. Advances in. Databases and Information Systems: 7th East-European Cotif. ADBIS'03, LNCS 2798, Heidelberg; Springer-Verlag, 2003, 278-292.
• [14] Tsymbal A., Puuronen S., Pechenizkiy M., Baumgarten M., Patterson D., Eigenvector-based feature extraction for classification, in: Proc. 15th Int. FLAIRS Conference on Artificial Intelligence, Pensacola, FL, USA, AAAI Press, 2002, 354-358.
• [15] Witten I. and Frank E., Data Mining: Practical machine learning toots with Java implementations, Morgan Kaufrnann, San Francisco, 2000,
• [16] William D.R., Goldstein M,, Multivariate Analysis. Methods and Applications, ISBN 0-471-08317-8, John Wiley & Sons, 1984.
• [17] Quinlan, J.R., 'C4.5 programs for machine learning, San Mateo CA: Morgan Kaufmann, 1993.