Multiplicative Algorithm for Correntropy-Based Nonnegative Matrix Factorization

• Autorzy: Hosseini-Asl E. , Zurada J. M.
• Języki publikacji: EN

Abstrakty

EN

Nonnegative matrix factorization (NMF) is a popular dimension reduction technique used for clustering by extracting latent features from highdimensional data and is widely used for text mining. Several optimization algorithms have been developed for NMF with different cost functions. In this paper we evaluate the correntropy similarity cost function. Correntropy is a nonlinear localized similarity measure which measures the similarity between two random variables using entropy-based criterion, and is especially robust to outliers. Some algorithms based on gradient descent have been used for correntropy cost function, but their convergence is highly dependent on proper initialization and step size and other parameter selection. The proposed general multiplicative factorization algorithm uses the gradient descent algorithm with adaptive step size to maximize the correntropy similarity between the data matrix and its factorization. After devising the algorithm, its performance has been evaluated for document clustering. Results were compared with constrained gradient descent method using steepest descent and L-BFGS methods. The simulations show that the performance of steepest descent and LBFGS convergence are highly dependent on gradient descent step size which depends on σ parameter of correntropy cost function. However, the multiplicative algorithm is shown to be less sensitive to σ parameterand yields better clustering results than other algorithms. The results demonstrate that clustering performance measured by entropy and purity improve the clustering. The multiplicative correntropy-based algorithm also shows less variation in accuracy of document clusters for variable number of clusters. The convergence of each algorithm is also investigated, and the experiments have shown that the multiplicative algorithm converges faster than L-BFGS and steepest descent method.

Słowa kluczowe

EN

Nonnegative Matrix Factorization (NMF); Correntropy; Multiplicative Algorithm; Document Clustering

• Wydawca: Społeczna Akademia Nauk w Łodzi
• Czasopismo: Journal of Applied Computer Science Methods
• Rocznik: 2013
• Tom: Vol. 5 No. 2
• Strony: 89--104
• Opis fizyczny: Bibliogr. 26 poz., rys., tab.

Twórcy

autor

Hosseini-Asl E.

• Department of Electrical and Computer Engineering University of Louisville, Louisville, KY, USA

autor

Zurada J. M.

• IT Institute, University of Social Sciences 9 Sienkiewicza St., 90-113 Łódź, Poland

Bibliografia

• 1. Lee D.D.,Seung H.S., 1999,Learning the parts of objects by non-negative matrix factorization, Nature, 401, 6755, pp. 788-791.
• 2. Seung D.,Lee L., 2001,Algorithms for non-negative matrix factorization, Advances in neural information processing systems, 13, pp. 556-562.
• 3. Hoyer P.O., 2002, Non-negative sparse coding, Proc. of 12th IEEE Workshop on Neural Networks for Signal Processing, pp. 557-565.
• 4. Xu W., Liu X., Gong Y., 2003, Document clustering based on non-negative matrix factorization, Proc. of the 26th Annual Int. ACM SIGIR Conf. on Research and development in informaion retrieval, pp. 267-273.
• 5. Hoyer P.O., 2004,Non-negative matrix factorization with sparseness constraints, The Journal of Machine Learning Research, 5, pp. 1457-1469.
• 6. Pauca V.P., Shahnaz F., Berry M.W., PlemmonsR.J., 2004,Text mining using nonnegative matrix factorizations, Proc. SIAM Int. Conf. on Data Mining, Orlando FL, pp. 22-24.
• 7. Sra S., DhillonI.S., 2005, Generalized nonnegative matrix approximations with Bregman divergences, Advances in neural information processing systems, pp. 283-290.
• 8. Tan P.N., Steinbach M., Kumar V., 2006,Introduction to Data Mining, Pearson Addison Wesley.
• 9. Shahnaz F., Berry M.W., Pauca V.P., PlemmonsR.J., 2006, Document clustering using nonnegative matrix factorization, Information Processing & Management, 42, 2, pp. 373-386.
• 10. Liu W., PokharelP.P., Principe J.C., 2006,Correntropy: A localized similarity measure, Int. Joint Conf. on Neural Networks, pp. 4919-4924.
• 11. Zdunek R., Cichocki A., 2006,Non-negative matrix factorization with quasi-Newton optimization, Int. Conf. on Artificial Intelligence and Soft Computing, Springer Berlin Heidelberg, 4029, pp. 870-879.
• 12. Berry M.W., Browne M., LangvilleA.N., Pauca V.P., PlemmonsR.J., 2007, Algorithms and applications for approximate nonnegative matrix factorization, Computational Statistics & Data Analysis, 52, 1, pp. 155-173.
• 13. KompassR., 2007,A generalized divergence measure for nonnegative matrix factorization, Neural computation, 19, 3, pp. 780-791.
• 14. Lin C.J., 2007, Projected gradient methods for nonnegative matrix factorization, Neural computation, 19, 10, pp. 2756-2779.
• 15. Liu W., PokharelP.P., Príncipe J.C., 2007, Correntropy: properties and applications in non-Gaussian signal processing, IEEE Trans. on Signal Processing, 55, 11, pp. 5286-5298.
• 16. Ding C., Li T., Peng W., Park H., 2006, Orthogonal nonnegative matrix tfactorizations for clustering, Proc. of the 12th ACM SIGKDD Int. Conf. on Knowledge discovery and data mining, ACM, pp. 126-135.
• 17. Kim H.,Park H., 2008,Nonnegative matrix factorization based on alternating nonnegativity constrained least squares and active set method, SIAM Journal on Matrix Analysis and Applications, 30, 2, pp. 713-730.
• 18. Matlab Software by Mark Schmidt, www.di.ens.fr/~mschmidt/Software/minConf.html
• 19. Cichocki A.,Anh-HuyP., 2009, Fast local algorithms for large scale nonnegative matrix and tensor factorizations, IEICE Trans. on fundamentals of electronics, communications and computer sciences,92, 3, pp. 708-721.
• 20. Kim J.,Park H., 2011, Fast nonnegative matrix factorization: An active-set-like method and comparisons, SIAM Journal on Scientific Computing, 33, 6, pp. 3261-3281.
• 21. FévotteC., BertinN., DurrieuJ.L., 2011,Nonnegative matrix factorization with the itakura-saito divergence: With application to music analysis, Neural computation, 21, 3, pp. 793-830.
• 22. FévotteC., IdierJ., 2011, Algorithms for nonnegative matrix factorization with the β-divergence, Neural Computation, 23, 9, pp. 2421-2456.
• 23. He R., Zheng W.S., Hu B.G., 2011, Maximum correntropy criterion for robust face recognition, IEEE Trans. on Pattern Analysis and Machine Intelligence, 33, 8, pp. 1561-1576.
• 24. Ensari T., Chorowski J., Zurada J.M., 2012, Correntropy-Based document clustering via nonnegative matrix factorization, Artificial Neural Networks and Machine LearningICANN 2012, Springer Berlin Heidelberg, pp. 347-354.
• 25. Ensari T., ChorowskiJ., Zurada J.M., 2012,Occluded Face Recognition Using Correntropy-Based Nonnegative Matrix Factorization, 11th International Conference on Machine Learning and Applications (ICMLA), 1, pp. 606-609.
• 26. Du L., Li X., Shen Y.D., 2012, Robust Nonnegative Matrix Factorization via HalfQuadratic Minimization, IEEE 12th International Conference on Data Mining (ICDM), pp. 201-210.