Optimization of convolutional neural networks using the fuzzy gravitational search algorithm

• Autorzy: Poma Yutzil , Melin Patricia , González Claudia I. , Martínez Gabriela E.

Identyfikatory

DOI

10.14313/JAMRIS/1-2020/12

• Języki publikacji: EN

Abstrakty

EN

This paper presents an approach to optimize a Convolutional Neural Network using the Fuzzy Gravitational Search Algorithm. The optimized parameters are the number of images per block that are used in the training phase, the number of filters and the filter size of the convolutional layer. The reason for optimizing these parameters is because they have a great impact on performance of the Convolutional Neural Networks. The neural network model presented in this work can be applied for any image recognition or classification applications; nevertheless, in this paper, the experiments are performed in the ORL and Cropped Yale databases. The results are compared with other neural networks, such as modular and monolithic neural networks. In addition, the experiments were performed manually, and the results were obtained (when the neural network is not optimized), and comparison was made with the optimized results to validate the advantage of using the Fuzzy Gravitational Search Algorithm.

Słowa kluczowe

EN

neural networks; convolutional neural network; fuzzy gravitational search algorithm; deep learning

• Wydawca: Łukasiewicz Industrial Research Institute for Automation and Measurements PIAP
• Czasopismo: Journal of Automation Mobile Robotics and Intelligent Systems
• Rocznik: 2020
• Tom: Vol. 14, No. 1
• Strony: 109--120
• Opis fizyczny: Bibliogr. 44 poz., rys.

Twórcy

autor

Poma Yutzil

• – Tijuana Institute of Technology, B.C., Tijuana, México

autor

Melin Patricia

• – Tijuana Institute of Technology, B.C., Tijuana, México

autor

González Claudia I.

• – Tijuana Institute of Technology, B.C., Tijuana, México

autor

Martínez Gabriela E.

• – Tijuana Institute of Technology, B.C., Tijuana, México

Bibliografia

• [1] D. H. Hubel and T. N. Wiesel, “Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex”, The Journal of Physiology, vol. 160, no. 1, 1962, 106–154.
• [2] K. Fukushima, “Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position”, Biological Cybernetics, vol. 36, no. 4, 1980,193–202,DOI: 10.1007/BF00344251.
• [3] Y. LeCun, B. E. Boser, J. S. Denker, D. Henderson,R. E. Howard, W. E. Hubbard and L. D. Jackel,“Handwritten Digit Recognition with a BackPropagation Network”. In: D. S. Touretzky (eds.), Advances in Neural Information Processing Systems 2, 1990, 396–404.
• [4] D. Sánchez, P. Melin and O. Castillo, “Fuzzy Adaptation for Particle Swarm Optimization for Modular Neural Networks Applied to Iris Recognition”. In: P. Melin, O. Castillo, J. Kacprzyk, M. Reformat and W. Melek (eds.), Fuzzy Logic in Intelligent System Design, 2018, 104–114,DOI: 10.1007/978-3-319-67137-6_11.
• [5] F. Valdez, O. Castillo and P. Melin, “Ant colony optimization for the design of Modular Neural Networks in pattern recognition”. In: 2016 International Joint Conference on Neural Networks (IJCNN), 2016, 163–168,DOI: 10.1109/IJCNN.2016.7727194.
• [6] C. I. Gonzalez, J. R. Castro, O. Mendoza and P. Melin, “General type-2 fuzzy edge detector applied on face recognition system using neural networks”. In: 2016 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), 2016,2325–2330,DOI: 10.1109/FUZZ-IEEE.2016.7737983.
• [7] G. E. Martínez, P. Melin, O. D. Mendoza and O. Castillo, “Face Recognition with a Sobel Edge Detector and the Choquet Integral as Integration Method in a Modular Neural Networks”. In: P. Melin, O. Castillo and J. Kacprzyk (eds.), Design of Intelligent Systems Based on Fuzzy Logic, Neural Networks and Nature-Inspired Optimization, 2015, 59–70.
• [8] P. Melin, I. Miramontes and G. Prado-Arechiga,“A hybrid model based on modular neural networks and fuzzy systems for classification of blood pressure and hypertension risk diagnosis”, Expert Systems with Applications, vol. 107,2018, 146–164,DOI: 10.1016/j.eswa.2018.04.023.
• [9] F. Valdez, P. Melin and O. Castillo, “Modular Neural Networks architecture optimization with a new nature inspired method using a fuzzy combination of Particle Swarm Optimization and Genetic Algorithms”, Information Sciences,vol. 270, 2014, 143–153, DOI: 10.1016/j.ins.2014.02.091.
• [10] P. Melin, D. Sánchez and O. Castillo, “Genetic optimization of modular neural networks with fuzzy response integration for human recognition”, Information Sciences, vol. 197, 2012, 1–19, DOI: 10.1016/j.ins.2012.02.027.
• [11] P. Melin and D. Sánchez, “Multi-objective optimization for modular granular neural networks applied to pattern recognition”, Information Sciences, vol. 460-461, 2018, 594–610,DOI: 10.1016/j.ins.2017.09.031.
• [12] D. Sánchez, P. Melin and O. Castillo, “Optimization of modular granular neural networks using a firefly algorithm for human recognition”,Engineering Applications of Artificial Intelligence, vol. 64, 2017, 172–186,DOI: 10.1016/j.engappai.2017.06.007.
• [13] F. Gaxiola, P. Melin, F. Valdez, J. R. Castro and O. Castillo, “Optimization of type-2 fuzzy weights in backpropagation learning for neural networks using GAs and PSO”, Applied Soft Computing, vol. 38, 2016, 860–871,DOI: 10.1016/j.asoc.2015.10.027.
• [14] P. Melin, J. Amezcua, F. Valdez and O. Castillo,“A new neural network model based on the LVQ algorithm for multi-class classification of arrhythmias”, Information Sciences, vol. 279, 2014, 483–497,DOI: 10.1016/j.ins.2014.04.003.
• [15] Y. LeCun, L. Bottou, Y. Bengio and P. Haffner,“Gradient-Based Learning Applied to Document Recognition”, Proceedings of the IEEE,vol. 86, no. 11, pp. 2278–2324,DOI: 10.1109/5.726791.
• [16] A. Frome, G. Cheung, A. Abdulkader, M. Zennaro, B. Wu, A. Bissacco, H. Adam, H. Neven and L. Vincent, “Large-scale privacy protection in Google Street View”. In: 2009 IEEE 12th International Conference on Computer Vision, 2009,2373–2380,DOI: 10.1109/ICCV.2009.5459413.
• [17] R. Hadsell, P. Sermanet, J. Ben, A. Erkan,M. Scoffier, K. Kavukcuoglu, U. Muller and Y. LeCun, “Learning long-range vision for autonomous off-road driving”, Journal of Field Robotics, vol. 26, no. 2, 2009, 120–144,DOI: 10.1002/rob.20276.
• [18] A. Sombra, F. Valdez, P. Melin and O. Castillo,“A new gravitational search algorithm using fuzzy logic to parameter adaptation”. In: 2013 IEEE Congress on Evolutionary Computation,2013, 1068–1074,DOI: 10.1109/CEC.2013.6557685.
• [19] E. Rashedi, H. Nezamabadi-pour and S. Saryazdi,“GSA: AGravitational Search Algorithm”, Information Sciences, vol. 179, no. 13, 2009, 2232–2248,DOI: 10.1016/j.ins.2009.03.004.
• [20] O. P. Verma and R. Sharma, “Newtonian Gravitational Edge Detection Using Gravitational Search Algorithm”. In: 2012 International Conference on Communication Systems and Network Technologies, 2012, 184–188,DOI: 10.1109/CSNT.2012.48.
• [21] A. Hatamlou, S. Abdullah and Z. Othman,“Gravitational search algorithm with heuristic search for clustering problems”. In: 2011 3rd Conference on Data Mining and Optimization (DMO), 2011, 190–193,DOI: 10.1109/DMO.2011.5976526.
• [22] S. Mirjalili, S. Z. Mohd Hashim and H. Moradian Sardroudi, “Training feedforward neural networks using hybrid particle swarm optimization and gravitational search algorithm”, Applied Mathematics and Computation, vol. 218,no. 22, 2012, 11125–11137,DOI: 10.1016/j.amc.2012.04.069.
• [23] S. Mirjalili and S. Z. M. Hashim, “A new hybrid PSOGSA algorithm for function optimization”. In: 2010 International Conference on Computer and Information Application, 2010,374–377,DOI: 10.1109/ICCIA.2010.6141614.
• [24] I. Goodfellow, Y. Bengio and A. Courville, Deep learning, MIT Press, 2016.
• [25] C. C. Aggarwal, Neural Networks and Deep Learning: A Textbook, Springer International Publishing, 2018.
• [26] “Greedy Layer-Wise Training of Deep Networks”. Y. Bengio, P. Lamblin, D. Popovici and H. Larochelle, http://papers.nips.cc/paper/3048-greedy-layer-wise-training-ofdeep-networks.pdf. Accessed on: 2020-06-23.
• [27] E. de la Rosa Montero, “El aprendizaje profundo para la identificación de sistemas no lineales,” Thesis, Centro de Investigación y de Estudios Avanzados del Instituto Politécnico Nacional, Mexico, 2014 (In Spanish), https://www.ctrl.cinvestav.mx/~yuw/pdf/MaTesER.pdf. Accessed on: 2020-06-23.
• [28] Y. LeCun and Y. Bengio, “Convolutional networks for images, speech, and time series”. In:The handbook of brain theory and neural networks, 1998, 255–258.
• [29] Y. Bengio and P. Lamblin, “Greedy layer-wise training of deep networks,” Adv. neural, no. 1,pp. 153–160, 2007.
• [30] K. Chellapilla, S. Puri and P. Simard, “High Performance Convolutional Neural Networks for Document Processing,” In: Tenth International Workshop on Frontiers in Handwriting Recognition, La Baule (France), Suvisoft, 2006.
• [31] V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines”. In:Proceedings of the 27th International Conference on Machine Learning, 2010, 807–814.
• [32] M. Ranzato, F. J. Huang, Y.-L. Boureau and Y. LeCun, “Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition”. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition, 2007, 1-8,DOI: 10.1109/CVPR.2007.383157.
• [33] J. Yang, K. Yu, Y. Gong and T. Huang, “Linear spatial pyramid matching using sparse coding for image classification”. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition,2009, 1794–1801,DOI: 10.1109/CVPR.2009.5206757.
• [34] T. Wang, D. J. Wu, A. Coates and A. Y. Ng, “Endto-end text recognition with convolutional eural networks”. In: Proceedings of the 21st International Conference on Pattern Recognition,2012, 3304–3308.
• [35] P. Kim, MATLAB Deep Learning: With Machine Learning, Neural Networks and Artificial Intelligence, Berkeley, California: Apress, 2017,DOI: 10.1007/978-1-4842-2845-6.
• [36] R. Venkatesan and B. Li, Convolutional Neural Networks in Visual Computing: A Concise Guide,CRC Press, 2017.
• [37] L. Lu, Y. Zheng, G. Carneiro, and L. Yang, (eds.),Deep Learning and Convolutional Neural Networks for Medical Image Computing: Precision Medicine, High Performance and Large-Scale Datasets, Springer International Publishing,2017.
• [38] B. González, F. Valdez, P. Melin and G. PradoArechiga, “Fuzzy logic in the gravitationalsearch algorithm for the optimization of modular neural networks in pattern recognition”, Expert Systems with Applications, vol. 42, no. 14,2015, 5839–5847,DOI: 10.1016/j.eswa.2015.03.034.
• [39] B. González, F. Valdez, P. Melin and G. PradoArechiga, “Fuzzy logic in the gravitational search algorithm enhanced using fuzzy logic with dynamic alpha parameter value adaptation for the optimization of modular neural networks in echocardiogram recognition”,Applied Soft Computing, vol. 37, 2015, 245–254,DOI: 10.1016/j.asoc.2015.08.034.
• [40] Y. Poma, P. Melin, C. I. González and G. E. Martínez, “Optimal Recognition Model Based on Convolutional Neural Networks and Fuzzy Gravitational Search Algorithm Method”. In: O. Castillo and P. Melin (eds.), Hybrid Intelligent Systems in Control, Pattern Recognition and Medicine, 2020, 71–81,DOI: 10.1007/978-3-030-34135-0_6.
• [41] C. I. González, P. Melin, J. R. Castro, O. Mendoza and O. Castillo, “General Type-2 Fuzzy Edge Detection in the Preprocessing of a Face Recognition System”. In: P. Melin, O. Castillo and J. Kacprzyk (eds.), Nature-Inspired Design of Hybrid Intelligent Systems, 2017, 3–18,DOI: 10.1007/978-3-319-47054-2_1.
• [42] G. E. Martínez, O. Mendoza, J. R. Castro, A. Rodríguez-Díaz, P. Melin and O. Castillo, “Comparison between Choquet and Sugeno integrals as aggregation operators for pattern recognition”. In: 2016 Annual Conference of the North American Fuzzy Information Processing Society (NAFIPS), 2016, 1–6,DOI: 10.1109/NAFIPS.2016.7851628.
• [43] P. Melin, J. Urias, D. Solano, M. Soto, M. Lopez and O. Castillo, “Voice Recognition with Neural Networks, Type-2 Fuzzy Logic and Genetic Algorithms”, Engineering Letters, vol. 13, no. 2, 2006.
• [44] F. Gaxiola, P. Melin, F. Valdez, J. R. Castro and O. Castillo, “Optimization of type-2 fuzzy weights in backpropagation learning for neural networks using GAs and PSO”, Applied Soft Computing, vol. 38, 2016, 860–871,DOI: 10.1016/j.asoc.2015.10.027.

Uwagi

Opracowanie rekordu ze środków MNiSW, umowa Nr 461252 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2020).