Survey on multi-objective-based parameter optimization for deep learning

• Autorzy: Chakraborty Mrittika , Pal Wreetbhas , Bandyopadhyay Sanghamitra , Maulik Ujjwal

Identyfikatory

DOI

10.7494/csci.2023.24.3.5479

• Języki publikacji: EN

Abstrakty

EN

Deep learning models form some of the most powerful machine-learning models for the extraction of important features. Most of the designs of deep neural models ( i.e., the initialization of parameters) are still manually tuned; hence, obtaining a model with high performance is exceedingly time-consuming and occasionally impossible. Optimizing the parameters of deep networks, therefore requires improved optimization algorithms with high convergence rates. The single objective-based optimization methods that are generally used are mostly time-consuming and do not guarantee optimum performance in all cases. Mathematical optimization problems that contain multiple objective functions that must be optimized simultaneously fall under the category of multi-objective optimization (sometimes referred to as Pareto optimization). Multi-objective optimization problems form one of the alternatives yet useful options for parameter optimization; however, this domain is a bit underexplored. In this survey, we focus on exploring the effectiveness of multi-objective optimization strategies for parameter optimization in conjunction with deep neural networks. The case studies that are used in this study focus on how the two methods are combined to provide valuable insights into the generation of predictions and analysis in multiple applications.

Słowa kluczowe

EN

deep learning; multi-objective optimization; parameter optimization; neural networks

• Wydawca: Wydawnictwa AGH
• Czasopismo: Computer Science
• Rocznik: 2023
• Tom: T. 24 (3)
• Strony: 327--359
• Opis fizyczny: Bibliogr. 99 poz., rys., tab., wykr.

Twórcy

autor

Chakraborty Mrittika

• Jadavpur University, Kolkata, India

autor

Pal Wreetbhas

• Jadavpur University, Kolkata, India

autor

Bandyopadhyay Sanghamitra

• Indian Statistical Institute, Kolkata, India

autor

Maulik Ujjwal

• Jadavpur University, Kolkata, India

Bibliografia

• [1] Abualkishik A.Z., Alwan A.A.: Multi-objective chaotic butterfly optimization with deep neural network based sustainable healthcare management systems, American Journal of Business and Operations Research, vol. 4(2),pp. 39–48, 2021.
• [2] Al-Wesabi F.N., Obayya M., Hilal A.M., Castillo O., Gupta D., Khanna A.:Multi-objective quantum tunicate swarm optimization with deep learning model for intelligent dystrophinopathies diagnosis, Soft Computing, pp. 1–16, 2022.
• [3] Ataei M., Osanloo M.: Using a combination of genetic algorithm and the gridsearch method to determine optimum cutoff grades of multiple metal deposits, International Journal of Surface Mining, Reclamation and Environment, vol. 18(1),pp. 60–78, 2004.
• [4] Bebis G., Georgiopoulos M.: Feed-forward neural networks, IEEE Potentials, vol. 13(4), pp. 27–31, 1994.
• [5] Bergstra J., Bengio Y.: Random search for hyper-parameter optimization, Journal of Machine Learning Research, vol. 13(2), pp. 281–305, 2012.
• [6] Boyd S.P., Vandenberghe L.: Convex optimization, Cambridge University Press, 2004.
• [7] Bui L.T., Alam S.:Multi-objective optimization in computational intelligence: theory and practice, Information Science Reference New York, 2008.
• [8] Chang C.T.:Multi-choice goal programming, Omega, vol. 35(4),pp. 389–396, 2007.
• [9] Charnes A., Clower R.W., Kortanek K.O.: Effective control through coherent decentralization with preemptive goals, Econometrica, pp. 294–320, 1967.
• [10] Charnes A., Cooper W.W., Ferguson R.O.: Optimal estimation of executive compensation by linear programming, Management Science, vol. 1(2),pp. 138–151, 1955.
• [11] Chong E.K., Żak S.H.:An introduction to optimization, vol. 75, John Wiley & Sons, 2013.
• [12] Cohanim B.E., Hewitt J.N., de Weck O.: The design of radio telescope ar-ray configurations using multiobjective optimization: Imaging performance versus cable length, The Astrophysical Journal Supplement Series, vol. 154(2),pp. 705–719, 2004.
• [13] Cohon J.L.: Multiobjective programming and planning, vol. 140, Courier Corporation, 2004.
• [14] Das I., Dennis J.E.: A closer look at drawbacks of minimizing weighted sums ofobjective for Pareto set generation in multicriteria optimization problems, Structural Optimization, vol. 14(1), pp. 63–69, 1997.
• [15] Dawes R.M., Corrigan B.: Linear models in decision making, Psychological Bulletin, vol. 81(2), pp. 95–106, 1974.
• [16] De Weck O.L.: Multiobjective optimization: History and promise. In: Invited Keynote Paper, GL2-2, The Third China-Japan-Korea Joint Symposium on Optimization of Structural and Mechanical Systems, Kanazawa, Japan, vol. 2, 2004.
• [17] Deb K.: Multi-objective Optimisation Using Evolutionary Algorithms: An Introduction. In: Multi-objective Evolutionary Optimisation for Product Design and Manufacturing, Springer, London, 2011. doi: 10.1007/978-0-85729-652-8_1.
• [18] Deb K.: Multi-objective Optimization. In: E. Burke, G. Kendall (eds.), Search Methodologies, pp. 403–449, Springer, Boston, MA, 2014. doi: 10.1007/978-1-4614-6940-7_15.
• [19] Deb K., Datta R.: Hybrid evolutionary multi-objective optimization and analysisof machining operations, Engineering Optimization, vol. 44(6), pp. 685–706, 2012.
• [20] Deb K., Jain H.: An Evolutionary Many-Objective Optimization Algorithm Using Reference-Point-Based Nondominated Sorting Approach, Part I: Solving Problems With Box Constraints, IEEE Transactions on Evolutionary Computation, vol. 18(4), pp. 577–601, 2014. doi: 10.1109/TEVC.2013.2281535.
• [21] Dodgson J.S., Spackman M., Pearman A., Phillips L.D.: Multi-criteria analysis:a manual, Department for Communities and Local Government, London, 2009.https://eprints.lse.ac.uk/12761/1/Multi-criteria_Analysis.pdf.
• [22] Ehrgott M.:Multicriteria optimization, vol. 491, Springer Science & Business Media, 2005.
• [23] Einhorn H.J., McCoach W.: A simple multiattribute utility procedure for evaluation, Behavioral Science, vol. 22(4), pp. 270–282, 1977.
• [24] Erivaldo Fernandes F. Jr., Yen G.G.: Particle swarm optimization of deep neural networks architectures for image classification, Swarm and Evolutionary Computation, vol. 49, pp. 62–74, 2019.
• [25] Fishburn P.C.: Utilities and decision rules: a survey, lexicographic orders, Management Science, vol. 20, pp. 1442–1471, 1974.
• [26] Ghosh S., Das N., Das I., Maulik U.: Understanding Deep Learning Techniques for Image Segmentation, ACM Computing Surveys, vol. 52(4), pp. 1–35, 2019.
• [27] Gino F., Moore D.A.: Effects of task difficulty on use of advice, Journal of Behavioral Decision Making, vol. 20(1), pp. 21–35, 2007.
• [28] Goel T., Murugan R., Mirjalili S., Chakrabartty D.K.: Multi-COVID-Net: Multi-objective optimized network for COVID-19 diagnosis from chest X-ray images, Applied Soft Computing, vol. 115, 108250, 2022.
• [29] Goldberg D.E.:Genetic Algorithms in Search Optimization and Machine Learning, Pearson Education India, 2013.
• [30] Gunantara N.: A review of multi-objective optimization: Methods and its applications, Cogent Engineering, vol. 5(1), 1502242, 2018.
• [31] Gunantara N., Hendrantoro G.: Multi-objective cross-layer optimization for selection of cooperative path pairs in multihop wireless ad hoc networks, Journal of Communications Software and Systems, vol. 9(3), pp. 170–177, 2013.
• [32] Gunantara N., Sastra N.P.: Cooperative diversity selection protocol using Paretomethod with multi objective criterion in wireless ad hoc networks, International Journal of Multimedia and Ubiquitous Engineering, vol. 11(5), pp. 43–54, 2016.
• [33] Gunantara N., Sastra N.P., Hendrantoro G.: Cooperative Diversity Paths selection Protocol Withmulti-Objective Criterion In Wireless Ad-Hoc Networks, International Journal of Applied Engineering Research, vol. 9(23), pp. 22395–22407,2014.
• [34] Gunasekara R.C., Mehrotra K., Mohan C.K.: Multi-objective optimization to identify key players in social networks. In: 2014 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2014),pp. 443–450, IEEE, 2014.
• [35] Haimes Y.: On a Bicriterion Formulation of the Problems of Integrated System Identification and System Optimization, IEEE Transactions on Systems, Man,and Cybernetics, vol. SMC-1(3), pp. 296–297, 1971.
• [36] Hardesty L.: MIT News Office. Explained: Neural networks, 2017.
• [37] Horn J., Nafpliotis N., Goldberg D.E.: A niched Pareto genetic algorithm for multiobjective optimization. In: Proceedings of the first IEEE conference one volutionary computation. IEEE World Congress on Computational Intelligence, pp. 82–87, Ieee, 1994.
• [38] Hsu C.H., Chang S.H., Liang J.H., Chou H.P., Liu C.H., Chang S.C., Pan J.Y.,Chen Y.T., Wei W., Juan D.C.: MONAS: Multi-Objective Neural Architecture Search using Reinforcement Learning, arXiv:180610332, 2018. doi: 10.48550/arXiv.1806.10332.
• [39] Ignizio J.P.: Generalized goal programming an overview, Computers & Operations Research, vol. 10(4), pp. 277–289, 1983.
• [40] İnik Ö., Altıok M., Ülker E., Koçer B.: MODE-CNN: A fast converging multi-objective optimization algorithm for CNN-based models, Applied Soft Computing, vol. 109, 107582, 2021.
• [41] Jain A.K., Mao J., Mohiuddin K.M.: Artificial neural networks: A tutorial, Computer, vol. 29(3), pp. 31–44, 1996.
• [42] Jain H., Deb K.: An evolutionary many-objective optimization algorithm using reference-point based nondominated sorting approach, part II: Handling constraints and extending to an adaptive approach, IEEE Transactions on Evolutionary Computation, vol. 18(4), pp. 602–622, 2013.
• [43] Jena S.:Multi-Objective Optimization of the Design Parameters of a Shell and Tube Type Heat Exchanger Based on Economic and Size Consideration, Ph.D.thesis, 2013.
• [44] Kim I.Y., Weck de O.L.: Adaptive weighted-sum method for bi-objective optimization: Pareto front generation, Structural and Multidisciplinary Optimization,vol. 29, pp. 149–158, 2005.
• [45] Krizhevsky A., Sutskever I., Hinton G.E.: ImageNet classification with deep convolutional neural networks, Communications of the ACM, vol. 60(6),pp. 84–90, 2017.
• [46] LeCun Y., Bengio Y., Hinton G.: Deep learning, Nature, vol. 521,pp. 436–444, 2015.
• [47] Li L., Qin L., Qu X., Zhang J., Wang Y., Ran B.: Day-ahead traffic flow fore-casting based on a deep belief network optimized by the multi-objective particleswarm algorithm, Knowledge-Based Systems, vol. 172, pp. 1–14, 2019.
• [48] Liu H., Simonyan K., Vinyals O., Fernando C., Kavukcuoglu K.: Hierarchical representations for efficient architecture search, arXiv preprint arXiv:171100436,2017.
• [49] Liu H., Simonyan K., Yang Y.: Darts: Differentiable architecture search,arXiv preprint arXiv:180609055, 2018.
• [50] Loni M., Sinaei S., Zoljodi A., Daneshtalab M., Sjödin M.: DeepMaker: A multi-objective optimization framework for deep neural networks in embedded systems, Microprocessors and Microsystems, vol. 73, 102989, 2020.
• [51] Lorenzo P.R., Nalepa J., Kawulok M., Ramos L.S., Pastor J.R.: Particle swarm optimization for hyper-parameter selection in deep neural networks. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 481–488,2017.
• [52] Lu Z., Whalen I., Boddeti V., Dhebar Y., Deb K., Goodman E., Banzhaf W.:NSGA-Net: neural architecture search using multi-objective genetic algorithm. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 419–427, 2019.
• [53] Lv S.X., Wang L.: Deep learning combined wind speed for ecasting with hybridtime series decomposition and multi-objective parameter optimization, Applied Energy, vol. 311, 118674, 2022.
• [54] Ma M., Sun C., Mao Z., Chen X.: Ensemble deep learning with multi-objective optimization for prognosis of rotating machinery, ISA Transactions, vol. 113,pp. 166–174, 2021.
• [55] Mardle S.J., Pascoe S., Tamiz M.: An investigation of genetic algorithms for the optimization of multi-objective fisheries bioeconomic models, International Transactions in Operational Research, vol. 7(1), pp. 33–49, 2000.
• [56] Maulik U.: Medical image segmentation using genetic algorithms, IEEE Transactions on Information Technology in Biomedicine, vol. 13(2), pp. 166–173, 2009.
• [57] Medsker L.R., Jain L.C.: Recurrent neural networks, Design and Applications, vol. 5, pp. 64–67, 2001.
• [58] Messac A., Puemi-Sukam C., Melachrinoudis E.: Aggregate objective functions and Pareto frontiers: required relationships and practical implications, Optimization and Engineering, vol. 1, pp. 171–188, 2000.
• [59] Messac A., Sundararaj G.J., Tappeta R.V., Renaud J.E.: Ability of objective functions to generate points on nonconvex Pareto frontiers, AIAA Journal,vol. 38(6), pp. 1084–1091, 2000.
• [60] Miettinen K.: Nonlinear multiobjective optimization, vol. 12, Springer Science & Business Media, 1999.
• [61] Miikkulainen R., Liang J., Meyerson E., Rawal A., Fink D., Francon O., Raju B.,et al.: Evolving deep neural networks. In:Artificial Intelligence in the Age of Neural Networks and Brain Computing, pp. 293–312, Elsevier, 2019.
• [62] Min H., Storbeck J.: On the origin and persistence of misconceptions ingoal programming, Journal of the Operational Research Society, vol. 42(4), pp. 301–312, 1991.
• [63] Mohammedqasem R., Mohammedqasim H., Biabani S.A.A., Ata O., Alomary M.N., Almehmadi M., Alsairi A.A.,et al.: Multi-Objective deep learning framework for COVID-19 dataset problems, Journal of King Saud University –Science, vol. 35(3), 102527, 2023.
• [64] Mukhopadhyay A., Maulik U., Bandyopadhyay S., Coello C.A.C.: A survey of multiobjective evolutionary algorithms for data mining: Part I,IEEE Transactions on Evolutionary Computation, vol. 18(1), pp. 4–19, 2013.
• [65] Murata T., Ishibuchi H., Tanaka H.: Multi-objective genetic algorithm and its applications to flowshop scheduling, Computers & Industrial Engineering, vol. 30(4), pp. 957–968, 1996.
• [66] Odu G., Charles-Owaba O.: Review of multi-criteria optimization methods – the-ory and applications, IOSR Journal of Engineering, vol. 3(10), pp. 01–14, 2013.
• [67] O’Shea K., Nash R.: An introduction to convolutional neural networks, arXivpreprint arXiv:151108458, 2015.
• [68] Özçelebi T.:Multi-Objective Optimization for Video Streaming, Ph.D. thesis, Graduate School of Sciences and Engineering, Koç University, 2006.
• [69] Pelikan M., Goldberg D.E., Cantú-Paz E.,et al.: BOA: The Bayesian optimization algorithm. In: GECCO-99: proceedings of the Genetic and Evolutionary Computation Conference: a joint meeting of the Eighth International Conferenceon Genetic Algorithms (ICGA-99) and the Fourth Annual Genetic Programming Conference (GP-99), July 13–17, 1999, Orlando, Florida,, pp. 525–532, 1999.
• [70] Pernodet F., Lahmidi H., Michel P.: Use of genetic algorithms for multicriteria optimization of building refurbishment. In:11th International IBPSA Conference, pp. 188–195, Citeseer, 2009.
• [71] Real E., Moore S., Selle A., Saxena S., Suematsu Y.L., Tan J., Le Q.V., Ku-rakin A.: Large-scale evolution of image classifiers. In: International Conferenceon Machine Learning, pp. 2902–2911, PMLR, 2017.
• [72] Rostami S., Neri F.: A fast hypervolume driven selection mechanism for many-objective optimisation problems, Swarm and Evolutionary Computation, vol. 34,pp. 50–67, 2017.
• [73] Ruspini E.H., Zwir I.S.: Automated qualitative description of measurements. In: IMTC/99. Proceedings of the 16th IEEE Instrumentation and Measurement Technology Conference (Cat. No. 99CH36309), vol. 2, pp. 1086–1091, IEEE, 1999.
• [74] Sakthivel N., Subasree S., Alias Priya M P., Tyagi A.K.: Breast lesion identifica-tion and categorization using mammography screening based on combined convolutional recursive neural network framework with parameters optimized usingmulti-objective seagull optimization algorithm, Concurrency and Computation: Practice and Experience, vol. 34(28), e7348, 2022.
• [75] Samek W., Montavon G., Lapuschkin S., Anders C.J., Müller K.R.: Explaining deep neural networks and beyond: A review of methods and applications, Proceedings of the IEEE, vol. 109(3), pp. 247–278, 2021.
• [76] Savic D.: Single-objective vs. multiobjective optimisation for integrated decision support. In: International Congress on Environmental Modelling and Software, vol. 119, pp. 7–12, 2002.
• [77] Schulz H., Behnke S.: Deep learning: Layer-wise learning of feature hierarchies, KI – Künstliche Intelligenz, vol. 26, pp. 357–363, 2012.
• [78] Shenfield A., Rostami S.: Multi-objective evolution of artificial neural networks in multi-class medical diagnosis problems with class imbalance. In: 2017 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB), pp. 1–8, IEEE, 2017.
• [79] Smithson S.C., Yang G., Gross W.J., Meyer B.H.: Neural networks design-ing neural networks: multi-objective hyper-parameter optimization. In: 2016 IEEE/ACM International Conference on Computer-Aided Design (ICCAD),pp. 1–8, IEEE, 2016.
• [80] Sonthi V.K., Nagarajan S., Krishnaraj N.: Automated Telugu Printed and Hand-written Character Recognition in Single Image using Aquila Optimizer based Deep Learning Model, International Journal of Advanced Computer Science and Applications, vol. 12(12), 2021.
• [81] Sonthi V.K., Nagarajan S., Krishnaraj N.: An intelligent Telugu handwritten character recognition using multi-objective mayfly optimization with deeplearning-based DenseNet model, ACM Transactions on Asian and Low-Resource Language Information Processing, vol. 22(3), pp. 1–16, 2022.
• [82] Steuer R.E.: Multiple Criteria Optimization: Theory, Computation, and Application, Wiley Series in Probability and Statistics, Wiley, New York, 1986.
• [83] Suganuma M., Shirakawa S., Nagao T.: A genetic programming approach to de-signing convolutional neural network architectures. In: Proceedings of the Geneticand Evolutionary Computation Conference, pp. 497–504, 2017.
• [84] Sun Y., Xue B., Zhang M., Yen G.G.: Evolving deep convolutional neural net-works for image classification, IEEE Transactions on Evolutionary Computation, vol. 24(2), pp. 394–407, 2019.
• [85] Tang Z., Zhang Z.: The multi-objective optimization of combustion systemoperations based on deep data-driven models, Energy, vol. 182, pp. 37–47, 2019.
• [86] Tao J., Sun G.: Application of deep learning based multi-fidelity surrogate modelto robust aerodynamic design optimization, Aerospace Science and Technology, vol. 92, pp. 722–737, 2019.
• [87] Tapia M.G.C., Coello C.A.C.: Applications of multi-objective evolutionary algorithms in economics and finance: A survey. In: 2007 IEEE Congress on Evolutionary Computation, pp. 532–539, IEEE, 2007.
• [88] Triantaphyllou E., Shu B., Sánchez S.N., Ray T.: Multi-criteria decision making: an operations research approach, Encyclopedia of Electrical and Electronics Engineering, vol. 15(1998), pp. 175–186, 1998.
• [89] Wang J., An Y., Li Z., Lu H.: A novel combined for ecasting model based onneural networks, deep learning approaches, and multi-objective optimization forshortterm wind speed forecasting, Energy, vol. 251, 123960, 2022.
• [90] Wang R., Weng Y., Zhou Z., Chen L., Hao H., Wang J.: Multi-objective ensemble deep learning using electronic health records to predict outcomes after lung cancer radiotherapy, Physics in Medicine & Biology, vol. 64(24), 245005, 2019.
• [91] Wang Y., Liu T., Zhang D., Xie Y.: Dual-convolutional neural network based aerodynamic prediction and multi-objective optimization of a compact turbinerotor, Aerospace Science and Technology, vol. 116, 106869, 2021.
• [92] Wikipedia contributors: Multi-objective optimization – Wikipedia, The Free Encyclopedia, https : / / en.wikipedia.org / w / index.php?title = Multi -objective_optimization&oldid=1147004915 [Online; accessed 3.04.2023], 2023.
• [93] Wu P., He Y., Li Y., He J., Liu X., Wang Y.: Multi-objective optimisation of machining process parameters using deep learning-based data-driven genetic algorithm and TOPSIS, Journal of Manufacturing Systems, vol. 64, pp. 40–52, 2022.
• [94] Xie L., Yuille A.: Genetic CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1379–1388, 2017.
• [95] Xu Y., Du J., Huang Z., Dai L.R., Lee C.H.: Multi-objective learning and mask-based post-processing for deep neural network based speech enhancement, arXivpreprint arXiv:170307172, 2017.
• [96] Yegnanarayana B.: Artificial neural networks, Prentice-Hall of India, Private Limited, New Delhi, 2009.
• [97] You Y.: Multi-objective optimal design of permanent magnet synchronous motor for electric vehicle based on deep learning, Applied Sciences, vol. 10(2), 482, 2020.
• [98] Zhou M., Wang B., Guo S., Watada J.: Multi-objective prediction intervalsfor wind power forecast based on deep neural networks, Information Sciences, vol. 550, pp. 207–220, 2021.
• [99] Zwir I.S., Ruspini E.H.: Qualitative object description: initial reports of the exploration of the frontier. In: Proceedings of EUROFUSE – SIC’99: Eurofuse’99, the fourth meeting of the EURO Working Group on Fuzzy Sets and SIC’99, the second International Conference on Soft and Intelligent Computing, vol. 99, pp. 485–490, 1999.