Selective maintenance optimization with stochastic break duration based on reinforcement learning

Liu, Yilai; Qian, Xinbo

doi:10.17531/ein.2022.4.17

Artykuł - szczegóły

Tytuł artykułu

Selective maintenance optimization with stochastic break duration based on reinforcement learning

Autorzy

Liu Yilai , Qian Xinbo

Treść / Zawartość

Pełne teksty:

Pobierz

Identyfikatory

DOI

10.17531/ein.2022.4.17

Warianty tytułu

Języki publikacji

Abstrakty

For industrial and military applications, a sequence of missions would be performed with a limited break between two adjacent missions. To improve the system reliability, selective maintenance may be performed on components during the break. Most studies on selective maintenance generally use minimal repair and replacement as maintenance actions while break duration is assumed to be deterministic. However, in practical engineering, many maintenance actions are imperfect maintenance, and the break duration is stochastic due to environmental and other factors. Therefore, a selective maintenance optimization model is proposed with imperfect maintenance for stochastic break duration. The model is aimed to maximize the reliability of system successfully completing the next mission. The reinforcement learning(RL) method is applied to optimally select maintenance actions for selected components. The proposed model and the advantages of the RL are verified by three case studies verify.

Słowa kluczowe

selective maintenance stochastic break duration imperfect maintenance reinforcement learning

Wydawca

Polskie Naukowo-Techniczne Towarzystwo Eksploatacyjne

Czasopismo

Eksploatacja i Niezawodność

Rocznik

2022

Tom

Vol. 24, no. 4

Strony

771--784

Opis fizyczny

Bibliogr. 49 poz., rys., tab.

Twórcy

autor

Liu Yilai

1454786628@qq.com

Key Laboratory of Metallurgical Equipment and Control Technology, Ministry of Education, Wuhan University of Science and Technology, Wuhan, China
Hubei Key Laboratory of Mechanical Transmission and Manufacturing Engineering, Wuhan University of Science and Technology, Wuhan , China

https://orcid.org/0000-0003-2370-9250

autor

Qian Xinbo

xinboqian@wust.edu.cn

Key Laboratory of Metallurgical Equipment and Control Technology, Ministry of Education, Wuhan University of Science and Technology, Wuhan, China
Precision Manufacturing Institute, Wuhan University of Science and Technology, Wuhan, China

Bibliografia

1. Andriotis C P, Papakonstantinou K G. Managing engineering systems with large state and action spaces through deep reinforcement learning. Reliability Engineering & System Safety, 2019; 191: 106483, https://doi.org/10.1016/j.ress.2019.04.036.
2. Barde S R A, Yacout S, Shin H. Optimal preventive maintenance policy based on reinforcement learning of a fleet of military trucks. Journal of Intelligent Manufacturing, 2019; 30(1): 147-161, https://doi.org/10.1007/s10845-016-1237-7.
3. Baxter L A, Kijima M, Tortorella M. A point process model for the reliability of a maintained system subject to general repair. Stochastic models, 1996; 12(1): 12-1, https://doi.org/10.1080/15326349608807372.
4. Cao W, Jia X, Hu Q, et al. A literature review on selective maintenance for multi-unit systems. Quality and Reliability Engineering International 2018; 34(5): 824-845, https://doi.org/10.1002/qre.2293.
5. Cao W, Jia X, Hu Q, et al. Selective maintenance for maximising system availability: a simulation approach. International Journal of Innovative Computing and Applications 2017; 8(1): 12-20, https://doi.org/10.1504/ijica.2017.082493.
6. Cao W, Li F, Ran Q. Study on selective maintenance optimization for multi-State systems confronting random missions. Journal of Ordnance Engineering College 2017; 29(2): 17-22.
7. Cassady C R, Murdock Jr W P, Pohl E A. Selective maintenance for support equipment involving multiple maintenance actions. European Journal of Operational Research 2001; 129(2): 252-258, https://doi.org/10.1016/s0377-2217(00)00222-8.
8. Chen Y, Jiang T, Liu Y. Selective maintenance optimization: research advances and challenges. Operations Research Transactions, 2019, 23(3): 27-46, 10.15960/j.cnki.issn.1007-6093.2019.03.003.
9. Chen Y, Ma Y, Liu Q, et al. Research on selective maintenance decision-making of equipment considering imperfect maintenance under sequential mission. AERO WEAPONRY 2019.
10. Chen Z, Zhang L, Tian G, et al. Economic maintenance planning of complex systems based on discrete artificial bee colony algorithm. IEEE Access 2020; 8: 108062-108071, https://doi.org/10.1109/ACCESS.2020.2999601.
11. U, Khatab A, et al. Optimal selective maintenance decisions for large serial k-out-of-n: G systems under imperfect maintenance. Reliability Engineering & System Safety 2018; 175: 234-245, https://doi.org/10.1016/j.ress.2018.03.023.
12. Dui H, Zheng X, Zhao QQ, Fang Y. Preventive maintenance of multiple components for hydraulic tension systems. Eksploatacja i Niezawodnosc – Maintenance and Reliability 2021; 23 (3): 489–497, https://doi.org/10.17531/ein.2021.3.9.
13. Gao H, Zhang X, Yang X, et al. Optimal selective maintenance decision-making for consecutive-mission systems with variable durations and limited maintenance Time. Mathematical Problems in Engineering, 2021; (2021), https://doi.org/10.1155/2021/5534659.
14. Hu M. Research on maintenance decision of wind turbine components based on reinforcement learning. School of Energy Power and Mechanical Engineer.
15. Huang J, Chang Q, Arinez J. Deep reinforcement learning based preventive maintenance policy for serial production lines. Expert Systems with Applications 2020; 160: 113701, https://doi.org/10.1016/j.eswa.2020.113701.
16. Hu Y, Miao X, Zhang J, et al. Reinforcement learning-driven maintenance strategy: A novel solution for long-term aircraft maintenance decision optimization. Computers & Industrial Engineering, 2021; 153: 107056, https://doi.org/10.1016/j.cie.2020.107056.
17. Khatab A, Aghezzaf E H, Djelloul I, et al. Selective maintenance for series-parallel systems when durations of missions and planned breaks are stochastic. IFAC-PapersOnLine 2016; 49(12): 1222-1227, https://doi.org/10.1016/j.ifacol.2016.07.677.
18. Khatab A, Aghezzaf E H, Djelloul I, et al. Selective maintenance optimization for systems operating missions and scheduled breaks with stochastic durations. Journal of manufacturing systems 2017; 43: 168-177, https://doi.org/10.1016/j.jmsy.2017.03.005.
19. Khatab A, Aghezzaf E H. Selective maintenance optimization when quality of imperfect maintenance actions are stochastic. Reliability engineering & system safety 2016; 150: 182-189, https://doi.org/10.1016/j.ress.2016.01.026.
20. Khatab A, Aghezzaf E L H, Diallo C, et al. Selective maintenance optimisation for series-parallel systems alternating missions and scheduled breaks with stochastic durations. International Journal of Production Research 2017; 55(10): 3008-3024, https://doi.org/10.1080/00207543.2017.1290295.
21. Li Z, Xu Y, Gao S, et al. Reliability modeling for repairable multi-state Elements based on Markov process. AERO WEAPONRY, 2018, 10.19297/j.cnki.41-1228/tj.2018.05.012.
22. Li Z, Zhong S, Lin L. An aero-engine life-cycle maintenance policy optimization algorithm: Reinforcement learning approach. Chinese Journal of Aeronautics 2019; 32(9): 2133-2150.
23. Lin D, Zuo M J, Yam R C M. General sequential imperfect preventive maintenance models. International Journal of reliability, Quality and safety Engineering 2000; 7(03): 253-266, https://doi.org/10.1142/S0218539300000213.
24. Liu Y, Chen Y, Jiang T. Dynamic selective maintenance optimization for multi-state systems over a finite horizon: A deep reinforcement learning approach. European Journal of Operational Research 2020; 283(1): 166-181, https://doi.org/10.1016/j.ejor.2019.10.049.
25. Liu Y, Chen Y, Jiang T. On sequence planning for selective maintenance of multi-state systems under stochastic maintenance durations. European Journal of Operational Research 2018; 268(1): 113-127, https://doi.org/10.1016/j.ejor.2017.12.036.
26. Liu Y, Huang H Z. Optimal selective maintenance strategy for multi-state systems under imperfect maintenance. IEEE Transactions on Reliability 2010; 59(2): 356-367, https://doi.org/10.1109/TR.2010.2046798.
27. Lust T, Roux O, Riane F. Exact and heuristic methods for the selective maintenance problem. European journal of operational research 2009; 197(3): 1166-1177, https://doi.org/10.1016/j.ejor.2008.03.047.
28. Lv X Z, Yu Y L, Zhang L, et al. Stochastic program for selective maintenance decision considering diagnostics uncertainty of built-in test equipment. IEEE 2011; 584-589, https://doi.org/10.1109/ICQR2MSE.2011.5976681.
29. Mahmoodzadeh Z, Wu K Y, Lopez Droguett E, et al. Condition-based maintenance with reinforcement learning for dry gas pipeline subject to internal corrosion. Sensors 2020; 20(19): 5708, https://doi.org/10.3390/s20195708.
30. Malik M A K. Reliable preventive maintenance scheduling.AIIE transactions, 1979; 11(3):221-228, https://doi.org/10.1080/05695557908974463.
31. Martínez-Tenor A, Fernández-Madrigal J A, Cruz-Martín A, et al. Towards a common implementation of reinforcement learning for multiple robotic tasks. Expert Systems with Applications 2018; 100: 246-259, https://doi.org/10.1016/j.eswa.2017.11.011.
32. Nakagawa T. Optimum policies when preventive maintenance is imperfect. IEEE Transactions on Reliability 1979; 28(4): 331-332, https://doi.org/10.1109/TR.1979.5220624.
33. Pandey M, Zuo M J, Moghaddass R, et al. Selective maintenance for binary systems under imperfect repair. Reliability Engineering & System Safety 2013; 113: 42-51, https://doi.org/10.1016/j.ress.2012.12.009.
34. Paraschos P D, Koulinas G K, Koulouriotis D E. Reinforcement learning for combined production-maintenance and quality control of a manufacturing system with deterioration failures. Journal of Manufacturing Systems, 2020; 56: 470-483, https://doi.org/10.1016/j.jmsy.2020.07.004.
35. Peng S. Reinforcement learning with Gaussian processes for condition-based maintenance. Computers & Industrial Engineering 2021; 158: 107321, https://doi.org/10.1016/j.cie.2021.107321.
36. Rajagopalan R, Cassady C R. An improved selective maintenance solution approach. Journal of Quality in Maintenance Engineering 2006; 12(2):172-185, https://doi.org/10.1108/13552510610667183.
37. Rice W F, Cassady C R, Nachlas J A. Optimal maintenance plans under limited maintenance time. Proceedings of the seventh industrial engineering research conference: 1998: 1-3.
38. Ruan J H, Wang Z X, Chan F T S, et al. A reinforcement learning-based algorithm for the aircraft maintenance routing problem. Expert Systems with Applications 2021; 169: 114399, https://doi.org/10.1016/j.eswa.2020.114399.
39. Su Y, Meng L, Kong X, et al. Generative adversarial networks for gearbox of wind turbine with unbalanced data sets in fault diagnosis. IEEE Sensors Journal, 2022; https://doi.org/10.1109/JSEN.2022.3178137.
40. Sun Y, Sun Z. Selective Maintenance on a Multi-State Transportation System Considering Maintenance Sequence Arrangement. IEEE Access 2021; 9: 70048-70060, https://doi.org/10.1109/ACCESS.2021.3078140.
41. Tanwar M, Rai R N, Bolia N. Imperfect repair modeling using Kijima type generalized renewal process. Reliability Engineering & System Safety 2014; 124: 24-31, https://doi.org/10.1016/j.ress.2013.10.007.
42. Wang H, Pham H. A quasi renewal process and its applications in imperfect maintenance. International journal of systems science 1996; 27(10): 1055-1062, https://doi.org/10.1080/00207729608929311.
43. Wang S, Zhang S, Li Y, et al. Selective maintenance decision-making of complex systems considering imperfect maintenance. International Journal of Performability Engineering 2018; 14(12): 2960, https://doi.org/10.23940/IJPE.18.12.P6.29602970.
44. Xu Q Z, LM Guo. Method for solving the selective maintenance problem for series-parallel system. Machinery Design & Manufacture 2016; 0(1): 61-65, https://doi.org/10.3969/j.issn.1001-3997.2016.01.017.
45. Yan J, Zhang Q, Hu X. Review of path planning techniques based on reinforcement learning. Computer Engineering 2021; 47(10):10, 10.19678/j.issn.1000-3428.0060683.
46. Yang Z, Qi C. Preventive maintenance of a multi-yield deteriorating machine:Using reinforcement learning . Systems Engineering-Theory & Practice, 2013, 33(7): 1647-1653.
47. Yousefi N, Tsianikas S, Coit D W. Dynamic maintenance model for a repairable multi-component system using deep reinforcement learning. Quality Engineering, 2022; 34(1):16-35, https://doi.org/10.1080/08982112.2021.1977950.
48. Zhao J, Liu J, Zhao Z, et al. A high‐performance maintenance strategy for stochastic selective maintenance. Concurr Comp-Pract E 2019; 31(12): e4840, https://doi.org/10.1002/cpe.4840.
49. Zhao X, Al-Khalifa K N, Hamouda A M, et al. Age replacement models: A summary with new perspectives and methods. Reliability Engineering & System Safety 2017; 161: 95-105, https://doi.org/10.1016/j.ress.2017.01.011.

Uwagi

Opracowanie rekordu ze środków MEiN, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2022-2023).

Typ dokumentu

Bibliografia

Identyfikator YADDA

bwmeta1.element.baztech-62351486-9f46-4944-9eb2-48989605d812