CNC machine control using deep reinforcement learning

• Autorzy: Kalandyk Dawid , Kwiatkowski Bogdan , Mazur Damian

Identyfikatory

DOI

10.24425/bpasts.2024.148940

• Języki publikacji: EN

Abstrakty

EN

Optimization of industrial processes such as manufacturing or processing of specific materials constitutes a point of interest for many researchers, and its application can lead not only to speeding up the processes in question, but also to reducing the energy cost incurred during them. This article presents a novel approach to optimizing the spindle motion of a computer numeric control (CNC) machine. The proposed solution is to use deep learning with reinforcement to map the performance of the reference points realization optimization (RPRO) algorithm used in the industry. A detailed study was conducted to see how well the proposed method performs the targeted task. In addition, the influence of a number of different factors and hyperparameters of the learning process on the performance of the trained agent was investigated. The proposed solution achieved very good results, not only satisfactorily replicating the performance of the benchmark algorithm, but also speeding up the machining process and providing significantly higher accuracy.

Słowa kluczowe

EN

deep reinforcement learning; CNC machining; machining optimization

PL

uczenie głębokie ze wzmocnieniem; obróbka CNC; optymalizacja obróbki

• Wydawca: Polska Akademia Nauk, Wydział IV Nauk Technicznych
• Czasopismo: Bulletin of the Polish Academy of Sciences. Technical Sciences
• Rocznik: 2024
• Tom: Vol. 72, nr 3
• Strony: art. no. e148940
• Opis fizyczny: Bibliogr. 60 poz., rys., tab.

Twórcy

autor

Kalandyk Dawid

• Doctoral School of the Rzeszów University of Technology, Powsta ´nców Warszawy Ave. 12, 35-959 Rzeszów, Poland

• https://orcid.org/0000-0002-7317-5499

autor

Kwiatkowski Bogdan

• Department of Electrical and Computer Engineering Fundamentals, Rzeszow University of Technology, W. Pola str. 2, 35-959 Rzeszów, Poland

• https://orcid.org/0000-0001-5287-2191

autor

Mazur Damian

• Department of Electrical and Computer Engineering Fundamentals, Rzeszow University of Technology, W. Pola str. 2, 35-959 Rzeszów, Poland

• https://orcid.org/0000-0002-3247-5903

Bibliografia

• [1] J.E. Bobrow, S. Dubowsky, and J.S. Gibson, “Time-optimal control of robotic manipulators along specified paths,” Int. J. Rob. Res., vol. 4, no. 3, pp. 3–17, 1985.
• [2] J. H. Lee, Y. Liu, and S.-H. Yang, “Accuracy improvement of miniaturized machine tool: geometric error modeling and compensation,” Int. J. Mach. Tools. Manuf., vol. 46, no. 12–13, pp. 1508–1516, 2006.
• [3] S.Z. Mansour and R. Seethaler, “Feedrate optimization for computer numerically controlled machine tools using modeled and measured process constraints,” J. Manuf. Sci. Eng., vol. 139, no. 1, p. 011012, 2017.
• [4] Q. Bi, N. Huang, C. Sun, Y. Wang, L. Zhu, and H. Ding, “Identification and compensation of geometric errors of rotary axes on five-axis machine by on-machine measurement,” Int. J. Mach. Tools. Manuf., vol. 89, pp. 182–191, 2015.
• [5] X. Li, H. Zhao, X. Zhao, and H. Ding, “Interpolation-based contour error estimation and component-based contouring control for five-axis CNC machine tools,” Sci. China. Technol. Sci., vol. 61, pp. 1666–1678, 2018.
• [6] B. Kwiatkowski, T. Kwater, D. Mazur, and J. Bartman, “An offline application that determines the maximum accuracy of the realization of reference points from G-code for given parameters of CNC machine dynamics,” Bull. Pol. Acad. Sci. Tech. Sci., vol. 72, p. e147345, 2024, doi: 10.24425/bpasts.2023.147345.
• [7] J.M. Langeron, E. Duc, C. Lartigue, and P. Bourdet, “A new format for 5-axis tool path computation, using Bspline curves,” Comput.-Aided Des., vol. 36, no. 12, pp. 1219–1229, 2004.
• [8] Y. Sun, S. Sun, J. Xu, and D. Guo, “A unified method of generating tool path based on multiple vector fields for CNC machining of compound NURBS surfaces,” Comput.-Aided Des., vol. 91, pp. 14–26, 2017.
• [9] M. Chen and Y. Sun, “A moving knot sequence-based feedrate scheduling method of parametric interpolator for CNC machining with contour error and drive constraints,” Int. J. Adv. Manuf. Technol., vol. 98, pp. 487–504, 2018.
• [10] B. Pękala, E. Rak, B. Kwiatkowski, A. Szczur, and R. Rak, “The use of concave and convex functions to optimize the feed-rate of numerically controlled machine tools,” in 2020 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), IEEE, 2020, pp. 1–8.
• [11] M. Rahaman, R. Seethaler, and I. Yellowley, “A new approach to contour error control in high speed machining,” Int. J. Mach. Tools Manuf., vol. 88, pp. 42–50, 2015.
• [12] S.D. Timar, R.T. Farouki, T.S. Smith, and C.L. Boyadjieff, “Algorithms for time–optimal control of CNC machines along curved tool paths,” Robot Comput. Integr. Manuf., vol. 21, no. 1, pp. 37–53, 2005.
• [13] J. Dong and J. A. Stori, “A generalized time-optimal bidirectional scan algorithm for constrained feed-rate optimization,” J. Dyn. Sys., Meas., Control., vol. 128, no. 2, pp. 379390, 2006.
• [14] Z. Shiller and H.-H. Lu, “Robust computation of path constrained time optimal motions,” in Proc. IEEE International Conference on Robotics and Automation, IEEE, 1990, pp. 144–149.
• [15] S.D. Timar and R.T. Farouki, “Time-optimal traversal of curved paths by Cartesian CNC machines under both constant and speed-dependent axis acceleration bounds,” Robot. Integr. Manuf., vol. 24, no. 1, pp. 16–31, 2008.
• [16] C. Wang, X.P. Tan, S.B. Tor, and C.S. Lim, “Machine learning in additive manufacturing: State-of-the-art and perspectives,” Addit. Manuf., vol. 36, p. 101538, 2020.
• [17] A. Molina, H. Ponce, P. Ponce, G. Tello, and M. Ramírez, “Artificial hydrocarbon networks fuzzy inference systems for CNC machines position controller,” Int. J. Adv. Manuf. Technol., vol. 72, pp. 1465–1479, 2014.
• [18] T. Kar, N.K. Mandal, and N.K. Singh, “Multi-response optimization and surface texture characterization for CNC milling of inconel 718 alloy,” Arab. J. Sci. Eng., vol. 45, pp. 1265–1277, 2020.
• [19] S. Datta, S.S. Mahapatra, B.C. Routara, and A. Bandyopadhyay, “The fuzzy inference system approach to a multi-performance characteristic index for surface quality improvement in CNC end milling,” Int. J. Exp. Des. Process Optim., vol. 2, no. 3, pp. 265–282, 2011.
• [20] M.F. Alam, M. Shtein, K. Barton, and D.J. Hoelzle, “Autonomous manufacturing using machine learning: A computational case study with a limited manufacturing budget,” in International Manufacturing Science and Engineering Conference, 2020, p. V002T07A009.
• [21] K. Arulkumaran, M.P. Deisenroth, M. Brundage, and A.A. Bharath, “Deep reinforcement learning: A brief survey,” IEEE Signal Process. Mag., vol. 34, no. 6, pp. 26–38, 2017.
• [22] V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015.
• [23] G. Lample and D.S. Chaplot, “Playing FPS games with deep reinforcement learning,” in Proc. AAAI Conference on Artificial Intelligence, 2017.
• [24] D. Kalandyk, “Reinforcement learning in car control: A brief survey,” in 2021 Selected Issues of Electrical Engineering and Electronics, WZEE 2021, 2021. doi: 10.1109/WZEE54157.2021.9576838.
• [25] J. Marquez, C. Sullivan, R.M. Price, and R.C. Roberts, “Hardware-in-the-Loop Soft Robotic Testing Framework using an Actor-Critic Deep Reinforcement Learning Algorithm,” IEEE Robot. Autom. Lett., vol. 8, no. 9, pp. 6076–6082, 2023, doi: 10.1109/LRA.2023.3301215.
• [26] Q. Su, B. Li, C. Wang, C. Qin, and W. Wang, “A power allocation scheme based on deep reinforcement learning in HetNets,” in 2020 international conference on computing, networking and communications (ICNC), IEEE, 2020, pp. 245–250.
• [27] X. Chen, H. Zhang, C. Wu, S. Mao, Y. Ji, and M. Bennis, “Optimized computation offloading performance in virtual edge computing systems via deep reinforcement learning,” IEEE Internet Things J., vol. 6, no. 3, pp. 4005–4018, 2018.
• [28] K. Li, Y. Zhang, K. Li, and Y. Fu, “Adversarial feature hallucination networks for few-shot learning,” in Proc. IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 13470–13479.
• [29] X. Wang et al., “Dynamic scheduling of tasks in cloud manufacturing with multi-agent reinforcement learning,” J. Manuf. Syst., vol. 65, pp. 130–145, 2022.
• [30] G. Velusamy and R. Lent, “Evaluating reinforcement learning methods for bundle routing control,” in 2019 IEEE Cognitive Communications for Aerospace Applications Workshop (CCAAW), IEEE, 2019, pp. 1–4.
• [31] A.M. Seid, G.O. Boateng, S. Anokye, T. Kwantwi, G. Sun, and G. Liu, “Collaborative computation offloading and resource allocation in multi-UAV-assisted IoT networks: A deep reinforcement learning approach,” IEEE Internet Things J., vol. 8, no. 15, pp. 12203–12218, 2021.
• [32] C. Li, P. Zheng, Y. Yin, B. Wang, and L. Wang, “Deep reinforcement learning in smart manufacturing: A review and prospects,” CIRP J. Manuf. Sci. Technol., vol. 40, pp. 75–101, 2023.
• [33] B. Fernandez-Gauna, I. Ansoategui, I. Etxeberria-Agiriano, and M. Graña, “Reinforcement learning of ball screw feed drive controllers,” Eng. Appl. Artif. Intell., vol. 30, pp. 107–117, 2014.
• [34] S. Baer, J. Bakakeu, R. Meyes, and T. Meisen, “Multi-agent reinforcement learning for job shop scheduling in flexible manufacturing systems,” in 2019 Second International Conference on Artificial Intelligence for Industries (AI4I), IEEE, 2019, pp. 22–25.
• [35] Y.-C. Wang and J. M. Usher, “Application of reinforcement learning for agent-based production scheduling,” Eng. Appl. Artif. Intell., vol. 18, no. 1, pp. 73–82, 2005.
• [36] T. Zhou, D. Tang, H. Zhu, and Z. Zhang, “Multi-agent reinforcement learning for online scheduling in smart factories,” Robot Comput. Integr. Manuf., vol. 72, p. 102202, 2021.
• [37] X. Jing, X. Yao, M. Liu, and J. Zhou, “Multi-agent reinforcement learning based on graph convolutional network for flexible job shop scheduling,” J. Intell. Manuf., pp. 1–19, 2022.
• [38] K. Chang, S.H. Park, and J.-G. Baek, “AGV dispatching algorithm based on deep Q-network in CNC machines environment,” Int. J. Comput. Integr. Manuf., vol. 35, no. 6, pp. 662–677, 2022.
• [39] J. Yao, B. Lu, and J. Zhang, “Tool remaining useful life prediction using deep transfer reinforcement learning based on long short-term memory networks,” Int. J. Adv. Manuf. Technol., vol. 2021, p. 9937846, 2022.
• [40] R. Lamprecht, F. Wurst, and M.F. Huber, “Reinforcement learning based condition-oriented maintenance scheduling for flow line systems,” in 2021 IEEE 19th International Conference on Industrial Informatics (INDIN), IEEE, 2021, pp. 1–7.
• [41] M.L.R. Rodríguez, S. Kubler, A. de Giorgio, M. Cordy, J. Robert, and Y. Le Traon, “Multi-agent deep reinforcement learning based Predictive Maintenance on parallel machines,” Robot. Comput. Integr. Manuf., vol. 78, p. 102406, 2022.
• [42] A. Mishra and V. S. Jatti, “Reinforcement learning based approach for the optimization of mechanical properties of additively manufactured specimens,” Int. J. Interact. Des. Manuf.-IJIDeM, vol. 17, pp. 2045–2053, 2023.
• [43] D. Limoge, A. Sunstrom, V. Pinskiy, and M. Putman, “Defending Industrial Production Using AI Process Control,” in 2020 IEEE Systems Security Symposium (SSS), IEEE, 2020, pp. 1–4.
• [44] Y. Zhang, Y. Li, and K. Xu, “Reinforcement learning–based tool orientation optimization for five-axis machining,” Int. J. Adv. Manuf. Technol., vol. 119, no. 11–12, pp. 7311–7326, 2022.
• [45] Q. Xiao, C. Li, Y. Tang, and L. Li, “Meta-reinforcement learning of machining parameters for energy-efficient process control of flexible turning operations,” IEEE Trans. Autom. Sci. Eng., vol. 18, no. 1, pp. 5–18, 2019.
• [46] W. Li et al., “A novel milling parameter optimization method based on improved deep reinforcement learning considering machining cost,” J. Manuf. Process., vol. 84, pp. 1362–1375, 2022.
• [47] H.A. Taha, S. Yacout, and Y. Shaban, “Deep Reinforcement Learning for autonomous pre-failure tool life improvement,” Int. J. Adv. Manuf. Technol., vol. 121, no. 9–10, pp. 6169–6192, 2022.
• [48] X. Liu and D. Chang, “An Improved Method for Optimizing CNC Laser Cutting Paths for Ship Hull Components with Thicknesses up to 24 mm,” J. Mar. Sci. Eng., vol. 11, no. 3, p. 652, 2023.
• [49] M.F. Alam, M. Shtein, K. Barton, and D. Hoelzle, “Reinforcement learning enabled autonomous manufacturing using transfer learning and probabilistic reward modeling,” IEEE Control Syst. Lett., vol. 7, pp. 508–513, 2022.
• [50] L. Jonath, J. Luderich, J. Brezina, A.M. Gonzalez Degetau, and S. Karaoglu, “Improving the Thermal Behavior of High-Speed Spindles Through the Use of an Active Controlled Heat Pipe System,” in International Conference on Thermal Issues in Machine Tools, 2023, pp. 203–218.
• [51] G. Singh and R. Sharma, “Cnc Machine Handling for Holes Servicing through Programming,” in Proc. International Conference on Innovative Computing & Communication (ICICC) 2022, 2023.
• [52] F. Jaensch, A. Csiszar, J. Sarbandi, and A. Verl, “Reinforcement learning of a robot cell control logic using a software-in-the-loop simulation as environment,” in 2019 Second International Conference on Artificial Intelligence for Industries (AI4I), IEEE, 2019, pp. 79–84.
• [53] Q. Chen, B. Heydari, and M. Moghaddam, “Leveraging task modularity in reinforcement learning for adaptable industry 4.0 automation,” J. Mech. Des., vol. 143, no. 7, p. 071701, 2021. doi: 10.1115/1.4049531.
• [54] A.A. Apolinarska et al., “Robotic assembly of timber joints using reinforcement learning,” Autom. Constr., vol. 125, p. 103569, 2021.
• [55] B. Li, H. Zhang, P. Ye, and J. Wang, “Trajectory smoothing method using reinforcement learning for computer numerical control machine tools,” Robot. Comput. Integr. Manuf., vol. 61, p. 101847, 2020.
• [56] Y. Jiang, J. Chen, H. Zhou, J. Yang, P. Hu, and J. Wang, “Contour error modeling and compensation of CNC machining based on deep learning and reinforcement learning,” Int. J. Adv. Manuf. Technol., pp. 1–20, 2022.
• [57] C. Dripke, S. Höhr, A. Csiszar, and A. Verl, “A concept for the application of reinforcement learning in the optimization of CAM-generated tool paths,” in Machine Learning for Cyber Physical Systems: Selected papers from the International Conference ML4CPS 2016, Springer, 2017, pp. 1–8.
• [58] V. Samsonov, E. Chrismarie, H.-G. Köpken, S. Bär, D. Lütticke, and T. Meisen, “Deep representation learning and reinforcement learning for workpiece setup optimization in CNC milling,” Prod. Eng., vol. 17, no. 6, pp. 847–859, 2023.
• [59] Z. Lin, T. Chen, Y. Jiang, H. Wang, S. Lin, and M. Zhu, “B-Spline-Based Curve Fitting to Cam Pitch Curve Using Reinforcement Learning,” Intell. Autom. Soft Comput., vol. 36, no. 2, p. 2145, 2023.
• [60] D. Kalandyk, B. Kwiatkowski, and D. Mazur, “Application of Mamdani Fuzzy Logic Inference System to Optimise CNC Machine Motion Dynamics,” in IEEE International Conference on Fuzzy Systems, 2023. doi: 10.1109/FUZZ52849.2023.10309802.

Uwagi

Opracowanie rekordu ze środków MNiSW, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2024).