Optimal control of dynamic systems using a new adjoining cell mapping method with reinforcement learning

• Autorzy: Arribas-Navarro T. , Prieto S. , Plaza M.
• Języki publikacji: EN

Abstrakty

EN

This work aims to improve and simplify the procedure used in the Control Adjoining Cell Mapping with Reinforcement Learning (CACM-RL) technique, for the tuning process of an optimal contro ller during the pre-learning stage (controller design), making easier the transition from a simulation environment to the real world. Common problems, encountered when working with CACM-RL, are the adjustment of the cell size and the long-term evolution error. In this sense, the main goal of the new approach, developed for CACM-RL that is proposed in this work (CACMRL*), is to give a response to both problems for helping engineers in defining of the control solution with accuracy and stability criteria instead of cell sizes. The new approach improves the mathematical analysis techniques and reduces the engineering effort during the design phase. In order to demonstrate the behaviour of CACM-RL*, three examples are described to show its application to real problems. In All the examples, CACM-RL* improves with respect to the considered alternatives. In some cases, CACM- RL* improves the average controllability by up to 100%.

Słowa kluczowe

EN

optimal control; cells mapping; state space; reinforcement learning; stability; nonlinear control; controllability

• Wydawca: Systems Research Institute, Polish Academy of Sciences
• Czasopismo: Control and Cybernetics
• Rocznik: 2015
• Tom: Vol. 44, no. 3
• Strony: 369--387
• Opis fizyczny: Bibliogr. 20 poz., rys.

Twórcy

autor

Arribas-Navarro T.

• SOTICOL Robotics Systems, S.L., Madrid, Spain

autor

Prieto S.

• Computer Engineering Department, Universidad de Alcal´a, Alcal´a de Henares, Spain

autor

Plaza M.

• SOTICOL Robotics Systems, S.L., Madrid, Spain

Bibliografia

• [1] BARTO, A. (1998) Reinforcement Learning: An Introduction. MIT Press.
• [2] BARTO, A., BRADTKE, S. J. and SINGH, S. P. (1995) Learning to act using Real-Time Dynamic Programming. Artificial Intelligence, Special Volume on Computational Research on Interaction and Agency, 72 (1), 81-138.
• [3] BELLMAN, R. E. (2010) Dynamic Programming. Princeton University Press.
• [4] BURSAL, F. H. and TONGUE, B. H. (1992) A New Method of Nonlinear System Identification using Interpolated Cell Mapping. American Control Conference, Chicago, USA, 3160-3164.IEEE.
• [5] GOMEZ, M. (2009) Planificacion optima de movimiento y aprendizaje por refuerzo en vehıculos moviles autonomos. PhD thesis, Universidad de Alcala.
• [6] GOMEZ, M., ARRIBAS, T. and S´ANCHEZ, S. (2012) Optimal Control based on CACM-RL in a Two-Wheeled Inverted Pendulum. International Journal of Advanced Robotic Systems, 9 (1), 1-8.
• [7] GOMEZ, M., GONZALEZ, R. V., MARTINEZ-MARIN, T., MEZIAT, D. and SANCHEZ, S. (2011) Optimal Motion Planning by Reinforcement Learning in Autonomous Mobile Vehicles. Robotica, 30 (2), 159-170.
• [8] GOMEZ, M., MARTINEZ-MARIN, T., SANCHEZ, S. and MEZIAT, D. (2007) Optimal control applied to Wheeled Mobile Vehicles. Proc. IEEE International Symposium on Intelligent Signal Processing, Alcal´a de Henares, Madrid, Spain, 83-88. IEEE.
• [9] GUTTALU, R. S. and ZUFIRIA, P. J. (1993) The adjoining cell mapping and its recursive unraveling, Part II: Application to selected problems. Nonlinear Dynamics, 4 (4), 309-336.
• [10] HSU C.S., (1985) A discrete method of optimal control based upon the cell state space concept. Journal of Optimization Theory and Applications, 46 (4), 547-569.
• [11] MO-HONG, C. (1993) A modified cell-to-cell mapping method for nonlinear systems. Computers & Mathematics with Applications, 25 (8), 47-57.
• [12] MOORE, A. (1990) Efficient Memory-Based Learning for Robot Control. PhD thesis, University of Cambridge.
• [13] MOORE, A. and ATKESON, C. (1995) The parti-game algorithm for variable resolution reinforcement learning in multidimensional state space. Machine Learning, 21 (3), 199-233.
• [14] PAPA, M., HENG-MING, T. and SHENOI, S. (1995) Evaluation of cell state techniques for optimal controller design. Proc. Int. Joint Conference of the Fourth IEEE International Conference on Fuzzy systems and The Second International Fuzzy Engineering Symposium, Yokohama, Japan, 3, 1331-1338. IEEE.
• [15] PAPA, M., HENG-MING, T. and SHENOI, S. (1997) Cell mapping for controller design and evaluation. IEEE Control Systems, 17 (2), 52-65.
• [16] SONG, F. and SMITH, S. M. (2002) Cell-state-space-based search. IEEE Control Systems, 22 (4), 42-56. TONGUE, B. H. (1987) On the Global Analysis of Nonlinear Systems through Interpolated Cell Mapping. Physica D, 28, 401-408.
• [17] WATKINS, C. J. C. H. and DAYAN, P. (1992) Technical note: Q-learning. Machine Learning, 8 (1), 279-292.
• [18] WILHELMUS, J. A. (1994) Cell Mapping methods: modifications and extensions. PhD thesis, Eindhoven University of Technology.
• [19] ZUFIRIA, P. J. and GUTTALU, R. S. (1993) The adjoining cell mapping and its recursive unraveling, Part I: Description of adaptive and recursive algorithms. Nonlinear Dynamics, 4 (3), 207-226.
• [20] ZUFIRIA, P. J. and MART´INEZ-MAR´IN, T. (2003) Improved Optimal Control Methods based upon the Adjoining Cell Mapping Technique. Journal of Optimization Theory and Applications, 118 (3), 657-680.