Conditional mean embedding and optimal feature selection via positive definite kernels

• Autorzy: Jorgensen Palle E.T. , Song Myung-Sin , Tian James

Identyfikatory

DOI

10.7494/OpMath.2024.44.1.79

• Języki publikacji: EN

Abstrakty

EN

Motivated by applications, we consider new operator-theoretic approaches to conditional mean embedding (CME). Our present results combine a spectral analysis-based optimization scheme with the use of kernels, stochastic processes, and constructive learning algorithms. For initially given non-linear data, we consider optimization-based feature selections. This entails the use of convex sets of kernels in a construction of optimal feature selection via regression algorithms from learning models. Thus, with initial inputs of training data (for a suitable learning algorithm), each choice of a kernel K in turn yields a variety of Hilbert spaces and realizations of features. A novel aspect of our work is the inclusion of a secondary optimization process over a specified convex set of positive definite kernels, resulting in the determination of “optimal” feature representations.

Słowa kluczowe

EN

positive-definite kernels; reproducing kernel Hilbert space; stochastic processes; frames; machine learning; embedding problem; optimization

• Wydawca: AGH University of Science and Technology Press
• Czasopismo: Opuscula Mathematica
• Rocznik: 2024
• Tom: Vol. 44, no. 1
• Strony: 79--103
• Opis fizyczny: Bibliogr. 30 poz., tab., wykr.

Twórcy

autor

Jorgensen Palle E.T.

• Department of Mathematics, The University of Iowa, Iowa City, IA 52242–1419, U.S.A.

autor

Song Myung-Sin

• Department of Mathematics and Statistics, Southern Illinois University Edwardsville, Edwardsville, IL 62026, U.S.A.

autor

Tian James

• Mathematical Reviews, 416–4th Street Ann Arbor, MI 48103–4816, U.S.A.

Bibliografia

• [1] N. Aronszajn, Theory of reproducing kernels, Trans. Amer. Math. Soc. 68 (1950), no. 3, 337–404.
• [2] J. Cerviño, J.A. Bazerque, M. Calvo-Fullana, A. Ribeiro, Multi-task reinforcement learning in reproducing kernel Hilbert spaces via cross-learning, IEEE Trans. Signal Process. 69 (2021), 5947–5962.
• [3] N. Dunford, J.T. Schwartz, Linear Operators. Part II, Wiley Classics Library, John Wiley & Sons, Inc., New York, 1988.
• [4] S. Grünewälder, G. Lever, L. Baldassarre, S. Patterson, A. Gretton, M. Pontil, Conditional mean embeddings as regressors, [in:] Proceedings of the 29th International Conference on International Conference on Machine Learning, Omnipress, Madison, WI, USA, 2012, ICML’12, 1803–1810.
• [5] D. He, J. Cheng, K. Xu, High-dimensional variable screening through kernel-based conditional mean dependence, J. Statist. Plann. Inference 224 (2023), 27–41.
• [6] P. Jorgensen, F. Tian, Non-commutative Analysis, World Scientific Publishing Co. Pte. Ltd., Hackensack, NJ, 2017.
• [7] P. Jorgensen, F. Tian, Decomposition of Gaussian processes, and factorization of positive definite kernels, Opuscula Math. 39 (2019), no. 4, 497–541.
• [8] P. Jorgensen, F. Tian, Realizations and factorizations of positive definite kernels, J. Theoret. Probab. 32 (2019), no. 4, 1925–1942.
• [9] P. Jorgensen, F. Tian, Sampling with positive definite kernels and an associated dichotomy, Adv. Theor. Math. Phys. 24 (2020), no. 1, 125–154.
• [10] P. Jorgensen, F. Tian, Reproducing kernels: harmonic analysis and some of their applications, Appl. Comput. Harmon. Anal. 52 (2021), 279–302.
• [11] P. Jorgensen, F. Tian, Infinite-dimensional Analysis – Operators in Hilbert Space; Stochastic Calculus via Representations, and Duality Theory, World Scientific Publishing Co. Pte. Ltd., Hackensack, NJ, 2021.
• [12] P. Jorgensen, F. Tian, Reproducing kernels and choices of associated feature spaces, in the form of L2-spaces, J. Math. Anal. Appl. 505 (2022), no. 2, 125535.
• [13] P.E.T. Jorgensen, M.-S. Song, J. Tian, Positive definite kernels, algorithms, frames, and approximations, (2021), arXiv:2104.11807.
• [14] I. Klebanov, I. Schuster, T.J. Sullivan, A rigorous theory of conditional mean embeddings, SIAM J. Math. Data Sci. 2 (2020), no. 3, 583–606.
• [15] T. Lai, Z. Zhang, Y. Wang, A kernel-based measure for conditional mean dependence, Comput. Statist. Data Anal. 160 (2021), Paper no. 107246.
• [16] T. Lai, Z. Zhang, Y. Wang, L. Kong, Testing independence of functional variables by angle covariance, J. Multivariate Anal. 182 (2021), Paper no. 104711.
• [17] Y.J. Lee, C.A. Micchelli, J. Yoon, On multivariate discrete least squares, J. Approx. Theory 211 (2016), 78–84.
• [18] G. Lever, J. Shawe-Taylor, R. Stafford, C. Szepesvári, Compressed conditional mean embeddings for model-based reinforcement learning, AAAI’16: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (2016), 1779–1787.
• [19] D.K. Lim, N.U. Rashid, J.G. Ibrahim, Model-based feature selection and clustering of RNA-seq data for unsupervised subtype discovery, Ann. Appl. Stat. 15 (2021), no. 1, 481–508.
• [20] C.-K. Lu, P. Shafto, Conditional deep Gaussian processes: Multi-fidelity kernel learning, Entropy 2021, 23(11), 1545.
• [21] E. Mehmanchi, A. Gómez, O.A. Prokopyev, Solving a class of feature selection problems via fractional 0–1 programming, Ann. Oper. Res. 303 (2021), 265–295.
• [22] C.A. Micchelli, M. Pontil, Q. Wu, D.-X. Zhou, Error bounds for learning the kernel, Anal. Appl. (Singap.) 14 (2016), no. 6, 849–868.
• [23] P. Niyogi, S. Smale, S. Weinberger, A topological view of unsupervised learning from noisy data, SIAM J. Comput. 40 (2011), no. 3, 646–663.
• [24] J. Park, K. Muandet, A measure-theoretic approach to kernel conditional mean embeddings, arXiv:2002.03689.
• [25] S. Ray Chowdhury, R. Oliveira, F. Ramos, Active learning of conditional mean embeddings via Bayesian optimisation, [in:] J. Peters, D. Sontag (eds) Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI), PMLR, 2020, volume 124 of Proceedings of Machine Learning Research, 1119–1128.
• [26] S. Smale, Y. Yao, Online learning algorithms, Found. Comput. Math. 6 (2006), 145–170.
• [27] S. Smale, D.-X. Zhou, Geometry on probability spaces, Constr. Approx. 30 (2009), 311–323.
• [28] P. Xu, Y. Wang, X. Chen, Z. Tian, COKE: communication-censored decentralized kernel learning, J. Mach. Learn. Res. 22 (2021), Paper no. 196.
• [29] Y. Zhang, Y.-C. Chen, Kernel smoothing, mean shift, and their learning theory with directional data, J. Mach. Learn. Res. 22 (2021), Paper no. 154.
• [30] P. Zhao, L. Lai, Minimax rate optimal adaptive nearest neighbor classification and regression, IEEE Trans. Inform. Theory 67 (2021), no. 5, 3155–3182.

Uwagi

Opracowanie rekordu ze środków MEiN, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2022-2023)