Tight Lower Bound on Differential Entropy for Mixed Gaussian Distributions

• Autorzy: Marconi Abdelrahman , Elghandour Ahmed H. , Elbayoumy Ashraf D. , Abdelaziz Amr

Identyfikatory

DOI

10.26636/jtit.2024.2.1444

• Języki publikacji: EN

Abstrakty

EN

In this paper, a tight lower bound for the differential entropy of the Gaussian mixture model is presented. First, the probability model of mixed Gaussian distribution that is created by mixing both discrete and continuous random variables is investigated in order to represent symmetric bimodal Gaussian distribution using the hyperbolic cosine function, on which a tighter upper bound is set. Then, this tight upper bound is used to derive a tight lower bound for the differential entropy of the Gaussian mixture model introduced. The proposed lower bound allows to maintain its tightness over the entire range of the model's parameters and shows more tightness when compared with other bounds that lose their tightness over certain parameter ranges. The presented results are then extended to introduce a more general tight lower bound for asymmetric bimodal Gaussian distribution, in which the two modes have a symmetric mean but differ in terms of their weights.

Słowa kluczowe

EN

differential entropy; lower bound; mixture random variable; multimodal Gaussian

• Wydawca: Instytut Łączności - Państwowy Instytut Badawczy
• Czasopismo: Journal of Telecommunications and Information Technology
• Rocznik: 2024
• Tom: nr 2
• Strony: 23--31
• Opis fizyczny: Bibliogr. 19 poz., rys.

Twórcy

autor

Marconi Abdelrahman

• Communications Department Military Technical College, Cairo, Egypt

• https://orcid.org/0009-0006-5181-4946

autor

Elghandour Ahmed H.

• Communications Department Military Technical College, Cairo, Egypt

• https://orcid.org/0000-0002-5250-4847

autor

Elbayoumy Ashraf D.

• Communications Department Military Technical College, Cairo, Egypt

• https://orcid.org/0000-0001-6356-5285

autor

Abdelaziz Amr

• Communications Department Military Technical College, Cairo, Egypt

• https://orcid.org/0000-0003-3709-3653

Bibliografia

• [1] J. Jose et al., "An Image Quality Enhancement Scheme Employing Adolescent Identity Search Algorithm in the NSST Domain for Multimodal Medical Image Fusion", Biomedical Signal Processing and Control, vol. 66, art. no. 102480, 2021.
• [2] J. Xu and Z. Cai, "Gaussian Mixture Deep Dynamic Latent Variable Model with Application to Soft Sensing for Multi-mode Industrial Processes", Applied Soft Computing, vol. 114, art. no. 108092, 2022.
• [3] J. Skvara and I. Nezbeda, "Thermodynamics and Structure of Supercooled Water. II.", Journal of Molecular Liquids, vol. 367, art. no. 120508, 2022.
• [4] K. Moshksar and A.K. Khandani, "Arbitrarily Tight Bounds on Differential Entropy of Gaussian Mixtures", IEEE Transactions on Information Theory, vol. 62, no. 6, pp. 3340-3354, 2016.
• [5] M. Wen, X. Cheng, and L. Yang, Index Modulation for 5G Wireless Communications, Springer, 154 p., 2017.
• [6] V. Bhatia and B. Mulgrew, "Non-parametric Likelihood Based Channel Estimator for Gaussian Mixture Noise", Signal Processing, vol. 87, no. 11, pp. 2569-2586, 2007.
• [7] D. Peel and G.J. McLachlan, Finite Mixture Models, John Wiley & Sons, 427 p., 2000.
• [8] G.J. McLachlan, S.X. Lee, and S.I. Rathnayake, "Finite Mixture Models", Annual Review of Statistics and its Application, vol. 6, pp. 335-378, 2019.
• [9] T.M. Cover and J.A. Thomas, Elements of Information Theory, John Wiley & Sons, 748 p., 2005.
• [10] M.F. Huber, T. Bailey, H. Durrant-Whyte, and U.D. Hanebeck, "On Entropy Approximation for Gaussian Mixture Random Vectors", 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, Seoul, South Korea, 2008.
• [11] J. Melbourne, S. Talukdar, S. Bhaban, and M.V. Salapaka, "Error Bounds on a Mixed Entropy Inequality", 2018 IEEE International Symposium on Information Theory (ISIT), Vail, USA, 2018.
• [12] A. Kolchinsky and B.D. Tracey, "Estimating Mixture Entropy with Pairwise Distances", Entropy, vol. 19, no. 7, art. no. 361, 2017.
• [13] L.V. Michalowicz, J.M. Nichols, and F. Bucholtz, "Calculation of Differential Entropy for a Mixed Gaussian Distribution", Entropy, vol. 10, no. 3, pp. 200-206, 2008.
• [14] J. Sandor, "On Some Inequalities for the Identric, Logarithmic and Related Means", Aequationes Mathematicae, vol. 40, pp. 261-270, 1990.
• [15] K.B. Stolarsky, "Holder Means, Lehmer Means, andx-1 log coshx", Journal of Mathematical Analysis and Applications, vol. 202, no. 3, pp. 810-818, 1996.
• [16] W.J. Weltner, P. Schuster, and K. Weltner, Mathematics for Engineers and Scientists, Stanley Thornes, 520 p., 1986 (ISBN: 9780859501200).
• [17] M. Raginsky, "On the Information Capacity of Gaussian Channels under Small Peak Power Constraints", 46th Annual Allerton Conference on Communication, Control, and Computing, Monticello, USA, 2008.
• [18] I.S. Gradshteyn and I.M. Ryzhik, Table of Integrals, Series, and Products, Elsevier, 1160 p., 1980.
• [19] J. Melbourne, S. Talukdar, S. Bhaban, M. Madiman, and M.V. Salapaka, "The Differential Entropy of Mixtures: New Bounds and Applications", Entropy, vol. 68, no. 4, pp. 2123-2146, 2022.