Teoria informacji a statystyka matematyczna

• Autorzy: Inglot T.

Identyfikatory

DOI

10.14708/ma.v42i1.521

Warianty tytułu

EN

Information Theory in the Mathematical Statistics

• Języki publikacji: PL

Abstrakty

PL

W niniejszym artykule przedstawiony jest zarys teorii informacji z probabilistycznego i statystycznego punktu widzenia. Ten nurt teorii informacji rozwijał się intensywnie w ostatnich dziesięcioleciach. Wpłynął tez w znaczący sposób na rozwój metod statystycznych. Celem artykułu jest wprowadzenie czytelnika w przystępny sposób w podana powyżej tematykę, dostarczenie mu pewnych intuicji i przybliżenie specyfiki podejścia teorio-informacyjnego w statystyce matematycznej.

EN

In the paper we present an outline of the information theory from the probabilistic and statistical point of view. Such a direction of the information theory has been intensively developed in recent decades and significantly influenced a progress in the statistical methodology. The aim of the article is to introduce the reader into these problems, provide some intuitions and acquaint with a specific information-theoretic approach to the mathematical statistics. The first part of the paper is devoted to brief and easy of approach introduction to the main notions of the information theory like entropy, relative entropy (Kullback- Leibler distance), information projection and Fisher information as well as presentation of their most important properties including de Bruijn’s identity, Fisher information inequalities and entropy power inequalities. In the short second part we give applications of the notions and results from the first part to limit theorems of the probability theory such as the asymptotic equipartition property, the convergence of empirical measures in the entropy distance, large deviation principle with emphasis to Sanov theorem, the convergence of distributions of homogeneous Markov chains in the entropy distance and the central limit theorem. The main, last part of the article shows some most significant and important applications of the information theory to the mathematical statistics. We discuss connections of the maximum likelihood estimators with the information projections and the notion of sufficient statistic from the information-theoretic point of view. The problems of source coding, channel capacity and an amount of information provided by statistical experiments are presented in a statistical framework. Some attention is paid to the expansion of Clarke and Barron and its corollaries e.g. in density estimation. Next, applications of the information theory to hypothesis testing is discussed. We give the classical Stein’s Lemma and its generalization to testing composite hypothesis obtained by Bahadur and show their connections with the asymptotic efficiency of statistical tests. Finally, we briefly mention the problem of information criteria in a model seletion including the most popular two-stage minimal description length criterion of Rissanen. The enclosed literature is limited only to papers and books which are referred to in the paper.

Słowa kluczowe

PL

entropia; odległość Kullbacka–Leiblera; informacja Fishera; zbieżność entropii; lemat Steina; model statystyczny; kodowanie źródła; estymacja gęstości

EN

entropy; Kullback-Leibler distance; Fisher information; entropy convergence; statistical model; source coding; Stein’s Lemma; density estimation

• Wydawca: Polskie Towarzystwo Matematyczne
• Czasopismo: Mathematica Applicanda
• Rocznik: 2014
• Tom: Vol. 42, No. 1
• Strony: 115--174
• Opis fizyczny: Bibliogr. 75 poz.

Twórcy

autor

Inglot T.

• Politechnika Wrocławska Instytut Matematyki i Informatyki Wybrzeze Wyspianskiego 27, 50-370 Wrocław

Bibliografia

• [1] Hirotugu Akaike. Markovian representation of stochastic processes and its application to the analysis of autoregressive moving average processes. Ann. Inst. Statist. Math., 26:363–387, 1974. ISSN 0020-3157.
• [2] Hirotugu Akaike. A new look at the statistical model identification. IEEE Trans. Automatic Control, AC-19:716–723, 1974. ISSN 0018-9286. doi: 10.1109/TAC.1974.1100705. System identification and time-series analysis.
• [3] Hirotugu Akaike. Stochastic theory of minimal realization. IEEE Trans. Automatic Control, AC-19:667–674, 1974. ISSN 0018-9286. System identification and time-series analysis.
• [4] R. R. Bahadur. An optimal property of the likelihood ratio statistic. In Proc. Fifth Berkeley Sympos. Math. Statist. and Probability (Berkeley, Calif., 1965/66), Vol. I: Statistics, pages 13–26. Univ. California Press, Berkeley, Calif., 1967.
• [5] R. R. Bahadur. Rates of convergence of estimates and test statistics. Ann. Math. Statist., 38:303–324, 1967. ISSN 0003-4851.
• [6] Ole Barndorff-Nielsen. Information and exponential families in statistical theory. John Wiley & Sons, Ltd., Chichester, 1978. ISBN 0-471- 99545-2. Wiley Series in Probability and Mathematical Statistics.
• [7] Andrew R. Barron. The strong ergodic theorem for densities: generalized Shannon-McMillan-Breiman theorem. Ann. Probab., 13(4):1292–1303, 1985. ISSN 0091-1798. doi: 10. 1214/aop.1176992813. URL http://links.jstor.org/sici?sici= 0091-1798(198511)13:4<1292:TSETFD>2.0.CO;2-J&origin=MSN.
• [8] Andrew R. Barron. Entropy and the central limit theorem. Ann. Probab., 14(1):336–342, 1986. ISSN 0091-1798. doi: 10. 1214/aop/1176982632. URL http://links.jstor.org/sici?sici= 0091-1798(198601)14:1<336:EATCLT>2.0.CO;2-5&origin=MSN.
• [9] Andrew R. Barron. Uniformly powerful goodness of fit tests. Ann. Statist., 17(1):107–124, 1989. ISSN 0090-5364. doi: 10.1214/aos/ 1176347005. URL http://dx.doi.org/10.1214/aos/1176347005.
• [10] Andrew R. Barron. Complexity regularization with application to artificial neural networks. In Nonparametric functional estimation and related topics (Spetses, 1990), volume 335 of NATO Adv. Sci. Inst. Ser. C Math. Phys. Sci., pages 561–576. Kluwer Acad. Publ., Dordrecht, 1991.
• [11] Andrew R. Barron and Thomas M. Cover. A bound on the financial value of information. IEEE Trans. Inform. Theory, 34(5, part 1):1097– 1100, 1988. ISSN 0018-9448. doi: 10.1109/18.21241. URL http: //dx.doi.org/10.1109/18.21241.
• [12] Andrew R. Barron and Thomas M. Cover. Correction to: “Minimum complexity density estimation”. IEEE Trans. Inform. Theory, 37(6): 1738, 1991. ISSN 0018-9448.
• [13] Andrew R. Barron and Thomas M. Cover. Minimum complexity density estimation. IEEE Trans. Inform. Theory, 37(4):1034–1054, 1991. ISSN 0018-9448. doi: 10.1109/18.86996. URL http://dx.doi.org/ 10.1109/18.86996.
• [14] Andrew R. Barron and Chyong-Hwa Sheu. Approximation of density functions by sequences of exponential families. Ann. Statist., 19(3):1347– 1369, 1991. ISSN 0090-5364. doi: 10.1214/aos/1176348252. URL http://dx.doi.org/10.1214/aos/1176348252.
• [15] Andrew R. Barron and Chyong-Hwa Sheu. Correction: “Approximation of density functions by sequences of exponential families”. Ann. Statist., 19(4):2284, 1991. ISSN 0090-5364. doi: 10.1214/aos/1176348405. URL http://dx.doi.org/10.1214/aos/1176348405.
• [16] Andrew Roger Barron. LOGICALLY SMOOTH DENSITY ESTIMATION (INFERENCE, DATA COMPRESSION). ProQuest LLC, Ann Arbor, MI, 1985. URL http://gateway.proquest.com/openurl? url_ver=Z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx: dissertation&res_dat=xri:pqdiss&rft_dat=xri:pqdiss:8602452. Thesis (Ph.D.)–Stanford University.
• [17] Jarosław Bartoszewicz. Wykłady ze statystyki matematycznej. Wydawnictwo Naukowe PWN, Warsaw, second edition, 1996. ISBN 83- 01-12095-9.
• [18] Nelson M. Blachman. The convolution inequality for entropy powers. IEEE Trans. Information Theory, IT-11:267–271, 1965. ISSN 0018-9448. doi: 10.1109/TIT.1965.1053768. 168 Teoria informacji a statystyka matematyczna
• [19] S. G. Bobkov, G. P. Chistyakov, and F. G¨otze. Convergence to stable laws in relative entropy. J. Theoret. Probab., 26(3):803–818, 2013. ISSN 0894-9840. doi: 10.1007/s10959-011-0377-0. URL http://dx.doi. org/10.1007/s10959-011-0377-0.
• [20] Sergey G. Bobkov. Entropic approach to E. Rio’s central limit theorem for W2 transport distance. Statist. Probab. Lett., 83(7):1644–1648, 2013. ISSN 0167-7152. doi: 10.1016/j.spl.2013.03.020. URL http://dx. doi.org/10.1016/j.spl.2013.03.020.
• [21] Sergey G. Bobkov and Mokshay M. Madiman. On the problem of reversibility of the entropy power inequality. In Limit theorems in probability, statistics and number theory, volume 42 of Springer Proc. Math. Stat., pages 61–74. Springer, Heidelberg, 2013. doi: 10.1007/978-3-642-36068-8_4. URL http://dx.doi.org/10.1007/ 978-3-642-36068-8_4.
• [22] Sergey G. Bobkov, Gennadiy P. Chistyakov, and Friedrich G¨otze. Rate of convergence and Edgeworth-type expansion in the entropic central limit theorem. Ann. Probab., 41(4):2479–2512, 2013. ISSN 0091-1798. doi: 10.1214/12-AOP780. URL http://dx.doi.org/10.1214/12-AOP780.
• [23] Vladimir Bobkov and Yavdat Il0yasov. Asymptotic behaviour of branches for ground states of elliptic systems. Electron. J. Differential Equations, pages No. 212, 21, 2013. ISSN 1072-6691.
• [24] Leo Breiman. A counterexample to a theorem of Kolmogorov. Ann. Math. Statist., 28:811–814, 1957. ISSN 0003-4851.
• [25] Leo Breiman. The individual ergodic theorem of information theory. Ann. Math. Statist., 28:809–811, 1957. ISSN 0003-4851. doi: 10.1214/ aoms/1177706899.
• [26] Leo Breiman. On transient Markov chains with application to the uniqueness problem for Markov processes. Ann. Math. Statist., 28:499–503, 1957. ISSN 0003-4851.
• [27] Qencov, N. N. Statistiqeskie rexa_wie pravila i optimalnye vyvody. 1972.
• [28] Herman Chernoff. A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. Ann. Math. Statistics, 23: 493–507, 1952. ISSN 0003-4851. doi: 10.1214/aoms/1177729330.
• [29] A. J. Chintschin, D. K. Faddejew, A. N. Kolmogoroff, A. R´enyi, and J. Balatoni. Arbeiten zur Informationstheorie, I. Mathematische Forschungsberichte, IV. VEB Deutsche Verlag der Wissenschaften, Berlin, 1957.
• [30] Bertrand S. Clarke and Andrew R. Barron. Information-theoretic asymptotics of Bayes methods. IEEE Trans. Inform. Theory, 36(3): 453–471, 1990. ISSN 0018-9448. doi: 10.1109/18.54897. URL http://dx.doi.org/10.1109/18.54897.
• [31] Bertrand S. Clarke and Andrew R. Barron. Jeffreys’ prior is asymptotically least favorable under entropy risk. J. Statist. Plann. Inference, 41(1):37–60, 1994. ISSN 0378-3758. doi: 10.1016/0378-3758(94) 90153-8. URL http://dx.doi.org/10.1016/0378-3758(94)90153-8.
• [32] Thomas M. Cover and Joy A. Thomas. Elements of information theory. 2nd ed. Hoboken, NJ: John Wiley & Sons, 2nd ed. edition, 2006. ISBN 0-471-24195-4/hbk; 0-471-74881-1/ebook. doi: 10.1002/047174882X.
• [33] I. Csiszar. Information-type measures of difference of probability distributions and indirect observations. Stud. Sci. Math. Hung., 2:299–318, 1967. ISSN 0081-6906; 1588-2896/e.
• [34] I. Csiszar. Information-type measures of divergence of probability distributions. II. Magyar Tud. Akad., Mat. Fiz. Tud. Oszt. K¨ozl., 17:267–290, 1967. ISSN 0200-3031; 0025-035X.
• [35] I. Csiszar. I-divergence geometry of probability distributions and minimization problems. Ann. Probab., 3:146–158, 1975. ISSN 0091-1798. doi: 10.1214/aop/1176996454.
• [36] Imre Csisz´ar and Frantiˇsek Mat´uˇs. Information projections revisited. IEEE Trans. Inf. Theory, 49(6):1474–1490, 2003. ISSN 0018-9448. doi: 10.1109/TIT.2003.810633.
• [37] Amir Dembo and Ofer Zeitouni. Large deviations and applications. In Handbook of stochastic analysis and applications, volume 163 of Statist. Textbooks Monogr., pages 361–416. Dekker, New York, 2002.
• [38] Ł Debowski. Information Theory and Statistics. Institute of Computer Science Polish Academy of Science, Warsaw, 2013.
• [39] D. K. Faddeev. On the concept of entropy of a finite probabilistic scheme. Uspehi Mat. Nauk (N.S.), 11(1(67)):227–231, 1956. ISSN 0042-1316.
• [40] Robert M. Gray. Entropy and information theory. Springer-Verlag, New York, 1990. ISBN 0-387-97371-0. doi: 10.1007/978-1-4757-3982-4. URL http://dx.doi.org/10.1007/978-1-4757-3982-4.
• [41] P. D. Gr¨unwald. The Minimum Description Length Principle. MIT Press, 2007. 170 Teoria informacji a statystyka matematyczna
• [42] Mark H. Hansen and Bin Yu. Model selection and the principle of minimum description length. J. Amer. Statist. Assoc., 96(454):746– 774, 2001. ISSN 0162-1459. doi: 10.1198/016214501753168398. URL http://dx.doi.org/10.1198/016214501753168398.
• [43] David Haussler. A general minimax result for relative entropy. IEEE Trans. Inform. Theory, 43(4):1276–1280, 1997. ISSN 0018-9448. doi: 10.1109/18.605594. URL http://dx.doi.org/10.1109/18.605594.
• [44] D.A. Huffman. A method for the construction of minimum-redundancy codes. Kibern. Sb. 3, 79-87 (1952)., 1952.
• [45] Tadeusz Inglot and Teresa Ledwina. Asymptotic optimality of datadriven Neyman’s tests for uniformity. Ann. Statist., 24(5):1982–2019, 1996. ISSN 0090-5364. doi: 10.1214/aos/1069362306. URL http: //dx.doi.org/10.1214/aos/1069362306.
• [46] Harold Jeffreys. An invariant form for the prior probability in estimation problems. Proc. Roy. Soc. London. Ser. A., 186:453–461, 1946. ISSN 0962-8444. doi: 10.1098/rspa.1946.0056.
• [47] Petr Jizba and Toshihico Arimitsu. The world according to R´enyi: thermodynamics of multifractal systems. Ann. Physics, 312(1):17–59, 2004. ISSN 0003-4916. doi: 10.1016/j.aop.2004.01.002. URL http://dx.doi.org/10.1016/j.aop.2004.01.002.
• [48] Oliver Johnson. Information theory and the central limit theorem. Imperial College Press, London, 2004. ISBN 1-86094-473-6. doi: 10.1142/ 9781860945373. URL http://dx.doi.org/10.1142/9781860945373.
• [49] Oliver Johnson and Andrew Barron. Fisher information inequalities and the central limit theorem. Probab. Theory Related Fields, 129(3):391– 409, 2004. ISSN 0178-8051. doi: 10.1007/s00440-004-0344-0. URL http://dx.doi.org/10.1007/s00440-004-0344-0.
• [50] Oliver Johnson and Yurii Suhov. Entropy and random vectors. J. Statist. Phys., 104(1-2):145–165, 2001. ISSN 0022-4715. doi: 10.1023/A:1010353526846. URL http://dx.doi.org/10.1023/A: 1010353526846.
• [51] David G. Kendall. Information theory and the limit-theorem for Markov chains and processes with a countable infinity of states. Ann. Inst. Statist. Math., 15:137–143, 1963. ISSN 0020-3157. doi: 10.1007/ BF02865910.
• [52] Solomon Kullback. Information theory and statistics. John Wiley and Sons, Inc., New York; Chapman and Hall, Ltd., London, 1959.
• [53] E. L. Lehmann and George Casella. Theory of point estimation. Springer Texts in Statistics. Springer-Verlag, New York, second edition, 1998. ISBN 0-387-98502-6.
• [54] Mokshay Madiman and Andrew Barron. Generalized entropy power inequalities and monotonicity properties of information. IEEE Trans. Inform. Theory, 53(7):2317–2329, 2007. ISSN 0018-9448. doi: 10.1109/TIT.2007.899484. URL http://dx.doi.org/10.1109/TIT. 2007.899484.
• [55] Brockway McMillan. The basic theorems of information theory. Ann. Math. Statistics, 24:196–219, 1953. ISSN 0003-4851. doi: 10.1214/ aoms/1177729028.
• [56] J. Neyman. Su un teorema concernente le cosiddette statistiche sufficienti. G. Ist. Ital. Attuari, 6:320–334, 1935.
• [57] Yakov Nikitin. Asymptotic efficiency of nonparametric tests. Cambridge University Press, Cambridge, 1995. ISBN 0-521-47029-3. doi: 10.1017/CBO9780511530081. URL http://dx.doi.org/10.1017/ CBO9780511530081.
• [58] Steven Orey. On the Shannon-Perez-Moy theorem. In Particle systems, random media and large deviations (Brunswick, Maine, 1984), volume 41 of Contemp. Math., pages 319–327. Amer. Math. Soc., Providence, RI, 1985. doi: 10.1090/conm/041/814721. URL http://dx. doi.org/10.1090/conm/041/814721.
• [59] M. Raghavachari. On a theorem of Bahadur on the rate of convergence of test statistics. Ann. Math. Statist., 41:1695–1699, 1970. ISSN 0003-4851. doi: 10.1214/aoms/1177696813.
• [60] A. R´enyi. Wahrscheinlichkeitsrechnung. Mit einem Anhang ¨uber Informationstheorie. Hochschulb¨ucher f¨ur Mathematik, Bd. 54. VEB Deutscher Verlag der Wissenschaften, Berlin, 1962.
• [61] Alfr´ed R´enyi. On measures of entropy and information. In Proc. 4th Berkeley Sympos. Math. Statist. and Prob., Vol. I, pages 547–561. Univ. California Press, Berkeley, Calif., 1961.
• [62] Alfr´ed R´enyi. On some basic problems of statistics from the point of view of information theory. In Proc. Fifth Berkeley Sympos. Math. Statist. and Probability (Berkeley, Calif., 1965/66), Vol. I: Statistics, pages 531–543. Univ. California Press, Berkeley, Calif., 1967.
• [63] Olivier Rioul. Information theoretic proofs of entropy power inequalities. IEEE Trans. Inform. Theory, 57(1):33–55, 2011. ISSN 0018-9448. doi: 10.1109/TIT.2010.2090193. URL http://dx.doi.org/10.1109/ TIT.2010.2090193.
• [64] J. Rissanen. Modeling by shortest data description. Automatica, 14:465– 471, 1978. ISSN 0005-1098. doi: 10.1016/0005-1098(78)90005-5.
• [65] Jorma Rissanen. A universal prior for integers and estimation by minimum description length. Ann. Statist., 11(2):416–431, 1983. ISSN 0090-5364. doi: 10.1214/aos/1176346150. URL http://dx.doi.org/ 10.1214/aos/1176346150.
• [66] Jorma Rissanen. Stochastic complexity and modeling. Ann. Statist., 14 (3):1080–1100, 1986. ISSN 0090-5364. doi: 10.1214/aos/1176350051. URL http://dx.doi.org/10.1214/aos/1176350051.
• [67] I. N. Sanov. On the probability of large deviations of random magnitudes. Mat. Sb. N. S., 42 (84):11–44, 1957.
• [68] Gideon Schwarz. Estimating the dimension of a model. Ann. Statist., 6 (2):461–464, 1978. ISSN 0090-5364. doi: 10.1214/aos/1176344136.
• [69] C. E. Shannon. A mathematical theory of communication. Bell System Tech. J., 27:379–423, 623–656, 1948. ISSN 0005-8580.
• [70] Claude E. Shannon and Warren Weaver. The mathematical theory of communication. Urbana, Ill.: University of Illinois Press. V, 117 p. (1949)., 1949.
• [71] R. Shimizu. On fisher’s ammount of information for location family. In G. P. Patil et al., editor, Statistical Distributions in Scientific Work, volume 3, pages 305–312. Reidel, 1975.
• [72] A. J. Stam. Some inequalities satisfied by the quantities of information of Fisher and Shannon. Information and Control, 2:101–112, 1959. ISSN 0890-5401. doi: 10.1016/50019-9958(59)90348-1.
• [73] Constantino Tsallis. Possible generalization of Boltzmann-Gibbs statistics. J. Statist. Phys., 52(1-2):479–487, 1988. ISSN 0022-4715. doi: 10. 1007/BF01016429. URL http://dx.doi.org/10.1007/BF01016429.
• [74] A. W. van der Vaart. Asymptotic statistics, volume 3 of Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 1998. ISBN 0-521-49603-9; 0-521-78450-6. doi: 10.1017/CBO9780511802256. URL http://dx.doi.org/10.1017/ CBO9780511802256.
• [75] Yuhong Yang and Andrew Barron. Information-theoretic determination of minimax rates of convergence. Ann. Statist., 27(5):1564–1599, 1999. ISSN 0090-5364. doi: 10.1214/aos/1017939142. URL http://dx.doi. org/10.1214/aos/1017939142.