Bounds for Validation

• Autorzy: Jaworski W.
• Języki publikacji: EN

Abstrakty

EN

In this paper we derive the bounds for Validation (known also as Hold-Out Estimate and Train-and-Test Method). We present the best possible bound in the case of 0-1 valued loss function. We also provide the tables where the least sample size is calculated that is necessary for obtaining the bound for a given estimation rate and reliability of estimation. For an arbitrary bounded loss function we present the optimal bound approximation with any given accuracy.

Słowa kluczowe

EN

computational learning theory; model assessment; hold-out-estimate; train-and-test; validation; Bernoulli sums

• Wydawca: IOS Press
• Czasopismo: Fundamenta Informaticae
• Rocznik: 2006
• Tom: Vol. 70, nr 3
• Strony: 261--275
• Opis fizyczny: tab., bibliogr. 10 poz.

Twórcy

autor

Jaworski W.

• Faculty of Mathematics,Computer Science and Mechanics, Warsaw University, ul. Banacha 2, 02-097 Warszawa, Poland,

Bibliografia

• [1] Cucker, F., Smale, S.: On the mathematical foundations of learning, Bull. Am. Math. Soc., New Ser., 39(1), 2002, 1-49.
• [2] Duda, R. O., Hart, P. E., Stork, D. G.: Pattern classification. 2nd ed, Chichester: Wiley-Interscience, 2001.
• [3] Fukunaga, K., Hayes, R.: Effects of sample size in classifier design, IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(8), 1989, 873-885.
• [4] Fukunaga, K., Hayes, R.: Estimation of classifier performance, IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(10), 1989, 1087-1101.
• [5] Guyon, I., Makhoul, J., Schwartz, R., Vapnik, V.: What size test set gives good error rate estimates?, IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(1), 1998, 52-64.
• [6] Hastie, T., Tibshirani, R., Friedman, J.: The elements of statistical learning. Data mining, inference, and prediction, Springer Series in Statistics. New York, NY: Springer, 2001.
• [7] Hoeffding,W.: Probability inequalities for sums of bounded random variables, J. Am. Stat. Assoc., 58, 1963, 13-30.
• [8] Michie, D., Spiegelhalter, D. J., Taylor, C. C.: Machine Learning, Neural and Statistical Classification, Ellis Horwood, 1994.
• [9] Raudys, S., Jain, A.: Small sample size effects in statistical pattern recognition: recommendations for practitioners, IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(3), 1991, 252-264.
• [10] Vapnik, V. N.: Statistical learning theory, Adaptive and Learning Systems for Signal Processing, Communications, and Control. Chichester: Wiley, 1998.