Regularized Semiparametric Neural Networks

• Autorzy: Enrico Capobianco

Warianty tytułu

Regularyzowane semiparametryczne sieci neuronowe

• Języki publikacji: EN

Abstrakty

W artykule dokonano przeglądu technik regularyzacji, które zastosowano do uczenia i estymacji sztucznych sieci neuronowych. Istotę tych technik zademonstrowano na przykładzie takich sieci, których struktura funkcjonalna jest tylko częściowo znana lub w ogóle nieznana. Na przykład, gdy funkcje aktywacji są nieznane, wtedy konstruowane są tzw. sieci nieparametryczne. Jeśli zaś struktura jest częściowo znana, są to sieci nazwane w artykule semiparametrycznymi. Wśród sugerowanych metod estymacji są metody największej wiarygodności z karą, uogólnione modelowanie addytywne, a także modelowanie nieliniowe. (abstrakt oryginalny)

EN

In this paper, regularization techniques are reviewed and applied to artiiial neural networks for learning or estimation purposes. The importance of these methodologies is emphasized when we deal with networks presenting a functional structure which is completely or partially unknown, as in the case of the presence of hidden layer activation functions of unspecified form. Under these last circumstances we have built a nonparametric network; and althgough the architecture at hand has both parametric and nonparametric components, we call the network semiparametric, as in the statistical language. Interesting cases can thus be investigated; for instance, when the activation functions are specified only up to some regularity conditions, i.e., smoothness, and allowed to be time-varying and spatially adaptable. Among the suggested estimation methods here addressed, local and penalized likelihood, additive generalized linear and nonlinear modelling are considered, given their relevance in dealing with the curse of dimensionality. (original abstract)

Słowa kluczowe

PL

Sieci neuronowe; Sztuczne sieci neuronowe (SSN); Metody estymacji

EN

Neural networks; Artificial neural networks (ANN); Estimation methods

• Czasopismo: Badania Operacyjne i Decyzje
• Rocznik: 1999
• Numer: nr 1
• Strony: 21-33

Twórcy

autor

Enrico Capobianco

• Parallel Distributed Processing Research Group at Stanford University

Bibliografia

• [1] BAEK J., WEHRLY T.E., Kernel Estimation for additive models under dependence, Stochastic Processes and their Applications, 47, 1993, 95-112.
• [2] BICKEL P., KLAASSEN C.A.J., RITOV Y., WELLNER J.A., Efficient and Adaptive Estimation of Semiparametric Models, Baltimore-London, The John Hopkins University Press, 1993.
• [3] FRIEDMAN J.H., STUETZLE W., Projection Pursuit Regression, Journal of the American Statistical Association, 76, 1981, 817-823.
• [4] GEMAN S., BIENENSTOCK E., DOURSAT R., Neural Networks and the bias/variance dilemma, Neural Computation, 4, 1992, 1-58.
• [5] GEMAN S., HWANG C.R., Nonparametric maximum likelihood estimation by the method of sieves, Annals of Statistics, 10, 1982, 401-414.
• [6] GIROSI F., JONES M., POGGIO T., Regularized Theory and Neural Networks Architectures, Neural Computation, 7, 1995, 219-269.
• [7] GREEN P.J., Penalized likelihood for generalized semi-parametric regression models, International Statistics Review, 55, 1987, 245-259.
• [8] GREEN P.J., SILVERMAN B.W., Nonparametric Regression and Generalized Linear Models, London, Chapman & Hall, 1994.
• [9] GRENANDER U., Abstract Inference, New York, Wiley, 1981.
• [10] GU C., RKPACK and its applications: fitting smoothing spline models, Proceedings of Statistical Computing Section, American Statistical Association, 1989, 42-51.
• [11] HASTIE T., TIBSHIRANI R., Generalized Additive Models, London, Chapman & Hall, 1990.
• [12] JOKINEN P.A., A nonlinear network model for continuous learning, Neurocomputing, 3, 1991, 157-176.
• [13] MAO J., JAIN A.K., Regularization techniques in Artificial Neural Networks, WCNN '93, INNS Press, 1993.
• [14] MCCULLAGH R., NELDER J.A., Generalized Linear Models, London, Chapman & Hall, 1989.
• [15] RIPLEY B.D., Statistical Aspects of Neural Networks, [in:] Networks and Chaos: statistical and probabilistic aspects, London, eds. O.E. Barndorff-Nielsen, J.L. Jenssen and W.J. Kendall, Chapman & Hall, 1993, 40-123.
• [16] ROOSEN C., HASTIE T., Automatic smoothing spline projection pursuit, Journal of Computational and Graphical Statistics, 3, 1994, 235-248.
• [17] RUMELHART D.E., HINTON G.E., WILLIAMS R.J., Learning internal representations by error propagation, In D.E. Rumelhart and J.L. McClelland eds.: Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Cambridge: MIT Press, 1, 1986, 318-362.
• [18] SILVERMAN B.W., Density estimation for statistics and data analysis, London, Chapman & Hall, United Kingdom, 1986.
• [19] SPALL J.C., CRISTION J.A., Nonlinear adaptive control using neural networks: estimation with a smoothed form of simultaneous perturbation gradient approximation perturbation gradient approximation, Statistica Sinica, 4, 1994, 1-27.
• [20] SPECHT D.F., Probabilistic Neural Networks, Neural Networks, 3, 1990, 109-118.
• [21] STANISWALIS J., The Kernel estimate of a regression function in likehood-based models, Journal of the American Statistical Association, 84, 1989, 276-283.
• [22] STANISWALIS J., YANDELL B., Locally Adaptive Smoothing Spline, Journal of Statistical Computing and Simulation, 43, 1992, 45-53.
• [23] STONE CH.J., Additive regression and other nonparametric models, The Annals of Statistics, vol. 13, nr 2, 1985, 689-705.
• [24] WAHBA G., Spline Functions for Observational Data, Philadelphia, CBMS-NSF Regional Conference, Series, SIAM, 1990.
• [25] WHITE H., Artificial Neural Networks. Approximation and learning theory, Blackwell Publishers, Oxford (UK) and Cambridge (USA), 1992.