Neural networks for localized approximation
Authors:
C. K. Chui, Xin Li and H. N. Mhaskar
Journal:
Math. Comp. 63 (1994), 607623
MSC:
Primary 65D15; Secondary 41A15, 41A30, 92B20
MathSciNet review:
1240656
Fulltext PDF Free Access
Abstract 
References 
Similar Articles 
Additional Information
Abstract: We prove that feedforward artificial neural networks with a single hidden layer and an ideal sigmoidal response function cannot provide localized approximation in a Euclidean space of dimension higher than one. We also show that networks with two hidden layers can be designed to provide localized approximation. Since wavelet bases are most effective for local approximation, we give a discussion of the implementation of spline wavelets using multilayered networks where the response function is a sigmoidal function of order at least two.
 [1]
Andrew
R. Barron, Universal approximation bounds for superpositions of a
sigmoidal function, IEEE Trans. Inform. Theory 39
(1993), no. 3, 930–945. MR 1237720
(94h:92001), http://dx.doi.org/10.1109/18.256500
 [2]
E. K. Blum and L. K. Li, Approximation theory and neural networks, Neural Networks 4 (1991), 511515.
 [3]
S. M. Caroll and S. M. Dickinson, Construction of neural nets using the Radon transform, preprint, 1990.
 [4]
T. P. Chen, H. Chen, and R. W. Liu, A constructive proof of approximation by superposition of sigmoidal functions for neutral networks, preprint, 1990.
 [5]
Charles
K. Chui and Xin
Li, Approximation by ridge functions and neural networks with one
hidden layer, J. Approx. Theory 70 (1992),
no. 2, 131–141. MR 1172015
(93d:41018), http://dx.doi.org/10.1016/00219045(92)90081X
 [6]
Charles
K. Chui and Xin
Li, Realization of neural networks with one hidden layer,
Multivariate approximation: from CAGD to wavelets (Santiago, 1992), Ser.
Approx. Decompos., vol. 3, World Sci. Publ., River Edge, NJ, 1993,
pp. 77–89. MR
1359545
 [7]
Charles
K. Chui and Jianzhong
Wang, On compactly supported spline wavelets
and a duality principle, Trans. Amer. Math.
Soc. 330 (1992), no. 2, 903–915. MR 1076613
(92f:41020), http://dx.doi.org/10.1090/S00029947199210766133
 [8]
G.
Cybenko, Approximation by superpositions of a sigmoidal
function, Math. Control Signals Systems 2 (1989),
no. 4, 303–314. MR 1015670
(90m:41033), http://dx.doi.org/10.1007/BF02551274
 [9]
W. Dahmen and C. A. Micchelli, Some remarks on ridge functions, Approx. Theory Appl. 3 (1987), 139143.
 [10]
K. I. Funahashi, On the approximate realization of continuous mappings by neural networks, Neural Networks 2 (1989), 183192.
 [11]
K. Hornik, M. Stinchcombe, and H. White, Multilayer feedforward networks are universal approximators, Neural Networks 2 (1989), 359366.
 [12]
B. Irie and S. Miyake, Capabilities of three layered perceptrons, IEEE Internat. Conf. on Neural Networks 1 (1988), 641648.
 [13]
Y. Ito, Representation of functions by superpositions of a step or sigmoid function and their applications to neural network theory, Neural Networks 4 (1991), 385394.
 [14]
, Approximation of functions on a compact set by finite sums of a sigmoid function without scaling, Neural Networks 4 (1991), 817826.
 [15]
H.
N. Mhaskar, Approximation properties of a multilayered feedforward
artificial neural network, Adv. Comput. Math. 1
(1993), no. 1, 61–80. MR 1230251
(94h:41020), http://dx.doi.org/10.1007/BF02070821
 [16]
H.
N. Mhaskar and Charles
A. Micchelli, Approximation by superposition of sigmoidal and
radial basis functions, Adv. in Appl. Math. 13
(1992), no. 3, 350–373. MR 1176581
(93f:41030), http://dx.doi.org/10.1016/01968858(92)90016P
 [17]
T.
Poggio and F.
Girosi, Regularization algorithms for learning that are equivalent
to multilayer networks, Science 247 (1990),
no. 4945, 978–982. MR 1038271
(90k:92076), http://dx.doi.org/10.1126/science.247.4945.978
 [18]
Walter
Rudin, Functional analysis, McGrawHill Book Co., New York,
1973. McGrawHill Series in Higher Mathematics. MR 0365062
(51 #1315)
 [19]
I.
J. Schoenberg, Cardinal spline interpolation, Society for
Industrial and Applied Mathematics, Philadelphia, Pa., 1973. Conference
Board of the Mathematical Sciences Regional Conference Series in Applied
Mathematics, No. 12. MR 0420078
(54 #8095)
 [20]
M. Stinchcombe and H. White, Universal approximation using feedforward network with nonsigmoid hidden layer activation functions, Proc. Internat. Joint Conference on Neural Networks (1989), 613618, San Diego, SOS printing.
 [21]
, Approximating and learning unknown mappings using multilayer feedforward networks with bounded weights, IEEE Internat. Conf. on Neural Networks 3 (1990), III7III16.
 [1]
 A. R. Barron, Universal approximation bounds for superposition of a sigmoidal function, preprint, November 1990. MR 1237720 (94h:92001)
 [2]
 E. K. Blum and L. K. Li, Approximation theory and neural networks, Neural Networks 4 (1991), 511515.
 [3]
 S. M. Caroll and S. M. Dickinson, Construction of neural nets using the Radon transform, preprint, 1990.
 [4]
 T. P. Chen, H. Chen, and R. W. Liu, A constructive proof of approximation by superposition of sigmoidal functions for neutral networks, preprint, 1990.
 [5]
 C. K. Chui and X. Li, Approximation by ridge functions and neural networks with one hidden layer, J. Approx. Theory 70 (1992), 131141. MR 1172015 (93d:41018)
 [6]
 , Realization of neural networks with one hidden layer, Multivariate approximations: From CAGD to Wavelets (K. Jetter and F. Utreras, eds.), World Scientific Publ., Singapore, 1993, pp. 7789. MR 1359545
 [7]
 C. K. Chui and J. Z. Wang, On compactly supported spline wavelets and a duality principle, Trans. Amer. Math. Soc. 330 (1992), 903916. MR 1076613 (92f:41020)
 [8]
 G. Cybenko, Approximation by superposition of sigmoidal functions, Math. Control Signals Systems 2 (4) (1989), 303314. MR 1015670 (90m:41033)
 [9]
 W. Dahmen and C. A. Micchelli, Some remarks on ridge functions, Approx. Theory Appl. 3 (1987), 139143.
 [10]
 K. I. Funahashi, On the approximate realization of continuous mappings by neural networks, Neural Networks 2 (1989), 183192.
 [11]
 K. Hornik, M. Stinchcombe, and H. White, Multilayer feedforward networks are universal approximators, Neural Networks 2 (1989), 359366.
 [12]
 B. Irie and S. Miyake, Capabilities of three layered perceptrons, IEEE Internat. Conf. on Neural Networks 1 (1988), 641648.
 [13]
 Y. Ito, Representation of functions by superpositions of a step or sigmoid function and their applications to neural network theory, Neural Networks 4 (1991), 385394.
 [14]
 , Approximation of functions on a compact set by finite sums of a sigmoid function without scaling, Neural Networks 4 (1991), 817826.
 [15]
 H. N. Mhaskar, Approximation properties of a multilayered feedforward artificial neural network, Adv. in Comput. Math. 1 (1993), 6180. MR 1230251 (94h:41020)
 [16]
 H. N. Mhaskar and C. A. Micchelli, Approximation by superposition of a signmoidal function, Adv. in Appl. Math. 13 (1992), 350373. MR 1176581 (93f:41030)
 [17]
 T. Poggio and F. Girosi, Regularization algorithms for learning that are equivalent to multilayer networks, Science 247 (1990), 978982. MR 1038271 (90k:92076)
 [18]
 W. Rudin, Functional analysis, McGrawHill, New York, 1973. MR 0365062 (51:1315)
 [19]
 I. J. Schoenberg, Cardinal spline interpolation, CBMSNSF Conf. Series in Appl. Math. #12, SIAM, Philadelphia, PA, 1973. MR 0420078 (54:8095)
 [20]
 M. Stinchcombe and H. White, Universal approximation using feedforward network with nonsigmoid hidden layer activation functions, Proc. Internat. Joint Conference on Neural Networks (1989), 613618, San Diego, SOS printing.
 [21]
 , Approximating and learning unknown mappings using multilayer feedforward networks with bounded weights, IEEE Internat. Conf. on Neural Networks 3 (1990), III7III16.
Similar Articles
Retrieve articles in Mathematics of Computation
with MSC:
65D15,
41A15,
41A30,
92B20
Retrieve articles in all journals
with MSC:
65D15,
41A15,
41A30,
92B20
Additional Information
DOI:
http://dx.doi.org/10.1090/S00255718199412406562
PII:
S 00255718(1994)12406562
Keywords:
Neural networks,
sigmoidal functions,
spline functions,
wavelets
Article copyright:
© Copyright 1994 American Mathematical Society
