Abstract.
We derive moderate deviation principles for the overlap parameter in the Hopfield model of spin glasses and neural networks. If the inverse temperature β is different from the critical inverse temperature β c =1 and the number of patterns M(N) satisfies M(N)/N → 0, the overlap parameter multiplied by Nγ, 1/2 < γ < 1, obeys a moderate deviation principle with speed N1−2γ and a quadratic rate function (i.e. the Gaussian limit for γ = 1/2 remains visible on the moderate deviation scale). At the critical temperature we need to multiply the overlap parameter by Nγ, 1/4 < γ < 1. If then M(N) satisfies (M(N)6 log N ∧ M(N)2N4γ log N)/N → 0, the rescaled overlap parameter obeys a moderate deviation principle with speed N1−4γ and a rate function that is basically a fourth power. The random term occurring in the Central Limit theorem for the overlap at β c = 1 is no longer present on a moderate deviation scale. If the scaling is even closer to N1/4, e.g. if we multiply the overlap parameter by N1/4 log log N the moderate deviation principle breaks down. The case of variable temperature converging to one is also considered. If β N converges to β c fast enough, i.e. faster than the non-Gaussian rate function persists, whereas for β N converging to one slower than the moderate deviations principle is given by the Gaussian rate. At the borderline the moderate deviation rate function is the one at criticality plus an additional Gaussian term.
Article PDF
Similar content being viewed by others
References
Bovier, A., Gayrard, V.: An almost sure large deviation principle for the Hopfield model. Ann. Probab. 24 (3), 1444–1475 (1996) MR98g:60055
Bovier, A., Gayrard, V.: An almost sure central limit theorem for the Hopfield model. Markov Process. Related Fields 3 (2), 151–173 (1997) MR98j:60044
Bovier, A., Gayrard, V.: The retrieval phase of the Hopfield model: a rigorous analysis of the overlap distribution. Probab. Theory Related Fields 107 (1), 61–98 (1997) MR98b:82041
Bovier, A., Gayrard, V.: Hopfield models as generalized random mean field models. Mathematical aspects of spin glasses and neural networks, Progr. Probab. 41, Birkhäuser Boston, Boston, MA, 1998, pp. 3–89. MR99a:82049
Bovier, A., Gayrard, V., Picco, P.: Gibbs states of the Hopfield model in the regime of perfect memory. Probab. Theory Related Fields 100 (3), 329–363 (1994) MR96c:60115
Bovier, A., Mason, D.M.: Extreme value behavior in the Hopfield model. Ann. Appl. Probab. 11 (1), 91–120 (2001) MR2002c:82049
Bovier, A., Picco, P. (eds.): Mathematical aspects of spin glasses and neural networks. Progress in Probability, 41, Birkhäuser Boston Inc., Boston, MA, 1998 MR98i:82002
Brascamp, H.J., Lieb, E.H.: On extensions of the Brunn-Minkowski and Prékopa-Leindler theorems, including inequalities for log concave functions, and with an application to the diffusion equation. J. Functional Analysis 22 (4), 366–389 (1976) MR56 #8774
Dembo, A., Zeitouni, O.: Large Deviations Techniques and Applications, Springer, New York, 1998
Deuschel, J.-D., Giacomin, G., Ioffe, D.: Large deviations and concentration properties for ∇φ interface models. Probab. Theory Related Fields 117 (1), 49–111 (2000) MR2001e:60206
Eichelsbacher, P., Löwe, M.: Moderate deviations for a class of mean-field models. Markov Process. Related Fields 10 (2), 345–366 (2004)
Ellis, R.S., Newman, C.M.: Limit theorems for sums of dependent random variables occurring in statistical mechanics. Z. Wahrsch. Verw. Gebiete 44 (2), 117–139 (1978) MR58 20113
Gentz, B.: An almost sure central limit theorem for the overlap parameters in the Hopfield model. Stochastic Process. Appl. 62 (2), 243–262 (1996) MR97e:60038
Gentz, B.: A central limit theorem for the overlap in the Hopfield model. Ann. Probab. 24 (4), 1809–1841 (1996) MR97j:60038
Gentz, B.: On the central limit theorem for the overlap in the Hopfield model. Mathematical aspects of spin glasses and neural networks, Birkhäuser Boston, Boston, MA, 1998, pp. 115–149 MR98m:82059
Gentz, B., Löwe, M.: Fluctuations in the Hopfield model at the critical temperature. Markov Process. Related Fields 5 (4), 423–449 (1999) MR2001k:60140
Gentz, B., Löwe, M.: The fluctuations of the overlap in the Hopfield model with finitely many patterns at the critical temperature. Probab. Theory Related Fields 115 (3), 357–381 (1999) MR2000m:60111
Hopfield, J.J.: Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. U.S.A. 79, 2554–2558 (1982)
Komlós, J., Paturi, R.: Convergence results in an associative memory model. Neural Networks 1, 239–250 (1988)
Külske, C.: Metastates in disordered mean-field models: random field and Hopfield models. J. Statist. Phys. 88 (5-6), 1257–1293 (1997) MR2000a:82038
Ledoux, M., Talagrand, M.: Probability in Banach Spaces, Springer-Verlag, Berlin, 1991
Loukianova, D.: Lower bounds on the restitution error in the Hopfield model. Probab. Theory Related Fields 107, 161–176 (1997)
Löwe, M.: On the storage capacity of Hopfield models with correlated patterns. Ann. Appl. Probab. 8 (4), 1216–1250 (1998) MR99k:82059
McEliece, R.J., Posner, E.C., Rodemich, E.R., Venkatesh, S.S.: The capacity of the Hopfield associative memory. IEEE Trans. Inform. Theory 33, 461–482 (1987)
Newman, C.M.: Memory capacity in neural network models: Rigorous lower bounds. Neural Networks 1, 223–238 (1988)
Pastur, L., Shcherbina, M., Tirozzi, B.: On the replica symmetric equations for the Hopfield model. J. Math. Phys. 40 (8), 3930–3947 (1999) MR2001a:82045
Pastur, L.A., Figotin, A.L.: Exactly soluble model of a spin glass. Sov. J. Low Temp. Phys. 3 (6), 378–383 (1977)
Pastur, L.A., Figotin, A.L.: On the theory of disordered spin systems. Theor. Math. Phys. 35, 403–414 (1977)
Sherrington, D., Kirkpatrick, S.: Solvable model of a spin–glass. Phys. Rev. Lett. 35, 1796–1972 (1975) MR31 4107
Talagrand, M.: Rigorous results for the Hopfield model with many patterns. Probab. Theory Related Fields 110 (2), 177–276 (1998) MR98k:82099
Talagrand, M.: Rigorous low-temperature results for the mean field p-spins interaction model. Probab. Theory Related Fields 117 (3), 303–360 (2000) MR2001f:82037
Talagrand, M.: On the Hopfield model at the critical temperature. Probab. Theory Related Fields 121 (2), 237–268 (2001) MR1 865 487
Author information
Authors and Affiliations
Corresponding author
Additional information
Research supported by the Volkswagen-Stiftung (RiP-program at Oberwolfach, Germany).
Mathematics Subject Classification (2000): 60F10 (primary), 60K35, 82B44, 82D30 (secondary)
Rights and permissions
About this article
Cite this article
Eichelsbacher, P., Löwe, M. Moderate Deviations for the overlap parameter in the Hopfield model. Probab. Theory Relat. Fields 130, 441–472 (2004). https://doi.org/10.1007/s00440-004-0349-8
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00440-004-0349-8