Replica Symmetry Breaking in Dense Hebbian Neural Networks

Understanding the glassy nature of neural networks is pivotal both for theoretical and computational advances in Machine Learning and Theoretical Artificial Intelligence. Keeping the focus on dense associative Hebbian neural networks (i.e. Hopfield networks with polynomial interactions of even degree P> 2), the purpose of this paper is twofold: at first we develop rigorous mathematical approaches to address properly a statistical mechanical picture of the phenomenon of replica symmetry breaking (RSB) in these networks, then—deepening results stemmed via these routes—we aim to inspect the glassiness that they hide. In particular, regarding the methodology, we provide two techniques: the former (closer to mathematical physics in spirit) is an adaptation of the transport PDE to this case, while the latter (more probabilistic in its nature) is an extension of Guerra’s interpolation breakthrough. Beyond coherence among the results, either in replica symmetric and in the one-step replica symmetry breaking level of description, we prove the Gardner’s picture (heuristically achieved through the replica trick) and we identify the maximal storage capacity by a ground-state analysis in the Baldi-Venkatesh high-storage regime. In the second part of the paper we investigate the glassy structure of these networks: at difference with the replica symmetric scenario (RS), RSB actually stabilizes the spin-glass phase. We report huge differences w.r.t. the standard pairwise Hopfield limit: in particular, it is known that it is possible to express the free energy of the Hopfield neural network (and, in a cascade fashion, all its properties) as a linear combination of the free energies of a hard spin glass (i.e. the Sherrington–Kirkpatrick model) and a soft spin glass (the Gaussian or ”spherical” model). While this continues to hold also in the first step of RSB for the Hopfield model, this is no longer true when interactions are more than pairwise (whatever the level of description, RS or RSB). For dense networks solely the free energy of the hard spin glass survives. As the Sherrington–Kirkpatrick spin glass is full-RSB (i.e. Parisi theory holds for that model), while the Gaussian spin-glass is replica symmetric, these different representation theorems prove a huge diversity in the underlying glassiness of associative neural networks.