Exploring the energy landscape of RBMs: reciprocal space insights into bosons, hierarchical learning and symmetry breaking
Deep generative models have become ubiquitous due to their ability to learn and sample from complex distributions. Despite the proliferation of various frameworks, the relationships among these models remain largely unexplored, a gap that hinders the development of a unified theory of AI learning. I...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IOP Publishing
2025-01-01
|
| Series: | Machine Learning: Science and Technology |
| Subjects: | |
| Online Access: | https://doi.org/10.1088/2632-2153/adf521 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849235782342017024 |
|---|---|
| author | J Quetzalcóatl Toledo-Marin Anindita Maiti Geoffrey C Fox Roger G Melko |
| author_facet | J Quetzalcóatl Toledo-Marin Anindita Maiti Geoffrey C Fox Roger G Melko |
| author_sort | J Quetzalcóatl Toledo-Marin |
| collection | DOAJ |
| description | Deep generative models have become ubiquitous due to their ability to learn and sample from complex distributions. Despite the proliferation of various frameworks, the relationships among these models remain largely unexplored, a gap that hinders the development of a unified theory of AI learning. In this work, we address two central challenges: clarifying the connections between different deep generative models and deepening our understanding of their learning mechanisms. We focus on Restricted Boltzmann Machines (RBMs), a class of generative models known for their universal approximation capabilities for discrete distributions. By introducing a reciprocal space formulation for RBMs, we reveal a connection between these models, diffusion processes, and systems of coupled bosons. Our analysis shows that at initialization, the RBM operates at a saddle point, where the local curvature is determined by the singular values of the weight matrix, whose distribution follows the Marc̆enko-Pastur law and exhibits rotational symmetry. During training, this rotational symmetry is broken due to hierarchical learning, where different degrees of freedom progressively capture features at multiple levels of abstraction. This leads to a symmetry breaking in the energy landscape, reminiscent of Landau’s theory. This symmetry breaking in the energy landscape is characterized by the singular values and the weight matrix eigenvector matrix. We derive the corresponding free energy in a mean-field approximation. We show that in the limit of infinite size RBM, the reciprocal variables are Gaussian distributed. Our findings indicate that in this regime, there will be some modes for which the diffusion process will not converge to the Boltzmann distribution. To illustrate our results, we trained replicas of RBMs with different hidden layer sizes using the MNIST dataset. Our findings not only bridge the gap between disparate generative frameworks but also shed light on the fundamental processes underpinning learning in deep generative models. |
| format | Article |
| id | doaj-art-5cfc386b90814b1ba82e57fdbb55a91b |
| institution | Kabale University |
| issn | 2632-2153 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IOP Publishing |
| record_format | Article |
| series | Machine Learning: Science and Technology |
| spelling | doaj-art-5cfc386b90814b1ba82e57fdbb55a91b2025-08-20T04:02:41ZengIOP PublishingMachine Learning: Science and Technology2632-21532025-01-016303503010.1088/2632-2153/adf521Exploring the energy landscape of RBMs: reciprocal space insights into bosons, hierarchical learning and symmetry breakingJ Quetzalcóatl Toledo-Marin0https://orcid.org/0000-0001-6212-1033Anindita Maiti1https://orcid.org/0000-0002-4712-6626Geoffrey C Fox2Roger G Melko3TRIUMF , Vancouver, BC V6T 2A3, Canada; Perimeter Institute for Theoretical Physics , Waterloo, Ontario N2L 2Y5, CanadaPerimeter Institute for Theoretical Physics , Waterloo, Ontario N2L 2Y5, CanadaUniversity of Virginia , Computer Science and Biocomplexity Institute, 994 Research Park Blvd, Charlottesville, VA 22911, United States of AmericaPerimeter Institute for Theoretical Physics , Waterloo, Ontario N2L 2Y5, Canada; Department of Physics and Astronomy, University of Waterloo , Waterloo, Ontario N2L 3G1, CanadaDeep generative models have become ubiquitous due to their ability to learn and sample from complex distributions. Despite the proliferation of various frameworks, the relationships among these models remain largely unexplored, a gap that hinders the development of a unified theory of AI learning. In this work, we address two central challenges: clarifying the connections between different deep generative models and deepening our understanding of their learning mechanisms. We focus on Restricted Boltzmann Machines (RBMs), a class of generative models known for their universal approximation capabilities for discrete distributions. By introducing a reciprocal space formulation for RBMs, we reveal a connection between these models, diffusion processes, and systems of coupled bosons. Our analysis shows that at initialization, the RBM operates at a saddle point, where the local curvature is determined by the singular values of the weight matrix, whose distribution follows the Marc̆enko-Pastur law and exhibits rotational symmetry. During training, this rotational symmetry is broken due to hierarchical learning, where different degrees of freedom progressively capture features at multiple levels of abstraction. This leads to a symmetry breaking in the energy landscape, reminiscent of Landau’s theory. This symmetry breaking in the energy landscape is characterized by the singular values and the weight matrix eigenvector matrix. We derive the corresponding free energy in a mean-field approximation. We show that in the limit of infinite size RBM, the reciprocal variables are Gaussian distributed. Our findings indicate that in this regime, there will be some modes for which the diffusion process will not converge to the Boltzmann distribution. To illustrate our results, we trained replicas of RBMs with different hidden layer sizes using the MNIST dataset. Our findings not only bridge the gap between disparate generative frameworks but also shed light on the fundamental processes underpinning learning in deep generative models.https://doi.org/10.1088/2632-2153/adf521RBMsgenerative modelsrestricted Boltzmann machinebosonssymmetry breakingdiffusion process |
| spellingShingle | J Quetzalcóatl Toledo-Marin Anindita Maiti Geoffrey C Fox Roger G Melko Exploring the energy landscape of RBMs: reciprocal space insights into bosons, hierarchical learning and symmetry breaking Machine Learning: Science and Technology RBMs generative models restricted Boltzmann machine bosons symmetry breaking diffusion process |
| title | Exploring the energy landscape of RBMs: reciprocal space insights into bosons, hierarchical learning and symmetry breaking |
| title_full | Exploring the energy landscape of RBMs: reciprocal space insights into bosons, hierarchical learning and symmetry breaking |
| title_fullStr | Exploring the energy landscape of RBMs: reciprocal space insights into bosons, hierarchical learning and symmetry breaking |
| title_full_unstemmed | Exploring the energy landscape of RBMs: reciprocal space insights into bosons, hierarchical learning and symmetry breaking |
| title_short | Exploring the energy landscape of RBMs: reciprocal space insights into bosons, hierarchical learning and symmetry breaking |
| title_sort | exploring the energy landscape of rbms reciprocal space insights into bosons hierarchical learning and symmetry breaking |
| topic | RBMs generative models restricted Boltzmann machine bosons symmetry breaking diffusion process |
| url | https://doi.org/10.1088/2632-2153/adf521 |
| work_keys_str_mv | AT jquetzalcoatltoledomarin exploringtheenergylandscapeofrbmsreciprocalspaceinsightsintobosonshierarchicallearningandsymmetrybreaking AT aninditamaiti exploringtheenergylandscapeofrbmsreciprocalspaceinsightsintobosonshierarchicallearningandsymmetrybreaking AT geoffreycfox exploringtheenergylandscapeofrbmsreciprocalspaceinsightsintobosonshierarchicallearningandsymmetrybreaking AT rogergmelko exploringtheenergylandscapeofrbmsreciprocalspaceinsightsintobosonshierarchicallearningandsymmetrybreaking |