Predicting neural scaling-law exponents from properties of the data
72No. 97 in all physics · No. 1 in Statistical and Nonlinear Physics
The error of large neural networks falls as a power of model size and data size, with exponents that are measured but not predicted. The task is to compute those exponents from features of the training data for networks that reshape their internal features while learning.
Status in the literature
Unverified note: it names recent work and was partly written without a live literature search; it awaits a check by an editor.
Linear random-feature and kernel models derive the exponents from power-law spectra (Maloney, Roberts and Sully 2022; Bahri et al. 2024; Paquette et al. 2024); solvable feature-learning models predict modified exponents (Bordelon, Atanasov and Pehlevan, arXiv:2409.17858; Ren, Nichani, Wu and Lee, arXiv:2504.19983), and synthetic-grammar theory ties data scaling to the decay of token correlations (Cagnetta and Wyart, arXiv:2406.00048); no theory yet predicts measured $\beta_N$ and $\beta_D$ for deep networks trained on natural data (2026).
No solutions, ideas or subproblems yet.
See also
- Related Origin of the 1/n spectrum of cortical population responses
- Related Test error of interpolating two-layer networks in the feature-learning regime
Details, signature and history
- Kind
- Phenomenon to explain
- Answer
- Mechanism
- Topic review
- L. Zdeborova and F. Krzakala, Statistical physics of inference: thresholds and algorithms, Advances in Physics 65, 2016. Link
- Record
- Revision 2 · notation repaired by the platform · compiled by AI, not yet reviewed by an editor
- Batch approval
- Approved for the batch by Victor Galitski (ORCID 0000-0001-5532-1096) · approval record · bundle This approves the batch, not each problem (what it means).
- AI systems
- AI systems used to prepare this seed revision, as listed in its record: ai:gpt-5.5 (how the seed revision was made)
- Earlier revisions
- revision 1 bundle
Proof
- Fingerprint
- ce8a1f574705…
- Public log
- Entry 2,094, in checkpoint 2,123
- In the log by
- Block 969,698, 2026-10-03
This text was in the PubPhys log by that date. The date says nothing about when the problem was first posed.
Comments (0)
Comments are not records: they are not signed, stamped or logged. How comments work
No comments yet.
Sign in to comment.