Predicting neural scaling-law exponents from properties of the data
72No. 97 in all physics · No. 1 in Statistical and Nonlinear Physics
For a deep network trained in the feature-learning regime on natural data, the test loss follows $L($$N$, $D$$) = L_{\infty} + A N^{-\beta_N} + B D^{-\beta_D}$ over several decades, with N parameters and D training examples or tokens. Derive $\beta_N, \beta_D$ and the compute-optimal allocation exponent from measurable statistics of the data distribution (for example the power-law decay exponent of the data covariance spectrum or of the target function's spectral decomposition). An answer is a theory that predicts measured exponents on a held-out dataset and architecture within quoted errors, or a proof that no data statistic of this kind fixes them.
What would settle it
A theory whose predicted $\beta_{N}$ and $\beta_{D}$ match exponents measured on several independent datasets and architectures trained in the feature-learning regime.
Status in the literature
Unverified note: it names recent work and was partly written without a live literature search; it awaits a check by an editor.
Linear random-feature and kernel models derive the exponents from power-law spectra (Maloney, Roberts and Sully 2022; Bahri et al. 2024; Paquette et al. 2024); solvable feature-learning models predict modified exponents (Bordelon, Atanasov and Pehlevan, arXiv:2409.17858; Ren, Nichani, Wu and Lee, arXiv:2504.19983), and synthetic-grammar theory ties data scaling to the decay of token correlations (Cagnetta and Wyart, arXiv:2406.00048); no theory yet predicts measured $\beta_N$ and $\beta_D$ for deep networks trained on natural data (2026).
No solutions, ideas or subproblems yet.
See also
- Related Origin of the 1/n spectrum of cortical population responses
- Related Test error of interpolating two-layer networks in the feature-learning regime
Details, signature and history
- Kind
- Phenomenon to explain
- Answer
- Mechanism
- Topic review
- L. Zdeborova and F. Krzakala, Statistical physics of inference: thresholds and algorithms, Advances in Physics 65, 2016. Link
- Record
- Revision 2 · notation repaired by the platform · compiled by AI, not yet reviewed by an editor
- Batch approval
- Approved for the batch by Victor Galitski (ORCID 0000-0001-5532-1096) · approval record · bundle This approves the batch, not each problem (what it means).
- AI systems
- AI systems used to prepare this seed revision, as listed in its record: ai:gpt-5.5 (how the seed revision was made)
- Earlier revisions
- revision 1 bundle
Proof
- Fingerprint
- ce8a1f574705…
- Public log
- Entry 2,094, in checkpoint 2,123
- In the log by
- Block 969,698, 2026-10-03
This text was in the PubPhys log by that date. The date says nothing about when the problem was first posed.
Comments (0)
Comments are not records: they are not signed, stamped or logged. How comments work
No comments yet.
Sign in to comment.