STAT On PubPhys: open In the literature: partially resolved Compiled by AI, not yet reviewed by an editor

Predicting neural scaling-law exponents from properties of the data

72

No. 97 in all physics · No. 1 in Statistical and Nonlinear Physics

A data centre aisle between rows of black server racks with blue and green status lights and overhead cables.
AI-generated illustration · About the figures

Non-expert version

For a deep network trained in the feature-learning regime on natural data, the test loss follows $L($$N$, $D$$) = L_{\infty} + A N^{-\beta_N} + B D^{-\beta_D}$ over several decades, with N parameters and D training examples or tokens. Derive $\beta_N, \beta_D$ and the compute-optimal allocation exponent from measurable statistics of the data distribution (for example the power-law decay exponent of the data covariance spectrum or of the target function's spectral decomposition). An answer is a theory that predicts measured exponents on a held-out dataset and architecture within quoted errors, or a proof that no data statistic of this kind fixes them.

What would settle it

A theory whose predicted $\beta_{N}$ and $\beta_{D}$ match exponents measured on several independent datasets and architectures trained in the feature-learning regime.

Status in the literature

Unverified note: it names recent work and was partly written without a live literature search; it awaits a check by an editor.

Linear random-feature and kernel models derive the exponents from power-law spectra (Maloney, Roberts and Sully 2022; Bahri et al. 2024; Paquette et al. 2024); solvable feature-learning models predict modified exponents (Bordelon, Atanasov and Pehlevan, arXiv:2409.17858; Ren, Nichani, Wu and Lee, arXiv:2504.19983), and synthetic-grammar theory ties data scaling to the decay of token correlations (Cagnetta and Wyart, arXiv:2406.00048); no theory yet predicts measured $\beta_N$ and $\beta_D$ for deep networks trained on natural data (2026).

No solutions, ideas or subproblems yet.

How results are checked

See also

Comments (0)

Comments are not records: they are not signed, stamped or logged. How comments work

No comments yet.

Sign in to comment.

Details, signature and history
Kind
Phenomenon to explain
Answer
Mechanism
Topic review
L. Zdeborova and F. Krzakala, Statistical physics of inference: thresholds and algorithms, Advances in Physics 65, 2016. Link
Record
Revision 2 · notation repaired by the platform · compiled by AI, not yet reviewed by an editor
Batch approval
Approved for the batch by Victor Galitski (ORCID 0000-0001-5532-1096) · approval record · bundle This approves the batch, not each problem (what it means).
AI systems
AI systems used to prepare this seed revision, as listed in its record: ai:gpt-5.5 (how the seed revision was made)
Earlier revisions
revision 1 bundle

Proof

Fingerprint
ce8a1f574705…
Public log
Entry 2,094, in checkpoint 2,123
In the log by
Block 969,698, 2026-10-03

This text was in the PubPhys log by that date. The date says nothing about when the problem was first posed.

History