Test error of interpolating two-layer networks in the feature-learning regime
52No. 581 in all physics · No. 11 in Statistical and Nonlinear Physics
Two-layer network $f(x) = \sum_{j=1..m} a_j \sigma(w_j . x)$ trained by gradient descent from small initialization to zero training loss on n samples $y = f_{*}(x) + \mathrm{noise}$ of variance $\sigma_n^2$, $x \sim \mathrm{N}(0, I_d)$ with $I_d$ the d x d identity, in the limit $n, d \to \infty$ with $\psi = n/d$ fixed and $m >> n$. For targets $f_{*}(x) = g(U x)$ with $U$ an r x d matrix and r fixed, compute the limiting test mean-squared error on noisy labels $E(\psi, \sigma_n^2, g)$ at fixed $\psi$, and classify the interpolation by the $\psi \to \infty$ limit of $E$: benign if $E \to \sigma_n^2$, tempered if $E \to c\,\sigma_n^2$ with $1 < c < \infty$, catastrophic if $E$ diverges. The answer is an asymptotic formula, proved or confirmed numerically to high precision.
What would settle it
An exact asymptotic formula for the test error of gradient-trained interpolating two-layer networks in the feature-learning regime, matched by large-scale simulation.
Status in the literature
Unverified note: it names recent work and was partly written without a live literature search; it awaits a check by an editor.
Exact answers exist for linear regression (Bartlett, Long, Lugosi and Tsigler 2020) and random-feature models (Mei and Montanari 2022), a benign, tempered, catastrophic taxonomy was proposed by Mallinar et al. 2022, and partial feature-learning results exist for classification and low-dimensional settings (Frei, Chatterji and Bartlett 2022; Kornowski, Yehudai and Shamir 2023; Joshi, Vardi and Srebro 2023); no asymptotic test-error formula for gradient-trained interpolating networks in the proportional regime exists (2026).
No solutions, ideas or subproblems yet.
See also
- Related Predicting neural scaling-law exponents from properties of the data
- Related Sample complexity of gradient training for multi-index targets
Details, signature and history
- Kind
- Well-posed question
- Answer
- Value
- Posed
- 2016
- Topic review
- L. Zdeborova and F. Krzakala, Statistical physics of inference: thresholds and algorithms, Advances in Physics 65, 2016. Link
- Record
- Revision 2 · notation repaired by the platform · compiled by AI, not yet reviewed by an editor
- Batch approval
- Approved for the batch by Victor Galitski (ORCID 0000-0001-5532-1096) · approval record · bundle This approves the batch, not each problem (what it means).
- Earlier revisions
- revision 1 bundle
Proof
- Fingerprint
- ad88fe5a9c92…
- Public log
- Entry 2,090, in checkpoint 2,123
- In the log by
- Block 969,698, 2026-10-03
This text was in the PubPhys log by that date. The date says nothing about when the problem was first posed.
Comments (0)
Comments are not records: they are not signed, stamped or logged. How comments work
No comments yet.
Sign in to comment.