Skip to content

Keep a gradient second moment when AdaBelief beta1 is 0 - #559

Open
shaneraphel wants to merge 1 commit into
jettify:masterfrom
shaneraphel:adabelief-beta1-zero
Open

shaneraphel wants to merge 1 commit into
jettify:masterfrom
shaneraphel:adabelief-beta1-zero

Conversation

@shaneraphel

@shaneraphel shaneraphel commented Sep 25, 2026 •

Copy link
Copy Markdown

Summary

With beta1 = 0 the first moment is set to the current gradient, so the belief residual g - m is 0 and the second moment stays at eps. The step is lr * g / sqrt(eps). This port's default eps is 0.001, which hides it, but the paper's eps is 1e-16 (and that is what AdaBelief(..., eps=1e-16) accepts). A scalar quadratic x^2 with learning rate 1e-2 and that eps reached -inf on step 10. The same overflow is juntang-zhuang/Adabelief-Optimizer#67; the fix there is the same idea, kept in that repository's latest package.

Only the degenerate case changes. When beta1 == 0 the residual is g, so the second moment tracks g^2 the way Adam does at beta1 = 0. For any positive beta1 the residual is still g - m after the first-moment update. One step at beta1 = 0.9 still stores (1 - beta2) (g - m)^2 plus the eps this implementation adds into the second-moment state.

Test plan

  • pytest tests/test_adabelief_beta1.py
  • Positive beta1 matches the residual variance, including the added eps.
  • Thirty steps of x^2 at beta1 = 0, eps = 1e-16, learning rate 1e-2 stay finite and the parameter moves from 0.5 toward 0.

This change was produced with assistance from an AI coding tool. I reproduced the overflow at the paper's eps, left the positive-beta1 residual alone, and ran the tests locally.

With beta1 = 0 the first moment equals the current gradient, so
(g - m) is 0 and the second moment stays at eps. The step is then
lr * g / sqrt(eps). At eps 1e-16 a scalar quadratic reached -inf
on step 10.

Adam with beta1 = 0 still tracks g^2. Use that residual only when
beta1 is 0. A positive beta1 still stores (1 - beta2) (g - m)^2
plus the eps that is added into the second-moment state.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant