conftrace_

gradient underflow

2 papers

Explore in graph

Co-occurring keywords · papers mentioning both

half precision floating point (1) learning rate scaling (1) floating point (1) parameterization (1) adam-atan2 (1) epsilon parameter (1) per-layer learning rate (1) gradient scaling (1) mixed precision training (1) numerical stability (1)

Papers

Scaling Exponents Across Parameterizations and Optimizers ICML 2024 Reducing Underflow in Mixed Precision Training by Gradient Scaling IJCAI 2020