conftrace_

second-order optimizer

5 papers

Explore in graph

Co-occurring keywords · papers mentioning both

stochastic optimization (2) preconditioner matrix (1) kronecker-factored curvature (1) curvature approximation (1) hessian estimation (1) gradient preconditioning (1) network training (1) language model pretraining (1) fisher matrix (1) nonlinear least square (1)

Papers

4-bit Shampoo for Memory-Efficient Network Training NeurIPS 2024 Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training ICLR 2024 KrADagrad: Kronecker approximation-domination gradient preconditioned stochastic optimization UAI 2023 Gradient Descent on Neurons and its Link to Approximate Second-order Optimization ICML 2022 Theseus: A Library for Differentiable Nonlinear Optimization NeurIPS 2022