conftrace_

gradient stabilization

2 papers

Explore in graph

Co-occurring keywords · papers mentioning both

svd parameterization (1) attention computation (1) long sequence (1) exploding gradient (1) vanishing gradient (1) linear transformer (1) recurrent neural network (1) singular value decomposition (1) language modeling (1) attention mechanism (1)

Papers

The Devil in Linear Transformer EMNLP 2022 Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization ICML 2018