conftrace_

llm pretraining

2 papers

Explore in graph

Co-occurring keywords · papers mentioning both

model embedding (1) gpt2 (1) llama (1) adamw (1) blockwise learning rate (1) sharpness disparity (1) data diversification (1) document deduplication (1) data selection (1) web corpus (1)

Papers

The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training ICML 2025 D4: Improving LLM Pretraining via Document De-Duplication and Diversification NeurIPS 2023