Skip to content
conftrace_

batch size optimization

3 papers

Explore in graph

Co-occurring keywords · papers mentioning both

inference energy (1) dynamic batch size (1) long-context generation (1) latency-throughput tradeoff (1) sparse kv cache (1) gradient direction (1) transformer training (1) draft model (1) speculative decoding (1) prompt optimization (1)

Papers

MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding ICLR 2025 Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models NAACL 2025 Dynamically Adjusting Transformer Batch Size by Monitoring Gradient Direction Change ACL 2020