FServe: FP8-Native W4A8KV4 Quantization and Attention Kernel Optimization for Efficient LLM Serving
Abstract
AbstractAs large language models (LLMs) scale, inference throughput is increasingly bottlenecked by memory bandwidth. While prior 4-bit quantization schemes promise memory savings, their dequantization overhead often negates these gains. We present FServe, a hardware-software co-designed framework featuring an FP8-native W4A8KV4 scheme that combines channel pre-scaling and group-wise scale relaxation for linear layers with post-RoPE key smoothing that aligns channel-wise means and ranges for attention layers. To translate these memory savings into FP8-native throughput, FServe further extends FlashAttention-3 kernels for prefill with packed-FP16 softmax instruction reordering and multi-warp asynchronous TMA fetching, and introduces a dynamic Stream-KV tile scheduler for decode phase workload imbalance. Our evaluation on Llama-3.1-8B and Qwen3-8B shows that FServe outperforms AWQ W4A16 and W4A8 baselines on accuracy despite additionally compressing the KV cache to INT4, achieves up to 1.41× TTFT speedup over BF16, and sustains effective decode bandwidth above the H100’s theoretical peak.