RAMP: Regime-Aware Merge-Prune Token Compression for ASR-Oriented Audio-Language Models
Abstract
AbstractASR-oriented audio-language models (ALMs) encode speech into long audio token sequences for LLM decoding, substantially increasing the inference cost of transcription. This motivates token compression methods that reduce audio token sequence length while preserving the acoustic evidence needed for transcription. However, existing compression methods often inherit vision-language criteria that are poorly matched to temporally ordered audio representations, or require attention-map access outside the standard FlashAttention path. Through a multi-axis geometric analysis, we show that ALM audio representations exhibit two distinct representation regimes: a locally redundant encoder regime and a dispersed post-projector regime. Motivated by this geometric shift, we propose **RAMP** (**R**egime-**A**ware **M**erge-**P**rune), a training-free method for FlashAttention-compatible audio token compression that applies bipartite L2 merging in the encoder and centroid-distance-guided pruning after the projector. Across seven ASR benchmarks and two primary ALMs, RAMP achieves the lowest average WER multiplier among the evaluated training-free compression baselines at every evaluated retention ratio. At the most aggressive setting (r=0.4) on Audio-Flamingo-3, RAMP attains a 2.36× average WER multiplier, compared with 4.26× for the best non-RAMP baseline, while reducing time-to-first-token (TTFT) by 24.8% over Vanilla.