Am I Audible Now? Dissecting Prosody in Neural Audio Codecs
Abstract
AbstractThe functional organization of prosody within the Residual Vector Quantization (RVQ) codebooks of neural audio codecs remains an open question. Prior work relies on correlational probing, showing that pitch features can be decoded from frozen codebook embeddings. Decodability, however, does not imply causal necessity: a codebook may encode pitch information the decoder ignores, while another may be essential despite containing little probe-recoverable signal. We address this gap through causal output-side ablation: for each codebook, we zero or mean-replace its discrete tokens, re-decode the waveform, and measure degradation in fundamental frequency F0 (CREPE), energy (frame-wise RMS), and intelligibility (STOI) against the unablated reconstruction. Experiments span 200 LibriSpeech test-clean utterances under clean, 15, 5, and 0 dB SNR with MUSAN noise across multiple neural audio codecs. Across different architectures, we find a consistent two-tier causal structure: a codec-specific subset of codebooks is reconstruction-critical, while the remaining intelligibility-preserving codebooks exhibit a stable pitch-contribution hierarchy that persists under additive noise. These findings provide new evidence for how neural audio codecs causally organize prosodic information.