Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Beyond Chunking: Efficient Global Pooling for Holistic Long-Document Representation

AbstractEffectively representing long documents is a persistent challenge in natural language processing, as foundational encoders are constrained by limited context windows. Prevailing methods like truncation and independent chunk aggregation can discard content or lose cross-chunk context, hindering downstream performance. To address this problem, we introduce **Spectral Attention Token Pooling (SATPool)**, a novel, encoder-agnostic module that generates a single, fixed-size vector from variable-length sequences of locally encoded tokens. SATPool operates in two stages: it first uses efficient linear attention to model global token interactions after chunk encoding, then employs **Spectral Token Compression (STC)** to compress these globally-aware token representations into a compact, context-aware vector. Across long-document classification, Retrieval-Augmented Generation (RAG), and multimodal RAG, SATPool achieves strong overall improvements and robust top-tier performance across datasets and encoder backbones. The pretrained encoder remains frozen, while the lightweight SATPool module and task-specific heads are trained for each downstream task.

More from EMNLP 2026

All EMNLP 2026 papers →