Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Brain Translation: Multitask Sequence-to-Sequence Modeling for Intracortical Attempted Speech Decoding

AbstractDecoding language from intracortical activity is a sequence transduction problem that stress-tests modern NLP methods: inputs are high-dimensional and nonstationary, alignment to linguistic units is unknown, and training data are orders of magnitude smaller than in standard translation or speech recognition. We study attempted-speech decoding from intracortical recordings as a neural-to-language translation task and ask what contextual sequence-to-sequence modeling contributes beyond the framewise connectionist temporal classification (CTC) pipelines that dominate this domain. We introduce a multitask Transformer encoder–decoder that jointly predicts phonemes, words (via a BART-initialized decoder), and auxiliary acoustic features, together with a day-conditioned calibration module (Neural Hammer Scalpel, NHS) that combines a global affine transform with feature-wise linear modulation to handle session-level distribution shift. On the Willet et al. dataset, our model achieves state-of-the-art phoneme error rate (14.3%); direct word decoding reaches 25.6% WER, falling to 19.4% with rescoring, however still trailing multi-stage systems with heavy language-model refinement. Ablations and held-out-day experiments quantify the contributions of multitask supervision and NHS and the cost of session drift, while attention analyses reveal recurring temporal chunking that phoneme and word decoders consume differently. Together, these results position seq2seq modeling as both a competitive decoder and an interpretable probe for how language evidence is organized in neural inputs.

More from EMNLP 2026

All EMNLP 2026 papers →