conftrace_

subword tokenization

131 papers

Explore in graph

Co-occurring keywords

neural machine translation (2310) byte pair encoding (60) language model (4599) morphological analysis (343) byte-pair encoding (53) low-resource language (2273) machine translation (2499) large language model (13587) cross-lingual transfer (1477) named entity recognition (2813)

Papers

Corpus-Dependent Subcharacter Encoding via HMM-Guided Code Assignment ACL 2026
AdaptBPE: From General Purpose to Specialized Tokenizers EACL 2026
When Multilingual Evaluation Assumptions Fail: Tokenization Effects Across Scripts EACL 2026
TokDrift: When LLM Speaks in Subwords but Code Speaks in Grammar ACL 2026
Modelling the Morphology of Verbal Paradigms: A Case Study in the Tokenization of Turkish and Hebrew EACL 2026
Contextual morphologically-guided tokenization for Latin encoder models EACL 2026
Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models EACL 2026
Tokenization Cost, Retention, and Orthography Robustness for Ladin and Italian Varieties EACL 2026
ReBPE: Iteratively Improving the Internal Structure of a Structured Tokeniser by Mining its Internal Structure EACL 2026
MUTANT: A Recipe for Multilingual Tokenizer Design ACL 2026
How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them ACL 2026
Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features EACL 2026
Tokenization and Morphological Fidelity in Uralic NLP: A Cross-Lingual Evaluation EACL 2026
Morpheme Matters: Morpheme-Based Subword Tokenization for Korean Language Models EACL 2026
Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inference NAACL 2025
When Every Token Counts: Optimal Segmentation for Low-Resource Language Models COLING 2025
Annotating and Inferring Compositional Structures in Numeral Systems Across Languages ACL 2025
The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages IJCNLP 2025
Byte Pair Encoding Is All You Need For Automatic Bengali Speech Recognition IJCNLP 2025
Segmentation Beyond Defaults: Asymmetrical Byte Pair Encoding for Optimal Machine Translation Performance IJCNLP 2025
R-BPE: Improving BPE-Tokenizers with Token Reuse EMNLP 2025
Adapting Large Language Models for Character-based Augmentative and Alternative Communication EMNLP 2025
Splintering Nonconcatenative Languages for Better Tokenization ACL 2025
TriEmbed: Bridge the Gap between Text and Token Indices with Embedding Reparameterization ACL 2025
False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models EMNLP 2025