byte pair encoding

MoVoC: Morphology-Aware Subword Construction for Ge’ez Script Languages EMNLP 2025 Multidimensional Byte Pair Encoding: Shortened Sequences for Improved Visual Data Generation ICCV 2025 R-BPE: Improving BPE-Tokenizers with Token Reuse EMNLP 2025 Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment IJCNLP 2025 Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removal AAAI 2025 Segmentation Beyond Defaults: Asymmetrical Byte Pair Encoding for Optimal Machine Translation Performance AACL 2025 Self-Vocabularizing Training for Neural Machine Translation NAACL 2025 Semitic Root Encoding: Tokenization Based on the Templatic Morphology of Semitic Languages in NMT EMNLP 2025 Tokenization is Sensitive to Language Variation ACL 2025 Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models EMNLP 2025 Transformers: Leveraging OpenNMT and Transfer Learning for Low-Resource Indian Language Translation EMNLP 2025 Unified Multimodal Understanding via Byte-Pair Visual Encoding ICCV 2025 Unlike “Likely”, “Unlike” is Unlikely: BPE-based Segmentation hurts Morphological Derivations in LLMs COLING 2025 When Every Token Counts: Optimal Segmentation for Low-Resource Language Models COLING 2025 An Analysis of BPE Vocabulary Trimming in Neural Machine Translation NAACL 2024 A Systematic Analysis of Subwords and Cross-Lingual Transfer in Multilingual Translation NAACL 2024 BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training EMNLP 2024 BPE-knockout: Pruning Pre-existing BPE Tokenisers with Backwards-compatible Morphological Semi-supervision NAACL 2024 Distributional Properties of Subword Regularization EMNLP 2024 DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomes ICLR 2024 Finding the Optimal Byte-Pair Encoding Merge Operations for Neural Machine Translation in a Low-Resource Setting EMNLP 2024 Getting the most out of your tokenizer for pre-training and domain adaptation ICML 2024 Greed is All You Need: An Evaluation of Tokenizer Inference Methods ACL 2024 How Important Is Tokenization in French Medical Masked Language Models? COLING 2024 Length-aware Byte Pair Encoding for Mitigating Over-segmentation in Korean Machine Translation ACL 2024