subword tokenization

False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models EMNLP 2025 From Characters to Tokens: Dynamic Grouping with Hierarchical BPE EMNLP 2025 From Tokens to Words: On the Inner Lexicon of LLMs ICLR 2025 Functional Lexicon in Subword Tokenization NAACL 2025 Jamo-Level Subword Tokenization in Low-Resource Korean Machine Translation NAACL 2025 Large Vocabulary Size Improves Large Language Models ACL 2025 MoVoC: Morphology-Aware Subword Construction for Ge’ez Script Languages EMNLP 2025 R-BPE: Improving BPE-Tokenizers with Token Reuse EMNLP 2025 Retrofitting Large Language Models with Dynamic Tokenization ACL 2025 Splintering Nonconcatenative Languages for Better Tokenization ACL 2025 The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages IJCNLP 2025 TriEmbed: Bridge the Gap between Text and Token Indices with Embedding Reparameterization ACL 2025 Unlike “Likely”, “Unlike” is Unlikely: BPE-based Segmentation hurts Morphological Derivations in LLMs COLING 2025 Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inference NAACL 2025 When Every Token Counts: Optimal Segmentation for Low-Resource Language Models COLING 2025 An Analysis of BPE Vocabulary Trimming in Neural Machine Translation NAACL 2024 BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training EMNLP 2024 BPE-knockout: Pruning Pre-existing BPE Tokenisers with Backwards-compatible Morphological Semi-supervision NAACL 2024 byteLLM@LT-EDI-2024: Homophobia/Transphobia Detection in Social Media Comments - Custom Subword Tokenization with Subword2Vec and BiLSTM EACL 2024 byteSizedLLM@DravidianLangTech 2024: Fake News Detection in Dravidian Languages - Unleashing the Power of Custom Subword Tokenization with Subword2Vec and BiLSTM EACL 2024 Finding the Optimal Byte-Pair Encoding Merge Operations for Neural Machine Translation in a Low-Resource Setting EMNLP 2024 Greed is All You Need: An Evaluation of Tokenizer Inference Methods ACL 2024 How Important Is Tokenization in French Medical Masked Language Models? COLING 2024 How to Compute the Probability of a Word EMNLP 2024 Integrating Multi-scale Contextualized Information for Byte-based Neural Machine Translation ACL 2024