conftrace_

tokenization efficiency

5 papers

Explore in graph

Co-occurring keywords · papers mentioning both

large language model (2) low-resource language (2) machine translation (2) subword fertility (1) vocabulary pruning (1) multi-word tokenization (1) tokenizer adaptation (1) token optimization (1) vocabulary extension (1) byte pair encoding (1)

Papers

Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pretrained Models EACL 2026 An Information-Theoretic Approach to Reducing Fertility in LLMs for Manipuri Machine Translation IJCNLP 2025 Confounding Factors in Relating Model Performance to Morphology EMNLP 2025 Cost-Performance Optimization for Processing Low-Resource Language Tasks Using Commercial LLMs EMNLP 2024 Multi-word Tokenization for Sequence Compression EMNLP 2023