conftrace_

multilingual tokenization

9 papers

Explore in graph

Co-occurring keywords · papers mentioning both

large language model (3) tokenizer evaluation (2) byte pair encoding (2) language model (2) neural machine translation (2) cross-lingual transfer (2) scaling consistency (1) intrinsic metric (1) zipf law (1) language-agnostic representation (1)

Papers

Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet ACL 2026 TokCollate: A Comprehensive Tool for Tokenizer Evaluation and Visualization across Languages ACL 2026 TReX: Tokenizer Regression for Optimal Data Mixture EACL 2026 Beyond Literal Token Overlap: Token Alignability for Multilinguality NAACL 2025 Beyond Text Compression: Evaluating Tokenizers Across Scales ACL 2025 Semitic Root Encoding: Tokenization Based on the Templatic Morphology of Semitic Languages in NMT EMNLP 2025 An Empirical Study of Multilingual Vocabulary for Neural Machine Translation Models EMNLP 2024 Tokenizer Choice For LLM Training: Negligible or Crucial? NAACL 2024 Wine is not v i n. On the Compatibility of Tokenizations across Languages EMNLP 2021