conftrace_

tokenizer evaluation

5 papers

Explore in graph

Co-occurring keywords · papers mentioning both

multilingual tokenization (2) language model (2) scaling consistency (1) intrinsic metric (1) japanese language processing (1) zipf law (1) large language model (1) multilingual modeling (1) rényi efficiency (1) multilingual tokenizer (1)

Papers

TokCollate: A Comprehensive Tool for Tokenizer Evaluation and Visualization across Languages ACL 2026 Beyond Text Compression: Evaluating Tokenizers Across Scales ACL 2025 Regional-TinyStories: A Small Language Model Framework for Evaluating Language Learning, Tokenizers, and Datasets IJCNLP 2025 Two Counterexamples to Tokenization and the Noiseless Channel COLING 2024 How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese ACL 2023