conftrace_

rényi efficiency

2 papers

Explore in graph

Co-occurring keywords · papers mentioning both

language model tokenization (1) grapheme cluster tokenizer (1) devanagari script (1) tokenizer evaluation (1) byte pair encoding (1) downstream performance (1) text representation (1) machine translation (1)

Papers

Comparative Analysis of the Intrinsic Metrics for Tokenizers and their effect on Downstream Tasks for Hindi and Marathi ACL 2026 Two Counterexamples to Tokenization and the Noiseless Channel COLING 2024