Skip to content
conftrace_

gradient-based tokenization

2 papers

Explore in graph

Co-occurring keywords · papers mentioning both

subword tokenization (2) byte token (1) multilingual learning (1) multilingual language model (1) byte-level tokenization (1) adaptive tokenization (1) non-latin script (1) token-free model (1) language fairness (1) character-level transformer (1)

Papers

MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization NeurIPS 2024 Charformer: Fast Character Transformers via Gradient-based Subword Tokenization ICLR 2022