conftrace_

bpe tokenization

7 papers

Explore in graph

Co-occurring keywords · papers mentioning both

subword tokenization (4) byte pair encoding (2) wordpiece tokenization (1) reading time prediction (1) tokenizer inference (1) cross-lingual vocabulary transfer (1) tokenization sampling (1) distribution analysis (1) stochastic dropout (1) large language model (1)

Papers

Dictionaries to the Rescue: Cross-Lingual Vocabulary Transfer for Low-Resource Languages Using Bilingual Dictionaries ACL 2025 Distributional Properties of Subword Regularization EMNLP 2024 Greed is All You Need: An Evaluation of Tokenizer Inference Methods ACL 2024 Subwords as Skills: Tokenization for Sparse-Reward Reinforcement Learning NeurIPS 2024 Tokenization via Language Modeling: the Role of Preceding Text COLING 2024 Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship? EMNLP 2023 BPE beyond Word Boundary: How NOT to use Multi Word Expressions in Neural Machine Translation ACL 2022