conftrace_

subword tokenization

131 papers

Explore in graph

Co-occurring keywords

neural machine translation (2310) byte pair encoding (60) language model (4599) morphological analysis (343) byte-pair encoding (53) low-resource language (2273) machine translation (2499) large language model (13587) cross-lingual transfer (1477) named entity recognition (2813)

Papers

Revisiting subword tokenization: A case study on affixal negation in large language models NAACL 2024
Learning Mutually Informed Representations for Characters and Subwords NAACL 2024
An Analysis of BPE Vocabulary Trimming in Neural Machine Translation NAACL 2024
Length-aware Byte Pair Encoding for Mitigating Over-segmentation in Korean Machine Translation ACL 2024
How to Compute the Probability of a Word EMNLP 2024
Tokenization Falling Short: On Subword Robustness in Large Language Models EMNLP 2024
Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs EMNLP 2024
Finding the Optimal Byte-Pair Encoding Merge Operations for Neural Machine Translation in a Low-Resource Setting EMNLP 2024
Word Boundary Information Isn’t Useful for Encoder Language Models ACL 2024
BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training EMNLP 2024
How Important Is Tokenization in French Medical Masked Language Models? COLING 2024
BPE-knockout: Pruning Pre-existing BPE Tokenisers with Backwards-compatible Morphological Semi-supervision NAACL 2024
The Effect of Model Capacity and Script Diversity on Subword Tokenization for Sorani Kurdish NAACL 2024
SpaceByte: Towards Deleting Tokenization from Large Language Modeling NIPS 2024
Optimizing the Size of Subword Vocabularies in Dialect Classification EACL 2023
CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models EMNLP 2023
Learn Your Tokens: Word-Pooled Tokenization for Language Modeling EMNLP 2023
Tokenization and the Noiseless Channel ACL 2023
Tokenization with Factorized Subword Encoding ACL 2023
Subword Segmental Machine Translation: Unifying Segmentation and Target Sentence Generation ACL 2023
PROMT Systems for WMT23 Shared General Translation Task EMNLP 2023
TranSFormer: Slow-Fast Transformer for Machine Translation ACL 2023
Entropy-guided Vocabulary Augmentation of Multilingual Language Models for Low-resource Tasks ACL 2023
The Helsinki-NLP Submissions at NADI 2023 Shared Task: Walking the Baseline EMNLP 2023
Analyzing Cognitive Plausibility of Subword Tokenization EMNLP 2023