subword tokenization

Learning Mutually Informed Representations for Characters and Subwords NAACL 2024 Length-aware Byte Pair Encoding for Mitigating Over-segmentation in Korean Machine Translation ACL 2024 MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization NeurIPS 2024 On the Effect of (Near) Duplicate Subwords in Language Modelling ACL 2024 Revisiting subword tokenization: A case study on affixal negation in large language models NAACL 2024 Semantics or spelling? Probing contextual word embeddings with orthographic noise ACL 2024 SpaceByte: Towards Deleting Tokenization from Large Language Modeling NeurIPS 2024 The Effect of Model Capacity and Script Diversity on Subword Tokenization for Sorani Kurdish NAACL 2024 Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs EMNLP 2024 Tokenization Falling Short: On Subword Robustness in Large Language Models EMNLP 2024 Token-length Bias in Minimal-pair Paradigm Datasets COLING 2024 Where are we Still Split on Tokenization? EACL 2024 Word Boundary Information Isn’t Useful for Encoder Language Models ACL 2024 Analyzing Cognitive Plausibility of Subword Tokenization EMNLP 2023 Byte-Level Grammatical Error Correction Using Synthetic and Curated Corpora ACL 2023 CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models EMNLP 2023 Entropy-guided Vocabulary Augmentation of Multilingual Language Models for Low-resource Tasks ACL 2023 Estimating Numbers without Regression EACL 2023 FramingFreaks at SemEval-2023 Task 3: Detecting the Category and the Framing of Texts as Subword Units with Traditional Machine Learning SemEval 2023 GUIT-NLP’s Submission to Shared Task: Low Resource Indic Language Translation EMNLP 2023 How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese ACL 2023 Incorporating Context into Subword Vocabularies EACL 2023 Learn Your Tokens: Word-Pooled Tokenization for Language Modeling EMNLP 2023 Local Byte Fusion for Neural Machine Translation ACL 2023 Optimizing the Size of Subword Vocabularies in Dialect Classification EACL 2023