conftrace_

subword tokenization

131 papers

Explore in graph

Co-occurring keywords

neural machine translation (2310) byte pair encoding (60) language model (4599) morphological analysis (343) byte-pair encoding (53) low-resource language (2273) machine translation (2499) large language model (13587) cross-lingual transfer (1477) named entity recognition (2813)

Papers

Exploring morphology-aware tokenization: A case study on Spanish language modeling EMNLP 2025
Retrofitting Large Language Models with Dynamic Tokenization ACL 2025
Annotating and Inferring Compositional Structures in Numeral Systems Across Languages ACL 2025
R-BPE: Improving BPE-Tokenizers with Token Reuse EMNLP 2025
Byte Pair Encoding Is All You Need For Automatic Bengali Speech Recognition IJCNLP 2025
From Characters to Tokens: Dynamic Grouping with Hierarchical BPE EMNLP 2025
TriEmbed: Bridge the Gap between Text and Token Indices with Embedding Reparameterization ACL 2025
Broken Words, Broken Performance: Effect of Tokenization on Performance of LLMs AACL 2025
MoVoC: Morphology-Aware Subword Construction for Ge’ez Script Languages EMNLP 2025
Dictionaries to the Rescue: Cross-Lingual Vocabulary Transfer for Low-Resource Languages Using Bilingual Dictionaries ACL 2025
Enhancing Character-Level Understanding in LLMs through Token Internal Structure Learning ACL 2025
Jamo-Level Subword Tokenization in Low-Resource Korean Machine Translation NAACL 2025
BabyLM’s First Words: Word Segmentation as a Phonological Probing Task ACL 2025
Adapting Large Language Models for Character-based Augmentative and Alternative Communication EMNLP 2025
Adversarial Tokenization ACL 2025
Byte Pair Encoding Is All You Need For Automatic Bengali Speech Recognition AACL 2025
Integrating Multi-scale Contextualized Information for Byte-based Neural Machine Translation ACL 2024
On the Effect of (Near) Duplicate Subwords in Language Modelling ACL 2024
Token-length Bias in Minimal-pair Paradigm Datasets COLING 2024
byteLLM@LT-EDI-2024: Homophobia/Transphobia Detection in Social Media Comments - Custom Subword Tokenization with Subword2Vec and BiLSTM EACL 2024
BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training EMNLP 2024
byteSizedLLM@DravidianLangTech 2024: Fake News Detection in Dravidian Languages - Unleashing the Power of Custom Subword Tokenization with Subword2Vec and BiLSTM EACL 2024
Where are we Still Split on Tokenization? EACL 2024
How to Compute the Probability of a Word EMNLP 2024
An Analysis of BPE Vocabulary Trimming in Neural Machine Translation NAACL 2024