conftrace_

data deduplication

14 papers

Explore in graph

Co-occurring keywords · papers mentioning both

large language model (3) language model (3) data curation (2) record linkage (2) data cleaning (2) training data (2) language identification (2) sampling weight (1) verbatim copying (1) statistical filtration (1)

Papers

XLL: Cross-Layer Logging for Data Deduplication in Consensus-Based Storage NSDI 2026 ZipLLM: Efficient LLM Storage via Model-Aware Synergistic Data Deduplication and Compression NSDI 2026 Quality over Quantity: Boosting Data Efficiency Through Ensembled Multimodal Data Curation AAAI 2025 Back to the Stats: Rescuing Low Resource Neural Machine Translation with Statistical Methods EMNLP 2024 BPID: A Benchmark for Personal Identity Deduplication EMNLP 2024 Enhancing Data Quality through Simple De-duplication: Navigating Responsible Computational Social Science Research EMNLP 2024 LinkTransformer: A Unified Package for Record Linkage with Transformer Language Models ACL 2024 SCIR-MT’s Submission for WMT24 General Machine Translation Task EMNLP 2024 SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training ACL 2024 Validating and Exploring Large Geographic Corpora COLING 2024 The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data Only NeurIPS 2023 Deduplicating Training Data Makes Language Models Better ACL 2022 Deduplicating Training Data Mitigates Privacy Risks in Language Models ICML 2022 Semi-supervised clustering for de-duplication AISTATS 2019