Knowledge Distillation

Leveraging Entity Information for Cross-Modality Correlation Learning: The Entity-Guided Multimodal Summarization ACL 2024 LLMs-as-Instructors: Learning from Errors Toward Automating Model Improvement EMNLP 2024 MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods ICLR 2024 Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging EMNLP 2024 Mixture-of-Subspaces in Low-Rank Adaptation EMNLP 2024 NeuroComparatives: Neuro-Symbolic Distillation of Comparative Knowledge NAACL 2024 PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning EMNLP 2024 Question-guided Knowledge Graph Re-scoring and Injection for Knowledge Graph Question Answering EMNLP 2024 READ-PVLA: Recurrent Adapter with Partial Video-Language Alignment for Parameter-Efficient Transfer Learning in Low-Resource Video-Language Modeling AAAI 2024 SEEKR: Selective Attention-Guided Knowledge Retention for Continual Learning of Large Language Models EMNLP 2024 SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors NeurIPS 2024 Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarios ACL 2024 Towards the Fundamental Limits of Knowledge Transfer over Finite Domains ICLR 2024 Towards Understanding Factual Knowledge of Large Language Models ICLR 2024 Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging NeurIPS 2024 VE-KD: Vocabulary-Expansion Knowledge-Distillation for Training Smaller Domain-Specific Language Models EMNLP 2024 X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model CVPR 2024 Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method ACL 2023 A Study on Knowledge Distillation from Weak Teacher for Scaling Up Pre-trained Language Models ACL 2023 Connective Prediction for Implicit Discourse Relation Recognition via Knowledge Distillation ACL 2023 DiGA: Distil To Generalize and Then Adapt for Domain Adaptive Semantic Segmentation CVPR 2023 Distilling Efficient Language-Specific Models for Cross-Lingual Transfer ACL 2023 Exploring All-In-One Knowledge Distillation Framework for Neural Machine Translation EMNLP 2023 Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting It into MLPs: An Effective GNN-to-MLP Distillation Framework AAAI 2023 How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectives ACL 2023