Knowledge Distillation

Attention Temperature Matters in Abstractive Summarization Distillation ACL 2022 Attribute Injection for Pretrained Language Models: A New Benchmark and an Efficient Method COLING 2022 Being Friends Instead of Adversaries: Deep Networks Learn from Data Simplified by Other Networks AAAI 2022 bert2BERT: Towards Reusable Pretrained Language Models ACL 2022 BERT Learns to Teach: Knowledge Distillation with Meta Learning ACL 2022 BiBERT: Accurate Fully Binarized BERT ICLR 2022 Bi-directional Weakly Supervised Knowledge Distillation for Whole Slide Image Classification NeurIPS 2022 Biomedical NER for the Enterprise with Distillated BERN2 and the Kazu Framework EMNLP 2022 BiT: Robustly Binarized Multi-distilled Transformer NeurIPS 2022 BMCook: A Task-agnostic Compression Toolkit for Big Models EMNLP 2022 Boosting Contrastive Learning with Relation Knowledge Distillation AAAI 2022 Bridging Fairness and Environmental Sustainability in Natural Language Processing EMNLP 2022 Calibrating Student Models for Emotion-related Tasks EMNLP 2022 CILDA: Contrastive Data Augmentation Using Intermediate Layer Knowledge Distillation COLING 2022 Class-Incremental Learning by Knowledge Distillation With Adaptive Feature Consolidation CVPR 2022 Class Incremental Learning for Intent Classification with Limited or No Old Data EMNLP 2022 CN-AutoMIC: Distilling Chinese Commonsense Knowledge from Pretrained Language Models EMNLP 2022 Co-Advise: Cross Inductive Bias Distillation CVPR 2022 Compressing Sentence Representation for Semantic Retrieval via Homomorphic Projective Distillation ACL 2022 Compression of Generative Pre-trained Language Models via Quantization ACL 2022 ConGen: Unsupervised Control and Generalization Distillation For Sentence Representation EMNLP 2022 Contact-Distil: Boosting Low Homologous Protein Contact Map Prediction by Self-Supervised Distillation AAAI 2022 Contextual Fine-to-Coarse Distillation for Coarse-grained Response Selection in Open-Domain Conversations ACL 2022 Contextual Similarity Distillation for Asymmetric Image Retrieval CVPR 2022 Co-Segmentation Aided Two-Stream Architecture for Video Captioning WACV 2022