← Keywords

knowledge distillation

3680 papers

Explore in graph

Also known as

CD DMD LORA KL DNA SELF-DISTILLATION TKD NBOD AD KD AOTD KI GID FD MKD SEQKD

Co-occurring keywords

model compression (3283) large language model (12755) transfer learning (5442) domain adaptation (4578) representation learning (6174) neural network (6616) language model (4573) continual learning (1164) catastrophic forgetting (939) contrastive learning (3979)

Papers

Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing? ICML 2022
From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression AAAI 2022
The Xiaomi Text-to-Text Simultaneous Speech Translation System for IWSLT 2022 ACL 2022
Pseudo-spherical Knowledge Distillation IJCAI 2022
In defense of dual-encoders for neural ranking ICML 2022
Empowering Dual-Encoder with Query Generator for Cross-Lingual Dense Retrieval EMNLP 2022
Weak-shot Semantic Segmentation via Dual Similarity Transfer NIPS 2022
Contact-Distil: Boosting Low Homologous Protein Contact Map Prediction by Self-Supervised Distillation AAAI 2022
Conformer with dual-mode chunked attention for joint online and offline ASR INTERSPEECH 2022
Knowledge distillation for In-memory keyword spotting model INTERSPEECH 2022
Data-Free Adversarial Knowledge Distillation for Graph Neural Networks IJCAI 2022
Can Functional Transfer Methods Capture Simple Inductive Biases? AISTATS 2022
TRILLsson: Distilled Universal Paralinguistic Speech Representations INTERSPEECH 2022
Device-Directed Speech Detection: Regularization via Distillation for Weakly-Supervised Models INTERSPEECH 2022
Adversarial Knowledge Distillation For Robust Spoken Language Understanding INTERSPEECH 2022
Performance-Aware Mutual Knowledge Distillation for Improving Neural Architecture Search CVPR 2022
Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition INTERSPEECH 2022
LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT INTERSPEECH 2022
Multi-Level Branched Regularization for Federated Learning ICML 2022
Rich Feature Construction for the Optimization-Generalization Dilemma ICML 2022
XDBERT: Distilling Visual Information to BERT from Cross-Modal Systems to Improve Language Understanding ACL 2022
The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimes NIPS 2022
Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection EMNLP 2022
Distillation-Resistant Watermarking for Model Protection in NLP EMNLP 2022
Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging EMNLP 2022