Skip to content
conftrace_
2026 EMNLP EMNLP 2026

LLM Fine-tuning with an Unconstrained Fine-tuned Teacher

AbstractFine-tuning large language models (LLMs) on a single target task requires balancing task adaptation with retention of prior capabilities. Existing teachers mainly guide preservation or transfer, but do not identify which target-task samples should be emphasized when fine-tuning must retain prior capabilities. We propose the **U**nconstrained **F**ine-tuned **T**eacher (**UFT**), shifting the teacher role from ”knowledge transfer” to ”value discovery”. UFT is obtained by unconstrained fine-tuning on the target task and then compared with the source model to identify high-value samples. These samples are upweighted during student fine-tuning and receive value-weighted UFT-guided distillation, allowing the student to prioritize task-critical signals while preserving prior capabilities. Experiments on Magicoder and MetaMathQA show that UFT improves the trade-off between target-task performance and prior-capability retention, while converging faster and more stably. By turning teacher behavior into sample-value discovery, UFT establishes a new teacher role for data-efficient fine-tuning.

More from EMNLP 2026

All EMNLP 2026 papers →