TADP: Structured Pruning of Large Language Models via Temporal Activation Dynamics
Abstract
AbstractStructured pruning of large language models provides a practical route to inference acceleration on standard hardware, but most existing methods rely on neuron importance criteria that compress activation behavior into static scalar scores, discarding the sequential structure of activations. We propose TADP (Temporal Activation Dynamics Pruning), a structured pruning method that models neuron importance along the temporal dimension of activations by jointly capturing mean activation, peak activation, positional variance, transition dynamics, and position-weighted response within a unified scoring framework. TADP requires no gradient computation or weight updates and performs pruning in a single forward pass over a calibration dataset, making it zero-shot and computationally lightweight. We evaluate TADP on eight models from the LLaMA, Mistral, Qwen, and Minerva families at 3B and 7B/8B scale, under pruning levels from 10% to 50%, measuring perplexity on WikiText-2 and zero-shot accuracy on ARC, HellaSwag, MMLU, and PIQA.