A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
Abstract.
Post-training pruning removes weights from a trained language model to cut inference cost, but the pruned model loses quality unless the remaining weights are adapted. For decades the standard answer was to prune and then retrain. However, for large language models (LLMs), retraining was declared infeasible, and the field responded with developing ever more elaborate rules for choosing which weights to remove so that no adaptation is needed afterwards. We argue that post-pruning adaptation is still feasible in the era of LLMs. We revisit local reconstruction: after pruning with a fixed mask, one submodel at a time is adapted on a few hundred calibration sequences to match the intermediate activations of the dense model. Across four model families from 0.5B to 72B parameters, we establish three findings. First, local reconstruction matches LoRA-style retraining in perplexity and zero-shot accuracy while using 64 times fewer samples and up to 130 times less compute, and it fits a 32B model on a single 80 GB GPU. Second, the size of the reconstructed submodel, from half a transformer block to a quarter of the network, has almost no effect on final quality, while peak memory varies by hundreds of gigabytes. The one exception is per-matrix reconstruction, the formulation most widely used in the literature, which consistently underperforms. We trace this failure to compositional error accumulation and show that including a single nonlinearity in the reconstructed submodel is what removes it. Third, once reconstruction is applied, the gap between sophisticated pruning criteria and plain magnitude pruning shrinks with model scale and essentially vanishes above 30B parameters. Part of what sophisticated criteria bought was compensation for a missing adaptation step. Together, these results establish local reconstruction as a practical default for post-pruning adaptation at LLM scale and shift the central question of LLM pruning from which weights to remove to how to adapt the ones that remain.
