Localizing Paragraph Memorization in Language Models

arXiv:2403.19851v1 Announce Type: cross
Abstract: Can we localize the weights and mechanisms used by a language model to memorize and recite entire paragraphs of its training data? In this paper, we show that while memorization is spread across multiple layers and model components, gradients of memorized paragraphs have a distinguishable spatial pattern, being larger in lower model layers than gradients of non-memorized examples. Moreover, the memorized examples can be unlearned by fine-tuning only the high-gradient weights. We localize a low-layer …

arxiv can components cs.cl cs.cr cs.lg data language language models stat.ml training training data