Unveiling and Mitigating Memorization in Text-to-image Diffusion Models through Cross Attention

arXiv:2403.11052v1 Announce Type: cross
Abstract: Recent advancements in text-to-image diffusion models have demonstrated their remarkable capability to generate high-quality images from textual prompts. However, increasing research indicates that these models memorize and replicate images from their training data, raising tremendous concerns about potential copyright infringement and privacy risks. In our study, we provide a novel perspective to understand this memorization phenomenon by examining its relationship with cross-attention mechanisms. We reveal that during memorization, the cross-attention tends to focus disproportionately on …

arxiv attention cs.cr cs.cv diffusion models image text