Investigating and Defending Shortcut Learning in Personalized Diffusion Models

arXiv:2406.18944v1 Announce Type: cross
Abstract: Personalized diffusion models have gained popularity for adapting pre-trained text-to-image models to generate images of specific topics with only a few images. However, recent studies find that these models are vulnerable to minor adversarial perturbation, and the fine-tuning performance is largely degraded on corrupted datasets. Such characteristics are further exploited to craft protective perturbation on sensitive images like portraits that prevent unauthorized generation. In response, diffusion-based purification methods have been proposed to remove these perturbations …

adversarial arxiv cs.ai cs.cr cs.cv datasets defending diffusion models find fine-tuning image images performance shortcut studies text topics vulnerable