Safeguarding Data in Multimodal AI: A Differentially Private Approach to CLIP Training

arXiv:2306.08173v2 Announce Type: replace-cross
Abstract: The surge in multimodal AI's success has sparked concerns over data privacy in vision-and-language tasks. While CLIP has revolutionized multimodal learning through joint training on images and text, its potential to unintentionally disclose sensitive information necessitates the integration of privacy-preserving mechanisms. We introduce a differentially private adaptation of the Contrastive Language-Image Pretraining (CLIP) model that effectively addresses privacy concerns while retaining accuracy. Our proposed method, Dp-CLIP, is rigorously evaluated on benchmark datasets encompassing diverse vision-and-language …

arxiv cs.cr cs.it cs.lg data data privacy images information integration language math.it privacy private safeguarding data sensitive sensitive information stat.ml text training