Pre-training Differentially Private Models with Limited Public Data

arXiv:2402.18752v1 Announce Type: cross
Abstract: The superior performance of large foundation models relies on the use of massive amounts of high-quality data, which often contain sensitive, private and copyrighted material that requires formal protection. While differential privacy (DP) is a prominent method to gauge the degree of security provided to the models, its application is commonly limited to the model fine-tuning stage, due to the performance degradation when applying DP during the pre-training stage. Consequently, DP is yet not capable …

arxiv cs.cr cs.lg data differential privacy foundation foundation models gauge high large material performance privacy private protection public quality security sensitive training