Differentially Private Knowledge Distillation via Synthetic Text Generation

arXiv:2403.00932v1 Announce Type: cross
Abstract: Large Language models (LLMs) are achieving state-of-the-art performance in many different downstream tasks. However, the increasing urgency of data privacy requires LLMs to train with Differential Privacy (DP) on private data. Concurrently it is also necessary to compress LLMs for real-life deployments on resource-constrained devices or latency-sensitive applications. Differential privacy and model compression generally must trade off utility loss to achieve their objectives. Moreover, concurrently achieving both can result in even more utility loss. To …

art arxiv cs.cl cs.cr cs.lg data data privacy devices differential privacy knowledge language language models large latency life llms performance privacy private private data real resource sensitive state synthetic text train