Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tails

arXiv:2405.17529v1 Announce Type: cross
Abstract: Differentially Private Stochastic Gradient Descent (DPSGD) is widely utilized to preserve training data privacy in deep learning, which first clips the gradients to a predefined norm and then injects calibrated noise into the training procedure. Existing DPSGD works typically assume the gradients follow sub-Gaussian distributions and design various clipping mechanisms to optimize training performance. However, recent studies have shown that the gradients in deep learning exhibit a heavy-tail phenomenon, that is, the tails of the …

arxiv body cs.cr cs.lg data data privacy deep learning high noise privacy private procedure tail tails training training data