DP-SGD for non-decomposable objective functions. (arXiv:2310.03104v1 [cs.LG])

Unsupervised pre-training is a common step in developing computer vision
models and large language models. In this setting, the absence of labels
requires the use of similarity-based loss functions, such as contrastive loss,
that favor minimizing the distance between similar inputs and maximizing the
distance between distinct inputs. As privacy concerns mount, training these
models using differential privacy has become more important. However, due to
how inputs are generated for these losses, one of their undesirable properties
is that their …

computer computer vision functions inputs language language models large loss non privacy privacy concerns similarity training