When Does Differentially Private Learning Not Suffer in High Dimensions?. (arXiv:2207.00160v3 [cs.LG] UPDATED)

Large pretrained models can be privately fine-tuned to achieve performance
approaching that of non-private models. A common theme in these results is the
surprising observation that high-dimensional models can achieve favorable
privacy-utility trade-offs. This seemingly contradicts known results on the
model-size dependence of differentially private convex learning and raises the
following research question: When does the performance of differentially
private learning not degrade with increasing model size? We identify that the
magnitudes of gradients projected onto subspaces is a key …

lg