Subsampling is not Magic: Why Large Batch Sizes Work for Differentially Private Stochastic Optimisation

We study the effect of the batch size to the total gradient variance in differentially private stochastic gradient descent (DP-SGD), seeking a theoretical explanation for the usefulness of large batch sizes. As DP-SGD is the basis of modern DP deep learning, its properties have been widely studied, and recent works have empirically found large batch sizes to be beneficial. However, theoretical explanations of this benefit are currently heuristic at best. We first observe that the total gradient variance in DP-SGD …

batch cs.cr cs.lg deep learning effect large magic private size stat.ml study work