A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimization

arXiv:2212.04486v3 Announce Type: replace-cross
Abstract: An open problem in differentially private deep learning is hyperparameter optimization (HPO). DP-SGD introduces new hyperparameters and complicates existing ones, forcing researchers to painstakingly tune hyperparameters with hundreds of trials, which in turn makes it impossible to account for the privacy cost of HPO without destroying the utility. We propose an adaptive HPO method that uses cheap trials (in terms of privacy cost and runtime) to estimate optimal hyperparameters and scales them up. We obtain …

account arxiv cost cs.ai cs.cr cs.lg deep learning linear optimization privacy private problem researchers scaling turn