Differentially private training of residual networks with scale normalisation. (arXiv:2203.00324v2 [cs.LG] UPDATED)

The training of neural networks with Differentially Private Stochastic
Gradient Descent offers formal Differential Privacy guarantees but introduces
accuracy trade-offs. In this work, we propose to alleviate these trade-offs in
residual networks with Group Normalisation through a simple architectural
modification termed ScaleNorm by which an additional normalisation layer is
introduced after the residual block's addition operation. Our method allows us
to further improve on the recently reported state-of-the art on CIFAR-10,
achieving a top-1 accuracy of 82.5% ({\epsilon}=8.0) when trained …

lg networks scale training