Vanilla Feature Distillation for Improving the Accuracy-Robustness Trade-Off in Adversarial Training. (arXiv:2206.02158v1 [cs.CV])

Adversarial training has been widely explored for mitigating attacks against
deep models. However, most existing works are still trapped in the dilemma
between higher accuracy and stronger robustness since they tend to fit a model
towards robust features (not easily tampered with by adversaries) while
ignoring those non-robust but highly predictive features. To achieve a better
robustness-accuracy trade-off, we propose the Vanilla Feature Distillation
Adversarial Training (VFD-Adv), which conducts knowledge distillation from a
pre-trained model (optimized towards high accuracy) to …

adversarial trade training vanilla