Tianlong Chen (陈天龙)

What does not kill you makes you stronger

(NeurIPS 2020) Once-for-All Adversarial Training, In-Situ Trade off between Robustness and Accuracy for Free

Once-for-All Adversarial Training: In-Situ Trade off between Robustness and Accuracy for Free

[Paper] [Code]

Abstract

Adversarial training and its many variants substantially improve deep network robustness, yet at the cost of compromising standard accuracy. Moreover, the training process is heavy and hence it becomes impractical to thoroughly explore the trade-off between accuracy and robustness. This paper asks this new question: how to quickly calibrate a trained model in-situ, to examine the achievable trade-offs between its standard and robust accuracies, without (re-)training it many times? Our proposed framework, Calibratable Adversarial Training (CAT), is built on an innovative model-conditional training framework, with a controlling hyper-parameter as the input. The resulting model could be adjusted among different standard and robust accuracies “for free” at testing time. As an important knob, we exploit dual batch normalization to separate standard and adversarial feature statistics, so that they can be learned in one model without degrading performance. We further extend CAT to a Calibratable Adversarial Training and Slimming (CATS) framework, that allows for the joint trade-off among accuracy, robustness and runtime efficiency. Experiments show that, without any re-training nor ensembling, CAT/CATS achieve similar or even superior performance compared to dedicatedly trained models at various configurations. All codes and pretrained models will be released upon acceptance. The codes are publicly available at: https://github.com/VITA-Group/Once-for-All-Adversarial-Training.