Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Basic Information

• Sergey Ioffe, Christian Szegedy from Google Reserch, @google.com
• 2015 ICML

Problem Description

在訓練Deep Neural Network的過程中，每層layer的input distribution都會隨著前一層的parameter而改變(Internal Covariate Shift)，容易導致Model收斂緩不容易train，而Batch Normalization就是透過在每一層的input都做一次normalization，來解決Internal Covariates Shift，減緩Vanishing Gradient的問題，同時也達到regularization的效果。