Various initializers and batch normalization

An implementation of weight/bias initializers and batch normalization in Tensorflow.

MNIST database is used to show performance-comparison

Network architecture

In order to examine the effect of initializers and batch normalization, a simple network architecture called multilayer perceptrons (MLP) is employed.

MLP has following architecture.

• input layer : 784 nodes (MNIST images size)
• first hidden layer : 256 nodes
• second hidden layer : 256 nodes
• output layer : 10 nodes (number of class for MNIST)

Various initializers

The following initializers for weights/biases of network are considered.

• Weight Initializer
  • normal, trucated normal
  • xaiver : Understanding the difficulty of training deep feedforward neural networks
  • he : Delving Deep into Rectifiers
• Bias Initializer
  • normal, constant

Simulation results

• Weight Initializer : he > trauncated normal = xaiver > normal
• Bias Initilaizer : zero > normal

Sample results are following.

Index	Weight Initializer	Bias Initializer	Accuracy
normal_w_normal_b_0.9451	normal	normal	0.9451
normal_w_zero_b_0.9485	normal	zero	0.9485
truncated_normal_w_normal_b_0.9788	truncated_normal	normal	0.9788
truncated_normal_w_zero_b_0.9790	truncated_normal	zero	0.9790
xavier_w_normal_b_0.9800	xavier	normal	0.9800
xavier_w_zero_b_0.9806	xavier	zero	0.9806
he_w_normal_b_0.9798	he	normal	0.9798
he_w_zero_b_0.9811	he	zero	0.9811

Batch normalization

Batch normalization improves performance of network in terms of final accuracy and convergence rate.
In this simulation, bias initalizer was zero-constant initializer and weight initializers were xavier / he.

Sample results are following.

Index	Weight Initializer	Batch Normalization	Accuracy
xavier_woBN_0.9806	xavier	Unused	0.9806
xavier_withBN_0.9812	xavier	Used	0.9812
he_woBN_0.9811	he	Unused	0.9811
he_withBN_0.9837	he	Used	0.9837

Usage

python run_main.py --weight-init <weight initializer> --bias-init <bias initializer> --batch-norm <True or False>

<weight initializer> must be selected in [normal, truncated_normal, xavier, he].
<bias initializer> must be selected in [normal, zero].

You may command like python run_main.py --weight-init xavier --bias-init zero --batch-norm True.

Acknowledgement

This implementation has been tested on Tensorflow r0.12.