Learning Goal - Regularization: Dropout
3 important questions on Learning Goal - Regularization: Dropout
What is dropout? Describe exactly what happens during a training step.
During each training mini-batch:
1. Randomly delete (drop) each hidden neuron with probability 0.5 (keep with 0.5)
2. Do a forward pass and backward pass through the reduced network (dropped neurons and their connections are temporarily gone)
3. Update weights only for the surviving neurons
4. Restore all neurons for the next mini-batch (a new random subset is dropped)
At test time: use the full network but halve all weights (to account for the fact that twice as many neurons are active compared to training).
1. Randomly delete (drop) each hidden neuron with probability 0.5 (keep with 0.5)
2. Do a forward pass and backward pass through the reduced network (dropped neurons and their connections are temporarily gone)
3. Update weights only for the surviving neurons
4. Restore all neurons for the next mini-batch (a new random subset is dropped)
At test time: use the full network but halve all weights (to account for the fact that twice as many neurons are active compared to training).
Why does dropout work? Explain the ensemble interpretation.
Each different random subset of dropped neurons produces a different sub-network. Training with dropout is like simultaneously training an enormous ensemble of different smaller networks. Networks overfit in different ways; averaging their predictions (which the full network with halved weights approximates) tends to cancel out individual overfitting.
Additionally, no neuron can rely on the guaranteed presence of specific other neurons — each neuron must learn robust, independent features that are useful across many network configurations.
Additionally, no neuron can rely on the guaranteed presence of specific other neurons — each neuron must learn robust, independent features that are useful across many network configurations.
Why are weights halved at test time in dropout?
During training, each neuron is only active ~50% of the time (dropped the other 50%). So the total incoming signal to any neuron is roughly half what the full network would produce. At test time, all neurons are active, so the expected input to each neuron would be twice what it experienced during training — disrupting all the learned thresholds. Halving the weights at test time rescales the signal back to the level the neuron was trained on.
The question on the page originate from the summary of the following study material:
- A unique study and practice tool
- Never study anything twice again
- Get the grades you hope for
- 100% sure, 100% understanding
Remember faster, study better. Scientifically proven.
