Learning Goal - Weight Initialisation
3 important questions on Learning Goal - Weight Initialisation
What is the problem with standard Gaussian initialisation (mean 0, std 1)?
For a neuron with n_in inputs, the weighted input z = Σⱼ wⱼxⱼ + b is a sum of n_in random variables, each with standard deviation ~1. By the variance addition rule, z has standard deviation ≈ √n_in.
For large n_in (e.g., 1000 inputs): |z| will typically be ~√1000 ≈ 32. At |z| = 32, σ(z) ≈ 0 or ≈ 1 — the neuron is saturated from the very first step. Training starts with a vanishing gradient and learns painfully slowly.
For large n_in (e.g., 1000 inputs): |z| will typically be ~√1000 ≈ 32. At |z| = 32, σ(z) ≈ 0 or ≈ 1 — the neuron is saturated from the very first step. Training starts with a vanishing gradient and learns painfully slowly.
What is the improved weight initialisation? Give the formula and the variance argument for why it works.
Initialise weights as Gaussians with:
w ~ N(0, 1/√n_in)
Variance argument: each weight has variance 1/n_in. The weighted sum z = Σⱼ wⱼxⱼ has variance:
Var(z) = n_in · (1/n_in) = 1
So z has standard deviation ~1 regardless of how many inputs there are. The neuron starts in the active, non-saturated region of the sigmoid — it can learn immediately.
w ~ N(0, 1/√n_in)
Variance argument: each weight has variance 1/n_in. The weighted sum z = Σⱼ wⱼxⱼ has variance:
Var(z) = n_in · (1/n_in) = 1
So z has standard deviation ~1 regardless of how many inputs there are. The neuron starts in the active, non-saturated region of the sigmoid — it can learn immediately.
Does improved weight initialisation affect final accuracy or just initial learning speed?
Primarily initial learning speed — networks with proper initialisation reach good accuracy much faster in early epochs. Final accuracy can also improve because a network that learns quickly may converge to better optima (escaping bad regions before they become sticky) compared to one that is stuck in saturation from the start.
The question on the page originate from the summary of the following study material:
- A unique study and practice tool
- Never study anything twice again
- Get the grades you hope for
- 100% sure, 100% understanding
Remember faster, study better. Scientifically proven.
