Learning Goal - Regularization: L2
3 important questions on Learning Goal - Regularization: L2
Write the L2-regularised cost function. What term is added and why?
C = C₀ + (λ/2n) Σ_w w²
The added term (λ/2n) Σ_w w² penalises large weights. This forces the network to prefer smaller weights, which means no single input dominates the decision — the network must learn a smoother, more generalisable function rather than fitting local noise.
The added term (λ/2n) Σ_w w² penalises large weights. This forces the network to prefer smaller weights, which means no single input dominates the decision — the network must learn a smoother, more generalisable function rather than fitting local noise.
Write the L2 weight update rule. What does "weight decay" mean physically?
w → (1 − η·λ/n) · w − η · ∂C₀/∂w
The weight is rescaled (multiplied by a number slightly less than 1) at every step before the gradient update. This is "weight decay" — weights are continuously pulled toward zero. Large weights shrink by large amounts; small weights shrink by small amounts (proportional decay).
The weight is rescaled (multiplied by a number slightly less than 1) at every step before the gradient update. This is "weight decay" — weights are continuously pulled toward zero. Large weights shrink by large amounts; small weights shrink by small amounts (proportional decay).
What is the regularisation parameter λ and how does it affect behaviour?
λ controls the trade-off between fitting the data and keeping weights small:
- λ = 0: no regularisation — minimise training error only → risk overfitting
- Large λ: strong penalty on weights → model is forced to be very simple → may underfit
- Tuning λ via validation data finds the sweet spot between these extremes
The question on the page originate from the summary of the following study material:
- A unique study and practice tool
- Never study anything twice again
- Get the grades you hope for
- 100% sure, 100% understanding
Remember faster, study better. Scientifically proven.
