Home / Summaries / COMP 3711 / l2-l1-weight

Learning Goal - Improving the Way Neural Networks Learn

Q: Why does L1 produce sparse models but L2 does not?

- L1 : shrinks by a fixed amount at every step. A small weight is just as aggressively pushed toward zero as a large weight. Eventually many weights hit zero and stay there — feature selection . - L2 : shrinks proportionally. A very small weight receives a tiny shrinkage — it asymptotically approaches zero but never quite reaches it. All weights survive, just at reduced magnitude. Geometrically: L1's constraint region is a diamond (pointy at axes → optimal solutions land exactly on an axis = zero weight). L2's constraint is a circle (smooth → optimal solutions almost never land exactly on an axis).

Q: What is artificial data expansion and why is it effective?

Artificial data expansion creates new training examples by applying realistic transformations to existing ones — for images: small rotations, translations, elastic distortions, skews. This dramatically increases the effective training set size. It is effective because these transformed examples represent real variations the network will encounter in deployment (e.g., handwriting varies in tilt, size, position). The network is explicitly trained to be invariant to these variations, rather than accidentally learning them as signal.

Q: What is Nielsen's broad strategy for hyperparameter tuning?

Simplify the problem first — use a subset (e.g., only digits 0 and 1), a simpler architecture, to get rapid feedback on what works Learning rate η : find the order of magnitude where training cost decreases immediately without large oscillations λ for regularisation : start with λ = 0 to find η, then try λ = 1.0 and scale by factors of 10 using validation accuracy Mini-batch size : balance between fast matrix operations (large batches) and frequent weight updates (small batches) Expand complexity only once basics are working

Q: What does momentum-based gradient descent add to the standard update rule?

Standard gradient descent: w → w − η·∇C Momentum adds a velocity term v and a friction coefficient μ: v → μv − η·∇C w → w + v The velocity accumulates in directions where the gradient consistently points — the 'ball' builds up speed. μ (between 0 and 1) acts as friction, gradually reducing speed in changing-gradient directions.

Q: How does momentum help escape local minima?

In standard gradient descent, the ball stops the moment it reaches a flat region or local minimum — it has no momentum to carry it further. With momentum, accumulated velocity from consistent gradient directions allows the ball to roll across small bumps or overshoot shallow local minima, continuing downhill into deeper valleys. It also speeds convergence in ravine-shaped cost surfaces where oscillation is a problem.

Q: A network is training but the cost barely changes for the first 20 epochs. Give three distinct diagnoses using techniques from this chapter.

Saturated neurons (wrong cost function) : using quadratic cost — neurons are saturated from the start, σ'(z) ≈ 0 kills the gradient. Fix: switch to cross-entropy . Bad weight initialisation : standard Gaussian initialisation with many inputs causes |z| >> 1 immediately — neurons saturate before learning begins. Fix: 1/√n_in initialisation . Learning rate too small : η is so tiny that each step moves the weights imperceptibly. Fix: increase η by an order of magnitude and observe whether the cost starts descending immediately.

7 important questions on Learning Goal - Improving the Way Neural Networks Learn

Write the L1 weight update rule. How does it differ from L2?

w → w − η·(λ/n)·sgn(w) − η·∂C₀/∂w
Where sgn(w) = +1 if w > 0, −1 if w < 0.
Key difference from L2: L1 shrinks weights by a fixed constant η·λ/n regardless of weight magnitude. L2 shrinks by a proportion of the current weight.
Consequence: L1 aggressively drives small weights all the way to exactly zero (sparse model). L2 makes all weights small but almost never zero.

Why does L1 produce sparse models but L2 does not?

- L1: shrinks by a fixed amount at every step. A small weight is just as aggressively pushed toward zero as a large weight. Eventually many weights hit zero and stay there — feature selection.

- L2: shrinks proportionally. A very small weight receives a tiny shrinkage — it asymptotically approaches zero but never quite reaches it. All weights survive, just at reduced magnitude.

Geometrically: L1's constraint region is a diamond (pointy at axes → optimal solutions land exactly on an axis = zero weight). L2's constraint is a circle (smooth → optimal solutions almost never land exactly on an axis).

What is artificial data expansion and why is it effective?

Artificial data expansion creates new training examples by applying realistic transformations to existing ones — for images: small rotations, translations, elastic distortions, skews. This dramatically increases the effective training set size.

It is effective because these transformed examples represent real variations the network will encounter in deployment (e.g., handwriting varies in tilt, size, position). The network is explicitly trained to be invariant to these variations, rather than accidentally learning them as signal.

What is Nielsen's broad strategy for hyperparameter tuning?

Simplify the problem first — use a subset (e.g., only digits 0 and 1), a simpler architecture, to get rapid feedback on what works
Learning rate η: find the order of magnitude where training cost decreases immediately without large oscillations
λ for regularisation: start with λ = 0 to find η, then try λ = 1.0 and scale by factors of 10 using validation accuracy
Mini-batch size: balance between fast matrix operations (large batches) and frequent weight updates (small batches)
Expand complexity only once basics are working

What does momentum-based gradient descent add to the standard update rule?

Standard gradient descent: w → w − η·∇C
Momentum adds a velocity term v and a friction coefficient μ:
v → μv − η·∇C w → w + v
The velocity accumulates in directions where the gradient consistently points — the "ball" builds up speed. μ (between 0 and 1) acts as friction, gradually reducing speed in changing-gradient directions.

How does momentum help escape local minima?

In standard gradient descent, the ball stops the moment it reaches a flat region or local minimum — it has no momentum to carry it further. With momentum, accumulated velocity from consistent gradient directions allows the ball to roll across small bumps or overshoot shallow local minima, continuing downhill into deeper valleys. It also speeds convergence in ravine-shaped cost surfaces where oscillation is a problem.

A network is training but the cost barely changes for the first 20 epochs. Give three distinct diagnoses using techniques from this chapter.

Saturated neurons (wrong cost function): using quadratic cost — neurons are saturated from the start, σ'(z) ≈ 0 kills the gradient. Fix: switch to cross-entropy.
Bad weight initialisation: standard Gaussian initialisation with many inputs causes |z| >> 1 immediately — neurons saturate before learning begins. Fix: 1/√n_in initialisation.
Learning rate too small: η is so tiny that each step moves the weights imperceptibly. Fix: increase η by an order of magnitude and observe whether the cost starts descending immediately.

The question on the page originate from the summary of the following study material:

COMP 3711

View summary

A unique study and practice tool
Never study anything twice again
Get the grades you hope for
100% sure, 100% understanding

Remember faster, study better. Scientifically proven.

Topics that are related to Learning Goal - Improving the Way Neural Networks Learn

Learning Goal - Improving the Way Neural Networks Learn

7 important questions on Learning Goal - Improving the Way Neural Networks Learn

Write the L1 weight update rule. How does it differ from L2?

Why does L1 produce sparse models but L2 does not?

What is artificial data expansion and why is it effective?

What is Nielsen's broad strategy for hyperparameter tuning?

What does momentum-based gradient descent add to the standard update rule?

How does momentum help escape local minima?

A network is training but the cost barely changes for the first 20 epochs. Give three distinct diagnoses using techniques from this chapter.

Topics that are related to Learning Goal - Improving the Way Neural Networks Learn

Learning Goal - Cross-Entropy Cost Function

Learning Goal - Softmax and Log-Likelihood

Learning Goal - Overfitting and Detection

Learning Goal - Regularization: L2

Learning Goal - Regularization: Dropout

Learning Goal - Weight Initialisation

Learning Goal - Alternative Activation Functions

COMP 3711

Global politics

Essentials of international relations

Behavioral genetics

Management and organisational behaviour

Follow Up Engels idioom 4/5 H

International Business

Marketing fundamentals

Projectmanagement, A practical Approach-Engl…

Basic Management Accounting for the Hospital…

International business

Organization theory and design

Learning Goal - Improving the Way Neural Networks Learn

7 important questions on Learning Goal - Improving the Way Neural Networks Learn

Write the L1 weight update rule. How does it differ from L2?

Why does L1 produce sparse models but L2 does not?

What is artificial data expansion and why is it effective?

What is Nielsen's broad strategy for hyperparameter tuning?

What does momentum-based gradient descent add to the standard update rule?

How does momentum help escape local minima?

A network is training but the cost barely changes for the first 20 epochs. Give three distinct diagnoses using techniques from this chapter.

Topics that are related to Learning Goal - Improving the Way Neural Networks Learn

Learning Goal - Cross-Entropy Cost Function

Learning Goal - Softmax and Log-Likelihood

Learning Goal - Overfitting and Detection

Learning Goal - Regularization: L2

Learning Goal - Regularization: Dropout

Learning Goal - Weight Initialisation

Learning Goal - Alternative Activation Functions

Summaries related to Classification - Impurity Measures

COMP 3711

Global politics

Essentials of international relations

Behavioral genetics

Management and organisational behaviour

Follow Up Engels idioom 4/5 H

International Business

Marketing fundamentals

Projectmanagement, A practical Approach-Engl…

Basic Management Accounting for the Hospital…

International business

Organization theory and design