Home / Summaries / COMP 3711 / backpropagation-gradient-c

Learning Goal - How the Backpropagation Algorithm Works

Q: What mathematical tool underlies all four backpropagation equations?

All four equations are consequences of the multivariable chain rule from calculus. Each partial derivative in the network is computed by tracing how a small change in one variable propagates through subsequent variables to the cost.

6 important questions on Learning Goal - How the Backpropagation Algorithm Works

What problem does backpropagation solve, and what does it compute?

Chapter 1 showed that gradient descent can train a network — but it didn't explain how to compute the gradient efficiently. Backpropagation is a fast algorithm that computes the partial derivatives of the cost function with respect to every weight and bias in the network:

∂C/∂w and ∂C/∂b

These gradients are exactly what gradient descent needs to update parameters.

Why is numerical gradient estimation impractical for large networks?

Numerical estimation perturbs each weight by a tiny ε and measures the change in cost. With n weights, this requires n forward passes through the network. For a network with 1 million weights, that is 1 million forward passes per gradient step.

Backpropagation computes all partial derivatives using just 1 forward pass + 1 backward pass — roughly 500,000× faster for a 1M-weight network.

What two assumptions must the cost function satisfy for backpropagation to apply?

Averaging: the cost must be an average over individual training examples:
C = (1/n) Σₓ Cₓ
This lets backpropagation compute per-example gradients, which can then be averaged to get the total gradient.
Output dependency: C must depend only on the network's output activations a^L (not on internal layer values directly).

Why must the cost function be an average over training examples?

Backpropagation computes ∂Cₓ/∂w for one training example x at a time. To perform gradient descent on the full dataset, we need the gradient of the total cost. If C = (1/n) Σₓ Cₓ, then by linearity:

∂C/∂w = (1/n) Σₓ ∂Cₓ/∂w

We simply average the per-example gradients. Without this structure, the per-example results can't be combined.

What mathematical tool underlies all four backpropagation equations?

All four equations are consequences of the multivariable chain rule from calculus. Each partial derivative in the network is computed by tracing how a small change in one variable propagates through subsequent variables to the cost.

Sketch the derivation of BP1 using the chain rule.

We want δ^L_j = ∂C/∂z^L_j.
Apply the chain rule: C depends on z^L_j through a^L_j = σ(z^L_j):
δ^L_j = ∂C/∂z^L_j = (∂C/∂a^L_j) · (∂a^L_j/∂z^L_j) = (∂C/∂a^L_j) · σ'(z^L_j)
In vector form: δ^L = ∇ₐC ⊙ σ'(z^L) — this is BP1.

The question on the page originate from the summary of the following study material:

COMP 3711

View summary

A unique study and practice tool
Never study anything twice again
Get the grades you hope for
100% sure, 100% understanding

Remember faster, study better. Scientifically proven.

Topics that are related to Learning Goal - How the Backpropagation Algorithm Works

Learning Goal - How the Backpropagation Algorithm Works

6 important questions on Learning Goal - How the Backpropagation Algorithm Works

What problem does backpropagation solve, and what does it compute?

Why is numerical gradient estimation impractical for large networks?

What two assumptions must the cost function satisfy for backpropagation to apply?

Why must the cost function be an average over training examples?

What mathematical tool underlies all four backpropagation equations?

Sketch the derivation of BP1 using the chain rule.

Topics that are related to Learning Goal - How the Backpropagation Algorithm Works

Learning Goal - Notation

Learning Goal - The Error δ

Learning Goal - The Backpropagation Algorithm

Learning Goal - Synthesis (Feynman Challenges)

COMP 3711

Global politics

Essentials of international relations

Behavioral genetics

Management and organisational behaviour

Follow Up Engels idioom 4/5 H

International Business

Marketing fundamentals

Projectmanagement, A practical Approach-Engl…

Basic Management Accounting for the Hospital…

International business

Organization theory and design

Learning Goal - How the Backpropagation Algorithm Works

6 important questions on Learning Goal - How the Backpropagation Algorithm Works

What problem does backpropagation solve, and what does it compute?

Why is numerical gradient estimation impractical for large networks?

What two assumptions must the cost function satisfy for backpropagation to apply?

Why must the cost function be an average over training examples?

What mathematical tool underlies all four backpropagation equations?

Sketch the derivation of BP1 using the chain rule.

Topics that are related to Learning Goal - How the Backpropagation Algorithm Works

Learning Goal - Notation

Learning Goal - The Error δ

Learning Goal - The Backpropagation Algorithm

Learning Goal - Synthesis (Feynman Challenges)

Summaries related to Classification - Impurity Measures

COMP 3711

Global politics

Essentials of international relations

Behavioral genetics

Management and organisational behaviour

Follow Up Engels idioom 4/5 H

International Business

Marketing fundamentals

Projectmanagement, A practical Approach-Engl…

Basic Management Accounting for the Hospital…

International business

Organization theory and design