Home / Summaries / COMP 3711 / network-forward-backpropagation

Learning Goal - Synthesis (Feynman Challenges)

Q: A weight in your network has been at nearly zero gradient for many epochs. Using the four BP equations, give two different reasons why this could be happening.

From BP4: ∂C/∂w^l_(jk) = a^(l−1)_k · δ^l_j a^(l−1)_k ≈ 0 (dead input) : the neuron feeding into this weight is barely activating — its output is near zero, so the gradient is near zero regardless of the error. The weight never gets a learning signal. δ^l_j ≈ 0 (saturated or near-optimal output) : from BP1/BP2, δ includes σ'(z) as a factor. If the output neuron is saturated (σ' ≈ 0), δ collapses, and the gradient collapses with it. The neuron is 'stuck'.

Q: You are told that two networks achieve the same test accuracy. Network A uses backpropagation; Network B estimates gradients numerically. Both have 100,000 weights. Compare their training time per epoch.

- Network B (numerical): requires ~100,000 forward passes per gradient update — one per weight - Network A (backpropagation): requires 1 forward pass + 1 backward pass per gradient update The backward pass costs roughly the same as a forward pass. So backpropagation is approximately 50,000× faster per update for this network. At scale (millions of weights), numerical differentiation is completely infeasible; backpropagation is what makes training deep networks possible.

Q: Why does the transpose (w^{l+1})^T appear in BP2 — what is it doing geometrically?

In the forward pass, w^(l+1) maps activations forward from layer l to layer l+1. During the backward pass, we need to distribute the error from layer l+1 back to layer l — the reverse direction. (w^(l+1))^T is the operation that 'reverses' the weight matrix. Geometrically, if w^(l+1) projects a vector from space l into space l+1, then (w^(l+1))^T projects a vector from space l+1 back into space l — sending the error signal backward through the same connections.

4 important questions on Learning Goal - Synthesis (Feynman Challenges)

A weight in your network has been at nearly zero gradient for many epochs. Using the four BP equations, give two different reasons why this could be happening.

From BP4: ∂C/∂w^l_(jk) = a^(l−1)_k · δ^l_j

a^(l−1)_k ≈ 0 (dead input): the neuron feeding into this weight is barely activating — its output is near zero, so the gradient is near zero regardless of the error. The weight never gets a learning signal.
δ^l_j ≈ 0 (saturated or near-optimal output): from BP1/BP2, δ includes σ'(z) as a factor. If the output neuron is saturated (σ' ≈ 0), δ collapses, and the gradient collapses with it. The neuron is "stuck".

You are told that two networks achieve the same test accuracy. Network A uses backpropagation; Network B estimates gradients numerically. Both have 100,000 weights. Compare their training time per epoch.

- Network B (numerical): requires ~100,000 forward passes per gradient update — one per weight
- Network A (backpropagation): requires 1 forward pass + 1 backward pass per gradient update

The backward pass costs roughly the same as a forward pass. So backpropagation is approximately 50,000× faster per update for this network. At scale (millions of weights), numerical differentiation is completely infeasible; backpropagation is what makes training deep networks possible.

Why does the transpose (w^{l+1})^T appear in BP2 — what is it doing geometrically?

In the forward pass, w^(l+1) maps activations forward from layer l to layer l+1. During the backward pass, we need to distribute the error from layer l+1 back to layer l — the reverse direction.
(w^(l+1))^T is the operation that "reverses" the weight matrix. Geometrically, if w^(l+1) projects a vector from space l into space l+1, then (w^(l+1))^T projects a vector from space l+1 back into space l — sending the error signal backward through the same connections.

Nielsen says backpropagation gives us "insight into how changing the weights and biases changes the overall behaviour" of the network. What does this mean concretely?

BP3 (∂C/∂b = δ) and BP4 (∂C/∂w = a^(l−1) · δ) give an exact formula for how sensitive the cost is to every parameter. Concretely:

A weight with a large positive gradient: increasing it raises the cost — decrease it
A weight with a large negative gradient: increasing it lowers the cost — increase it
A weight with near-zero gradient: changing it barely affects the cost — it is either near-optimal or stuck (saturated/dead)

This tells you not just which direction to move each weight, but also which weights matter most to the current prediction error.

The question on the page originate from the summary of the following study material:

COMP 3711

View summary

A unique study and practice tool
Never study anything twice again
Get the grades you hope for
100% sure, 100% understanding

Remember faster, study better. Scientifically proven.

Topics that are related to Learning Goal - Synthesis (Feynman Challenges)

Learning Goal - How the Backpropagation Algorithm Works

Learning Goal - Synthesis (Feynman Challenges)

4 important questions on Learning Goal - Synthesis (Feynman Challenges)

A weight in your network has been at nearly zero gradient for many epochs. Using the four BP equations, give two different reasons why this could be happening.

You are told that two networks achieve the same test accuracy. Network A uses backpropagation; Network B estimates gradients numerically. Both have 100,000 weights. Compare their training time per epoch.

Why does the transpose (w^{l+1})^T appear in BP2 — what is it doing geometrically?

Nielsen says backpropagation gives us "insight into how changing the weights and biases changes the overall behaviour" of the network. What does this mean concretely?

Topics that are related to Learning Goal - Synthesis (Feynman Challenges)

Learning Goal - How the Backpropagation Algorithm Works

Summaries related to Classification - Impurity Measures

COMP 3711

Global politics

Essentials of international relations

Behavioral genetics

Management and organisational behaviour

Follow Up Engels idioom 4/5 H

International Business

Marketing fundamentals

Projectmanagement, A practical Approach-Engl…

Basic Management Accounting for the Hospital…

International business

Organization theory and design