Learning Goal - The Backpropagation Algorithm
3 important questions on Learning Goal - The Backpropagation Algorithm
Write out the five steps of the backpropagation algorithm in order.
- Input
- Set the activation of the input layer: a¹
- Feedforward
- For each layer l = 2, 3, …, L:
- Compute: zˡ = wˡ · a^(l−1) + bˡ
- Compute: aˡ = σ(zˡ)
- Save both zˡ and aˡ at every layer
- Compute: zˡ = wˡ · a^(l−1) + bˡ
- For each layer l = 2, 3, …, L:
- Output error
- Compute δᴸ using BP1
- Backpropagate
- For each layer l = L−1, L−2, …, 2:
- Compute δˡ using BP2
- For each layer l = L−1, L−2, …, 2:
- Output gradient
- Use BP3 and BP4 to compute:
- ∂C / ∂bˡⱼ
- ∂C / ∂wˡⱼₖ
- ∂C / ∂bˡⱼ
- Do this for every layer
- Use BP3 and BP4 to compute:
Why must zˡ and aˡ be saved during the forward pass?
The backward pass requires:
Key idea:
- σ′(zˡ) — used in BP1 and BP2
- This requires the saved zˡ values
- a^(l−1)_k — used in BP4
- This requires the saved activations from the previous layer
- This requires the saved activations from the previous layer
Key idea:
- If these values (zˡ and activations) are not saved during the forward pass,
the backward pass would need to recompute them - This would roughly double the amount of computation needed
What does "one forward pass + one backward pass" mean in terms of computational cost?
- Forward pass: propagate activations from input to output — O(w) where w = number of weights
- Backward pass: propagate deltas from output back to input — also O(w)
Total: 2 * O(w) per training example. Compare to numerical differentiation which requires O(w^2) — one forward pass per weight. Backpropagation is asymptotically much faster.
- Backward pass: propagate deltas from output back to input — also O(w)
Total: 2 * O(w) per training example. Compare to numerical differentiation which requires O(w^2) — one forward pass per weight. Backpropagation is asymptotically much faster.
The question on the page originate from the summary of the following study material:
- A unique study and practice tool
- Never study anything twice again
- Get the grades you hope for
- 100% sure, 100% understanding
Remember faster, study better. Scientifically proven.
