Learning Goal - The Error δ
3 important questions on Learning Goal - The Error δ
How is the "error" δ^l_j defined? What does it actually measure?
δ^l_j ≡ ∂C/∂z^l_j
Despite being called "error," δ^l_j is not prediction error in the everyday sense. It measures the sensitivity of the cost function to the weighted input z^l_j — how much the cost changes if you perturb that neuron's pre-activation.
Despite being called "error," δ^l_j is not prediction error in the everyday sense. It measures the sensitivity of the cost function to the weighted input z^l_j — how much the cost changes if you perturb that neuron's pre-activation.
Explain the "demon" analogy for δ^l_j. What happens when δ is large vs. small?
Imagine a demon sitting at neuron j in layer l who can perturb the weighted input by Δz^l_j to change the cost:
- Large |δ^l_j|: the demon can make a big dent in the cost — this neuron's weighted input matters a lot. The network will adjust its weights heavily.
- Near-zero δ^l_j: the demon can barely change the cost — this neuron is already near-optimal. The network will barely adjust the weights connected to it.
Why is δ defined with respect to z (the weighted input) rather than a (the activation)?
Defining error via z rather than a makes the algebra of backpropagation much cleaner. The sigmoid σ always sits as a neat factor σ'(z^l) in all four equations. If δ were defined via a, the chain rule would produce less symmetric, harder-to-manipulate expressions.
The question on the page originate from the summary of the following study material:
- A unique study and practice tool
- Never study anything twice again
- Get the grades you hope for
- 100% sure, 100% understanding
Remember faster, study better. Scientifically proven.
