Home / Summaries / COMP 3711 / learning-cross-entropy-quadratic

Learning Goal - Cross-Entropy Cost Function

Q: When should you use cross-entropy instead of quadratic cost?

Almost always for classification with sigmoid output neurons . Cross-entropy is the default choice because: 1. It avoids the learning slowdown from saturation 2. Its gradient has a clean probabilistic interpretation 3. It remains positive and well-behaved throughout training The quadratic cost is only preferable in narrow cases (e.g., regression with linear output neurons, where there is no sigmoid and no saturation issue).

5 important questions on Learning Goal - Cross-Entropy Cost Function

What is the learning slowdown problem with the quadratic cost function?

The weight update rule for the quadratic cost contains σ'(z) as a factor:
∂C/∂w ∝ σ'(z) · (a − y)
When a neuron is badly wrong (output saturated near 0 or 1), the sigmoid is nearly flat — σ'(z) ≈ 0 — which drives the gradient to near zero. The network barely updates its weights even though its output is very wrong.
This is the learning slowdown: worst performance → slowest learning.

Write the cross-entropy cost function. What makes it different from the quadratic cost?

C = −(1/n) Σₓ [ y · ln(a) + (1−y) · ln(1−a) ]
Where a = σ(z) is the network's output and y is the target.
Key difference: it is always non-negative and approaches zero as the network learns. Crucially, its gradient does not contain σ'(z), so it avoids the learning slowdown.

Why does cross-entropy eliminate the σ'(z) slowdown? Walk through the algebra intuition.

Differentiating cross-entropy with respect to wⱼ via the chain rule produces a σ(z)(1−σ(z)) term. Since σ'(z) = σ(z)(1−σ(z)), this term cancels the σ'(z) that would otherwise appear:
∂C/∂wⱼ = (1/n) Σₓ xⱼ · (σ(z) − y)
The learning rate is now controlled solely by the error (σ(z) − y): bigger mistake → faster learning. The flat regions of the sigmoid no longer slow things down.

Cross-entropy learning mirrors human learning. How?

Humans tend to learn fastest from their biggest mistakes — a decisive error triggers more reflection and adjustment than a small slip. Cross-entropy replicates this: the larger the output error (σ(z) − y), the larger the gradient, and the faster the weights update. Quadratic cost does the opposite — the biggest mistakes (saturated neurons) cause the slowest learning.

When should you use cross-entropy instead of quadratic cost?

Almost always for classification with sigmoid output neurons. Cross-entropy is the default choice because:
1. It avoids the learning slowdown from saturation
2. Its gradient has a clean probabilistic interpretation
3. It remains positive and well-behaved throughout training

The quadratic cost is only preferable in narrow cases (e.g., regression with linear output neurons, where there is no sigmoid and no saturation issue).

The question on the page originate from the summary of the following study material:

COMP 3711

View summary