Home / Summaries / COMP 3711 / softmax-output-outputs

Learning Goal - Softmax and Log-Likelihood

Q: What is a softmax output layer? Write the activation formula.

A softmax layer defines the activation of each output neuron as: a^L_j = e^(z^L_j) / Σₖ e^(z^L_k) Each output is the exponentiated weighted input normalised by the sum of all exponentiated weighted inputs. The denominator ensures all outputs sum to 1.

Q: Why are softmax outputs interpreted as probabilities?

Two properties guarantee a valid probability distribution: Positivity : e^z > 0 for any z, so all outputs are positive Sum to 1 : the denominator normalises the outputs so Σⱼ a^L_j = 1 So softmax output a^L_j = the network's estimated probability that the input belongs to class j.

Q: What is the key difference between sigmoid + cross-entropy vs. softmax + log-likelihood?

Sigmoid + Cross-entropy: Output range: each output independently between 0 and 1 Interpretation: independent probabilities per class Best for: multi-label problems (multiple correct classes at once) Softmax + Log-likelihood: Output range: all outputs sum to exactly 1 Interpretation: mutually exclusive class probabilities Best for: single-label classification (only one correct class) Both eliminate the learning slowdown. Softmax is preferred when classes are mutually exclusive and you want a proper probability distribution.

4 important questions on Learning Goal - Softmax and Log-Likelihood

What is a softmax output layer? Write the activation formula.

A softmax layer defines the activation of each output neuron as:
a^L_j = e^(z^L_j) / Σₖ e^(z^L_k)
Each output is the exponentiated weighted input normalised by the sum of all exponentiated weighted inputs. The denominator ensures all outputs sum to 1.

Why are softmax outputs interpreted as probabilities?

Two properties guarantee a valid probability distribution:

Positivity: e^z > 0 for any z, so all outputs are positive
Sum to 1: the denominator normalises the outputs so Σⱼ a^L_j = 1

So softmax output a^L_j = the network's estimated probability that the input belongs to class j.

What is the log-likelihood cost, and why does it work well with softmax?

C ≡ −ln(a^L_y)
Where y is the correct class label. The cost = the negative log of the probability assigned to the correct answer.
It works well with softmax because, like cross-entropy with sigmoid, the σ'(z)-like terms cancel during differentiation. The learning rate is again governed purely by the prediction error — no saturation slowdown.