Learning Goal - Alternative Activation Functions
3 important questions on Learning Goal - Alternative Activation Functions
What is tanh, and what advantage does it have over sigmoid?
tanh(z) = (e^z − e^(−z)) / (e^z + e^(−z))
It is a rescaled sigmoid ranging from −1 to +1 (instead of 0 to 1). Its output is centred at 0, which means weight updates are symmetric — gradients can be positive or negative with equal likelihood. This prevents systematic bias in gradient updates that can slow learning with sigmoid neurons. In practice, tanh and sigmoid perform similarly, but tanh is often preferred for hidden layers.
It is a rescaled sigmoid ranging from −1 to +1 (instead of 0 to 1). Its output is centred at 0, which means weight updates are symmetric — gradients can be positive or negative with equal likelihood. This prevents systematic bias in gradient updates that can slow learning with sigmoid neurons. In practice, tanh and sigmoid perform similarly, but tanh is often preferred for hidden layers.
What is a ReLU neuron? Write its formula and explain why it avoids the saturation problem.
ReLU(z) = max(0, z)
For z > 0: the output is z itself — the gradient is a constant 1. ReLU never saturates for positive inputs; the slope never flattens out. This prevents the vanishing gradient problem in deep networks.
For z ≤ 0: output is 0 (the neuron is "dead" for that input), but this is a different issue from saturation and can be addressed with variants like Leaky ReLU.
For z > 0: the output is z itself — the gradient is a constant 1. ReLU never saturates for positive inputs; the slope never flattens out. This prevents the vanishing gradient problem in deep networks.
For z ≤ 0: output is 0 (the neuron is "dead" for that input), but this is a different issue from saturation and can be addressed with variants like Leaky ReLU.
Compare sigmoid, tanh, and ReLU on three axes: output range, saturation risk, and typical use case.
Here are the key differences as bullet points:
Sigmoid:
Sigmoid:
- Output range: (0, 1)
- Saturation risk: high (both ends)
- Gradient: σ(1−σ)
- Typical use: output layer (classification)
- Output range: (−1, 1)
- Saturation risk: high (both ends)
- Gradient: 1 − tanh²
- Typical use: hidden layers
- Output range: [0, ∞)
- Saturation risk: low (only at z ≤ 0)
- Gradient: 1 for z > 0, 0 for z ≤ 0
- Typical use: hidden layers (deep networks)
The question on the page originate from the summary of the following study material:
- A unique study and practice tool
- Never study anything twice again
- Get the grades you hope for
- 100% sure, 100% understanding
Remember faster, study better. Scientifically proven.
